perf: stream proxied artifacts instead of buffering the full body in memory #66

New Issue

2026-07-02T00:20:15+10:00

unkinben commented

2026-07-02 00:20:15 +10:00

internal/proxy/engine.go:187 reads every upstream response with io.ReadAll, hashes it in memory, uploads from memory and serves from memory. Large immutable blobs (Docker layers, RPMs, tarballs, Go module zips) — or several concurrent ones — can OOM the process.

A streaming tempfile-backed CAS already exists (internal/storage/cas.go) and is used by local uploads, but the proxy path bypasses it; Engine.cas is even assigned in NewEngine and never used.

Fix: stream immutable fetches through the CAS (tee to sha256 + S3), only buffer mutable index files (small, need RewriteResponse).

`internal/proxy/engine.go:187` reads every upstream response with `io.ReadAll`, hashes it in memory, uploads from memory and serves from memory. Large immutable blobs (Docker layers, RPMs, tarballs, Go module zips) — or several concurrent ones — can OOM the process. A streaming tempfile-backed CAS already exists (`internal/storage/cas.go`) and is used by local uploads, but the proxy path bypasses it; `Engine.cas` is even assigned in `NewEngine` and never used. Fix: stream immutable fetches through the CAS (tee to sha256 + S3), only buffer mutable index files (small, need RewriteResponse).

unkinben referenced this issue from a commit

2026-07-02 00:49:11 +10:00

perf: stream immutable blobs through the CAS instead of buffering

unkinben referenced a pull request that will close this issue

2026-07-02 00:49:24 +10:00

perf: stream proxied artifacts instead of buffering the full body in memory #94

benvin closed this issue

2026-07-02 21:33:42 +10:00

benvin referenced this issue from a commit

2026-07-02 21:33:43 +10:00

perf: stream proxied artifacts instead of buffering the full body in memory (#94)

Sign in to join this conversation.