perf: stream proxied artifacts instead of buffering the full body in memory #94

Merged
benvin merged 1 commits from benvin/stream-immutable-blobs into master 2026-07-02 21:33:42 +10:00

1 Commits

Author SHA1 Message Date
unkinben 2fc9ae6a31 perf: stream immutable blobs through the CAS instead of buffering
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
fetchFromUpstream read every upstream response fully into memory before
storing and serving it, so a large blob (or several concurrent ones) could
OOM the process. Immutable artifacts now stream through the tempfile-backed
CAS (tee to sha256 + S3), keeping memory bounded regardless of size, and
are served back from the store. Mutable indexes are small and may be
rewritten, so they keep the in-memory path. Immutable content is never
rewritten in the proxy path (proxyBaseURL is empty), so skipping
RewriteResponse there is behaviour-preserving.

Refs #66
2026-07-02 00:49:10 +10:00