fix: make local docker uploads replica-independent #104

2026-07-05T17:31:43+10:00

unkinben commented

2026-07-05 17:31:43 +10:00

Why

Chunked blob uploads kept the in-progress session in process memory keyed by upload UUID, so the POST/PATCH/PUT of a single docker push had to land on the same replica. The API runs at minReplicas: 2 with no session affinity (see argocd-apps api-hpa.yaml), so a real push — which streams the layer via PATCH then finalises with PUT — intermittently 404s with BLOB_UPLOAD_UNKNOWN when a chunk hits a replica that never saw the POST. This was flagged when the local docker registry landed (#103).

Changes

Stage chunked uploads in object storage under uploads/<uuid> instead of an in-memory temp file. The UUID travels in the Location URL handed to the client, so any replica reconstructs the staging key with no shared in-process state. Finalise streams the staged bytes plus any trailing PUT body through the CAS in one pass; monolithic uploads are unchanged.
Support DELETE of an in-progress upload (cancel) by dropping its staging object.
Reap abandoned staging objects in the GC (uploads/ older than 24h) via a new S3.ListStaleObjects, so cancelled/interrupted pushes don't leak.

Verification

Split a single push across two instances sharing one Postgres+MinIO: POST→A, PATCH→B, PUT→A finalises with the correct digest, and the blob pulls back byte-identical from both replicas. Config-blob and manifest pushes split the same way succeed; tags/list is correct. (Pre-fix, the cross-replica PATCH 404s.)
scripts/docker-e2e.sh still passes (incl. TestLocalDockerPushPull); unit tests + go vet clean.

## Why Chunked blob uploads kept the in-progress session in **process memory** keyed by upload UUID, so the `POST`/`PATCH`/`PUT` of a single `docker push` had to land on the same replica. The API runs at `minReplicas: 2` with no session affinity (see argocd-apps `api-hpa.yaml`), so a real push — which streams the layer via `PATCH` then finalises with `PUT` — intermittently 404s with `BLOB_UPLOAD_UNKNOWN` when a chunk hits a replica that never saw the `POST`. This was flagged when the local docker registry landed (#103). ## Changes - Stage chunked uploads in object storage under `uploads/<uuid>` instead of an in-memory temp file. The UUID travels in the `Location` URL handed to the client, so any replica reconstructs the staging key with no shared in-process state. Finalise streams the staged bytes plus any trailing `PUT` body through the CAS in one pass; monolithic uploads are unchanged. - Support `DELETE` of an in-progress upload (cancel) by dropping its staging object. - Reap abandoned staging objects in the GC (`uploads/` older than 24h) via a new `S3.ListStaleObjects`, so cancelled/interrupted pushes don't leak. ## Verification - Split a single push across **two instances sharing one Postgres+MinIO**: `POST`→A, `PATCH`→B, `PUT`→A finalises with the correct digest, and the blob pulls back **byte-identical from both** replicas. Config-blob and manifest pushes split the same way succeed; `tags/list` is correct. (Pre-fix, the cross-replica `PATCH` 404s.) - `scripts/docker-e2e.sh` still passes (incl. `TestLocalDockerPushPull`); unit tests + `go vet` clean.

unkinben added 1 commit 2026-07-05 17:31:44 +10:00

fix: make local docker uploads replica-independent

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

f9f78955e6

Chunked blob uploads kept the in-progress session in process memory keyed by
upload UUID, so the POST/PATCH/PUT of a single `docker push` had to land on the
same replica. In production the API runs at minReplicas 2 with no session
affinity, so a real push (which streams the layer via PATCH then finalises with
PUT) intermittently 404s with BLOB_UPLOAD_UNKNOWN when chunks hit a replica that
never saw the POST.

- Stage chunked uploads in object storage under uploads/<uuid> instead of an
  in-memory temp file. The UUID travels in the Location URL handed to the
  client, so any replica reconstructs the staging key with no shared in-process
  state. Finalise streams the staged bytes plus any trailing PUT body through
  the CAS in one pass; monolithic uploads are unchanged.
- Support DELETE of an in-progress upload (cancel) by dropping its staging
  object.
- Reap abandoned staging objects in the GC (uploads/ older than 24h) via a new
  S3 ListStaleObjects, so cancelled/interrupted pushes don't leak.

Verified by splitting a single push across two instances sharing one
Postgres+MinIO: POST->A, PATCH->B, PUT->A finalises and the blob pulls back
byte-identical from both. The dockerised e2e suite still passes.

benvin merged commit 649f89f58b into master

2026-07-05 17:39:49 +10:00

benvin deleted branch benvin/docker-upload-sessions-s3

2026-07-05 17:39:49 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#104