Chunked blob uploads kept the in-progress session in process memory keyed by
upload UUID, so the POST/PATCH/PUT of a single `docker push` had to land on the
same replica. In production the API runs at minReplicas 2 with no session
affinity, so a real push (which streams the layer via PATCH then finalises with
PUT) intermittently 404s with BLOB_UPLOAD_UNKNOWN when chunks hit a replica that
never saw the POST.
- Stage chunked uploads in object storage under uploads/<uuid> instead of an
in-memory temp file. The UUID travels in the Location URL handed to the
client, so any replica reconstructs the staging key with no shared in-process
state. Finalise streams the staged bytes plus any trailing PUT body through
the CAS in one pass; monolithic uploads are unchanged.
- Support DELETE of an in-progress upload (cancel) by dropping its staging
object.
- Reap abandoned staging objects in the GC (uploads/ older than 24h) via a new
S3 ListStaleObjects, so cancelled/interrupted pushes don't leak.
Verified by splitting a single push across two instances sharing one
Postgres+MinIO: POST->A, PATCH->B, PUT->A finalises and the blob pulls back
byte-identical from both. The dockerised e2e suite still passes.
Raises statement coverage of the core packages (all of `internal/` except the interactive `tui/`, plus `pkg/`) from **8.7% to 90.1%**.
## Approach
- **Pure-go unit tests** for all providers, virtual mergers, classifier, config, auth, models, and the API client (httptest).
- **Testcontainers-backed** tests (new `internal/testsupport` helper: Postgres/Redis/MinIO, Ryuk disabled) for database, storage, cache, the proxy engine, the GC, and a full-stack `server` test that drives the whole HTTP API. These `t.Skip` when Docker is absent so `go test` still runs locally without it.
## Measuring
```
go test -coverpkg=./internal/...,./pkg/... -coverprofile=cover.out ./internal/... ./pkg/...
grep -v /internal/tui/ cover.out | go tool cover -func=/dev/stdin | tail -1 # 90.1%
```
Run with `-p 1` (containers are heavy).
## Notes
- The interactive `tui/` package and `cmd/main` are excluded from the target per the agreed scope.
- Some defensive error branches are covered via fault injection (closed DB pool, killing MinIO mid-upload).
Reviewed-on: #98
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#71
## Why
`FindOrphanedBlobs` returned any blob not currently referenced. Because CAS dedups (the blob row can exist before its artifact/local_files row is written), a concurrent upload reusing an existing hash could have its S3 object deleted mid-flight by the GC.
## Changes
- `FindOrphanedBlobs` now takes a `minAge` and only returns blobs with `created_at < now()-minAge`.
- The collector passes a 1h `blobGracePeriod`.
## Validation
- `go test ./internal/gc/...` and `make e2e` pass.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #86
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkinben
benvin