First increment toward 90% core-package coverage. Adds Docker-free unit
tests taking these packages to full or near-full coverage:
- provider/npm, provider/alpine, provider/puppet: 100%
- provider/pypi: index generation via a fake FileStore, upload validation,
name parsing, classification, rewrite
- config: defaults, overrides, DSN, invalid port
- auth: basic header with/without credentials
Infra-backed packages (database, storage, cache, proxy engine, api
handlers, server) still need the testcontainers batch to reach 90%.
Fixes#74
## Why
`internal/proxy/circuit.go` implemented and tested a circuit breaker, but nothing ever called it — a repeatedly-failing upstream was still hit on every request.
## Changes
- Construct a `CircuitBreaker` in `NewEngine`.
- In `Engine.Fetch`: short-circuit when the breaker is open (serve stale from the store if present, otherwise return 503), `RecordFailure` on each `UpstreamError`, and `RecordSuccess` on a successful fetch.
## Validation
- `go test ./internal/proxy/` and `make e2e` pass.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #90
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#71
## Why
`FindOrphanedBlobs` returned any blob not currently referenced. Because CAS dedups (the blob row can exist before its artifact/local_files row is written), a concurrent upload reusing an existing hash could have its S3 object deleted mid-flight by the GC.
## Changes
- `FindOrphanedBlobs` now takes a `minAge` and only returns blobs with `created_at < now()-minAge`.
- The collector passes a 1h `blobGracePeriod`.
## Validation
- `go test ./internal/gc/...` and `make e2e` pass.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #86
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`master` does not compile. Three PRs that each built individually combined badly:
- #92 changed `fetchBearerToken` to return `(string, time.Duration, error)` and added `cachedBearerToken` (which hashes the challenge via `sha256Hash`).
- #94 (streaming) removed the now-unused-in-that-PR `sha256Hash` helper and its `crypto/sha256` / `encoding/hex` imports.
- #89 (HEAD) added `headUpstream`, which calls `fetchBearerToken` expecting two return values.
Result on `master`: `internal/proxy/engine.go` fails to build (`assignment mismatch: 2 variables but fetchBearerToken returns 3 values`; `undefined: sha256Hash`).
## Changes
- Re-add the `sha256Hash` helper and its `crypto/sha256` + `encoding/hex` imports.
- Fix the `headUpstream` 401 path to handle `fetchBearerToken`s three return values.
## Validation
- `go build ./...`, `go vet`, and `make e2e` all pass.
Should merge before the other in-flight branches so they rebase onto a compiling `master`.
Reviewed-on: #96
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#67
## Why
The proxy used `http.DefaultClient` for all upstream GET/HEAD and bearer-token requests. It has no timeouts, so a slow or hung upstream holds a goroutine and connection indefinitely.
## Changes
- Add a shared `upstreamClient` (`internal/proxy/httpclient.go`) with dial, TLS-handshake, response-header and idle-connection timeouts, plus connection pooling.
- Deliberately no overall `Client.Timeout`, so large artifact bodies can still stream; total time is bounded by the request context.
- Route all four upstream calls in the engine through it.
## Validation
- `make e2e` passes.
Reviewed-on: #83
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#69
## Why
`getenv` returned the fallback whenever `os.Getenv` was empty, so an intentionally-empty env var could not override a non-empty default.
## Changes
- Use `os.LookupEnv` to distinguish unset from set-but-empty.
## Validation
- `make e2e` passes.
Reviewed-on: #85
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#75
## Why
On a fetch-lock miss, `Engine.Fetch` slept a flat 500ms once, tried the store, and otherwise fell through to fetch upstream unlocked. A cold-cache stampede therefore still hit upstream once per waiter.
## Changes
- Add `waitForStore`, which polls the store every 100ms for up to 5s (stopping on context cancellation) so waiters pick up the lock leaders populated result.
- Only fall through to an upstream fetch if the leader has not populated the store within the wait budget.
## Validation
- `make e2e` passes.
Reviewed-on: #93
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#76
## Why
Every proxied request spawned a goroutine running a 5s-timeout single-row INSERT. Under load this is unbounded goroutines and connection-pool pressure.
## Changes
- Add `database.AccessLogEntry` + `InsertAccessLogBatch` (bulk `COPY`).
- The engine starts one background writer that drains a buffered channel and flushes every 128 entries or 2s.
- `logAccess` is now a non-blocking channel send (drops on full buffer), so the request path never blocks on the DB. Best-effort telemetry: a small tail may be lost on abrupt shutdown.
## Validation
- `make e2e` passes.
Reviewed-on: #91
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#70
## Why
Docker `HEAD` routes mapped to `handleProxy`, which ran a full `Fetch` + `io.Copy` — downloading the entire blob (and fetching upstream on a miss) only for net/http to discard the body. HEAD existence checks (manifests, blobs) are common.
## Changes
- Add `Engine.Head`: answers cached artifacts/indexes from store metadata (no blob download); on a miss issues an upstream `HEAD` (with bearer-token handling) and never caches a body.
- Route `HEAD /v2/{remote}/*` to a dedicated `handleProxyHead` that writes headers only.
- Add e2e tests for HEAD on a blocklisted path (403) and an unknown remote (404).
## Note
`headUpstream` uses `http.DefaultClient` to build cleanly on master; it will pick up the shared timeout-configured client from #67 once that merges.
## Validation
- `make e2e` passes (includes new HEAD tests).
Reviewed-on: #89
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#77
## Why
Each upstream 401 re-ran the full token-endpoint request, even though a single Docker pull triggers many blob/manifest requests sharing one scope.
## Changes
- Add Redis `GetToken`/`SetToken`.
- `fetchBearerToken` now also parses `expires_in` and returns a TTL.
- New `Engine.cachedBearerToken` reuses a cached token keyed by remote + challenge (hashed), caching for `expires_in` minus a safety margin (default 60s when absent).
## Validation
- `make e2e` passes.
Reviewed-on: #92
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#66
## Why
`fetchFromUpstream` read every upstream response with `io.ReadAll`, hashed it in memory, uploaded from memory and served from memory. A single large immutable blob (Docker layer, RPM, tarball, Go module zip) — or several concurrent ones — could OOM the process. The streaming, tempfile-backed CAS already existed but the proxy path bypassed it (and `Engine.cas` was assigned but unused).
## Changes
- Immutable fetches now stream through `CAS.Store` (tempfile -> sha256 -> S3), so memory stays bounded regardless of artifact size, and are served back from the store.
- Mutable indexes stay on the in-memory path (small, and subject to `RewriteResponse`).
- Skipping `RewriteResponse` for immutable content is behaviour-preserving: the proxy path always passes an empty `proxyBaseURL`, under which every providers `RewriteResponse` is a no-op.
- Remove the now-unused in-memory `sha256Hash` helper.
## Validation
- `make e2e` passes.
- Live smoke test against Postgres/Redis/MinIO: proxied a 12 MB blob through a generic remote — fetch #1 `X-Artifact-Source: remote`, fetch #2 `X-Artifact-Source: cache`, both byte-identical (sha256) to the origin.
Reviewed-on: #94
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#73
## Why
`Classifier.Classify` runs on every proxied request and recompiled the Blocklist/Patterns/Immutable/Mutable regex lists each time. Regex compilation is expensive and fully redundant.
## Changes
- Memoise compilation in a `sync.Map` keyed by pattern text (`compileCached`); each distinct pattern compiles once and is reused. Patterns that fail to compile are cached as a typed nil so they are not retried. No invalidation needed since the pattern text is the key.
## Validation
- `go test ./internal/proxy/` and `make e2e` pass.
Reviewed-on: #88
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#72
## Why
`compilePatterns` silently discards any pattern that fails to compile. A typo in a blocklist entry therefore turns a deny rule into a no-op — a fail-open with security impact.
## Changes
- Add `Remote.ValidatePatterns`, which compiles every pattern list (patterns, blocklist, mutable/immutable patterns, ban_tags) and returns an error on the first invalid regex.
- Reject invalid patterns with 400 at remote create and update time.
- Unit test for valid and invalid patterns.
## Validation
- `go test ./pkg/models/` and `make e2e` pass.
Reviewed-on: #87
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#68
## Why
`isNetworkError` type-asserted `err.(*UpstreamError)` directly. If the error is ever wrapped, stale-on-error handling silently stops triggering.
## Changes
- Use `errors.As` to detect `*UpstreamError` through wrapping.
## Validation
- `make e2e` passes.
Reviewed-on: #84
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#78
## Why
`serveFromStore` first called `store.Download` with the bare content hash as the S3 key, which never matches real object keys (`blobs/sha256/<hash>`). Every cached blob serve therefore paid an extra guaranteed-404 round-trip before retrying with the correct `BlobKey`.
## Changes
- Remove the dead first `Download` attempt; go straight to the `BlobKey` lookup, then fall back to the index key.
## Validation
- `make e2e` passes (proxy cache-hit paths exercised end-to-end).
Reviewed-on: #82
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#80
## Why
`make e2e` did not compile against master: `e2e/e2e_test.go` called `server.New(cfg)` but the signature is `New(cfg, version string)`. This blocked all end-to-end validation.
## Changes
- Pass a static `"e2e-test"` version to `server.New` in the e2e bootstrap.
## Validation
- `make e2e` builds and passes (testcontainers: postgres/redis/minio).
Reviewed-on: #81
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Shows total bytes served from cache (instead of upstream) over the last 30 days. Queries `SUM(size_bytes) WHERE cache_hit = TRUE` from access_log.
Reviewed-on: #65
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
When a helm repo base URL includes a path component (e.g. \`stakater.github.io/stakater-charts\`), the merger was extracting the full URL path (\`stakater-charts/reloader-2.2.8.tgz\`) and the proxy then constructed \`base_url/stakater-charts/reloader-2.2.8.tgz\` = double path = 404.
Fix: \`extractPathRelativeToBase()\` strips the shared base path prefix so only the filename portion is used as the proxy path.
Reviewed-on: #64
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The / endpoint was hardcoded to return 3.0.0-dev. Now uses the git tag version set via ldflags at build time.
Reviewed-on: #63
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Problems
1. Docker daemon sends specific Accept headers to negotiate manifest format, but the proxy dropped them — registries defaulted to OCI format, causing "mediaType should be manifest.v2+json not oci.image.index" errors
2. Upstream Content-Type was only used when the provider returned "application/octet-stream" — Docker manifests got the wrong Content-Type
## Fixes
- Forward client Accept header to upstream (both initial request and Bearer token retry)
- Always prefer upstream Content-Type when present
- Fetch signature now accepts variadic clientHeaders for backwards compat
## E2E tested
- DockerHub: redis:7-alpine, alpine:3 — skopeo inspect OK
- GHCR: OCI-only images work with docker pull (GHCR 404s Docker v2 Accept, which is expected)
- Quay: prometheus/node-exporter — skopeo inspect OK
Reviewed-on: #62
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Vite's \`base: /ui\` makes HTML reference \`/ui/assets/...\` but files are at \`/usr/share/nginx/html/assets/\` (no \`ui/\` subdir). The previous \`location /ui { try_files ... }\` couldn't find the files.
Fix: rewrite strips the base path prefix before try_files, so \`/ui/assets/foo.js\` resolves to \`/usr/share/nginx/html/assets/foo.js\`.
Reviewed-on: #61
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Docker Hub (and other registries) return 401 with a `Www-Authenticate: Bearer realm=...` challenge even for public images. The proxy now:
1. Detects 401 + Bearer challenge
2. Parses realm/service/scope from the header
3. Fetches an anonymous token (or authenticated if username/password configured)
4. Retries the original request with the Bearer token
Fixes: `docker pull artifactapi.../dockerhub/library/redis:latest` returning "unauthorized: upstream returned 401"
Reviewed-on: #60
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The woodpecker docker-buildx plugin expects build_args as a YAML map (KEY: VALUE), not a list (- KEY=VALUE). The list format was silently ignored, so BASE_PATH was never passed to the Docker build.
Reviewed-on: #59
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Serves the UI under /ui instead of /. This pairs with the argocd route simplification (argocd-apps#201) where /ui → UI service and everything else → API.
- Vite: `base` set from `BASE_PATH` env var at build time
- React Router: `basename` set from injected `__BASE_PATH__`
- Nginx: location block uses `${BASE_PATH}`, substituted by sed at build
- Dockerfile: `ARG BASE_PATH=/` (default preserves existing behavior)
- Woodpecker: passes `BASE_PATH=/ui` to docker-web build
Tested: assets serve at `/ui/assets/...`, SPA routing works at `/ui/remotes`, etc.
Reviewed-on: #58
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The v3 Go rewrite removed the /v2/ Docker Registry compatibility endpoint. Docker clients need:
- GET/HEAD /v2/ → 200 (registry ping)
- GET/HEAD /v2/{remoteName}/* → proxy to the docker remote
Usage: `docker pull artifactapi.example.com/{remoteName}/image:tag`
Reviewed-on: #57
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Problem
Helm charts like Intel device plugins have download URLs on `github.com` but the chart index is served from `intel.github.io`. The merger rewrites all URLs through the proxy, constructing:
```
https://artifactapi/api/v1/remote/intel-helm/intel/helm-charts/releases/download/...
```
Which proxies to `https://intel.github.io/helm-charts/intel/helm-charts/releases/download/...` — a 404.
## Fix
Compare the download URL host against the remote's base URL host. If they differ, leave the URL as-is so helm downloads directly from the source. Same-host URLs are still rewritten through the proxy.
Also adds `BaseURL` to `MemberIndex` so the merger has the context it needs, and uses the correct `/local/` vs `/remote/` route prefix.
Reviewed-on: #56
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- New `.pre-commit-config.yaml` with standard Go hooks (gofmt, go vet, go mod tidy) plus file hygiene checks (trailing whitespace, end-of-file, yaml, large files, merge conflicts)
- go vet runs as a local hook with `./...` since the dnephin per-file hook doesn't work with Go module layouts
- Woodpecker pre-commit pipeline updated to use `almalinux9-gobuilder` image with `uvx pre-commit run --all-files`
- Pre-commit hooks installed into the repo
Reviewed-on: #55
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- New "Locals" sidebar nav item with list + detail + browse pages
- Remotes page filters out local repos (repo_type=local hidden)
- LocalDetail: simplified view — just name, type, description + "Browse Files" button
- Virtuals: member links resolve to /locals/ or /remotes/ based on repo_type
- Objects page detects context for correct back-navigation
## Test plan
- [ ] Visual check: locals page shows only local repos
- [ ] Remotes page hides local repos
- [ ] Virtual member links point to correct pages
- [ ] Browse files works from local detail page
Reviewed-on: #54
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- Upload RPMs to local repos, metadata parsed async via cavaliergopher/rpm
- Repodata (repomd.xml, primary/filelists/other.xml.gz) generated on-demand from DB — nothing stored in S3
- RPM provider implements LocalUploader, PostUploadHook, and LocalIndexer
- New rpm_metadata table for parsed RPM header data (name, version, deps, etc.)
- New provider interfaces: PostUploadHook, BlobReader, MetadataStore, RPMMetadataReader
## Test plan
- [x] Upload cowsay RPM from epel → async metadata parse confirmed in logs
- [x] repomd.xml generated with correct hashes → primary.xml.gz has correct metadata
- [x] `dnf install` from local repo: download + install successful
- [x] Bad file rejection (.txt → 400), overwrite rejection (409)
Reviewed-on: #53
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
Move package-type-specific local repo logic out of centralized handlers into provider packages via optional Go interfaces.
**New interfaces in `provider` package:**
- \`LocalUploader\`: \`ValidateUpload(filePath) → (storagePath, contentType, error)\` + \`UploadResponse(...)\`
- \`LocalIndexer\`: \`ServeLocalIndex(w, r, files, repoName, path) → bool\` + \`GenerateLocalIndex(ctx, files, repoName, path) → ([]byte, error)\`
- \`FileStore\`: \`ListFilesByPrefix\` + \`ListPackages\` (implemented by database.DB)
**Providers implement these interfaces:**
- PyPI: upload validation (wheel/sdist naming), simple index serving + generation
- Terraform: upload validation (provider zip naming), mirror protocol serving
**Handlers simplified to generic dispatch:**
- \`local.go\`: type-asserts to \`LocalUploader\`, falls back to generic upload
- \`proxy.go\`: type-asserts to \`LocalIndexer\`, falls back to raw file serving
- \`engine.go\`: type-asserts to \`LocalIndexer\` for local virtual members
Adding a new local repo type (e.g. RPM) = implement the interfaces in its provider package. Zero handler changes.
## Test plan
- [x] Build + unit tests pass
- [x] E2E: PyPI local upload → simple index → uv pip install (smoke test after refactor)
Reviewed-on: #52
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- Virtual engine detects local members and generates indexes in-memory
- MemberIndex.RepoType drives correct URL prefix in merged output
- PyPI merger rewrites links to /api/v1/local/ or /api/v1/remote/ appropriately
- Includes local PyPI support (cherry-picked from #50)
## Test plan
- [x] Upload wheel to local PyPI → install from direct local URL
- [x] Create virtual with local + remote → install from virtual URL
- [x] Both paths produce correct absolute download URLs
Reviewed-on: #51
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkinben