A GitHub release scan that hit a transient asset error still saved the release-list ETag, so the failed asset was never retried. Assets that are corrupt, deleted or truncated upstream fail on every retry, so they must not keep the remote failed.
- fail the rpm/deb/alpine scan on transient asset fetch, short read or metadata insert errors, so the ETag is not saved and the backoff retry re-derives the asset
- add `provider.ErrInvalidPackage` for unparseable packages and 404/410/416 asset downloads; such assets are logged and skipped
- decide a ranged fetch holds the whole object from Content-Length or the Content-Range total, or a clean EOF when the total is unknown
- add a NOT VALID cascading FK from the GitHub sync-state tables to `remotes`
Reviewed-on: #135
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
AcquireLock stored a constant value and ReleaseLock was an unconditional DEL, so a holder that outlived its TTL deleted the next holder's lock. This hits Fetch, Evict and gems push/yank.
- store a random owner token with SET NX PX and return it from AcquireLock
- release via a Lua compare-and-delete keyed on that token
- carry the token through Fetch, Evict and Engine.Lock's release func
- release the Fetch lock with an uncancelled context
- make Engine.Lock's busy 503 message generic
Reviewed-on: #156
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Yank hard-deleted a local gem version, so the same name-version-platform could be re-pushed with different bytes and break lockfiles pinning the old checksum. rubygems.org forbids re-pushing a yanked version.
- write a `_yanked/<name>/<full>` tombstone before deleting the gem, quick spec and index line
- reject a push of a tombstoned version with 409
- serialise push and yank of one version on the engine Redis lock; 503 when it stays busy
- drop partial push writes on failure
- return 404 for `_index/` and `_yanked/` paths on gems locals
- encode out-of-range gemspec dates (before 1900) as the epoch instead of wrapping the Time#_dump year field
- extend the ruby:3.3 e2e to yank, re-push different bytes and fail to install the yanked version
Reviewed-on: #155
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Bundler needs one source for internal and public gems, but gems virtuals have no merger, so a local gems repo and rubygems.org cannot be served together.
- merge `info/<gem>` across members (member order wins a duplicate version-platform); pass a single-member gem through verbatim
- regenerate `versions` lines and md5s for gems several members list; union `names`
- 302 `gems/*.gem` and `quick/Marshal.4.8/*` to the member owning that version, via the shared member-path validation
- return 502 on a member error; cache merges in the shared virtual cache; route HEAD on virtuals
- add `docs/gems/virtual.md` and a stock `ruby:3.3` bundler e2e-docker case
Reviewed-on: #154
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
In-house gems have nowhere to publish: gems locals only take raw file uploads, which `gem push`, `gem install` and bundler cannot use.
- accept `gem push` (`POST <repo>/api/v1/gems`, 256MiB cap, 8MiB metadata cap) and `gem yank` (`DELETE <repo>/api/v1/gems/yank`); any API key works
- parse the gemspec from the `.gem`; reject an existing name-version-platform with 409
- serve the compact index (`versions`, `info/<gem>`, `names`), `.gem` files and generated `quick/Marshal.4.8` specs; no legacy `specs.4.8.gz`
- route HEAD on locals (gem probes `versions` with HEAD)
- add `docs/gems/local.md`, a UI snippet and a stock `ruby:3.3` e2e-docker case
Reviewed-on: #153
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
npm clients need one registry for internal and public packages, but npm virtuals have no merger, so a local npm repo and npmjs cannot be served together.
- merge packuments across members: union versions (member order wins), first-defined dist-tags minus ones naming missing versions
- serve full and abbreviated (`install-v1`) packuments, scoped names included, via the shared virtual merge cache
- point tarballs at `<virtual>/<member>/<path>` and 302 them to the owning member
- reject an invalid Host with 400; serve virtual and local packuments `no-store` with `Vary: Host, X-Forwarded-Proto, Accept`
- return 502 when a member errors; skip members without the package
- add `docs/npm/virtual.md` and a stock `node:22` e2e-docker case
Reviewed-on: #152
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
NuGet index rewrites embed the client Host and X-Forwarded-Proto into per-request JSON, so a shared cache could store one host's URLs, a forged header could inject content, HEAD skipped the rewrite, and immutable flat-container blobs were re-validated as mutable.
- set `no-store`, `Vary: Host, X-Forwarded-Proto` and the rewritten Content-Length on GET and HEAD
- share `v1.Scheme()` (first X-Forwarded-Proto value, `http`/`https` only) across nuget, cargo, virtual redirects and the terraform registry
- reject non host[:port] Host headers and JSON-escape the injected base URL
- classify `.snupkg`, `readme` and `icon` flat-container blobs as immutable
requires unkin/artifactapi#148 merged first
Reviewed-on: #149
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Remotes answer only GET, so HEAD probes get 405, and there is no RubyGems remote. Synthesized (github_*) and rewritten (npm, helm, pypi, puppet, terraform, nuget) mutable responses need the GET path for HEAD lengths to match.
- add a HEAD route for /api/v1/remote/{name}/*; rewritten and synthesized mutable HEADs go through the GET path
- mark rewriting providers with a ResponseRewriter capability; set Content-Length on proxied GETs
- use the remote's upstream client for upstream HEAD
- add a gems remote type: compact index and specs mutable, .gem and gemspec immutable
- add gems docs, UI snippet and e2e caching case; add HEAD-vs-GET e2e checks
Reviewed-on: #148
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
In-house npm packages have nowhere to publish: npm locals only accept raw file uploads, which `npm publish` and `npm install` cannot use.
- accept `npm publish` (`PUT /api/v1/local/<repo>/<pkg>`, scoped included) on local npm repos, capping the body at 256MiB and verifying the tarball against dist integrity
- write tarball, then manifest, then dist-tags; publishing an existing version returns 409 and leaves tags alone, while a matching leftover tarball from a failed publish is reused so retries complete
- serve packuments with tarball URLs pointing back at the local repo, dropping dist-tags that name unpublished versions
- document the `.npmrc` registry + `_authToken` setup in the UI and `docs/npm/local.md`
- add a stock `node:22` publish/install e2e-docker case
Reviewed-on: #139
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A remote created through the API without `mutable_ttl` was stored as 0, so its mutable paths never refreshed. With `check_mutable`, the ETag expired together with the freshness key, so an expired mutable path was always refetched in full and never revalidated.
- default an omitted `mutable_ttl` to 3600 on create; keep the stored value on update; explicit 0 still means never expire
- keep mutable ETags for `max(10 × mutable_ttl, 7d)`, re-armed on each 304, only when `check_mutable` is set and `mutable_ttl` is non-zero
- drop the stored ETag when a refetch returns none; never store one for immutable paths
- flush a remote's Redis keys on delete
- update docs/common/caching.md
Reviewed-on: #151
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Parallel package-type PRs conflict because every type edits the same shared lines (README tables, provider.go, server.go imports, the package type list, UI maps, the e2e caching table). A new type now only adds files.
- move per-type README sections into docs/<type>/{remote,local,virtual}.md
- split provider.go capability and metadata types into cap_*.go and metadata_*.go
- import providers via internal/provider/all, one file per type
- declare each package type in package_type_<type>.go, registered via init()
- split the e2e caching cases and UI colours/snippets into per-type files
Reviewed-on: #150
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
In-house charts have nowhere to publish: a helm local stores uploads but serves no index.yaml, and the helm virtual merger drops every chart field except name/version/urls.
- add helm local uploads (`PUT .../files/<name>-<version>.tgz`) recording each Chart.yaml in a new `helm_metadata` table; mismatched filenames stay out of the index
- bound chart parsing (50MiB decompressed, 10k entries, 1MiB Chart.yaml)
- generate a deterministic `index.yaml` with sha256 digests and path-escaped relative urls
- keep all chart fields when merging helm virtual indexes, expanding YAML aliases
- add a real-helm e2e-docker test, UI snippet and README section
Reviewed-on: #140
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The rpm virtual's merge cache, singleflight and two-generation retention are hard-wired to RPMRepo, so other merged index types (deb next) can't reuse them.
- add a mergedRepo interface (index + referenced files) implemented by RPMRepo
- key the merge cache by type-scoped string and serve files from both retained generations via cachedFile
- extract parallel fail-closed member fetching into a generic fetchMembers
- add tests for generation retention and per-key isolation
Reviewed-on: #147
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Puppet modules can only be proxied from a remote Forge; there is nowhere to publish internal modules for puppet module install, r10k and g10k.
- add puppet local uploads at <owner>-<module>-<version>.tar.gz, rejecting existing versions
- record each release's metadata.json, md5 and sha256 in a puppet_metadata table, streaming the tarball under size caps
- serve Forge v3 /v3/modules, paginated /v3/releases, /v3/releases/<slug> and /v3/files
- add an e2e-docker case driving real r10k and puppet module install
- document local puppet in README and UI usage snippets
Reviewed-on: #145
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
.NET builds restore from nuget.org directly because artifactapi has no NuGet remote type.
- add `nuget` package type proxying a NuGet v3 feed; `index.json` maps to the upstream service index
- add a `ServeRewriter` provider hook so cached JSON has upstream-origin URLs pointed back at the requesting host per request
- treat `.nupkg`/`.nuspec` as immutable and service, registration and flat container indexes as mutable
- add a `nuget.config` usage snippet to the UI and a nuget e2e caching case
Reviewed-on: #141
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
- add ansible package type proxying the Galaxy v3 API
- rewrite href/links/download_url back to the remote so tarballs flow through it
- forward the query string for providers that opt in, keeping Galaxy pagination working
- add a UI usage snippet and an ansible e2e caching case
Remote and virtual configuration writes are unauthenticated, so anything that can reach the API can repoint or delete repos. Vault identity OIDC tokens give callers a verifiable identity.
- add `ARTIFACTAPI_OIDC_ISSUER`/`ARTIFACTAPI_OIDC_AUDIENCE`; unset issuer keeps auth off
- require a JWKS-verified bearer token on POST/PUT/DELETE of `/api/v2/remotes` and `/api/v2/virtuals`, 401 otherwise
- leave uploads, object deletes, probe, reads and `/v2` open
- discover the issuer lazily with a 10s client timeout; fail fast with 503 for 15s after a failed discovery
- log the verified `sub`; allow `Authorization` in CORS
Reviewed-on: #136
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Releasing a GitHub sync lease always advanced `last_synced_at`, even after a failed scan (e.g. a rate-limit 403). A remote that failed once waited a full `mutable_ttl` before retrying, and a cold remote kept returning 503 until then.
- record scan outcomes in one shared lease helper for github_rpm/deb/alpine
- keep `last_synced_at` and the ETag on failure; retry from 60s with exponential backoff, capped at min(10m, ttl/4)
- honour `Retry-After` / `X-RateLimit-Reset`, clamped to `mutable_ttl`
- add `sync_failures` / `next_retry_at` columns (migration 0002)
Reviewed-on: #133
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Gitea's registry rejects blob HEADs with 401, so the v3.13.0 tag pipeline cannot push. Release images move to the in-cluster artifactapi docker-internal registry, which trusts the runner without credentials.
- push docker-api to docker-internal/artifactapi and docker-web to docker-internal/artifactapi-ui
- use the CA-baked docker-internal/plugin-docker-buildx image
- drop the droneci credentials
- set kubernetes resources and service account on both steps
Reviewed-on: #134
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Virtual repos only merge helm and pypi, so several rpm repos (e.g. many github_rpm remotes) cannot be served as one yum repo.
- merge member primary/filelists/other into one repodata set; member order wins duplicate NEVRAs
- read member repodata through local, github_rpm and proxied remote paths
- prefix package locations (incl. xml:base under a member upstream) with the member name and 302 them to the member route
- reject absolute and dot-segment member paths; escape the redirect and keep its query
- return 502 when any member's repodata is unavailable
- reuse each virtual's merge for 60s behind singleflight; serve data files from its current and previous merge (per replica)
- add merger/engine unit tests and a dockerised dnf e2e case
Reviewed-on: #131
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Rust crate builds (rpmbuilder CI) fetch from crates.io directly because artifactapi has no Cargo remote type.
- add `cargo` package type proxying the sparse registry protocol (RFC 2789)
- synthesize `config.json` so `dl` points crate downloads back at the remote
- treat index files as mutable and `crates/*/*.crate` as immutable, fetched from static.crates.io for crates.io
- add a `~/.cargo/config.toml` usage snippet to the UI and a cargo e2e caching case
Reviewed-on: #132
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
ObjectsHandler.Routes now takes an Evictor, but TestObjectsListPrefix still calls it with no arguments, so the api/v2 test package fails to compile on master.
- pass the existing fakeEvictor to Routes in objects_test.go
Reviewed-on: #130
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
`GET /api/v2/remotes/{name}/objects` dropped the `prefix` query parameter, so a filtered listing returned the first page of every artifact in the remote.
- add a `prefix` argument to `ListArtifacts` that matches path prefixes
- pass the `prefix` query parameter through from the objects list handler
Reviewed-on: #129
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Evicting a remote object only deleted its Postgres row. The S3 index and the Redis TTL/ETag keys survived, so stale mirror metadata such as EPEL repodata kept being served.
- evict the artifact row, S3 index and Redis keys via `Engine.Evict`
- evict a directory with `<dir>/*`; other wildcards return 400, unknown remotes 404
- wait on the per-path fetch lock for single-path evicts; return 503 on timeout or a Redis error
- wildcard evicts take no lock; a fetch already in flight may re-cache its path
- return S3 list errors from `DeletePrefix` instead of deleting nothing
Reviewed-on: #128
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Packer-built almalinux9-gobuilder is retiring. Moves test and
pre-commit steps to artifactapi docker-internal/gobuilder 0.1.2-alma9
(estate CA, go1.26.7, go-cache-plugin baked in).
- Repoint both steps at the new image, quoted (has a colon)
- Drop curl/sha256sum bootstrap for go-cache-plugin; it's on PATH now
- Set GOCACHEPROG as a static env var instead of a conditional export
- Verified build/test still pass with unreachable/invalid S3 creds —
best-effort degrade is a plugin runtime property, unaffected
Reviewed-on: #127
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Every pipeline run recompiled the whole tree from scratch; the shared gocache S3 bucket already holds those objects.
- Bootstrap the go-cache-plugin GOCACHEPROG in the test and pre-commit steps, keyed `ci-artifactapi`.
- Best-effort: a failed plugin fetch leaves GOCACHEPROG unset, and S3 errors degrade to cache misses, so an outage never fails a build.
- Run tests on the gobuilder image, which trusts the internal CA the S3 endpoint presents.
Container builds still compile uncached; plumbing the cache into the Dockerfile is a follow-up.
Reviewed-on: #126
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The Postgres schema was a ~180-line DDL blob inlined in `internal/database/postgres.go` and re-executed on every start. It could only grow — each change appended another `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` — and nothing recorded what had ever been applied. `golib/pg` already owns that mechanic for the estate (arrproxy#10), so artifactapi keeps the SQL and hands over the machinery.
## How
- Depend on `git.unkin.net/unkin/golib` v0.1.0.
- Move the DDL **verbatim** into `migrations/0001_init.sql`, embedded by the new `migrations` package, and build the pool with `pg.NewMigrated` (`LockName: "artifactapi-migrations"`). The runner takes a cluster-wide advisory lock the old blob never took, so replicas starting together queue instead of racing each other through the DDL.
- **No-op on the live database.** Production has the schema but no `schema_migrations`, so its first start on this build re-runs `0001`. Every statement is `IF NOT EXISTS`-guarded, so that run changes nothing and only lands the tracking row. `TestMigratingAnAlreadyPopulatedSchemaIsANoOp` drops the row from a migrated database and asserts exactly that; `TestMigrationsAreIdempotent` keeps future migrations additive and idempotent.
- **Env var contract unchanged.** `config.DatabaseDSN` stays the DSN source rather than `pg.DSNFromEnv`: the variable names match, but golib deliberately has no default user or database name, and artifactapi documents and ships `DBUSER`/`DBNAME` defaults of `artifacts`.
- Drift guards: the embedded set is asserted equal to `migrations/`, and the derived advisory key is pinned so a rename cannot silently let two builds migrate at once.
- Plumb `GOPRIVATE=git.unkin.net` for the first cross-repo Go dependency — exported by the `Makefile`, set in the `Dockerfile` and both woodpecker Go steps, documented in the README.
`database.New`'s signature and every caller are untouched.
Transitive bumps come with golib's `go.mod` (testcontainers-go 0.42 → 0.44, otel, x/crypto, x/net); `go build`, `go vet`, `golangci-lint`, `go test -race ./pkg/... ./internal/...` and `pre-commit run --all-files` are clean.
Reviewed-on: #125
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
`least_conn` mirror selection (`baseURLAttemptOrder`) scaled super-linearly. After rotating the pool by the round-robin cursor it `sort.SliceStable`d with a comparator that called `inflightCounter` on **every comparison** — and each call did a `remoteName+"\x00"+url` concat plus a `sync.Map` `LoadOrStore` with a speculative `new(atomic.Int64)`. So each selection cost O(n·log n) map lookups + allocations, all on the cache-miss/upstream path.
## How
Snapshot each mirror's in-flight count **once**, then sort the snapshot — O(n) map loads, zero comparator allocations.
- Add read-only `inflightCount(name, url) int64`: plain `sync.Map` `Load`, returns 0 when the gauge is absent (no `LoadOrStore`, no speculative allocation).
- `least_conn` branch builds a `{url, count}` snapshot via one `inflightCount` per rotated URL, `sort.SliceStable` by `count` ascending, then extracts the URLs.
- `beginAttempt`/`endAttempt` keep the create-on-write `inflightCounter` path — they legitimately need to create the gauge.
## Numbers (`BenchmarkBaseURLAttemptOrder_LeastConn`, Ryzen 7 4700U, best of 3)
| pool | before ns/op | after ns/op | before allocs | after allocs | before B/op | after B/op |
|------|-------------:|------------:|--------------:|-------------:|------------:|-----------:|
| 3 | ~2516 | ~1492 | 22 | 8 | 474 | 296 |
| 8 | ~14647 | ~3319 | 131–132 | 8 | 2688 | 568 |
Allocs are now **constant** regardless of pool size; pool-8 is ~4.8x faster with ~16x fewer allocations.
## Behavior
Unchanged: least-loaded first, RR rotation as the stable tie-break, `round_robin` and single-URL paths untouched. Pure internal optimization — no API/schema/DB change. Added a multi-mirror tie-break test asserting all-equal load yields the RR rotation; `make test` (`-race`) green, vet/fmt clean.
Reviewed-on: #124
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
We added a mirror load-balancing strategy (round_robin / least_conn) in #122 and need hard numbers on whether mirror *selection* adds meaningful request latency. This PR adds microbenchmarks that isolate the selection cost (no network/DB/redis) and commits the results as a permanent regression guard.
## How
`internal/proxy/selection_bench_test.go` builds a zero-value `Engine` (same setup as `leastconn_test.go` / `multibaseurl_test.go`) and calls the selection functions directly:
- `BenchmarkBaseURLAttemptOrder_SingleURL` — the `len<=1` early-return no-op path
- `BenchmarkBaseURLAttemptOrder_RoundRobin` — pools of 3 and 8
- `BenchmarkBaseURLAttemptOrder_LeastConn` — pools of 3 and 8, with in-flight skew preloaded on the gauges
- `BenchmarkBeginEndAttempt` — the gauge inc/dec pair
- `_Parallel` (`RunParallel`) variants of RR / least_conn / begin-end to surface sync.Map + atomic contention
Run: `go test -run=^$ -bench='BaseURLAttemptOrder|BeginEndAttempt' -benchmem -benchtime=1s -count=6 -cpu=8 ./internal/proxy/`
`internal/proxy/BENCHMARKS.md` has the median-of-6 table (Ryzen 7 4700U, go1.26.5) plus the raw runs.
## Headline numbers (median of 6, ns/op — sequential | parallel@8)
| selection | seq ns/op | parallel ns/op | allocs/op |
|---|---:|---:|---:|
| single-URL | ~133 | — | 1 |
| round_robin (3) | ~462 | ~67 | 4 |
| round_robin (8) | ~682 | ~130 | 4 |
| least_conn (3) | ~2690 | ~292 | 22 |
| least_conn (8) | ~15200 | ~1673 | 132 |
| beginAttempt+endAttempt | ~514 | ~72 | 4 |
**Key caveat:** `baseURLAttemptOrder` is only called from `headUpstream` / `fetchFromUpstream` / `checkUpstream` — the cache-miss/upstream path. A cache hit returns `Source: "cache"` before any selection runs, so the **cache-hit hot path pays zero** selection cost regardless of strategy.
**Verdict:** even the worst case (least_conn across 8 mirrors, ~15 µs) is <1% of the multi-millisecond upstream round-trip it accompanies; round_robin (~0.5 µs) is negligible. The strategy adds no meaningful latency. least_conn's cost scales super-linearly because the `sort.SliceStable` comparator re-derives each mirror's gauge via `inflightCounter` (string-concat key + speculative `new(atomic.Int64)`) O(n·log n) times — a possible future micro-opt (snapshot loads before sorting), out of scope here.
Reviewed-on: #123
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The mirrorlist (PR #121) always load-balances round-robin. Round-robin is oblivious to how busy each mirror is, so a slow or saturated mirror keeps getting its fair share of new requests. This adds an opt-in **least-connections** strategy that favors the mirror currently handling the fewest in-flight requests, steering new work toward idle mirrors. Round-robin stays the default, so existing remotes are unchanged.
## How
- **Model**: `models.Remote` gains `mirror_strategy` (`round_robin` default/empty, or `least_conn`). `ValidateMirrorStrategy` checks the enum and requires a non-empty mirrorlist for `least_conn`. Empty behaves as `round_robin` for back-compat.
- **DB**: additive `mirror_strategy TEXT NOT NULL DEFAULT 'round_robin'` column (CREATE TABLE + `ADD COLUMN IF NOT EXISTS`), wired through remoteCols/scanRemote/CreateRemote/UpdateRemote; empty normalized to `round_robin` on write.
- **Engine**: a per-remote, per-pool-URL atomic in-flight gauge is incremented around each upstream call (head/fetch/checkUpstream). For a `least_conn` remote the attempt order starts with the least-loaded pool URL (ties broken by the existing round-robin rotation). Round-robin path and failover order are unchanged; single-URL pools are a no-op.
- **Tests**: unit tests for least-loaded selection, round-robin default, gauge inc/dec, single-URL no-op, and validation; DB round-trip covers the new column; docker e2e adds a `least_conn` distribution test plus a real `dnf` install through a `least_conn` remote.
Back-compat: unset/empty `mirror_strategy` is `round_robin`, so all existing remotes keep their current behavior.
Reviewed-on: #122
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.
**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).
## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.
## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.
## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.
Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Switching a remote's backend (`base_url`) left artifactapi serving the previously-cached mutable metadata (repodata / Release / APKINDEX) until TTL expiry, so requests could keep pointing at the old upstream. `cache.FlushRemote` already existed but was wired to nothing.
## How
- Inject a `MetadataFlusher` (satisfied by `*cache.Redis`) into `RemotesHandler` via `NewRemotesHandler`; `server.go` passes `s.cache`.
- On update, read the existing remote first, then after a successful DB update flush the remote's cached metadata when `base_url` changed, so the next request re-fetches fresh from the new upstream.
- Keep scope to `base_url` (upstream identity); a flush failure is logged as a warning and does not fail the request since the DB update already landed.
- Add tests: a `base_url` change flushes exactly once, an unchanged `base_url` does not flush, and a flush error still returns 200.
Reviewed-on: #120
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Part of #117. Local generated repodata must be byte-identical across the two no-affinity replicas and across every regeneration, so apt/apk never hit a checksum mismatch between an index's advertised hash and the bytes actually served. This does the deb+apk half (the rpm half landed in #118).
How:
- Derive the deb `Release` `Date:` from the newest persisted `created_at` (RFC1123Z, UTC) instead of `time.Now()`; carry `created_at` through the deb metadata SELECT and `DebMetadata`. An empty repo falls back to the Unix epoch. This also stops `Date:` running ahead of wall clock.
- Pin the apk `APKINDEX` tar header `ModTime` to the Unix epoch instead of the zero-value `time.Time`, so it is never wall-clock derived.
- Give both list queries a genuine total order by adding a `file_path` tiebreak (name/version/arch is not unique).
- Add guard tests: deb generators byte-identical across generations; the `Release` checksum/size matches the served `Packages`/`Packages.gz` bytes (the exact apt invariant); `Date:` pinned to `created_at`; apk index byte-identical and tar `ModTime` pinned to epoch.
Reviewed-on: #119
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Local RPM repos regenerated `repomd.xml` on every request and advertised a `primary.xml.gz` sha256 that drifted every second, because `time.Now().Unix()` was embedded inside the gzipped `primary.xml` (and in `repomd` `<revision>`/`<timestamp>`). The advertised hash therefore never matched the content-addressed `<sha256>-primary.xml.gz` bytes a second later or on the other replica, so `dnf` failed with a checksum mismatch. Part of #117.
How:
- Derives `<time file=>` in `primary.xml` from the persisted `rpm_metadata.created_at` instead of the wall clock; unset timestamps collapse to a fixed `0`.
- Derives `repomd` `<revision>`/`<timestamp>` from the newest package upload time, so `repomd.xml` is byte-identical across replicas and requests.
- Adds `file_path` as a total-order tiebreak to the metadata `ORDER BY`.
- Pins the gzip header (`OS: 255`) so compressed bytes depend only on the payload.
- Adds regression tests: generators are byte-identical across two runs, and the sha256 in `repomd.xml` equals the sha256 of the bytes each `serve*` handler returns.
Reviewed-on: #118
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The alpine local repo (a real apk repo with auto-generated per-arch APKINDEX) and the new github_alpine remote (metadata-only apk index synthesized from a GitHub repo's release .apk assets) shipped without any "How do I use this?" UI guidance, unlike deb/github_deb. This adds the matching client instructions so users can consume and publish to these repos.
## How
- Extend `case 'alpine':` in `UsageInstructions.tsx` to branch on `isLocal` (mirroring rpm/deb):
- LOCAL: "Add the apk repo" snippet appends `${url}/api/v1/local/<name>` to `/etc/apk/repositories` with `apk update/add --allow-untrusted` (served unsigned, parity with rpm gpgcheck=0), plus a "Publish a .apk" snippet uploading to the canonical `<arch>/<name>-<version>.apk` path (APKINDEX carries no filename, so path must match or install 404s).
- REMOTE: existing caching-proxy snippet kept unchanged.
- Add `case 'github_alpine':` (always remote-class): "Add the apk repo (metadata-only, from GitHub releases)" snippet with `--allow-untrusted`; note explains the per-arch APKINDEX is synthesized from release .apk assets and downloads redirect to the backing releases remote.
- No other cases touched. `npm run build` (tsc typecheck + vite) passes; pre-commit passes.
Reviewed-on: #116
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
`github_alpine` is the Alpine/apk analog of the existing `github_deb`/`github_rpm` metadata-only remotes. It lets a plain GitHub-releases repo of `.apk` files be consumed as a real apk repository without artifactapi ever precaching the packages: it scans the repo's releases, derives each package's `.PKGINFO` from a ranged prefix fetch, synthesizes a per-arch `APKINDEX.tar.gz` from the cached metadata, and redirects the actual `.apk` downloads to a backend `releases_remote`. It stacks on the apk-local branch, reusing that work's alpine APKINDEX generator, `.apk`/`.PKGINFO` parser, Q1 pull-checksum, and `AlpineMetadata` store.
## How
- **`pkg/models`**: add `PackageGitHubAlpine` to the enum + validators (and test).
- **`internal/provider/alpine/github.go`**: the `github_alpine` provider. `ServeRemote` serves per-arch `<arch>/APKINDEX.tar.gz` (reusing `generateAPKIndex` over arch-filtered `AlpineMetadata` rows, `normalizeIndexPath` for apk's `./` dot-segment), 302-redirects `*.apk` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`, and cold-starts with a 503 + `Retry-After`. `scanWithState` lists releases with ETag/If-None-Match and incrementally derives/prunes. `deriveAsset` does a **ranged GET of just the front of the `.apk`** — the control gzip stream carrying `.PKGINFO` sits near the front — doubling the range on truncation; it parses `.PKGINFO` and computes the `C:` Q1 checksum (`Q1`+base64(sha1(control stream))). `FilePath` = the github-relative asset path so the redirect resolves.
- **`internal/provider/alpine/syncer.go`**: a parallel background `Syncer` (own worker pool, shared rate limiter, deduped queue, DB lease), separate from the deb/rpm syncers.
- **`internal/database/alpine_github_sync.go`** + `github_alpine_sync_state` table: `ListGitHubAlpineRemotes` + Claim/Release per-remote sync lease, kept separate from the deb/rpm tables.
- **`internal/server/server.go`**: construct + `Run` the alpine syncer alongside deb/rpm and register it in the `PackageType→Primer` map (priming on create then flows through the existing generic `remotes.go` path).
The rpm/deb providers, syncers, and tables are untouched — this adds parallel alpine equivalents and reuses shared helpers already present in the alpine package.
## Tests
Mirror the deb github tests: scan derives `.PKGINFO`/Q1 from a ranged prefix of a `testsupport.MinimalApk` served over an httptest range server (no full download); diff/prune; pattern filter; per-arch `ServeRemote` routing (index served per-arch and grouped, dot-segment collapse, `.apk` → 302, cold-start 503, warm 200, canceled-request-serves-cache); DB lease prevents a second replica; prime bypasses the recency window. `go build`/`go vet`/`go mod tidy` clean, `make test` (-race) green, pre-commit green.
Reviewed-on: #115
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The alpine provider only supported remote (proxy) repositories, so there was no way to publish first-party `.apk` packages the way `rpm-local` and `deb-local` already allow. This extends the existing alpine provider into a real apk repository: uploaded `.apk` files are parsed in pure Go and a per-arch `APKINDEX.tar.gz` is generated on demand, at parity with rpm repodata and deb Packages generation. (The metadata-only `github_alpine` type is a separate follow-up and is not part of this PR.)
## How
- Implements `LocalUploader` / `LocalIndexer` / `PostUploadHook` / `PostDeleteHook` on the existing `alpine` provider, leaving the remote proxy methods (`UpstreamURL`/`ContentType`/`AuthHeaders`/`RewriteResponse`/`Classify`) intact.
- Parses the `.apk` (up to three concatenated, independently gzipped tar streams) in pure Go: locates the control stream by its `.PKGINFO` member, reads the `key = value` fields, and computes the apk pull checksum `C:` = `Q1` + base64(sha1(**control gzip stream bytes**)) — the sha1 of the second gzip member, not of the whole file.
- Derives arch from `.PKGINFO` and records download size (`S:` blob size) and installed size (`I:` from `.PKGINFO size`).
- Generates an **unsigned** per-arch `APKINDEX.tar.gz` = gzip(tar(`APKINDEX`)) filtered by requested arch (clients use `--allow-untrusted`, matching rpm `gpgcheck=0` / deb `[trusted=yes]`), applying the same dot-segment normalization as deb so `./<arch>/APKINDEX.tar.gz` resolves. Non-index / `.apk` paths return `false` so the generic file streamer serves the stored blob.
- Adds `AlpineMetadata` plus **separate** `AlpineMetadataStore` / `AlpineMetadataReader` / `AlpineMetadataDeleter` interfaces (type-asserted from the generic hooks) so the shared rpm/deb metadata interfaces and their test doubles are untouched.
- Adds the `alpine_metadata` table (keyed by `repo_name` + `file_path`, per-arch index) and its `Insert`/`Delete`/`List` DB methods.
- Adds `testsupport.MinimalApk`, unit tests (`.PKGINFO` parse, Q1 checksum over the control stream, per-arch filtering, empty-field omission, `./` dot-segment handling, ValidateUpload accept/reject), and a `dockere2e` `TestLocalAlpineIndex`.
## Consumption
`/etc/apk/repositories` line = `<url>/api/v1/local/<name>` (apk appends `/<arch>/APKINDEX.tar.gz`); `apk update --allow-untrusted && apk add --allow-untrusted <pkg>`. Packages live at `/api/v1/local/<name>/<arch>/<file>.apk`.
## Verification
`go build ./...`, `go vet ./...` (incl. `-tags dockere2e`), `go mod tidy` (no change), `make test` (`-race`), and `pre-commit run --all-files` all pass.
Reviewed-on: #114
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The `deb` and `github_deb` repository types had no tailored "How do I use this?" snippets in the UI — they fell through to the generic default (curl-a-file-by-path), which does not tell a user how to actually wire the repo into apt. This adds real, end-to-end-validated apt instructions mirroring how the existing package types (rpm, npm, etc.) are handled.
## How
- Adds a `case 'deb':` to `buildSnippets` that branches on `isLocal`: a flat real apt repo served at `/api/v1/local/<name>/ ./` with `[trusted=yes]` plus a publish-a-.deb snippet for locals, and a caching-proxy `deb <proxy> <suite> <component>` form (upstream-signed, no `[trusted=yes]`) for remotes.
- Adds a `case 'github_deb':` emitting the metadata-only flat form `deb [trusted=yes] <proxy>/ ./`, whose index is synthesized from a GitHub repo's release .deb assets.
- Reuses the existing `url`/`proxy`/`isLocal` helpers and matches the surrounding case style; leaves the rpm and other cases untouched.
Reviewed-on: #113
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
This stacks the Debian/apt analog of `github_rpm` on top of the deb local+remote work (#111). It lets a GitHub repo's `.deb` release assets be consumed as a real apt repository without artifactapi ever precaching whole packages: it derives per-asset control metadata from a ranged prefix fetch, synthesizes a flat apt repo from the cache, and redirects the actual `.deb` downloads to a backend `releases_remote` (the generic github.com remote).
Base is `benvin/deb-local-remote` (stacked) to keep the diff atomic.
## How
- Adds `github_deb` to the package-type enum and validity map.
- Adds the `github_deb` provider mirroring `github_rpm`: `ServeRemote` serves `Packages`/`Packages.gz`/`Release`, returns 404 for `InRelease`/`Release.gpg` (unsigned, consumed via `[trusted=yes]`), and 302-redirects `*.deb` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`; cold-start prime with a retryable 503.
- `deriveAsset` ranged-GETs the front of the `.deb` (an `ar` archive), locates and fully reads `control.tar.*`, and parses the control paragraph — doubling the range if the control member is truncated. The Packages `SHA256` comes from the GitHub asset `digest` when present, else a one-time full stream; `MD5sum` is left unset (apt verifies against SHA256 under `[trusted=yes]`).
- Adds a `github_deb` background Syncer (own worker pool, shared rate limiter, deduped queue) with per-remote DB-lease-gated scans so only one replica scans per window.
- Adds the `github_deb_sync_state` table plus `ListGitHubDebRemotes` / `ClaimGitHubDebSyncLease` / `ReleaseGitHubDebSyncLease` DB helpers, kept separate from the rpm ones.
- Primes `github_deb` remotes on create and runs the deb syncer alongside the rpm one; prime-on-create is routed by package type.
- Reuses the deb apt-index generators and control parser; the Packages generator now skips empty hash lines so a SHA256-only entry is valid.
## Notes / deviations
- **Filename convention:** the `Filename` stored in the Packages index is the **github-relative** asset path (same as rpm's `assetPath`), not `pool/<asset>`. This is required for the `.deb` 302 to `{releases_remote=github}/{path}` to resolve against github.com; it still matches the `*.deb` redirect rule.
- **GitHub client helpers** (releases pagination, ranged GET, auth headers) are duplicated into the deb package rather than shared, because the rpm equivalents are unexported in `package rpm` and the task requires not modifying the rpm provider.
- `go build`, `go vet`, `go mod tidy`, and `make test` (`-race`, incl. the Postgres lease integration tests) all pass; pre-commit clean.
Do not merge — for review.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #112
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Brings Debian/apt to artifactapi with feature parity to the existing rpm support (local + remote), so `.deb` packages can be hosted as a flat apt repo and a Debian/Ubuntu mirror can be cached through the proxy.
- Adds `deb` to the package-type enum and registers a new `internal/provider/deb` provider.
- Classifies `.deb` blobs immutable and the apt index surface (`Packages`, `Release`, `InRelease`, `dists/`, by-hash) mutable so the caching engine revalidates it.
- Parses the `.deb` in pure Go (ar archive to `control.tar.{gz,xz,zst}` to `./control`), storing the raw control stanza plus computed size/md5/sha256 as `deb_metadata`.
- Serves a flat apt repo (`deb [trusted=yes] .../ ./`): generates `Packages`, `Packages.gz` and an unsigned `Release` (returns 404 for `InRelease`/`Release.gpg`), mirroring rpm unsigned repodata / gpgcheck=0 trust model.
- Proxies a remote mirror via `UpstreamURL`/`ContentType`/`AuthHeaders` (HTTP Basic).
- Adds the `deb_metadata` table to `migrate()`, DB access methods, a `MinimalDeb` pure-Go fixture, unit tests, and a `dockere2e` `TestLocalDebRepo`.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #111
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
## Why
Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories. A machine credential usable by a free (non-enterprise) account is needed to lift the request budget to ~5000/hr and to read private-repo release assets.
Builds on the background syncer (#108, now merged to `master`); this diff is the auth changes only.
## How
- Add `internal/githubauth`: a process-wide GitHub credential delivered via env/secret, applied by default to every outbound GitHub request (releases scan, ranged asset-header GETs, and the generic-github byte proxy for private assets).
- Support two modes:
- **PAT** — `GITHUB_TOKEN` sent as `Authorization: Bearer <token>`.
- **GitHub App** — `GITHUB_APP_ID` + `GITHUB_APP_INSTALLATION_ID` + private key (`GITHUB_APP_PRIVATE_KEY` inline PEM or `GITHUB_APP_PRIVATE_KEY_PATH`). Mint a short-lived RS256 JWT with stdlib `crypto/rsa` (no new dependency), exchange it at `POST /app/installations/{id}/access_tokens` for a ~1h installation token, cache it, and single-flight a refresh a few minutes before expiry.
- Inject at the two GitHub call paths: the rpm github provider header builder (releases + ranged fetches) and the generic provider `AuthHeaders` (byte proxy, github.com hosts only; the pre-signed `objects.githubusercontent.com` redirect deliberately gets no Authorization).
- Honor precedence: a remote's own `username`/`password` overrides the server credential; no credential configured stays anonymous (current behavior).
- Fail closed at startup on partial App configuration (e.g. App id without a private key); a token-and-App conflict is also rejected.
- Never persist the credential to the DB, return it from an API, or log it (token-exchange failures never echo the response body).
- Read config via the existing `getenv` convention; document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication.
## Rate limit
Authenticated requests share the syncer's single global limiter — no second limiter is added. A token raises the effective GitHub ceiling (~5000/hr vs ~60/hr), so the limiter defaults stay safe.
## Tests
`internal/githubauth` and `internal/provider/{rpm,generic}`:
- PAT attaches the correct `Authorization` header to releases + asset-header requests.
- App mints a valid RS256 JWT (verified against the app public key), exchanges it at a mocked endpoint, reuses the cached token without re-exchanging, refreshes near expiry, and single-flights concurrent callers.
- Per-remote credential overrides the server credential (rpm + generic).
- No credential → no `Authorization` header, requests still succeed anonymously.
- ETag/304 flow still works with auth attached.
- The credential does not appear in a remote's serialized JSON.
- Config validation: no-config is anonymous; partial App config and token/App conflict both error.
Verified fail-before/pass-after for the injection tests. `gofmt -l`, `go build ./...`, `go vet ./...`, `go test ./...` all clean (26 packages).
Reviewed-on: #109
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.
## How
- Add a single per-process background syncer (started at boot, cleanly stopped
on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the
github provider so every GitHub call (releases list + each ranged asset GET)
acquires a token first.
- Re-check each `github_rpm` remote for new/changed releases on its existing
`mutable_ttl` cadence; derive only new/changed assets incrementally and prune
assets that disappear upstream. Repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call returns
immediately.
- Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives
nothing and is not counted against GitHub's rate limit, so an unchanged repo
is nearly free.
- Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`,
`etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only
for the replica that atomically claims the lease, bounding total GitHub load
to ~once per `mutable_ttl` regardless of replica count; the ETag is shared
through the same row.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
cache, and return a bounded wait then a retryable `503` rather than blocking
on a cold derive.
- Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` /
`GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst
5, 3 workers, 60s tick) and document the syncer in the README.
## Tests
- Unit (httptest, Range/ETag-aware fixture): `304` releases response derives
nothing; incremental derive fetches only the newly added asset; the shared
limiter caps request rate; work-queue enqueues coalesce to one job; prime
enqueues a job; a held lease stops a second replica from scanning; cold-start
serves `503` while warm cache serves `200`.
- DB integration (testcontainers postgres): the real lease SQL — one holder at a
time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
bypasses recency but respects a live lease.
- Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the
background at ~1 req/s (global limiter); `dnf makecache` served fast from the
priming cache (no cold block); clean shutdown mid-scan, no panics.
## Notes
- Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.
Reviewed-on: #108
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Publishing RPMs to GitHub releases is common, but consuming them with `dnf` requires repodata GitHub does not provide, and mirroring every package into a local repo wastes storage and staleness-tracking on artifacts that already have a durable home. This exposes GitHub releases as a first-class RPM source that synthesizes repodata on the fly and **never precaches the packages**.
## What
Add a `github_rpm` remote package type backed by a metadata-only provider.
- Introduce a `RemoteServer` interception hook (the remote-side analog of `LocalIndexer`): `handleProxy` lets a provider fully answer a request before the byte-proxy engine, passing the request-derived proxy base URL and the DB as a `RemoteMetadataStore`.
- Scan a repo's releases via the GitHub API (`base_url` = the releases API root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset filename), and reuse the existing local-rpm repodata generators to emit `repomd.xml`/`primary`/`filelists`/`other`.
- Derive per-asset metadata without precaching: fetch only the RPM header via a ranged GET (retrying with a larger range on a truncated-header parse) for NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from the GitHub asset `digest` when present, else compute it once by streaming.
- Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more often than `mutable_ttl`, pruning assets that disappear upstream.
- Serve each package's `<location>` as the github-relative download path so the client comes back to this remote, which **302-redirects** to the `releases_remote` (an existing generic github.com remote) that streams the actual bytes.
Reuse the existing `releases_remote` field as the redirect target — it already carries exactly this "downloads served by remote X" semantic end to end, so no new schema/model field is needed.
Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns, added idempotently) so both local and `github_rpm` repodata resolve upgrades and conflicts; the local upload path records them too.
## No-precache mechanics
- **Dependency metadata**: always from the ranged header fetch (header precedes payload; `rpm.Read` stops at the payload boundary), giving `dnf` full resolution. Default range 1 MiB, doubling to 16 MiB.
- **Checksum**: prefer the GitHub asset `digest` (no download); fall back to a one-time streamed sha256 only when absent. Header-only "minimal mode" (no deps) is rejected as a default because `dnf` needs accurate provides/requires and a correct pkgid checksum to install.
## Tests
Header-range parsing incl. the retry loop, digest-vs-computed checksum selection, repodata synthesis with the redirect-able `<location href>`, the 302 redirect path (and the guard when `releases_remote` is unset), asset pattern filtering, and stale-asset pruning. `go build`/`vet`/`test` green; pre-commit clean.
## Follow-ups
- `github_apk` / `github_deb` metadata-only remotes (same pattern; not in this PR).
- Terraform provider support for `artifactapi_remote_github_rpm` ships as a separate PR against `terraform-provider-artifactapi` (depends on this API surface).
Reviewed-on: #107
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
A repository detail page in the ArtifactAPI UI showed configuration and stats, but nothing that told a user how to actually *consume* the repo. You had to already know the per-package-type URL scheme (yum baseurl, pip index-url, docker registry host, terraform source address, ...) by hand. This adds an in-page, copy-pasteable "How do I use this?" panel so each repo page tells you exactly how to point a Linux host at it.
## Changes
- Add a `UsageInstructions` component: a collapsible "How do I use this?" panel with monospace code boxes and copy-to-clipboard buttons, styled to match the existing detail-section / badge theme.
- Generate instructions per package type (rpm, pypi, npm, docker, terraform, helm, alpine, goproxy, puppet, generic) and per class:
- **remote** — consume via the caching proxy (`/api/v1/remote/<name>/...`).
- **local** — consume via the real registry endpoint, plus a publish/push example (rpm `PUT .../files/`, docker Registry V2 push, terraform provider upload).
- **virtual** — consume the merged index via `/api/v1/virtual/<name>/...` using the same per-type client config.
- Interpolate the repository's real name into every snippet so it is genuinely copy-pasteable.
- Resolve the instance base URL from `window.location.origin` (the UI is served on the API origin, client `BASE=''`) instead of hardcoding a hostname; falls back to the public host only when `window` is unavailable.
- Render the panel on `RemoteDetail`, `LocalDetail`, and the `Virtuals` member-expand panel.
## Verification
- `npm run build` (tsc typecheck + vite build) passes.
- Rendered the component in headless Chromium against real repo data for rpm remote, rpm local (with publish), docker local (with push), terraform local (HCL `required_providers` + signing note), and a pypi virtual — all snippets render with the correct URLs and theme.
Reviewed-on: #105
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The local-repo object browser renders every file name as inert text, so there is no way to grab a file from the UI even though local rpm repos already serve their contents as real yum repos. Users have to hand-construct URLs. This adds one-click downloads, built as a modular per-repo-type capability so other types can be switched on later with a single map entry.
## Changes
- Adds a `downloadableTypes` capability map (`ui/src/components/downloads.ts`) keyed by `package_type`; each entry builds the direct-download URL for a file.
- Enables `rpm`, pointing at the yum files route `/api/v2/remotes/<repo>/files/<path>` (path-segment-encoded).
- Renders file names in the object browser as `<a href download>` links when the repo's type is downloadable, and as plain text otherwise.
- Fetches the repo's `package_type` on the Objects page as the modularity hook and threads it through the tree rows.
## Notes
Download links are same-origin unauthenticated GETs (the API serves local repos with no token on reads, matching how yum clients fetch), so a bare `href` carries no credentials. Adding a future type is one entry in `downloadableTypes`.
---------
Co-authored-by: Ben Vin <neotheo@gmail.com>
Reviewed-on: #106
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Chunked blob uploads kept the in-progress session in **process memory** keyed by upload UUID, so the `POST`/`PATCH`/`PUT` of a single `docker push` had to land on the same replica. The API runs at `minReplicas: 2` with no session affinity (see argocd-apps `api-hpa.yaml`), so a real push — which streams the layer via `PATCH` then finalises with `PUT` — intermittently 404s with `BLOB_UPLOAD_UNKNOWN` when a chunk hits a replica that never saw the `POST`. This was flagged when the local docker registry landed (#103).
## Changes
- Stage chunked uploads in object storage under `uploads/<uuid>` instead of an in-memory temp file. The UUID travels in the `Location` URL handed to the client, so any replica reconstructs the staging key with no shared in-process state. Finalise streams the staged bytes plus any trailing `PUT` body through the CAS in one pass; monolithic uploads are unchanged.
- Support `DELETE` of an in-progress upload (cancel) by dropping its staging object.
- Reap abandoned staging objects in the GC (`uploads/` older than 24h) via a new `S3.ListStaleObjects`, so cancelled/interrupted pushes don't leak.
## Verification
- Split a single push across **two instances sharing one Postgres+MinIO**: `POST`→A, `PATCH`→B, `PUT`→A finalises with the correct digest, and the blob pulls back **byte-identical from both** replicas. Config-blob and manifest pushes split the same way succeed; `tags/list` is correct. (Pre-fix, the cross-replica `PATCH` 404s.)
- `scripts/docker-e2e.sh` still passes (incl. `TestLocalDockerPushPull`); unit tests + `go vet` clean.
Reviewed-on: #104
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Local `docker` repos had no write path — the `/v2` Docker Registry API only proxied to upstreams. This makes a local docker repo a genuine container registry so `docker push`/`docker pull` (and podman/skopeo/buildah) work against it directly, matching the project principle that a local repo is *the real thing* rather than a mirror.
## Changes
- Implement the Docker Registry HTTP API V2 read/write half for local docker repos: blob uploads (monolithic and chunked POST/PATCH/PUT), manifest push, `tags/list`, and blob/manifest GET/HEAD.
- Store blobs and manifests through the existing content-addressable store; keep a `local_files` reference per (repo, image) so the GC does not reap them. Tags are mutable (`UpsertLocalFile`); digests and blobs are immutable.
- Dispatch `/v2` reads to the local handler for local docker repos and fall through to the upstream proxy otherwise; writes are local-docker only.
- Add `UpsertLocalFile` for mutable tag references.
- Cover the push/pull round-trip with a dockerised e2e test and unit-test the registry path parser. Document the registry in the README.
## Verification
- `scripts/docker-e2e.sh` passes, including the new `TestLocalDockerPushPull`.
- Verified a real end-to-end round-trip with skopeo against a live instance: pushed `hello-world`, pulled it back, loaded it into the docker daemon, and ran it successfully.
Reviewed-on: #103
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Local terraform repos already served the Terraform **network mirror** protocol, but consuming that requires every user to add a `provider_installation { network_mirror }` block to `~/.terraformrc`. A `source = "artifactapi.k8s.../ns/type"` address instead triggers the **provider registry** protocol (service discovery at `/.well-known/terraform.json` + GPG-signed SHA256SUMS), which returned 404 — hence *"does not offer a provider registry."*
Local repos are meant to be the real thing, so this makes a terraform local repo a first-class provider registry: `terraform init` installs from a bare source address with no client config.
## What
- Serve `/.well-known/terraform.json` service discovery and the `providers.v1` endpoints under `/terraform/v1/providers`: `versions`, `download/{os}/{arch}`, `sha256sums`, `sha256sums.sig`.
- Map the Terraform **namespace** segment to the artifactapi **repo name**; locate the provider by **type**. `download_url` points back at the existing `/api/v1/local/...` path.
- Generate `SHA256SUMS` per version and sign it with a GPG key loaded from `TF_SIGNING_KEY_PATH` (optional `TF_SIGNING_KEY_PASSPHRASE`); advertise the public key + key id in the download response. **No key → registry stays disabled (endpoints 404)**, so behaviour is unchanged until the signing secret is present.
- New `internal/tfsign` (key load + detached signing, via `x/crypto/openpgp`) and `internal/api/terraform` (registry handler). Export `ParseProviderZip` for reuse.
- `TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
- README section documenting usage.
## Consumer
```hcl
terraform {
required_providers {
artifactapi = {
source = "artifactapi.k8s.syd1.au.unkin.net/terraform-unkin/artifactapi"
version = "0.1.2"
}
}
}
```
## Tests
- `internal/tfsign`: sign + verify round-trip, disabled/missing-key paths.
- `internal/api/terraform`: dockerised full flow (discovery → versions → download → sha256sums → sig), verifying the signature against the advertised public key.
## Follow-ups (separate PRs)
- **argocd-apps**: mount the signing K8s secret into the api deployment + set `TF_SIGNING_KEY_PATH`. The `/` HTTPRoute already routes `/.well-known` and `/terraform` to the API, so no gateway change is needed.
- Image/version bump once tagged.
## Note
Anchored the `terraform/` gitignore to the repo root (`/terraform/`) so it stops matching `internal/*/terraform/`. This surfaced `internal/provider/terraform/terraform_extra_test.go`, which had been silently untracked — now committed.
Reviewed-on: #102
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The web UI ships as a separate image served under \`/ui\` (built with \`BASE_PATH=/ui\`). Hitting the bare domain (e.g. \`https://artifactapi.k8s.syd1.au.unkin.net/\`) returned the API's JSON identity blob instead of the app, so browsers never landed on the UI.
## Changes
- Redirect \`GET /\` to \`/ui/\` (302 Found).
- Preserve the former root JSON (\`{"name","version"}\`) at \`/version\`, so health/monitoring can still read the running version.
- Update the server integration test to assert the redirect and the \`/version\` payload.
Reviewed-on: #101
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #99.
## Why
Evicting or deleting a local RPM removed the \`local_files\` row but left its \`rpm_metadata\` behind. Since generated repodata is built from \`rpm_metadata\`, \`primary.xml\` kept advertising a package that no longer exists, producing 404s for clients that tried to fetch it.
## Changes
- Add \`PostDeleteHook\` and \`MetadataDeleter\` provider interfaces (symmetric to the existing \`PostUploadHook\`/\`MetadataStore\`), plus a \`DeleteRPMMetadata\` DB method.
- Implement \`AfterDelete\` in the RPM provider to drop the metadata row for the deleted file.
- Route both local delete paths — the new \`evictLocal\` and the existing files handler's \`remove\` — through a shared \`deleteLocalFile\` helper that removes the file then runs the provider's post-delete hook. Non-RPM providers have no hook, so nothing changes for them.
- Cover the cleanup with a dockerised test.
Reviewed-on: #100
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Local repos store uploaded files in the \`local_files\` table, whereas remote/proxy repos cache into the \`artifacts\` table. The shared **Cached Objects** page always queried the artifacts table via \`/api/v2/remotes/{name}/objects\`, so files uploaded to a local repo (e.g. an internal RPM) were fully stored and servable but showed as **0 objects** in the UI.
## Changes
- Add \`ListLocalArtifacts\`, joining \`local_files\` with \`blobs\` and returning \`models.Artifact\`-shaped rows (size from the blob; access/fetch counters zero and timestamps derived from \`created_at\`, since local files track no access).
- Add \`LocalRoutes\` to the objects handler: \`listLocal\` reads \`local_files\`, \`evictLocal\` deletes via \`DeleteLocalFile\`. Extract shared page/per_page parsing into \`pageBounds\`.
- Mount \`/api/v2/locals/{name}/objects\` (GET + DELETE) in the server.
- Add \`listLocalObjects\`/\`evictLocalObject\` to the UI client and route the Objects page to them when viewing a local repo.
- Cover the listing and eviction paths with a dockerised test.
## Notes
Generated \`repodata/*\` files are not listed — they are produced on the fly from \`rpm_metadata\` and never stored in \`local_files\`, which matches how the repo serves them.
Reviewed-on: #99
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Adds a black-box e2e suite that runs against the **built container image** via docker-compose (complementing the in-process `e2e/` testcontainers suite).
## What it does
`make docker-e2e` → `scripts/docker-e2e.sh`: builds the image, brings up the full stack (postgres, redis, minio, artifactapi) plus a static nginx **mock upstream** for hermetic caching, waits for `/health`, runs `go test -tags=dockere2e ./e2e-docker/...`, and tears everything down.
## Coverage
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact for **each of the 10 remote package types** (generic, docker, helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/` index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge across two members.
## Notes
- The artifactapi host port is parameterised (`ARTIFACTAPI_PORT`, default `8000`; the e2e run uses `8001`) so it does not collide with a locally-running instance. This is the only change to the production `docker-compose.yml`.
- Fixtures under `e2e-docker/fixtures/` are real package files (incl. a real RPM so repodata parsing works); a `.gitignore` negation tracks them over the global ignore of those extensions.
## Validation
Ran `make docker-e2e` locally: **all suites pass** against the containerised product.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #97
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Raises statement coverage of the core packages (all of `internal/` except the interactive `tui/`, plus `pkg/`) from **8.7% to 90.1%**.
## Approach
- **Pure-go unit tests** for all providers, virtual mergers, classifier, config, auth, models, and the API client (httptest).
- **Testcontainers-backed** tests (new `internal/testsupport` helper: Postgres/Redis/MinIO, Ryuk disabled) for database, storage, cache, the proxy engine, the GC, and a full-stack `server` test that drives the whole HTTP API. These `t.Skip` when Docker is absent so `go test` still runs locally without it.
## Measuring
```
go test -coverpkg=./internal/...,./pkg/... -coverprofile=cover.out ./internal/... ./pkg/...
grep -v /internal/tui/ cover.out | go tool cover -func=/dev/stdin | tail -1 # 90.1%
```
Run with `-p 1` (containers are heavy).
## Notes
- The interactive `tui/` package and `cmd/main` are excluded from the target per the agreed scope.
- Some defensive error branches are covered via fault injection (closed DB pool, killing MinIO mid-upload).
Reviewed-on: #98
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#74
## Why
`internal/proxy/circuit.go` implemented and tested a circuit breaker, but nothing ever called it — a repeatedly-failing upstream was still hit on every request.
## Changes
- Construct a `CircuitBreaker` in `NewEngine`.
- In `Engine.Fetch`: short-circuit when the breaker is open (serve stale from the store if present, otherwise return 503), `RecordFailure` on each `UpstreamError`, and `RecordSuccess` on a successful fetch.
## Validation
- `go test ./internal/proxy/` and `make e2e` pass.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #90
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#71
## Why
`FindOrphanedBlobs` returned any blob not currently referenced. Because CAS dedups (the blob row can exist before its artifact/local_files row is written), a concurrent upload reusing an existing hash could have its S3 object deleted mid-flight by the GC.
## Changes
- `FindOrphanedBlobs` now takes a `minAge` and only returns blobs with `created_at < now()-minAge`.
- The collector passes a 1h `blobGracePeriod`.
## Validation
- `go test ./internal/gc/...` and `make e2e` pass.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #86
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`master` does not compile. Three PRs that each built individually combined badly:
- #92 changed `fetchBearerToken` to return `(string, time.Duration, error)` and added `cachedBearerToken` (which hashes the challenge via `sha256Hash`).
- #94 (streaming) removed the now-unused-in-that-PR `sha256Hash` helper and its `crypto/sha256` / `encoding/hex` imports.
- #89 (HEAD) added `headUpstream`, which calls `fetchBearerToken` expecting two return values.
Result on `master`: `internal/proxy/engine.go` fails to build (`assignment mismatch: 2 variables but fetchBearerToken returns 3 values`; `undefined: sha256Hash`).
## Changes
- Re-add the `sha256Hash` helper and its `crypto/sha256` + `encoding/hex` imports.
- Fix the `headUpstream` 401 path to handle `fetchBearerToken`s three return values.
## Validation
- `go build ./...`, `go vet`, and `make e2e` all pass.
Should merge before the other in-flight branches so they rebase onto a compiling `master`.
Reviewed-on: #96
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#67
## Why
The proxy used `http.DefaultClient` for all upstream GET/HEAD and bearer-token requests. It has no timeouts, so a slow or hung upstream holds a goroutine and connection indefinitely.
## Changes
- Add a shared `upstreamClient` (`internal/proxy/httpclient.go`) with dial, TLS-handshake, response-header and idle-connection timeouts, plus connection pooling.
- Deliberately no overall `Client.Timeout`, so large artifact bodies can still stream; total time is bounded by the request context.
- Route all four upstream calls in the engine through it.
## Validation
- `make e2e` passes.
Reviewed-on: #83
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#69
## Why
`getenv` returned the fallback whenever `os.Getenv` was empty, so an intentionally-empty env var could not override a non-empty default.
## Changes
- Use `os.LookupEnv` to distinguish unset from set-but-empty.
## Validation
- `make e2e` passes.
Reviewed-on: #85
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#75
## Why
On a fetch-lock miss, `Engine.Fetch` slept a flat 500ms once, tried the store, and otherwise fell through to fetch upstream unlocked. A cold-cache stampede therefore still hit upstream once per waiter.
## Changes
- Add `waitForStore`, which polls the store every 100ms for up to 5s (stopping on context cancellation) so waiters pick up the lock leaders populated result.
- Only fall through to an upstream fetch if the leader has not populated the store within the wait budget.
## Validation
- `make e2e` passes.
Reviewed-on: #93
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#76
## Why
Every proxied request spawned a goroutine running a 5s-timeout single-row INSERT. Under load this is unbounded goroutines and connection-pool pressure.
## Changes
- Add `database.AccessLogEntry` + `InsertAccessLogBatch` (bulk `COPY`).
- The engine starts one background writer that drains a buffered channel and flushes every 128 entries or 2s.
- `logAccess` is now a non-blocking channel send (drops on full buffer), so the request path never blocks on the DB. Best-effort telemetry: a small tail may be lost on abrupt shutdown.
## Validation
- `make e2e` passes.
Reviewed-on: #91
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#70
## Why
Docker `HEAD` routes mapped to `handleProxy`, which ran a full `Fetch` + `io.Copy` — downloading the entire blob (and fetching upstream on a miss) only for net/http to discard the body. HEAD existence checks (manifests, blobs) are common.
## Changes
- Add `Engine.Head`: answers cached artifacts/indexes from store metadata (no blob download); on a miss issues an upstream `HEAD` (with bearer-token handling) and never caches a body.
- Route `HEAD /v2/{remote}/*` to a dedicated `handleProxyHead` that writes headers only.
- Add e2e tests for HEAD on a blocklisted path (403) and an unknown remote (404).
## Note
`headUpstream` uses `http.DefaultClient` to build cleanly on master; it will pick up the shared timeout-configured client from #67 once that merges.
## Validation
- `make e2e` passes (includes new HEAD tests).
Reviewed-on: #89
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#77
## Why
Each upstream 401 re-ran the full token-endpoint request, even though a single Docker pull triggers many blob/manifest requests sharing one scope.
## Changes
- Add Redis `GetToken`/`SetToken`.
- `fetchBearerToken` now also parses `expires_in` and returns a TTL.
- New `Engine.cachedBearerToken` reuses a cached token keyed by remote + challenge (hashed), caching for `expires_in` minus a safety margin (default 60s when absent).
## Validation
- `make e2e` passes.
Reviewed-on: #92
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#66
## Why
`fetchFromUpstream` read every upstream response with `io.ReadAll`, hashed it in memory, uploaded from memory and served from memory. A single large immutable blob (Docker layer, RPM, tarball, Go module zip) — or several concurrent ones — could OOM the process. The streaming, tempfile-backed CAS already existed but the proxy path bypassed it (and `Engine.cas` was assigned but unused).
## Changes
- Immutable fetches now stream through `CAS.Store` (tempfile -> sha256 -> S3), so memory stays bounded regardless of artifact size, and are served back from the store.
- Mutable indexes stay on the in-memory path (small, and subject to `RewriteResponse`).
- Skipping `RewriteResponse` for immutable content is behaviour-preserving: the proxy path always passes an empty `proxyBaseURL`, under which every providers `RewriteResponse` is a no-op.
- Remove the now-unused in-memory `sha256Hash` helper.
## Validation
- `make e2e` passes.
- Live smoke test against Postgres/Redis/MinIO: proxied a 12 MB blob through a generic remote — fetch #1 `X-Artifact-Source: remote`, fetch #2 `X-Artifact-Source: cache`, both byte-identical (sha256) to the origin.
Reviewed-on: #94
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#73
## Why
`Classifier.Classify` runs on every proxied request and recompiled the Blocklist/Patterns/Immutable/Mutable regex lists each time. Regex compilation is expensive and fully redundant.
## Changes
- Memoise compilation in a `sync.Map` keyed by pattern text (`compileCached`); each distinct pattern compiles once and is reused. Patterns that fail to compile are cached as a typed nil so they are not retried. No invalidation needed since the pattern text is the key.
## Validation
- `go test ./internal/proxy/` and `make e2e` pass.
Reviewed-on: #88
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#72
## Why
`compilePatterns` silently discards any pattern that fails to compile. A typo in a blocklist entry therefore turns a deny rule into a no-op — a fail-open with security impact.
## Changes
- Add `Remote.ValidatePatterns`, which compiles every pattern list (patterns, blocklist, mutable/immutable patterns, ban_tags) and returns an error on the first invalid regex.
- Reject invalid patterns with 400 at remote create and update time.
- Unit test for valid and invalid patterns.
## Validation
- `go test ./pkg/models/` and `make e2e` pass.
Reviewed-on: #87
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#68
## Why
`isNetworkError` type-asserted `err.(*UpstreamError)` directly. If the error is ever wrapped, stale-on-error handling silently stops triggering.
## Changes
- Use `errors.As` to detect `*UpstreamError` through wrapping.
## Validation
- `make e2e` passes.
Reviewed-on: #84
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#78
## Why
`serveFromStore` first called `store.Download` with the bare content hash as the S3 key, which never matches real object keys (`blobs/sha256/<hash>`). Every cached blob serve therefore paid an extra guaranteed-404 round-trip before retrying with the correct `BlobKey`.
## Changes
- Remove the dead first `Download` attempt; go straight to the `BlobKey` lookup, then fall back to the index key.
## Validation
- `make e2e` passes (proxy cache-hit paths exercised end-to-end).
Reviewed-on: #82
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#80
## Why
`make e2e` did not compile against master: `e2e/e2e_test.go` called `server.New(cfg)` but the signature is `New(cfg, version string)`. This blocked all end-to-end validation.
## Changes
- Pass a static `"e2e-test"` version to `server.New` in the e2e bootstrap.
## Validation
- `make e2e` builds and passes (testcontainers: postgres/redis/minio).
Reviewed-on: #81
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Shows total bytes served from cache (instead of upstream) over the last 30 days. Queries `SUM(size_bytes) WHERE cache_hit = TRUE` from access_log.
Reviewed-on: #65
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
When a helm repo base URL includes a path component (e.g. \`stakater.github.io/stakater-charts\`), the merger was extracting the full URL path (\`stakater-charts/reloader-2.2.8.tgz\`) and the proxy then constructed \`base_url/stakater-charts/reloader-2.2.8.tgz\` = double path = 404.
Fix: \`extractPathRelativeToBase()\` strips the shared base path prefix so only the filename portion is used as the proxy path.
Reviewed-on: #64
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 08:02:52 +10:00
347 changed files with 31574 additions and 690 deletions
| `GITHUB_SYNC_RATE` | `1` | `github_rpm` syncer global GitHub request rate (req/s), shared across all remotes. `1`/s = 3600/hr, under an authenticated token's ~5000/hr; unauthenticated (~60/hr) relies on ETag/304 |
| `GITHUB_SYNC_BURST` | `5` | Token-bucket burst for the shared limiter |
| `GITHUB_SYNC_POLL_INTERVAL` | `60` | Base scheduler tick in seconds; per-remote cadence is its `mutable_ttl`, enforced by the DB lease |
| `GITHUB_TOKEN` | | Server-level GitHub PAT (fine-grained or classic), sent as `Authorization: Bearer`. Applies to every GitHub request; per-remote creds override it. See [GitHub authentication](#github-authentication) |
| `GITHUB_APP_ID` | | GitHub App id (App auth mode; mutually exclusive with `GITHUB_TOKEN`) |
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
| `ARTIFACTAPI_OIDC_ISSUER` | | OIDC issuer URL (e.g. Vault `https://vault.service.consul:8200/v1/identity/oidc`). When set, `POST /api/v2/remotes`, `POST /api/v2/virtuals` and `PUT`/`DELETE` on `/api/v2/{remotes,virtuals}/{name}` require `Authorization: Bearer <jwt>` verified against the issuer's JWKS. Uploads, object/file deletes, probe, reads and `/v2` stay open. Unset = no auth |
| `ARTIFACTAPI_OIDC_AUDIENCE` | | Comma-separated allowed `aud` values (e.g. the Vault OIDC role's client_id). Required when `ARTIFACTAPI_OIDC_ISSUER` is set |
## Development
@@ -160,6 +220,22 @@ make lint # golangci-lint + go vet
make fmt # gofmt + goimports
```
### `GOPRIVATE`
artifactapi depends on `git.unkin.net/unkin/golib`, which is served by Gitea and
is unknown to `proxy.golang.org` / `sum.golang.org`. Module resolution therefore
needs:
```
export GOPRIVATE=git.unkin.net
```
The `Makefile` exports it for every target, and the `Dockerfile` and the
woodpecker Go steps set it themselves, so `make build|test|lint` and CI work on
a clean checkout. Only bare `go` commands run outside `make` need it in your
shell — set it there (or in your shell profile) rather than with `go env -w`,
| `mutable_ttl` | Seconds a mutable path stays fresh. `0` also means never expires. Omitted on create, it defaults to `3600`; omitted on update, the stored value is kept. |
| `check_mutable` | When a mutable path's `mutable_ttl` expires, sends a conditional `HEAD` (`If-None-Match`) with the stored upstream ETag. A `304` re-arms the TTL and serves the stored copy; anything else refetches in full. The ETag is kept for `max(10 × mutable_ttl, 7 days)`, re-armed on each `304`, so idle paths age out. It is only stored when `check_mutable` is set and `mutable_ttl` is non-zero, and never for immutable paths. |
| `stale_on_error` | On an upstream network error, serve the stored copy instead of failing. |
| `quarantine_enabled` / `quarantine_days` | Stored on the remote but not enforced by the proxy yet. |
| `mirrorlist` / `mirror_strategy` | rpm/deb/alpine only. Upstream pool is `[base_url] + mirrorlist`; `round_robin` (default) rotates the starting mirror, `least_conn` starts at the one with the fewest in-flight requests. Network errors and 5xx fail over to the next mirror; other statuses return as-is. |
## Precedence
The first rule that matches wins:
1.`blocklist` matches: **denied**
2.`patterns` is non-empty and nothing matches: **denied**
3.`immutable_patterns` matches: **immutable**
4.`mutable_patterns` matches: **mutable**
5. the provider's built-in `Classify` says mutable: **mutable**
6. anything else: **immutable**
So `blocklist` beats the allowlist, and `immutable_patterns` beats
`mutable_patterns`. Either user list overrides the built-in rules. `GET` and
`HEAD` both apply this order.
## Matching rules
- Regexes are Go `regexp`, matched **unanchored** (`MatchString`). Add `^` or `$` yourself when you need them.
- They match the path after `/api/v1/remote/<name>/`, without a leading slash, e.g. `9/BaseOS/x86_64/os/repodata/repomd.xml`.
- The query string is not part of the path. It is dropped and not forwarded upstream.
-`create` and `update` reject an invalid regex in `patterns`, `blocklist`, `mutable_patterns`, `immutable_patterns` or `ban_tags` with a 400. Only invalid regexes already stored before that check are skipped at match time.
## Lifecycles
| | Immutable | Mutable |
|---|---|---|
| Stored as | content-addressed blob (sha256), streamed | per-path index object, buffered and rewritten by the provider |
| After expiry | refetched from upstream | revalidated with `If-None-Match` when `check_mutable` is set and an ETag is stored, else refetched |
| Upstream down | `stale_on_error` serves the stored copy and re-arms the TTL; an open circuit breaker always serves stale if a copy exists, else 503 | same |
Concurrent misses on one path share a fetch lock. Waiters poll the store for
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
unkin-agent
unkinben