github_rpm remotes regenerate repomd and all three gz files from every metadata row on each request, which is expensive for large repos under repeated dnf traffic.
- add a per-remote `cache_repodata` toggle (column, model, v2 API, docs)
- cache built repodata per replica, rebuilt via singleflight when the `count(*)`+`max(id)` row fingerprint moves
- store a build only if no newer build has landed; evict idle entries hourly
- map cached-path DB errors like the uncached path: 503 + Retry-After on cancel/deadline, generic 500 otherwise
- check `cacheEmpty` via the fingerprint for all github_rpm remotes, not only cached ones
Reviewed-on: #168
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
github_* remotes sharing a mutable_ttl all become due at the same moment, so many remotes created or synced together burst GitHub at once.
- add per-remote `sync_interval` (seconds) for github_rpm/github_deb/github_alpine remotes
- offset each remote's slots by a stable name-hashed phase so remotes on one interval spread out
- compute slots and the first-sync gate in the lease claim from the DB clock so replicas agree
- wait for the first slot on a scheduled first sync and skip the create-time prime; cold-cache primes stay immediate
- keep the mutable_ttl freshness predicate unchanged when `sync_interval` is unset
- document `sync_interval` in README, rpm docs and UI types
Reviewed-on: #166
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Virtual merges expire after 60s and the next request blocks on every member. One slow member (a large github_rpm remote) makes the re-merge outlast dnf's 30s timeout.
- add `serve_stale` to remotes (model, v2 API, migration 0008, UI, TUI), default false
- serve a flagged member's last good fetch to rpm/deb/alpine merges at once
- refresh it in the background, single-flighted, detached from the request
- block on the member again once its copy is over an hour old, or the flag is off
- prune copies with their merge cache; drop refreshes for evicted entries
Reviewed-on: #167
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Local repo writes (file uploads/deletes, object evictions, native publishes, docker push) are open to anyone who can reach the service; only remote/virtual config is token-gated.
- add `ARTIFACTAPI_AUTH_UPLOADS` (default false) gating every local write path behind the OIDC verifier
- accept the token as Bearer, Token, Basic password (any user) or bare header value on every gated path, including remote/virtual config writes
- serve a `/v2/token` Bearer challenge flow so `docker login` with the token enables push; missing, expired or invalid credentials get an anonymous token so pulls stay open
- document minting a Vault identity token and passing it per client
Reviewed-on: #169
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Remotes store `quarantine_enabled`/`quarantine_days` but the proxy never enforces them, so freshly published (possibly malicious) artifacts are cached immediately.
- refuse uncached immutable paths published under `quarantine_days` ago with a JSON 403
- take publish time from `Last-Modified` capped at now, else a Redis first-seen key; 503 + `Retry-After` when that lookup fails
- reject enabled quarantine with `quarantine_days<1`; backfill stored `<=0` to 3
- default `quarantine_days` to 3 on create and keep it on update
- let goproxy virtual probes skip a quarantining member
Reviewed-on: #170
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Virtual repos cover rpm, deb, npm and others but not alpine, so a local alpine repo and github_alpine remotes cannot be served as one apk repository.
- merge members' `<dir>/APKINDEX.tar.gz` per directory; member order wins duplicate (P, V, A); a member 404 is skipped
- 302 `<dir>/<P>-<V>.apk` downloads to the member whose index listed the package
- serve the merged index unsigned, like local and github_alpine indexes
- cache merges per directory on the shared virtual cache; member errors fail with 502
- add docs/alpine/virtual.md, unit tests and an apk e2e with abuild-built packages
Requires unkin/artifactapi#163.
Reviewed-on: #164
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The generic `github` remote only allowlists URL patterns: every repo needs hand-written mutable patterns for branch archives, and a moved branch re-downloads the whole archive under a ref-named key.
- add `vcs` package type for GitHub or Gitea (picked from `base_url`) tag, commit and release-download paths, cached immutable
- redirect `archive/refs/heads/<branch>` to the commit archive, resolving the branch via the branch API with one shared, per-remote bounded lookup cache reused for `mutable_ttl`
- cache missing branches for at least 60s and map API rate limits to `503` with `Retry-After`
- reject any other path shape, escaping segments and denied redirect targets
- add `docs/vcs/remote.md`, a UI type def and a vcs e2e caching case
Reviewed-on: #162
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Virtual repos merge helm, pypi, rpm, npm, gems, ansible, goproxy and puppet but not deb, so several apt repos (e.g. github_deb remotes plus a local deb) cannot be served as one apt source.
- merge member Packages for flat (`./`) and `dists/<suite>` layouts; member order wins duplicate (Package, Version, Architecture)
- generate an unsigned Release with Acquire-By-Hash, serving indexes by name and by SHA256 from both retained generations
- prefix Filename with the member name and 302 downloads to the member route
- cache merges per suite on the shared virtual cache, capped at 32 suites per virtual (LRU)
- add docs/deb/virtual.md, unit tests and dockerised apt e2e for flat and suite layouts
Reviewed-on: #138
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A puppet remote drops the query string on `/v3/releases`, so `?module=<slug>` reaches the Forge unfiltered and clients get releases of every module instead of the one they asked for.
- forward validated `module`, `limit` (capped at 100) and `offset` on `v3/releases` via `QueryForwarder`
- document the forwarded params and the Forge API `base_url` in `docs/puppet/remote.md`
Reviewed-on: #161
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Clients need one Forge to install internal modules and forgeapi.puppet.com ones with dependencies resolved across both; puppet has no virtual type.
- merge member `/v3/releases?module=` lists newest first, the earlier member winning a duplicate version
- cache merges, and modules no member has, for the merge TTL
- paginate over the merged list with Forge-root-relative links and `file_uri`
- serve `/v3/modules/<slug>` with the highest merged version as `current_release`
- 302 release details and downloads to the owning member; fail closed with 502 on a member error
- add docs/puppet/virtual.md and an e2e-docker case driving real `puppet module install` and r10k
Reviewed-on: #160
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Go clients need one GOPROXY for internal and public modules; goproxy has no virtual type, so a local goproxy repo and proxy.golang.org cannot be served together.
- union member `@v/list`s; compute `@latest` over the union (highest release, else highest prerelease)
- 302 `@latest` and `.info`/`.mod`/`.zip` to the first member with that version, root-relative and `no-store`; probe `.info` for unlisted pseudo-versions
- validate case-encoded module paths and versions
- fail closed with 502 on a member error, skip members answering 404/410
- cache merged lists, keyed on list and version owners, and module misses for the merge TTL
- add `docs/goproxy/virtual.md` and a stock `golang:1.25` e2e-docker case
requires unkin/artifactapi#143
Reviewed-on: #159
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Private Go modules have nowhere to live: goproxy is remote-only, so internal modules need VCS access or a hand-rolled proxy.
- serve local goproxy repos over the GOPROXY protocol (`@v/list`, `@latest`, `.info`/`.mod`/`.zip`) with case-encoded module paths
- accept PUT uploads of `.zip`/`.mod`/`.info` per version; synthesize `.info` when absent
- validate zips with `golang.org/x/mod/zip.CheckZip`, `.mod` module path and `.info` version before storing
- add a ContentValidator hook so providers can check upload bytes
- document publishing in the UI and `docs/goproxy/local.md`; add an e2e-docker case
Reviewed-on: #143
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Clients need one Galaxy server to install both internal collections and galaxy.ansible.com ones; ansible has no virtual type.
- merge member collection version lists in member order, the earlier member winning a duplicate version, cached with the shared virtual merge cache
- paginate limit/offset over the merged list, following remote member pages
- serve version detail from the owning member with href and download_url via the virtual; downloads 302 to the member through the member-redirect validation
- fail closed with 502 on a member error or a member versions list shorter than its count
- add docs/ansible/virtual.md and an e2e-docker case driving real ansible-galaxy
requires unkin/artifactapi#157
Reviewed-on: #158
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Internal Ansible collections can only be installed from galaxy.ansible.com through the remote; there is nowhere to publish them for ansible-galaxy.
- add the Galaxy v3 publish endpoint (multipart, base64 parts) with an import task that is already completed
- read collection_info from MANIFEST.json under size and decompression caps, rejecting escaping paths and links
- serve collection, paginated versions (root-relative links, as Galaxy does), version detail and artifact downloads
- reject an existing namespace.name:version with 409
- add docs/ansible/local.md, a UI snippet and an e2e-docker case driving real ansible-galaxy
requires unkin/artifactapi#137
Reviewed-on: #157
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
ansible-galaxy installs pull collections straight from galaxy.ansible.com because artifactapi has no Galaxy remote type.
- add `ansible` package type proxying the Galaxy v3 API (discovery, collection versions, artifact downloads)
- forward digits-only `limit` (capped at 100) and `offset` on metadata paths, canonicalised; tarball paths forward no query
- rewrite only `href`, `download_url` and `links.*` to root-relative remote paths so tarballs flow through artifactapi
- treat `artifacts/*.tar.gz` as immutable and JSON metadata as mutable
- add `docs/ansible/remote.md`, a UI type def and an ansible e2e caching case
requires unkin/artifactapi#146
Reviewed-on: #137
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Bundler needs one source for internal and public gems, but gems virtuals have no merger, so a local gems repo and rubygems.org cannot be served together.
- merge `info/<gem>` across members (member order wins a duplicate version-platform); pass a single-member gem through verbatim
- regenerate `versions` lines and md5s for gems several members list; union `names`
- 302 `gems/*.gem` and `quick/Marshal.4.8/*` to the member owning that version, via the shared member-path validation
- return 502 on a member error; cache merges in the shared virtual cache; route HEAD on virtuals
- add `docs/gems/virtual.md` and a stock `ruby:3.3` bundler e2e-docker case
Reviewed-on: #154
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
In-house gems have nowhere to publish: gems locals only take raw file uploads, which `gem push`, `gem install` and bundler cannot use.
- accept `gem push` (`POST <repo>/api/v1/gems`, 256MiB cap, 8MiB metadata cap) and `gem yank` (`DELETE <repo>/api/v1/gems/yank`); any API key works
- parse the gemspec from the `.gem`; reject an existing name-version-platform with 409
- serve the compact index (`versions`, `info/<gem>`, `names`), `.gem` files and generated `quick/Marshal.4.8` specs; no legacy `specs.4.8.gz`
- route HEAD on locals (gem probes `versions` with HEAD)
- add `docs/gems/local.md`, a UI snippet and a stock `ruby:3.3` e2e-docker case
Reviewed-on: #153
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
npm clients need one registry for internal and public packages, but npm virtuals have no merger, so a local npm repo and npmjs cannot be served together.
- merge packuments across members: union versions (member order wins), first-defined dist-tags minus ones naming missing versions
- serve full and abbreviated (`install-v1`) packuments, scoped names included, via the shared virtual merge cache
- point tarballs at `<virtual>/<member>/<path>` and 302 them to the owning member
- reject an invalid Host with 400; serve virtual and local packuments `no-store` with `Vary: Host, X-Forwarded-Proto, Accept`
- return 502 when a member errors; skip members without the package
- add `docs/npm/virtual.md` and a stock `node:22` e2e-docker case
Reviewed-on: #152
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
NuGet index rewrites embed the client Host and X-Forwarded-Proto into per-request JSON, so a shared cache could store one host's URLs, a forged header could inject content, HEAD skipped the rewrite, and immutable flat-container blobs were re-validated as mutable.
- set `no-store`, `Vary: Host, X-Forwarded-Proto` and the rewritten Content-Length on GET and HEAD
- share `v1.Scheme()` (first X-Forwarded-Proto value, `http`/`https` only) across nuget, cargo, virtual redirects and the terraform registry
- reject non host[:port] Host headers and JSON-escape the injected base URL
- classify `.snupkg`, `readme` and `icon` flat-container blobs as immutable
requires unkin/artifactapi#148 merged first
Reviewed-on: #149
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Remotes answer only GET, so HEAD probes get 405, and there is no RubyGems remote. Synthesized (github_*) and rewritten (npm, helm, pypi, puppet, terraform, nuget) mutable responses need the GET path for HEAD lengths to match.
- add a HEAD route for /api/v1/remote/{name}/*; rewritten and synthesized mutable HEADs go through the GET path
- mark rewriting providers with a ResponseRewriter capability; set Content-Length on proxied GETs
- use the remote's upstream client for upstream HEAD
- add a gems remote type: compact index and specs mutable, .gem and gemspec immutable
- add gems docs, UI snippet and e2e caching case; add HEAD-vs-GET e2e checks
Reviewed-on: #148
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
In-house npm packages have nowhere to publish: npm locals only accept raw file uploads, which `npm publish` and `npm install` cannot use.
- accept `npm publish` (`PUT /api/v1/local/<repo>/<pkg>`, scoped included) on local npm repos, capping the body at 256MiB and verifying the tarball against dist integrity
- write tarball, then manifest, then dist-tags; publishing an existing version returns 409 and leaves tags alone, while a matching leftover tarball from a failed publish is reused so retries complete
- serve packuments with tarball URLs pointing back at the local repo, dropping dist-tags that name unpublished versions
- document the `.npmrc` registry + `_authToken` setup in the UI and `docs/npm/local.md`
- add a stock `node:22` publish/install e2e-docker case
Reviewed-on: #139
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A remote created through the API without `mutable_ttl` was stored as 0, so its mutable paths never refreshed. With `check_mutable`, the ETag expired together with the freshness key, so an expired mutable path was always refetched in full and never revalidated.
- default an omitted `mutable_ttl` to 3600 on create; keep the stored value on update; explicit 0 still means never expire
- keep mutable ETags for `max(10 × mutable_ttl, 7d)`, re-armed on each 304, only when `check_mutable` is set and `mutable_ttl` is non-zero
- drop the stored ETag when a refetch returns none; never store one for immutable paths
- flush a remote's Redis keys on delete
- update docs/common/caching.md
Reviewed-on: #151
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Parallel package-type PRs conflict because every type edits the same shared lines (README tables, provider.go, server.go imports, the package type list, UI maps, the e2e caching table). A new type now only adds files.
- move per-type README sections into docs/<type>/{remote,local,virtual}.md
- split provider.go capability and metadata types into cap_*.go and metadata_*.go
- import providers via internal/provider/all, one file per type
- declare each package type in package_type_<type>.go, registered via init()
- split the e2e caching cases and UI colours/snippets into per-type files
Reviewed-on: #150
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
unkin-agent