## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.
**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).
## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.
## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.
## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.
Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Switching a remote's backend (`base_url`) left artifactapi serving the previously-cached mutable metadata (repodata / Release / APKINDEX) until TTL expiry, so requests could keep pointing at the old upstream. `cache.FlushRemote` already existed but was wired to nothing.
## How
- Inject a `MetadataFlusher` (satisfied by `*cache.Redis`) into `RemotesHandler` via `NewRemotesHandler`; `server.go` passes `s.cache`.
- On update, read the existing remote first, then after a successful DB update flush the remote's cached metadata when `base_url` changed, so the next request re-fetches fresh from the new upstream.
- Keep scope to `base_url` (upstream identity); a flush failure is logged as a warning and does not fail the request since the DB update already landed.
- Add tests: a `base_url` change flushes exactly once, an unchanged `base_url` does not flush, and a flush error still returns 200.
Reviewed-on: #120
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Part of #117. Local generated repodata must be byte-identical across the two no-affinity replicas and across every regeneration, so apt/apk never hit a checksum mismatch between an index's advertised hash and the bytes actually served. This does the deb+apk half (the rpm half landed in #118).
How:
- Derive the deb `Release` `Date:` from the newest persisted `created_at` (RFC1123Z, UTC) instead of `time.Now()`; carry `created_at` through the deb metadata SELECT and `DebMetadata`. An empty repo falls back to the Unix epoch. This also stops `Date:` running ahead of wall clock.
- Pin the apk `APKINDEX` tar header `ModTime` to the Unix epoch instead of the zero-value `time.Time`, so it is never wall-clock derived.
- Give both list queries a genuine total order by adding a `file_path` tiebreak (name/version/arch is not unique).
- Add guard tests: deb generators byte-identical across generations; the `Release` checksum/size matches the served `Packages`/`Packages.gz` bytes (the exact apt invariant); `Date:` pinned to `created_at`; apk index byte-identical and tar `ModTime` pinned to epoch.
Reviewed-on: #119
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Local RPM repos regenerated `repomd.xml` on every request and advertised a `primary.xml.gz` sha256 that drifted every second, because `time.Now().Unix()` was embedded inside the gzipped `primary.xml` (and in `repomd` `<revision>`/`<timestamp>`). The advertised hash therefore never matched the content-addressed `<sha256>-primary.xml.gz` bytes a second later or on the other replica, so `dnf` failed with a checksum mismatch. Part of #117.
How:
- Derives `<time file=>` in `primary.xml` from the persisted `rpm_metadata.created_at` instead of the wall clock; unset timestamps collapse to a fixed `0`.
- Derives `repomd` `<revision>`/`<timestamp>` from the newest package upload time, so `repomd.xml` is byte-identical across replicas and requests.
- Adds `file_path` as a total-order tiebreak to the metadata `ORDER BY`.
- Pins the gzip header (`OS: 255`) so compressed bytes depend only on the payload.
- Adds regression tests: generators are byte-identical across two runs, and the sha256 in `repomd.xml` equals the sha256 of the bytes each `serve*` handler returns.
Reviewed-on: #118
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The alpine local repo (a real apk repo with auto-generated per-arch APKINDEX) and the new github_alpine remote (metadata-only apk index synthesized from a GitHub repo's release .apk assets) shipped without any "How do I use this?" UI guidance, unlike deb/github_deb. This adds the matching client instructions so users can consume and publish to these repos.
## How
- Extend `case 'alpine':` in `UsageInstructions.tsx` to branch on `isLocal` (mirroring rpm/deb):
- LOCAL: "Add the apk repo" snippet appends `${url}/api/v1/local/<name>` to `/etc/apk/repositories` with `apk update/add --allow-untrusted` (served unsigned, parity with rpm gpgcheck=0), plus a "Publish a .apk" snippet uploading to the canonical `<arch>/<name>-<version>.apk` path (APKINDEX carries no filename, so path must match or install 404s).
- REMOTE: existing caching-proxy snippet kept unchanged.
- Add `case 'github_alpine':` (always remote-class): "Add the apk repo (metadata-only, from GitHub releases)" snippet with `--allow-untrusted`; note explains the per-arch APKINDEX is synthesized from release .apk assets and downloads redirect to the backing releases remote.
- No other cases touched. `npm run build` (tsc typecheck + vite) passes; pre-commit passes.
Reviewed-on: #116
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
`github_alpine` is the Alpine/apk analog of the existing `github_deb`/`github_rpm` metadata-only remotes. It lets a plain GitHub-releases repo of `.apk` files be consumed as a real apk repository without artifactapi ever precaching the packages: it scans the repo's releases, derives each package's `.PKGINFO` from a ranged prefix fetch, synthesizes a per-arch `APKINDEX.tar.gz` from the cached metadata, and redirects the actual `.apk` downloads to a backend `releases_remote`. It stacks on the apk-local branch, reusing that work's alpine APKINDEX generator, `.apk`/`.PKGINFO` parser, Q1 pull-checksum, and `AlpineMetadata` store.
## How
- **`pkg/models`**: add `PackageGitHubAlpine` to the enum + validators (and test).
- **`internal/provider/alpine/github.go`**: the `github_alpine` provider. `ServeRemote` serves per-arch `<arch>/APKINDEX.tar.gz` (reusing `generateAPKIndex` over arch-filtered `AlpineMetadata` rows, `normalizeIndexPath` for apk's `./` dot-segment), 302-redirects `*.apk` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`, and cold-starts with a 503 + `Retry-After`. `scanWithState` lists releases with ETag/If-None-Match and incrementally derives/prunes. `deriveAsset` does a **ranged GET of just the front of the `.apk`** — the control gzip stream carrying `.PKGINFO` sits near the front — doubling the range on truncation; it parses `.PKGINFO` and computes the `C:` Q1 checksum (`Q1`+base64(sha1(control stream))). `FilePath` = the github-relative asset path so the redirect resolves.
- **`internal/provider/alpine/syncer.go`**: a parallel background `Syncer` (own worker pool, shared rate limiter, deduped queue, DB lease), separate from the deb/rpm syncers.
- **`internal/database/alpine_github_sync.go`** + `github_alpine_sync_state` table: `ListGitHubAlpineRemotes` + Claim/Release per-remote sync lease, kept separate from the deb/rpm tables.
- **`internal/server/server.go`**: construct + `Run` the alpine syncer alongside deb/rpm and register it in the `PackageType→Primer` map (priming on create then flows through the existing generic `remotes.go` path).
The rpm/deb providers, syncers, and tables are untouched — this adds parallel alpine equivalents and reuses shared helpers already present in the alpine package.
## Tests
Mirror the deb github tests: scan derives `.PKGINFO`/Q1 from a ranged prefix of a `testsupport.MinimalApk` served over an httptest range server (no full download); diff/prune; pattern filter; per-arch `ServeRemote` routing (index served per-arch and grouped, dot-segment collapse, `.apk` → 302, cold-start 503, warm 200, canceled-request-serves-cache); DB lease prevents a second replica; prime bypasses the recency window. `go build`/`go vet`/`go mod tidy` clean, `make test` (-race) green, pre-commit green.
Reviewed-on: #115
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The alpine provider only supported remote (proxy) repositories, so there was no way to publish first-party `.apk` packages the way `rpm-local` and `deb-local` already allow. This extends the existing alpine provider into a real apk repository: uploaded `.apk` files are parsed in pure Go and a per-arch `APKINDEX.tar.gz` is generated on demand, at parity with rpm repodata and deb Packages generation. (The metadata-only `github_alpine` type is a separate follow-up and is not part of this PR.)
## How
- Implements `LocalUploader` / `LocalIndexer` / `PostUploadHook` / `PostDeleteHook` on the existing `alpine` provider, leaving the remote proxy methods (`UpstreamURL`/`ContentType`/`AuthHeaders`/`RewriteResponse`/`Classify`) intact.
- Parses the `.apk` (up to three concatenated, independently gzipped tar streams) in pure Go: locates the control stream by its `.PKGINFO` member, reads the `key = value` fields, and computes the apk pull checksum `C:` = `Q1` + base64(sha1(**control gzip stream bytes**)) — the sha1 of the second gzip member, not of the whole file.
- Derives arch from `.PKGINFO` and records download size (`S:` blob size) and installed size (`I:` from `.PKGINFO size`).
- Generates an **unsigned** per-arch `APKINDEX.tar.gz` = gzip(tar(`APKINDEX`)) filtered by requested arch (clients use `--allow-untrusted`, matching rpm `gpgcheck=0` / deb `[trusted=yes]`), applying the same dot-segment normalization as deb so `./<arch>/APKINDEX.tar.gz` resolves. Non-index / `.apk` paths return `false` so the generic file streamer serves the stored blob.
- Adds `AlpineMetadata` plus **separate** `AlpineMetadataStore` / `AlpineMetadataReader` / `AlpineMetadataDeleter` interfaces (type-asserted from the generic hooks) so the shared rpm/deb metadata interfaces and their test doubles are untouched.
- Adds the `alpine_metadata` table (keyed by `repo_name` + `file_path`, per-arch index) and its `Insert`/`Delete`/`List` DB methods.
- Adds `testsupport.MinimalApk`, unit tests (`.PKGINFO` parse, Q1 checksum over the control stream, per-arch filtering, empty-field omission, `./` dot-segment handling, ValidateUpload accept/reject), and a `dockere2e` `TestLocalAlpineIndex`.
## Consumption
`/etc/apk/repositories` line = `<url>/api/v1/local/<name>` (apk appends `/<arch>/APKINDEX.tar.gz`); `apk update --allow-untrusted && apk add --allow-untrusted <pkg>`. Packages live at `/api/v1/local/<name>/<arch>/<file>.apk`.
## Verification
`go build ./...`, `go vet ./...` (incl. `-tags dockere2e`), `go mod tidy` (no change), `make test` (`-race`), and `pre-commit run --all-files` all pass.
Reviewed-on: #114
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The `deb` and `github_deb` repository types had no tailored "How do I use this?" snippets in the UI — they fell through to the generic default (curl-a-file-by-path), which does not tell a user how to actually wire the repo into apt. This adds real, end-to-end-validated apt instructions mirroring how the existing package types (rpm, npm, etc.) are handled.
## How
- Adds a `case 'deb':` to `buildSnippets` that branches on `isLocal`: a flat real apt repo served at `/api/v1/local/<name>/ ./` with `[trusted=yes]` plus a publish-a-.deb snippet for locals, and a caching-proxy `deb <proxy> <suite> <component>` form (upstream-signed, no `[trusted=yes]`) for remotes.
- Adds a `case 'github_deb':` emitting the metadata-only flat form `deb [trusted=yes] <proxy>/ ./`, whose index is synthesized from a GitHub repo's release .deb assets.
- Reuses the existing `url`/`proxy`/`isLocal` helpers and matches the surrounding case style; leaves the rpm and other cases untouched.
Reviewed-on: #113
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
This stacks the Debian/apt analog of `github_rpm` on top of the deb local+remote work (#111). It lets a GitHub repo's `.deb` release assets be consumed as a real apt repository without artifactapi ever precaching whole packages: it derives per-asset control metadata from a ranged prefix fetch, synthesizes a flat apt repo from the cache, and redirects the actual `.deb` downloads to a backend `releases_remote` (the generic github.com remote).
Base is `benvin/deb-local-remote` (stacked) to keep the diff atomic.
## How
- Adds `github_deb` to the package-type enum and validity map.
- Adds the `github_deb` provider mirroring `github_rpm`: `ServeRemote` serves `Packages`/`Packages.gz`/`Release`, returns 404 for `InRelease`/`Release.gpg` (unsigned, consumed via `[trusted=yes]`), and 302-redirects `*.deb` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`; cold-start prime with a retryable 503.
- `deriveAsset` ranged-GETs the front of the `.deb` (an `ar` archive), locates and fully reads `control.tar.*`, and parses the control paragraph — doubling the range if the control member is truncated. The Packages `SHA256` comes from the GitHub asset `digest` when present, else a one-time full stream; `MD5sum` is left unset (apt verifies against SHA256 under `[trusted=yes]`).
- Adds a `github_deb` background Syncer (own worker pool, shared rate limiter, deduped queue) with per-remote DB-lease-gated scans so only one replica scans per window.
- Adds the `github_deb_sync_state` table plus `ListGitHubDebRemotes` / `ClaimGitHubDebSyncLease` / `ReleaseGitHubDebSyncLease` DB helpers, kept separate from the rpm ones.
- Primes `github_deb` remotes on create and runs the deb syncer alongside the rpm one; prime-on-create is routed by package type.
- Reuses the deb apt-index generators and control parser; the Packages generator now skips empty hash lines so a SHA256-only entry is valid.
## Notes / deviations
- **Filename convention:** the `Filename` stored in the Packages index is the **github-relative** asset path (same as rpm's `assetPath`), not `pool/<asset>`. This is required for the `.deb` 302 to `{releases_remote=github}/{path}` to resolve against github.com; it still matches the `*.deb` redirect rule.
- **GitHub client helpers** (releases pagination, ranged GET, auth headers) are duplicated into the deb package rather than shared, because the rpm equivalents are unexported in `package rpm` and the task requires not modifying the rpm provider.
- `go build`, `go vet`, `go mod tidy`, and `make test` (`-race`, incl. the Postgres lease integration tests) all pass; pre-commit clean.
Do not merge — for review.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #112
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Brings Debian/apt to artifactapi with feature parity to the existing rpm support (local + remote), so `.deb` packages can be hosted as a flat apt repo and a Debian/Ubuntu mirror can be cached through the proxy.
- Adds `deb` to the package-type enum and registers a new `internal/provider/deb` provider.
- Classifies `.deb` blobs immutable and the apt index surface (`Packages`, `Release`, `InRelease`, `dists/`, by-hash) mutable so the caching engine revalidates it.
- Parses the `.deb` in pure Go (ar archive to `control.tar.{gz,xz,zst}` to `./control`), storing the raw control stanza plus computed size/md5/sha256 as `deb_metadata`.
- Serves a flat apt repo (`deb [trusted=yes] .../ ./`): generates `Packages`, `Packages.gz` and an unsigned `Release` (returns 404 for `InRelease`/`Release.gpg`), mirroring rpm unsigned repodata / gpgcheck=0 trust model.
- Proxies a remote mirror via `UpstreamURL`/`ContentType`/`AuthHeaders` (HTTP Basic).
- Adds the `deb_metadata` table to `migrate()`, DB access methods, a `MinimalDeb` pure-Go fixture, unit tests, and a `dockere2e` `TestLocalDebRepo`.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #111
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
## Why
Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories. A machine credential usable by a free (non-enterprise) account is needed to lift the request budget to ~5000/hr and to read private-repo release assets.
Builds on the background syncer (#108, now merged to `master`); this diff is the auth changes only.
## How
- Add `internal/githubauth`: a process-wide GitHub credential delivered via env/secret, applied by default to every outbound GitHub request (releases scan, ranged asset-header GETs, and the generic-github byte proxy for private assets).
- Support two modes:
- **PAT** — `GITHUB_TOKEN` sent as `Authorization: Bearer <token>`.
- **GitHub App** — `GITHUB_APP_ID` + `GITHUB_APP_INSTALLATION_ID` + private key (`GITHUB_APP_PRIVATE_KEY` inline PEM or `GITHUB_APP_PRIVATE_KEY_PATH`). Mint a short-lived RS256 JWT with stdlib `crypto/rsa` (no new dependency), exchange it at `POST /app/installations/{id}/access_tokens` for a ~1h installation token, cache it, and single-flight a refresh a few minutes before expiry.
- Inject at the two GitHub call paths: the rpm github provider header builder (releases + ranged fetches) and the generic provider `AuthHeaders` (byte proxy, github.com hosts only; the pre-signed `objects.githubusercontent.com` redirect deliberately gets no Authorization).
- Honor precedence: a remote's own `username`/`password` overrides the server credential; no credential configured stays anonymous (current behavior).
- Fail closed at startup on partial App configuration (e.g. App id without a private key); a token-and-App conflict is also rejected.
- Never persist the credential to the DB, return it from an API, or log it (token-exchange failures never echo the response body).
- Read config via the existing `getenv` convention; document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication.
## Rate limit
Authenticated requests share the syncer's single global limiter — no second limiter is added. A token raises the effective GitHub ceiling (~5000/hr vs ~60/hr), so the limiter defaults stay safe.
## Tests
`internal/githubauth` and `internal/provider/{rpm,generic}`:
- PAT attaches the correct `Authorization` header to releases + asset-header requests.
- App mints a valid RS256 JWT (verified against the app public key), exchanges it at a mocked endpoint, reuses the cached token without re-exchanging, refreshes near expiry, and single-flights concurrent callers.
- Per-remote credential overrides the server credential (rpm + generic).
- No credential → no `Authorization` header, requests still succeed anonymously.
- ETag/304 flow still works with auth attached.
- The credential does not appear in a remote's serialized JSON.
- Config validation: no-config is anonymous; partial App config and token/App conflict both error.
Verified fail-before/pass-after for the injection tests. `gofmt -l`, `go build ./...`, `go vet ./...`, `go test ./...` all clean (26 packages).
Reviewed-on: #109
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.
## How
- Add a single per-process background syncer (started at boot, cleanly stopped
on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the
github provider so every GitHub call (releases list + each ranged asset GET)
acquires a token first.
- Re-check each `github_rpm` remote for new/changed releases on its existing
`mutable_ttl` cadence; derive only new/changed assets incrementally and prune
assets that disappear upstream. Repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call returns
immediately.
- Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives
nothing and is not counted against GitHub's rate limit, so an unchanged repo
is nearly free.
- Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`,
`etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only
for the replica that atomically claims the lease, bounding total GitHub load
to ~once per `mutable_ttl` regardless of replica count; the ETag is shared
through the same row.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
cache, and return a bounded wait then a retryable `503` rather than blocking
on a cold derive.
- Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` /
`GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst
5, 3 workers, 60s tick) and document the syncer in the README.
## Tests
- Unit (httptest, Range/ETag-aware fixture): `304` releases response derives
nothing; incremental derive fetches only the newly added asset; the shared
limiter caps request rate; work-queue enqueues coalesce to one job; prime
enqueues a job; a held lease stops a second replica from scanning; cold-start
serves `503` while warm cache serves `200`.
- DB integration (testcontainers postgres): the real lease SQL — one holder at a
time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
bypasses recency but respects a live lease.
- Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the
background at ~1 req/s (global limiter); `dnf makecache` served fast from the
priming cache (no cold block); clean shutdown mid-scan, no panics.
## Notes
- Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.
Reviewed-on: #108
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Publishing RPMs to GitHub releases is common, but consuming them with `dnf` requires repodata GitHub does not provide, and mirroring every package into a local repo wastes storage and staleness-tracking on artifacts that already have a durable home. This exposes GitHub releases as a first-class RPM source that synthesizes repodata on the fly and **never precaches the packages**.
## What
Add a `github_rpm` remote package type backed by a metadata-only provider.
- Introduce a `RemoteServer` interception hook (the remote-side analog of `LocalIndexer`): `handleProxy` lets a provider fully answer a request before the byte-proxy engine, passing the request-derived proxy base URL and the DB as a `RemoteMetadataStore`.
- Scan a repo's releases via the GitHub API (`base_url` = the releases API root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset filename), and reuse the existing local-rpm repodata generators to emit `repomd.xml`/`primary`/`filelists`/`other`.
- Derive per-asset metadata without precaching: fetch only the RPM header via a ranged GET (retrying with a larger range on a truncated-header parse) for NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from the GitHub asset `digest` when present, else compute it once by streaming.
- Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more often than `mutable_ttl`, pruning assets that disappear upstream.
- Serve each package's `<location>` as the github-relative download path so the client comes back to this remote, which **302-redirects** to the `releases_remote` (an existing generic github.com remote) that streams the actual bytes.
Reuse the existing `releases_remote` field as the redirect target — it already carries exactly this "downloads served by remote X" semantic end to end, so no new schema/model field is needed.
Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns, added idempotently) so both local and `github_rpm` repodata resolve upgrades and conflicts; the local upload path records them too.
## No-precache mechanics
- **Dependency metadata**: always from the ranged header fetch (header precedes payload; `rpm.Read` stops at the payload boundary), giving `dnf` full resolution. Default range 1 MiB, doubling to 16 MiB.
- **Checksum**: prefer the GitHub asset `digest` (no download); fall back to a one-time streamed sha256 only when absent. Header-only "minimal mode" (no deps) is rejected as a default because `dnf` needs accurate provides/requires and a correct pkgid checksum to install.
## Tests
Header-range parsing incl. the retry loop, digest-vs-computed checksum selection, repodata synthesis with the redirect-able `<location href>`, the 302 redirect path (and the guard when `releases_remote` is unset), asset pattern filtering, and stale-asset pruning. `go build`/`vet`/`test` green; pre-commit clean.
## Follow-ups
- `github_apk` / `github_deb` metadata-only remotes (same pattern; not in this PR).
- Terraform provider support for `artifactapi_remote_github_rpm` ships as a separate PR against `terraform-provider-artifactapi` (depends on this API surface).
Reviewed-on: #107
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
A repository detail page in the ArtifactAPI UI showed configuration and stats, but nothing that told a user how to actually *consume* the repo. You had to already know the per-package-type URL scheme (yum baseurl, pip index-url, docker registry host, terraform source address, ...) by hand. This adds an in-page, copy-pasteable "How do I use this?" panel so each repo page tells you exactly how to point a Linux host at it.
## Changes
- Add a `UsageInstructions` component: a collapsible "How do I use this?" panel with monospace code boxes and copy-to-clipboard buttons, styled to match the existing detail-section / badge theme.
- Generate instructions per package type (rpm, pypi, npm, docker, terraform, helm, alpine, goproxy, puppet, generic) and per class:
- **remote** — consume via the caching proxy (`/api/v1/remote/<name>/...`).
- **local** — consume via the real registry endpoint, plus a publish/push example (rpm `PUT .../files/`, docker Registry V2 push, terraform provider upload).
- **virtual** — consume the merged index via `/api/v1/virtual/<name>/...` using the same per-type client config.
- Interpolate the repository's real name into every snippet so it is genuinely copy-pasteable.
- Resolve the instance base URL from `window.location.origin` (the UI is served on the API origin, client `BASE=''`) instead of hardcoding a hostname; falls back to the public host only when `window` is unavailable.
- Render the panel on `RemoteDetail`, `LocalDetail`, and the `Virtuals` member-expand panel.
## Verification
- `npm run build` (tsc typecheck + vite build) passes.
- Rendered the component in headless Chromium against real repo data for rpm remote, rpm local (with publish), docker local (with push), terraform local (HCL `required_providers` + signing note), and a pypi virtual — all snippets render with the correct URLs and theme.
Reviewed-on: #105
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The local-repo object browser renders every file name as inert text, so there is no way to grab a file from the UI even though local rpm repos already serve their contents as real yum repos. Users have to hand-construct URLs. This adds one-click downloads, built as a modular per-repo-type capability so other types can be switched on later with a single map entry.
## Changes
- Adds a `downloadableTypes` capability map (`ui/src/components/downloads.ts`) keyed by `package_type`; each entry builds the direct-download URL for a file.
- Enables `rpm`, pointing at the yum files route `/api/v2/remotes/<repo>/files/<path>` (path-segment-encoded).
- Renders file names in the object browser as `<a href download>` links when the repo's type is downloadable, and as plain text otherwise.
- Fetches the repo's `package_type` on the Objects page as the modularity hook and threads it through the tree rows.
## Notes
Download links are same-origin unauthenticated GETs (the API serves local repos with no token on reads, matching how yum clients fetch), so a bare `href` carries no credentials. Adding a future type is one entry in `downloadableTypes`.
---------
Co-authored-by: Ben Vin <neotheo@gmail.com>
Reviewed-on: #106
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Chunked blob uploads kept the in-progress session in **process memory** keyed by upload UUID, so the `POST`/`PATCH`/`PUT` of a single `docker push` had to land on the same replica. The API runs at `minReplicas: 2` with no session affinity (see argocd-apps `api-hpa.yaml`), so a real push — which streams the layer via `PATCH` then finalises with `PUT` — intermittently 404s with `BLOB_UPLOAD_UNKNOWN` when a chunk hits a replica that never saw the `POST`. This was flagged when the local docker registry landed (#103).
## Changes
- Stage chunked uploads in object storage under `uploads/<uuid>` instead of an in-memory temp file. The UUID travels in the `Location` URL handed to the client, so any replica reconstructs the staging key with no shared in-process state. Finalise streams the staged bytes plus any trailing `PUT` body through the CAS in one pass; monolithic uploads are unchanged.
- Support `DELETE` of an in-progress upload (cancel) by dropping its staging object.
- Reap abandoned staging objects in the GC (`uploads/` older than 24h) via a new `S3.ListStaleObjects`, so cancelled/interrupted pushes don't leak.
## Verification
- Split a single push across **two instances sharing one Postgres+MinIO**: `POST`→A, `PATCH`→B, `PUT`→A finalises with the correct digest, and the blob pulls back **byte-identical from both** replicas. Config-blob and manifest pushes split the same way succeed; `tags/list` is correct. (Pre-fix, the cross-replica `PATCH` 404s.)
- `scripts/docker-e2e.sh` still passes (incl. `TestLocalDockerPushPull`); unit tests + `go vet` clean.
Reviewed-on: #104
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Local `docker` repos had no write path — the `/v2` Docker Registry API only proxied to upstreams. This makes a local docker repo a genuine container registry so `docker push`/`docker pull` (and podman/skopeo/buildah) work against it directly, matching the project principle that a local repo is *the real thing* rather than a mirror.
## Changes
- Implement the Docker Registry HTTP API V2 read/write half for local docker repos: blob uploads (monolithic and chunked POST/PATCH/PUT), manifest push, `tags/list`, and blob/manifest GET/HEAD.
- Store blobs and manifests through the existing content-addressable store; keep a `local_files` reference per (repo, image) so the GC does not reap them. Tags are mutable (`UpsertLocalFile`); digests and blobs are immutable.
- Dispatch `/v2` reads to the local handler for local docker repos and fall through to the upstream proxy otherwise; writes are local-docker only.
- Add `UpsertLocalFile` for mutable tag references.
- Cover the push/pull round-trip with a dockerised e2e test and unit-test the registry path parser. Document the registry in the README.
## Verification
- `scripts/docker-e2e.sh` passes, including the new `TestLocalDockerPushPull`.
- Verified a real end-to-end round-trip with skopeo against a live instance: pushed `hello-world`, pulled it back, loaded it into the docker daemon, and ran it successfully.
Reviewed-on: #103
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Local terraform repos already served the Terraform **network mirror** protocol, but consuming that requires every user to add a `provider_installation { network_mirror }` block to `~/.terraformrc`. A `source = "artifactapi.k8s.../ns/type"` address instead triggers the **provider registry** protocol (service discovery at `/.well-known/terraform.json` + GPG-signed SHA256SUMS), which returned 404 — hence *"does not offer a provider registry."*
Local repos are meant to be the real thing, so this makes a terraform local repo a first-class provider registry: `terraform init` installs from a bare source address with no client config.
## What
- Serve `/.well-known/terraform.json` service discovery and the `providers.v1` endpoints under `/terraform/v1/providers`: `versions`, `download/{os}/{arch}`, `sha256sums`, `sha256sums.sig`.
- Map the Terraform **namespace** segment to the artifactapi **repo name**; locate the provider by **type**. `download_url` points back at the existing `/api/v1/local/...` path.
- Generate `SHA256SUMS` per version and sign it with a GPG key loaded from `TF_SIGNING_KEY_PATH` (optional `TF_SIGNING_KEY_PASSPHRASE`); advertise the public key + key id in the download response. **No key → registry stays disabled (endpoints 404)**, so behaviour is unchanged until the signing secret is present.
- New `internal/tfsign` (key load + detached signing, via `x/crypto/openpgp`) and `internal/api/terraform` (registry handler). Export `ParseProviderZip` for reuse.
- `TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
- README section documenting usage.
## Consumer
```hcl
terraform {
required_providers {
artifactapi = {
source = "artifactapi.k8s.syd1.au.unkin.net/terraform-unkin/artifactapi"
version = "0.1.2"
}
}
}
```
## Tests
- `internal/tfsign`: sign + verify round-trip, disabled/missing-key paths.
- `internal/api/terraform`: dockerised full flow (discovery → versions → download → sha256sums → sig), verifying the signature against the advertised public key.
## Follow-ups (separate PRs)
- **argocd-apps**: mount the signing K8s secret into the api deployment + set `TF_SIGNING_KEY_PATH`. The `/` HTTPRoute already routes `/.well-known` and `/terraform` to the API, so no gateway change is needed.
- Image/version bump once tagged.
## Note
Anchored the `terraform/` gitignore to the repo root (`/terraform/`) so it stops matching `internal/*/terraform/`. This surfaced `internal/provider/terraform/terraform_extra_test.go`, which had been silently untracked — now committed.
Reviewed-on: #102
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The web UI ships as a separate image served under \`/ui\` (built with \`BASE_PATH=/ui\`). Hitting the bare domain (e.g. \`https://artifactapi.k8s.syd1.au.unkin.net/\`) returned the API's JSON identity blob instead of the app, so browsers never landed on the UI.
## Changes
- Redirect \`GET /\` to \`/ui/\` (302 Found).
- Preserve the former root JSON (\`{"name","version"}\`) at \`/version\`, so health/monitoring can still read the running version.
- Update the server integration test to assert the redirect and the \`/version\` payload.
Reviewed-on: #101
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #99.
## Why
Evicting or deleting a local RPM removed the \`local_files\` row but left its \`rpm_metadata\` behind. Since generated repodata is built from \`rpm_metadata\`, \`primary.xml\` kept advertising a package that no longer exists, producing 404s for clients that tried to fetch it.
## Changes
- Add \`PostDeleteHook\` and \`MetadataDeleter\` provider interfaces (symmetric to the existing \`PostUploadHook\`/\`MetadataStore\`), plus a \`DeleteRPMMetadata\` DB method.
- Implement \`AfterDelete\` in the RPM provider to drop the metadata row for the deleted file.
- Route both local delete paths — the new \`evictLocal\` and the existing files handler's \`remove\` — through a shared \`deleteLocalFile\` helper that removes the file then runs the provider's post-delete hook. Non-RPM providers have no hook, so nothing changes for them.
- Cover the cleanup with a dockerised test.
Reviewed-on: #100
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Local repos store uploaded files in the \`local_files\` table, whereas remote/proxy repos cache into the \`artifacts\` table. The shared **Cached Objects** page always queried the artifacts table via \`/api/v2/remotes/{name}/objects\`, so files uploaded to a local repo (e.g. an internal RPM) were fully stored and servable but showed as **0 objects** in the UI.
## Changes
- Add \`ListLocalArtifacts\`, joining \`local_files\` with \`blobs\` and returning \`models.Artifact\`-shaped rows (size from the blob; access/fetch counters zero and timestamps derived from \`created_at\`, since local files track no access).
- Add \`LocalRoutes\` to the objects handler: \`listLocal\` reads \`local_files\`, \`evictLocal\` deletes via \`DeleteLocalFile\`. Extract shared page/per_page parsing into \`pageBounds\`.
- Mount \`/api/v2/locals/{name}/objects\` (GET + DELETE) in the server.
- Add \`listLocalObjects\`/\`evictLocalObject\` to the UI client and route the Objects page to them when viewing a local repo.
- Cover the listing and eviction paths with a dockerised test.
## Notes
Generated \`repodata/*\` files are not listed — they are produced on the fly from \`rpm_metadata\` and never stored in \`local_files\`, which matches how the repo serves them.
Reviewed-on: #99
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Adds a black-box e2e suite that runs against the **built container image** via docker-compose (complementing the in-process `e2e/` testcontainers suite).
## What it does
`make docker-e2e` → `scripts/docker-e2e.sh`: builds the image, brings up the full stack (postgres, redis, minio, artifactapi) plus a static nginx **mock upstream** for hermetic caching, waits for `/health`, runs `go test -tags=dockere2e ./e2e-docker/...`, and tears everything down.
## Coverage
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact for **each of the 10 remote package types** (generic, docker, helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/` index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge across two members.
## Notes
- The artifactapi host port is parameterised (`ARTIFACTAPI_PORT`, default `8000`; the e2e run uses `8001`) so it does not collide with a locally-running instance. This is the only change to the production `docker-compose.yml`.
- Fixtures under `e2e-docker/fixtures/` are real package files (incl. a real RPM so repodata parsing works); a `.gitignore` negation tracks them over the global ignore of those extensions.
## Validation
Ran `make docker-e2e` locally: **all suites pass** against the containerised product.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #97
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Raises statement coverage of the core packages (all of `internal/` except the interactive `tui/`, plus `pkg/`) from **8.7% to 90.1%**.
## Approach
- **Pure-go unit tests** for all providers, virtual mergers, classifier, config, auth, models, and the API client (httptest).
- **Testcontainers-backed** tests (new `internal/testsupport` helper: Postgres/Redis/MinIO, Ryuk disabled) for database, storage, cache, the proxy engine, the GC, and a full-stack `server` test that drives the whole HTTP API. These `t.Skip` when Docker is absent so `go test` still runs locally without it.
## Measuring
```
go test -coverpkg=./internal/...,./pkg/... -coverprofile=cover.out ./internal/... ./pkg/...
grep -v /internal/tui/ cover.out | go tool cover -func=/dev/stdin | tail -1 # 90.1%
```
Run with `-p 1` (containers are heavy).
## Notes
- The interactive `tui/` package and `cmd/main` are excluded from the target per the agreed scope.
- Some defensive error branches are covered via fault injection (closed DB pool, killing MinIO mid-upload).
Reviewed-on: #98
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#74
## Why
`internal/proxy/circuit.go` implemented and tested a circuit breaker, but nothing ever called it — a repeatedly-failing upstream was still hit on every request.
## Changes
- Construct a `CircuitBreaker` in `NewEngine`.
- In `Engine.Fetch`: short-circuit when the breaker is open (serve stale from the store if present, otherwise return 503), `RecordFailure` on each `UpstreamError`, and `RecordSuccess` on a successful fetch.
## Validation
- `go test ./internal/proxy/` and `make e2e` pass.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #90
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#71
## Why
`FindOrphanedBlobs` returned any blob not currently referenced. Because CAS dedups (the blob row can exist before its artifact/local_files row is written), a concurrent upload reusing an existing hash could have its S3 object deleted mid-flight by the GC.
## Changes
- `FindOrphanedBlobs` now takes a `minAge` and only returns blobs with `created_at < now()-minAge`.
- The collector passes a 1h `blobGracePeriod`.
## Validation
- `go test ./internal/gc/...` and `make e2e` pass.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #86
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`master` does not compile. Three PRs that each built individually combined badly:
- #92 changed `fetchBearerToken` to return `(string, time.Duration, error)` and added `cachedBearerToken` (which hashes the challenge via `sha256Hash`).
- #94 (streaming) removed the now-unused-in-that-PR `sha256Hash` helper and its `crypto/sha256` / `encoding/hex` imports.
- #89 (HEAD) added `headUpstream`, which calls `fetchBearerToken` expecting two return values.
Result on `master`: `internal/proxy/engine.go` fails to build (`assignment mismatch: 2 variables but fetchBearerToken returns 3 values`; `undefined: sha256Hash`).
## Changes
- Re-add the `sha256Hash` helper and its `crypto/sha256` + `encoding/hex` imports.
- Fix the `headUpstream` 401 path to handle `fetchBearerToken`s three return values.
## Validation
- `go build ./...`, `go vet`, and `make e2e` all pass.
Should merge before the other in-flight branches so they rebase onto a compiling `master`.
Reviewed-on: #96
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#67
## Why
The proxy used `http.DefaultClient` for all upstream GET/HEAD and bearer-token requests. It has no timeouts, so a slow or hung upstream holds a goroutine and connection indefinitely.
## Changes
- Add a shared `upstreamClient` (`internal/proxy/httpclient.go`) with dial, TLS-handshake, response-header and idle-connection timeouts, plus connection pooling.
- Deliberately no overall `Client.Timeout`, so large artifact bodies can still stream; total time is bounded by the request context.
- Route all four upstream calls in the engine through it.
## Validation
- `make e2e` passes.
Reviewed-on: #83
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#69
## Why
`getenv` returned the fallback whenever `os.Getenv` was empty, so an intentionally-empty env var could not override a non-empty default.
## Changes
- Use `os.LookupEnv` to distinguish unset from set-but-empty.
## Validation
- `make e2e` passes.
Reviewed-on: #85
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#75
## Why
On a fetch-lock miss, `Engine.Fetch` slept a flat 500ms once, tried the store, and otherwise fell through to fetch upstream unlocked. A cold-cache stampede therefore still hit upstream once per waiter.
## Changes
- Add `waitForStore`, which polls the store every 100ms for up to 5s (stopping on context cancellation) so waiters pick up the lock leaders populated result.
- Only fall through to an upstream fetch if the leader has not populated the store within the wait budget.
## Validation
- `make e2e` passes.
Reviewed-on: #93
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#76
## Why
Every proxied request spawned a goroutine running a 5s-timeout single-row INSERT. Under load this is unbounded goroutines and connection-pool pressure.
## Changes
- Add `database.AccessLogEntry` + `InsertAccessLogBatch` (bulk `COPY`).
- The engine starts one background writer that drains a buffered channel and flushes every 128 entries or 2s.
- `logAccess` is now a non-blocking channel send (drops on full buffer), so the request path never blocks on the DB. Best-effort telemetry: a small tail may be lost on abrupt shutdown.
## Validation
- `make e2e` passes.
Reviewed-on: #91
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#70
## Why
Docker `HEAD` routes mapped to `handleProxy`, which ran a full `Fetch` + `io.Copy` — downloading the entire blob (and fetching upstream on a miss) only for net/http to discard the body. HEAD existence checks (manifests, blobs) are common.
## Changes
- Add `Engine.Head`: answers cached artifacts/indexes from store metadata (no blob download); on a miss issues an upstream `HEAD` (with bearer-token handling) and never caches a body.
- Route `HEAD /v2/{remote}/*` to a dedicated `handleProxyHead` that writes headers only.
- Add e2e tests for HEAD on a blocklisted path (403) and an unknown remote (404).
## Note
`headUpstream` uses `http.DefaultClient` to build cleanly on master; it will pick up the shared timeout-configured client from #67 once that merges.
## Validation
- `make e2e` passes (includes new HEAD tests).
Reviewed-on: #89
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#77
## Why
Each upstream 401 re-ran the full token-endpoint request, even though a single Docker pull triggers many blob/manifest requests sharing one scope.
## Changes
- Add Redis `GetToken`/`SetToken`.
- `fetchBearerToken` now also parses `expires_in` and returns a TTL.
- New `Engine.cachedBearerToken` reuses a cached token keyed by remote + challenge (hashed), caching for `expires_in` minus a safety margin (default 60s when absent).
## Validation
- `make e2e` passes.
Reviewed-on: #92
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#66
## Why
`fetchFromUpstream` read every upstream response with `io.ReadAll`, hashed it in memory, uploaded from memory and served from memory. A single large immutable blob (Docker layer, RPM, tarball, Go module zip) — or several concurrent ones — could OOM the process. The streaming, tempfile-backed CAS already existed but the proxy path bypassed it (and `Engine.cas` was assigned but unused).
## Changes
- Immutable fetches now stream through `CAS.Store` (tempfile -> sha256 -> S3), so memory stays bounded regardless of artifact size, and are served back from the store.
- Mutable indexes stay on the in-memory path (small, and subject to `RewriteResponse`).
- Skipping `RewriteResponse` for immutable content is behaviour-preserving: the proxy path always passes an empty `proxyBaseURL`, under which every providers `RewriteResponse` is a no-op.
- Remove the now-unused in-memory `sha256Hash` helper.
## Validation
- `make e2e` passes.
- Live smoke test against Postgres/Redis/MinIO: proxied a 12 MB blob through a generic remote — fetch #1 `X-Artifact-Source: remote`, fetch #2 `X-Artifact-Source: cache`, both byte-identical (sha256) to the origin.
Reviewed-on: #94
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#73
## Why
`Classifier.Classify` runs on every proxied request and recompiled the Blocklist/Patterns/Immutable/Mutable regex lists each time. Regex compilation is expensive and fully redundant.
## Changes
- Memoise compilation in a `sync.Map` keyed by pattern text (`compileCached`); each distinct pattern compiles once and is reused. Patterns that fail to compile are cached as a typed nil so they are not retried. No invalidation needed since the pattern text is the key.
## Validation
- `go test ./internal/proxy/` and `make e2e` pass.
Reviewed-on: #88
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#72
## Why
`compilePatterns` silently discards any pattern that fails to compile. A typo in a blocklist entry therefore turns a deny rule into a no-op — a fail-open with security impact.
## Changes
- Add `Remote.ValidatePatterns`, which compiles every pattern list (patterns, blocklist, mutable/immutable patterns, ban_tags) and returns an error on the first invalid regex.
- Reject invalid patterns with 400 at remote create and update time.
- Unit test for valid and invalid patterns.
## Validation
- `go test ./pkg/models/` and `make e2e` pass.
Reviewed-on: #87
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#68
## Why
`isNetworkError` type-asserted `err.(*UpstreamError)` directly. If the error is ever wrapped, stale-on-error handling silently stops triggering.
## Changes
- Use `errors.As` to detect `*UpstreamError` through wrapping.
## Validation
- `make e2e` passes.
Reviewed-on: #84
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#78
## Why
`serveFromStore` first called `store.Download` with the bare content hash as the S3 key, which never matches real object keys (`blobs/sha256/<hash>`). Every cached blob serve therefore paid an extra guaranteed-404 round-trip before retrying with the correct `BlobKey`.
## Changes
- Remove the dead first `Download` attempt; go straight to the `BlobKey` lookup, then fall back to the index key.
## Validation
- `make e2e` passes (proxy cache-hit paths exercised end-to-end).
Reviewed-on: #82
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes#80
## Why
`make e2e` did not compile against master: `e2e/e2e_test.go` called `server.New(cfg)` but the signature is `New(cfg, version string)`. This blocked all end-to-end validation.
## Changes
- Pass a static `"e2e-test"` version to `server.New` in the e2e bootstrap.
## Validation
- `make e2e` builds and passes (testcontainers: postgres/redis/minio).
Reviewed-on: #81
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Shows total bytes served from cache (instead of upstream) over the last 30 days. Queries `SUM(size_bytes) WHERE cache_hit = TRUE` from access_log.
Reviewed-on: #65
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
When a helm repo base URL includes a path component (e.g. \`stakater.github.io/stakater-charts\`), the merger was extracting the full URL path (\`stakater-charts/reloader-2.2.8.tgz\`) and the proxy then constructed \`base_url/stakater-charts/reloader-2.2.8.tgz\` = double path = 404.
Fix: \`extractPathRelativeToBase()\` strips the shared base path prefix so only the filename portion is used as the proxy path.
Reviewed-on: #64
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The / endpoint was hardcoded to return 3.0.0-dev. Now uses the git tag version set via ldflags at build time.
Reviewed-on: #63
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Problems
1. Docker daemon sends specific Accept headers to negotiate manifest format, but the proxy dropped them — registries defaulted to OCI format, causing "mediaType should be manifest.v2+json not oci.image.index" errors
2. Upstream Content-Type was only used when the provider returned "application/octet-stream" — Docker manifests got the wrong Content-Type
## Fixes
- Forward client Accept header to upstream (both initial request and Bearer token retry)
- Always prefer upstream Content-Type when present
- Fetch signature now accepts variadic clientHeaders for backwards compat
## E2E tested
- DockerHub: redis:7-alpine, alpine:3 — skopeo inspect OK
- GHCR: OCI-only images work with docker pull (GHCR 404s Docker v2 Accept, which is expected)
- Quay: prometheus/node-exporter — skopeo inspect OK
Reviewed-on: #62
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Vite's \`base: /ui\` makes HTML reference \`/ui/assets/...\` but files are at \`/usr/share/nginx/html/assets/\` (no \`ui/\` subdir). The previous \`location /ui { try_files ... }\` couldn't find the files.
Fix: rewrite strips the base path prefix before try_files, so \`/ui/assets/foo.js\` resolves to \`/usr/share/nginx/html/assets/foo.js\`.
Reviewed-on: #61
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Docker Hub (and other registries) return 401 with a `Www-Authenticate: Bearer realm=...` challenge even for public images. The proxy now:
1. Detects 401 + Bearer challenge
2. Parses realm/service/scope from the header
3. Fetches an anonymous token (or authenticated if username/password configured)
4. Retries the original request with the Bearer token
Fixes: `docker pull artifactapi.../dockerhub/library/redis:latest` returning "unauthorized: upstream returned 401"
Reviewed-on: #60
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The woodpecker docker-buildx plugin expects build_args as a YAML map (KEY: VALUE), not a list (- KEY=VALUE). The list format was silently ignored, so BASE_PATH was never passed to the Docker build.
Reviewed-on: #59
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Serves the UI under /ui instead of /. This pairs with the argocd route simplification (argocd-apps#201) where /ui → UI service and everything else → API.
- Vite: `base` set from `BASE_PATH` env var at build time
- React Router: `basename` set from injected `__BASE_PATH__`
- Nginx: location block uses `${BASE_PATH}`, substituted by sed at build
- Dockerfile: `ARG BASE_PATH=/` (default preserves existing behavior)
- Woodpecker: passes `BASE_PATH=/ui` to docker-web build
Tested: assets serve at `/ui/assets/...`, SPA routing works at `/ui/remotes`, etc.
Reviewed-on: #58
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The v3 Go rewrite removed the /v2/ Docker Registry compatibility endpoint. Docker clients need:
- GET/HEAD /v2/ → 200 (registry ping)
- GET/HEAD /v2/{remoteName}/* → proxy to the docker remote
Usage: `docker pull artifactapi.example.com/{remoteName}/image:tag`
Reviewed-on: #57
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Problem
Helm charts like Intel device plugins have download URLs on `github.com` but the chart index is served from `intel.github.io`. The merger rewrites all URLs through the proxy, constructing:
```
https://artifactapi/api/v1/remote/intel-helm/intel/helm-charts/releases/download/...
```
Which proxies to `https://intel.github.io/helm-charts/intel/helm-charts/releases/download/...` — a 404.
## Fix
Compare the download URL host against the remote's base URL host. If they differ, leave the URL as-is so helm downloads directly from the source. Same-host URLs are still rewritten through the proxy.
Also adds `BaseURL` to `MemberIndex` so the merger has the context it needs, and uses the correct `/local/` vs `/remote/` route prefix.
Reviewed-on: #56
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- New `.pre-commit-config.yaml` with standard Go hooks (gofmt, go vet, go mod tidy) plus file hygiene checks (trailing whitespace, end-of-file, yaml, large files, merge conflicts)
- go vet runs as a local hook with `./...` since the dnephin per-file hook doesn't work with Go module layouts
- Woodpecker pre-commit pipeline updated to use `almalinux9-gobuilder` image with `uvx pre-commit run --all-files`
- Pre-commit hooks installed into the repo
Reviewed-on: #55
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- New "Locals" sidebar nav item with list + detail + browse pages
- Remotes page filters out local repos (repo_type=local hidden)
- LocalDetail: simplified view — just name, type, description + "Browse Files" button
- Virtuals: member links resolve to /locals/ or /remotes/ based on repo_type
- Objects page detects context for correct back-navigation
## Test plan
- [ ] Visual check: locals page shows only local repos
- [ ] Remotes page hides local repos
- [ ] Virtual member links point to correct pages
- [ ] Browse files works from local detail page
Reviewed-on: #54
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben