Compare commits

..

11 Commits

Author SHA1 Message Date
unkin-agent 734195e54e proxy: snapshot in-flight counts before sorting (least_conn selection O(n)) (#124)
ci/woodpecker/tag/docker Pipeline was successful
## Why

`least_conn` mirror selection (`baseURLAttemptOrder`) scaled super-linearly. After rotating the pool by the round-robin cursor it `sort.SliceStable`d with a comparator that called `inflightCounter` on **every comparison** — and each call did a `remoteName+"\x00"+url` concat plus a `sync.Map` `LoadOrStore` with a speculative `new(atomic.Int64)`. So each selection cost O(n·log n) map lookups + allocations, all on the cache-miss/upstream path.

## How

Snapshot each mirror's in-flight count **once**, then sort the snapshot — O(n) map loads, zero comparator allocations.

- Add read-only `inflightCount(name, url) int64`: plain `sync.Map` `Load`, returns 0 when the gauge is absent (no `LoadOrStore`, no speculative allocation).
- `least_conn` branch builds a `{url, count}` snapshot via one `inflightCount` per rotated URL, `sort.SliceStable` by `count` ascending, then extracts the URLs.
- `beginAttempt`/`endAttempt` keep the create-on-write `inflightCounter` path — they legitimately need to create the gauge.

## Numbers (`BenchmarkBaseURLAttemptOrder_LeastConn`, Ryzen 7 4700U, best of 3)

| pool | before ns/op | after ns/op | before allocs | after allocs | before B/op | after B/op |
|------|-------------:|------------:|--------------:|-------------:|------------:|-----------:|
| 3    | ~2516        | ~1492       | 22            | 8            | 474         | 296        |
| 8    | ~14647       | ~3319       | 131–132       | 8            | 2688        | 568        |

Allocs are now **constant** regardless of pool size; pool-8 is ~4.8x faster with ~16x fewer allocations.

## Behavior

Unchanged: least-loaded first, RR rotation as the stable tie-break, `round_robin` and single-URL paths untouched. Pure internal optimization — no API/schema/DB change. Added a multi-mirror tie-break test asserting all-equal load yields the RR rotation; `make test` (`-race`) green, vet/fmt clean.
Reviewed-on: #124
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-13 19:55:21 +10:00
unkin-agent bc8a72e5cc proxy: add mirror-selection benchmarks (#123)
## Why

We added a mirror load-balancing strategy (round_robin / least_conn) in #122 and need hard numbers on whether mirror *selection* adds meaningful request latency. This PR adds microbenchmarks that isolate the selection cost (no network/DB/redis) and commits the results as a permanent regression guard.

## How

`internal/proxy/selection_bench_test.go` builds a zero-value `Engine` (same setup as `leastconn_test.go` / `multibaseurl_test.go`) and calls the selection functions directly:

- `BenchmarkBaseURLAttemptOrder_SingleURL` — the `len<=1` early-return no-op path
- `BenchmarkBaseURLAttemptOrder_RoundRobin` — pools of 3 and 8
- `BenchmarkBaseURLAttemptOrder_LeastConn` — pools of 3 and 8, with in-flight skew preloaded on the gauges
- `BenchmarkBeginEndAttempt` — the gauge inc/dec pair
- `_Parallel` (`RunParallel`) variants of RR / least_conn / begin-end to surface sync.Map + atomic contention

Run: `go test -run=^$ -bench='BaseURLAttemptOrder|BeginEndAttempt' -benchmem -benchtime=1s -count=6 -cpu=8 ./internal/proxy/`

`internal/proxy/BENCHMARKS.md` has the median-of-6 table (Ryzen 7 4700U, go1.26.5) plus the raw runs.

## Headline numbers (median of 6, ns/op — sequential | parallel@8)

| selection | seq ns/op | parallel ns/op | allocs/op |
|---|---:|---:|---:|
| single-URL | ~133 | — | 1 |
| round_robin (3) | ~462 | ~67 | 4 |
| round_robin (8) | ~682 | ~130 | 4 |
| least_conn (3) | ~2690 | ~292 | 22 |
| least_conn (8) | ~15200 | ~1673 | 132 |
| beginAttempt+endAttempt | ~514 | ~72 | 4 |

**Key caveat:** `baseURLAttemptOrder` is only called from `headUpstream` / `fetchFromUpstream` / `checkUpstream` — the cache-miss/upstream path. A cache hit returns `Source: "cache"` before any selection runs, so the **cache-hit hot path pays zero** selection cost regardless of strategy.

**Verdict:** even the worst case (least_conn across 8 mirrors, ~15 µs) is <1% of the multi-millisecond upstream round-trip it accompanies; round_robin (~0.5 µs) is negligible. The strategy adds no meaningful latency. least_conn's cost scales super-linearly because the `sort.SliceStable` comparator re-derives each mirror's gauge via `inflightCounter` (string-concat key + speculative `new(atomic.Int64)`) O(n·log n) times — a possible future micro-opt (snapshot loads before sorting), out of scope here.

Reviewed-on: #123
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-13 17:40:22 +10:00
unkin-agent cd7c2c4383 remotes: add least-connections mirror strategy (round-robin remains default) (#122)
ci/woodpecker/tag/docker Pipeline was successful
## Why

The mirrorlist (PR #121) always load-balances round-robin. Round-robin is oblivious to how busy each mirror is, so a slow or saturated mirror keeps getting its fair share of new requests. This adds an opt-in **least-connections** strategy that favors the mirror currently handling the fewest in-flight requests, steering new work toward idle mirrors. Round-robin stays the default, so existing remotes are unchanged.

## How

- **Model**: `models.Remote` gains `mirror_strategy` (`round_robin` default/empty, or `least_conn`). `ValidateMirrorStrategy` checks the enum and requires a non-empty mirrorlist for `least_conn`. Empty behaves as `round_robin` for back-compat.
- **DB**: additive `mirror_strategy TEXT NOT NULL DEFAULT 'round_robin'` column (CREATE TABLE + `ADD COLUMN IF NOT EXISTS`), wired through remoteCols/scanRemote/CreateRemote/UpdateRemote; empty normalized to `round_robin` on write.
- **Engine**: a per-remote, per-pool-URL atomic in-flight gauge is incremented around each upstream call (head/fetch/checkUpstream). For a `least_conn` remote the attempt order starts with the least-loaded pool URL (ties broken by the existing round-robin rotation). Round-robin path and failover order are unchanged; single-URL pools are a no-op.
- **Tests**: unit tests for least-loaded selection, round-robin default, gauge inc/dec, single-URL no-op, and validation; DB round-trip covers the new column; docker e2e adds a `least_conn` distribution test plus a real `dnf` install through a `least_conn` remote.

Back-compat: unset/empty `mirror_strategy` is `round_robin`, so all existing remotes keep their current behavior.
Reviewed-on: #122
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-13 17:23:59 +10:00
unkin-agent f1820fd104 remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk) (#121)
## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.

**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).

## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.

## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.

## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.

Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-13 16:08:55 +10:00
unkin-agent 822f356881 remotes: flush cached metadata when a remote's base_url changes (#120)
## Why

Switching a remote's backend (`base_url`) left artifactapi serving the previously-cached mutable metadata (repodata / Release / APKINDEX) until TTL expiry, so requests could keep pointing at the old upstream. `cache.FlushRemote` already existed but was wired to nothing.

## How

- Inject a `MetadataFlusher` (satisfied by `*cache.Redis`) into `RemotesHandler` via `NewRemotesHandler`; `server.go` passes `s.cache`.
- On update, read the existing remote first, then after a successful DB update flush the remote's cached metadata when `base_url` changed, so the next request re-fetches fresh from the new upstream.
- Keep scope to `base_url` (upstream identity); a flush failure is logged as a warning and does not fail the request since the DB update already landed.
- Add tests: a `base_url` change flushes exactly once, an unchanged `base_url` does not flush, and a flush error still returns 200.

Reviewed-on: #120
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-13 08:29:24 +10:00
unkin-agent 73c0bfc670 deb/apk: make local repodata deterministic (#119)
ci/woodpecker/tag/docker Pipeline was successful
Part of #117. Local generated repodata must be byte-identical across the two no-affinity replicas and across every regeneration, so apt/apk never hit a checksum mismatch between an index's advertised hash and the bytes actually served. This does the deb+apk half (the rpm half landed in #118).

How:
- Derive the deb `Release` `Date:` from the newest persisted `created_at` (RFC1123Z, UTC) instead of `time.Now()`; carry `created_at` through the deb metadata SELECT and `DebMetadata`. An empty repo falls back to the Unix epoch. This also stops `Date:` running ahead of wall clock.
- Pin the apk `APKINDEX` tar header `ModTime` to the Unix epoch instead of the zero-value `time.Time`, so it is never wall-clock derived.
- Give both list queries a genuine total order by adding a `file_path` tiebreak (name/version/arch is not unique).
- Add guard tests: deb generators byte-identical across generations; the `Release` checksum/size matches the served `Packages`/`Packages.gz` bytes (the exact apt invariant); `Date:` pinned to `created_at`; apk index byte-identical and tar `ModTime` pinned to epoch.

Reviewed-on: #119
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 23:32:07 +10:00
unkin-agent a8aa0c231b rpm: make local repodata deterministic (fixes #117) (#118)
Local RPM repos regenerated `repomd.xml` on every request and advertised a `primary.xml.gz` sha256 that drifted every second, because `time.Now().Unix()` was embedded inside the gzipped `primary.xml` (and in `repomd` `<revision>`/`<timestamp>`). The advertised hash therefore never matched the content-addressed `<sha256>-primary.xml.gz` bytes a second later or on the other replica, so `dnf` failed with a checksum mismatch. Part of #117.

How:
- Derives `<time file=>` in `primary.xml` from the persisted `rpm_metadata.created_at` instead of the wall clock; unset timestamps collapse to a fixed `0`.
- Derives `repomd` `<revision>`/`<timestamp>` from the newest package upload time, so `repomd.xml` is byte-identical across replicas and requests.
- Adds `file_path` as a total-order tiebreak to the metadata `ORDER BY`.
- Pins the gzip header (`OS: 255`) so compressed bytes depend only on the payload.
- Adds regression tests: generators are byte-identical across two runs, and the sha256 in `repomd.xml` equals the sha256 of the bytes each `serve*` handler returns.

Reviewed-on: #118
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 23:14:51 +10:00
unkin-agent 0e26d99228 ui: add alpine local and github_alpine usage instructions (#116)
ci/woodpecker/tag/docker Pipeline was successful
## Why
The alpine local repo (a real apk repo with auto-generated per-arch APKINDEX) and the new github_alpine remote (metadata-only apk index synthesized from a GitHub repo's release .apk assets) shipped without any "How do I use this?" UI guidance, unlike deb/github_deb. This adds the matching client instructions so users can consume and publish to these repos.

## How
- Extend `case 'alpine':` in `UsageInstructions.tsx` to branch on `isLocal` (mirroring rpm/deb):
  - LOCAL: "Add the apk repo" snippet appends `${url}/api/v1/local/<name>` to `/etc/apk/repositories` with `apk update/add --allow-untrusted` (served unsigned, parity with rpm gpgcheck=0), plus a "Publish a .apk" snippet uploading to the canonical `<arch>/<name>-<version>.apk` path (APKINDEX carries no filename, so path must match or install 404s).
  - REMOTE: existing caching-proxy snippet kept unchanged.
- Add `case 'github_alpine':` (always remote-class): "Add the apk repo (metadata-only, from GitHub releases)" snippet with `--allow-untrusted`; note explains the per-arch APKINDEX is synthesized from release .apk assets and downloads redirect to the backing releases remote.
- No other cases touched. `npm run build` (tsc typecheck + vite) passes; pre-commit passes.

Reviewed-on: #116
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 20:53:23 +10:00
unkin-agent 5a06c16797 Add github_alpine metadata-only package type (#115)
## Why

`github_alpine` is the Alpine/apk analog of the existing `github_deb`/`github_rpm` metadata-only remotes. It lets a plain GitHub-releases repo of `.apk` files be consumed as a real apk repository without artifactapi ever precaching the packages: it scans the repo's releases, derives each package's `.PKGINFO` from a ranged prefix fetch, synthesizes a per-arch `APKINDEX.tar.gz` from the cached metadata, and redirects the actual `.apk` downloads to a backend `releases_remote`. It stacks on the apk-local branch, reusing that work's alpine APKINDEX generator, `.apk`/`.PKGINFO` parser, Q1 pull-checksum, and `AlpineMetadata` store.

## How

- **`pkg/models`**: add `PackageGitHubAlpine` to the enum + validators (and test).
- **`internal/provider/alpine/github.go`**: the `github_alpine` provider. `ServeRemote` serves per-arch `<arch>/APKINDEX.tar.gz` (reusing `generateAPKIndex` over arch-filtered `AlpineMetadata` rows, `normalizeIndexPath` for apk's `./` dot-segment), 302-redirects `*.apk` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`, and cold-starts with a 503 + `Retry-After`. `scanWithState` lists releases with ETag/If-None-Match and incrementally derives/prunes. `deriveAsset` does a **ranged GET of just the front of the `.apk`** — the control gzip stream carrying `.PKGINFO` sits near the front — doubling the range on truncation; it parses `.PKGINFO` and computes the `C:` Q1 checksum (`Q1`+base64(sha1(control stream))). `FilePath` = the github-relative asset path so the redirect resolves.
- **`internal/provider/alpine/syncer.go`**: a parallel background `Syncer` (own worker pool, shared rate limiter, deduped queue, DB lease), separate from the deb/rpm syncers.
- **`internal/database/alpine_github_sync.go`** + `github_alpine_sync_state` table: `ListGitHubAlpineRemotes` + Claim/Release per-remote sync lease, kept separate from the deb/rpm tables.
- **`internal/server/server.go`**: construct + `Run` the alpine syncer alongside deb/rpm and register it in the `PackageType→Primer` map (priming on create then flows through the existing generic `remotes.go` path).

The rpm/deb providers, syncers, and tables are untouched — this adds parallel alpine equivalents and reuses shared helpers already present in the alpine package.

## Tests

Mirror the deb github tests: scan derives `.PKGINFO`/Q1 from a ranged prefix of a `testsupport.MinimalApk` served over an httptest range server (no full download); diff/prune; pattern filter; per-arch `ServeRemote` routing (index served per-arch and grouped, dot-segment collapse, `.apk` → 302, cold-start 503, warm 200, canceled-request-serves-cache); DB lease prevents a second replica; prime bypasses the recency window. `go build`/`go vet`/`go mod tidy` clean, `make test` (-race) green, pre-commit green.

Reviewed-on: #115
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 20:48:42 +10:00
unkin-agent 7f77666709 Add Alpine/apk local repository support (#114)
## Why

The alpine provider only supported remote (proxy) repositories, so there was no way to publish first-party `.apk` packages the way `rpm-local` and `deb-local` already allow. This extends the existing alpine provider into a real apk repository: uploaded `.apk` files are parsed in pure Go and a per-arch `APKINDEX.tar.gz` is generated on demand, at parity with rpm repodata and deb Packages generation. (The metadata-only `github_alpine` type is a separate follow-up and is not part of this PR.)

## How

- Implements `LocalUploader` / `LocalIndexer` / `PostUploadHook` / `PostDeleteHook` on the existing `alpine` provider, leaving the remote proxy methods (`UpstreamURL`/`ContentType`/`AuthHeaders`/`RewriteResponse`/`Classify`) intact.
- Parses the `.apk` (up to three concatenated, independently gzipped tar streams) in pure Go: locates the control stream by its `.PKGINFO` member, reads the `key = value` fields, and computes the apk pull checksum `C:` = `Q1` + base64(sha1(**control gzip stream bytes**)) — the sha1 of the second gzip member, not of the whole file.
- Derives arch from `.PKGINFO` and records download size (`S:` blob size) and installed size (`I:` from `.PKGINFO size`).
- Generates an **unsigned** per-arch `APKINDEX.tar.gz` = gzip(tar(`APKINDEX`)) filtered by requested arch (clients use `--allow-untrusted`, matching rpm `gpgcheck=0` / deb `[trusted=yes]`), applying the same dot-segment normalization as deb so `./<arch>/APKINDEX.tar.gz` resolves. Non-index / `.apk` paths return `false` so the generic file streamer serves the stored blob.
- Adds `AlpineMetadata` plus **separate** `AlpineMetadataStore` / `AlpineMetadataReader` / `AlpineMetadataDeleter` interfaces (type-asserted from the generic hooks) so the shared rpm/deb metadata interfaces and their test doubles are untouched.
- Adds the `alpine_metadata` table (keyed by `repo_name` + `file_path`, per-arch index) and its `Insert`/`Delete`/`List` DB methods.
- Adds `testsupport.MinimalApk`, unit tests (`.PKGINFO` parse, Q1 checksum over the control stream, per-arch filtering, empty-field omission, `./` dot-segment handling, ValidateUpload accept/reject), and a `dockere2e` `TestLocalAlpineIndex`.

## Consumption

`/etc/apk/repositories` line = `<url>/api/v1/local/<name>` (apk appends `/<arch>/APKINDEX.tar.gz`); `apk update --allow-untrusted && apk add --allow-untrusted <pkg>`. Packages live at `/api/v1/local/<name>/<arch>/<file>.apk`.

## Verification

`go build ./...`, `go vet ./...` (incl. `-tags dockere2e`), `go mod tidy` (no change), `make test` (`-race`), and `pre-commit run --all-files` all pass.

Reviewed-on: #114
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 20:39:27 +10:00
unkin-agent 26c37024ae ui: add deb and github_deb usage instructions (#113)
ci/woodpecker/tag/docker Pipeline was successful
## Why

The `deb` and `github_deb` repository types had no tailored "How do I use this?" snippets in the UI — they fell through to the generic default (curl-a-file-by-path), which does not tell a user how to actually wire the repo into apt. This adds real, end-to-end-validated apt instructions mirroring how the existing package types (rpm, npm, etc.) are handled.

## How

- Adds a `case 'deb':` to `buildSnippets` that branches on `isLocal`: a flat real apt repo served at `/api/v1/local/<name>/ ./` with `[trusted=yes]` plus a publish-a-.deb snippet for locals, and a caching-proxy `deb <proxy> <suite> <component>` form (upstream-signed, no `[trusted=yes]`) for remotes.
- Adds a `case 'github_deb':` emitting the metadata-only flat form `deb [trusted=yes] <proxy>/ ./`, whose index is synthesized from a GitHub repo's release .deb assets.
- Reuses the existing `url`/`proxy`/`isLocal` helpers and matches the surrounding case style; leaves the rpm and other cases untouched.

Reviewed-on: #113
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-11 23:54:34 +10:00
50 changed files with 4966 additions and 65 deletions
+16
View File
@@ -9,6 +9,18 @@ services:
# No host port needed: only the artifactapi container talks to it, and the # No host port needed: only the artifactapi container talks to it, and the
# tests compare served bytes against the on-disk fixtures. # tests compare served bytes against the on-disk fixtures.
# Two constant-body upstreams for the multi-base_url suite: each returns a
# distinct, upstream-identifying body for any path, so round-robin
# distribution across a two-mirror remote is directly observable.
mockupstreama:
image: nginx:alpine
volumes:
- ./e2e-docker/mirror-conf/a.conf:/etc/nginx/conf.d/default.conf:ro,z
mockupstreamb:
image: nginx:alpine
volumes:
- ./e2e-docker/mirror-conf/b.conf:/etc/nginx/conf.d/default.conf:ro,z
artifactapi: artifactapi:
# The host port is set via ARTIFACTAPI_PORT (see scripts/docker-e2e.sh), # The host port is set via ARTIFACTAPI_PORT (see scripts/docker-e2e.sh),
# defaulting to 8000; the e2e run uses 8001 to avoid colliding with a # defaulting to 8000; the e2e run uses 8001 to avoid colliding with a
@@ -16,3 +28,7 @@ services:
depends_on: depends_on:
mockupstream: mockupstream:
condition: service_started condition: service_started
mockupstreama:
condition: service_started
mockupstreamb:
condition: service_started
+12
View File
@@ -30,6 +30,18 @@ already-running stack.
index), rpm (real package + **automatic repodata** generation). index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge - **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge
across two members. across two members.
- **Mirrorlist** — an rpm remote with a `mirrorlist` of extra upstream mirrors
(pool = `base_url` + `mirrorlist`): round-robin distribution across both mirrors
(constant-body `mockupstreama` / `mockupstreamb`), failover past a dead primary,
no-mirrorlist regression, and a real `dnf` (stock `rockylinux:9` container)
`makecache` + `install` through a two-mirror rpm remote whose `base_url` is dead
— a dead mirror must not break the client.
- **Mirror strategy (`least_conn`)** — a `mirror_strategy: least_conn` rpm remote
over the two constant-body mirrors exercises the least-connections selection
path end-to-end (both mirrors serve, all requests succeed), plus a real `dnf`
install through a `least_conn` remote with a dead primary (failover unchanged).
The precise least-loaded pick is asserted deterministically in the proxy unit
test, since an in-flight-skew assertion over HTTP is timing-sensitive.
## Fixtures ## Fixtures
@@ -0,0 +1,55 @@
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
<revision>1786573032</revision>
<data type="primary">
<checksum type="sha256">d82f717e4da1afe96b8e7857de9e852f5785c0d75734e50d0f8afdcbc6261b08</checksum>
<open-checksum type="sha256">3345bb631380ae6c0620fe2a29cf7dff2ed4e28c5cb06c91e8a1628ba1979bcb</open-checksum>
<location href="repodata/d82f717e4da1afe96b8e7857de9e852f5785c0d75734e50d0f8afdcbc6261b08-primary.xml.gz"/>
<timestamp>1786573032</timestamp>
<size>631</size>
<open-size>1192</open-size>
</data>
<data type="filelists">
<checksum type="sha256">daa313cc5eeb7df556e1d4885d7701b10b9f012f436ef239fa46827f966222be</checksum>
<open-checksum type="sha256">648bd0ce00fda09abbc6e9c3ff3278518a76f258576ac24cd10c12e41e0e5bd7</open-checksum>
<location href="repodata/daa313cc5eeb7df556e1d4885d7701b10b9f012f436ef239fa46827f966222be-filelists.xml.gz"/>
<timestamp>1786573032</timestamp>
<size>256</size>
<open-size>338</open-size>
</data>
<data type="other">
<checksum type="sha256">8510c74a6f288828bbc92abee5d0d8ae9687d3c31a2579ea95e31a4c3a320d85</checksum>
<open-checksum type="sha256">c42cfd3843e9c53a60ad84bded44aa46c65faae7b3da99a099a9ca0b018872a9</open-checksum>
<location href="repodata/8510c74a6f288828bbc92abee5d0d8ae9687d3c31a2579ea95e31a4c3a320d85-other.xml.gz"/>
<timestamp>1786573032</timestamp>
<size>296</size>
<open-size>399</open-size>
</data>
<data type="primary_db">
<checksum type="sha256">f6bd7755da13d9726381048f467992869104a4c5521338ef740dc35eb85b9b71</checksum>
<open-checksum type="sha256">c45c85d12ccb0f8172b7bfae466362c08ac1867559574a9fb9cb2118c04daddb</open-checksum>
<location href="repodata/f6bd7755da13d9726381048f467992869104a4c5521338ef740dc35eb85b9b71-primary.sqlite.bz2"/>
<timestamp>1786573032</timestamp>
<size>1740</size>
<open-size>106496</open-size>
<database_version>10</database_version>
</data>
<data type="filelists_db">
<checksum type="sha256">be3c6e4c7a13ece48bd5d6a4d6d5e6a2395fe006ef9c5f217f5b87144f465e57</checksum>
<open-checksum type="sha256">1ccfa3dff532d782ce3225aae807506a4ce4534291386f1c47455dcc6b70cfd6</open-checksum>
<location href="repodata/be3c6e4c7a13ece48bd5d6a4d6d5e6a2395fe006ef9c5f217f5b87144f465e57-filelists.sqlite.bz2"/>
<timestamp>1786573032</timestamp>
<size>764</size>
<open-size>28672</open-size>
<database_version>10</database_version>
</data>
<data type="other_db">
<checksum type="sha256">ba593cd8ab5ec1e127888707c1fd882920996f1ce173fd7a589d647918fd7da4</checksum>
<open-checksum type="sha256">5d4d38380f0e359bfc0a50033d4faa84c75c5ae8fe2ef11c1c82d64748e9b8e2</open-checksum>
<location href="repodata/ba593cd8ab5ec1e127888707c1fd882920996f1ce173fd7a589d647918fd7da4-other.sqlite.bz2"/>
<timestamp>1786573032</timestamp>
<size>738</size>
<open-size>24576</open-size>
<database_version>10</database_version>
</data>
</repomd>
+105
View File
@@ -0,0 +1,105 @@
//go:build dockere2e
package e2edocker
import (
"fmt"
"net/http"
"os"
"os/exec"
"strings"
"testing"
)
// TestLeastConnMultiBaseURL configures an rpm remote with mirror_strategy =
// least_conn over a two-mirror pool and drives distinct cache-miss paths through
// it, asserting every request succeeds and both mirrors serve traffic. This
// exercises the least-connections selection path (leastConnOrder + the in-flight
// gauge inc/dec around each upstream call) end-to-end through a real HTTP client.
// A precise least-loaded assertion is timing-sensitive over HTTP and is covered
// deterministically by the proxy unit test (TestLeastConnPicksLeastLoaded);
// here, with requests issued serially, in-flight counts return to zero between
// them so equal-load mirrors are spread by the round-robin tie-break.
func TestLeastConnMultiBaseURL(t *testing.T) {
name := "e2e-leastconn"
createRepo(t, fmt.Sprintf(`{
"name": %q,
"package_type": "rpm",
"repo_type": "remote",
"base_url": %q,
"mirrorlist": [%q],
"mirror_strategy": "least_conn",
"stale_on_error": false
}`, name, mockUpstreamA(), mockUpstreamB()))
defer deleteRepo(t, name)
seenA, seenB := false, false
const n = 12
for i := 0; i < n; i++ {
url := api(fmt.Sprintf("/api/v1/remote/%s/lc/%d", name, i))
resp, body := doRequest(t, http.MethodGet, url, nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("request %d: status %d: %s", i, resp.StatusCode, body)
}
switch strings.TrimSpace(string(body)) {
case "UPSTREAM-A":
seenA = true
case "UPSTREAM-B":
seenB = true
default:
t.Fatalf("request %d: unexpected body %q", i, body)
}
}
if !seenA || !seenB {
t.Fatalf("least_conn remote did not reach both upstreams: A=%v B=%v", seenA, seenB)
}
}
// TestLeastConnDnfInstall drives a real dnf (stock rockylinux container) at a
// two-mirror rpm remote configured with mirror_strategy = least_conn whose
// primary base_url is dead: makecache + install must succeed via the live mirror.
// This proves a real package-manager client installs correctly through a
// least_conn remote and that failover semantics are unchanged under the new
// strategy. Requires the compose network exported by scripts/docker-e2e.sh.
func TestLeastConnDnfInstall(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := os.Getenv("ARTIFACTAPI_INTERNAL")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
name := "e2e-leastconn-dnf"
createRepo(t, fmt.Sprintf(`{
"name": %q,
"package_type": "rpm",
"repo_type": "remote",
"base_url": "http://mockupstream-dead:80",
"mirrorlist": [%q],
"mirror_strategy": "least_conn",
"stale_on_error": false
}`, name, mockUpstream()))
defer deleteRepo(t, name)
repoURL := strings.TrimRight(internal, "/") + "/api/v1/remote/" + name + "/rpm-mirror"
repoConf := fmt.Sprintf("[dnflc]\nname=dnflc\nbaseurl=%s\nenabled=1\ngpgcheck=0\nsslverify=0\nmetadata_expire=0\n", repoURL)
script := "set -euo pipefail; " +
"printf '%s' \"$REPO\" > /etc/yum.repos.d/dnflc.repo; " +
"dnf -y --disablerepo='*' --enablerepo=dnflc makecache; " +
"dnf -y --disablerepo='*' --enablerepo=dnflc install e2e-testpkg; " +
"rpm -q e2e-testpkg"
cmd := exec.Command("docker", "run", "--rm",
"--network", network,
"-e", "REPO="+repoConf,
"rockylinux:9", "bash", "-c", script)
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("real dnf install through a least_conn remote failed: %v\n%s", err, out)
}
if !strings.Contains(string(out), "e2e-testpkg-1.0-1") {
t.Fatalf("dnf did not install the expected package via least_conn remote; output:\n%s", out)
}
}
+55
View File
@@ -3,7 +3,10 @@
package e2edocker package e2edocker
import ( import (
"archive/tar"
"bytes" "bytes"
"compress/gzip"
"io"
"net/http" "net/http"
"strings" "strings"
"testing" "testing"
@@ -136,3 +139,55 @@ func TestLocalDebRepo(t *testing.T) {
t.Fatalf("deb content mismatch") t.Fatalf("deb content mismatch")
} }
} }
// TestLocalAlpineIndex uploads an .apk to an alpine local repo and validates
// that a per-arch APKINDEX.tar.gz is generated automatically from the parsed
// .PKGINFO (the apk-local analog of rpm repodata / deb Packages generation).
func TestLocalAlpineIndex(t *testing.T) {
createRepo(t, `{"name":"local-alpine","package_type":"alpine","repo_type":"local"}`)
defer deleteRepo(t, "local-alpine")
apk := testsupport.MinimalApk("e2e-testpkg", "1.0-r0", "x86_64")
uploadFile(t, "local-alpine", "x86_64/e2e-testpkg-1.0-r0.apk", apk, "application/vnd.android.package-archive")
// The index is generated asynchronously after upload; poll for it.
resp, body := getEventually(t, api("/api/v1/local/local-alpine/x86_64/APKINDEX.tar.gz"), 15*time.Second)
if resp.StatusCode != http.StatusOK {
t.Fatalf("APKINDEX: status %d: %s", resp.StatusCode, body)
}
zr, err := gzip.NewReader(bytes.NewReader(body))
if err != nil {
t.Fatalf("APKINDEX not gzip: %v", err)
}
tarBytes, _ := io.ReadAll(zr)
tr := tar.NewReader(bytes.NewReader(tarBytes))
var index string
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("APKINDEX not tar: %v", err)
}
if hdr.Name == "APKINDEX" {
b, _ := io.ReadAll(tr)
index = string(b)
}
}
for _, want := range []string{"P:e2e-testpkg", "V:1.0-r0", "A:x86_64", "C:Q1", "S:", "I:"} {
if !strings.Contains(index, want) {
t.Fatalf("APKINDEX missing %q:\n%s", want, index)
}
}
// The .apk downloads back byte-identical from its arch path.
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-alpine/x86_64/e2e-testpkg-1.0-r0.apk"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("download apk: status %d: %s", resp.StatusCode, body)
}
if !bytes.Equal(body, apk) {
t.Fatalf("apk content mismatch")
}
}
+9
View File
@@ -0,0 +1,9 @@
# Mock upstream A for the multi-base_url e2e: any path returns a constant,
# upstream-identifying body so round-robin distribution is observable.
server {
listen 80;
location / {
default_type text/plain;
return 200 "UPSTREAM-A";
}
}
+8
View File
@@ -0,0 +1,8 @@
# Mock upstream B for the multi-base_url e2e (see a.conf).
server {
listen 80;
location / {
default_type text/plain;
return 200 "UPSTREAM-B";
}
}
+163
View File
@@ -0,0 +1,163 @@
//go:build dockere2e
package e2edocker
import (
"fmt"
"net/http"
"os"
"os/exec"
"strings"
"testing"
)
// mockUpstreamA/B are the constant-body upstreams (see docker-compose.e2e.yml)
// that let the round-robin test observe which mirror served each request.
func mockUpstreamA() string {
if v := os.Getenv("MOCK_UPSTREAM_A_INTERNAL"); v != "" {
return strings.TrimRight(v, "/")
}
return "http://mockupstreama"
}
func mockUpstreamB() string {
if v := os.Getenv("MOCK_UPSTREAM_B_INTERNAL"); v != "" {
return strings.TrimRight(v, "/")
}
return "http://mockupstreamb"
}
// TestMultiBaseURLRoundRobin configures an rpm remote with base_url = mirror A
// and mirrorlist = [mirror B] and drives distinct paths through it, asserting
// both mirrors serve traffic. Each path is a cache miss, so every request reaches
// upstream and the round-robin cursor alternates mirrors.
func TestMultiBaseURLRoundRobin(t *testing.T) {
name := "e2e-rr"
createRepo(t, fmt.Sprintf(`{
"name": %q,
"package_type": "rpm",
"repo_type": "remote",
"base_url": %q,
"mirrorlist": [%q],
"stale_on_error": false
}`, name, mockUpstreamA(), mockUpstreamB()))
defer deleteRepo(t, name)
seenA, seenB := false, false
const n = 12
for i := 0; i < n; i++ {
url := api(fmt.Sprintf("/api/v1/remote/%s/rr/%d", name, i))
resp, body := doRequest(t, http.MethodGet, url, nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("request %d: status %d: %s", i, resp.StatusCode, body)
}
switch strings.TrimSpace(string(body)) {
case "UPSTREAM-A":
seenA = true
case "UPSTREAM-B":
seenB = true
default:
t.Fatalf("request %d: unexpected body %q", i, body)
}
}
if !seenA || !seenB {
t.Fatalf("round-robin did not reach both upstreams: A=%v B=%v", seenA, seenB)
}
}
// TestMultiBaseURLFailover points a two-mirror remote at a dead primary and a
// healthy secondary and asserts every request still succeeds via the secondary.
func TestMultiBaseURLFailover(t *testing.T) {
name := "e2e-failover"
createRepo(t, fmt.Sprintf(`{
"name": %q,
"package_type": "rpm",
"repo_type": "remote",
"base_url": "http://mockupstream-dead:80",
"mirrorlist": [%q],
"stale_on_error": false
}`, name, mockUpstreamB()))
defer deleteRepo(t, name)
for i := 0; i < 6; i++ {
url := api(fmt.Sprintf("/api/v1/remote/%s/fo/%d", name, i))
resp, body := doRequest(t, http.MethodGet, url, nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("request %d: dead primary broke fetch: status %d: %s", i, resp.StatusCode, body)
}
if got := strings.TrimSpace(string(body)); got != "UPSTREAM-B" {
t.Fatalf("request %d: body %q, want UPSTREAM-B (served via failover)", i, got)
}
}
}
// TestSingleBaseURLRegression asserts a remote with no mirrorlist works exactly
// as before the mirrorlist change.
func TestSingleBaseURLRegression(t *testing.T) {
name := "e2e-single"
createRepo(t, fmt.Sprintf(`{
"name": %q,
"package_type": "rpm",
"repo_type": "remote",
"base_url": %q,
"stale_on_error": false
}`, name, mockUpstreamA()))
defer deleteRepo(t, name)
resp, body := doRequest(t, http.MethodGet, api("/api/v1/remote/"+name+"/solo/0"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("single-url fetch: status %d: %s", resp.StatusCode, body)
}
if got := strings.TrimSpace(string(body)); got != "UPSTREAM-A" {
t.Fatalf("single-url body %q, want UPSTREAM-A", got)
}
}
// TestMultiBaseURLDnfFailover drives a real dnf (stock rockylinux container) at
// a two-mirror rpm remote whose primary is dead: makecache + install must
// succeed via the live secondary mirror, proving a dead mirror does not break a
// real package-manager client. Requires the compose network and internal API
// URL exported by scripts/docker-e2e.sh; skipped when run standalone.
func TestMultiBaseURLDnfFailover(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := os.Getenv("ARTIFACTAPI_INTERNAL")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
name := "e2e-dnf-failover"
// Primary base_url is dead; the live mirror serves the real yum repo under
// fixtures/rpm-mirror via the shared mock upstream.
createRepo(t, fmt.Sprintf(`{
"name": %q,
"package_type": "rpm",
"repo_type": "remote",
"base_url": "http://mockupstream-dead:80",
"mirrorlist": [%q],
"stale_on_error": false
}`, name, mockUpstream()))
defer deleteRepo(t, name)
repoURL := strings.TrimRight(internal, "/") + "/api/v1/remote/" + name + "/rpm-mirror"
repoConf := fmt.Sprintf("[dnffo]\nname=dnffo\nbaseurl=%s\nenabled=1\ngpgcheck=0\nsslverify=0\nmetadata_expire=0\n", repoURL)
script := "set -euo pipefail; " +
"printf '%s' \"$REPO\" > /etc/yum.repos.d/dnffo.repo; " +
"dnf -y --disablerepo='*' --enablerepo=dnffo makecache; " +
"dnf -y --disablerepo='*' --enablerepo=dnffo install e2e-testpkg; " +
"rpm -q e2e-testpkg"
cmd := exec.Command("docker", "run", "--rm",
"--network", network,
"-e", "REPO="+repoConf,
"rockylinux:9", "bash", "-c", script)
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("real dnf install through a dead primary mirror failed: %v\n%s", err, out)
}
if !strings.Contains(string(out), "e2e-testpkg-1.0-1") {
t.Fatalf("dnf did not install the expected package via failover; output:\n%s", out)
}
}
+1 -1
View File
@@ -57,7 +57,7 @@ func do(t *testing.T, h http.Handler, method, path, body string) int {
} }
func TestRemotesErrorPaths(t *testing.T) { func TestRemotesErrorPaths(t *testing.T) {
h := NewRemotesHandler(closedDB(t), nil).Routes() h := NewRemotesHandler(closedDB(t), nil, nil).Routes()
if c := do(t, h, "GET", "/", ""); c != 500 { if c := do(t, h, "GET", "/", ""); c != 500 {
t.Errorf("list with dead db = %d, want 500", c) t.Errorf("list with dead db = %d, want 500", c)
} }
+50 -4
View File
@@ -1,8 +1,10 @@
package v2 package v2
import ( import (
"context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"log/slog"
"net/http" "net/http"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
@@ -17,15 +19,23 @@ type Primer interface {
EnqueuePrime(remote models.Remote) EnqueuePrime(remote models.Remote)
} }
// MetadataFlusher purges a remote's cached mutable metadata (repodata / Release
// / APKINDEX freshness keys). *cache.Redis satisfies it.
type MetadataFlusher interface {
FlushRemote(ctx context.Context, remote string) error
}
type RemotesHandler struct { type RemotesHandler struct {
db *database.DB db *database.DB
cache MetadataFlusher
primers map[models.PackageType]Primer primers map[models.PackageType]Primer
} }
// NewRemotesHandler wires the handler to the per-type metadata primers. primers // NewRemotesHandler wires the handler to the metadata cache and per-type
// may be nil; a package type with no registered primer simply skips priming. // primers. cache may be nil (flush-on-backend-change is skipped); primers may
func NewRemotesHandler(db *database.DB, primers map[models.PackageType]Primer) *RemotesHandler { // be nil (a package type with no registered primer simply skips priming).
return &RemotesHandler{db: db, primers: primers} func NewRemotesHandler(db *database.DB, cache MetadataFlusher, primers map[models.PackageType]Primer) *RemotesHandler {
return &RemotesHandler{db: db, cache: cache, primers: primers}
} }
func (h *RemotesHandler) Routes() chi.Router { func (h *RemotesHandler) Routes() chi.Router {
@@ -78,6 +88,14 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
http.Error(w, "base_url is required for remote repositories", http.StatusBadRequest) http.Error(w, "base_url is required for remote repositories", http.StatusBadRequest)
return return
} }
if err := remote.ValidateMirrorlist(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := remote.ValidateMirrorStrategy(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := remote.ValidatePatterns(); err != nil { if err := remote.ValidatePatterns(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest) http.Error(w, err.Error(), http.StatusBadRequest)
return return
@@ -102,14 +120,42 @@ func (h *RemotesHandler) update(w http.ResponseWriter, r *http.Request) {
return return
} }
remote.Name = name remote.Name = name
if err := remote.ValidateMirrorlist(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := remote.ValidateMirrorStrategy(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := remote.ValidatePatterns(); err != nil { if err := remote.ValidatePatterns(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest) http.Error(w, err.Error(), http.StatusBadRequest)
return return
} }
// Capture the current backend before the update so we can tell whether the
// remote's base_url (its upstream) changed. A read failure just means we
// skip the freshness flush; it must not block the update.
oldBaseURL, oldKnown := "", false
if existing, err := h.db.GetRemote(r.Context(), name); err == nil {
oldBaseURL, oldKnown = existing.BaseURL, true
}
if err := h.db.UpdateRemote(r.Context(), &remote); err != nil { if err := h.db.UpdateRemote(r.Context(), &remote); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError) http.Error(w, err.Error(), http.StatusInternalServerError)
return return
} }
// Changing the backend invalidates any cached mutable metadata (repodata /
// Release / APKINDEX): purge it so the next request re-fetches from the new
// upstream instead of serving stale data until TTL expiry. A flush failure
// is logged but does not fail the request — the DB update already landed.
if oldKnown && oldBaseURL != remote.BaseURL && h.cache != nil {
if err := h.cache.FlushRemote(r.Context(), name); err != nil {
slog.Warn("flush cached metadata after base_url change failed",
"remote", name, "error", err)
} else {
slog.Info("flushed cached metadata after base_url change",
"remote", name, "old_base_url", oldBaseURL, "new_base_url", remote.BaseURL)
}
}
writeJSON(w, http.StatusOK, remote) writeJSON(w, http.StatusOK, remote)
} }
+96
View File
@@ -0,0 +1,96 @@
package v2
import (
"context"
"errors"
"testing"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeFlusher records FlushRemote calls so a test can assert whether — and how
// often — a remote's cached metadata was purged.
type fakeFlusher struct {
calls []string
err error
}
func (f *fakeFlusher) FlushRemote(_ context.Context, remote string) error {
f.calls = append(f.calls, remote)
return f.err
}
func seedRemote(t *testing.T, db *database.DB, name, baseURL string) {
t.Helper()
err := db.CreateRemote(context.Background(), &models.Remote{
Name: name,
PackageType: models.PackageRPM,
RepoType: models.RepoTypeRemote,
BaseURL: baseURL,
})
if err != nil {
t.Fatalf("seed remote: %v", err)
}
}
// A base_url change must flush the remote's cached metadata exactly once, while
// an update that leaves base_url untouched must not flush at all.
func TestUpdateFlushesCacheOnBaseURLChange(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
const name = "rpm-flush-change"
seedRemote(t, db, name, "https://old.example.com/repo")
ff := &fakeFlusher{}
h := NewRemotesHandler(db, ff, nil).Routes()
if c := do(t, h, "PUT", "/"+name, `{"package_type":"rpm","repo_type":"remote","base_url":"https://new.example.com/repo"}`); c != 200 {
t.Fatalf("update (backend change) = %d, want 200", c)
}
if len(ff.calls) != 1 || ff.calls[0] != name {
t.Fatalf("flush calls = %v, want exactly one flush of %q", ff.calls, name)
}
// Re-updating with the same (now current) base_url must not flush again.
ff.calls = nil
if c := do(t, h, "PUT", "/"+name, `{"package_type":"rpm","repo_type":"remote","base_url":"https://new.example.com/repo"}`); c != 200 {
t.Fatalf("update (no backend change) = %d, want 200", c)
}
if len(ff.calls) != 0 {
t.Fatalf("flush calls = %v, want no flush when base_url is unchanged", ff.calls)
}
}
// A flush error must be swallowed: the DB update already succeeded, so the
// request still returns 200.
func TestUpdateFlushFailureStillSucceeds(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
const name = "rpm-flush-error"
seedRemote(t, db, name, "https://old.example.com/repo")
ff := &fakeFlusher{err: errors.New("redis down")}
h := NewRemotesHandler(db, ff, nil).Routes()
if c := do(t, h, "PUT", "/"+name, `{"package_type":"rpm","repo_type":"remote","base_url":"https://new.example.com/repo"}`); c != 200 {
t.Fatalf("update with failing flush = %d, want 200", c)
}
if len(ff.calls) != 1 {
t.Fatalf("flush calls = %v, want exactly one attempted flush", ff.calls)
}
}
+71
View File
@@ -0,0 +1,71 @@
package database
import (
"context"
"errors"
"time"
"github.com/jackc/pgx/v5"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// ListGitHubAlpineRemotes returns every github_alpine remote so the syncer can
// sweep them on each poll tick.
func (db *DB) ListGitHubAlpineRemotes(ctx context.Context) ([]models.Remote, error) {
rows, err := db.Pool.Query(ctx, `SELECT `+remoteCols+` FROM remotes WHERE package_type = $1 ORDER BY name`, models.PackageGitHubAlpine)
if err != nil {
return nil, err
}
defer rows.Close()
var remotes []models.Remote
for rows.Next() {
var r models.Remote
if err := scanRemote(rows, &r); err != nil {
return nil, err
}
remotes = append(remotes, r)
}
return remotes, rows.Err()
}
// ClaimGitHubAlpineSyncLease atomically claims the per-remote sync lease. It
// succeeds only when the remote is due (never synced, or synced longer than
// freshness ago) and no live lease is held by another replica. A zero freshness
// (prime scans) ignores the recency gate. The returned etag is the stored
// releases-list ETag, shared across replicas.
func (db *DB) ClaimGitHubAlpineSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
row := db.Pool.QueryRow(ctx, `
INSERT INTO github_alpine_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
VALUES ($1, $2, now() + make_interval(secs => $4))
ON CONFLICT (remote_name) DO UPDATE
SET sync_lease_owner = $2,
sync_lease_expires = now() + make_interval(secs => $4)
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
RETURNING s.etag
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
var etag string
if err := row.Scan(&etag); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return false, "", nil
}
return false, "", err
}
return true, etag, nil
}
// ReleaseGitHubAlpineSyncLease records the completed scan and frees the lease.
// Only the owning replica may release; last_synced_at advances so the next poll
// waits a full freshness window, and etag is persisted for the next conditional
// request.
func (db *DB) ReleaseGitHubAlpineSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
_, err := db.Pool.Exec(ctx, `
UPDATE github_alpine_sync_state
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
WHERE remote_name = $1 AND sync_lease_owner = $2
`, remoteName, owner, syncedAt, etag)
return err
}
+70
View File
@@ -0,0 +1,70 @@
package database
import (
"context"
"strings"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func (db *DB) InsertAlpineMetadata(ctx context.Context, meta *provider.AlpineMetadata) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO alpine_metadata (
repo_name, file_path, content_hash, checksum,
name, version, arch, download_size, installed_size,
description, url, license, origin, maintainer,
build_time, commit_hash, provider_priority,
depends, provides, install_if
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20)
ON CONFLICT (repo_name, file_path) DO NOTHING
`,
meta.RepoName, meta.FilePath, meta.ContentHash, meta.Checksum,
meta.Name, meta.Version, meta.Arch, meta.DownloadSize, meta.InstalledSize,
meta.Description, meta.URL, meta.License, meta.Origin, meta.Maintainer,
meta.BuildTime, meta.Commit, meta.ProviderPriority,
strings.Join(meta.Depends, " "), strings.Join(meta.Provides, " "), strings.Join(meta.InstallIf, " "),
)
return err
}
func (db *DB) DeleteAlpineMetadata(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM alpine_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
func (db *DB) ListAlpineMetadataEntries(ctx context.Context, repoName string) ([]provider.AlpineMetadata, error) {
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash, checksum,
name, version, arch, download_size, installed_size,
description, url, license, origin, maintainer,
build_time, commit_hash, provider_priority,
depends, provides, install_if
FROM alpine_metadata
WHERE repo_name = $1
ORDER BY name, version, arch, file_path
`, repoName)
if err != nil {
return nil, err
}
defer rows.Close()
var result []provider.AlpineMetadata
for rows.Next() {
var m provider.AlpineMetadata
var depends, provides, installIf string
if err := rows.Scan(
&m.RepoName, &m.FilePath, &m.ContentHash, &m.Checksum,
&m.Name, &m.Version, &m.Arch, &m.DownloadSize, &m.InstalledSize,
&m.Description, &m.URL, &m.License, &m.Origin, &m.Maintainer,
&m.BuildTime, &m.Commit, &m.ProviderPriority,
&depends, &provides, &installIf,
); err != nil {
return nil, err
}
m.Depends = strings.Fields(depends)
m.Provides = strings.Fields(provides)
m.InstallIf = strings.Fields(installIf)
result = append(result, m)
}
return result, rows.Err()
}
+47
View File
@@ -99,6 +99,53 @@ func TestRemotesCRUD(t *testing.T) {
} }
} }
func TestRemoteMirrorlistRoundTrip(t *testing.T) {
requireDB(t)
mirrors := []string{"https://b.example", "https://c.example"}
if err := testDB.CreateRemote(ctx(), &models.Remote{
Name: "r-mirror", PackageType: models.PackageRPM, RepoType: models.RepoTypeRemote,
BaseURL: "https://a.example", Mirrorlist: mirrors, MutableTTL: 3600,
}); err != nil {
t.Fatalf("create mirrorlist remote: %v", err)
}
defer testDB.DeleteRemote(ctx(), "r-mirror")
got, err := testDB.GetRemote(ctx(), "r-mirror")
if err != nil {
t.Fatalf("get: %v", err)
}
if got.BaseURL != "https://a.example" {
t.Fatalf("BaseURL = %q, want https://a.example", got.BaseURL)
}
if len(got.Mirrorlist) != 2 || got.Mirrorlist[0] != mirrors[0] || got.Mirrorlist[1] != mirrors[1] {
t.Fatalf("Mirrorlist round-trip = %v, want %v", got.Mirrorlist, mirrors)
}
// An unset strategy is stored as the round_robin default.
if got.MirrorStrategy != models.MirrorStrategyRoundRobin {
t.Fatalf("MirrorStrategy default = %q, want %q", got.MirrorStrategy, models.MirrorStrategyRoundRobin)
}
// Updating to least_conn round-trips.
got.MirrorStrategy = models.MirrorStrategyLeastConn
if err := testDB.UpdateRemote(ctx(), got); err != nil {
t.Fatalf("update to least_conn: %v", err)
}
got, _ = testDB.GetRemote(ctx(), "r-mirror")
if got.MirrorStrategy != models.MirrorStrategyLeastConn {
t.Fatalf("MirrorStrategy after update = %q, want least_conn", got.MirrorStrategy)
}
// Clearing the mirrorlist on update persists an empty list.
got.Mirrorlist = nil
if err := testDB.UpdateRemote(ctx(), got); err != nil {
t.Fatalf("update clearing mirrorlist: %v", err)
}
got, _ = testDB.GetRemote(ctx(), "r-mirror")
if len(got.Mirrorlist) != 0 {
t.Fatalf("mirrorlist after clear = %v, want empty", got.Mirrorlist)
}
}
func TestArtifactsAndBlobs(t *testing.T) { func TestArtifactsAndBlobs(t *testing.T) {
requireDB(t) requireDB(t)
seedRemote(t, "r-art") seedRemote(t, "r-art")
+3 -3
View File
@@ -31,10 +31,10 @@ func (db *DB) ListDebMetadataEntries(ctx context.Context, repoName string) ([]pr
rows, err := db.Pool.Query(ctx, ` rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash, SELECT repo_name, file_path, content_hash,
name, version, architecture, control, name, version, architecture, control,
size, md5, sha256 size, md5, sha256, created_at
FROM deb_metadata FROM deb_metadata
WHERE repo_name = $1 WHERE repo_name = $1
ORDER BY name, version, architecture ORDER BY name, version, architecture, file_path
`, repoName) `, repoName)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -47,7 +47,7 @@ func (db *DB) ListDebMetadataEntries(ctx context.Context, repoName string) ([]pr
if err := rows.Scan( if err := rows.Scan(
&m.RepoName, &m.FilePath, &m.ContentHash, &m.RepoName, &m.FilePath, &m.ContentHash,
&m.Name, &m.Version, &m.Architecture, &m.Control, &m.Name, &m.Version, &m.Architecture, &m.Control,
&m.Size, &m.MD5, &m.SHA256, &m.Size, &m.MD5, &m.SHA256, &m.CreatedAt,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
+41
View File
@@ -44,6 +44,8 @@ func (db *DB) migrate() error {
package_type TEXT NOT NULL, package_type TEXT NOT NULL,
repo_type TEXT DEFAULT 'remote', repo_type TEXT DEFAULT 'remote',
base_url TEXT NOT NULL DEFAULT '', base_url TEXT NOT NULL DEFAULT '',
mirrorlist TEXT[] DEFAULT '{}',
mirror_strategy TEXT NOT NULL DEFAULT 'round_robin',
description TEXT DEFAULT '', description TEXT DEFAULT '',
username TEXT DEFAULT '', username TEXT DEFAULT '',
password TEXT DEFAULT '', password TEXT DEFAULT '',
@@ -124,6 +126,8 @@ func (db *DB) migrate() error {
CREATE INDEX IF NOT EXISTS idx_access_log_remote_time ON access_log(remote_name, created_at); CREATE INDEX IF NOT EXISTS idx_access_log_remote_time ON access_log(remote_name, created_at);
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS repo_type TEXT DEFAULT 'remote'; ALTER TABLE remotes ADD COLUMN IF NOT EXISTS repo_type TEXT DEFAULT 'remote';
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS mirrorlist TEXT[] DEFAULT '{}';
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS mirror_strategy TEXT NOT NULL DEFAULT 'round_robin';
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_dial_timeout INTEGER DEFAULT 0; ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_dial_timeout INTEGER DEFAULT 0;
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_tls_timeout INTEGER DEFAULT 0; ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_tls_timeout INTEGER DEFAULT 0;
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_response_header_timeout INTEGER DEFAULT 0; ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_response_header_timeout INTEGER DEFAULT 0;
@@ -182,6 +186,35 @@ func (db *DB) migrate() error {
CREATE INDEX IF NOT EXISTS idx_deb_metadata_repo ON deb_metadata(repo_name); CREATE INDEX IF NOT EXISTS idx_deb_metadata_repo ON deb_metadata(repo_name);
CREATE TABLE IF NOT EXISTS alpine_metadata (
id BIGSERIAL PRIMARY KEY,
repo_name TEXT NOT NULL,
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
checksum TEXT NOT NULL,
name TEXT NOT NULL,
version TEXT NOT NULL,
arch TEXT NOT NULL,
download_size BIGINT DEFAULT 0,
installed_size BIGINT DEFAULT 0,
description TEXT DEFAULT '',
url TEXT DEFAULT '',
license TEXT DEFAULT '',
origin TEXT DEFAULT '',
maintainer TEXT DEFAULT '',
build_time BIGINT DEFAULT 0,
commit_hash TEXT DEFAULT '',
provider_priority TEXT DEFAULT '',
depends TEXT DEFAULT '',
provides TEXT DEFAULT '',
install_if TEXT DEFAULT '',
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(repo_name, file_path)
);
CREATE INDEX IF NOT EXISTS idx_alpine_metadata_repo ON alpine_metadata(repo_name);
CREATE INDEX IF NOT EXISTS idx_alpine_metadata_repo_arch ON alpine_metadata(repo_name, arch);
CREATE TABLE IF NOT EXISTS github_rpm_sync_state ( CREATE TABLE IF NOT EXISTS github_rpm_sync_state (
remote_name TEXT PRIMARY KEY, remote_name TEXT PRIMARY KEY,
etag TEXT DEFAULT '', etag TEXT DEFAULT '',
@@ -198,6 +231,14 @@ func (db *DB) migrate() error {
sync_lease_expires TIMESTAMPTZ sync_lease_expires TIMESTAMPTZ
); );
CREATE TABLE IF NOT EXISTS github_alpine_sync_state (
remote_name TEXT PRIMARY KEY,
etag TEXT DEFAULT '',
last_synced_at TIMESTAMPTZ,
sync_lease_owner TEXT DEFAULT '',
sync_lease_expires TIMESTAMPTZ
);
CREATE TABLE IF NOT EXISTS signing_keys ( CREATE TABLE IF NOT EXISTS signing_keys (
purpose TEXT PRIMARY KEY, purpose TEXT PRIMARY KEY,
private_key_armor TEXT NOT NULL, private_key_armor TEXT NOT NULL,
+21 -7
View File
@@ -6,7 +6,7 @@ import (
"git.unkin.net/unkin/artifactapi/pkg/models" "git.unkin.net/unkin/artifactapi/pkg/models"
) )
const remoteCols = `name, package_type, repo_type, base_url, description, username, password, const remoteCols = `name, package_type, repo_type, base_url, mirrorlist, mirror_strategy, description, username, password,
immutable_ttl, mutable_ttl, check_mutable, immutable_ttl, mutable_ttl, check_mutable,
patterns, blocklist, mutable_patterns, immutable_patterns, patterns, blocklist, mutable_patterns, immutable_patterns,
ban_tags_enabled, ban_tags, ban_tags_enabled, ban_tags,
@@ -15,9 +15,18 @@ const remoteCols = `name, package_type, repo_type, base_url, description, userna
upstream_dial_timeout, upstream_tls_timeout, upstream_response_header_timeout, upstream_dial_timeout, upstream_tls_timeout, upstream_response_header_timeout,
created_at, updated_at` created_at, updated_at`
// normalizeMirrorStrategy maps an empty strategy to the round_robin default so
// the NOT NULL mirror_strategy column always stores a canonical value.
func normalizeMirrorStrategy(s string) string {
if s == "" {
return models.MirrorStrategyRoundRobin
}
return s
}
func scanRemote(scanner interface{ Scan(...any) error }, r *models.Remote) error { func scanRemote(scanner interface{ Scan(...any) error }, r *models.Remote) error {
return scanner.Scan( return scanner.Scan(
&r.Name, &r.PackageType, &r.RepoType, &r.BaseURL, &r.Description, &r.Username, &r.Password, &r.Name, &r.PackageType, &r.RepoType, &r.BaseURL, &r.Mirrorlist, &r.MirrorStrategy, &r.Description, &r.Username, &r.Password,
&r.ImmutableTTL, &r.MutableTTL, &r.CheckMutable, &r.ImmutableTTL, &r.MutableTTL, &r.CheckMutable,
&r.Patterns, &r.Blocklist, &r.MutablePatterns, &r.ImmutablePatterns, &r.Patterns, &r.Blocklist, &r.MutablePatterns, &r.ImmutablePatterns,
&r.BanTagsEnabled, &r.BanTags, &r.BanTagsEnabled, &r.BanTags,
@@ -58,22 +67,24 @@ func (db *DB) ListRemotes(ctx context.Context) ([]models.Remote, error) {
func (db *DB) CreateRemote(ctx context.Context, r *models.Remote) error { func (db *DB) CreateRemote(ctx context.Context, r *models.Remote) error {
_, err := db.Pool.Exec(ctx, ` _, err := db.Pool.Exec(ctx, `
INSERT INTO remotes ( INSERT INTO remotes (
name, package_type, repo_type, base_url, description, username, password, name, package_type, repo_type, base_url, mirrorlist, description, username, password,
immutable_ttl, mutable_ttl, check_mutable, immutable_ttl, mutable_ttl, check_mutable,
patterns, blocklist, mutable_patterns, immutable_patterns, patterns, blocklist, mutable_patterns, immutable_patterns,
ban_tags_enabled, ban_tags, ban_tags_enabled, ban_tags,
quarantine_enabled, quarantine_days, stale_on_error, quarantine_enabled, quarantine_days, stale_on_error,
releases_remote, managed_by, releases_remote, managed_by,
upstream_dial_timeout, upstream_tls_timeout, upstream_response_header_timeout upstream_dial_timeout, upstream_tls_timeout, upstream_response_header_timeout,
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24) mirror_strategy
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26)
`, `,
r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Description, r.Username, r.Password, r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Mirrorlist, r.Description, r.Username, r.Password,
r.ImmutableTTL, r.MutableTTL, r.CheckMutable, r.ImmutableTTL, r.MutableTTL, r.CheckMutable,
r.Patterns, r.Blocklist, r.MutablePatterns, r.ImmutablePatterns, r.Patterns, r.Blocklist, r.MutablePatterns, r.ImmutablePatterns,
r.BanTagsEnabled, r.BanTags, r.BanTagsEnabled, r.BanTags,
r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError, r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError,
r.ReleasesRemote, r.ManagedBy, r.ReleasesRemote, r.ManagedBy,
r.UpstreamDialTimeout, r.UpstreamTLSTimeout, r.UpstreamResponseHeaderTimeout, r.UpstreamDialTimeout, r.UpstreamTLSTimeout, r.UpstreamResponseHeaderTimeout,
normalizeMirrorStrategy(r.MirrorStrategy),
) )
return err return err
} }
@@ -81,13 +92,14 @@ func (db *DB) CreateRemote(ctx context.Context, r *models.Remote) error {
func (db *DB) UpdateRemote(ctx context.Context, r *models.Remote) error { func (db *DB) UpdateRemote(ctx context.Context, r *models.Remote) error {
_, err := db.Pool.Exec(ctx, ` _, err := db.Pool.Exec(ctx, `
UPDATE remotes SET UPDATE remotes SET
package_type=$2, repo_type=$3, base_url=$4, description=$5, username=$6, password=$7, package_type=$2, repo_type=$3, base_url=$4, mirrorlist=$25, description=$5, username=$6, password=$7,
immutable_ttl=$8, mutable_ttl=$9, check_mutable=$10, immutable_ttl=$8, mutable_ttl=$9, check_mutable=$10,
patterns=$11, blocklist=$12, mutable_patterns=$13, immutable_patterns=$14, patterns=$11, blocklist=$12, mutable_patterns=$13, immutable_patterns=$14,
ban_tags_enabled=$15, ban_tags=$16, ban_tags_enabled=$15, ban_tags=$16,
quarantine_enabled=$17, quarantine_days=$18, stale_on_error=$19, quarantine_enabled=$17, quarantine_days=$18, stale_on_error=$19,
releases_remote=$20, managed_by=$21, releases_remote=$20, managed_by=$21,
upstream_dial_timeout=$22, upstream_tls_timeout=$23, upstream_response_header_timeout=$24, upstream_dial_timeout=$22, upstream_tls_timeout=$23, upstream_response_header_timeout=$24,
mirror_strategy=$26,
updated_at=NOW() updated_at=NOW()
WHERE name=$1 WHERE name=$1
`, `,
@@ -98,6 +110,8 @@ func (db *DB) UpdateRemote(ctx context.Context, r *models.Remote) error {
r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError, r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError,
r.ReleasesRemote, r.ManagedBy, r.ReleasesRemote, r.ManagedBy,
r.UpstreamDialTimeout, r.UpstreamTLSTimeout, r.UpstreamResponseHeaderTimeout, r.UpstreamDialTimeout, r.UpstreamTLSTimeout, r.UpstreamResponseHeaderTimeout,
r.Mirrorlist,
normalizeMirrorStrategy(r.MirrorStrategy),
) )
return err return err
} }
+7 -2
View File
@@ -3,6 +3,7 @@ package database
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider" "git.unkin.net/unkin/artifactapi/internal/provider"
) )
@@ -65,6 +66,7 @@ type RPMMetadataRow struct {
Obsoletes json.RawMessage Obsoletes json.RawMessage
Files json.RawMessage Files json.RawMessage
Changelogs json.RawMessage Changelogs json.RawMessage
CreatedAt time.Time
} }
func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]provider.RPMMetadata, error) { func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]provider.RPMMetadata, error) {
@@ -94,6 +96,7 @@ func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]pr
SourceRPM: r.SourceRPM, SourceRPM: r.SourceRPM,
URL: r.URL, URL: r.URL,
Packager: r.Packager, Packager: r.Packager,
CreatedAt: r.CreatedAt,
} }
json.Unmarshal(r.Requires, &meta.Requires) json.Unmarshal(r.Requires, &meta.Requires)
json.Unmarshal(r.Provides, &meta.Provides) json.Unmarshal(r.Provides, &meta.Provides)
@@ -112,10 +115,11 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
name, epoch, version, release, arch, name, epoch, version, release, arch,
summary, description, rpm_size, installed_size, summary, description, rpm_size, installed_size,
license, vendor, build_group, build_host, source_rpm, url, packager, license, vendor, build_group, build_host, source_rpm, url, packager,
requires, provides, conflicts, obsoletes, files, changelogs requires, provides, conflicts, obsoletes, files, changelogs,
created_at
FROM rpm_metadata FROM rpm_metadata
WHERE repo_name = $1 WHERE repo_name = $1
ORDER BY name, epoch, version, release, arch ORDER BY name, epoch, version, release, arch, file_path
`, repoName) `, repoName)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -131,6 +135,7 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
&r.Summary, &r.Description, &r.RPMSize, &r.InstalledSize, &r.Summary, &r.Description, &r.RPMSize, &r.InstalledSize,
&r.License, &r.Vendor, &r.Group, &r.BuildHost, &r.SourceRPM, &r.URL, &r.Packager, &r.License, &r.Vendor, &r.Group, &r.BuildHost, &r.SourceRPM, &r.URL, &r.Packager,
&r.Requires, &r.Provides, &r.Conflicts, &r.Obsoletes, &r.Files, &r.Changelogs, &r.Requires, &r.Provides, &r.Conflicts, &r.Obsoletes, &r.Files, &r.Changelogs,
&r.CreatedAt,
); err != nil { ); err != nil {
return nil, err return nil, err
} }
+361
View File
@@ -1,12 +1,27 @@
package alpine package alpine
import ( import (
"bufio"
"bytes"
"compress/gzip"
"context" "context"
"crypto/sha1"
"encoding/base64"
"errors"
"fmt"
"io"
"log/slog"
"net/http" "net/http"
"path"
"strconv"
"strings" "strings"
"time"
"archive/tar"
"git.unkin.net/unkin/artifactapi/internal/auth" "git.unkin.net/unkin/artifactapi/internal/auth"
"git.unkin.net/unkin/artifactapi/internal/provider" "git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/pkg/models" "git.unkin.net/unkin/artifactapi/pkg/models"
) )
@@ -46,3 +61,349 @@ func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte,
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) { func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
return auth.BasicHeaders(remote), nil return auth.BasicHeaders(remote), nil
} }
// --- LocalUploader: hosting real .apk packages -----------------------------
// ValidateUpload accepts any *.apk and preserves the client-supplied directory
// (the arch prefix) as the storage path, since arch cannot be parsed from the
// filename alone and the generic uploader hands us only the path. apk clients
// fetch packages at <arch>/<file>.apk, so publishers upload to that same path;
// AfterUpload records the true arch (from .PKGINFO) for index filtering.
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
clean := strings.TrimPrefix(path.Clean("/"+filePath), "/")
filename := clean
if i := strings.LastIndex(clean, "/"); i >= 0 {
filename = clean[i+1:]
}
if !strings.HasSuffix(strings.ToLower(filename), ".apk") {
return "", "", fmt.Errorf("file must be a .apk package")
}
return clean, "application/vnd.android.package-archive", nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
filename := storagePath
if i := strings.LastIndex(storagePath, "/"); i >= 0 {
filename = storagePath[i+1:]
}
return map[string]any{
"filename": filename,
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs provider.BlobReader, db provider.MetadataStore) {
s3Key := storage.BlobKey(strings.TrimPrefix(contentHash, "sha256:"))
reader, blobSize, err := blobs.Download(ctx, s3Key)
if err != nil {
slog.Error("alpine metadata: download failed", "repo", repoName, "path", storagePath, "error", err)
return
}
defer reader.Close()
raw, err := io.ReadAll(reader)
if err != nil {
slog.Error("alpine metadata: read failed", "repo", repoName, "path", storagePath, "error", err)
return
}
meta, err := parseApk(raw)
if err != nil {
slog.Error("alpine metadata: parse failed", "repo", repoName, "path", storagePath, "error", err)
return
}
meta.RepoName = repoName
meta.FilePath = storagePath
meta.ContentHash = contentHash
meta.DownloadSize = blobSize
if meta.Name == "" || meta.Arch == "" {
slog.Error("alpine metadata: .PKGINFO missing pkgname/arch", "repo", repoName, "path", storagePath)
return
}
store, ok := db.(provider.AlpineMetadataStore)
if !ok {
slog.Error("alpine metadata: store does not support alpine metadata", "repo", repoName)
return
}
if err := store.InsertAlpineMetadata(ctx, meta); err != nil {
slog.Error("alpine metadata: insert failed", "repo", repoName, "path", storagePath, "error", err)
return
}
slog.Info("alpine metadata: parsed", "repo", repoName, "name", meta.Name, "version", meta.Version, "arch", meta.Arch)
}
func (p *Provider) AfterDelete(ctx context.Context, repoName, storagePath string, db provider.MetadataDeleter) error {
deleter, ok := db.(provider.AlpineMetadataDeleter)
if !ok {
return nil
}
if err := deleter.DeleteAlpineMetadata(ctx, repoName, storagePath); err != nil {
slog.Error("alpine metadata: delete failed", "repo", repoName, "path", storagePath, "error", err)
return err
}
slog.Info("alpine metadata: deleted", "repo", repoName, "path", storagePath)
return nil
}
// --- LocalIndexer: generating a per-arch APKINDEX.tar.gz -------------------
// normalizeIndexPath collapses apk's dot-segment prefix: an /etc/apk/repositories
// line of "<url>/api/v1/local/<name>" makes apk request "./<arch>/APKINDEX.tar.gz".
// Mirrors deb's flat-repo normalization.
func normalizeIndexPath(p string) string {
return strings.TrimPrefix(path.Clean("/"+p), "/")
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, reqPath string) bool {
clean := normalizeIndexPath(reqPath)
if !strings.HasSuffix(clean, "APKINDEX.tar.gz") {
return false
}
arch := strings.TrimSuffix(clean, "APKINDEX.tar.gz")
arch = strings.Trim(arch, "/")
if arch == "" || strings.Contains(arch, "/") {
http.Error(w, "APKINDEX must be requested per-arch: <arch>/APKINDEX.tar.gz", http.StatusNotFound)
return true
}
reader, ok := files.(provider.AlpineMetadataReader)
if !ok {
http.Error(w, "alpine metadata not available", http.StatusInternalServerError)
return true
}
metas, err := reader.ListAlpineMetadataEntries(r.Context(), repoName)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
slog.Warn("alpine: metadata read canceled", "repo", repoName, "error", err)
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
return true
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return true
}
var filtered []provider.AlpineMetadata
for _, m := range metas {
if m.Arch == arch {
filtered = append(filtered, m)
}
}
w.Header().Set("Content-Type", "application/gzip")
w.WriteHeader(http.StatusOK)
w.Write(generateAPKIndex(filtered))
return true
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
return nil, fmt.Errorf("alpine local index generation for virtual repos not supported")
}
// --- pure-Go .apk parsing --------------------------------------------------
// parseApk reads an .apk (up to three concatenated, independently gzipped tar
// streams: optional signature, control, data). It locates the control stream by
// its .PKGINFO member, computes the apk pull checksum C: = "Q1" +
// base64(sha1(<control gzip stream bytes>)), and reads the .PKGINFO fields.
func parseApk(raw []byte) (*provider.AlpineMetadata, error) {
members, err := gzipMembers(raw)
if err != nil {
return nil, err
}
for _, m := range members {
pkginfo, ok := pkginfoFromTar(m.tar)
if !ok {
continue
}
meta := parsePkginfo(pkginfo)
sum := sha1.Sum(m.raw)
meta.Checksum = "Q1" + base64.StdEncoding.EncodeToString(sum[:])
return meta, nil
}
return nil, errors.New("no .PKGINFO found in any .apk gzip stream")
}
type gzMember struct {
raw []byte // the raw bytes of this gzip stream (for the Q1 checksum)
tar []byte // the decompressed tar payload
}
// gzipMembers splits the concatenated gzip streams, returning each stream's raw
// bytes alongside its decompressed tar. It relies on bytes.Reader being an
// io.ByteReader (so compress/gzip does not over-read past a member's trailer)
// to recover exact stream boundaries via Multistream(false)+Reset.
func gzipMembers(data []byte) ([]gzMember, error) {
br := bytes.NewReader(data)
zr, err := gzip.NewReader(br)
if err != nil {
return nil, err
}
var members []gzMember
prev := 0
for {
zr.Multistream(false)
out, err := io.ReadAll(zr)
if err != nil {
return nil, err
}
end := len(data) - br.Len()
members = append(members, gzMember{raw: data[prev:end], tar: out})
prev = end
if err := zr.Reset(br); err != nil {
if err == io.EOF {
break
}
return nil, err
}
}
return members, nil
}
func pkginfoFromTar(tarBytes []byte) (string, bool) {
tr := tar.NewReader(bytes.NewReader(tarBytes))
for {
hdr, err := tr.Next()
if err != nil {
return "", false
}
if strings.TrimPrefix(hdr.Name, "./") == ".PKGINFO" {
b, err := io.ReadAll(tr)
if err != nil {
return "", false
}
return string(b), true
}
}
}
// parsePkginfo reads the "key = value" .PKGINFO text, collecting the repeated
// depend/provides/install_if keys into slices.
func parsePkginfo(text string) *provider.AlpineMetadata {
m := &provider.AlpineMetadata{}
sc := bufio.NewScanner(strings.NewReader(text))
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
idx := strings.Index(line, "=")
if idx < 0 {
continue
}
key := strings.TrimSpace(line[:idx])
val := strings.TrimSpace(line[idx+1:])
switch key {
case "pkgname":
m.Name = val
case "pkgver":
m.Version = val
case "arch":
m.Arch = val
case "pkgdesc":
m.Description = val
case "url":
m.URL = val
case "license":
m.License = val
case "origin":
m.Origin = val
case "maintainer":
m.Maintainer = val
case "builddate":
if n, err := strconv.ParseInt(val, 10, 64); err == nil {
m.BuildTime = n
}
case "commit":
m.Commit = val
case "size":
if n, err := strconv.ParseInt(val, 10, 64); err == nil {
m.InstalledSize = n
}
case "provider_priority":
m.ProviderPriority = val
case "depend":
if val != "" {
m.Depends = append(m.Depends, val)
}
case "provides":
if val != "" {
m.Provides = append(m.Provides, val)
}
case "install_if":
if val != "" {
m.InstallIf = append(m.InstallIf, val)
}
}
}
return m
}
// generateAPKIndex builds the APKINDEX.tar.gz = gzip(tar(APKINDEX)) for the
// given (already arch-filtered) rows. Records are blank-line separated; fields
// follow the canonical C/P/V/A/S/I/T/U/L/o/m/t/c/k/D/p/i order and empties are
// omitted. Unsigned (clients use --allow-untrusted), matching rpm gpgcheck=0.
func generateAPKIndex(metas []provider.AlpineMetadata) []byte {
var idx bytes.Buffer
for i, m := range metas {
if i > 0 {
idx.WriteString("\n")
}
writeField(&idx, "C", m.Checksum)
writeField(&idx, "P", m.Name)
writeField(&idx, "V", m.Version)
writeField(&idx, "A", m.Arch)
writeField(&idx, "S", intField(m.DownloadSize))
writeField(&idx, "I", intField(m.InstalledSize))
writeField(&idx, "T", m.Description)
writeField(&idx, "U", m.URL)
writeField(&idx, "L", m.License)
writeField(&idx, "o", m.Origin)
writeField(&idx, "m", m.Maintainer)
writeField(&idx, "t", intField(m.BuildTime))
writeField(&idx, "c", m.Commit)
writeField(&idx, "k", m.ProviderPriority)
writeField(&idx, "D", strings.Join(m.Depends, " "))
writeField(&idx, "p", strings.Join(m.Provides, " "))
writeField(&idx, "i", strings.Join(m.InstallIf, " "))
}
var tarBuf bytes.Buffer
tw := tar.NewWriter(&tarBuf)
body := idx.Bytes()
// ModTime is pinned to the Unix epoch (never wall clock) so APKINDEX.tar.gz
// is byte-identical across replicas and regenerations (issue #117); apk
// clients ignore the tar mtime.
tw.WriteHeader(&tar.Header{Name: "APKINDEX", Mode: 0o644, Size: int64(len(body)), Typeflag: tar.TypeReg, ModTime: time.Unix(0, 0)})
tw.Write(body)
tw.Close()
var gzBuf bytes.Buffer
gz := gzip.NewWriter(&gzBuf)
gz.Write(tarBuf.Bytes())
gz.Close()
return gzBuf.Bytes()
}
func writeField(b *bytes.Buffer, key, val string) {
if val == "" {
return
}
b.WriteString(key)
b.WriteString(":")
b.WriteString(val)
b.WriteString("\n")
}
func intField(n int64) string {
if n == 0 {
return ""
}
return strconv.FormatInt(n, 10)
}
@@ -0,0 +1,72 @@
package alpine
import (
"archive/tar"
"bytes"
"compress/gzip"
"io"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func apkFixture() []provider.AlpineMetadata {
return []provider.AlpineMetadata{
{
RepoName: "r", FilePath: "x86_64/aaa-1.0-r0.apk", Checksum: "Q1aaa",
Name: "aaa", Version: "1.0-r0", Arch: "x86_64", DownloadSize: 100, InstalledSize: 10,
Description: "pkg aaa", URL: "https://a", License: "MIT",
Depends: []string{"so:libc"}, Provides: []string{"cmd:aaa"}, BuildTime: 1710000000,
},
{
RepoName: "r", FilePath: "x86_64/bbb-2.0-r0.apk", Checksum: "Q1bbb",
Name: "bbb", Version: "2.0-r0", Arch: "x86_64", DownloadSize: 200, InstalledSize: 20,
},
}
}
// TestAPKIndexDeterministic asserts APKINDEX.tar.gz is byte-identical across two
// generations separated by wall-clock time, so the two no-affinity replicas and
// every regeneration serve the same bytes (issue #117).
func TestAPKIndexDeterministic(t *testing.T) {
metas := apkFixture()
first := generateAPKIndex(metas)
time.Sleep(10 * time.Millisecond)
second := generateAPKIndex(metas)
if !bytes.Equal(first, second) {
t.Error("APKINDEX.tar.gz differs across generations")
}
}
// TestAPKIndexTarModTimePinned guards the tar header: its ModTime must be the
// pinned Unix epoch, never wall clock. Fails if a future edit stamps time.Now().
func TestAPKIndexTarModTimePinned(t *testing.T) {
metas := apkFixture()
zr, err := gzip.NewReader(bytes.NewReader(generateAPKIndex(metas)))
if err != nil {
t.Fatalf("gzip: %v", err)
}
if !zr.ModTime.IsZero() && zr.ModTime.Unix() != 0 {
t.Errorf("gzip header ModTime = %v, want zero/epoch", zr.ModTime)
}
tarBytes, err := io.ReadAll(zr)
if err != nil {
t.Fatalf("gunzip: %v", err)
}
tr := tar.NewReader(bytes.NewReader(tarBytes))
hdr, err := tr.Next()
if err != nil {
t.Fatalf("tar: %v", err)
}
if hdr.Name != "APKINDEX" {
t.Fatalf("tar entry = %q, want APKINDEX", hdr.Name)
}
if hdr.ModTime.Unix() != 0 {
t.Errorf("APKINDEX tar ModTime = %v (unix %d), want epoch (0)", hdr.ModTime, hdr.ModTime.Unix())
}
}
@@ -0,0 +1,324 @@
package alpine
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/sha1"
"encoding/base64"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
type fakeBlobReader struct{ data []byte }
func (f fakeBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
return io.NopCloser(bytes.NewReader(f.data)), int64(len(f.data)), nil
}
type errBlobReader struct{}
func (errBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
return nil, 0, io.ErrUnexpectedEOF
}
// fakeAlpineStore satisfies provider.MetadataStore (shared) and
// provider.AlpineMetadataStore, recording the row AfterUpload writes.
type fakeAlpineStore struct{ inserted *provider.AlpineMetadata }
func (f *fakeAlpineStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
func (f *fakeAlpineStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
func (f *fakeAlpineStore) InsertAlpineMetadata(_ context.Context, m *provider.AlpineMetadata) error {
f.inserted = m
return nil
}
// fakeAlpineDeleter satisfies provider.MetadataDeleter and AlpineMetadataDeleter.
type fakeAlpineDeleter struct{ deleted bool }
func (f *fakeAlpineDeleter) DeleteRPMMetadata(context.Context, string, string) error { return nil }
func (f *fakeAlpineDeleter) DeleteDebMetadata(context.Context, string, string) error { return nil }
func (f *fakeAlpineDeleter) DeleteAlpineMetadata(context.Context, string, string) error {
f.deleted = true
return nil
}
// fakeAlpineReader is a FileStore that also serves alpine metadata rows.
type fakeAlpineReader struct{ metas []provider.AlpineMetadata }
func (f fakeAlpineReader) ListAlpineMetadataEntries(context.Context, string) ([]provider.AlpineMetadata, error) {
return f.metas, nil
}
func (f fakeAlpineReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
return nil, nil
}
func (f fakeAlpineReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
type errAlpineReader struct{}
func (errAlpineReader) ListAlpineMetadataEntries(context.Context, string) ([]provider.AlpineMetadata, error) {
return nil, io.ErrUnexpectedEOF
}
func (errAlpineReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
return nil, nil
}
func (errAlpineReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
func TestAlpineValidateUpload(t *testing.T) {
p := &Provider{}
sp, ct, err := p.ValidateUpload("x86_64/foo-1.0-r0.apk")
if err != nil || sp != "x86_64/foo-1.0-r0.apk" || ct != "application/vnd.android.package-archive" {
t.Errorf("sp=%q ct=%q err=%v", sp, ct, err)
}
// Dot-segment prefix is normalized away.
if sp, _, err := p.ValidateUpload("./aarch64/bar-2.0-r1.apk"); err != nil || sp != "aarch64/bar-2.0-r1.apk" {
t.Errorf("dot-seg: sp=%q err=%v", sp, err)
}
if _, _, err := p.ValidateUpload("foo.rpm"); err == nil {
t.Error("expected error for non-apk")
}
resp := p.UploadResponse("x86_64/foo-1.0-r0.apk", "sha256:abc", 42)
if resp["filename"] != "foo-1.0-r0.apk" || resp["content_hash"] != "sha256:abc" || resp["size_bytes"] != int64(42) {
t.Errorf("upload response %v", resp)
}
}
func TestAlpineAfterUpload(t *testing.T) {
data := testsupport.MinimalApk("hello", "1.0-r0", "x86_64")
store := &fakeAlpineStore{}
(&Provider{}).AfterUpload(context.Background(), "myrepo", "x86_64/hello-1.0-r0.apk",
"sha256:deadbeef", fakeBlobReader{data: data}, store)
m := store.inserted
if m == nil {
t.Fatal("no metadata inserted")
}
if m.Name != "hello" || m.Version != "1.0-r0" || m.Arch != "x86_64" {
t.Errorf("unexpected metadata: %+v", m)
}
if m.DownloadSize != int64(len(data)) {
t.Errorf("DownloadSize = %d, want %d", m.DownloadSize, len(data))
}
if m.InstalledSize != 4 {
t.Errorf("InstalledSize = %d, want 4", m.InstalledSize)
}
if m.License != "MIT" || m.Origin != "hello" || !strings.HasPrefix(m.Maintainer, "e2e") {
t.Errorf("scalar fields not parsed: %+v", m)
}
if len(m.Depends) != 1 || m.Depends[0] != "so:libc.musl-x86_64.so.1" {
t.Errorf("Depends = %v", m.Depends)
}
if len(m.Provides) != 1 || m.Provides[0] != "cmd:hello=1.0-r0" {
t.Errorf("Provides = %v", m.Provides)
}
// The Q1 checksum is the sha1 of the CONTROL gzip stream (the member whose
// tar carries .PKGINFO), not of the whole file.
controlRaw := controlStreamBytes(t, data)
sum := sha1.Sum(controlRaw)
want := "Q1" + base64.StdEncoding.EncodeToString(sum[:])
if m.Checksum != want {
t.Errorf("Checksum = %q, want %q (sha1 of control stream)", m.Checksum, want)
}
// And explicitly NOT the sha1 of the whole apk.
whole := sha1.Sum(data)
if m.Checksum == "Q1"+base64.StdEncoding.EncodeToString(whole[:]) {
t.Error("Checksum was computed over the whole file, not the control stream")
}
}
func TestAlpineAfterUploadErrors(t *testing.T) {
store := &fakeAlpineStore{}
(&Provider{}).AfterUpload(context.Background(), "r", "x86_64/p.apk", "sha256:x", errBlobReader{}, store)
if store.inserted != nil {
t.Error("no metadata should be inserted on download error")
}
store2 := &fakeAlpineStore{}
(&Provider{}).AfterUpload(context.Background(), "r", "x86_64/p.apk", "sha256:x", fakeBlobReader{data: []byte("not an apk")}, store2)
if store2.inserted != nil {
t.Error("no metadata should be inserted on parse error")
}
}
func TestAlpineAfterDelete(t *testing.T) {
d := &fakeAlpineDeleter{}
if err := (&Provider{}).AfterDelete(context.Background(), "r", "x86_64/p.apk", d); err != nil {
t.Fatalf("AfterDelete: %v", err)
}
if !d.deleted {
t.Error("DeleteAlpineMetadata not called")
}
}
func TestAlpineServeLocalIndex(t *testing.T) {
p := &Provider{}
reader := fakeAlpineReader{metas: []provider.AlpineMetadata{
{Name: "aaa", Version: "1.0-r0", Arch: "x86_64", Checksum: "Q1aaa", DownloadSize: 100, InstalledSize: 10,
Description: "pkg aaa", URL: "https://a", License: "MIT", Depends: []string{"so:libc"}, Provides: []string{"cmd:aaa"}},
{Name: "bbb", Version: "2.0-r0", Arch: "aarch64", Checksum: "Q1bbb", DownloadSize: 200, InstalledSize: 20},
}}
// x86_64 index contains only aaa, with its fields, and not bbb.
w := serveIndex(t, p, reader, "x86_64/APKINDEX.tar.gz")
if w.Code != 200 {
t.Fatalf("code %d", w.Code)
}
idx := untarIndex(t, w.Body.Bytes())
for _, want := range []string{"C:Q1aaa", "P:aaa", "V:1.0-r0", "A:x86_64", "S:100", "I:10", "T:pkg aaa", "U:https://a", "L:MIT", "D:so:libc", "p:cmd:aaa"} {
if !strings.Contains(idx, want) {
t.Errorf("x86_64 APKINDEX missing %q:\n%s", want, idx)
}
}
if strings.Contains(idx, "P:bbb") {
t.Errorf("x86_64 APKINDEX leaked aarch64 package:\n%s", idx)
}
// aarch64 index contains only bbb.
w = serveIndex(t, p, reader, "aarch64/APKINDEX.tar.gz")
idx = untarIndex(t, w.Body.Bytes())
if !strings.Contains(idx, "P:bbb") || strings.Contains(idx, "P:aaa") {
t.Errorf("aarch64 filtering wrong:\n%s", idx)
}
// Non-index and .apk paths are not owned by the indexer.
for _, path := range []string{"x86_64/foo-1.0-r0.apk", "x86_64/", "README"} {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if p.ServeLocalIndex(w, r, reader, "repo", path) {
t.Errorf("ServeLocalIndex should return false for %q", path)
}
}
}
// Empty fields are omitted from the record (bbb has no description/url).
func TestAlpineIndexOmitsEmptyFields(t *testing.T) {
p := &Provider{}
reader := fakeAlpineReader{metas: []provider.AlpineMetadata{
{Name: "bbb", Version: "2.0-r0", Arch: "x86_64", Checksum: "Q1bbb", DownloadSize: 200, InstalledSize: 20},
}}
idx := untarIndex(t, serveIndex(t, p, reader, "x86_64/APKINDEX.tar.gz").Body.Bytes())
for _, absent := range []string{"T:", "U:", "L:", "D:", "p:", "i:", "o:", "m:", "c:", "k:"} {
if strings.Contains(idx, absent) {
t.Errorf("empty field %q should be omitted:\n%s", absent, idx)
}
}
}
// apk requests "./<arch>/APKINDEX.tar.gz" for a bare repo base URL; the
// dot-segment must be collapsed and yield the same bytes as the plain path.
func TestAlpineServeLocalIndexDotSegment(t *testing.T) {
p := &Provider{}
reader := fakeAlpineReader{metas: []provider.AlpineMetadata{
{Name: "aaa", Version: "1.0-r0", Arch: "x86_64", Checksum: "Q1aaa", DownloadSize: 100, InstalledSize: 10},
}}
plain := untarIndex(t, serveIndex(t, p, reader, "x86_64/APKINDEX.tar.gz").Body.Bytes())
dotted := untarIndex(t, serveIndex(t, p, reader, "./x86_64/APKINDEX.tar.gz").Body.Bytes())
if plain != dotted {
t.Errorf("dot-segment path differs:\nplain=%q\ndotted=%q", plain, dotted)
}
}
func TestAlpineServeLocalIndexArchRequired(t *testing.T) {
p := &Provider{}
reader := fakeAlpineReader{}
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/APKINDEX.tar.gz", nil)
if !p.ServeLocalIndex(w, r, reader, "repo", "APKINDEX.tar.gz") {
t.Fatal("bare APKINDEX should be owned (and rejected) by the indexer")
}
if w.Code != http.StatusNotFound {
t.Errorf("bare APKINDEX code = %d, want 404", w.Code)
}
}
func TestAlpineServeMetadataError(t *testing.T) {
p := &Provider{}
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/x86_64/APKINDEX.tar.gz", nil)
p.ServeLocalIndex(w, r, errAlpineReader{}, "repo", "x86_64/APKINDEX.tar.gz")
if w.Code != 500 {
t.Errorf("failing reader code = %d, want 500", w.Code)
}
}
func TestAlpineGenerateLocalIndexUnsupported(t *testing.T) {
if _, err := (&Provider{}).GenerateLocalIndex(context.Background(), fakeAlpineReader{}, "r", "x86_64/APKINDEX.tar.gz"); err == nil {
t.Error("expected unsupported error")
}
}
func serveIndex(t *testing.T, p *Provider, files provider.FileStore, path string) *httptest.ResponseRecorder {
t.Helper()
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if !p.ServeLocalIndex(w, r, files, "repo", path) {
t.Fatalf("ServeLocalIndex returned false for %q", path)
}
return w
}
// untarIndex un-gzips and un-tars an APKINDEX.tar.gz and returns the APKINDEX text.
func untarIndex(t *testing.T, gzTar []byte) string {
t.Helper()
zr, err := gzip.NewReader(bytes.NewReader(gzTar))
if err != nil {
t.Fatalf("APKINDEX not gzip: %v", err)
}
tarBytes, _ := io.ReadAll(zr)
tr := tar.NewReader(bytes.NewReader(tarBytes))
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("APKINDEX not tar: %v", err)
}
if hdr.Name == "APKINDEX" {
b, _ := io.ReadAll(tr)
return string(b)
}
}
t.Fatal("no APKINDEX member in tarball")
return ""
}
// controlStreamBytes returns the raw bytes of the gzip stream whose tar carries
// .PKGINFO, so the test can independently compute the expected Q1 checksum.
func controlStreamBytes(t *testing.T, apk []byte) []byte {
t.Helper()
br := bytes.NewReader(apk)
zr, err := gzip.NewReader(br)
if err != nil {
t.Fatalf("gzip: %v", err)
}
prev := 0
for {
zr.Multistream(false)
out, _ := io.ReadAll(zr)
end := len(apk) - br.Len()
tr := tar.NewReader(bytes.NewReader(out))
for {
h, err := tr.Next()
if err != nil {
break
}
if strings.TrimPrefix(h.Name, "./") == ".PKGINFO" {
return apk[prev:end]
}
}
prev = end
if err := zr.Reset(br); err != nil {
break
}
}
t.Fatal("no control stream found")
return nil
}
+714
View File
@@ -0,0 +1,714 @@
package alpine
import (
"bytes"
"compress/gzip"
"context"
"crypto/sha1"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/url"
"regexp"
"strings"
"sync"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/githubauth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// gitHubProvider is the process-wide singleton for github_alpine. The background
// Syncer binds its shared rate limiter and work queue onto this instance so the
// request path and the syncer drive the same derive machinery.
var gitHubProvider = newGitHubProvider()
func init() {
provider.Register(gitHubProvider)
}
// Tuning knobs for the no-precache control fetch. An .apk is up to three
// concatenated gzip streams (optional signature, control, data); the control
// stream carrying .PKGINFO sits near the front, so a small prefix reliably
// covers it.
const (
defaultHeaderRangeInitial = 32 << 10 // 32 KiB — covers the control stream of almost every .apk
defaultHeaderRangeMax = 16 << 20 // 16 MiB — give up past this and skip the asset
defaultReleasePageCap = 10 // 100 releases/page * 10 pages
defaultScanTimeout = 10 * time.Minute
defaultServeTimeout = 30 * time.Second
defaultColdWait = 8 * time.Second
)
// GitHubProvider is a metadata-only remote: it scans a GitHub repo's releases
// for .apk assets, derives per-asset .PKGINFO metadata via a ranged prefix fetch
// (never downloading whole packages), synthesizes a per-arch APKINDEX from that
// cached metadata, and redirects package downloads to a backend "releases_remote"
// (the generic github.com remote) that serves the actual bytes.
type GitHubProvider struct {
client *http.Client
headerInitial int64
headerMax int64
pageCap int
scanTimeout time.Duration
serveTimeout time.Duration
coldWait time.Duration
limiter *rate.Limiter
syncer *Syncer
serverCred githubauth.Credential
mu sync.Mutex
scanning map[string]bool
lastScan map[string]time.Time
}
func newGitHubProvider() *GitHubProvider {
return &GitHubProvider{
client: &http.Client{},
headerInitial: defaultHeaderRangeInitial,
headerMax: defaultHeaderRangeMax,
pageCap: defaultReleasePageCap,
scanTimeout: defaultScanTimeout,
serveTimeout: defaultServeTimeout,
coldWait: defaultColdWait,
scanning: map[string]bool{},
lastScan: map[string]time.Time{},
}
}
func (p *GitHubProvider) limiterWait(ctx context.Context) error {
if p.limiter == nil {
return nil
}
return p.limiter.Wait(ctx)
}
func (p *GitHubProvider) Type() models.PackageType { return models.PackageGitHubAlpine }
func (p *GitHubProvider) Classify(path string) provider.Mutability {
if strings.HasSuffix(path, "APKINDEX.tar.gz") {
return provider.Mutable
}
return provider.Immutable
}
func (p *GitHubProvider) ContentType(path string) string {
switch {
case strings.HasSuffix(path, ".apk"):
return "application/vnd.android.package-archive"
case strings.HasSuffix(path, ".tar.gz"):
return "application/gzip"
}
return "application/octet-stream"
}
func (p *GitHubProvider) UpstreamURL(remote models.Remote, path string) string {
return strings.TrimRight(remote.BaseURL, "/") + "/" + strings.TrimLeft(path, "/")
}
func (p *GitHubProvider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
return nil, nil
}
func (p *GitHubProvider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
return p.githubHeaders(ctx, remote, false)
}
// ServeRemote answers a request against a github_alpine remote. It refreshes the
// derived metadata (bounded by mutable_ttl), serves a synthesized per-arch
// APKINDEX.tar.gz, and 302-redirects .apk downloads to the backend
// releases_remote. Returns false only for paths it does not own.
func (p *GitHubProvider) ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, reqPath, proxyBaseURL string, store provider.RemoteMetadataStore) bool {
p.onRequest(remote, store)
// apk requests the index at "./<arch>/APKINDEX.tar.gz"; collapse the
// dot-segment before matching, mirroring the local indexer.
path := normalizeIndexPath(reqPath)
if strings.HasSuffix(path, "APKINDEX.tar.gz") {
p.serveIndex(w, r, remote, path, store)
return true
}
if strings.HasSuffix(path, ".apk") {
if remote.ReleasesRemote == "" {
http.Error(w, "github_alpine remote has no releases_remote configured for downloads", http.StatusInternalServerError)
return true
}
p.serveApkRedirect(w, r, remote, path, proxyBaseURL, store)
return true
}
return false
}
// serveApkRedirect resolves an apk-reconstructed download path — apk builds
// "<arch>/<name>-<version>.apk" itself because APKINDEX carries no filename — to
// the real github-relative asset path stored on the metadata row, then redirects
// to the backend releases_remote. Passing the inbound path through verbatim would
// point at a nonexistent, allowlist-denied github.com path.
func (p *GitHubProvider) serveApkRedirect(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store provider.RemoteMetadataStore) {
arch := strings.TrimSuffix(path[:strings.LastIndex(path, "/")+1], "/")
basename := path[strings.LastIndex(path, "/")+1:]
if arch == "" || strings.Contains(arch, "/") {
http.Error(w, "apk download must be requested per-arch: <arch>/<name>-<version>.apk", http.StatusNotFound)
return
}
reader, ok := store.(provider.AlpineMetadataReader)
if !ok {
http.Error(w, "alpine metadata not available", http.StatusInternalServerError)
return
}
sctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), p.serveTimeout)
defer cancel()
rows, err := reader.ListAlpineMetadataEntries(sctx, remote.Name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
for _, row := range rows {
if row.Arch == arch && row.Name+"-"+row.Version+".apk" == basename {
loc := strings.TrimRight(proxyBaseURL, "/") + "/api/v1/remote/" + remote.ReleasesRemote + "/" + strings.TrimLeft(row.FilePath, "/")
http.Redirect(w, r, loc, http.StatusFound)
return
}
}
http.Error(w, "package not found", http.StatusNotFound)
}
func (p *GitHubProvider) serveIndex(w http.ResponseWriter, r *http.Request, remote models.Remote, path string, store provider.RemoteMetadataStore) {
arch := strings.TrimSuffix(path, "APKINDEX.tar.gz")
arch = strings.Trim(arch, "/")
if arch == "" || strings.Contains(arch, "/") {
http.Error(w, "APKINDEX must be requested per-arch: <arch>/APKINDEX.tar.gz", http.StatusNotFound)
return
}
// Serve on a context detached from the inbound request so a client disconnect
// never cancels the metadata DB read and surfaces as a 500.
sctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), p.serveTimeout)
defer cancel()
if p.syncer != nil && !p.ensurePrimed(sctx, remote, store) {
w.Header().Set("Retry-After", "5")
http.Error(w, "metadata is being prepared, retry shortly", http.StatusServiceUnavailable)
return
}
reader, ok := store.(provider.AlpineMetadataReader)
if !ok {
http.Error(w, "alpine metadata not available", http.StatusInternalServerError)
return
}
metas, err := reader.ListAlpineMetadataEntries(sctx, remote.Name)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
return
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
var filtered []provider.AlpineMetadata
for _, m := range metas {
if m.Arch == arch {
filtered = append(filtered, m)
}
}
w.Header().Set("Content-Type", "application/gzip")
w.WriteHeader(http.StatusOK)
w.Write(generateAPKIndex(filtered))
}
// onRequest keeps a remote's derived metadata fresh off the request path.
func (p *GitHubProvider) onRequest(remote models.Remote, store provider.RemoteMetadataStore) {
if p.syncer != nil {
p.syncer.enqueue(remote, false)
return
}
p.refresh(remote, store)
}
// ensurePrimed returns true once the remote has at least one cached row. On an
// empty cache it enqueues a prime and polls briefly for it to land.
func (p *GitHubProvider) ensurePrimed(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) bool {
if !p.cacheEmpty(ctx, store, remote.Name) {
return true
}
if p.syncer != nil {
p.syncer.enqueue(remote, true)
}
deadline := time.Now().Add(p.coldWait)
for time.Now().Before(deadline) {
select {
case <-ctx.Done():
return false
case <-time.After(400 * time.Millisecond):
}
if !p.cacheEmpty(ctx, store, remote.Name) {
return true
}
}
return false
}
func (p *GitHubProvider) cacheEmpty(ctx context.Context, store provider.RemoteMetadataStore, name string) bool {
reader, ok := store.(provider.AlpineMetadataReader)
if !ok {
return false
}
rows, err := reader.ListAlpineMetadataEntries(ctx, name)
if err != nil {
return false
}
return len(rows) == 0
}
// refresh brings the derived metadata up to date without coupling the scan to
// the inbound request (legacy inline path used without a syncer / in unit tests).
func (p *GitHubProvider) refresh(remote models.Remote, store provider.RemoteMetadataStore) {
ttl := time.Duration(remote.MutableTTL) * time.Second
if ttl <= 0 {
ttl = 5 * time.Minute
}
p.mu.Lock()
last, ok := p.lastScan[remote.Name]
fresh := ok && time.Since(last) < ttl
if fresh || p.scanning[remote.Name] {
p.mu.Unlock()
return
}
p.scanning[remote.Name] = true
p.mu.Unlock()
if p.cacheEmpty(context.Background(), store, remote.Name) {
p.runScan(remote, store)
return
}
go p.runScan(remote, store)
}
func (p *GitHubProvider) runScan(remote models.Remote, store provider.RemoteMetadataStore) {
defer func() {
p.mu.Lock()
delete(p.scanning, remote.Name)
p.mu.Unlock()
}()
ctx, cancel := context.WithTimeout(context.Background(), p.scanTimeout)
defer cancel()
if err := p.scan(ctx, remote, store); err != nil {
slog.Error("github_alpine: release scan failed", "remote", remote.Name, "error", err)
return
}
p.mu.Lock()
p.lastScan[remote.Name] = time.Now()
p.mu.Unlock()
}
// scan runs a full unconditional derive. Retained for the legacy inline refresh
// path and existing tests; the syncer uses scanWithState.
func (p *GitHubProvider) scan(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) error {
_, _, err := p.scanWithState(ctx, remote, store, "")
return err
}
// scanWithState derives metadata incrementally. It sends the prior releases-list
// ETag as a conditional request: a 304 means nothing changed. On a 200 it diffs
// the release assets against the cache, derives only new/changed assets, prunes
// assets that disappeared, and returns the new ETag.
func (p *GitHubProvider) scanWithState(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore, etag string) (newEtag string, changed bool, err error) {
inserter, ok := store.(provider.AlpineMetadataStore)
if !ok {
return etag, false, errors.New("store does not support alpine metadata writes")
}
deleter, ok := store.(provider.AlpineMetadataDeleter)
if !ok {
return etag, false, errors.New("store does not support alpine metadata deletes")
}
reader, ok := store.(provider.AlpineMetadataReader)
if !ok {
return etag, false, errors.New("store does not support alpine metadata reads")
}
releases, newEtag, notModified, err := p.fetchReleases(ctx, remote, etag)
if err != nil {
return etag, false, err
}
if notModified {
return etag, false, nil
}
existing, err := reader.ListAlpineMetadataEntries(ctx, remote.Name)
if err != nil {
return newEtag, false, err
}
existingByPath := make(map[string]provider.AlpineMetadata, len(existing))
for _, m := range existing {
existingByPath[m.FilePath] = m
}
allow, err := compilePatterns(remote.Patterns)
if err != nil {
return newEtag, false, err
}
seen := map[string]bool{}
for _, rel := range releases {
if rel.Draft {
continue
}
for _, asset := range rel.Assets {
if !strings.HasSuffix(strings.ToLower(asset.Name), ".apk") {
continue
}
if !matchesAny(allow, asset.Name) {
continue
}
fp := assetPath(asset)
if fp == "" {
continue
}
seen[fp] = true
if cur, ok := existingByPath[fp]; ok {
if asset.Digest == "" || cur.ContentHash == asset.Digest {
continue
}
_ = deleter.DeleteAlpineMetadata(ctx, remote.Name, fp)
}
meta, err := p.deriveAsset(ctx, remote, asset, fp)
if err != nil {
slog.Warn("github_alpine: derive asset failed", "remote", remote.Name, "asset", asset.Name, "error", err)
continue
}
if err := inserter.InsertAlpineMetadata(ctx, meta); err != nil {
slog.Error("github_alpine: insert metadata failed", "remote", remote.Name, "asset", asset.Name, "error", err)
continue
}
slog.Info("github_alpine: derived asset", "remote", remote.Name, "name", meta.Name, "version", meta.Version, "arch", meta.Arch)
}
}
for fp := range existingByPath {
if !seen[fp] {
_ = deleter.DeleteAlpineMetadata(ctx, remote.Name, fp)
}
}
return newEtag, true, nil
}
type ghRelease struct {
TagName string `json:"tag_name"`
Draft bool `json:"draft"`
Assets []ghAsset `json:"assets"`
}
type ghAsset struct {
Name string `json:"name"`
Size int64 `json:"size"`
BrowserDownloadURL string `json:"browser_download_url"`
Digest string `json:"digest"`
}
// fetchReleases lists a repo's releases, sending the prior ETag as If-None-Match
// on page 1 so an unchanged repo short-circuits to notModified. Every call waits
// on the shared limiter first.
func (p *GitHubProvider) fetchReleases(ctx context.Context, remote models.Remote, etag string) (all []ghRelease, newEtag string, notModified bool, err error) {
base := strings.TrimRight(remote.BaseURL, "/") + "/releases"
for page := 1; page <= p.pageCap; page++ {
u := fmt.Sprintf("%s?per_page=100&page=%d", base, page)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return nil, "", false, err
}
hdr, err := p.githubHeaders(ctx, remote, true)
if err != nil {
return nil, "", false, err
}
copyHeaders(req, hdr)
if page == 1 && etag != "" {
req.Header.Set("If-None-Match", etag)
}
if err := p.limiterWait(ctx); err != nil {
return nil, "", false, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, "", false, err
}
if page == 1 && resp.StatusCode == http.StatusNotModified {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
return nil, etag, true, nil
}
body, err := io.ReadAll(resp.Body)
respEtag := resp.Header.Get("ETag")
resp.Body.Close()
if err != nil {
return nil, "", false, err
}
if resp.StatusCode != http.StatusOK {
return nil, "", false, fmt.Errorf("github releases API %s: status %d", u, resp.StatusCode)
}
if page == 1 {
newEtag = respEtag
}
var releases []ghRelease
if err := json.Unmarshal(body, &releases); err != nil {
return nil, "", false, fmt.Errorf("decode releases: %w", err)
}
if len(releases) == 0 {
break
}
all = append(all, releases...)
if len(releases) < 100 {
break
}
}
return all, newEtag, false, nil
}
func (p *GitHubProvider) deriveAsset(ctx context.Context, remote models.Remote, asset ghAsset, fp string) (*provider.AlpineMetadata, error) {
meta, err := p.fetchPkginfo(ctx, remote, asset.BrowserDownloadURL)
if err != nil {
return nil, err
}
if meta.Name == "" || meta.Arch == "" {
return nil, errors.New(".PKGINFO missing pkgname/arch")
}
meta.RepoName = remote.Name
meta.FilePath = fp
// S: the on-disk .apk size comes straight from the releases API, so we never
// download the body just to size it.
meta.DownloadSize = asset.Size
// ContentHash records the GitHub asset digest (when present) purely so the
// next scan can detect a changed asset; unlike deb it is not the index
// checksum (that is the Q1 control-stream sum already set in fetchPkginfo).
if asset.Digest != "" {
meta.ContentHash = asset.Digest
}
return meta, nil
}
// fetchPkginfo pulls only the front of the .apk with a ranged GET and derives the
// .PKGINFO fields plus the apk pull checksum (C: = Q1 + base64(sha1(control gzip
// stream))). The control stream sits near the front, so a small prefix suffices;
// a prefix that truncates it doubles the range and retries.
func (p *GitHubProvider) fetchPkginfo(ctx context.Context, remote models.Remote, downloadURL string) (*provider.AlpineMetadata, error) {
n := p.headerInitial
for {
body, full, err := p.rangeGet(ctx, remote, downloadURL, n)
if err != nil {
return nil, err
}
meta, complete, perr := pkginfoFromPrefix(body)
if perr != nil {
return nil, fmt.Errorf("parse apk .PKGINFO: %w", perr)
}
if complete {
return meta, nil
}
if full || n >= p.headerMax {
return nil, fmt.Errorf(".PKGINFO not found within %d bytes of %s", n, downloadURL)
}
n *= 2
if n > p.headerMax {
n = p.headerMax
}
}
}
// pkginfoFromPrefix parses the concatenated gzip streams present in a front
// prefix of an .apk. It walks each fully-covered gzip member until it finds the
// control stream (the one whose tar carries .PKGINFO), computes the Q1 pull
// checksum from that stream's raw bytes, and reads the .PKGINFO fields. A prefix
// too short to fully cover the control stream returns complete=false so the
// caller can widen the range.
func pkginfoFromPrefix(prefix []byte) (meta *provider.AlpineMetadata, complete bool, err error) {
br := bytes.NewReader(prefix)
zr, zerr := gzip.NewReader(br)
if zerr != nil {
if zerr == io.EOF || zerr == io.ErrUnexpectedEOF {
return nil, false, nil
}
return nil, false, zerr
}
prev := 0
for {
zr.Multistream(false)
out, rerr := io.ReadAll(zr)
if rerr != nil {
// A member truncated by the range boundary is not an error — widen.
if rerr == io.ErrUnexpectedEOF || rerr == io.EOF {
return nil, false, nil
}
return nil, false, rerr
}
end := len(prefix) - br.Len()
raw := prefix[prev:end]
if pkginfo, ok := pkginfoFromTar(out); ok {
m := parsePkginfo(pkginfo)
sum := sha1.Sum(raw)
m.Checksum = "Q1" + base64.StdEncoding.EncodeToString(sum[:])
return m, true, nil
}
prev = end
if rsterr := zr.Reset(br); rsterr != nil {
if rsterr == io.EOF {
// No more complete members in the prefix; the control stream is
// either not covered yet or genuinely absent — let the caller
// decide by widening (or hitting the full-object guard).
return nil, false, nil
}
if rsterr == io.ErrUnexpectedEOF {
return nil, false, nil
}
return nil, false, rsterr
}
}
}
// rangeGet returns the first n bytes of downloadURL. full is true when the
// response body was shorter than n (i.e. we already have the whole object).
func (p *GitHubProvider) rangeGet(ctx context.Context, remote models.Remote, downloadURL string, n int64) ([]byte, bool, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
if err != nil {
return nil, false, err
}
hdr, err := p.githubHeaders(ctx, remote, false)
if err != nil {
return nil, false, err
}
copyHeaders(req, hdr)
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", n-1))
if err := p.limiterWait(ctx); err != nil {
return nil, false, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, false, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
return nil, false, fmt.Errorf("range GET %s: status %d", downloadURL, resp.StatusCode)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, n))
if err != nil {
return nil, false, err
}
full := int64(len(body)) < n
return body, full, nil
}
// assetPath is the package's location relative to github.com — the path the
// backend releases_remote (base https://github.com) proxies. It doubles as the
// alpine_metadata key and the redirect target, so an .apk download resolves back
// to this remote and redirects to the backend.
func assetPath(asset ghAsset) string {
u, err := url.Parse(asset.BrowserDownloadURL)
if err != nil {
return ""
}
return strings.TrimPrefix(u.Path, "/")
}
// githubHeaders builds the outbound headers for a GitHub request, attaching a
// bearer credential when one is available. A per-remote credential wins; absent
// that, the process-wide server credential is used; absent both, the request is
// unauthenticated.
func (p *GitHubProvider) githubHeaders(ctx context.Context, remote models.Remote, api bool) (http.Header, error) {
h := http.Header{}
if api {
h.Set("Accept", "application/vnd.github+json")
h.Set("X-GitHub-Api-Version", "2022-11-28")
}
tok, err := p.githubToken(ctx, remote)
if err != nil {
return nil, err
}
if tok != "" {
h.Set("Authorization", "Bearer "+tok)
}
return h, nil
}
// githubToken resolves the bearer token for a remote. Precedence: a per-remote
// credential (password, then username) overrides the server credential.
func (p *GitHubProvider) githubToken(ctx context.Context, remote models.Remote) (string, error) {
if remote.Password != "" {
return remote.Password, nil
}
if remote.Username != "" {
return remote.Username, nil
}
if c := p.serverCredential(); c != nil {
return c.Token(ctx)
}
return "", nil
}
func (p *GitHubProvider) serverCredential() githubauth.Credential {
if p.serverCred != nil {
return p.serverCred
}
return githubauth.Server()
}
func copyHeaders(req *http.Request, h http.Header) {
for k, vals := range h {
for _, v := range vals {
req.Header.Add(k, v)
}
}
}
func compilePatterns(patterns []string) ([]*regexp.Regexp, error) {
var out []*regexp.Regexp
for _, p := range patterns {
re, err := regexp.Compile(p)
if err != nil {
return nil, fmt.Errorf("invalid pattern %q: %w", p, err)
}
out = append(out, re)
}
return out, nil
}
func matchesAny(res []*regexp.Regexp, s string) bool {
if len(res) == 0 {
return true
}
for _, re := range res {
if re.MatchString(s) {
return true
}
}
return false
}
+497
View File
@@ -0,0 +1,497 @@
package alpine
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeStore is an in-memory provider.RemoteMetadataStore + AlpineMetadata
// store/reader/deleter keyed by file_path, mirroring the (repo_name, file_path)
// uniqueness of the real alpine_metadata table.
type fakeStore struct {
mu sync.Mutex
rows map[string]provider.AlpineMetadata
}
func newFakeStore() *fakeStore { return &fakeStore{rows: map[string]provider.AlpineMetadata{}} }
func (f *fakeStore) InsertAlpineMetadata(_ context.Context, m *provider.AlpineMetadata) error {
f.mu.Lock()
defer f.mu.Unlock()
if _, ok := f.rows[m.FilePath]; ok {
return nil // ON CONFLICT DO NOTHING
}
f.rows[m.FilePath] = *m
return nil
}
func (f *fakeStore) DeleteAlpineMetadata(_ context.Context, _, filePath string) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.rows, filePath)
return nil
}
func (f *fakeStore) ListAlpineMetadataEntries(ctx context.Context, _ string) ([]provider.AlpineMetadata, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
out := make([]provider.AlpineMetadata, 0, len(f.rows))
for _, m := range f.rows {
out = append(out, m)
}
return out, nil
}
// The generic RemoteMetadataStore surface (rpm/deb) is unused by the alpine
// github provider but required to satisfy the interface passed to ServeRemote.
func (f *fakeStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
func (f *fakeStore) DeleteRPMMetadata(context.Context, string, string) error { return nil }
func (f *fakeStore) ListRPMMetadataEntries(context.Context, string) ([]provider.RPMMetadata, error) {
return nil, nil
}
func (f *fakeStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
func (f *fakeStore) DeleteDebMetadata(context.Context, string, string) error { return nil }
var _ provider.RemoteMetadataStore = (*fakeStore)(nil)
// githubFixture serves the releases API and the .apk asset downloads (with Range
// support) for a set of packages. digest controls whether the asset carries a
// sha256 digest (change-detection path) or not.
type githubFixture struct {
srv *httptest.Server
apkBytes map[string][]byte
rangeHit map[string]int
fullHit map[string]int
etag string
releasesHit int
notModHit int
releaseAuth string
assetAuth string
mu sync.Mutex
}
func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
t.Helper()
f := &githubFixture{
apkBytes: map[string][]byte{},
rangeHit: map[string]int{},
fullHit: map[string]int{},
}
f.apkBytes["demo-1.2.3-r0.apk"] = testsupport.MinimalApk("demo", "1.2.3-r0", "x86_64")
mux := http.NewServeMux()
mux.HandleFunc("/repos/acme/tools/releases", func(w http.ResponseWriter, r *http.Request) {
page := r.URL.Query().Get("page")
if page != "" && page != "1" {
w.Write([]byte("[]"))
return
}
f.mu.Lock()
f.releasesHit++
f.releaseAuth = r.Header.Get("Authorization")
etag := f.etag
if etag != "" && r.Header.Get("If-None-Match") == etag {
f.notModHit++
f.mu.Unlock()
w.WriteHeader(http.StatusNotModified)
return
}
f.mu.Unlock()
if etag != "" {
w.Header().Set("ETag", etag)
}
var assets []map[string]any
for name := range f.apkBytes {
a := map[string]any{
"name": name,
"size": len(f.apkBytes[name]),
"browser_download_url": f.srv.URL + "/acme/tools/releases/download/v1.2.3/" + name,
}
if withDigest {
sum := sha256.Sum256(f.apkBytes[name])
a["digest"] = "sha256:" + hex.EncodeToString(sum[:])
}
assets = append(assets, a)
}
rel := []map[string]any{{"tag_name": "v1.2.3", "draft": false, "assets": assets}}
json.NewEncoder(w).Encode(rel)
})
mux.HandleFunc("/acme/tools/releases/download/", func(w http.ResponseWriter, r *http.Request) {
name := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
body, ok := f.apkBytes[name]
if !ok {
http.Error(w, "not found", 404)
return
}
rng := r.Header.Get("Range")
f.mu.Lock()
f.assetAuth = r.Header.Get("Authorization")
if rng != "" {
f.rangeHit[name]++
} else {
f.fullHit[name]++
}
f.mu.Unlock()
if rng == "" {
w.WriteHeader(200)
w.Write(body)
return
}
var end int
fmt.Sscanf(rng, "bytes=0-%d", &end)
if end >= len(body)-1 {
end = len(body) - 1
}
w.Header().Set("Content-Range", fmt.Sprintf("bytes 0-%d/%d", end, len(body)))
w.Header().Set("Content-Length", strconv.Itoa(end+1))
w.WriteHeader(http.StatusPartialContent)
w.Write(body[:end+1])
})
f.srv = httptest.NewServer(mux)
t.Cleanup(f.srv.Close)
return f
}
func (f *githubFixture) remote() models.Remote {
return models.Remote{
Name: "acme-apk",
PackageType: models.PackageGitHubAlpine,
BaseURL: f.srv.URL + "/repos/acme/tools",
ReleasesRemote: "github",
MutableTTL: 3600,
}
}
func newTestProvider() *GitHubProvider {
p := newGitHubProvider()
p.headerInitial = 32 // force the ranged-fetch retry loop against the tiny fixture
p.headerMax = 1 << 20
return p
}
const demoPath = "acme/tools/releases/download/v1.2.3/demo-1.2.3-r0.apk"
func TestGitHubScanDerivesPkginfoFromPrefix(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
metas, _ := store.ListAlpineMetadataEntries(context.Background(), "acme-apk")
if len(metas) != 1 {
t.Fatalf("want 1 metadata row, got %d", len(metas))
}
m := metas[0]
if m.Name != "demo" || m.Version != "1.2.3-r0" || m.Arch != "x86_64" {
t.Fatalf("bad .PKGINFO fields: %+v", m)
}
if m.FilePath != demoPath {
t.Fatalf("FilePath = %q, want %q", m.FilePath, demoPath)
}
if int(m.DownloadSize) != len(fx.apkBytes["demo-1.2.3-r0.apk"]) {
t.Fatalf("DownloadSize = %d, want %d", m.DownloadSize, len(fx.apkBytes["demo-1.2.3-r0.apk"]))
}
if !strings.HasPrefix(m.Checksum, "Q1") {
t.Fatalf("Checksum not a Q1 pull checksum: %q", m.Checksum)
}
// The C: checksum must equal Q1 over the raw control gzip stream, matching the
// local-upload parser applied to the same bytes.
want, err := parseApk(fx.apkBytes["demo-1.2.3-r0.apk"])
if err != nil {
t.Fatalf("reference parseApk: %v", err)
}
if m.Checksum != want.Checksum {
t.Fatalf("Checksum = %q, want %q (Q1 of control stream)", m.Checksum, want.Checksum)
}
if fx.fullHit["demo-1.2.3-r0.apk"] != 0 {
t.Fatalf("expected no full download, got %d", fx.fullHit["demo-1.2.3-r0.apk"])
}
if fx.rangeHit["demo-1.2.3-r0.apk"] == 0 {
t.Fatalf("expected ranged .PKGINFO fetch")
}
}
func TestGitHubServeRemoteIndexAndRedirect(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
const proxyBase = "https://artifactapi.example"
// The per-arch index is served and triggers the initial scan.
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-apk/x86_64/APKINDEX.tar.gz", nil)
if !p.ServeRemote(rec, req, remote, "x86_64/APKINDEX.tar.gz", proxyBase, store) {
t.Fatal("ServeRemote did not handle APKINDEX")
}
if rec.Code != 200 {
t.Fatalf("APKINDEX bad: code=%d body=%s", rec.Code, rec.Body.String())
}
idx := readAPKIndex(t, rec.Body.Bytes())
if !strings.Contains(idx, "P:demo") || !strings.Contains(idx, "A:x86_64") {
t.Fatalf("APKINDEX missing package record: %s", idx)
}
if !strings.Contains(idx, "C:Q1") {
t.Fatalf("APKINDEX missing pull checksum: %s", idx)
}
// A different arch yields an empty (but valid) index.
rec = httptest.NewRecorder()
req = httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, "aarch64/APKINDEX.tar.gz", proxyBase, store) {
t.Fatal("ServeRemote did not handle aarch64 APKINDEX")
}
if rec.Code != 200 {
t.Fatalf("empty-arch index bad: %d", rec.Code)
}
if got := readAPKIndex(t, rec.Body.Bytes()); strings.Contains(got, "P:demo") {
t.Fatalf("aarch64 index should not carry the x86_64 package: %s", got)
}
// An .apk request arrives in apk's reconstructed shape
// "<arch>/<name>-<version>.apk" (APKINDEX carries no filename), NOT as the
// github-relative FilePath. ServeRemote must resolve it back to the stored
// FilePath before redirecting to the backend releases_remote.
rec = httptest.NewRecorder()
req = httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-apk/x86_64/demo-1.2.3-r0.apk", nil)
if !p.ServeRemote(rec, req, remote, "x86_64/demo-1.2.3-r0.apk", proxyBase, store) {
t.Fatal("ServeRemote did not handle .apk")
}
if rec.Code != http.StatusFound {
t.Fatalf("want 302, got %d", rec.Code)
}
wantLoc := proxyBase + "/api/v1/remote/github/" + demoPath
if got := rec.Header().Get("Location"); got != wantLoc {
t.Fatalf("Location = %q, want %q (must be the stored FilePath, not the inbound path)", got, wantLoc)
}
}
// An apk download whose reconstructed "<arch>/<name>-<version>.apk" matches no
// cached row must 404, never redirect to a bad path.
func TestGitHubServeRemoteApkRedirectNotFound(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
// Warm the cache so the store is populated but lacks the requested package.
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("warm scan: %v", err)
}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-apk/x86_64/nope-9.9.9.apk", nil)
if !p.ServeRemote(rec, req, remote, "x86_64/nope-9.9.9.apk", "https://x", store) {
t.Fatal("ServeRemote did not handle .apk")
}
if rec.Code != http.StatusNotFound {
t.Fatalf("want 404 for unknown package, got %d (Location=%q)", rec.Code, rec.Header().Get("Location"))
}
}
// apk requests the index at "./<arch>/APKINDEX.tar.gz"; ServeRemote must collapse
// the dot-segment and synthesize the same index as the un-prefixed request.
func TestGitHubServeRemoteApkDotSegment(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
const proxyBase = "https://artifactapi.example"
serve := func(path string) *httptest.ResponseRecorder {
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-apk/"+path, nil)
if !p.ServeRemote(rec, req, remote, path, proxyBase, store) {
t.Fatalf("ServeRemote did not handle %q", path)
}
return rec
}
plain, dotted := serve("x86_64/APKINDEX.tar.gz"), serve("./x86_64/APKINDEX.tar.gz")
if plain.Code != 200 || dotted.Code != 200 {
t.Fatalf("index: plain=%d dotted=%d, want 200/200", plain.Code, dotted.Code)
}
if !bytes.Equal(plain.Body.Bytes(), dotted.Body.Bytes()) {
t.Error("./<arch>/APKINDEX.tar.gz body differs from the un-prefixed body")
}
}
func TestGitHubServeRemoteRejectsNonPerArchIndex(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, fx.remote(), "APKINDEX.tar.gz", "https://x", store) {
t.Fatal("expected handled")
}
if rec.Code != http.StatusNotFound {
t.Fatalf("bare APKINDEX must 404 (per-arch required), got %d", rec.Code)
}
}
// A canceled inbound request must still serve the warm cache (detached context),
// not turn the metadata read into a 500.
func TestGitHubServeRemoteCanceledRequestServesCache(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("warm scan: %v", err)
}
p.mu.Lock()
p.lastScan[remote.Name] = time.Now()
p.mu.Unlock()
ctx, cancel := context.WithCancel(context.Background())
cancel()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-apk/x86_64/APKINDEX.tar.gz", nil).WithContext(ctx)
if !p.ServeRemote(rec, req, remote, "x86_64/APKINDEX.tar.gz", "https://x", store) {
t.Fatal("ServeRemote did not handle APKINDEX")
}
if rec.Code != http.StatusOK {
t.Fatalf("canceled request must serve cache, not error; got code=%d body=%s", rec.Code, rec.Body.String())
}
if got := readAPKIndex(t, rec.Body.Bytes()); !strings.Contains(got, "P:demo") {
t.Fatalf("expected index served from cache, got %s", got)
}
}
func TestGitHubServeRemoteRedirectRequiresReleasesRemote(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
remote.ReleasesRemote = ""
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, demoPath, "https://x", store) {
t.Fatal("expected handled")
}
if rec.Code != http.StatusInternalServerError {
t.Fatalf("want 500 when releases_remote unset, got %d", rec.Code)
}
}
func TestGitHubScanPrunesRemovedAssets(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
if rows, _ := store.ListAlpineMetadataEntries(context.Background(), "acme-apk"); len(rows) != 1 {
t.Fatalf("want 1 row after first scan, got %d", len(rows))
}
delete(fx.apkBytes, "demo-1.2.3-r0.apk")
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("rescan: %v", err)
}
if rows, _ := store.ListAlpineMetadataEntries(context.Background(), "acme-apk"); len(rows) != 0 {
t.Fatalf("want 0 rows after prune, got %d", len(rows))
}
}
func TestGitHubAssetPatternFilter(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.apkBytes["other-9-r0.apk"] = testsupport.MinimalApk("other", "9-r0", "aarch64")
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
remote.Patterns = []string{`^demo-.*\.apk$`}
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("scan: %v", err)
}
rows, _ := store.ListAlpineMetadataEntries(context.Background(), "acme-apk")
if len(rows) != 1 || rows[0].Name != "demo" {
t.Fatalf("pattern filter failed, rows=%+v", rows)
}
}
// Multi-arch: each asset's index record lands under its own arch bucket.
func TestGitHubServeRemotePerArchGrouping(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.apkBytes["demo-1.2.3-r0-aarch64.apk"] = testsupport.MinimalApk("demo", "1.2.3-r0", "aarch64")
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
const proxyBase = "https://x"
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("scan: %v", err)
}
serve := func(arch string) string {
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, arch+"/APKINDEX.tar.gz", proxyBase, store) {
t.Fatalf("ServeRemote did not handle %s", arch)
}
return readAPKIndex(t, rec.Body.Bytes())
}
x86 := serve("x86_64")
if !strings.Contains(x86, "A:x86_64") || strings.Contains(x86, "A:aarch64") {
t.Fatalf("x86_64 index leaked another arch: %s", x86)
}
arm := serve("aarch64")
if !strings.Contains(arm, "A:aarch64") || strings.Contains(arm, "A:x86_64") {
t.Fatalf("aarch64 index leaked another arch: %s", arm)
}
}
func readAPKIndex(t *testing.T, gzBytes []byte) string {
t.Helper()
gz, err := gzip.NewReader(bytes.NewReader(gzBytes))
if err != nil {
t.Fatalf("gzip: %v", err)
}
tr := tar.NewReader(gz)
for {
hdr, err := tr.Next()
if err != nil {
t.Fatal("APKINDEX member missing from tar.gz")
}
if strings.TrimPrefix(hdr.Name, "./") == "APKINDEX" {
body, err := io.ReadAll(tr)
if err != nil {
t.Fatalf("read APKINDEX: %v", err)
}
return string(body)
}
}
}
+238
View File
@@ -0,0 +1,238 @@
package alpine
import (
"context"
"crypto/rand"
"encoding/hex"
"log/slog"
"os"
"sync"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
const (
syncLeaseDuration = 15 * time.Minute
defaultSyncFreshness = 5 * time.Minute
jobQueueDepth = 256
)
// SyncStore is the persistence surface the alpine syncer needs: the metadata
// cache it primes plus the shared sync-state coordination (remote enumeration
// and the per-remote lease). *database.DB satisfies it.
type SyncStore interface {
provider.RemoteMetadataStore
ListGitHubAlpineRemotes(ctx context.Context) ([]models.Remote, error)
ClaimGitHubAlpineSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (claimed bool, etag string, err error)
ReleaseGitHubAlpineSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error
}
// SyncConfig tunes the shared syncer. Zero values fall back to safe defaults.
type SyncConfig struct {
RatePerSec float64
Burst int
Workers int
PollInterval time.Duration
}
type syncJob struct {
remote models.Remote
prime bool
}
// Syncer is the single per-process background worker that keeps every
// github_alpine remote's derived metadata fresh. It owns a deduped work queue, a
// pool of workers, and a global token-bucket rate limiter shared across all
// remotes and bound onto the github_alpine provider. Periodic checks are gated by
// a shared DB lease so, across replicas, only one performs each scan.
type Syncer struct {
store SyncStore
prov *GitHubProvider
limiter *rate.Limiter
cfg SyncConfig
owner string
jobs chan syncJob
mu sync.Mutex
active map[string]bool
}
// NewSyncer builds the syncer bound to the process-wide github_alpine provider
// singleton. Call Run to start it.
func NewSyncer(store SyncStore, cfg SyncConfig) *Syncer {
return newSyncer(store, gitHubProvider, cfg)
}
func newSyncer(store SyncStore, prov *GitHubProvider, cfg SyncConfig) *Syncer {
if cfg.RatePerSec <= 0 {
cfg.RatePerSec = 1
}
if cfg.Burst <= 0 {
cfg.Burst = 5
}
if cfg.Workers <= 0 {
cfg.Workers = 3
}
if cfg.PollInterval <= 0 {
cfg.PollInterval = 60 * time.Second
}
lim := rate.NewLimiter(rate.Limit(cfg.RatePerSec), cfg.Burst)
s := &Syncer{
store: store,
prov: prov,
limiter: lim,
cfg: cfg,
owner: leaseOwner(),
jobs: make(chan syncJob, jobQueueDepth),
active: map[string]bool{},
}
prov.limiter = lim
prov.syncer = s
return s
}
// Run starts the worker pool and the periodic scheduler and blocks until ctx is
// canceled, at which point it drains in-flight scans and returns.
func (s *Syncer) Run(ctx context.Context) {
slog.Info("github_alpine syncer started",
"rate_per_sec", s.cfg.RatePerSec, "burst", s.cfg.Burst,
"workers", s.cfg.Workers, "poll_interval", s.cfg.PollInterval, "owner", s.owner)
var wg sync.WaitGroup
for i := 0; i < s.cfg.Workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
s.worker(ctx)
}()
}
ticker := time.NewTicker(s.cfg.PollInterval)
defer ticker.Stop()
s.schedule(ctx)
for {
select {
case <-ctx.Done():
wg.Wait()
slog.Info("github_alpine syncer stopped")
return
case <-ticker.C:
s.schedule(ctx)
}
}
}
// schedule enqueues a periodic check for every github_alpine remote. The DB lease
// enforces the per-remote mutable_ttl cadence and cross-replica coordination.
func (s *Syncer) schedule(ctx context.Context) {
remotes, err := s.store.ListGitHubAlpineRemotes(ctx)
if err != nil {
slog.Error("github_alpine syncer: list remotes", "error", err)
return
}
for _, r := range remotes {
s.enqueue(r, false)
}
}
// EnqueuePrime queues an immediate background prime for a freshly created remote.
func (s *Syncer) EnqueuePrime(remote models.Remote) {
if s == nil {
return
}
s.enqueue(remote, true)
}
// enqueue adds a job unless the remote is already queued or in-flight, coalescing
// duplicate requests down to one scan. It never blocks.
func (s *Syncer) enqueue(remote models.Remote, prime bool) {
s.mu.Lock()
if s.active[remote.Name] {
s.mu.Unlock()
return
}
s.active[remote.Name] = true
s.mu.Unlock()
select {
case s.jobs <- syncJob{remote: remote, prime: prime}:
default:
s.mu.Lock()
delete(s.active, remote.Name)
s.mu.Unlock()
}
}
func (s *Syncer) worker(ctx context.Context) {
for {
select {
case <-ctx.Done():
return
case job := <-s.jobs:
s.process(ctx, job)
}
}
}
// process claims the shared lease and, if won, runs an incremental scan. Losing
// the claim (another replica scanning, or not yet due) is a no-op.
func (s *Syncer) process(ctx context.Context, job syncJob) {
defer func() {
s.mu.Lock()
delete(s.active, job.remote.Name)
s.mu.Unlock()
}()
freshness := time.Duration(job.remote.MutableTTL) * time.Second
if freshness <= 0 {
freshness = defaultSyncFreshness
}
if job.prime {
freshness = 0
}
claimed, etag, err := s.store.ClaimGitHubAlpineSyncLease(ctx, job.remote.Name, s.owner, freshness, syncLeaseDuration)
if err != nil {
slog.Error("github_alpine syncer: claim lease", "remote", job.remote.Name, "error", err)
return
}
if !claimed {
return
}
scanCtx, cancel := context.WithTimeout(ctx, s.prov.scanTimeout)
defer cancel()
newEtag, changed, scanErr := s.prov.scanWithState(scanCtx, job.remote, s.store, etag)
releaseEtag := etag
if scanErr == nil {
releaseEtag = newEtag
} else {
slog.Error("github_alpine syncer: scan failed", "remote", job.remote.Name, "error", scanErr)
}
relCtx, relCancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer relCancel()
if err := s.store.ReleaseGitHubAlpineSyncLease(relCtx, job.remote.Name, s.owner, releaseEtag, time.Now()); err != nil {
slog.Warn("github_alpine syncer: release lease", "remote", job.remote.Name, "error", err)
}
if scanErr == nil && changed {
slog.Info("github_alpine syncer: refreshed", "remote", job.remote.Name, "prime", job.prime)
}
}
// leaseOwner is a per-replica identity for the lease: hostname plus a random
// suffix so restarts and colocated replicas never collide.
func leaseOwner() string {
host, _ := os.Hostname()
var b [6]byte
_, _ = rand.Read(b[:])
return host + "-" + hex.EncodeToString(b[:])
}
+300
View File
@@ -0,0 +1,300 @@
package alpine
import (
"context"
"net/http"
"net/http/httptest"
"sync"
"testing"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeSyncStore is an in-memory SyncStore: the metadata cache (via the embedded
// fakeStore) plus the shared sync-state lease, whose claim mirrors the atomic
// semantics of the real SQL (recency gate AND no live lease).
type fakeSyncStore struct {
*fakeStore
mu sync.Mutex
remotes []models.Remote
leaseOwner map[string]string
leaseExp map[string]time.Time
lastSynced map[string]time.Time
etags map[string]string
}
func newFakeSyncStore() *fakeSyncStore {
return &fakeSyncStore{
fakeStore: newFakeStore(),
leaseOwner: map[string]string{},
leaseExp: map[string]time.Time{},
lastSynced: map[string]time.Time{},
etags: map[string]string{},
}
}
func (f *fakeSyncStore) ListGitHubAlpineRemotes(_ context.Context) ([]models.Remote, error) {
f.mu.Lock()
defer f.mu.Unlock()
return append([]models.Remote(nil), f.remotes...), nil
}
func (f *fakeSyncStore) ClaimGitHubAlpineSyncLease(_ context.Context, name, owner string, freshness, lease time.Duration) (bool, string, error) {
f.mu.Lock()
defer f.mu.Unlock()
now := time.Now()
ls, hasLS := f.lastSynced[name]
exp, hasExp := f.leaseExp[name]
freshOK := !hasLS || now.Sub(ls) >= freshness
leaseOK := !hasExp || exp.Before(now)
if freshOK && leaseOK {
f.leaseOwner[name] = owner
f.leaseExp[name] = now.Add(lease)
return true, f.etags[name], nil
}
return false, "", nil
}
func (f *fakeSyncStore) ReleaseGitHubAlpineSyncLease(_ context.Context, name, owner, etag string, syncedAt time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.leaseOwner[name] != owner {
return nil
}
f.lastSynced[name] = syncedAt
f.etags[name] = etag
delete(f.leaseOwner, name)
delete(f.leaseExp, name)
return nil
}
func testSyncConfig() SyncConfig {
return SyncConfig{RatePerSec: 1000, Burst: 100, Workers: 1, PollInterval: time.Hour}
}
// (a) A 304 conditional response must derive nothing: no asset fetches and
// changed=false, so an unchanged repo is nearly free.
func TestSyncerConditionalNotModifiedSkipsDerive(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
etag1, changed, err := p.scanWithState(context.Background(), remote, store, "")
if err != nil {
t.Fatalf("first scan: %v", err)
}
if !changed || etag1 != `"v1"` {
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag1)
}
priorRange := fx.rangeHit["demo-1.2.3-r0.apk"]
if priorRange == 0 {
t.Fatal("first scan should have fetched the asset .PKGINFO")
}
etag2, changed2, err := p.scanWithState(context.Background(), remote, store, etag1)
if err != nil {
t.Fatalf("second scan: %v", err)
}
if changed2 {
t.Fatal("304 scan must report changed=false")
}
if etag2 != etag1 {
t.Fatalf("etag changed across 304: %q -> %q", etag1, etag2)
}
if fx.notModHit != 1 {
t.Fatalf("want exactly one 304 releases response, got %d", fx.notModHit)
}
if got := fx.rangeHit["demo-1.2.3-r0.apk"]; got != priorRange {
t.Fatalf("304 scan re-fetched asset .PKGINFO: %d -> %d", priorRange, got)
}
}
// (b) On a real change, only the newly added asset is derived.
func TestSyncerIncrementalDerivesOnlyNewAsset(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
if _, _, err := p.scanWithState(context.Background(), remote, store, ""); err != nil {
t.Fatalf("first scan: %v", err)
}
demoRange := fx.rangeHit["demo-1.2.3-r0.apk"]
fx.apkBytes["other-9-r0.apk"] = testsupport.MinimalApk("other", "9-r0", "aarch64")
fx.etag = `"v2"`
if _, changed, err := p.scanWithState(context.Background(), remote, store, `"v1"`); err != nil || !changed {
t.Fatalf("second scan changed=%v err=%v", changed, err)
}
rows, _ := store.ListAlpineMetadataEntries(context.Background(), remote.Name)
if len(rows) != 2 {
t.Fatalf("want 2 cached rows after incremental derive, got %d", len(rows))
}
if got := fx.rangeHit["demo-1.2.3-r0.apk"]; got != demoRange {
t.Fatalf("already-cached asset was re-fetched: %d -> %d", demoRange, got)
}
if fx.rangeHit["other-9-r0.apk"] == 0 {
t.Fatal("newly added asset was not derived")
}
}
// (c) The shared limiter caps the request rate.
func TestRateLimiterCapsRequestRate(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
p.limiter = rate.NewLimiter(rate.Every(120*time.Millisecond), 1)
remote := fx.remote()
start := time.Now()
for i := 0; i < 3; i++ {
if _, _, _, err := p.fetchReleases(context.Background(), remote, ""); err != nil {
t.Fatalf("fetchReleases %d: %v", i, err)
}
}
if elapsed := time.Since(start); elapsed < 200*time.Millisecond {
t.Fatalf("rate limiter did not throttle: 3 calls took %v, want >= 200ms", elapsed)
}
}
// (d) Concurrent enqueues for the same remote coalesce to a single queued job.
func TestSyncerEnqueueDedup(t *testing.T) {
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := models.Remote{Name: "acme-apk", PackageType: models.PackageGitHubAlpine, MutableTTL: 3600}
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() { defer wg.Done(); s.enqueue(remote, false) }()
}
wg.Wait()
if got := len(s.jobs); got != 1 {
t.Fatalf("want exactly 1 coalesced job, got %d", got)
}
}
// (e) Prime-on-create enqueues a prime job.
func TestSyncerEnqueuePrime(t *testing.T) {
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := models.Remote{Name: "acme-apk", PackageType: models.PackageGitHubAlpine, MutableTTL: 3600}
s.EnqueuePrime(remote)
select {
case job := <-s.jobs:
if !job.prime || job.remote.Name != "acme-apk" {
t.Fatalf("bad prime job: %+v", job)
}
default:
t.Fatal("EnqueuePrime did not enqueue a job")
}
}
// (f) A held lease prevents a second replica from scanning.
func TestSyncerLeasePreventsSecondReplica(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := fx.remote()
claimed, _, err := store.ClaimGitHubAlpineSyncLease(context.Background(), remote.Name, "replica-1", time.Duration(remote.MutableTTL)*time.Second, syncLeaseDuration)
if err != nil || !claimed {
t.Fatalf("replica-1 claim: claimed=%v err=%v", claimed, err)
}
s.process(context.Background(), syncJob{remote: remote})
if fx.releasesHit != 0 {
t.Fatalf("second replica scanned while lease held: %d releases calls", fx.releasesHit)
}
if rows, _ := store.ListAlpineMetadataEntries(context.Background(), remote.Name); len(rows) != 0 {
t.Fatalf("second replica derived metadata while lease held: %d rows", len(rows))
}
}
// With the syncer wired and the cache empty, an index request enqueues a prime
// and returns a retryable 503 when it has not landed within the cold wait.
func TestServeRemoteColdStartReturns503(t *testing.T) {
fx := newGitHubFixture(t, true)
store := newFakeSyncStore()
p := newTestProvider()
p.coldWait = 300 * time.Millisecond
_ = newSyncer(store, p, testSyncConfig()) // binds p.syncer, but no workers running
remote := fx.remote()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-apk/x86_64/APKINDEX.tar.gz", nil)
if !p.ServeRemote(rec, req, remote, "x86_64/APKINDEX.tar.gz", "https://x", store) {
t.Fatal("ServeRemote did not handle APKINDEX")
}
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("cold empty cache must return 503, got %d", rec.Code)
}
if rec.Header().Get("Retry-After") == "" {
t.Fatal("503 should carry Retry-After")
}
if got := len(p.syncer.jobs); got != 1 {
t.Fatalf("cold start did not enqueue a prime, jobs=%d", got)
}
}
// With the cache warm, the same request serves the index immediately (no 503).
func TestServeRemoteWarmCacheServesImmediately(t *testing.T) {
fx := newGitHubFixture(t, true)
store := newFakeSyncStore()
p := newTestProvider()
_ = newSyncer(store, p, testSyncConfig())
remote := fx.remote()
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("warm scan: %v", err)
}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-apk/x86_64/APKINDEX.tar.gz", nil)
if !p.ServeRemote(rec, req, remote, "x86_64/APKINDEX.tar.gz", "https://x", store) {
t.Fatal("ServeRemote did not handle APKINDEX")
}
if rec.Code != http.StatusOK {
t.Fatalf("warm cache must serve 200, got %d body=%s", rec.Code, rec.Body.String())
}
}
// A prime job (freshness 0) runs even right after a sync; a periodic job at the
// same moment is gated by the recency window.
func TestSyncerPrimeBypassesRecencyPeriodicDoesNot(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := fx.remote()
var _ provider.RemoteMetadataStore = store
s.process(context.Background(), syncJob{remote: remote, prime: true})
if rows, _ := store.ListAlpineMetadataEntries(context.Background(), remote.Name); len(rows) != 1 {
t.Fatalf("prime did not derive: %d rows", len(rows))
}
releasesAfterPrime := fx.releasesHit
s.process(context.Background(), syncJob{remote: remote, prime: false})
if fx.releasesHit != releasesAfterPrime {
t.Fatalf("periodic scan ran inside recency window: %d -> %d releases calls", releasesAfterPrime, fx.releasesHit)
}
}
+16 -1
View File
@@ -395,7 +395,7 @@ func generateRelease(metas []provider.DebMetadata) []byte {
arches := uniqueArches(metas) arches := uniqueArches(metas)
var b bytes.Buffer var b bytes.Buffer
fmt.Fprintf(&b, "Date: %s\n", time.Now().UTC().Format(time.RFC1123Z)) fmt.Fprintf(&b, "Date: %s\n", releaseDate(metas).Format(time.RFC1123Z))
fmt.Fprintf(&b, "Architectures: %s\n", strings.Join(arches, " ")) fmt.Fprintf(&b, "Architectures: %s\n", strings.Join(arches, " "))
b.WriteString("Acquire-By-Hash: no\n") b.WriteString("Acquire-By-Hash: no\n")
@@ -410,6 +410,21 @@ func generateRelease(metas []provider.DebMetadata) []byte {
return b.Bytes() return b.Bytes()
} }
// releaseDate derives the Release Date: from the newest package's persisted
// created_at (in UTC) so the file is byte-identical across the no-affinity
// replicas and across regenerations (issue #117); an empty repo falls back to
// the Unix epoch. This never uses wall clock, which also keeps Date: from
// running ahead of any Valid-Until logic.
func releaseDate(metas []provider.DebMetadata) time.Time {
newest := time.Unix(0, 0)
for _, m := range metas {
if m.CreatedAt.After(newest) {
newest = m.CreatedAt
}
}
return newest.UTC()
}
func writeReleaseEntry(b *bytes.Buffer, hash string, size int, name string) { func writeReleaseEntry(b *bytes.Buffer, hash string, size int, name string) {
fmt.Fprintf(b, " %s %d %s\n", hash, size, name) fmt.Fprintf(b, " %s %d %s\n", hash, size, name)
} }
@@ -0,0 +1,172 @@
package deb
import (
"bytes"
"strconv"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
// debFixture returns a fixed set of rows with persisted created_at values, in
// the total order ListDebMetadataEntries produces (name, version, arch,
// file_path), so the generators are exercised on a stable input.
func debFixture() []provider.DebMetadata {
t1 := time.Date(2026, 3, 1, 8, 30, 0, 0, time.UTC)
t2 := time.Date(2026, 4, 15, 12, 0, 0, 0, time.UTC) // newest
return []provider.DebMetadata{
{
RepoName: "r", FilePath: "pool/aaa_1.0_amd64.deb", ContentHash: "sha256:aa",
Name: "aaa", Version: "1.0", Architecture: "amd64",
Control: "Package: aaa\nVersion: 1.0\nArchitecture: amd64",
Size: 100, MD5: "d41d8cd98f00b204e9800998ecf8427e", SHA256: "aa", CreatedAt: t1,
},
{
RepoName: "r", FilePath: "pool/bbb_2.0_arm64.deb", ContentHash: "sha256:bb",
Name: "bbb", Version: "2.0", Architecture: "arm64",
Control: "Package: bbb\nVersion: 2.0\nArchitecture: arm64",
Size: 200, MD5: "0cc175b9c0f1b6a831c399e269772661", SHA256: "bb", CreatedAt: t2,
},
}
}
// TestDebGeneratorsDeterministic asserts the served bytes are a pure function of
// DB state: Packages, Packages.gz and Release are byte-identical across two
// generations separated by wall-clock time. Fails against the old
// time.Now()-stamped Release Date:.
func TestDebGeneratorsDeterministic(t *testing.T) {
metas := debFixture()
pkgs1 := generatePackages(metas)
rel1 := generateRelease(metas)
gz1 := gzipBytes(pkgs1)
time.Sleep(10 * time.Millisecond)
pkgs2 := generatePackages(metas)
rel2 := generateRelease(metas)
gz2 := gzipBytes(pkgs2)
if !bytes.Equal(pkgs1, pkgs2) {
t.Error("Packages differs across generations")
}
if !bytes.Equal(gz1, gz2) {
t.Error("Packages.gz differs across generations")
}
if !bytes.Equal(rel1, rel2) {
t.Errorf("Release differs across generations:\n--- first ---\n%s\n--- second ---\n%s", rel1, rel2)
}
}
// TestDebReleaseDateUsesPersistedCreatedAt pins the Release Date: to the newest
// persisted created_at (RFC1123Z, UTC), not wall clock. Fails against the old
// time.Now() code.
func TestDebReleaseDateUsesPersistedCreatedAt(t *testing.T) {
metas := debFixture()
want := time.Date(2026, 4, 15, 12, 0, 0, 0, time.UTC).Format(time.RFC1123Z)
rel := string(generateRelease(metas))
var got string
for _, line := range strings.Split(rel, "\n") {
if strings.HasPrefix(line, "Date:") {
got = strings.TrimSpace(strings.TrimPrefix(line, "Date:"))
break
}
}
if got != want {
t.Errorf("Release Date: = %q, want %q (newest created_at)", got, want)
}
}
// TestDebReleaseDateEmptyRepoIsEpoch guards the fallback: an empty repo yields a
// deterministic epoch Date: rather than wall clock.
func TestDebReleaseDateEmptyRepoIsEpoch(t *testing.T) {
want := time.Unix(0, 0).UTC().Format(time.RFC1123Z)
rel := string(generateRelease(nil))
if !strings.Contains(rel, "Date: "+want+"\n") {
t.Errorf("empty-repo Release missing epoch Date: %q\n%s", want, rel)
}
}
// TestDebReleaseChecksumsMatchServedBytes is the exact apt invariant: the
// sha256/size (and md5/size) advertised for Packages and Packages.gz in Release
// equal the sha256/size of the actual bytes ServeLocalIndex serves. apt rejects
// any mismatch.
func TestDebReleaseChecksumsMatchServedBytes(t *testing.T) {
metas := debFixture()
packages := generatePackages(metas)
packagesGz := gzipBytes(packages)
rel := string(generateRelease(metas))
wantSHA := map[string]struct {
hash string
size int
}{
"Packages": {sha256Hex(packages), len(packages)},
"Packages.gz": {sha256Hex(packagesGz), len(packagesGz)},
}
wantMD5 := map[string]struct {
hash string
size int
}{
"Packages": {md5Hex(packages), len(packages)},
"Packages.gz": {md5Hex(packagesGz), len(packagesGz)},
}
sha := parseReleaseSection(rel, "SHA256:")
md5s := parseReleaseSection(rel, "MD5Sum:")
for name, w := range wantSHA {
got, ok := sha[name]
if !ok {
t.Fatalf("Release SHA256 section missing %q", name)
}
if got.hash != w.hash || got.size != w.size {
t.Errorf("Release SHA256 %s = (%s, %d), served bytes are (%s, %d)", name, got.hash, got.size, w.hash, w.size)
}
}
for name, w := range wantMD5 {
got, ok := md5s[name]
if !ok {
t.Fatalf("Release MD5Sum section missing %q", name)
}
if got.hash != w.hash || got.size != w.size {
t.Errorf("Release MD5Sum %s = (%s, %d), served bytes are (%s, %d)", name, got.hash, got.size, w.hash, w.size)
}
}
}
type releaseEntry struct {
hash string
size int
}
// parseReleaseSection reads the indented " <hash> <size> <name>" lines that
// follow a "SHA256:" / "MD5Sum:" header until the next non-indented line.
func parseReleaseSection(release, header string) map[string]releaseEntry {
out := map[string]releaseEntry{}
lines := strings.Split(release, "\n")
in := false
for _, line := range lines {
if line == header {
in = true
continue
}
if !in {
continue
}
if !strings.HasPrefix(line, " ") {
break
}
fields := strings.Fields(line)
if len(fields) != 3 {
continue
}
size, _ := strconv.Atoi(fields[1])
out[fields[2]] = releaseEntry{hash: fields[0], size: size}
}
return out
}
+52
View File
@@ -5,6 +5,7 @@ import (
"fmt" "fmt"
"io" "io"
"net/http" "net/http"
"time"
"git.unkin.net/unkin/artifactapi/pkg/models" "git.unkin.net/unkin/artifactapi/pkg/models"
) )
@@ -113,6 +114,54 @@ type DebMetadata struct {
Size int64 Size int64
MD5 string MD5 string
SHA256 string SHA256 string
// CreatedAt is the persisted insert time; the Release Date: is derived from
// the newest value so the index is byte-identical across replicas and
// regenerations (issue #117) rather than stamped from wall clock.
CreatedAt time.Time
}
// AlpineMetadataStore / AlpineMetadataDeleter / AlpineMetadataReader are the
// Alpine-specific persistence surfaces. They are kept separate from the shared
// RPM/Deb metadata interfaces so the apk provider can type-assert the generic
// MetadataStore/MetadataDeleter/FileStore it is handed without widening (and
// thus perturbing the test doubles of) the rpm and deb providers. *database.DB
// satisfies all three.
type AlpineMetadataStore interface {
InsertAlpineMetadata(ctx context.Context, meta *AlpineMetadata) error
}
type AlpineMetadataDeleter interface {
DeleteAlpineMetadata(ctx context.Context, repoName, filePath string) error
}
type AlpineMetadataReader interface {
ListAlpineMetadataEntries(ctx context.Context, repoName string) ([]AlpineMetadata, error)
}
// AlpineMetadata is the derived per-package metadata for an Alpine .apk, holding
// the fields an APKINDEX record carries plus the apk pull checksum (Q1…, the
// sha1 of the control gzip stream) and the download/installed sizes.
type AlpineMetadata struct {
RepoName string
FilePath string
ContentHash string
Checksum string // C: "Q1" + base64(sha1(control gzip stream))
Name string // P:
Version string // V:
Arch string // A:
DownloadSize int64 // S: on-disk .apk size
InstalledSize int64 // I: unpacked size from .PKGINFO
Description string // T:
URL string // U:
License string // L:
Origin string // o:
Maintainer string // m:
BuildTime int64 // t:
Commit string // c:
ProviderPriority string // k:
Depends []string // D:
Provides []string // p:
InstallIf []string // i:
} }
type RPMMetadata struct { type RPMMetadata struct {
@@ -141,6 +190,9 @@ type RPMMetadata struct {
Obsoletes []RPMDep Obsoletes []RPMDep
Files []RPMFile Files []RPMFile
Changelogs []RPMChangelog Changelogs []RPMChangelog
// CreatedAt is the persisted upload timestamp; used as a stable, replica-independent
// value for the repodata <time>/<revision> fields so generated indexes are deterministic.
CreatedAt time.Time
} }
type RPMDep struct { type RPMDep struct {
+31 -4
View File
@@ -275,7 +275,7 @@ func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader pr
filelistsHash := sha256Hex(filelists) filelistsHash := sha256Hex(filelists)
otherHash := sha256Hex(other) otherHash := sha256Hex(other)
repomd := generateRepomd(primaryHash, len(primary), filelistsHash, len(filelists), otherHash, len(other)) repomd := generateRepomd(repomdRevision(metas), primaryHash, len(primary), filelistsHash, len(filelists), otherHash, len(other))
w.Header().Set("Content-Type", "application/xml") w.Header().Set("Content-Type", "application/xml")
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
@@ -315,8 +315,32 @@ func (p *Provider) serveOther(w http.ResponseWriter, r *http.Request, reader pro
w.Write(generateOtherXMLGZ(metas)) w.Write(generateOtherXMLGZ(metas))
} }
func generateRepomd(primaryHash string, primarySize int, filelistsHash string, filelistsSize int, otherHash string, otherSize int) []byte { // stableUnix maps a persisted timestamp to a fixed integer for repodata's
ts := fmt.Sprintf("%d", time.Now().Unix()) // informational <time>/<timestamp> fields. Zero times (unset) collapse to 0 so
// output stays byte-identical across replicas and requests. dnf does not
// validate these values.
func stableUnix(t time.Time) int64 {
if t.IsZero() {
return 0
}
return t.Unix()
}
// repomdRevision derives repomd.xml's <revision>/<timestamp> from persisted
// state: the newest package upload time in the repo. It changes only when the
// repo's package set does, and is identical on every replica reading the same
// rows, so repomd.xml is byte-stable.
func repomdRevision(metas []provider.RPMMetadata) string {
var max int64
for _, m := range metas {
if u := stableUnix(m.CreatedAt); u > max {
max = u
}
}
return fmt.Sprintf("%d", max)
}
func generateRepomd(ts string, primaryHash string, primarySize int, filelistsHash string, filelistsSize int, otherHash string, otherSize int) []byte {
var b bytes.Buffer var b bytes.Buffer
b.WriteString(xml.Header) b.WriteString(xml.Header)
b.WriteString(`<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">` + "\n") b.WriteString(`<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">` + "\n")
@@ -359,7 +383,7 @@ func generatePrimaryXMLGZ(metas []provider.RPMMetadata) []byte {
if m.URL != "" { if m.URL != "" {
fmt.Fprintf(&xmlBuf, " <url>%s</url>\n", xmlEscape(m.URL)) fmt.Fprintf(&xmlBuf, " <url>%s</url>\n", xmlEscape(m.URL))
} }
fmt.Fprintf(&xmlBuf, " <time file=\"%d\" build=\"0\"/>\n", time.Now().Unix()) fmt.Fprintf(&xmlBuf, " <time file=\"%d\" build=\"0\"/>\n", stableUnix(m.CreatedAt))
fmt.Fprintf(&xmlBuf, " <size package=\"%d\" installed=\"%d\" archive=\"0\"/>\n", m.RPMSize, m.InstalledSize) fmt.Fprintf(&xmlBuf, " <size package=\"%d\" installed=\"%d\" archive=\"0\"/>\n", m.RPMSize, m.InstalledSize)
fmt.Fprintf(&xmlBuf, " <location href=\"%s\"/>\n", xmlEscape(m.FilePath)) fmt.Fprintf(&xmlBuf, " <location href=\"%s\"/>\n", xmlEscape(m.FilePath))
fmt.Fprintf(&xmlBuf, " <format>\n") fmt.Fprintf(&xmlBuf, " <format>\n")
@@ -484,6 +508,9 @@ func xmlEscape(s string) string {
func gzipBytes(data []byte) []byte { func gzipBytes(data []byte) []byte {
var buf bytes.Buffer var buf bytes.Buffer
gz := gzip.NewWriter(&buf) gz := gzip.NewWriter(&buf)
// Pin every header field so the compressed bytes (and their sha256) depend
// only on the payload, never on wall-clock time or the Go version's gzip defaults.
gz.Header = gzip.Header{OS: 255}
gz.Write(data) gz.Write(data)
gz.Close() gz.Close()
return buf.Bytes() return buf.Bytes()
@@ -0,0 +1,148 @@
package rpm
import (
"bytes"
"compress/gzip"
"encoding/xml"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func gunzip(t *testing.T, data []byte) string {
t.Helper()
zr, err := gzip.NewReader(bytes.NewReader(data))
if err != nil {
t.Fatalf("gzip reader: %v", err)
}
out, err := io.ReadAll(zr)
if err != nil {
t.Fatalf("gunzip: %v", err)
}
return string(out)
}
// sampleMetas returns a fixed two-package repo state whose upload timestamps are
// pinned, so any nondeterminism must come from the generators themselves.
func sampleMetas() []provider.RPMMetadata {
base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
return []provider.RPMMetadata{
{
Name: "alpha", Version: "1.0", Release: "1", Arch: "x86_64",
Summary: "a", Description: "d", ContentHash: "sha256:aaa",
FilePath: "Packages/alpha-1.0-1.x86_64.rpm", RPMSize: 10, InstalledSize: 20,
Provides: []provider.RPMDep{{Name: "alpha"}},
Requires: []provider.RPMDep{{Name: "libc", Flags: "GE", Version: "2.0"}},
CreatedAt: base,
},
{
Name: "beta", Version: "2.0", Release: "3", Arch: "noarch",
Summary: "b", Description: "d2", ContentHash: "sha256:bbb",
FilePath: "Packages/beta-2.0-3.noarch.rpm", RPMSize: 30, InstalledSize: 40,
CreatedAt: base.Add(time.Hour),
},
}
}
// TestRepodataGeneratorsDeterministic is the direct regression guard for #117:
// generating each metadata document twice from identical state must yield
// byte-identical output (hence an identical sha256). The old code embedded
// time.Now() inside primary.xml.gz, so its bytes/hash drifted every second.
func TestRepodataGeneratorsDeterministic(t *testing.T) {
metas := sampleMetas()
gens := map[string]func([]provider.RPMMetadata) []byte{
"primary": generatePrimaryXMLGZ,
"filelists": generateFilelistsXMLGZ,
"other": generateOtherXMLGZ,
}
for name, gen := range gens {
a := gen(metas)
b := gen(metas)
if sha256Hex(a) != sha256Hex(b) {
t.Errorf("%s: sha256 differs between two generations (nondeterministic): %s != %s",
name, sha256Hex(a), sha256Hex(b))
}
}
// repomd.xml itself must also be byte-stable across regenerations.
r1 := generateRepomd(repomdRevision(metas), sha256Hex(generatePrimaryXMLGZ(metas)), 1, "f", 2, "o", 3)
r2 := generateRepomd(repomdRevision(metas), sha256Hex(generatePrimaryXMLGZ(metas)), 1, "f", 2, "o", 3)
if string(r1) != string(r2) {
t.Error("repomd.xml differs between two generations")
}
}
// TestPrimaryTimeUsesPersistedCreatedAt proves the <time> element is a pure
// function of the persisted upload timestamp, not the wall clock.
func TestPrimaryTimeUsesPersistedCreatedAt(t *testing.T) {
metas := sampleMetas()
out := gunzip(t, generatePrimaryXMLGZ(metas))
if want := `<time file="1767323045" build="0"/>`; !strings.Contains(out, want) {
t.Errorf("primary.xml missing persisted <time> %q; got:\n%s", want, out)
}
// A zero (unset) CreatedAt collapses to a fixed 0, never a live clock value.
metas[0].CreatedAt = time.Time{}
out = gunzip(t, generatePrimaryXMLGZ(metas))
if !strings.Contains(out, `<time file="0" build="0"/>`) {
t.Errorf("zero CreatedAt should emit file=\"0\"; got:\n%s", out)
}
}
type repomdDoc struct {
Revision string `xml:"revision"`
Data []struct {
Type string `xml:"type,attr"`
Checksum struct {
Value string `xml:",chardata"`
} `xml:"checksum"`
Location struct {
Href string `xml:"href,attr"`
} `xml:"location"`
} `xml:"data"`
}
// TestRepomdHashMatchesServedBytes asserts the exact invariant #117 violated:
// the sha256 advertised in repomd.xml equals the sha256 of the bytes the
// content-addressed serve* handler returns for the same repo state.
func TestRepomdHashMatchesServedBytes(t *testing.T) {
p := &Provider{}
reader := fakeRPMReader{metas: sampleMetas()}
serve := func(path string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if !p.ServeLocalIndex(w, r, reader, "repo", path) {
t.Fatalf("ServeLocalIndex false for %q", path)
}
if w.Code != http.StatusOK {
t.Fatalf("%s: code %d", path, w.Code)
}
return w
}
var doc repomdDoc
if err := xml.Unmarshal(serve("repodata/repomd.xml").Body.Bytes(), &doc); err != nil {
t.Fatalf("parse repomd: %v", err)
}
if len(doc.Data) != 3 {
t.Fatalf("expected 3 <data> entries, got %d", len(doc.Data))
}
for _, d := range doc.Data {
// The advertised location is content-addressed: repodata/<sha256>-<type>.xml.gz.
body := serve("repodata/" + d.Location.Href[len("repodata/"):]).Body.Bytes()
got := sha256Hex(body)
if got != d.Checksum.Value {
t.Errorf("%s: repomd advertises %s but served bytes hash to %s (dnf would reject)",
d.Type, d.Checksum.Value, got)
}
if d.Location.Href != "repodata/"+d.Checksum.Value+"-"+d.Type+".xml.gz" {
t.Errorf("%s: location %q not addressed by its checksum %s", d.Type, d.Location.Href, d.Checksum.Value)
}
}
}
+162
View File
@@ -0,0 +1,162 @@
# Mirror-selection benchmarks
These benchmarks (`selection_bench_test.go`) isolate the **mirror load-balancing
selection overhead** — no network, no DB, no Redis. They build a zero-value
`Engine` and call `baseURLAttemptOrder` / `beginAttempt` / `endAttempt`
directly, the same way `leastconn_test.go` and `multibaseurl_test.go` do.
Goal: quantify how much latency the load-balancing strategy (`round_robin` vs
`least_conn`) adds versus a plain single-URL remote, and give a permanent
regression guard.
## Key context: selection is cache-miss-only
`baseURLAttemptOrder` is called from exactly three places — `headUpstream`,
`fetchFromUpstream`, and `checkUpstream` — all on the **upstream / cache-miss
path**. A cache hit returns `Source: "cache"` from `GetArtifact` / `store.Stat`
*before* any selection code runs. So none of the numbers below apply to the hot
cache-hit path: cache hits pay **zero** selection cost regardless of strategy.
The overhead here is paid once per upstream fetch, alongside a network round-trip
measured in milliseconds.
## How to run
```
go test -run=^$ -bench='BaseURLAttemptOrder|BeginEndAttempt' -benchmem \
-benchtime=1s -count=6 -cpu=8 ./internal/proxy/
```
## Results
Machine: AMD Ryzen 7 4700U (8 threads), linux/amd64, go1.26.5.
`-benchtime=1s -count=6`; figures below are the **median of 6 runs**.
### Sequential (single-goroutine)
| Benchmark | ns/op | B/op | allocs/op |
|----------------------------------|-------:|-----:|----------:|
| BaseURLAttemptOrder_SingleURL | ~133 | 16 | 1 |
| BaseURLAttemptOrder_RoundRobin/3 | ~462 | 120 | 4 |
| BaseURLAttemptOrder_RoundRobin/8 | ~682 | 280 | 4 |
| BaseURLAttemptOrder_LeastConn/3 | ~2690 | 474 | 22 |
| BaseURLAttemptOrder_LeastConn/8 | ~15200 | 2688 | 132 |
| BeginEndAttempt (gauge inc/dec) | ~514 | 104 | 4 |
### Parallel (`RunParallel`, GOMAXPROCS=8) — ns/op is wall-time across 8 cores
| Benchmark | ns/op | B/op | allocs/op |
|-------------------------------------------|------:|-----:|----------:|
| BaseURLAttemptOrder_RoundRobin_Parallel/3 | ~67.5 | 120 | 4 |
| BaseURLAttemptOrder_RoundRobin_Parallel/8 | ~130 | 280 | 4 |
| BaseURLAttemptOrder_LeastConn_Parallel/3 | ~292 | 474 | 22 |
| BaseURLAttemptOrder_LeastConn_Parallel/8 | ~1673 | 2688 | 132 |
| BeginEndAttempt_Parallel | ~71.5 | 104 | 4 |
## Reading the numbers
- **Single-URL is a near-no-op** (~133 ns, 1 alloc): the `len(urls) <= 1`
early return just returns the pool slice. Every non-mirrored remote takes this
path.
- **round_robin is cheap**: ~462 ns for a 3-mirror pool, ~682 ns for 8. Cost is
one atomic cursor increment plus building the rotated `[]string`. Allocs are
constant at 4 (the ordered slice + its backing string headers), size grows
with pool length.
- **least_conn is more expensive and scales super-linearly**: ~2.7 µs / 22
allocs at 3 mirrors, ~15 µs / 132 allocs at 8. The cost is the per-call
`sort.SliceStable`, whose comparator calls `inflightCounter` (a
`sync.Map.LoadOrStore` with a `remoteName\x00url` string-concat key plus a
speculative `new(atomic.Int64)`) O(n·log n) times. That is where the alloc
count and the time come from — not the sort itself. A future optimization
could snapshot each mirror's load once before sorting; out of scope for this
measurement PR.
- **beginAttempt/endAttempt** (~514 ns seq, ~72 ns parallel) is one
`LoadOrStore` + two atomic adds; it only runs for least_conn multi-mirror
remotes, once per upstream attempt.
- **Under concurrency the atomics/sync.Map do not collapse**: every parallel
variant reports *lower* ns/op than its sequential twin because work spreads
across 8 cores (RunParallel reports aggregate wall-time-per-op). No contention
cliff on the shared rrCounters cursor, the inflight `sync.Map`, or the
per-mirror `atomic.Int64` gauges.
## Verdict
At the per-request scale that matters (a cache-miss that is *already* doing a
multi-millisecond network fetch), even the worst case here — least_conn across 8
mirrors at ~15 µs — is <1% of a single upstream round-trip, and round_robin
(~0.5 µs) is negligible. The strategy adds no meaningful latency, and it adds
**exactly zero** to the cache-hit hot path because selection never runs there.
## Raw output (all 6 runs)
```
goos: linux
goarch: amd64
pkg: git.unkin.net/unkin/artifactapi/internal/proxy
cpu: AMD Ryzen 7 4700U with Radeon Graphics
BenchmarkBaseURLAttemptOrder_SingleURL-8 8635875 138.4 ns/op 16 B/op 1 allocs/op
BenchmarkBaseURLAttemptOrder_SingleURL-8 9673108 126.7 ns/op 16 B/op 1 allocs/op
BenchmarkBaseURLAttemptOrder_SingleURL-8 8001002 147.3 ns/op 16 B/op 1 allocs/op
BenchmarkBaseURLAttemptOrder_SingleURL-8 11303490 135.0 ns/op 16 B/op 1 allocs/op
BenchmarkBaseURLAttemptOrder_SingleURL-8 10125138 132.0 ns/op 16 B/op 1 allocs/op
BenchmarkBaseURLAttemptOrder_SingleURL-8 8025687 130.1 ns/op 16 B/op 1 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool3-8 2706013 453.2 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool3-8 2498718 444.4 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool3-8 2605516 471.6 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool3-8 2799928 487.6 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool3-8 2463375 418.6 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool3-8 2472265 474.3 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool8-8 1741789 689.4 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool8-8 1775467 602.1 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool8-8 1829398 688.4 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool8-8 1781149 679.3 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool8-8 1795680 599.4 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin/pool8-8 1739122 684.5 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool3-8 424184 2675 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool3-8 426796 2498 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool3-8 427116 2716 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool3-8 430540 2681 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool3-8 418156 2700 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool3-8 423009 2711 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool8-8 163642 14007 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool8-8 78501 15457 ns/op 2688 B/op 131 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool8-8 76380 14928 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool8-8 183634 16091 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool8-8 73809 15561 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn/pool8-8 74546 13962 ns/op 2688 B/op 132 allocs/op
BenchmarkBeginEndAttempt-8 2350348 519.7 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt-8 2321659 514.0 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt-8 2284635 438.0 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt-8 2287051 513.9 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt-8 2286481 520.3 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt-8 2837775 512.9 ns/op 104 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool3-8 16052568 67.78 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool3-8 17452791 66.07 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool3-8 17549858 69.25 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool3-8 18845167 64.55 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool3-8 16285608 69.81 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool3-8 17382639 67.12 ns/op 120 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool8-8 9734368 120.6 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool8-8 10154736 133.3 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool8-8 10061422 131.8 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool8-8 10212364 127.4 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool8-8 10259030 132.5 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel/pool8-8 10069576 122.4 ns/op 280 B/op 4 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool3-8 4288112 292.7 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool3-8 4009249 295.9 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool3-8 4176378 291.3 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool3-8 4104871 289.9 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool3-8 4245262 296.4 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool3-8 4079778 290.3 ns/op 474 B/op 22 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool8-8 748200 1636 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool8-8 763029 1653 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool8-8 663717 1772 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool8-8 739677 1676 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool8-8 763148 1669 ns/op 2688 B/op 132 allocs/op
BenchmarkBaseURLAttemptOrder_LeastConn_Parallel/pool8-8 610597 1684 ns/op 2688 B/op 132 allocs/op
BenchmarkBeginEndAttempt_Parallel-8 17266058 69.46 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt_Parallel-8 17151303 72.05 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt_Parallel-8 16919542 74.17 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt_Parallel-8 16948015 72.49 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt_Parallel-8 16918693 69.52 ns/op 104 B/op 4 allocs/op
BenchmarkBeginEndAttempt_Parallel-8 17376012 70.99 ns/op 104 B/op 4 allocs/op
```
+197
View File
@@ -10,7 +10,10 @@ import (
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
"sort"
"strings" "strings"
"sync"
"sync/atomic"
"time" "time"
"git.unkin.net/unkin/artifactapi/internal/cache" "git.unkin.net/unkin/artifactapi/internal/cache"
@@ -35,6 +38,15 @@ type Engine struct {
cas *storage.CAS cas *storage.CAS
circuit *CircuitBreaker circuit *CircuitBreaker
accessLog chan database.AccessLogEntry accessLog chan database.AccessLogEntry
// rrCounters holds a per-remote round-robin cursor (remoteName ->
// *atomic.Uint64) used to rotate the starting mirror across upstream base
// URLs. Distribution is per-replica and approximate, which is fine.
rrCounters sync.Map
// inflight holds a per-remote, per-upstream-URL in-flight request gauge
// (key "remoteName\x00baseURL" -> *atomic.Int64) used by the least_conn
// mirror strategy to prefer the mirror currently handling the fewest
// requests. Per-replica and approximate, which is fine.
inflight sync.Map
} }
func NewEngine(db *database.DB, c *cache.Redis, s *storage.S3) *Engine { func NewEngine(db *database.DB, c *cache.Redis, s *storage.S3) *Engine {
@@ -222,7 +234,32 @@ func (e *Engine) Head(ctx context.Context, remote models.Remote, path string, pr
return e.headUpstream(ctx, remote, path, prov) return e.headUpstream(ctx, remote, path, prov)
} }
// headUpstream issues an upstream HEAD, load-balancing across the remote's base
// URLs and failing over to the next mirror on a network error or 5xx.
func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*HeadResult, error) { func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*HeadResult, error) {
order := e.baseURLAttemptOrder(remote)
if len(order) == 0 {
return nil, &ProxyError{Status: http.StatusBadGateway, Message: "no upstream base_url configured"}
}
var lastErr error
for i, url := range order {
ctr := e.beginAttempt(remote, url)
result, err := e.headUpstreamOnce(ctx, withBaseURL(remote, url), path, prov)
endAttempt(ctr)
if err == nil {
return result, nil
}
lastErr = err
if i < len(order)-1 && shouldFailover(err) {
slog.Warn("upstream HEAD failed, failing over", "remote", remote.Name, "base_url", url, "error", err)
continue
}
return nil, err
}
return nil, lastErr
}
func (e *Engine) headUpstreamOnce(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*HeadResult, error) {
url := prov.UpstreamURL(remote, path) url := prov.UpstreamURL(remote, path)
authHeaders, err := prov.AuthHeaders(ctx, remote) authHeaders, err := prov.AuthHeaders(ctx, remote)
@@ -277,7 +314,33 @@ func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path st
return &HeadResult{ContentType: contentType, Size: resp.ContentLength, Source: "remote"}, nil return &HeadResult{ContentType: contentType, Size: resp.ContentLength, Source: "remote"}, nil
} }
// fetchFromUpstream fetches an artifact from upstream, load-balancing across the
// remote's base URLs and failing over to the next mirror on a network error or
// 5xx before returning an error.
func (e *Engine) fetchFromUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider, class Classification, ttl time.Duration, clientHeaders http.Header) (*FetchResult, error) { func (e *Engine) fetchFromUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider, class Classification, ttl time.Duration, clientHeaders http.Header) (*FetchResult, error) {
order := e.baseURLAttemptOrder(remote)
if len(order) == 0 {
return nil, &ProxyError{Status: http.StatusBadGateway, Message: "no upstream base_url configured"}
}
var lastErr error
for i, url := range order {
ctr := e.beginAttempt(remote, url)
result, err := e.fetchFromUpstreamOnce(ctx, withBaseURL(remote, url), path, prov, class, ttl, clientHeaders)
endAttempt(ctr)
if err == nil {
return result, nil
}
lastErr = err
if i < len(order)-1 && shouldFailover(err) {
slog.Warn("upstream fetch failed, failing over", "remote", remote.Name, "base_url", url, "error", err)
continue
}
return nil, err
}
return nil, lastErr
}
func (e *Engine) fetchFromUpstreamOnce(ctx context.Context, remote models.Remote, path string, prov provider.Provider, class Classification, ttl time.Duration, clientHeaders http.Header) (*FetchResult, error) {
url := prov.UpstreamURL(remote, path) url := prov.UpstreamURL(remote, path)
authHeaders, err := prov.AuthHeaders(ctx, remote) authHeaders, err := prov.AuthHeaders(ctx, remote)
@@ -454,7 +517,33 @@ func (e *Engine) serveFromStore(ctx context.Context, remote models.Remote, path
}, nil }, nil
} }
// checkUpstream issues a conditional upstream HEAD (If-None-Match), load
// balancing across the remote's base URLs and failing over to the next mirror on
// a network error or 5xx.
func (e *Engine) checkUpstream(ctx context.Context, remote models.Remote, path, etag string, prov provider.Provider) (bool, error) { func (e *Engine) checkUpstream(ctx context.Context, remote models.Remote, path, etag string, prov provider.Provider) (bool, error) {
order := e.baseURLAttemptOrder(remote)
if len(order) == 0 {
return false, &ProxyError{Status: http.StatusBadGateway, Message: "no upstream base_url configured"}
}
var lastErr error
for i, url := range order {
ctr := e.beginAttempt(remote, url)
notModified, err := e.checkUpstreamOnce(ctx, withBaseURL(remote, url), path, etag, prov)
endAttempt(ctr)
if err == nil {
return notModified, nil
}
lastErr = err
if i < len(order)-1 && shouldFailover(err) {
slog.Warn("upstream revalidation failed, failing over", "remote", remote.Name, "base_url", url, "error", err)
continue
}
return false, err
}
return false, lastErr
}
func (e *Engine) checkUpstreamOnce(ctx context.Context, remote models.Remote, path, etag string, prov provider.Provider) (bool, error) {
url := prov.UpstreamURL(remote, path) url := prov.UpstreamURL(remote, path)
req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil) req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
@@ -649,3 +738,111 @@ func isNetworkError(err error) bool {
var ue *UpstreamError var ue *UpstreamError
return errors.As(err, &ue) return errors.As(err, &ue)
} }
// baseURLAttemptOrder returns the ordered upstream base URLs to try for a single
// request, drawn from the remote's pool ([base_url] + mirrorlist). A multi-mirror
// remote starts at a strategy-chosen position and advances linearly for
// failover; a remote with no mirrorlist yields exactly [base_url], preserving the
// original single-attempt behavior. The default (round_robin) rotates the
// starting mirror; least_conn starts with the mirror handling the fewest
// in-flight requests. Failover order after the first pick is unchanged.
func (e *Engine) baseURLAttemptOrder(remote models.Remote) []string {
urls := remote.UpstreamPool()
if len(urls) <= 1 {
return urls
}
// Rotate by the round-robin cursor first so equal-load mirrors still spread
// evenly; least_conn then stable-sorts this rotation by in-flight count.
v, _ := e.rrCounters.LoadOrStore(remote.Name, new(atomic.Uint64))
start := int(v.(*atomic.Uint64).Add(1) - 1)
ordered := make([]string, len(urls))
for i := range urls {
ordered[i] = urls[(start+i)%len(urls)]
}
if remote.MirrorStrategy == models.MirrorStrategyLeastConn {
// Snapshot each mirror's in-flight count once, then sort the snapshot.
// Reading the gauge inside the comparator would repeat an allocating
// sync.Map lookup on every comparison (O(n log n) lookups); this is O(n).
snap := make([]inflightSnapshot, len(ordered))
for i, url := range ordered {
snap[i] = inflightSnapshot{url: url, count: e.inflightCount(remote.Name, url)}
}
sort.SliceStable(snap, func(a, b int) bool {
return snap[a].count < snap[b].count
})
for i := range snap {
ordered[i] = snap[i].url
}
}
return ordered
}
// inflightSnapshot pairs a mirror URL with its sampled in-flight count so the
// least_conn sort compares plain ints instead of re-reading the gauge.
type inflightSnapshot struct {
url string
count int64
}
// inflightCount reads the in-flight request gauge for a given (remote, upstream
// URL) without creating it, returning 0 when the counter is absent. This keeps
// the selection read path allocation-free (plain Load, no LoadOrStore).
func (e *Engine) inflightCount(remoteName, url string) int64 {
v, ok := e.inflight.Load(remoteName + "\x00" + url)
if !ok {
return 0
}
return v.(*atomic.Int64).Load()
}
// inflightCounter returns the shared in-flight request gauge for a given
// (remote, upstream URL), creating it on first use.
func (e *Engine) inflightCounter(remoteName, url string) *atomic.Int64 {
v, _ := e.inflight.LoadOrStore(remoteName+"\x00"+url, new(atomic.Int64))
return v.(*atomic.Int64)
}
// beginAttempt increments the in-flight gauge for a least_conn multi-mirror
// remote before an upstream call and returns the counter to release; it is a
// no-op (returns nil) for round-robin remotes and single-URL pools.
func (e *Engine) beginAttempt(remote models.Remote, url string) *atomic.Int64 {
if remote.MirrorStrategy != models.MirrorStrategyLeastConn {
return nil
}
if len(remote.UpstreamPool()) <= 1 {
return nil
}
ctr := e.inflightCounter(remote.Name, url)
ctr.Add(1)
return ctr
}
// endAttempt decrements a gauge returned by beginAttempt, tolerating nil.
func endAttempt(ctr *atomic.Int64) {
if ctr != nil {
ctr.Add(-1)
}
}
// withBaseURL narrows a remote's active BaseURL to a single selected mirror so
// providers (UpstreamURL/AuthHeaders/RewriteResponse) operate on exactly that
// upstream for this attempt.
func withBaseURL(remote models.Remote, url string) models.Remote {
remote.BaseURL = url
remote.Mirrorlist = nil
return remote
}
// shouldFailover reports whether an upstream attempt error is worth retrying
// against the next mirror: network errors/timeouts and upstream 5xx responses.
// Definitive statuses (404/403/401/...) are returned to the caller unchanged.
func shouldFailover(err error) bool {
if isNetworkError(err) {
return true
}
var pe *ProxyError
if errors.As(err, &pe) {
return pe.Status >= 500
}
return false
}
+148
View File
@@ -0,0 +1,148 @@
package proxy
import (
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// baseURLAttemptOrder and the in-flight gauge only touch the engine's sync.Map
// fields, so these tests run against a zero-value Engine without a DB/S3/redis
// stack and can drive the gauge deterministically.
// TestLeastConnPicksLeastLoaded pre-loads one mirror's in-flight gauge and
// asserts a least_conn remote starts its attempt order with the idle mirror.
func TestLeastConnPicksLeastLoaded(t *testing.T) {
e := &Engine{}
r := models.Remote{
Name: "lc",
BaseURL: "https://a.example",
Mirrorlist: []string{"https://b.example"},
MirrorStrategy: models.MirrorStrategyLeastConn,
}
// Make A appear busy: least_conn must prefer B regardless of RR rotation.
e.inflightCounter(r.Name, "https://a.example").Add(3)
for i := 0; i < 5; i++ {
order := e.baseURLAttemptOrder(r)
if len(order) != 2 {
t.Fatalf("attempt %d: order len = %d, want 2", i, len(order))
}
if order[0] != "https://b.example" {
t.Fatalf("attempt %d: least_conn started with %q, want idle mirror https://b.example", i, order[0])
}
}
// Once B is the busier mirror, the starting pick flips to A.
e.inflightCounter(r.Name, "https://b.example").Add(10)
if order := e.baseURLAttemptOrder(r); order[0] != "https://a.example" {
t.Fatalf("after loading B, least_conn started with %q, want https://a.example", order[0])
}
}
// TestLeastConnStableTieBreak asserts that when every mirror carries equal
// in-flight load, least_conn falls back to the round-robin rotation: the
// snapshot sort is stable, so tied mirrors keep the RR-rotated order and the
// starting pick advances across the whole pool on successive calls.
func TestLeastConnStableTieBreak(t *testing.T) {
e := &Engine{}
r := models.Remote{
Name: "lc-tie",
BaseURL: "https://a.example",
Mirrorlist: []string{"https://b.example", "https://c.example"},
MirrorStrategy: models.MirrorStrategyLeastConn,
}
pool := r.UpstreamPool()
// Equal (zero) load on every mirror: order must equal the RR rotation.
starts := map[string]int{}
for i := 0; i < len(pool); i++ {
order := e.baseURLAttemptOrder(r)
if len(order) != len(pool) {
t.Fatalf("attempt %d: order len = %d, want %d", i, len(order), len(pool))
}
// A stable sort of an all-tied slice is a pure RR rotation: for the
// call whose cursor selects start s, order must be pool rotated by s.
start := indexOf(pool, order[0])
for j := range order {
if want := pool[(start+j)%len(pool)]; order[j] != want {
t.Fatalf("attempt %d: order[%d] = %q, want RR-rotated %q", i, j, order[j], want)
}
}
starts[order[0]]++
}
if len(starts) != len(pool) {
t.Fatalf("tied least_conn did not rotate across the whole pool: %v", starts)
}
}
func indexOf(s []string, v string) int {
for i := range s {
if s[i] == v {
return i
}
}
return -1
}
// TestRoundRobinDefaultUnchanged asserts an unset strategy still rotates the
// starting mirror across the pool and ignores the in-flight gauge.
func TestRoundRobinDefaultUnchanged(t *testing.T) {
e := &Engine{}
r := models.Remote{
Name: "rr",
BaseURL: "https://a.example",
Mirrorlist: []string{"https://b.example"},
}
// Even with A heavily loaded, round-robin must still rotate (not avoid A).
e.inflightCounter(r.Name, "https://a.example").Add(100)
starts := map[string]int{}
for i := 0; i < 4; i++ {
starts[e.baseURLAttemptOrder(r)[0]]++
}
if starts["https://a.example"] == 0 || starts["https://b.example"] == 0 {
t.Fatalf("round-robin did not rotate starting mirror: %v", starts)
}
}
// TestLeastConnSingleURLNoOp asserts a single-URL pool yields exactly [base_url]
// and beginAttempt is a no-op there and for round-robin remotes.
func TestLeastConnSingleURLNoOp(t *testing.T) {
e := &Engine{}
solo := models.Remote{Name: "solo", BaseURL: "https://a.example", MirrorStrategy: models.MirrorStrategyLeastConn}
if order := e.baseURLAttemptOrder(solo); len(order) != 1 || order[0] != "https://a.example" {
t.Fatalf("single-url order = %v, want [base_url]", order)
}
if ctr := e.beginAttempt(solo, "https://a.example"); ctr != nil {
t.Fatal("beginAttempt on single-url pool should be a no-op (nil)")
}
rr := models.Remote{Name: "rr2", BaseURL: "https://a.example", Mirrorlist: []string{"https://b.example"}}
if ctr := e.beginAttempt(rr, "https://a.example"); ctr != nil {
t.Fatal("beginAttempt on round-robin remote should be a no-op (nil)")
}
}
// TestBeginEndAttemptGauge asserts the gauge increments on begin and returns to
// zero after endAttempt, so it tracks live in-flight requests.
func TestBeginEndAttemptGauge(t *testing.T) {
e := &Engine{}
r := models.Remote{
Name: "g",
BaseURL: "https://a.example",
Mirrorlist: []string{"https://b.example"},
MirrorStrategy: models.MirrorStrategyLeastConn,
}
c1 := e.beginAttempt(r, "https://a.example")
c2 := e.beginAttempt(r, "https://a.example")
if got := e.inflightCounter(r.Name, "https://a.example").Load(); got != 2 {
t.Fatalf("gauge after two begins = %d, want 2", got)
}
endAttempt(c1)
endAttempt(c2)
if got := e.inflightCounter(r.Name, "https://a.example").Load(); got != 0 {
t.Fatalf("gauge after matching ends = %d, want 0", got)
}
endAttempt(nil) // tolerated
}
+188
View File
@@ -0,0 +1,188 @@
package proxy
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// TestFetchMultiBaseURLRoundRobin drives distinct artifact paths through a
// remote configured with two upstreams and asserts both receive traffic.
func TestFetchMultiBaseURLRoundRobin(t *testing.T) {
requireStack(t)
ctx := context.Background()
var hitsA, hitsB atomic.Int64
upA := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hitsA.Add(1)
w.Write([]byte("A"))
}))
defer upA.Close()
upB := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hitsB.Add(1)
w.Write([]byte("B"))
}))
defer upB.Close()
r := seed(t, models.Remote{
Name: "eng-rr",
PackageType: models.PackageGeneric,
RepoType: models.RepoTypeRemote,
BaseURL: upA.URL,
Mirrorlist: []string{upB.URL},
StaleOnError: true,
})
p := prov(t, models.PackageGeneric)
const n = 10
for i := 0; i < n; i++ {
res, err := testEngine.Fetch(ctx, r, fmt.Sprintf("rr-%d.bin", i), p)
if err != nil {
t.Fatalf("fetch %d: %v", i, err)
}
res.Reader.Close()
}
if hitsA.Load() == 0 || hitsB.Load() == 0 {
t.Fatalf("round-robin did not spread across both upstreams: A=%d B=%d", hitsA.Load(), hitsB.Load())
}
if total := hitsA.Load() + hitsB.Load(); total != n {
t.Fatalf("expected %d upstream hits total, got %d (A=%d B=%d)", n, total, hitsA.Load(), hitsB.Load())
}
}
// TestFetchMultiBaseURLFailover asserts that a dead/erroring primary mirror
// transparently fails over to a healthy secondary, for both a 5xx primary and a
// network-unreachable primary.
func TestFetchMultiBaseURLFailover(t *testing.T) {
requireStack(t)
ctx := context.Background()
var hitsB atomic.Int64
upB := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hitsB.Add(1)
w.Write([]byte("served-by-B"))
}))
defer upB.Close()
up500 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer up500.Close()
p := prov(t, models.PackageGeneric)
// Primary returns 5xx: every request must still succeed via the secondary.
r5xx := seed(t, models.Remote{
Name: "eng-failover-5xx",
PackageType: models.PackageGeneric,
RepoType: models.RepoTypeRemote,
BaseURL: up500.URL,
Mirrorlist: []string{upB.URL},
})
for i := 0; i < 6; i++ {
res, err := testEngine.Fetch(ctx, r5xx, fmt.Sprintf("fo5-%d.bin", i), p)
if err != nil {
t.Fatalf("5xx failover fetch %d: %v", i, err)
}
if got := readAll(t, res); got != "served-by-B" {
t.Fatalf("5xx failover fetch %d body=%q, want served-by-B", i, got)
}
}
// Primary is network-unreachable: failover must still reach the secondary.
rNet := seed(t, models.Remote{
Name: "eng-failover-net",
PackageType: models.PackageGeneric,
RepoType: models.RepoTypeRemote,
BaseURL: "http://127.0.0.1:1",
Mirrorlist: []string{upB.URL},
})
res, err := testEngine.Fetch(ctx, rNet, "fonet.bin", p)
if err != nil {
t.Fatalf("network failover fetch: %v", err)
}
if got := readAll(t, res); got != "served-by-B" {
t.Fatalf("network failover body=%q, want served-by-B", got)
}
if hitsB.Load() == 0 {
t.Fatal("secondary upstream never served during failover")
}
}
// TestFetchDefinitiveStatusNoFailover asserts a definitive 404 from the first
// mirror is returned as-is (not failed over): a missing artifact is not a mirror
// outage. The remote is fresh so its round-robin cursor starts at index 0.
func TestFetchDefinitiveStatusNoFailover(t *testing.T) {
requireStack(t)
ctx := context.Background()
var hitsB atomic.Int64
up404 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.NotFound(w, r)
}))
defer up404.Close()
upB := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hitsB.Add(1)
w.Write([]byte("B"))
}))
defer upB.Close()
r := seed(t, models.Remote{
Name: "eng-no-failover-404",
PackageType: models.PackageGeneric,
RepoType: models.RepoTypeRemote,
BaseURL: up404.URL,
Mirrorlist: []string{upB.URL},
})
_, err := testEngine.Fetch(ctx, r, "missing.bin", prov(t, models.PackageGeneric))
var pe *ProxyError
if err == nil || !asProxyError(err, &pe) || pe.Status != http.StatusNotFound {
t.Fatalf("expected 404 ProxyError without failover, got %v", err)
}
if hitsB.Load() != 0 {
t.Fatalf("404 from primary must not fail over, but secondary was hit %d times", hitsB.Load())
}
}
// TestFetchSingleBaseURLUnchanged asserts a single-URL remote behaves exactly as
// before: one healthy URL succeeds, and one dead URL errors with no failover.
func TestFetchSingleBaseURLUnchanged(t *testing.T) {
requireStack(t)
ctx := context.Background()
upB := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("solo"))
}))
defer upB.Close()
p := prov(t, models.PackageGeneric)
rOK := seed(t, models.Remote{
Name: "eng-solo",
PackageType: models.PackageGeneric,
RepoType: models.RepoTypeRemote,
BaseURL: upB.URL,
})
res, err := testEngine.Fetch(ctx, rOK, "solo.bin", p)
if err != nil {
t.Fatalf("single-url fetch: %v", err)
}
if got := readAll(t, res); got != "solo" {
t.Fatalf("single-url body=%q, want solo", got)
}
rDead := seed(t, models.Remote{
Name: "eng-solo-dead",
PackageType: models.PackageGeneric,
RepoType: models.RepoTypeRemote,
BaseURL: "http://127.0.0.1:1",
})
if _, err := testEngine.Fetch(ctx, rDead, "x.bin", p); err == nil {
t.Fatal("single dead upstream should error, not succeed")
}
}
+156
View File
@@ -0,0 +1,156 @@
package proxy
import (
"fmt"
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// These benchmarks isolate the mirror-selection overhead only: they construct a
// zero-value Engine (no DB/S3/redis) and call baseURLAttemptOrder /
// beginAttempt / endAttempt directly, mirroring leastconn_test.go and
// multibaseurl_test.go. This quantifies how much latency the load-balancing
// strategy (round_robin vs least_conn) adds versus a single-URL remote. Note
// that in the live proxy this selection runs only on the cache-miss/upstream
// path; a cache hit never calls it.
// mirrorPool builds a remote with n upstreams (base_url + n-1 mirrorlist
// entries) under the given strategy.
func mirrorPool(name, strategy string, n int) models.Remote {
r := models.Remote{
Name: name,
BaseURL: "https://mirror0.example/repo",
MirrorStrategy: strategy,
}
for i := 1; i < n; i++ {
r.Mirrorlist = append(r.Mirrorlist, fmt.Sprintf("https://mirror%d.example/repo", i))
}
return r
}
// skewInflight sets an ascending in-flight load across the pool so least_conn's
// stable sort has real work to do (mirror0 busiest, last mirror idle).
func skewInflight(e *Engine, r models.Remote) {
pool := r.UpstreamPool()
for i, u := range pool {
e.inflightCounter(r.Name, u).Add(int64(len(pool) - i))
}
}
// BenchmarkBaseURLAttemptOrder_SingleURL measures the early-return no-op path
// (pool of 1): the branch that preserves original single-attempt behavior and
// must add effectively zero overhead. This is the same code the cache-miss path
// takes for every non-mirrored remote.
func BenchmarkBaseURLAttemptOrder_SingleURL(b *testing.B) {
e := &Engine{}
r := models.Remote{Name: "solo", BaseURL: "https://mirror0.example/repo"}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = e.baseURLAttemptOrder(r)
}
}
// BenchmarkBaseURLAttemptOrder_RoundRobin measures the default strategy: rotate
// the starting mirror by an atomic cursor and materialize the ordered slice. No
// in-flight sort.
func BenchmarkBaseURLAttemptOrder_RoundRobin(b *testing.B) {
for _, n := range []int{3, 8} {
b.Run(fmt.Sprintf("pool%d", n), func(b *testing.B) {
e := &Engine{}
r := mirrorPool("rr", models.MirrorStrategyRoundRobin, n)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = e.baseURLAttemptOrder(r)
}
})
}
}
// BenchmarkBaseURLAttemptOrder_LeastConn measures the least_conn strategy: RR
// rotation plus a stable sort of the pool by the atomic in-flight gauges. Skew
// is preloaded so the sort compares distinct loads.
func BenchmarkBaseURLAttemptOrder_LeastConn(b *testing.B) {
for _, n := range []int{3, 8} {
b.Run(fmt.Sprintf("pool%d", n), func(b *testing.B) {
e := &Engine{}
r := mirrorPool("lc", models.MirrorStrategyLeastConn, n)
skewInflight(e, r)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
_ = e.baseURLAttemptOrder(r)
}
})
}
}
// BenchmarkBeginEndAttempt measures the gauge inc/dec pair that brackets each
// least_conn upstream attempt (LoadOrStore + atomic add, then atomic add back).
func BenchmarkBeginEndAttempt(b *testing.B) {
e := &Engine{}
r := mirrorPool("g", models.MirrorStrategyLeastConn, 3)
url := r.UpstreamPool()[0]
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
ctr := e.beginAttempt(r, url)
endAttempt(ctr)
}
}
// BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel surfaces atomic-cursor
// contention on the shared rrCounters entry under concurrent selection.
func BenchmarkBaseURLAttemptOrder_RoundRobin_Parallel(b *testing.B) {
for _, n := range []int{3, 8} {
b.Run(fmt.Sprintf("pool%d", n), func(b *testing.B) {
e := &Engine{}
r := mirrorPool("rrp", models.MirrorStrategyRoundRobin, n)
b.ReportAllocs()
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
_ = e.baseURLAttemptOrder(r)
}
})
})
}
}
// BenchmarkBaseURLAttemptOrder_LeastConn_Parallel surfaces sync.Map read
// contention on the in-flight gauges plus the per-call sort under concurrency.
func BenchmarkBaseURLAttemptOrder_LeastConn_Parallel(b *testing.B) {
for _, n := range []int{3, 8} {
b.Run(fmt.Sprintf("pool%d", n), func(b *testing.B) {
e := &Engine{}
r := mirrorPool("lcp", models.MirrorStrategyLeastConn, n)
skewInflight(e, r)
b.ReportAllocs()
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
_ = e.baseURLAttemptOrder(r)
}
})
})
}
}
// BenchmarkBeginEndAttempt_Parallel exercises the gauge inc/dec pair under
// concurrency: all goroutines hammer the same atomic.Int64, the realistic
// hot-mirror case, to surface counter contention.
func BenchmarkBeginEndAttempt_Parallel(b *testing.B) {
e := &Engine{}
r := mirrorPool("gp", models.MirrorStrategyLeastConn, 3)
url := r.UpstreamPool()[0]
b.ReportAllocs()
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
ctr := e.beginAttempt(r, url)
endAttempt(ctr)
}
})
}
+13 -2
View File
@@ -20,7 +20,7 @@ import (
"git.unkin.net/unkin/artifactapi/internal/database" "git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/gc" "git.unkin.net/unkin/artifactapi/internal/gc"
"git.unkin.net/unkin/artifactapi/internal/githubauth" "git.unkin.net/unkin/artifactapi/internal/githubauth"
_ "git.unkin.net/unkin/artifactapi/internal/provider/alpine" "git.unkin.net/unkin/artifactapi/internal/provider/alpine"
"git.unkin.net/unkin/artifactapi/internal/provider/deb" "git.unkin.net/unkin/artifactapi/internal/provider/deb"
_ "git.unkin.net/unkin/artifactapi/internal/provider/docker" _ "git.unkin.net/unkin/artifactapi/internal/provider/docker"
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic" _ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
@@ -52,6 +52,7 @@ type Server struct {
gc *gc.Collector gc *gc.Collector
syncer *rpm.Syncer syncer *rpm.Syncer
debSyncer *deb.Syncer debSyncer *deb.Syncer
alpineSyncer *alpine.Syncer
} }
func New(cfg *config.Config, version string) (*Server, error) { func New(cfg *config.Config, version string) (*Server, error) {
@@ -105,6 +106,12 @@ func New(cfg *config.Config, version string) (*Server, error) {
Workers: cfg.GitHubSyncWorkers, Workers: cfg.GitHubSyncWorkers,
PollInterval: time.Duration(cfg.GitHubSyncPollInterval) * time.Second, PollInterval: time.Duration(cfg.GitHubSyncPollInterval) * time.Second,
}) })
alpineSyncer := alpine.NewSyncer(db, alpine.SyncConfig{
RatePerSec: cfg.GitHubSyncRatePerSec,
Burst: cfg.GitHubSyncBurst,
Workers: cfg.GitHubSyncWorkers,
PollInterval: time.Duration(cfg.GitHubSyncPollInterval) * time.Second,
})
// The terraform registry signs with a GPG key. A configured file wins (BYO // The terraform registry signs with a GPG key. A configured file wins (BYO
// key); otherwise artifactapi generates one on first start and persists it in // key); otherwise artifactapi generates one on first start and persists it in
@@ -138,6 +145,7 @@ func New(cfg *config.Config, version string) (*Server, error) {
gc: collector, gc: collector,
syncer: syncer, syncer: syncer,
debSyncer: debSyncer, debSyncer: debSyncer,
alpineSyncer: alpineSyncer,
} }
s.router = s.routes() s.router = s.routes()
@@ -167,9 +175,10 @@ func (s *Server) routes() chi.Router {
r.Mount("/api/v1", proxyHandler.Routes()) r.Mount("/api/v1", proxyHandler.Routes())
r.Mount("/v2", proxyHandler.DockerV2Routes()) r.Mount("/v2", proxyHandler.DockerV2Routes())
remotesHandler := v2.NewRemotesHandler(s.db, map[models.PackageType]v2.Primer{ remotesHandler := v2.NewRemotesHandler(s.db, s.cache, map[models.PackageType]v2.Primer{
models.PackageGitHubRPM: s.syncer, models.PackageGitHubRPM: s.syncer,
models.PackageGitHubDeb: s.debSyncer, models.PackageGitHubDeb: s.debSyncer,
models.PackageGitHubAlpine: s.alpineSyncer,
}) })
virtualsHandler := v2.NewVirtualsHandler(s.db) virtualsHandler := v2.NewVirtualsHandler(s.db)
healthHandler := v2.NewHealthHandler(s.db, s.cache, s.store) healthHandler := v2.NewHealthHandler(s.db, s.cache, s.store)
@@ -239,6 +248,7 @@ func (s *Server) Run(ctx context.Context) error {
go s.gc.Run(ctx) go s.gc.Run(ctx)
go s.syncer.Run(ctx) go s.syncer.Run(ctx)
go s.debSyncer.Run(ctx) go s.debSyncer.Run(ctx)
go s.alpineSyncer.Run(ctx)
httpServer := s.newHTTPServer() httpServer := s.newHTTPServer()
@@ -261,6 +271,7 @@ func (s *Server) RunOnListener(ctx context.Context, ln net.Listener) error {
go s.gc.Run(ctx) go s.gc.Run(ctx)
go s.syncer.Run(ctx) go s.syncer.Run(ctx)
go s.debSyncer.Run(ctx) go s.debSyncer.Run(ctx)
go s.alpineSyncer.Run(ctx)
httpServer := s.newHTTPServer() httpServer := s.newHTTPServer()
+38
View File
@@ -0,0 +1,38 @@
package testsupport
import (
"bytes"
"fmt"
)
// MinimalApk builds a valid-enough Alpine package in pure Go (no committed
// binary fixture, no abuild): two concatenated, independently gzipped tar
// streams -- a control stream carrying .PKGINFO and a data stream carrying a
// single payload file. It mirrors MinimalDeb/MinimalRPM and is parseable by the
// alpine provider (which derives arch/name/version and the Q1 pull checksum from
// the control stream).
func MinimalApk(name, version, arch string) []byte {
pkginfo := fmt.Sprintf(
"# generated by testsupport\n"+
"pkgname = %s\n"+
"pkgver = %s\n"+
"arch = %s\n"+
"pkgdesc = minimal test package\n"+
"url = https://example.com/%s\n"+
"license = MIT\n"+
"origin = %s\n"+
"maintainer = e2e <e2e@example.com>\n"+
"builddate = 1700000000\n"+
"size = 4\n"+
"depend = so:libc.musl-x86_64.so.1\n"+
"provides = cmd:%s=%s\n",
name, version, arch, name, name, name, version)
control := gzipBytes(tarSingle(".PKGINFO", []byte(pkginfo)))
data := gzipBytes(tarSingle("usr/bin/"+name, []byte("body")))
var buf bytes.Buffer
buf.Write(control)
buf.Write(data)
return buf.Bytes()
}
+2
View File
@@ -18,6 +18,7 @@ const (
PackageGoProxy PackageType = "goproxy" PackageGoProxy PackageType = "goproxy"
PackageGitHubRPM PackageType = "github_rpm" PackageGitHubRPM PackageType = "github_rpm"
PackageGitHubDeb PackageType = "github_deb" PackageGitHubDeb PackageType = "github_deb"
PackageGitHubAlpine PackageType = "github_alpine"
) )
var validPackageTypes = map[PackageType]bool{ var validPackageTypes = map[PackageType]bool{
@@ -34,6 +35,7 @@ var validPackageTypes = map[PackageType]bool{
PackageGoProxy: true, PackageGoProxy: true,
PackageGitHubRPM: true, PackageGitHubRPM: true,
PackageGitHubDeb: true, PackageGitHubDeb: true,
PackageGitHubAlpine: true,
} }
func (p PackageType) Valid() bool { func (p PackageType) Valid() bool {
+2
View File
@@ -19,6 +19,8 @@ func TestPackageTypeValid(t *testing.T) {
models.PackageTerraform, models.PackageTerraform,
models.PackageGoProxy, models.PackageGoProxy,
models.PackageGitHubRPM, models.PackageGitHubRPM,
models.PackageGitHubDeb,
models.PackageGitHubAlpine,
} }
for _, pt := range valid { for _, pt := range valid {
if !pt.Valid() { if !pt.Valid() {
+77
View File
@@ -2,6 +2,7 @@ package models
import ( import (
"fmt" "fmt"
"net/url"
"regexp" "regexp"
"time" "time"
) )
@@ -39,6 +40,14 @@ type Remote struct {
PackageType PackageType `json:"package_type"` PackageType PackageType `json:"package_type"`
RepoType RepoType `json:"repo_type"` RepoType RepoType `json:"repo_type"`
BaseURL string `json:"base_url"` BaseURL string `json:"base_url"`
// Mirrorlist holds additional upstream mirror base URLs. The effective
// upstream pool is [base_url] + mirrorlist, load-balanced round-robin with
// failover by the proxy engine. Only valid on remote rpm/deb/apk repos.
Mirrorlist []string `json:"mirrorlist,omitempty"`
// MirrorStrategy selects how the proxy engine picks the starting upstream
// from the pool: round_robin (default/empty) rotates, least_conn favors the
// mirror with the fewest in-flight requests. Failover order is unchanged.
MirrorStrategy string `json:"mirror_strategy,omitempty"`
Description string `json:"description,omitempty"` Description string `json:"description,omitempty"`
Username string `json:"-"` Username string `json:"-"`
Password string `json:"-"` Password string `json:"-"`
@@ -72,6 +81,74 @@ type Remote struct {
UpdatedAt time.Time `json:"updated_at"` UpdatedAt time.Time `json:"updated_at"`
} }
// Mirror balancing strategies for MirrorStrategy. An empty value is treated as
// round_robin, so existing remotes keep their current behavior.
const (
MirrorStrategyRoundRobin = "round_robin"
MirrorStrategyLeastConn = "least_conn"
)
// mirrorlistPackageTypes are the package types for which a mirrorlist is
// allowed: OS package repos (rpm, deb, apk/alpine) that fetch many small files
// and benefit most from mirror load-balancing and failover.
var mirrorlistPackageTypes = map[PackageType]bool{
PackageRPM: true,
PackageDeb: true,
PackageAlpine: true,
}
// UpstreamPool returns the ordered upstream base URLs for this remote: the
// primary base_url first, followed by any mirrorlist entries. The proxy engine
// load-balances round-robin across the pool and fails over between them.
func (r Remote) UpstreamPool() []string {
pool := make([]string, 0, 1+len(r.Mirrorlist))
if r.BaseURL != "" {
pool = append(pool, r.BaseURL)
}
pool = append(pool, r.Mirrorlist...)
return pool
}
// ValidateMirrorlist enforces that a mirrorlist is only configured on remote
// rpm/deb/apk repositories and that every entry is a parseable http/https URL.
func (r *Remote) ValidateMirrorlist() error {
if len(r.Mirrorlist) == 0 {
return nil
}
if r.RepoType != RepoTypeRemote {
return fmt.Errorf("mirrorlist is only allowed on remote repositories")
}
if !mirrorlistPackageTypes[r.PackageType] {
return fmt.Errorf("mirrorlist is only allowed for rpm, deb and alpine package types, not %q", r.PackageType)
}
for _, u := range r.Mirrorlist {
parsed, err := url.ParseRequestURI(u)
if err != nil {
return fmt.Errorf("invalid mirrorlist url %q: %w", u, err)
}
if parsed.Scheme != "http" && parsed.Scheme != "https" {
return fmt.Errorf("mirrorlist url %q must be http or https", u)
}
}
return nil
}
// ValidateMirrorStrategy enforces that mirror_strategy is one of the allowed
// values and that a non-default strategy (least_conn) is only set alongside a
// non-empty mirrorlist, where balancing is meaningful. An empty strategy is
// accepted and behaves as round_robin.
func (r *Remote) ValidateMirrorStrategy() error {
switch r.MirrorStrategy {
case "", MirrorStrategyRoundRobin, MirrorStrategyLeastConn:
default:
return fmt.Errorf("invalid mirror_strategy %q: must be %q or %q", r.MirrorStrategy, MirrorStrategyRoundRobin, MirrorStrategyLeastConn)
}
if r.MirrorStrategy == MirrorStrategyLeastConn && len(r.Mirrorlist) == 0 {
return fmt.Errorf("mirror_strategy %q requires a non-empty mirrorlist", r.MirrorStrategy)
}
return nil
}
// ValidatePatterns ensures every configured regex compiles. Storing an // ValidatePatterns ensures every configured regex compiles. Storing an
// invalid pattern would otherwise be silently dropped at match time, which // invalid pattern would otherwise be silently dropped at match time, which
// for the blocklist is a fail-open: a mistyped deny rule becomes a no-op. // for the blocklist is a fail-open: a mistyped deny rule becomes a no-op.
+107 -1
View File
@@ -1,6 +1,10 @@
package models package models
import "testing" import (
"encoding/json"
"strings"
"testing"
)
func TestRemote_ValidatePatterns(t *testing.T) { func TestRemote_ValidatePatterns(t *testing.T) {
valid := &Remote{ valid := &Remote{
@@ -17,3 +21,105 @@ func TestRemote_ValidatePatterns(t *testing.T) {
t.Fatal("expected error for invalid blocklist regex, got nil") t.Fatal("expected error for invalid blocklist regex, got nil")
} }
} }
func TestRemoteMirrorlistJSON(t *testing.T) {
// base_url stays a plain string; mirrorlist round-trips as an array.
var r Remote
body := `{"name":"x","package_type":"rpm","repo_type":"remote","base_url":"https://a.example","mirrorlist":["https://b.example","https://c.example"]}`
if err := json.Unmarshal([]byte(body), &r); err != nil {
t.Fatal(err)
}
if r.BaseURL != "https://a.example" {
t.Errorf("BaseURL = %q, want https://a.example", r.BaseURL)
}
if len(r.Mirrorlist) != 2 || r.Mirrorlist[0] != "https://b.example" || r.Mirrorlist[1] != "https://c.example" {
t.Errorf("Mirrorlist = %v, want two entries", r.Mirrorlist)
}
out, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(out), `"base_url":"https://a.example"`) {
t.Errorf("marshal lost base_url: %s", out)
}
if !strings.Contains(string(out), `"mirrorlist":["https://b.example","https://c.example"]`) {
t.Errorf("marshal lost mirrorlist: %s", out)
}
}
func TestRemoteMirrorlistOmitempty(t *testing.T) {
out, err := json.Marshal(Remote{Name: "x", PackageType: PackageRPM, RepoType: RepoTypeRemote, BaseURL: "https://a.example"})
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(out), "mirrorlist") {
t.Errorf("empty mirrorlist should be omitted: %s", out)
}
}
func TestUpstreamPool(t *testing.T) {
// base_url first, then mirrorlist.
r := Remote{BaseURL: "https://a.example", Mirrorlist: []string{"https://b.example", "https://c.example"}}
pool := r.UpstreamPool()
want := []string{"https://a.example", "https://b.example", "https://c.example"}
if strings.Join(pool, ",") != strings.Join(want, ",") {
t.Errorf("UpstreamPool = %v, want %v", pool, want)
}
// No mirrorlist ⇒ pool is just [base_url].
solo := Remote{BaseURL: "https://a.example"}
if got := solo.UpstreamPool(); len(got) != 1 || got[0] != "https://a.example" {
t.Errorf("solo UpstreamPool = %v, want [base_url]", got)
}
}
func TestValidateMirrorStrategy(t *testing.T) {
ml := []string{"https://m.example"}
cases := []struct {
name string
remote Remote
wantErr bool
}{
{"empty defaults ok", Remote{Mirrorlist: ml}, false},
{"explicit round_robin ok", Remote{MirrorStrategy: MirrorStrategyRoundRobin, Mirrorlist: ml}, false},
{"round_robin without mirrorlist ok", Remote{MirrorStrategy: MirrorStrategyRoundRobin}, false},
{"least_conn with mirrorlist ok", Remote{MirrorStrategy: MirrorStrategyLeastConn, Mirrorlist: ml}, false},
{"least_conn without mirrorlist rejected", Remote{MirrorStrategy: MirrorStrategyLeastConn}, true},
{"unknown strategy rejected", Remote{MirrorStrategy: "random", Mirrorlist: ml}, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
err := tc.remote.ValidateMirrorStrategy()
if (err != nil) != tc.wantErr {
t.Errorf("ValidateMirrorStrategy() err = %v, wantErr = %v", err, tc.wantErr)
}
})
}
}
func TestValidateMirrorlist(t *testing.T) {
cases := []struct {
name string
remote Remote
wantErr bool
}{
{"empty is ok on anything", Remote{RepoType: RepoTypeRemote, PackageType: PackageGeneric}, false},
{"rpm remote ok", Remote{RepoType: RepoTypeRemote, PackageType: PackageRPM, Mirrorlist: []string{"https://m.example"}}, false},
{"deb remote ok", Remote{RepoType: RepoTypeRemote, PackageType: PackageDeb, Mirrorlist: []string{"http://m.example"}}, false},
{"alpine remote ok", Remote{RepoType: RepoTypeRemote, PackageType: PackageAlpine, Mirrorlist: []string{"https://m.example"}}, false},
{"generic remote rejected", Remote{RepoType: RepoTypeRemote, PackageType: PackageGeneric, Mirrorlist: []string{"https://m.example"}}, true},
{"docker remote rejected", Remote{RepoType: RepoTypeRemote, PackageType: PackageDocker, Mirrorlist: []string{"https://m.example"}}, true},
{"local rpm rejected", Remote{RepoType: RepoTypeLocal, PackageType: PackageRPM, Mirrorlist: []string{"https://m.example"}}, true},
{"bad scheme rejected", Remote{RepoType: RepoTypeRemote, PackageType: PackageRPM, Mirrorlist: []string{"ftp://m.example"}}, true},
{"unparseable rejected", Remote{RepoType: RepoTypeRemote, PackageType: PackageRPM, Mirrorlist: []string{"://nope"}}, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
err := tc.remote.ValidateMirrorlist()
if (err != nil) != tc.wantErr {
t.Errorf("ValidateMirrorlist() err = %v, wantErr = %v", err, tc.wantErr)
}
})
}
}
+13 -3
View File
@@ -17,8 +17,8 @@ cleanup() {
} }
trap cleanup EXIT trap cleanup EXIT
echo "==> building and starting stack (postgres, redis, minio, mockupstream, artifactapi)" echo "==> building and starting stack (postgres, redis, minio, mockupstream(s), artifactapi)"
"${COMPOSE[@]}" up -d --build postgres redis minio mockupstream artifactapi "${COMPOSE[@]}" up -d --build postgres redis minio mockupstream mockupstreama mockupstreamb artifactapi
echo "==> waiting for artifactapi health at ${API_URL}" echo "==> waiting for artifactapi health at ${API_URL}"
for i in $(seq 1 60); do for i in $(seq 1 60); do
@@ -34,7 +34,17 @@ for i in $(seq 1 60); do
sleep 1 sleep 1
done done
echo "==> running dockerised e2e suite" # Resolve the compose network the artifactapi container is attached to, so the
# real-package-manager test can launch a stock distro container on the same
# network and reach artifactapi by service name.
API_CID="$("${COMPOSE[@]}" ps -q artifactapi)"
COMPOSE_NETWORK="$(docker inspect -f '{{range $k,$_ := .NetworkSettings.Networks}}{{$k}}{{end}}' "${API_CID}" 2>/dev/null || true)"
echo "==> running dockerised e2e suite (compose network: ${COMPOSE_NETWORK:-unknown})"
ARTIFACTAPI_URL="${API_URL}" \ ARTIFACTAPI_URL="${API_URL}" \
MOCK_UPSTREAM_INTERNAL="${MOCK_UPSTREAM_INTERNAL:-http://mockupstream}" \ MOCK_UPSTREAM_INTERNAL="${MOCK_UPSTREAM_INTERNAL:-http://mockupstream}" \
MOCK_UPSTREAM_A_INTERNAL="${MOCK_UPSTREAM_A_INTERNAL:-http://mockupstreama}" \
MOCK_UPSTREAM_B_INTERNAL="${MOCK_UPSTREAM_B_INTERNAL:-http://mockupstreamb}" \
ARTIFACTAPI_INTERNAL="${ARTIFACTAPI_INTERNAL:-http://artifactapi:8000}" \
COMPOSE_NETWORK="${COMPOSE_NETWORK}" \
go test -tags=dockere2e -count=1 -timeout=10m -v ./e2e-docker/... go test -tags=dockere2e -count=1 -timeout=10m -v ./e2e-docker/...
+72 -1
View File
@@ -176,7 +176,25 @@ helm install <release> ${name}/<chart>`,
]; ];
case 'alpine': case 'alpine':
return [ return isLocal
? [
{
title: 'Add the apk repo (real apk repo, APKINDEX auto-generated)',
language: 'bash',
code: `echo '${url}/api/v1/local/${name}' | sudo tee -a /etc/apk/repositories
sudo apk update --allow-untrusted
sudo apk add --allow-untrusted <package>`,
note: `Served unsigned (parity with the rpm repo's gpgcheck=0) — use --allow-untrusted, or install a signing key. apk fetches <arch>/APKINDEX.tar.gz under this base.`,
},
{
title: 'Publish a .apk (index regenerates automatically)',
language: 'bash',
code: `curl -fsSL --upload-file ./mypkg-1.0-r0.apk \\
${url}/api/v2/remotes/${name}/files/x86_64/mypkg-1.0-r0.apk`,
note: 'Upload each package at <arch>/<name>-<version>.apk — apk reconstructs that exact path from the index (APKINDEX carries no filename), so a mismatched path will 404 on install.',
},
]
: [
{ {
title: 'Add the APK repository', title: 'Add the APK repository',
language: 'bash', language: 'bash',
@@ -187,6 +205,18 @@ sudo apk add <package>`,
}, },
]; ];
case 'github_alpine':
return [
{
title: 'Add the apk repo (metadata-only, from GitHub releases)',
language: 'bash',
code: `echo '${proxy}' | sudo tee -a /etc/apk/repositories
sudo apk update --allow-untrusted
sudo apk add --allow-untrusted <package>`,
note: "The per-arch APKINDEX is synthesized from the configured GitHub repo's release .apk assets; package downloads are redirected to the backing releases remote. Served unsigned, so --allow-untrusted.",
},
];
case 'goproxy': case 'goproxy':
return [ return [
{ {
@@ -208,6 +238,47 @@ go mod download`,
}, },
]; ];
case 'deb':
return isLocal
? [
{
title: 'Add the apt repo (real apt repo, flat — Packages/Release auto-generated)',
language: 'bash',
code: `echo 'deb [trusted=yes] ${url}/api/v1/local/${name}/ ./' | sudo tee /etc/apt/sources.list.d/${name}.list
sudo apt-get update
sudo apt-get install <package>`,
note: '[trusted=yes]: artifactapi serves the flat repo unsigned (matches the rpm repo\'s gpgcheck=0). The `./` is the flat-repo suite — apt fetches Packages/Release from the repo root.',
},
{
title: 'Publish a .deb (index regenerates automatically)',
language: 'bash',
code: `curl -fsSL --upload-file ./my-package_1.0_amd64.deb \\
${url}/api/v2/remotes/${name}/files/my-package_1.0_amd64.deb`,
},
]
: [
{
title: 'Add the apt repo (caching proxy)',
language: 'bash',
code: `echo 'deb ${proxy} <suite> <component>' | sudo tee /etc/apt/sources.list.d/${name}.list
sudo apt-get update
sudo apt-get install <package>`,
note: "Signatures are verified against the upstream mirror's real signed Release through the proxy (no [trusted=yes] needed). Example suite/component: bookworm main.",
},
];
case 'github_deb':
return [
{
title: 'Add the apt repo (metadata-only, from GitHub releases)',
language: 'bash',
code: `echo 'deb [trusted=yes] ${proxy}/ ./' | sudo tee /etc/apt/sources.list.d/${name}.list
sudo apt-get update
sudo apt-get install <package>`,
note: "The apt index is synthesized from the configured GitHub repo's release .deb assets; package downloads are redirected to the backing releases remote. Served unsigned, so [trusted=yes].",
},
];
case 'generic': case 'generic':
default: default:
return [ return [