Compare commits

..

11 Commits

Author SHA1 Message Date
unkin-agent 26c37024ae ui: add deb and github_deb usage instructions (#113)
ci/woodpecker/tag/docker Pipeline was successful
## Why

The `deb` and `github_deb` repository types had no tailored "How do I use this?" snippets in the UI — they fell through to the generic default (curl-a-file-by-path), which does not tell a user how to actually wire the repo into apt. This adds real, end-to-end-validated apt instructions mirroring how the existing package types (rpm, npm, etc.) are handled.

## How

- Adds a `case 'deb':` to `buildSnippets` that branches on `isLocal`: a flat real apt repo served at `/api/v1/local/<name>/ ./` with `[trusted=yes]` plus a publish-a-.deb snippet for locals, and a caching-proxy `deb <proxy> <suite> <component>` form (upstream-signed, no `[trusted=yes]`) for remotes.
- Adds a `case 'github_deb':` emitting the metadata-only flat form `deb [trusted=yes] <proxy>/ ./`, whose index is synthesized from a GitHub repo's release .deb assets.
- Reuses the existing `url`/`proxy`/`isLocal` helpers and matches the surrounding case style; leaves the rpm and other cases untouched.

Reviewed-on: #113
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-11 23:54:34 +10:00
unkin-agent 5fde0ee58e Add github_deb metadata-only package type (#112)
ci/woodpecker/tag/docker Pipeline was successful
## Why

This stacks the Debian/apt analog of `github_rpm` on top of the deb local+remote work (#111). It lets a GitHub repo's `.deb` release assets be consumed as a real apt repository without artifactapi ever precaching whole packages: it derives per-asset control metadata from a ranged prefix fetch, synthesizes a flat apt repo from the cache, and redirects the actual `.deb` downloads to a backend `releases_remote` (the generic github.com remote).

Base is `benvin/deb-local-remote` (stacked) to keep the diff atomic.

## How

- Adds `github_deb` to the package-type enum and validity map.
- Adds the `github_deb` provider mirroring `github_rpm`: `ServeRemote` serves `Packages`/`Packages.gz`/`Release`, returns 404 for `InRelease`/`Release.gpg` (unsigned, consumed via `[trusted=yes]`), and 302-redirects `*.deb` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`; cold-start prime with a retryable 503.
- `deriveAsset` ranged-GETs the front of the `.deb` (an `ar` archive), locates and fully reads `control.tar.*`, and parses the control paragraph — doubling the range if the control member is truncated. The Packages `SHA256` comes from the GitHub asset `digest` when present, else a one-time full stream; `MD5sum` is left unset (apt verifies against SHA256 under `[trusted=yes]`).
- Adds a `github_deb` background Syncer (own worker pool, shared rate limiter, deduped queue) with per-remote DB-lease-gated scans so only one replica scans per window.
- Adds the `github_deb_sync_state` table plus `ListGitHubDebRemotes` / `ClaimGitHubDebSyncLease` / `ReleaseGitHubDebSyncLease` DB helpers, kept separate from the rpm ones.
- Primes `github_deb` remotes on create and runs the deb syncer alongside the rpm one; prime-on-create is routed by package type.
- Reuses the deb apt-index generators and control parser; the Packages generator now skips empty hash lines so a SHA256-only entry is valid.

## Notes / deviations

- **Filename convention:** the `Filename` stored in the Packages index is the **github-relative** asset path (same as rpm's `assetPath`), not `pool/<asset>`. This is required for the `.deb` 302 to `{releases_remote=github}/{path}` to resolve against github.com; it still matches the `*.deb` redirect rule.
- **GitHub client helpers** (releases pagination, ranged GET, auth headers) are duplicated into the deb package rather than shared, because the rpm equivalents are unexported in `package rpm` and the task requires not modifying the rpm provider.
- `go build`, `go vet`, `go mod tidy`, and `make test` (`-race`, incl. the Postgres lease integration tests) all pass; pre-commit clean.

Do not merge — for review.

---------

Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #112
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-11 23:28:11 +10:00
unkin-agent 60f008debc Add deb (Debian/apt) local and remote repository support (#111)
Brings Debian/apt to artifactapi with feature parity to the existing rpm support (local + remote), so `.deb` packages can be hosted as a flat apt repo and a Debian/Ubuntu mirror can be cached through the proxy.

- Adds `deb` to the package-type enum and registers a new `internal/provider/deb` provider.
- Classifies `.deb` blobs immutable and the apt index surface (`Packages`, `Release`, `InRelease`, `dists/`, by-hash) mutable so the caching engine revalidates it.
- Parses the `.deb` in pure Go (ar archive to `control.tar.{gz,xz,zst}` to `./control`), storing the raw control stanza plus computed size/md5/sha256 as `deb_metadata`.
- Serves a flat apt repo (`deb [trusted=yes] .../ ./`): generates `Packages`, `Packages.gz` and an unsigned `Release` (returns 404 for `InRelease`/`Release.gpg`), mirroring rpm unsigned repodata / gpgcheck=0 trust model.
- Proxies a remote mirror via `UpstreamURL`/`ContentType`/`AuthHeaders` (HTTP Basic).
- Adds the `deb_metadata` table to `migrate()`, DB access methods, a `MinimalDeb` pure-Go fixture, unit tests, and a `dockere2e` `TestLocalDebRepo`.

---------

Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #111
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
2026-08-11 23:21:08 +10:00
unkinben 109ba2ce27 feat: server-level GitHub machine credential for authenticated requests (#109)
ci/woodpecker/tag/docker Pipeline was successful
## Why

Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories. A machine credential usable by a free (non-enterprise) account is needed to lift the request budget to ~5000/hr and to read private-repo release assets.

Builds on the background syncer (#108, now merged to `master`); this diff is the auth changes only.

## How

- Add `internal/githubauth`: a process-wide GitHub credential delivered via env/secret, applied by default to every outbound GitHub request (releases scan, ranged asset-header GETs, and the generic-github byte proxy for private assets).
- Support two modes:
  - **PAT** — `GITHUB_TOKEN` sent as `Authorization: Bearer <token>`.
  - **GitHub App** — `GITHUB_APP_ID` + `GITHUB_APP_INSTALLATION_ID` + private key (`GITHUB_APP_PRIVATE_KEY` inline PEM or `GITHUB_APP_PRIVATE_KEY_PATH`). Mint a short-lived RS256 JWT with stdlib `crypto/rsa` (no new dependency), exchange it at `POST /app/installations/{id}/access_tokens` for a ~1h installation token, cache it, and single-flight a refresh a few minutes before expiry.
- Inject at the two GitHub call paths: the rpm github provider header builder (releases + ranged fetches) and the generic provider `AuthHeaders` (byte proxy, github.com hosts only; the pre-signed `objects.githubusercontent.com` redirect deliberately gets no Authorization).
- Honor precedence: a remote's own `username`/`password` overrides the server credential; no credential configured stays anonymous (current behavior).
- Fail closed at startup on partial App configuration (e.g. App id without a private key); a token-and-App conflict is also rejected.
- Never persist the credential to the DB, return it from an API, or log it (token-exchange failures never echo the response body).
- Read config via the existing `getenv` convention; document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication.

## Rate limit

Authenticated requests share the syncer's single global limiter — no second limiter is added. A token raises the effective GitHub ceiling (~5000/hr vs ~60/hr), so the limiter defaults stay safe.

## Tests

`internal/githubauth` and `internal/provider/{rpm,generic}`:
- PAT attaches the correct `Authorization` header to releases + asset-header requests.
- App mints a valid RS256 JWT (verified against the app public key), exchanges it at a mocked endpoint, reuses the cached token without re-exchanging, refreshes near expiry, and single-flights concurrent callers.
- Per-remote credential overrides the server credential (rpm + generic).
- No credential → no `Authorization` header, requests still succeed anonymously.
- ETag/304 flow still works with auth attached.
- The credential does not appear in a remote's serialized JSON.
- Config validation: no-config is anonymous; partial App config and token/App conflict both error.

Verified fail-before/pass-after for the injection tests. `gofmt -l`, `go build ./...`, `go vet ./...`, `go test ./...` all clean (26 packages).

Reviewed-on: #109
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-10 21:42:39 +10:00
unkinben e24c35f534 feat: background syncer for github_rpm remotes (#108)
## Why

Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.

## How

- Add a single per-process background syncer (started at boot, cleanly stopped
  on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
  global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the
  github provider so every GitHub call (releases list + each ranged asset GET)
  acquires a token first.
- Re-check each `github_rpm` remote for new/changed releases on its existing
  `mutable_ttl` cadence; derive only new/changed assets incrementally and prune
  assets that disappear upstream. Repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call returns
  immediately.
- Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives
  nothing and is not counted against GitHub's rate limit, so an unchanged repo
  is nearly free.
- Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`,
  `etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only
  for the replica that atomically claims the lease, bounding total GitHub load
  to ~once per `mutable_ttl` regardless of replica count; the ETag is shared
  through the same row.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
  cache, and return a bounded wait then a retryable `503` rather than blocking
  on a cold derive.
- Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` /
  `GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst
  5, 3 workers, 60s tick) and document the syncer in the README.

## Tests

- Unit (httptest, Range/ETag-aware fixture): `304` releases response derives
  nothing; incremental derive fetches only the newly added asset; the shared
  limiter caps request rate; work-queue enqueues coalesce to one job; prime
  enqueues a job; a held lease stops a second replica from scanning; cold-start
  serves `503` while warm cache serves `200`.
- DB integration (testcontainers postgres): the real lease SQL — one holder at a
  time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
  bypasses recency but respects a live lease.
- Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the
  background at ~1 req/s (global limiter); `dnf makecache` served fast from the
  priming cache (no cold block); clean shutdown mid-scan, no panics.

## Notes

- Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.

Reviewed-on: #108
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-10 21:31:24 +10:00
unkinben d154fbf3f3 feat: github_rpm metadata-only remote (GitHub releases as a yum repo, no precache) (#107)
## Why

Publishing RPMs to GitHub releases is common, but consuming them with `dnf` requires repodata GitHub does not provide, and mirroring every package into a local repo wastes storage and staleness-tracking on artifacts that already have a durable home. This exposes GitHub releases as a first-class RPM source that synthesizes repodata on the fly and **never precaches the packages**.

## What

Add a `github_rpm` remote package type backed by a metadata-only provider.

- Introduce a `RemoteServer` interception hook (the remote-side analog of `LocalIndexer`): `handleProxy` lets a provider fully answer a request before the byte-proxy engine, passing the request-derived proxy base URL and the DB as a `RemoteMetadataStore`.
- Scan a repo's releases via the GitHub API (`base_url` = the releases API root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset filename), and reuse the existing local-rpm repodata generators to emit `repomd.xml`/`primary`/`filelists`/`other`.
- Derive per-asset metadata without precaching: fetch only the RPM header via a ranged GET (retrying with a larger range on a truncated-header parse) for NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from the GitHub asset `digest` when present, else compute it once by streaming.
- Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more often than `mutable_ttl`, pruning assets that disappear upstream.
- Serve each package's `<location>` as the github-relative download path so the client comes back to this remote, which **302-redirects** to the `releases_remote` (an existing generic github.com remote) that streams the actual bytes.

Reuse the existing `releases_remote` field as the redirect target — it already carries exactly this "downloads served by remote X" semantic end to end, so no new schema/model field is needed.

Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns, added idempotently) so both local and `github_rpm` repodata resolve upgrades and conflicts; the local upload path records them too.

## No-precache mechanics

- **Dependency metadata**: always from the ranged header fetch (header precedes payload; `rpm.Read` stops at the payload boundary), giving `dnf` full resolution. Default range 1 MiB, doubling to 16 MiB.
- **Checksum**: prefer the GitHub asset `digest` (no download); fall back to a one-time streamed sha256 only when absent. Header-only "minimal mode" (no deps) is rejected as a default because `dnf` needs accurate provides/requires and a correct pkgid checksum to install.

## Tests

Header-range parsing incl. the retry loop, digest-vs-computed checksum selection, repodata synthesis with the redirect-able `<location href>`, the 302 redirect path (and the guard when `releases_remote` is unset), asset pattern filtering, and stale-asset pruning. `go build`/`vet`/`test` green; pre-commit clean.

## Follow-ups

- `github_apk` / `github_deb` metadata-only remotes (same pattern; not in this PR).
- Terraform provider support for `artifactapi_remote_github_rpm` ships as a separate PR against `terraform-provider-artifactapi` (depends on this API surface).

Reviewed-on: #107
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-10 20:54:44 +10:00
unkinben eee8ee1c31 feat(ui): add "How do I use this?" usage instructions to repo detail pages (#105)
ci/woodpecker/tag/docker Pipeline was successful
## Why

A repository detail page in the ArtifactAPI UI showed configuration and stats, but nothing that told a user how to actually *consume* the repo. You had to already know the per-package-type URL scheme (yum baseurl, pip index-url, docker registry host, terraform source address, ...) by hand. This adds an in-page, copy-pasteable "How do I use this?" panel so each repo page tells you exactly how to point a Linux host at it.

## Changes

- Add a `UsageInstructions` component: a collapsible "How do I use this?" panel with monospace code boxes and copy-to-clipboard buttons, styled to match the existing detail-section / badge theme.
- Generate instructions per package type (rpm, pypi, npm, docker, terraform, helm, alpine, goproxy, puppet, generic) and per class:
  - **remote** — consume via the caching proxy (`/api/v1/remote/<name>/...`).
  - **local** — consume via the real registry endpoint, plus a publish/push example (rpm `PUT .../files/`, docker Registry V2 push, terraform provider upload).
  - **virtual** — consume the merged index via `/api/v1/virtual/<name>/...` using the same per-type client config.
- Interpolate the repository's real name into every snippet so it is genuinely copy-pasteable.
- Resolve the instance base URL from `window.location.origin` (the UI is served on the API origin, client `BASE=''`) instead of hardcoding a hostname; falls back to the public host only when `window` is unavailable.
- Render the panel on `RemoteDetail`, `LocalDetail`, and the `Virtuals` member-expand panel.

## Verification

- `npm run build` (tsc typecheck + vite build) passes.
- Rendered the component in headless Chromium against real repo data for rpm remote, rpm local (with publish), docker local (with push), terraform local (HCL `required_providers` + signing note), and a pypi virtual — all snippets render with the correct URLs and theme.

Reviewed-on: #105
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 14:16:22 +10:00
unkinben f6b0afc5d6 feat(ui): direct-download links for downloadable local repo files (#106)
## Why

The local-repo object browser renders every file name as inert text, so there is no way to grab a file from the UI even though local rpm repos already serve their contents as real yum repos. Users have to hand-construct URLs. This adds one-click downloads, built as a modular per-repo-type capability so other types can be switched on later with a single map entry.

## Changes

- Adds a `downloadableTypes` capability map (`ui/src/components/downloads.ts`) keyed by `package_type`; each entry builds the direct-download URL for a file.
- Enables `rpm`, pointing at the yum files route `/api/v2/remotes/<repo>/files/<path>` (path-segment-encoded).
- Renders file names in the object browser as `<a href download>` links when the repo's type is downloadable, and as plain text otherwise.
- Fetches the repo's `package_type` on the Objects page as the modularity hook and threads it through the tree rows.

## Notes

Download links are same-origin unauthenticated GETs (the API serves local repos with no token on reads, matching how yum clients fetch), so a bare `href` carries no credentials. Adding a future type is one entry in `downloadableTypes`.

---------

Co-authored-by: Ben Vin <neotheo@gmail.com>
Reviewed-on: #106
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 14:15:25 +10:00
unkinben 649f89f58b fix: make local docker uploads replica-independent (#104)
ci/woodpecker/tag/docker Pipeline was successful
## Why

Chunked blob uploads kept the in-progress session in **process memory** keyed by upload UUID, so the `POST`/`PATCH`/`PUT` of a single `docker push` had to land on the same replica. The API runs at `minReplicas: 2` with no session affinity (see argocd-apps `api-hpa.yaml`), so a real push — which streams the layer via `PATCH` then finalises with `PUT` — intermittently 404s with `BLOB_UPLOAD_UNKNOWN` when a chunk hits a replica that never saw the `POST`. This was flagged when the local docker registry landed (#103).

## Changes

- Stage chunked uploads in object storage under `uploads/<uuid>` instead of an in-memory temp file. The UUID travels in the `Location` URL handed to the client, so any replica reconstructs the staging key with no shared in-process state. Finalise streams the staged bytes plus any trailing `PUT` body through the CAS in one pass; monolithic uploads are unchanged.
- Support `DELETE` of an in-progress upload (cancel) by dropping its staging object.
- Reap abandoned staging objects in the GC (`uploads/` older than 24h) via a new `S3.ListStaleObjects`, so cancelled/interrupted pushes don't leak.

## Verification

- Split a single push across **two instances sharing one Postgres+MinIO**: `POST`→A, `PATCH`→B, `PUT`→A finalises with the correct digest, and the blob pulls back **byte-identical from both** replicas. Config-blob and manifest pushes split the same way succeed; `tags/list` is correct. (Pre-fix, the cross-replica `PATCH` 404s.)
- `scripts/docker-e2e.sh` still passes (incl. `TestLocalDockerPushPull`); unit tests + `go vet` clean.

Reviewed-on: #104
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-05 17:39:49 +10:00
unkinben a92ede23f6 feat: serve local docker repos as a real registry (#103)
ci/woodpecker/tag/docker Pipeline was successful
## Why

Local `docker` repos had no write path — the `/v2` Docker Registry API only proxied to upstreams. This makes a local docker repo a genuine container registry so `docker push`/`docker pull` (and podman/skopeo/buildah) work against it directly, matching the project principle that a local repo is *the real thing* rather than a mirror.

## Changes

- Implement the Docker Registry HTTP API V2 read/write half for local docker repos: blob uploads (monolithic and chunked POST/PATCH/PUT), manifest push, `tags/list`, and blob/manifest GET/HEAD.
- Store blobs and manifests through the existing content-addressable store; keep a `local_files` reference per (repo, image) so the GC does not reap them. Tags are mutable (`UpsertLocalFile`); digests and blobs are immutable.
- Dispatch `/v2` reads to the local handler for local docker repos and fall through to the upstream proxy otherwise; writes are local-docker only.
- Add `UpsertLocalFile` for mutable tag references.
- Cover the push/pull round-trip with a dockerised e2e test and unit-test the registry path parser. Document the registry in the README.

## Verification

- `scripts/docker-e2e.sh` passes, including the new `TestLocalDockerPushPull`.
- Verified a real end-to-end round-trip with skopeo against a live instance: pushed `hello-world`, pulled it back, loaded it into the docker daemon, and ran it successfully.

Reviewed-on: #103
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-05 16:55:53 +10:00
unkinben 936cf8846a feat: serve local terraform repos as a provider registry (#102)
ci/woodpecker/tag/docker Pipeline was successful
## Why

Local terraform repos already served the Terraform **network mirror** protocol, but consuming that requires every user to add a `provider_installation { network_mirror }` block to `~/.terraformrc`. A `source = "artifactapi.k8s.../ns/type"` address instead triggers the **provider registry** protocol (service discovery at `/.well-known/terraform.json` + GPG-signed SHA256SUMS), which returned 404 — hence *"does not offer a provider registry."*

Local repos are meant to be the real thing, so this makes a terraform local repo a first-class provider registry: `terraform init` installs from a bare source address with no client config.

## What

- Serve `/.well-known/terraform.json` service discovery and the `providers.v1` endpoints under `/terraform/v1/providers`: `versions`, `download/{os}/{arch}`, `sha256sums`, `sha256sums.sig`.
- Map the Terraform **namespace** segment to the artifactapi **repo name**; locate the provider by **type**. `download_url` points back at the existing `/api/v1/local/...` path.
- Generate `SHA256SUMS` per version and sign it with a GPG key loaded from `TF_SIGNING_KEY_PATH` (optional `TF_SIGNING_KEY_PASSPHRASE`); advertise the public key + key id in the download response. **No key → registry stays disabled (endpoints 404)**, so behaviour is unchanged until the signing secret is present.
- New `internal/tfsign` (key load + detached signing, via `x/crypto/openpgp`) and `internal/api/terraform` (registry handler). Export `ParseProviderZip` for reuse.
- `TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
- README section documenting usage.

## Consumer

```hcl
terraform {
  required_providers {
    artifactapi = {
      source  = "artifactapi.k8s.syd1.au.unkin.net/terraform-unkin/artifactapi"
      version = "0.1.2"
    }
  }
}
```

## Tests

- `internal/tfsign`: sign + verify round-trip, disabled/missing-key paths.
- `internal/api/terraform`: dockerised full flow (discovery → versions → download → sha256sums → sig), verifying the signature against the advertised public key.

## Follow-ups (separate PRs)

- **argocd-apps**: mount the signing K8s secret into the api deployment + set `TF_SIGNING_KEY_PATH`. The `/` HTTPRoute already routes `/.well-known` and `/terraform` to the API, so no gateway change is needed.
- Image/version bump once tagged.

## Note

Anchored the `terraform/` gitignore to the repo root (`/terraform/`) so it stops matching `internal/*/terraform/`. This surfaced `internal/provider/terraform/terraform_extra_test.go`, which had been silently untracked — now committed.

Reviewed-on: #102
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 18:55:35 +10:00
62 changed files with 8490 additions and 36 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
bin/
terraform/
/terraform/
# e2e-docker fixtures are real package files (.rpm, .tgz, .whl, .zip, ...) that
# are intentionally tracked, overriding any global ignore of those extensions.
+198
View File
@@ -32,9 +32,150 @@ API: `http://localhost:8000` | Frontend: `http://localhost:5173`
| `puppet` | `v3/modules/*`, `v3/releases*` | `.tar.gz` |
| `terraform` | `*/versions` | `*/download/*/*` |
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
Providers classify paths automatically. Users only configure what to proxy and TTLs.
### `github_rpm` — GitHub releases as a yum repo (metadata-only, no precache)
A `github_rpm` remote turns a GitHub repo's **releases** into a real `dnf`/`yum`
repository without ever caching the packages. It scans releases for `.rpm`
assets, derives each package's metadata (NEVRA, requires/provides/conflicts/
obsoletes, files, checksum) and **synthesizes `repodata/` on the fly**. Package
metadata comes from a **ranged GET of just the RPM header** (the header sits at
the front of the file, so the whole package is never downloaded); the sha256
checksum comes from the GitHub asset `digest` when present, else a one-time
lazy stream. Derived metadata is cached (keyed by asset) so repodata generation
is served from primed DB rows, never a cold on-demand derive.
Each package's `<location>` points back at the remote, which **302-redirects**
the download to the `releases_remote` — an existing generic `github.com` remote
that streams the actual bytes. `dnf` follows the redirect transparently.
#### Background syncer
A single process-wide **background syncer** keeps every `github_rpm` remote's
derived metadata current off the client request path:
- **Prime on create.** Creating a `github_rpm` remote enqueues a background prime
scan, so its metadata is derived right away without blocking the create call.
The first `dnf` request is served from cache. If a request arrives before the
prime lands, it returns a retryable `503` (with `Retry-After`) rather than
serving an empty repo or blocking on a multi-minute derive.
- **Periodic re-check, driven by `mutable_ttl`.** Each remote is re-checked for
new or changed releases no more often than its `mutable_ttl`. New/changed
assets are derived incrementally; assets already cached are never re-fetched,
and assets that disappear upstream are pruned.
- **ETag / 304 conditional requests.** The releases-list `ETag` is stored per
remote and sent as `If-None-Match`; a `304 Not Modified` means nothing changed
and the syncer derives nothing. GitHub does not count `304` conditional
responses against the rate limit, so an unchanged repo is nearly free — this is
the main lever keeping GitHub traffic low.
- **Global rate limit.** Every GitHub call (releases list + each ranged asset
header GET) passes through a single token-bucket limiter **shared across all
remotes**, so GitHub is never hammered. Configure a token (`password`) on the
remote for the higher authenticated rate limit (~5000/hr vs ~60/hr
unauthenticated).
- **Multi-replica coordination.** State is shared through the database. Before a
periodic scan a replica must atomically claim a per-remote lease
(`github_rpm_sync_state`: `last_synced_at`, `etag`, `sync_lease_owner`,
`sync_lease_expires`); only the winner scans. This bounds total GitHub load to
~once per `mutable_ttl` regardless of replica count, and the shared `etag`
lets any replica issue the conditional request.
```hcl
# Backend that serves the actual .rpm bytes from github.com.
resource "artifactapi_remote_generic" "github" {
name = "github"
base_url = "https://github.com"
patterns = [
"acme/tools/releases/download/.*\\.rpm$", # allowlist the repo's release assets
]
}
resource "artifactapi_remote_github_rpm" "acme-tools" {
name = "acme-tools"
base_url = "https://api.github.com/repos/acme/tools" # the releases API root
releases_remote = "github" # backend for downloads
mutable_ttl = 3600 # release re-scan interval
# Optional: restrict which release assets become packages (regex on filename).
patterns = [".*\\.x86_64\\.rpm$", ".*\\.noarch\\.rpm$"]
# Optional: a token for private repos / higher API rate limits.
# password = "ghp_..."
}
```
`dnf` config: `baseurl=https://artifactapi.example/api/v1/remote/acme-tools`.
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
from the synthesized metadata.
### GitHub authentication
Anonymous GitHub is capped at **60 requests/hour** and cannot read private
repositories. Configure a **server-level GitHub credential** to raise the ceiling
to roughly **5000 requests/hour** and to read private-repo release assets. The
credential is a process-wide machine identity applied by default to *every*
outbound GitHub request — the releases scan, the ranged asset-header fetches, and
the generic-github byte proxy that streams private release assets.
The credential is read from the environment (deliver it from a Vault or
Kubernetes secret). It is **never** stored per-remote in the database, **never**
returned by any API, and **never** logged. Configure **exactly one** mode.
**Precedence.** A remote's own `username`/`password` credential still wins for
that remote's requests; the server credential is the default for everything else.
With no credential configured at all, requests stay anonymous (current behavior).
Partial configuration (e.g. an App id with no private key) is a **startup error**
— artifactapi fails closed rather than silently falling back to anonymous.
Both modes share the syncer's single global rate limiter, so a token simply
raises the effective GitHub ceiling; the default limiter settings stay safe.
#### Mode 1 — Personal Access Token (minimum viable, recommended for free accounts)
Set `GITHUB_TOKEN`. It is sent as `Authorization: Bearer <token>`.
Recommended free-account setup — a **fine-grained PAT** scoped to just the target
repositories:
1. GitHub → *Settings → Developer settings → Personal access tokens →
Fine-grained tokens → Generate new token*.
2. Limit *Repository access* to the specific repo(s) serving releases.
3. Grant repository permissions **Contents: Read-only** and **Metadata:
Read-only** (Metadata is mandatory and auto-selected).
A classic PAT with the `repo` scope also works but is broader than necessary.
```bash
GITHUB_TOKEN=github_pat_xxxxxxxx
```
#### Mode 2 — GitHub App installation token (proper machine identity)
A GitHub App is not tied to a personal account and can be created and installed on
free personal repos. artifactapi mints a short-lived RS256 **JWT** from the app
private key, exchanges it at `POST /app/installations/{id}/access_tokens` for a
~1-hour **installation access token**, caches that token, and refreshes it a few
minutes before expiry (thread-safe, single-flighted).
1. GitHub → *Settings → Developer settings → GitHub Apps → New GitHub App*.
2. Under *Permissions → Repository permissions* grant **Contents: Read-only**
(Metadata: Read-only is implied).
3. Generate a **private key** (downloads a PEM) and note the **App ID**.
4. *Install* the App on the account and select the target repositories, then read
the **Installation ID** from the installation URL
(`.../settings/installations/<installation-id>`).
```bash
GITHUB_APP_ID=123456
GITHUB_APP_INSTALLATION_ID=7654321
GITHUB_APP_PRIVATE_KEY_PATH=/etc/artifactapi/github-app.pem
# or inline PEM (e.g. mounted from a secret):
# GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
```
## Terraform
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
@@ -89,6 +230,54 @@ resource "artifactapi_virtual" "helm" {
Provider: [terraform-provider-artifactapi](../terraform-provider-artifactapi)
### Serving providers as a registry
A local `terraform` repo is a real provider registry: upload
`terraform-provider-{type}_{version}_{os}_{arch}.zip` files under
`{namespace}/{type}/`, and Terraform installs them from a bare source address —
no `.terraformrc` mirror config:
```hcl
terraform {
required_providers {
artifactapi = {
source = "artifactapi.k8s.syd1.au.unkin.net/<repo>/<type>"
version = "0.1.2"
}
}
}
```
The Terraform *namespace* segment is the artifactapi repo name; the provider is
matched by *type*. The registry serves service discovery
(`/.well-known/terraform.json`), the `providers.v1` version/download endpoints,
and a GPG-signed `SHA256SUMS` per the provider registry protocol.
Signing needs a GPG key. By default artifactapi generates one on first start and
stores it in the database (`signing_keys` table), so every replica shares it and
there's nothing to provision. To bring your own key instead, point
`TF_SIGNING_KEY_PATH` at an armored private key (optionally
`TF_SIGNING_KEY_PASSPHRASE`), which takes precedence over the generated one.
`TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
### Local docker registry
A local `docker` repo is a real container registry, not a mirror: it serves the
Docker Registry HTTP API V2 for both push and pull, so any client (`docker`,
`podman`, `skopeo`, `buildah`) can use it directly.
```sh
docker tag myapp:latest artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
docker push artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
docker pull artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
```
The first path segment after `/v2/` is the artifactapi repo name; the remainder
is the image name. Blobs and manifests are stored through the shared
content-addressable store (deduplicated by digest, reaped by GC once
unreferenced); tags are mutable references and re-pushing a tag moves it. Blob
uploads support both the monolithic and chunked (`POST`/`PATCH`/`PUT`) flows.
## Access Control
| Field | Default | Behaviour |
@@ -149,6 +338,15 @@ S3 client supports MinIO, Ceph RGW, and AWS S3 (via minio-go).
| `MINIO_BUCKET` | `artifacts` | S3 bucket |
| `MINIO_SECURE` | `false` | Use HTTPS for S3 |
| `MINIO_REGION` | | S3 region (AWS) |
| `GITHUB_SYNC_RATE` | `1` | `github_rpm` syncer global GitHub request rate (req/s), shared across all remotes. `1`/s = 3600/hr, under an authenticated token's ~5000/hr; unauthenticated (~60/hr) relies on ETag/304 |
| `GITHUB_SYNC_BURST` | `5` | Token-bucket burst for the shared limiter |
| `GITHUB_SYNC_WORKERS` | `3` | Concurrent `github_rpm` scan workers |
| `GITHUB_SYNC_POLL_INTERVAL` | `60` | Base scheduler tick in seconds; per-remote cadence is its `mutable_ttl`, enforced by the DB lease |
| `GITHUB_TOKEN` | | Server-level GitHub PAT (fine-grained or classic), sent as `Authorization: Bearer`. Applies to every GitHub request; per-remote creds override it. See [GitHub authentication](#github-authentication) |
| `GITHUB_APP_ID` | | GitHub App id (App auth mode; mutually exclusive with `GITHUB_TOKEN`) |
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
## Development
+177
View File
@@ -0,0 +1,177 @@
//go:build dockere2e
package e2edocker
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"fmt"
"net/http"
"strings"
"testing"
)
func digestOf(b []byte) string {
sum := sha256.Sum256(b)
return "sha256:" + hex.EncodeToString(sum[:])
}
// pushBlobMonolithic uploads a blob with POST (open session) then PUT?digest
// (whole body) — the monolithic-after-POST flow.
func pushBlobMonolithic(t *testing.T, repo, image string, blob []byte) {
t.Helper()
dgst := digestOf(blob)
resp, body := doRequest(t, http.MethodPost, api("/v2/"+repo+"/"+image+"/blobs/uploads/"), nil, "")
if resp.StatusCode != http.StatusAccepted {
t.Fatalf("start upload: status %d: %s", resp.StatusCode, body)
}
loc := resp.Header.Get("Location")
if loc == "" {
t.Fatalf("start upload: no Location header")
}
resp, body = doRequest(t, http.MethodPut, baseURL()+loc+"?digest="+dgst, blob, "application/octet-stream")
if resp.StatusCode != http.StatusCreated {
t.Fatalf("finish upload: status %d: %s", resp.StatusCode, body)
}
if got := resp.Header.Get("Docker-Content-Digest"); got != dgst {
t.Fatalf("finish upload: digest mismatch: got %q want %q", got, dgst)
}
}
// pushBlobChunked uploads a blob with POST then PATCH (body) then PUT?digest
// (empty) — the chunked flow a real docker daemon uses.
func pushBlobChunked(t *testing.T, repo, image string, blob []byte) {
t.Helper()
dgst := digestOf(blob)
resp, body := doRequest(t, http.MethodPost, api("/v2/"+repo+"/"+image+"/blobs/uploads/"), nil, "")
if resp.StatusCode != http.StatusAccepted {
t.Fatalf("start upload: status %d: %s", resp.StatusCode, body)
}
loc := resp.Header.Get("Location")
resp, body = doRequest(t, http.MethodPatch, baseURL()+loc, blob, "application/octet-stream")
if resp.StatusCode != http.StatusAccepted {
t.Fatalf("patch upload: status %d: %s", resp.StatusCode, body)
}
if got := resp.Header.Get("Range"); got != fmt.Sprintf("0-%d", len(blob)-1) {
t.Fatalf("patch upload: unexpected Range %q", got)
}
loc = resp.Header.Get("Location")
resp, body = doRequest(t, http.MethodPut, baseURL()+loc+"?digest="+dgst, nil, "")
if resp.StatusCode != http.StatusCreated {
t.Fatalf("finish upload: status %d: %s", resp.StatusCode, body)
}
}
// TestLocalDockerPushPull exercises a full container push and pull against a
// local docker repo using the Docker Registry HTTP API V2, the way a docker
// client would: upload the config and layer blobs, push the manifest under a
// tag, then pull the manifest and blobs back byte-identically.
func TestLocalDockerPushPull(t *testing.T) {
createRepo(t, `{"name":"docker-internal","package_type":"docker","repo_type":"local"}`)
defer deleteRepo(t, "docker-internal")
const image = "team/app"
const tag = "v1.0.0"
// /v2/ version check.
resp, _ := doRequest(t, http.MethodGet, api("/v2/"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("/v2/ ping: status %d", resp.StatusCode)
}
config := []byte(`{"architecture":"amd64","os":"linux","config":{},"rootfs":{"type":"layers","diff_ids":["sha256:0000000000000000000000000000000000000000000000000000000000000000"]}}`)
layer := bytes.Repeat([]byte("artifactapi-layer-data-"), 4096) // ~90 KB opaque layer
configDigest := digestOf(config)
layerDigest := digestOf(layer)
// A brand-new blob should be absent (this is the client's mount check).
resp, _ = doRequest(t, http.MethodHead, api("/v2/"+"docker-internal/"+image+"/blobs/"+configDigest), nil, "")
if resp.StatusCode != http.StatusNotFound {
t.Fatalf("pre-push blob HEAD: expected 404, got %d", resp.StatusCode)
}
pushBlobMonolithic(t, "docker-internal", image, config)
pushBlobChunked(t, "docker-internal", image, layer)
manifest := []byte(fmt.Sprintf(`{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":%d,"digest":%q},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":%d,"digest":%q}]}`,
len(config), configDigest, len(layer), layerDigest))
manifestDigest := digestOf(manifest)
manifestType := "application/vnd.docker.distribution.manifest.v2+json"
resp, body := doRequest(t, http.MethodPut, api("/v2/docker-internal/"+image+"/manifests/"+tag), manifest, manifestType)
if resp.StatusCode != http.StatusCreated {
t.Fatalf("push manifest: status %d: %s", resp.StatusCode, body)
}
if got := resp.Header.Get("Docker-Content-Digest"); got != manifestDigest {
t.Fatalf("push manifest: digest %q want %q", got, manifestDigest)
}
// --- pull back ---
// Manifest by tag.
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/manifests/"+tag), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("pull manifest by tag: status %d: %s", resp.StatusCode, body)
}
if !bytes.Equal(body, manifest) {
t.Fatalf("pulled manifest bytes differ from pushed")
}
if ct := resp.Header.Get("Content-Type"); ct != manifestType {
t.Fatalf("pulled manifest content-type %q want %q", ct, manifestType)
}
if got := resp.Header.Get("Docker-Content-Digest"); got != manifestDigest {
t.Fatalf("pulled manifest digest %q want %q", got, manifestDigest)
}
// Manifest by digest.
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/manifests/"+manifestDigest), nil, "")
if resp.StatusCode != http.StatusOK || !bytes.Equal(body, manifest) {
t.Fatalf("pull manifest by digest: status %d, equal=%v", resp.StatusCode, bytes.Equal(body, manifest))
}
// Blobs by digest.
for _, tc := range []struct {
name string
digest string
want []byte
}{
{"config", configDigest, config},
{"layer", layerDigest, layer},
} {
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/blobs/"+tc.digest), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("pull %s blob: status %d", tc.name, resp.StatusCode)
}
if !bytes.Equal(body, tc.want) {
t.Fatalf("pulled %s blob bytes differ", tc.name)
}
if got := resp.Header.Get("Docker-Content-Digest"); got != tc.digest {
t.Fatalf("pulled %s blob digest %q want %q", tc.name, got, tc.digest)
}
}
// tags/list reflects the pushed tag.
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/tags/list"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("tags/list: status %d: %s", resp.StatusCode, body)
}
if !strings.Contains(string(body), `"`+tag+`"`) {
t.Fatalf("tags/list missing tag %q: %s", tag, body)
}
if !strings.Contains(string(body), `"docker-internal/`+image+`"`) {
t.Fatalf("tags/list wrong repository name: %s", body)
}
// A now-present blob HEAD should succeed (client would skip re-upload).
resp, _ = doRequest(t, http.MethodHead, api("/v2/docker-internal/"+image+"/blobs/"+layerDigest), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("post-push blob HEAD: expected 200, got %d", resp.StatusCode)
}
}
+45
View File
@@ -8,6 +8,8 @@ import (
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
func uploadFile(t *testing.T, repo, filePath string, body []byte, contentType string) {
@@ -91,3 +93,46 @@ func TestLocalRPMRepodata(t *testing.T) {
t.Fatalf("repomd.xml not a valid repodata document: %s", s)
}
}
// TestLocalDebRepo uploads a .deb and validates that the flat apt index
// (Packages / Release) is generated automatically from the parsed control
// stanza (the deb-local analog of rpm repodata generation).
func TestLocalDebRepo(t *testing.T) {
createRepo(t, `{"name":"local-deb","package_type":"deb","repo_type":"local"}`)
defer deleteRepo(t, "local-deb")
deb := testsupport.MinimalDeb("e2e-testpkg", "1.0.0", "amd64")
uploadFile(t, "local-deb", "e2e-testpkg_1.0.0_amd64.deb", deb, "application/vnd.debian.binary-package")
// The index is generated asynchronously after upload; poll for it.
resp, body := getEventually(t, api("/api/v1/local/local-deb/Packages"), 15*time.Second)
if resp.StatusCode != http.StatusOK {
t.Fatalf("Packages: status %d: %s", resp.StatusCode, body)
}
pkgs := string(body)
for _, want := range []string{"Package: e2e-testpkg", "Version: 1.0.0", "Architecture: amd64", "Filename: pool/e2e-testpkg_1.0.0_amd64.deb", "SHA256:"} {
if !strings.Contains(pkgs, want) {
t.Fatalf("Packages missing %q:\n%s", want, pkgs)
}
}
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-deb/Release"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("Release: status %d: %s", resp.StatusCode, body)
}
rel := string(body)
for _, want := range []string{"Architectures: amd64", "SHA256:", "Packages"} {
if !strings.Contains(rel, want) {
t.Fatalf("Release missing %q:\n%s", want, rel)
}
}
// The .deb downloads back byte-identical from its pool path.
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-deb/pool/e2e-testpkg_1.0.0_amd64.deb"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("download deb: status %d: %s", resp.StatusCode, body)
}
if !bytes.Equal(body, deb) {
t.Fatalf("deb content mismatch")
}
}
+5 -3
View File
@@ -7,12 +7,17 @@ require (
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/go-chi/chi/v5 v5.3.0
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.10.0
github.com/klauspost/compress v1.19.2
github.com/minio/minio-go/v7 v7.2.0
github.com/redis/go-redis/v9 v9.20.0
github.com/testcontainers/testcontainers-go v0.42.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
github.com/testcontainers/testcontainers-go/modules/redis v0.42.0
github.com/ulikunitz/xz v0.5.16
golang.org/x/crypto v0.51.0
golang.org/x/time v0.15.0
gopkg.in/yaml.v3 v3.0.1
)
@@ -45,11 +50,9 @@ require (
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.18.6 // indirect
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
@@ -96,7 +99,6 @@ require (
go.opentelemetry.io/otel/trace v1.41.0 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/crypto v0.51.0 // indirect
golang.org/x/net v0.53.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.44.0 // indirect
+6 -2
View File
@@ -85,8 +85,8 @@ github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
@@ -189,6 +189,8 @@ github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYI
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
github.com/ulikunitz/xz v0.5.16 h1:ld6NyySjx5lowVKwJvMRLnW5nxKX/xnpSiFYZ/Lxur0=
github.com/ulikunitz/xz v0.5.16/go.mod h1:H9Rt/W6/Qj27PGauhQc6nfCDy7vHpzsOThBSaYDoEhw=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
@@ -234,6 +236,8 @@ golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
+301
View File
@@ -0,0 +1,301 @@
// Package terraform serves local terraform repos as a real Terraform provider
// registry: service discovery, version listing, and GPG-signed downloads, so
// `terraform init` installs from a bare source address with no client config.
package terraform
import (
"encoding/json"
"fmt"
"net/http"
"path"
"sort"
"strings"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
tfprov "git.unkin.net/unkin/artifactapi/internal/provider/terraform"
"git.unkin.net/unkin/artifactapi/internal/tfsign"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// ProvidersV1Path is the base the service-discovery document advertises (Terraform
// appends "{namespace}/{type}/versions" etc). MountPath is the same prefix without
// the trailing slash, for chi.Mount.
const (
ProvidersV1Path = "/terraform/v1/providers/"
MountPath = "/terraform/v1/providers"
)
type Handler struct {
db *database.DB
signer *tfsign.Signer
protocols []string
}
func NewHandler(db *database.DB, signer *tfsign.Signer, protocols string) *Handler {
var protos []string
for _, p := range strings.Split(protocols, ",") {
if p = strings.TrimSpace(p); p != "" {
protos = append(protos, p)
}
}
if len(protos) == 0 {
protos = []string{"5.0", "6.0"}
}
return &Handler{db: db, signer: signer, protocols: protos}
}
// Enabled reports whether a signing key is configured. Without one the registry
// cannot produce the signed SHA256SUMS the protocol requires, so it stays off.
func (h *Handler) Enabled() bool { return h.signer != nil }
func (h *Handler) Routes() chi.Router {
r := chi.NewRouter()
r.Get("/{namespace}/{type}/versions", h.versions)
r.Get("/{namespace}/{type}/{version}/download/{os}/{arch}", h.download)
r.Get("/{namespace}/{type}/{version}/sha256sums", h.sha256sums)
r.Get("/{namespace}/{type}/{version}/sha256sums.sig", h.sha256sumsSig)
return r
}
// ServiceDiscovery answers /.well-known/terraform.json, pointing Terraform at the
// providers.v1 protocol base.
func (h *Handler) ServiceDiscovery(w http.ResponseWriter, r *http.Request) {
if !h.Enabled() {
http.NotFound(w, r)
return
}
writeJSON(w, map[string]string{"providers.v1": ProvidersV1Path})
}
// providerFile is one resolved platform artifact within a repo.
type providerFile struct {
version string
os string
arch string
filePath string // path within the repo, e.g. unkin/artifactapi/...zip
sha256 string // hex, no "sha256:" prefix
}
// resolve finds every provider zip of the given type in the repo (namespace).
// The Terraform source namespace maps to the artifactapi repo name; the provider
// is matched by type across whatever in-repo folder it was uploaded under.
func (h *Handler) resolve(r *http.Request, namespace, typeName string) ([]providerFile, error) {
remote, err := h.db.GetRemote(r.Context(), namespace)
if err != nil || remote.PackageType != models.PackageTerraform {
return nil, nil
}
rows, err := h.db.ListLocalFiles(r.Context(), namespace, 10000, 0)
if err != nil {
return nil, err
}
var out []providerFile
for _, row := range rows {
parsed := tfprov.ParseProviderZip(path.Base(row.FilePath))
if !parsed.Ok || parsed.Type != typeName {
continue
}
out = append(out, providerFile{
version: parsed.Version,
os: parsed.OS,
arch: parsed.Arch,
filePath: row.FilePath,
sha256: strings.TrimPrefix(row.ContentHash, "sha256:"),
})
}
return out, nil
}
func (h *Handler) versions(w http.ResponseWriter, r *http.Request) {
if !h.Enabled() {
http.NotFound(w, r)
return
}
namespace := chi.URLParam(r, "namespace")
typeName := chi.URLParam(r, "type")
files, err := h.resolve(r, namespace, typeName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if len(files) == 0 {
http.NotFound(w, r)
return
}
// Group platforms by version, de-duplicated and stably ordered.
type platform struct {
OS string `json:"os"`
Arch string `json:"arch"`
}
platforms := map[string]map[string]platform{}
for _, f := range files {
if platforms[f.version] == nil {
platforms[f.version] = map[string]platform{}
}
platforms[f.version][f.os+"_"+f.arch] = platform{OS: f.os, Arch: f.arch}
}
type versionEntry struct {
Version string `json:"version"`
Protocols []string `json:"protocols"`
Platforms []platform `json:"platforms"`
}
out := struct {
Versions []versionEntry `json:"versions"`
}{}
for version, plats := range platforms {
entry := versionEntry{Version: version, Protocols: h.protocols}
for _, p := range plats {
entry.Platforms = append(entry.Platforms, p)
}
sort.Slice(entry.Platforms, func(i, j int) bool {
return entry.Platforms[i].OS+entry.Platforms[i].Arch < entry.Platforms[j].OS+entry.Platforms[j].Arch
})
out.Versions = append(out.Versions, entry)
}
sort.Slice(out.Versions, func(i, j int) bool { return out.Versions[i].Version < out.Versions[j].Version })
writeJSON(w, out)
}
func (h *Handler) download(w http.ResponseWriter, r *http.Request) {
if !h.Enabled() {
http.NotFound(w, r)
return
}
namespace := chi.URLParam(r, "namespace")
typeName := chi.URLParam(r, "type")
version := chi.URLParam(r, "version")
osName := chi.URLParam(r, "os")
arch := chi.URLParam(r, "arch")
files, err := h.resolve(r, namespace, typeName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
var match *providerFile
for i := range files {
if files[i].version == version && files[i].os == osName && files[i].arch == arch {
match = &files[i]
break
}
}
if match == nil {
http.NotFound(w, r)
return
}
base := baseURL(r)
verBase := fmt.Sprintf("%s%s/%s/%s", base+ProvidersV1Path, namespace, typeName, version)
type gpgKey struct {
KeyID string `json:"key_id"`
ASCIIArmor string `json:"ascii_armor"`
}
resp := struct {
Protocols []string `json:"protocols"`
OS string `json:"os"`
Arch string `json:"arch"`
Filename string `json:"filename"`
DownloadURL string `json:"download_url"`
SHASumsURL string `json:"shasums_url"`
SHASumsSignatureURL string `json:"shasums_signature_url"`
SHASum string `json:"shasum"`
SigningKeys struct {
GPGPublicKeys []gpgKey `json:"gpg_public_keys"`
} `json:"signing_keys"`
}{
Protocols: h.protocols,
OS: match.os,
Arch: match.arch,
Filename: path.Base(match.filePath),
DownloadURL: fmt.Sprintf("%s/api/v1/local/%s/%s", base, namespace, match.filePath),
SHASumsURL: verBase + "/sha256sums",
SHASumsSignatureURL: verBase + "/sha256sums.sig",
SHASum: match.sha256,
}
resp.SigningKeys.GPGPublicKeys = []gpgKey{{
KeyID: h.signer.KeyID(),
ASCIIArmor: h.signer.PublicKeyArmor(),
}}
writeJSON(w, resp)
}
func (h *Handler) sha256sums(w http.ResponseWriter, r *http.Request) {
sums, ok := h.buildSums(w, r)
if !ok {
return
}
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.Write(sums)
}
func (h *Handler) sha256sumsSig(w http.ResponseWriter, r *http.Request) {
sums, ok := h.buildSums(w, r)
if !ok {
return
}
sig, err := h.signer.Sign(sums)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/octet-stream")
w.Write(sig)
}
// buildSums renders the SHA256SUMS body for one version: one "<hex> <filename>"
// line per platform zip, sorted by filename so the signed bytes are stable.
func (h *Handler) buildSums(w http.ResponseWriter, r *http.Request) ([]byte, bool) {
if !h.Enabled() {
http.NotFound(w, r)
return nil, false
}
namespace := chi.URLParam(r, "namespace")
typeName := chi.URLParam(r, "type")
version := chi.URLParam(r, "version")
files, err := h.resolve(r, namespace, typeName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return nil, false
}
var lines []string
for _, f := range files {
if f.version != version {
continue
}
lines = append(lines, fmt.Sprintf("%s %s", f.sha256, path.Base(f.filePath)))
}
if len(lines) == 0 {
http.NotFound(w, r)
return nil, false
}
sort.Strings(lines)
return []byte(strings.Join(lines, "\n") + "\n"), true
}
func writeJSON(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(v)
}
func baseURL(r *http.Request) string {
scheme := "http"
if r.TLS != nil {
scheme = "https"
}
if fwd := r.Header.Get("X-Forwarded-Proto"); fwd != "" {
scheme = fwd
}
return scheme + "://" + r.Host
}
+186
View File
@@ -0,0 +1,186 @@
package terraform
import (
"bytes"
"context"
"encoding/json"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"github.com/go-chi/chi/v5"
"golang.org/x/crypto/openpgp"
"golang.org/x/crypto/openpgp/armor"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/internal/tfsign"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
var testDSN string
func TestMain(m *testing.M) {
ctx := context.Background()
dsn, terminate, err := testsupport.StartPostgres(ctx)
if err != nil {
os.Exit(m.Run())
}
testDSN = dsn
code := m.Run()
terminate()
os.Exit(code)
}
// testSigner writes a throwaway armored key and loads it.
func testSigner(t *testing.T) *tfsign.Signer {
t.Helper()
e, err := openpgp.NewEntity("artifactapi test", "tf", "tf@example.com", nil)
if err != nil {
t.Fatal(err)
}
var buf bytes.Buffer
w, _ := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
if err := e.SerializePrivate(w, nil); err != nil {
t.Fatal(err)
}
w.Close()
p := filepath.Join(t.TempDir(), "private-key.asc")
if err := os.WriteFile(p, buf.Bytes(), 0o600); err != nil {
t.Fatal(err)
}
s, err := tfsign.Load(p, "")
if err != nil {
t.Fatal(err)
}
return s
}
func TestProviderRegistryFlow(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
ctx := context.Background()
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
const repo = "tf-reg" // Terraform namespace == repo name
const filePath = "unkin/artifactapi/terraform-provider-artifactapi_1.2.3_linux_amd64.zip"
const hash = "sha256:983cdb25cb7b976538e4334d26e52dee5f44749b9be1500c760cf5cf66be659b"
const wantSha = "983cdb25cb7b976538e4334d26e52dee5f44749b9be1500c760cf5cf66be659b"
if err := db.CreateRemote(ctx, &models.Remote{Name: repo, PackageType: models.PackageTerraform, RepoType: models.RepoTypeLocal}); err != nil {
t.Fatal(err)
}
if err := db.UpsertBlob(ctx, hash, "blobs/98/3c", 6381007, "application/zip"); err != nil {
t.Fatal(err)
}
if err := db.CreateLocalFile(ctx, repo, filePath, hash); err != nil {
t.Fatal(err)
}
signer := testSigner(t)
h := NewHandler(db, signer, "5.0,6.0")
router := chi.NewRouter()
router.Get("/.well-known/terraform.json", h.ServiceDiscovery)
router.Mount(MountPath, h.Routes())
get := func(p string) *httptest.ResponseRecorder {
req := httptest.NewRequest("GET", p, nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
return w
}
// Service discovery.
w := get("/.well-known/terraform.json")
if w.Code != 200 {
t.Fatalf("discovery = %d", w.Code)
}
var disc map[string]string
json.Unmarshal(w.Body.Bytes(), &disc)
if disc["providers.v1"] != ProvidersV1Path {
t.Errorf("providers.v1 = %q", disc["providers.v1"])
}
// Versions.
w = get("/terraform/v1/providers/tf-reg/artifactapi/versions")
if w.Code != 200 {
t.Fatalf("versions = %d %s", w.Code, w.Body)
}
var vresp struct {
Versions []struct {
Version string `json:"version"`
Protocols []string `json:"protocols"`
Platforms []map[string]string `json:"platforms"`
} `json:"versions"`
}
json.Unmarshal(w.Body.Bytes(), &vresp)
if len(vresp.Versions) != 1 || vresp.Versions[0].Version != "1.2.3" {
t.Fatalf("unexpected versions: %+v", vresp)
}
if len(vresp.Versions[0].Platforms) != 1 || vresp.Versions[0].Platforms[0]["os"] != "linux" {
t.Fatalf("unexpected platforms: %+v", vresp.Versions[0].Platforms)
}
// Download.
w = get("/terraform/v1/providers/tf-reg/artifactapi/1.2.3/download/linux/amd64")
if w.Code != 200 {
t.Fatalf("download = %d %s", w.Code, w.Body)
}
var dl struct {
Filename string `json:"filename"`
DownloadURL string `json:"download_url"`
SHASumsURL string `json:"shasums_url"`
SHASumsSignatureURL string `json:"shasums_signature_url"`
SHASum string `json:"shasum"`
SigningKeys struct {
GPGPublicKeys []struct {
KeyID string `json:"key_id"`
ASCIIArmor string `json:"ascii_armor"`
} `json:"gpg_public_keys"`
} `json:"signing_keys"`
}
json.Unmarshal(w.Body.Bytes(), &dl)
if dl.SHASum != wantSha {
t.Errorf("shasum = %q", dl.SHASum)
}
wantURL := "http://example.com/api/v1/local/tf-reg/" + filePath
if dl.DownloadURL != wantURL {
t.Errorf("download_url = %q, want %q", dl.DownloadURL, wantURL)
}
if len(dl.SigningKeys.GPGPublicKeys) != 1 || dl.SigningKeys.GPGPublicKeys[0].KeyID != signer.KeyID() {
t.Errorf("signing key mismatch: %+v", dl.SigningKeys)
}
// SHA256SUMS + signature verify against the advertised key.
sums := get("/terraform/v1/providers/tf-reg/artifactapi/1.2.3/sha256sums")
wantLine := wantSha + " terraform-provider-artifactapi_1.2.3_linux_amd64.zip\n"
if sums.Body.String() != wantLine {
t.Errorf("sha256sums = %q, want %q", sums.Body.String(), wantLine)
}
sig := get("/terraform/v1/providers/tf-reg/artifactapi/1.2.3/sha256sums.sig")
keyring, err := openpgp.ReadArmoredKeyRing(bytes.NewReader([]byte(dl.SigningKeys.GPGPublicKeys[0].ASCIIArmor)))
if err != nil {
t.Fatal(err)
}
if _, err := openpgp.CheckDetachedSignature(keyring, bytes.NewReader(sums.Body.Bytes()), bytes.NewReader(sig.Body.Bytes())); err != nil {
t.Errorf("sha256sums.sig did not verify: %v", err)
}
}
func TestRegistryDisabledWithoutSigner(t *testing.T) {
h := NewHandler(nil, nil, "")
router := chi.NewRouter()
router.Get("/.well-known/terraform.json", h.ServiceDiscovery)
req := httptest.NewRequest("GET", "/.well-known/terraform.json", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 404 {
t.Errorf("disabled discovery = %d, want 404", w.Code)
}
}
+486
View File
@@ -0,0 +1,486 @@
package v1
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"os"
"sort"
"strings"
"github.com/go-chi/chi/v5"
"github.com/google/uuid"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// This file implements the write half of the Docker Registry HTTP API V2 for
// *local* docker repositories, so a `docker push` / `docker pull` against
// artifactapi treats a local docker repo as a genuine registry (matching the
// project's "local repos are the real thing" principle) rather than a mirror.
//
// Storage reuses the existing content-addressable primitives:
// - blob and manifest bytes are stored via the CAS (deduplicated by sha256)
// - a local_files row per (repo, "<image>/blobs/<digest>") and
// (repo, "<image>/manifests/<ref>") keeps the blob referenced so the GC
// does not reap it, and lets pulls resolve a reference back to a blob.
// Tags are mutable references (UpsertLocalFile); digests and blobs are
// immutable (CreateLocalFile, tolerating an already-exists on re-push).
const dockerAPIVersionHeader = "registry/2.0"
// Chunked blob uploads are staged in object storage under uploads/<uuid> rather
// than in process memory, so the POST / PATCH / PUT of a single push can each be
// served by a different replica (the API runs with minReplicas>1 and no session
// affinity). The upload UUID travels in the Location URL handed back to the
// client, so any replica reconstructs the staging key with no shared in-process
// state. Abandoned stages are dropped by the GC's uploads sweep.
func uploadKey(id string) string { return "uploads/" + id }
var errUploadUnknown = errors.New("unknown upload")
// appendUpload appends a chunk to the staged upload object and returns the new
// total size. The staged bytes live entirely in object storage (download,
// append to a per-request temp file, re-upload), which keeps the session state
// replica-independent. Docker sends the whole layer in one PATCH, so this is a
// single append in the common case.
func (h *ProxyHandler) appendUpload(ctx context.Context, id string, chunk io.Reader) (int64, error) {
key := uploadKey(id)
reader, info, err := h.store.Download(ctx, key)
if err != nil {
return 0, errUploadUnknown
}
tmp, err := os.CreateTemp("", "docker-upload-*")
if err != nil {
reader.Close()
return 0, err
}
defer os.Remove(tmp.Name())
defer tmp.Close()
if _, err := io.Copy(tmp, reader); err != nil {
reader.Close()
return 0, err
}
reader.Close()
n, err := io.Copy(tmp, chunk)
if err != nil {
return 0, err
}
size := info.Size + n
if _, err := tmp.Seek(0, io.SeekStart); err != nil {
return 0, err
}
if err := h.store.Upload(ctx, key, tmp, size, "application/octet-stream"); err != nil {
return 0, err
}
return size, nil
}
// dockerReq is a parsed /v2/<remote>/<image>/... request. kind is one of
// "manifest", "blob", "upload", "tags".
type dockerReq struct {
image string
kind string
ref string // tag, digest, or upload uuid depending on kind
}
// parseDockerPath splits the chi "*" remainder (everything after the repo name)
// into the image name and the registry operation. The image name may itself
// contain slashes, so operations are located by their well-known infixes.
func parseDockerPath(rest string) (dockerReq, bool) {
rest = strings.TrimPrefix(rest, "/")
switch {
case strings.HasSuffix(rest, "/tags/list"):
return dockerReq{image: strings.TrimSuffix(rest, "/tags/list"), kind: "tags"}, true
case rest == "tags/list":
return dockerReq{}, false // no image
}
if i := strings.Index(rest, "/blobs/uploads"); i >= 0 {
image := rest[:i]
ref := strings.TrimPrefix(rest[i+len("/blobs/uploads"):], "/")
return dockerReq{image: image, kind: "upload", ref: ref}, image != ""
}
if i := strings.LastIndex(rest, "/manifests/"); i >= 0 {
return dockerReq{image: rest[:i], kind: "manifest", ref: rest[i+len("/manifests/"):]}, true
}
if i := strings.LastIndex(rest, "/blobs/"); i >= 0 {
return dockerReq{image: rest[:i], kind: "blob", ref: rest[i+len("/blobs/"):]}, true
}
return dockerReq{}, false
}
func isDigest(ref string) bool { return strings.HasPrefix(ref, "sha256:") }
// localDockerRemote returns the repo if name is a local docker repository.
func (h *ProxyHandler) localDockerRemote(r *http.Request, name string) (*models.Remote, bool) {
remote, err := h.db.GetRemote(r.Context(), name)
if err != nil {
return nil, false
}
return remote, remote.RepoType == models.RepoTypeLocal && remote.PackageType == models.PackageDocker
}
func dockerError(w http.ResponseWriter, status int, code, msg string) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(status)
fmt.Fprintf(w, `{"errors":[{"code":%q,"message":%q}]}`, code, msg)
}
// dockerGet dispatches a registry GET to the local handler for local docker
// repos and falls through to the upstream proxy for everything else.
func (h *ProxyHandler) dockerGet(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "remoteName")
if remote, ok := h.localDockerRemote(r, name); ok {
h.dockerLocalGet(w, r, remote, false)
return
}
h.handleProxy(w, r)
}
func (h *ProxyHandler) dockerHead(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "remoteName")
if remote, ok := h.localDockerRemote(r, name); ok {
h.dockerLocalGet(w, r, remote, true)
return
}
h.handleProxyHead(w, r)
}
func (h *ProxyHandler) dockerPost(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "remoteName")
remote, ok := h.localDockerRemote(r, name)
if !ok {
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "push is only supported for local docker repositories")
return
}
h.dockerStartUpload(w, r, remote)
}
func (h *ProxyHandler) dockerPatch(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "remoteName")
remote, ok := h.localDockerRemote(r, name)
if !ok {
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "push is only supported for local docker repositories")
return
}
h.dockerPatchUpload(w, r, remote)
}
func (h *ProxyHandler) dockerPut(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "remoteName")
remote, ok := h.localDockerRemote(r, name)
if !ok {
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "push is only supported for local docker repositories")
return
}
req, ok := parseDockerPath(chi.URLParam(r, "*"))
if !ok {
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
return
}
switch req.kind {
case "upload":
h.dockerFinishUpload(w, r, remote, req)
case "manifest":
h.dockerPutManifest(w, r, remote, req)
default:
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "PUT not supported for this path")
}
}
func (h *ProxyHandler) dockerDelete(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "remoteName")
remote, ok := h.localDockerRemote(r, name)
if !ok {
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "delete is only supported for local docker repositories")
return
}
req, ok := parseDockerPath(chi.URLParam(r, "*"))
if !ok {
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
return
}
// Cancel an in-progress upload: drop its staging object.
if req.kind == "upload" && req.ref != "" {
_ = h.store.Delete(r.Context(), uploadKey(req.ref))
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(http.StatusNoContent)
return
}
if req.kind != "manifest" && req.kind != "blob" {
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
return
}
filePath := req.image + "/" + req.kind + "s/" + req.ref
if err := h.db.DeleteLocalFile(r.Context(), remote.Name, filePath); err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(http.StatusAccepted)
}
// dockerLocalGet serves manifest / blob / tags-list reads for a local repo.
func (h *ProxyHandler) dockerLocalGet(w http.ResponseWriter, r *http.Request, remote *models.Remote, head bool) {
req, ok := parseDockerPath(chi.URLParam(r, "*"))
if !ok {
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
return
}
switch req.kind {
case "tags":
h.dockerTagsList(w, r, remote, req.image)
case "manifest":
h.dockerServeRef(w, r, remote, req.image+"/manifests/"+req.ref, head, true)
case "blob":
h.dockerServeRef(w, r, remote, req.image+"/blobs/"+req.ref, head, false)
default:
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
}
}
// dockerServeRef streams the blob backing a local_files path. isManifest
// controls only the default content type; the stored blob content type wins.
func (h *ProxyHandler) dockerServeRef(w http.ResponseWriter, r *http.Request, remote *models.Remote, filePath string, head, isManifest bool) {
file, err := h.db.GetLocalFile(r.Context(), remote.Name, filePath)
if err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if file == nil {
code := "BLOB_UNKNOWN"
if isManifest {
code = "MANIFEST_UNKNOWN"
}
dockerError(w, http.StatusNotFound, code, "not found")
return
}
s3Key := storage.BlobKey(file.ContentHash[len("sha256:"):])
reader, info, err := h.store.Download(r.Context(), s3Key)
if err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
defer reader.Close()
contentType := info.ContentType
if contentType == "" {
if isManifest {
contentType = "application/vnd.docker.distribution.manifest.v2+json"
} else {
contentType = "application/octet-stream"
}
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size))
w.Header().Set("Docker-Content-Digest", file.ContentHash)
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.Header().Set("X-Artifact-Source", "local")
if head {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusOK)
io.Copy(w, reader)
}
func (h *ProxyHandler) dockerTagsList(w http.ResponseWriter, r *http.Request, remote *models.Remote, image string) {
prefix := image + "/manifests/"
files, err := h.db.ListLocalFilesByPrefix(r.Context(), remote.Name, prefix)
if err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
tags := []string{}
for _, f := range files {
ref := strings.TrimPrefix(f.FilePath, prefix)
if ref == "" || isDigest(ref) {
continue
}
tags = append(tags, ref)
}
sort.Strings(tags)
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(http.StatusOK)
fmt.Fprintf(w, `{"name":%q,"tags":`, remote.Name+"/"+image)
writeJSONStringList(w, tags)
fmt.Fprint(w, "}")
}
func writeJSONStringList(w io.Writer, items []string) {
fmt.Fprint(w, "[")
for i, s := range items {
if i > 0 {
fmt.Fprint(w, ",")
}
fmt.Fprintf(w, "%q", s)
}
fmt.Fprint(w, "]")
}
// dockerStartUpload begins a blob upload. It honours a monolithic
// POST?digest=... (blob in the POST body) and otherwise opens a chunked
// session, returning its Location for the client's PATCH/PUT.
func (h *ProxyHandler) dockerStartUpload(w http.ResponseWriter, r *http.Request, remote *models.Remote) {
req, ok := parseDockerPath(chi.URLParam(r, "*"))
if !ok || req.kind != "upload" {
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
return
}
if digest := r.URL.Query().Get("digest"); digest != "" {
h.dockerCommitBlob(w, r, remote, req.image, digest, r.Body)
return
}
// Stage an empty object keyed by the upload UUID; PATCH/PUT append to it.
id := uuid.NewString()
if err := h.store.Upload(r.Context(), uploadKey(id), bytes.NewReader(nil), 0, "application/octet-stream"); err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
loc := fmt.Sprintf("/v2/%s/%s/blobs/uploads/%s", remote.Name, req.image, id)
w.Header().Set("Location", loc)
w.Header().Set("Docker-Upload-UUID", id)
w.Header().Set("Range", "0-0")
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(http.StatusAccepted)
}
func (h *ProxyHandler) dockerPatchUpload(w http.ResponseWriter, r *http.Request, remote *models.Remote) {
req, ok := parseDockerPath(chi.URLParam(r, "*"))
if !ok || req.kind != "upload" || req.ref == "" {
dockerError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "unknown upload")
return
}
size, err := h.appendUpload(r.Context(), req.ref, r.Body)
if err != nil {
if errors.Is(err, errUploadUnknown) {
dockerError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "unknown upload")
return
}
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
loc := fmt.Sprintf("/v2/%s/%s/blobs/uploads/%s", remote.Name, req.image, req.ref)
w.Header().Set("Location", loc)
w.Header().Set("Docker-Upload-UUID", req.ref)
w.Header().Set("Range", fmt.Sprintf("0-%d", size-1))
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(http.StatusAccepted)
}
// dockerFinishUpload completes a chunked upload: appends any final PUT body,
// stores the assembled blob, and verifies its digest.
func (h *ProxyHandler) dockerFinishUpload(w http.ResponseWriter, r *http.Request, remote *models.Remote, req dockerReq) {
digest := r.URL.Query().Get("digest")
if digest == "" {
dockerError(w, http.StatusBadRequest, "DIGEST_INVALID", "digest query parameter required")
return
}
if req.ref == "" {
// Monolithic PUT with no prior session: body is the whole blob.
h.dockerCommitBlob(w, r, remote, req.image, digest, r.Body)
return
}
key := uploadKey(req.ref)
reader, _, err := h.store.Download(r.Context(), key)
if err != nil {
dockerError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "unknown upload")
return
}
defer reader.Close()
// Drop the staging object once we're done, regardless of outcome; a fresh
// context so cleanup still runs if the client disconnects.
defer h.store.Delete(context.Background(), key)
// Stream the staged bytes plus any trailing PUT body through the CAS in one
// pass — no extra round trip to re-assemble.
combined := io.MultiReader(reader, r.Body)
h.dockerCommitBlob(w, r, remote, req.image, digest, combined)
}
// dockerCommitBlob stores blob bytes through the CAS, verifies the client's
// declared digest, and records the per-image local_files reference.
func (h *ProxyHandler) dockerCommitBlob(w http.ResponseWriter, r *http.Request, remote *models.Remote, image, digest string, body io.Reader) {
result, err := h.cas.Store(r.Context(), body, "application/octet-stream")
if err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", fmt.Sprintf("store failed: %v", err))
return
}
if result.ContentHash != digest {
dockerError(w, http.StatusBadRequest, "DIGEST_INVALID", fmt.Sprintf("digest mismatch: got %s, declared %s", result.ContentHash, digest))
return
}
if err := h.db.UpsertBlob(r.Context(), result.ContentHash, result.S3Key, result.SizeBytes, "application/octet-stream"); err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
if err := h.db.CreateLocalFile(r.Context(), remote.Name, image+"/blobs/"+digest, result.ContentHash); err != nil && !errors.Is(err, database.ErrAlreadyExists) {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
w.Header().Set("Location", fmt.Sprintf("/v2/%s/%s/blobs/%s", remote.Name, image, digest))
w.Header().Set("Docker-Content-Digest", digest)
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(http.StatusCreated)
}
// dockerPutManifest stores a manifest and points its reference (tag or digest)
// at it. Tags are mutable so a re-push moves the tag; digests are immutable.
func (h *ProxyHandler) dockerPutManifest(w http.ResponseWriter, r *http.Request, remote *models.Remote, req dockerReq) {
body, err := io.ReadAll(r.Body)
if err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
contentType := r.Header.Get("Content-Type")
if contentType == "" {
contentType = "application/vnd.docker.distribution.manifest.v2+json"
}
sum := sha256.Sum256(body)
digest := "sha256:" + hex.EncodeToString(sum[:])
result, err := h.cas.Store(r.Context(), strings.NewReader(string(body)), contentType)
if err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", fmt.Sprintf("store failed: %v", err))
return
}
if err := h.db.UpsertBlob(r.Context(), result.ContentHash, result.S3Key, result.SizeBytes, contentType); err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
// Always addressable by digest (immutable).
if err := h.db.CreateLocalFile(r.Context(), remote.Name, req.image+"/manifests/"+digest, result.ContentHash); err != nil && !errors.Is(err, database.ErrAlreadyExists) {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
// If pushed under a tag, (re)point the tag at this manifest.
if !isDigest(req.ref) {
if err := h.db.UpsertLocalFile(r.Context(), remote.Name, req.image+"/manifests/"+req.ref, result.ContentHash); err != nil {
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
return
}
}
slog.Info("local docker manifest pushed", "repo", remote.Name, "image", req.image, "ref", req.ref, "digest", digest)
w.Header().Set("Location", fmt.Sprintf("/v2/%s/%s/manifests/%s", remote.Name, req.image, req.ref))
w.Header().Set("Docker-Content-Digest", digest)
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
w.WriteHeader(http.StatusCreated)
}
+50
View File
@@ -0,0 +1,50 @@
package v1
import "testing"
func TestParseDockerPath(t *testing.T) {
tests := []struct {
name string
rest string
wantOK bool
wantImage string
wantKind string
wantRef string
}{
{"start upload trailing slash", "team/app/blobs/uploads/", true, "team/app", "upload", ""},
{"start upload no slash", "team/app/blobs/uploads", true, "team/app", "upload", ""},
{"patch upload with uuid", "team/app/blobs/uploads/abc-123", true, "team/app", "upload", "abc-123"},
{"single-segment image upload", "app/blobs/uploads/", true, "app", "upload", ""},
{"blob by digest", "team/app/blobs/sha256:deadbeef", true, "team/app", "blob", "sha256:deadbeef"},
{"manifest by tag", "team/app/manifests/v1.0.0", true, "team/app", "manifest", "v1.0.0"},
{"manifest by digest", "team/app/manifests/sha256:cafe", true, "team/app", "manifest", "sha256:cafe"},
{"tags list", "team/app/tags/list", true, "team/app", "tags", ""},
{"leading slash tolerated", "/team/app/manifests/latest", true, "team/app", "manifest", "latest"},
{"deep image name", "a/b/c/manifests/latest", true, "a/b/c", "manifest", "latest"},
{"unrecognised", "team/app/whatever", false, "", "", ""},
{"tags list without image", "tags/list", false, "", "", ""},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
got, ok := parseDockerPath(tc.rest)
if ok != tc.wantOK {
t.Fatalf("ok = %v, want %v", ok, tc.wantOK)
}
if !tc.wantOK {
return
}
if got.image != tc.wantImage || got.kind != tc.wantKind || got.ref != tc.wantRef {
t.Fatalf("got %+v, want image=%q kind=%q ref=%q", got, tc.wantImage, tc.wantKind, tc.wantRef)
}
})
}
}
func TestIsDigest(t *testing.T) {
if !isDigest("sha256:abc") {
t.Fatal("sha256: prefix should be a digest")
}
if isDigest("v1.0.0") {
t.Fatal("a tag is not a digest")
}
}
+28 -3
View File
@@ -23,10 +23,18 @@ type ProxyHandler struct {
db *database.DB
store *storage.S3
local *v2.LocalHandler
cas *storage.CAS
}
func NewProxyHandler(engine *proxy.Engine, virtualEngine *virtual.Engine, db *database.DB, store *storage.S3, local *v2.LocalHandler) *ProxyHandler {
return &ProxyHandler{engine: engine, virtualEngine: virtualEngine, db: db, store: store, local: local}
return &ProxyHandler{
engine: engine,
virtualEngine: virtualEngine,
db: db,
store: store,
local: local,
cas: storage.NewCAS(store),
}
}
func (h *ProxyHandler) Routes() chi.Router {
@@ -37,12 +45,20 @@ func (h *ProxyHandler) Routes() chi.Router {
return r
}
// DockerV2Routes mounts the Docker Registry HTTP API V2. Reads (GET/HEAD)
// dispatch to a local registry implementation for local docker repos and fall
// through to the upstream proxy otherwise; writes (POST/PATCH/PUT/DELETE) are
// only valid for local docker repos and drive push.
func (h *ProxyHandler) DockerV2Routes() chi.Router {
r := chi.NewRouter()
r.Get("/", h.handleDockerPing)
r.Head("/", h.handleDockerPing)
r.Get("/{remoteName}/*", h.handleProxy)
r.Head("/{remoteName}/*", h.handleProxyHead)
r.Get("/{remoteName}/*", h.dockerGet)
r.Head("/{remoteName}/*", h.dockerHead)
r.Post("/{remoteName}/*", h.dockerPost)
r.Patch("/{remoteName}/*", h.dockerPatch)
r.Put("/{remoteName}/*", h.dockerPut)
r.Delete("/{remoteName}/*", h.dockerDelete)
return r
}
@@ -67,6 +83,15 @@ func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
return
}
// Metadata-only remotes (e.g. github_rpm) synthesize their own responses and
// redirect package downloads to a backend remote instead of proxying bytes.
if rs, ok := prov.(provider.RemoteServer); ok {
proxyBaseURL := fmt.Sprintf("%s://%s", scheme(r), r.Host)
if rs.ServeRemote(w, r, *remote, path, proxyBaseURL, h.db) {
return
}
}
result, err := h.engine.Fetch(r.Context(), *remote, path, prov, r.Header)
if err != nil {
var proxyErr *proxy.ProxyError
+1 -1
View File
@@ -57,7 +57,7 @@ func do(t *testing.T, h http.Handler, method, path, body string) int {
}
func TestRemotesErrorPaths(t *testing.T) {
h := NewRemotesHandler(closedDB(t)).Routes()
h := NewRemotesHandler(closedDB(t), nil).Routes()
if c := do(t, h, "GET", "/", ""); c != 500 {
t.Errorf("list with dead db = %d, want 500", c)
}
+18 -4
View File
@@ -11,12 +11,21 @@ import (
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type RemotesHandler struct {
db *database.DB
// Primer enqueues a background metadata prime for a newly created remote so the
// create call never blocks on a derive. *rpm.Syncer and *deb.Syncer satisfy it.
type Primer interface {
EnqueuePrime(remote models.Remote)
}
func NewRemotesHandler(db *database.DB) *RemotesHandler {
return &RemotesHandler{db: db}
type RemotesHandler struct {
db *database.DB
primers map[models.PackageType]Primer
}
// NewRemotesHandler wires the handler to the per-type metadata primers. primers
// may be nil; a package type with no registered primer simply skips priming.
func NewRemotesHandler(db *database.DB, primers map[models.PackageType]Primer) *RemotesHandler {
return &RemotesHandler{db: db, primers: primers}
}
func (h *RemotesHandler) Routes() chi.Router {
@@ -77,6 +86,11 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Prime a metadata-only remote (github_rpm/github_deb) in the background so
// its first index request is served from cache instead of a cold derive.
if primer := h.primers[remote.PackageType]; primer != nil {
primer.EnqueuePrime(remote)
}
writeJSON(w, http.StatusCreated, remote)
}
+64
View File
@@ -24,6 +24,38 @@ type Config struct {
S3Bucket string
S3Secure bool
S3Region string
// Terraform provider registry signing. When TFSigningKeyPath points at a
// readable armored GPG private key, artifactapi serves local terraform
// repos as a real provider registry (service discovery + signed
// SHA256SUMS). Left empty, the registry endpoints stay disabled.
TFSigningKeyPath string
TFSigningKeyPassphrase string
TFProviderProtocols string
// github_rpm background syncer. The syncer keeps derived RPM metadata for
// every github_rpm remote fresh off the client request path, sharing a
// single global token-bucket limiter across all remotes so GitHub is never
// hammered. Defaults are conservative: 1 req/s (3600/hr) sits well under an
// authenticated token's 5000/hr. Unauthenticated remotes (60/hr) lean on
// ETag/304 — an unchanged repo costs nothing — so keep those repos small or
// configure a token.
GitHubSyncRatePerSec float64
GitHubSyncBurst int
GitHubSyncWorkers int
GitHubSyncPollInterval int
// Server-level GitHub machine credential, applied by default to every
// outbound GitHub request (releases scan, ranged asset fetches, and the
// generic-github byte proxy for private assets). Delivered via env/secret
// only — never stored per-remote, never returned by an API, never logged.
// Configure exactly one mode: a Personal Access Token, or a GitHub App
// (id + installation id + private key). Partial App config fails at startup.
GitHubToken string
GitHubAppID string
GitHubAppInstallationID string
GitHubAppPrivateKey string
GitHubAppPrivateKeyPath string
}
func (c *Config) DatabaseDSN() string {
@@ -41,6 +73,23 @@ func Load() (*Config, error) {
s3Secure, _ := strconv.ParseBool(getenv("MINIO_SECURE", "false"))
syncRate, err := strconv.ParseFloat(getenv("GITHUB_SYNC_RATE", "1"), 64)
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_RATE: %w", err)
}
syncBurst, err := strconv.Atoi(getenv("GITHUB_SYNC_BURST", "5"))
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_BURST: %w", err)
}
syncWorkers, err := strconv.Atoi(getenv("GITHUB_SYNC_WORKERS", "3"))
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_WORKERS: %w", err)
}
syncPoll, err := strconv.Atoi(getenv("GITHUB_SYNC_POLL_INTERVAL", "60"))
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_POLL_INTERVAL: %w", err)
}
cfg := &Config{
ListenAddr: getenv("LISTEN_ADDR", ":8000"),
@@ -59,6 +108,21 @@ func Load() (*Config, error) {
S3Bucket: getenv("MINIO_BUCKET", "artifacts"),
S3Secure: s3Secure,
S3Region: getenv("MINIO_REGION", ""),
TFSigningKeyPath: getenv("TF_SIGNING_KEY_PATH", ""),
TFSigningKeyPassphrase: getenv("TF_SIGNING_KEY_PASSPHRASE", ""),
TFProviderProtocols: getenv("TF_PROVIDER_PROTOCOLS", "5.0,6.0"),
GitHubSyncRatePerSec: syncRate,
GitHubSyncBurst: syncBurst,
GitHubSyncWorkers: syncWorkers,
GitHubSyncPollInterval: syncPoll,
GitHubToken: getenv("GITHUB_TOKEN", ""),
GitHubAppID: getenv("GITHUB_APP_ID", ""),
GitHubAppInstallationID: getenv("GITHUB_APP_INSTALLATION_ID", ""),
GitHubAppPrivateKey: getenv("GITHUB_APP_PRIVATE_KEY", ""),
GitHubAppPrivateKeyPath: getenv("GITHUB_APP_PRIVATE_KEY_PATH", ""),
}
return cfg, nil
+70
View File
@@ -0,0 +1,70 @@
package database
import (
"context"
"errors"
"time"
"github.com/jackc/pgx/v5"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// ListGitHubDebRemotes returns every github_deb remote so the syncer can sweep
// them on each poll tick.
func (db *DB) ListGitHubDebRemotes(ctx context.Context) ([]models.Remote, error) {
rows, err := db.Pool.Query(ctx, `SELECT `+remoteCols+` FROM remotes WHERE package_type = $1 ORDER BY name`, models.PackageGitHubDeb)
if err != nil {
return nil, err
}
defer rows.Close()
var remotes []models.Remote
for rows.Next() {
var r models.Remote
if err := scanRemote(rows, &r); err != nil {
return nil, err
}
remotes = append(remotes, r)
}
return remotes, rows.Err()
}
// ClaimGitHubDebSyncLease atomically claims the per-remote sync lease. It
// succeeds only when the remote is due (never synced, or synced longer than
// freshness ago) and no live lease is held by another replica. A zero freshness
// (prime scans) ignores the recency gate. The returned etag is the stored
// releases-list ETag, shared across replicas.
func (db *DB) ClaimGitHubDebSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
row := db.Pool.QueryRow(ctx, `
INSERT INTO github_deb_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
VALUES ($1, $2, now() + make_interval(secs => $4))
ON CONFLICT (remote_name) DO UPDATE
SET sync_lease_owner = $2,
sync_lease_expires = now() + make_interval(secs => $4)
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
RETURNING s.etag
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
var etag string
if err := row.Scan(&etag); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return false, "", nil
}
return false, "", err
}
return true, etag, nil
}
// ReleaseGitHubDebSyncLease records the completed scan and frees the lease. Only
// the owning replica may release; last_synced_at advances so the next poll waits
// a full freshness window, and etag is persisted for the next conditional request.
func (db *DB) ReleaseGitHubDebSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
_, err := db.Pool.Exec(ctx, `
UPDATE github_deb_sync_state
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
WHERE remote_name = $1 AND sync_lease_owner = $2
`, remoteName, owner, syncedAt, etag)
return err
}
+90
View File
@@ -0,0 +1,90 @@
package database
import (
"testing"
"time"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func seedGitHubDebRemote(t *testing.T, name string) {
t.Helper()
if err := testDB.CreateRemote(ctx(), &models.Remote{
Name: name, PackageType: models.PackageGitHubDeb, RepoType: models.RepoTypeRemote,
BaseURL: "https://api.github.com/repos/acme/tools", ReleasesRemote: "github", MutableTTL: 3600,
}); err != nil {
t.Fatalf("seed github_deb remote: %v", err)
}
}
// TestGitHubDebSyncLease exercises the real SQL: exactly one replica may hold the
// lease, the recency window blocks a too-soon periodic re-claim, and a prime
// (freshness 0) bypasses recency but still respects a live lease.
func TestGitHubDebSyncLease(t *testing.T) {
requireDB(t)
name := "ghdeb-lease-" + time.Now().Format("150405.000000")
seedGitHubDebRemote(t, name)
const lease = 15 * time.Minute
freshness := time.Hour
claimed, etag, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-1", freshness, lease)
if err != nil || !claimed {
t.Fatalf("replica-1 first claim: claimed=%v err=%v", claimed, err)
}
if etag != "" {
t.Fatalf("initial etag should be empty, got %q", etag)
}
claimed2, _, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-2", freshness, lease)
if err != nil {
t.Fatalf("replica-2 claim err: %v", err)
}
if claimed2 {
t.Fatal("replica-2 claimed while replica-1 holds the lease")
}
if err := testDB.ReleaseGitHubDebSyncLease(ctx(), name, "replica-1", `"etag-1"`, time.Now()); err != nil {
t.Fatalf("release: %v", err)
}
claimed3, _, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-2", freshness, lease)
if err != nil {
t.Fatalf("replica-2 recency claim err: %v", err)
}
if claimed3 {
t.Fatal("periodic claim succeeded inside the freshness window")
}
claimed4, etag4, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-2", 0, lease)
if err != nil || !claimed4 {
t.Fatalf("prime claim: claimed=%v err=%v", claimed4, err)
}
if etag4 != `"etag-1"` {
t.Fatalf("prime claim etag = %q, want persisted \"etag-1\"", etag4)
}
}
func TestListGitHubDebRemotes(t *testing.T) {
requireDB(t)
name := "ghdeb-list-" + time.Now().Format("150405.000000")
seedGitHubDebRemote(t, name)
seedRemote(t, "generic-"+time.Now().Format("150405.000000"))
remotes, err := testDB.ListGitHubDebRemotes(ctx())
if err != nil {
t.Fatalf("list: %v", err)
}
found := false
for _, r := range remotes {
if r.PackageType != models.PackageGitHubDeb {
t.Fatalf("non-github_deb remote returned: %s (%s)", r.Name, r.PackageType)
}
if r.Name == name {
found = true
}
}
if !found {
t.Fatalf("seeded remote %q not returned", name)
}
}
+57
View File
@@ -0,0 +1,57 @@
package database
import (
"context"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func (db *DB) InsertDebMetadata(ctx context.Context, meta *provider.DebMetadata) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO deb_metadata (
repo_name, file_path, content_hash,
name, version, architecture, control,
size, md5, sha256
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
ON CONFLICT (repo_name, file_path) DO NOTHING
`,
meta.RepoName, meta.FilePath, meta.ContentHash,
meta.Name, meta.Version, meta.Architecture, meta.Control,
meta.Size, meta.MD5, meta.SHA256,
)
return err
}
func (db *DB) DeleteDebMetadata(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM deb_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
func (db *DB) ListDebMetadataEntries(ctx context.Context, repoName string) ([]provider.DebMetadata, error) {
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash,
name, version, architecture, control,
size, md5, sha256
FROM deb_metadata
WHERE repo_name = $1
ORDER BY name, version, architecture
`, repoName)
if err != nil {
return nil, err
}
defer rows.Close()
var result []provider.DebMetadata
for rows.Next() {
var m provider.DebMetadata
if err := rows.Scan(
&m.RepoName, &m.FilePath, &m.ContentHash,
&m.Name, &m.Version, &m.Architecture, &m.Control,
&m.Size, &m.MD5, &m.SHA256,
); err != nil {
return nil, err
}
result = append(result, m)
}
return result, rows.Err()
}
+73
View File
@@ -0,0 +1,73 @@
package database
import (
"context"
"errors"
"time"
"github.com/jackc/pgx/v5"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// ListGitHubRPMRemotes returns every github_rpm remote so the syncer can sweep
// them on each poll tick.
func (db *DB) ListGitHubRPMRemotes(ctx context.Context) ([]models.Remote, error) {
rows, err := db.Pool.Query(ctx, `SELECT `+remoteCols+` FROM remotes WHERE package_type = $1 ORDER BY name`, models.PackageGitHubRPM)
if err != nil {
return nil, err
}
defer rows.Close()
var remotes []models.Remote
for rows.Next() {
var r models.Remote
if err := scanRemote(rows, &r); err != nil {
return nil, err
}
remotes = append(remotes, r)
}
return remotes, rows.Err()
}
// ClaimGitHubSyncLease atomically claims the per-remote sync lease. It succeeds
// (claimed=true) only when the remote is due — never synced, or synced longer
// than freshness ago — and no live lease is held by another replica. This bounds
// total GitHub load to roughly one scan per freshness window regardless of how
// many replicas poll. The returned etag is the stored releases-list ETag, shared
// across replicas so a conditional request can short-circuit an unchanged repo.
// A zero freshness (used for prime scans) ignores the recency gate and claims
// whenever no live lease is held.
func (db *DB) ClaimGitHubSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
row := db.Pool.QueryRow(ctx, `
INSERT INTO github_rpm_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
VALUES ($1, $2, now() + make_interval(secs => $4))
ON CONFLICT (remote_name) DO UPDATE
SET sync_lease_owner = $2,
sync_lease_expires = now() + make_interval(secs => $4)
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
RETURNING s.etag
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
var etag string
if err := row.Scan(&etag); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return false, "", nil
}
return false, "", err
}
return true, etag, nil
}
// ReleaseGitHubSyncLease records the completed scan and frees the lease. Only the
// owning replica may release; last_synced_at advances so the next poll waits a
// full freshness window, and etag is persisted for the next conditional request.
func (db *DB) ReleaseGitHubSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
_, err := db.Pool.Exec(ctx, `
UPDATE github_rpm_sync_state
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
WHERE remote_name = $1 AND sync_lease_owner = $2
`, remoteName, owner, syncedAt, etag)
return err
}
+95
View File
@@ -0,0 +1,95 @@
package database
import (
"testing"
"time"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func seedGitHubRPMRemote(t *testing.T, name string) {
t.Helper()
if err := testDB.CreateRemote(ctx(), &models.Remote{
Name: name, PackageType: models.PackageGitHubRPM, RepoType: models.RepoTypeRemote,
BaseURL: "https://api.github.com/repos/acme/tools", ReleasesRemote: "github", MutableTTL: 3600,
}); err != nil {
t.Fatalf("seed github_rpm remote: %v", err)
}
}
// TestGitHubSyncLease exercises the real SQL: exactly one replica may hold the
// lease, the recency window blocks a too-soon periodic re-claim, and a prime
// (freshness 0) bypasses recency but still respects a live lease.
func TestGitHubSyncLease(t *testing.T) {
requireDB(t)
name := "gh-lease-" + time.Now().Format("150405.000000")
seedGitHubRPMRemote(t, name)
const lease = 15 * time.Minute
freshness := time.Hour
// First claim on a never-synced remote wins; etag starts empty.
claimed, etag, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-1", freshness, lease)
if err != nil || !claimed {
t.Fatalf("replica-1 first claim: claimed=%v err=%v", claimed, err)
}
if etag != "" {
t.Fatalf("initial etag should be empty, got %q", etag)
}
// A second replica cannot claim while the lease is held.
claimed2, _, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-2", freshness, lease)
if err != nil {
t.Fatalf("replica-2 claim err: %v", err)
}
if claimed2 {
t.Fatal("replica-2 claimed while replica-1 holds the lease")
}
// Replica 1 finishes: record the sync and persist an etag.
if err := testDB.ReleaseGitHubSyncLease(ctx(), name, "replica-1", `"etag-1"`, time.Now()); err != nil {
t.Fatalf("release: %v", err)
}
// A periodic re-claim inside the freshness window is blocked by recency.
claimed3, _, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-2", freshness, lease)
if err != nil {
t.Fatalf("replica-2 recency claim err: %v", err)
}
if claimed3 {
t.Fatal("periodic claim succeeded inside the freshness window")
}
// A prime (freshness 0) bypasses recency and reads the persisted etag.
claimed4, etag4, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-2", 0, lease)
if err != nil || !claimed4 {
t.Fatalf("prime claim: claimed=%v err=%v", claimed4, err)
}
if etag4 != `"etag-1"` {
t.Fatalf("prime claim etag = %q, want persisted \"etag-1\"", etag4)
}
}
func TestListGitHubRPMRemotes(t *testing.T) {
requireDB(t)
name := "gh-list-" + time.Now().Format("150405.000000")
seedGitHubRPMRemote(t, name)
seedRemote(t, "generic-"+time.Now().Format("150405.000000"))
remotes, err := testDB.ListGitHubRPMRemotes(ctx())
if err != nil {
t.Fatalf("list: %v", err)
}
found := false
for _, r := range remotes {
if r.PackageType != models.PackageGitHubRPM {
t.Fatalf("non-github_rpm remote returned: %s (%s)", r.Name, r.PackageType)
}
if r.Name == name {
found = true
}
}
if !found {
t.Fatalf("seeded remote %q not returned", name)
}
}
+14
View File
@@ -38,6 +38,20 @@ func (db *DB) CreateLocalFile(ctx context.Context, repoName, filePath, contentHa
return nil
}
// UpsertLocalFile inserts a local file or repoints an existing path at a new
// blob. Unlike CreateLocalFile it never errors on a duplicate path — it is for
// mutable references such as Docker tags, where re-pushing a tag must move it to
// the newly-pushed manifest rather than being rejected as an overwrite.
func (db *DB) UpsertLocalFile(ctx context.Context, repoName, filePath, contentHash string) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO local_files (repo_name, file_path, content_hash)
VALUES ($1, $2, $3)
ON CONFLICT (repo_name, file_path)
DO UPDATE SET content_hash = EXCLUDED.content_hash, created_at = NOW()
`, repoName, filePath, contentHash)
return err
}
func (db *DB) GetLocalFile(ctx context.Context, repoName, filePath string) (*LocalFile, error) {
row := db.Pool.QueryRow(ctx, `
SELECT id, repo_name, file_path, content_hash, created_at
+46
View File
@@ -151,6 +151,8 @@ func (db *DB) migrate() error {
packager TEXT DEFAULT '',
requires JSONB DEFAULT '[]',
provides JSONB DEFAULT '[]',
conflicts JSONB DEFAULT '[]',
obsoletes JSONB DEFAULT '[]',
files JSONB DEFAULT '[]',
changelogs JSONB DEFAULT '[]',
created_at TIMESTAMPTZ DEFAULT NOW(),
@@ -158,6 +160,50 @@ func (db *DB) migrate() error {
);
CREATE INDEX IF NOT EXISTS idx_rpm_metadata_repo ON rpm_metadata(repo_name);
ALTER TABLE rpm_metadata ADD COLUMN IF NOT EXISTS conflicts JSONB DEFAULT '[]';
ALTER TABLE rpm_metadata ADD COLUMN IF NOT EXISTS obsoletes JSONB DEFAULT '[]';
CREATE TABLE IF NOT EXISTS deb_metadata (
id BIGSERIAL PRIMARY KEY,
repo_name TEXT NOT NULL,
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
name TEXT NOT NULL,
version TEXT NOT NULL,
architecture TEXT NOT NULL,
control TEXT NOT NULL,
size BIGINT DEFAULT 0,
md5 TEXT DEFAULT '',
sha256 TEXT DEFAULT '',
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(repo_name, file_path)
);
CREATE INDEX IF NOT EXISTS idx_deb_metadata_repo ON deb_metadata(repo_name);
CREATE TABLE IF NOT EXISTS github_rpm_sync_state (
remote_name TEXT PRIMARY KEY,
etag TEXT DEFAULT '',
last_synced_at TIMESTAMPTZ,
sync_lease_owner TEXT DEFAULT '',
sync_lease_expires TIMESTAMPTZ
);
CREATE TABLE IF NOT EXISTS github_deb_sync_state (
remote_name TEXT PRIMARY KEY,
etag TEXT DEFAULT '',
last_synced_at TIMESTAMPTZ,
sync_lease_owner TEXT DEFAULT '',
sync_lease_expires TIMESTAMPTZ
);
CREATE TABLE IF NOT EXISTS signing_keys (
purpose TEXT PRIMARY KEY,
private_key_armor TEXT NOT NULL,
key_id TEXT NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW()
);
`)
return err
}
+11 -5
View File
@@ -10,6 +10,8 @@ import (
func (db *DB) InsertRPMMetadata(ctx context.Context, meta *provider.RPMMetadata) error {
requiresJSON, _ := json.Marshal(meta.Requires)
providesJSON, _ := json.Marshal(meta.Provides)
conflictsJSON, _ := json.Marshal(meta.Conflicts)
obsoletesJSON, _ := json.Marshal(meta.Obsoletes)
filesJSON, _ := json.Marshal(meta.Files)
changelogsJSON, _ := json.Marshal(meta.Changelogs)
@@ -19,15 +21,15 @@ func (db *DB) InsertRPMMetadata(ctx context.Context, meta *provider.RPMMetadata)
name, epoch, version, release, arch,
summary, description, rpm_size, installed_size,
license, vendor, build_group, build_host, source_rpm, url, packager,
requires, provides, files, changelogs
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23)
requires, provides, conflicts, obsoletes, files, changelogs
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25)
ON CONFLICT (repo_name, file_path) DO NOTHING
`,
meta.RepoName, meta.FilePath, meta.ContentHash,
meta.Name, meta.Epoch, meta.Version, meta.Release, meta.Arch,
meta.Summary, meta.Description, meta.RPMSize, meta.InstalledSize,
meta.License, meta.Vendor, meta.Group, meta.BuildHost, meta.SourceRPM, meta.URL, meta.Packager,
requiresJSON, providesJSON, filesJSON, changelogsJSON,
requiresJSON, providesJSON, conflictsJSON, obsoletesJSON, filesJSON, changelogsJSON,
)
return err
}
@@ -59,6 +61,8 @@ type RPMMetadataRow struct {
Packager string
Requires json.RawMessage
Provides json.RawMessage
Conflicts json.RawMessage
Obsoletes json.RawMessage
Files json.RawMessage
Changelogs json.RawMessage
}
@@ -93,6 +97,8 @@ func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]pr
}
json.Unmarshal(r.Requires, &meta.Requires)
json.Unmarshal(r.Provides, &meta.Provides)
json.Unmarshal(r.Conflicts, &meta.Conflicts)
json.Unmarshal(r.Obsoletes, &meta.Obsoletes)
json.Unmarshal(r.Files, &meta.Files)
json.Unmarshal(r.Changelogs, &meta.Changelogs)
result[i] = meta
@@ -106,7 +112,7 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
name, epoch, version, release, arch,
summary, description, rpm_size, installed_size,
license, vendor, build_group, build_host, source_rpm, url, packager,
requires, provides, files, changelogs
requires, provides, conflicts, obsoletes, files, changelogs
FROM rpm_metadata
WHERE repo_name = $1
ORDER BY name, epoch, version, release, arch
@@ -124,7 +130,7 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
&r.Name, &r.Epoch, &r.Version, &r.Release, &r.Arch,
&r.Summary, &r.Description, &r.RPMSize, &r.InstalledSize,
&r.License, &r.Vendor, &r.Group, &r.BuildHost, &r.SourceRPM, &r.URL, &r.Packager,
&r.Requires, &r.Provides, &r.Files, &r.Changelogs,
&r.Requires, &r.Provides, &r.Conflicts, &r.Obsoletes, &r.Files, &r.Changelogs,
); err != nil {
return nil, err
}
+35
View File
@@ -0,0 +1,35 @@
package database
import (
"context"
"errors"
"github.com/jackc/pgx/v5"
)
// GetSigningKey returns the stored armored private key and key id for a purpose.
// found is false when no key has been generated yet.
func (db *DB) GetSigningKey(ctx context.Context, purpose string) (armor, keyID string, found bool, err error) {
row := db.Pool.QueryRow(ctx, `
SELECT private_key_armor, key_id FROM signing_keys WHERE purpose = $1
`, purpose)
if err := row.Scan(&armor, &keyID); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return "", "", false, nil
}
return "", "", false, err
}
return armor, keyID, true, nil
}
// InsertSigningKeyIfAbsent stores a freshly generated key, doing nothing if
// another replica already inserted one. Callers re-read with GetSigningKey to
// pick up whichever key won the race.
func (db *DB) InsertSigningKeyIfAbsent(ctx context.Context, purpose, armor, keyID string) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO signing_keys (purpose, private_key_armor, key_id)
VALUES ($1, $2, $3)
ON CONFLICT (purpose) DO NOTHING
`, purpose, armor, keyID)
return err
}
+31
View File
@@ -0,0 +1,31 @@
package database
import "testing"
func TestSigningKeyRoundTripAndIdempotency(t *testing.T) {
requireDB(t)
const purpose = "terraform-provider-test"
// Absent to start.
if _, _, found, err := testDB.GetSigningKey(ctx(), purpose); err != nil || found {
t.Fatalf("expected no key, got found=%v err=%v", found, err)
}
if err := testDB.InsertSigningKeyIfAbsent(ctx(), purpose, "ARMOR-1", "KEYID1"); err != nil {
t.Fatal(err)
}
// A second insert must not overwrite (models the replica race).
if err := testDB.InsertSigningKeyIfAbsent(ctx(), purpose, "ARMOR-2", "KEYID2"); err != nil {
t.Fatal(err)
}
armor, keyID, found, err := testDB.GetSigningKey(ctx(), purpose)
if err != nil || !found {
t.Fatalf("expected key, found=%v err=%v", found, err)
}
if armor != "ARMOR-1" || keyID != "KEYID1" {
t.Errorf("key was overwritten: armor=%q key_id=%q", armor, keyID)
}
}
+28
View File
@@ -14,6 +14,11 @@ import (
// before the referencing artifact/local_files row exists.
const blobGracePeriod = 1 * time.Hour
// uploadGracePeriod is how long a docker blob-upload staging object
// (uploads/<uuid>) may sit idle before GC treats it as an abandoned push and
// reaps it. Generous so a slow but live push is never cut off mid-flight.
const uploadGracePeriod = 24 * time.Hour
type Collector struct {
db *database.DB
store *storage.S3
@@ -43,6 +48,8 @@ func (c *Collector) Run(ctx context.Context) {
func (c *Collector) sweep(ctx context.Context) {
start := time.Now()
c.sweepUploads(ctx)
orphaned, err := c.db.FindOrphanedBlobs(ctx, blobGracePeriod)
if err != nil {
slog.Error("gc: find orphaned blobs", "error", err)
@@ -70,3 +77,24 @@ func (c *Collector) sweep(ctx context.Context) {
)
}
}
// sweepUploads reaps docker blob-upload staging objects abandoned longer than
// uploadGracePeriod (cancelled or interrupted pushes that never finalised).
func (c *Collector) sweepUploads(ctx context.Context) {
stale, err := c.store.ListStaleObjects(ctx, "uploads/", time.Now().Add(-uploadGracePeriod))
if err != nil {
slog.Error("gc: list stale uploads", "error", err)
return
}
reaped := 0
for _, key := range stale {
if err := c.store.Delete(ctx, key); err != nil {
slog.Warn("gc: delete stale upload", "key", key, "error", err)
continue
}
reaped++
}
if reaped > 0 {
slog.Info("gc: reaped stale docker uploads", "count", reaped)
}
}
+199
View File
@@ -0,0 +1,199 @@
package githubauth
import (
"context"
"crypto"
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"io"
"net/http"
"strings"
"sync"
"time"
)
const (
defaultAPIBase = "https://api.github.com"
// jwtLifetime is how long the app JWT is valid. GitHub caps it at 10 minutes;
// 9 leaves headroom for clock skew.
jwtLifetime = 9 * time.Minute
// jwtBackdate backdates iat to tolerate the app server's clock running behind
// GitHub's, which otherwise rejects the JWT.
jwtBackdate = 60 * time.Second
// refreshSkew refreshes the installation token this long before it expires so
// a request never races an expiry.
refreshSkew = 5 * time.Minute
)
type httpDoer interface {
Do(*http.Request) (*http.Response, error)
}
// appCredential mints installation access tokens for a GitHub App. It signs a
// short-lived RS256 JWT with the app private key, exchanges it for a ~1h
// installation token, caches that token, and refreshes it shortly before expiry.
// Refreshes are single-flighted by holding the mutex across the exchange, so
// concurrent callers coalesce onto one HTTP request and reuse the cached token.
type appCredential struct {
appID string
installationID string
key *rsa.PrivateKey
apiBase string
client httpDoer
mu sync.Mutex
token string
expiry time.Time
}
func newAppCredential(opts Options) (*appCredential, error) {
if opts.AppID == "" {
return nil, errors.New("github app: GITHUB_APP_ID is required")
}
if opts.InstallationID == "" {
return nil, errors.New("github app: GITHUB_APP_INSTALLATION_ID is required")
}
pemBytes, err := loadPrivateKeyPEM(opts)
if err != nil {
return nil, err
}
key, err := parseRSAPrivateKey(pemBytes)
if err != nil {
return nil, err
}
apiBase := opts.apiBaseURL
if apiBase == "" {
apiBase = defaultAPIBase
}
client := opts.httpClient
if client == nil {
client = &http.Client{Timeout: 30 * time.Second}
}
return &appCredential{
appID: opts.AppID,
installationID: opts.InstallationID,
key: key,
apiBase: strings.TrimRight(apiBase, "/"),
client: client,
}, nil
}
// Token returns a cached installation token, refreshing it under a single-flight
// lock when it is missing or within refreshSkew of expiry.
func (a *appCredential) Token(ctx context.Context) (string, error) {
a.mu.Lock()
defer a.mu.Unlock()
if a.token != "" && time.Now().Before(a.expiry.Add(-refreshSkew)) {
return a.token, nil
}
if err := a.refreshLocked(ctx); err != nil {
return "", err
}
return a.token, nil
}
func (a *appCredential) refreshLocked(ctx context.Context) error {
jwt, err := mintJWT(a.appID, a.key, time.Now())
if err != nil {
return fmt.Errorf("github app: mint jwt: %w", err)
}
u := fmt.Sprintf("%s/app/installations/%s/access_tokens", a.apiBase, a.installationID)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, u, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+jwt)
req.Header.Set("Accept", "application/vnd.github+json")
req.Header.Set("X-GitHub-Api-Version", "2022-11-28")
resp, err := a.client.Do(req)
if err != nil {
return fmt.Errorf("github app: token exchange: %w", err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if resp.StatusCode != http.StatusCreated && resp.StatusCode != http.StatusOK {
// Never echo the body verbatim — it can contain sensitive material.
return fmt.Errorf("github app: token exchange status %d", resp.StatusCode)
}
var out struct {
Token string `json:"token"`
ExpiresAt time.Time `json:"expires_at"`
}
if err := json.Unmarshal(body, &out); err != nil {
return fmt.Errorf("github app: decode token response: %w", err)
}
if out.Token == "" {
return errors.New("github app: token exchange returned an empty token")
}
a.token = out.Token
a.expiry = out.ExpiresAt
if a.expiry.IsZero() {
// Defensive: assume the documented ~1h lifetime if GitHub omits it.
a.expiry = time.Now().Add(time.Hour)
}
return nil
}
// mintJWT builds and RS256-signs a GitHub App JWT (iss=app id, backdated iat,
// ≤10m exp) using stdlib crypto — no third-party JWT dependency.
func mintJWT(appID string, key *rsa.PrivateKey, now time.Time) (string, error) {
header := map[string]string{"alg": "RS256", "typ": "JWT"}
claims := map[string]any{
"iat": now.Add(-jwtBackdate).Unix(),
"exp": now.Add(jwtLifetime).Unix(),
"iss": appID,
}
hb, err := json.Marshal(header)
if err != nil {
return "", err
}
cb, err := json.Marshal(claims)
if err != nil {
return "", err
}
signingInput := b64url(hb) + "." + b64url(cb)
digest := sha256.Sum256([]byte(signingInput))
sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, digest[:])
if err != nil {
return "", err
}
return signingInput + "." + b64url(sig), nil
}
func b64url(b []byte) string {
return base64.RawURLEncoding.EncodeToString(b)
}
// parseRSAPrivateKey accepts PKCS#1 ("RSA PRIVATE KEY") and PKCS#8 ("PRIVATE
// KEY") PEM, covering both GitHub App key export formats.
func parseRSAPrivateKey(pemBytes []byte) (*rsa.PrivateKey, error) {
block, _ := pem.Decode(pemBytes)
if block == nil {
return nil, errors.New("github app: private key is not valid PEM")
}
if key, err := x509.ParsePKCS1PrivateKey(block.Bytes); err == nil {
return key, nil
}
keyAny, err := x509.ParsePKCS8PrivateKey(block.Bytes)
if err != nil {
return nil, errors.New("github app: private key is not a supported RSA PKCS#1/PKCS#8 key")
}
rsaKey, ok := keyAny.(*rsa.PrivateKey)
if !ok {
return nil, errors.New("github app: private key is not an RSA key")
}
return rsaKey, nil
}
+207
View File
@@ -0,0 +1,207 @@
package githubauth
import (
"context"
"crypto"
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
)
func testRSAKeyPEM(t *testing.T) string {
t.Helper()
key, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
t.Fatalf("generate key: %v", err)
}
der := x509.MarshalPKCS1PrivateKey(key)
return string(pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: der}))
}
// appFixture serves the installation-token exchange endpoint, records requests,
// verifies the presented JWT against the app public key, and returns tokens with
// a controllable expiry.
type appFixture struct {
srv *httptest.Server
pub *rsa.PublicKey
mu sync.Mutex
exchanges int
lastJWT string
expiresAt func() time.Time
tokenSeq int
}
func newAppFixture(t *testing.T, pemKey string) *appFixture {
t.Helper()
block, _ := pem.Decode([]byte(pemKey))
key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
if err != nil {
t.Fatalf("parse test key: %v", err)
}
f := &appFixture{
pub: &key.PublicKey,
expiresAt: func() time.Time { return time.Now().Add(time.Hour) },
}
mux := http.NewServeMux()
mux.HandleFunc("/app/installations/456/access_tokens", func(w http.ResponseWriter, r *http.Request) {
auth := r.Header.Get("Authorization")
jwt := strings.TrimPrefix(auth, "Bearer ")
f.mu.Lock()
f.exchanges++
f.lastJWT = jwt
f.tokenSeq++
seq := f.tokenSeq
exp := f.expiresAt()
f.mu.Unlock()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
json.NewEncoder(w).Encode(map[string]any{
"token": fmt.Sprintf("ghs_installation_%d", seq),
"expires_at": exp.UTC().Format(time.RFC3339),
})
})
f.srv = httptest.NewServer(mux)
t.Cleanup(f.srv.Close)
return f
}
func (f *appFixture) verifyJWT(t *testing.T) {
t.Helper()
f.mu.Lock()
jwt := f.lastJWT
f.mu.Unlock()
parts := strings.Split(jwt, ".")
if len(parts) != 3 {
t.Fatalf("jwt not three-part: %q", jwt)
}
signingInput := parts[0] + "." + parts[1]
sig, err := base64.RawURLEncoding.DecodeString(parts[2])
if err != nil {
t.Fatalf("decode sig: %v", err)
}
digest := sha256.Sum256([]byte(signingInput))
if err := rsa.VerifyPKCS1v15(f.pub, crypto.SHA256, digest[:], sig); err != nil {
t.Fatalf("jwt signature invalid: %v", err)
}
var claims struct {
Iss string `json:"iss"`
Iat int64 `json:"iat"`
Exp int64 `json:"exp"`
}
cb, _ := base64.RawURLEncoding.DecodeString(parts[1])
if err := json.Unmarshal(cb, &claims); err != nil {
t.Fatalf("decode claims: %v", err)
}
if claims.Iss != "123" {
t.Fatalf("iss = %q, want 123", claims.Iss)
}
if claims.Exp-claims.Iat > int64((10*time.Minute)/time.Second) {
t.Fatalf("jwt lifetime exceeds 10m: iat=%d exp=%d", claims.Iat, claims.Exp)
}
if claims.Iat > time.Now().Unix() {
t.Fatalf("iat not backdated: %d", claims.Iat)
}
}
func newAppCred(t *testing.T, f *appFixture, pemKey string) *appCredential {
t.Helper()
c, err := newAppCredential(Options{
AppID: "123",
InstallationID: "456",
PrivateKeyPEM: pemKey,
apiBaseURL: f.srv.URL,
httpClient: f.srv.Client(),
})
if err != nil {
t.Fatalf("newAppCredential: %v", err)
}
return c
}
func TestApp_MintsJWTAndExchangesForInstallationToken(t *testing.T) {
pemKey := testRSAKeyPEM(t)
f := newAppFixture(t, pemKey)
c := newAppCred(t, f, pemKey)
tok, err := c.Token(context.Background())
if err != nil {
t.Fatalf("token: %v", err)
}
if tok != "ghs_installation_1" {
t.Fatalf("token = %q, want ghs_installation_1", tok)
}
if f.exchanges != 1 {
t.Fatalf("exchanges = %d, want 1", f.exchanges)
}
f.verifyJWT(t)
}
func TestApp_CachesInstallationToken(t *testing.T) {
pemKey := testRSAKeyPEM(t)
f := newAppFixture(t, pemKey)
c := newAppCred(t, f, pemKey)
for i := 0; i < 5; i++ {
if _, err := c.Token(context.Background()); err != nil {
t.Fatalf("token: %v", err)
}
}
if f.exchanges != 1 {
t.Fatalf("exchanges = %d, want 1 (token should be cached)", f.exchanges)
}
}
func TestApp_RefreshesNearExpiry(t *testing.T) {
pemKey := testRSAKeyPEM(t)
f := newAppFixture(t, pemKey)
// Token expires within refreshSkew, so every call must re-exchange.
f.expiresAt = func() time.Time { return time.Now().Add(2 * time.Minute) }
c := newAppCred(t, f, pemKey)
t1, err := c.Token(context.Background())
if err != nil {
t.Fatalf("token 1: %v", err)
}
t2, err := c.Token(context.Background())
if err != nil {
t.Fatalf("token 2: %v", err)
}
if f.exchanges != 2 {
t.Fatalf("exchanges = %d, want 2 (near-expiry token must refresh)", f.exchanges)
}
if t1 == t2 {
t.Fatalf("expected a fresh token after refresh, both = %q", t1)
}
}
func TestApp_ConcurrentTokenSingleFlights(t *testing.T) {
pemKey := testRSAKeyPEM(t)
f := newAppFixture(t, pemKey)
c := newAppCred(t, f, pemKey)
var wg sync.WaitGroup
for i := 0; i < 20; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if _, err := c.Token(context.Background()); err != nil {
t.Errorf("token: %v", err)
}
}()
}
wg.Wait()
if f.exchanges != 1 {
t.Fatalf("exchanges = %d, want 1 (concurrent calls must coalesce)", f.exchanges)
}
}
+106
View File
@@ -0,0 +1,106 @@
// Package githubauth provides the process-wide GitHub machine credential used to
// authenticate every outbound GitHub request (releases scan, ranged asset header
// fetches, and the generic-github byte proxy for private assets). The credential
// is delivered via env/secret only — it is never stored per-remote in the DB,
// never returned by any API, and never logged.
package githubauth
import (
"context"
"errors"
"fmt"
"os"
"strings"
"sync"
)
// Credential yields a bearer token for GitHub requests. Token may block to mint
// or refresh (the GitHub App path); an empty string means "no auth", which only
// happens when no credential is configured.
type Credential interface {
Token(ctx context.Context) (string, error)
}
// Options is the raw, env-sourced auth configuration. Exactly one mode may be
// configured: a static token, or a GitHub App (id + installation id + private
// key). Partial App configuration is an error (fail closed); no fields at all is
// fine and yields a nil credential (anonymous, current behavior).
type Options struct {
// Token is a Personal Access Token (fine-grained or classic) sent verbatim
// as "Authorization: Bearer <token>".
Token string
// GitHub App fields. PrivateKeyPEM and PrivateKeyPath are alternatives; the
// inline PEM wins when both are set.
AppID string
InstallationID string
PrivateKeyPEM string
PrivateKeyPath string
// apiBaseURL overrides https://api.github.com for tests. Empty uses the real
// endpoint. httpClient likewise overrides the default client for tests.
apiBaseURL string
httpClient httpDoer
}
// New builds the process credential from options, validating that auth is either
// fully configured or fully absent. It returns (nil, nil) when nothing is set.
func New(opts Options) (Credential, error) {
hasToken := opts.Token != ""
hasAppField := opts.AppID != "" || opts.InstallationID != "" ||
opts.PrivateKeyPEM != "" || opts.PrivateKeyPath != ""
switch {
case !hasToken && !hasAppField:
return nil, nil // no auth configured — anonymous is fine
case hasToken && hasAppField:
return nil, errors.New("github auth: both a token and GitHub App fields are set; configure exactly one")
case hasToken:
return staticToken{token: opts.Token}, nil
default:
return newAppCredential(opts)
}
}
// staticToken is a fixed PAT credential.
type staticToken struct{ token string }
func (s staticToken) Token(context.Context) (string, error) { return s.token, nil }
// server is the process-wide credential set once at startup. A nil value means
// no server credential (anonymous). Access is guarded so a late SetServer in a
// test is race-free.
var (
serverMu sync.RWMutex
server Credential
)
// SetServer installs the process credential. Call once during startup.
func SetServer(c Credential) {
serverMu.Lock()
server = c
serverMu.Unlock()
}
// Server returns the process credential, or nil if none is configured.
func Server() Credential {
serverMu.RLock()
defer serverMu.RUnlock()
return server
}
// loadPrivateKeyPEM resolves the App private key bytes from the inline PEM or a
// file path, without ever returning the key material in an error message.
func loadPrivateKeyPEM(opts Options) ([]byte, error) {
if strings.TrimSpace(opts.PrivateKeyPEM) != "" {
return []byte(opts.PrivateKeyPEM), nil
}
if opts.PrivateKeyPath != "" {
b, err := os.ReadFile(opts.PrivateKeyPath)
if err != nil {
return nil, fmt.Errorf("github app: read private key file: %w", err)
}
return b, nil
}
return nil, errors.New("github app: no private key configured")
}
+77
View File
@@ -0,0 +1,77 @@
package githubauth
import (
"context"
"testing"
)
func TestNew_NoConfigIsAnonymous(t *testing.T) {
c, err := New(Options{})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if c != nil {
t.Fatalf("expected nil credential when nothing configured, got %T", c)
}
}
func TestNew_TokenMode(t *testing.T) {
c, err := New(Options{Token: "ghp_example"})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
tok, err := c.Token(context.Background())
if err != nil {
t.Fatalf("token: %v", err)
}
if tok != "ghp_example" {
t.Fatalf("token = %q, want ghp_example", tok)
}
}
func TestNew_TokenAndAppConflict(t *testing.T) {
_, err := New(Options{Token: "ghp_example", AppID: "123"})
if err == nil {
t.Fatal("expected error when both token and app fields are set")
}
}
func TestNew_PartialAppFailsClosed(t *testing.T) {
cases := map[string]Options{
"app id without key": {AppID: "123", InstallationID: "456"},
"key without app id": {InstallationID: "456", PrivateKeyPEM: testRSAKeyPEM(t)},
"app id without inst": {AppID: "123", PrivateKeyPEM: testRSAKeyPEM(t)},
}
for name, opts := range cases {
t.Run(name, func(t *testing.T) {
if _, err := New(opts); err == nil {
t.Fatalf("expected fail-closed error for %q", name)
}
})
}
}
func TestNew_AppModeParsesKey(t *testing.T) {
c, err := New(Options{
AppID: "123",
InstallationID: "456",
PrivateKeyPEM: testRSAKeyPEM(t),
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if _, ok := c.(*appCredential); !ok {
t.Fatalf("expected *appCredential, got %T", c)
}
}
func TestNew_AppModeRejectsBadKey(t *testing.T) {
_, err := New(Options{
AppID: "123",
InstallationID: "456",
PrivateKeyPEM: "-----BEGIN RSA PRIVATE KEY-----\nnope\n-----END RSA PRIVATE KEY-----",
})
if err == nil {
t.Fatal("expected error for malformed private key")
}
}
+447
View File
@@ -0,0 +1,447 @@
package deb
import (
"archive/tar"
"bufio"
"bytes"
"compress/gzip"
"context"
"crypto/md5"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"path"
"regexp"
"strconv"
"strings"
"time"
"github.com/klauspost/compress/zstd"
"github.com/ulikunitz/xz"
"git.unkin.net/unkin/artifactapi/internal/auth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func init() {
provider.Register(&Provider{})
}
// mutableRe marks the apt index surface (both the flat local repo and a proxied
// Debian/Ubuntu mirror's dists/ tree) so the caching engine revalidates it
// instead of freezing it like an immutable .deb.
var mutableRe = []*regexp.Regexp{
regexp.MustCompile(`(^|/)Packages(\.gz|\.xz|\.bz2)?$`),
regexp.MustCompile(`(^|/)Sources(\.gz|\.xz|\.bz2)?$`),
regexp.MustCompile(`(^|/)Release$`),
regexp.MustCompile(`(^|/)InRelease$`),
regexp.MustCompile(`(^|/)Release\.gpg$`),
regexp.MustCompile(`(^|/)Contents-`),
regexp.MustCompile(`^dists/`),
regexp.MustCompile(`/by-hash/`),
}
type Provider struct{}
func (p *Provider) Type() models.PackageType { return models.PackageDeb }
func (p *Provider) Classify(path string) provider.Mutability {
for _, re := range mutableRe {
if re.MatchString(path) {
return provider.Mutable
}
}
return provider.Immutable
}
func (p *Provider) ContentType(path string) string {
switch {
case strings.HasSuffix(path, ".deb"):
return "application/vnd.debian.binary-package"
case strings.HasSuffix(path, ".gz"):
return "application/gzip"
case strings.HasSuffix(path, ".xz"):
return "application/x-xz"
case strings.HasSuffix(path, "Packages"), strings.HasSuffix(path, "Release"),
strings.HasSuffix(path, "InRelease"), strings.HasSuffix(path, "Sources"):
return "text/plain"
}
return "application/octet-stream"
}
func (p *Provider) UpstreamURL(remote models.Remote, path string) string {
return strings.TrimRight(remote.BaseURL, "/") + "/" + strings.TrimLeft(path, "/")
}
func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
return nil, nil
}
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
return auth.BasicHeaders(remote), nil
}
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
filename := filePath
if idx := strings.LastIndex(filePath, "/"); idx >= 0 {
filename = filePath[idx+1:]
}
if !strings.HasSuffix(strings.ToLower(filename), ".deb") {
return "", "", fmt.Errorf("file must be a .deb package")
}
return "pool/" + filename, "application/vnd.debian.binary-package", nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
filename := strings.TrimPrefix(storagePath, "pool/")
return map[string]any{
"filename": filename,
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs provider.BlobReader, db provider.MetadataStore) {
s3Key := storage.BlobKey(strings.TrimPrefix(contentHash, "sha256:"))
reader, blobSize, err := blobs.Download(ctx, s3Key)
if err != nil {
slog.Error("deb metadata: download failed", "repo", repoName, "path", storagePath, "error", err)
return
}
defer reader.Close()
raw, err := io.ReadAll(reader)
if err != nil {
slog.Error("deb metadata: read failed", "repo", repoName, "path", storagePath, "error", err)
return
}
control, err := extractControl(raw)
if err != nil {
slog.Error("deb metadata: parse failed", "repo", repoName, "path", storagePath, "error", err)
return
}
fields := parseControlFields(control)
sum := md5.Sum(raw)
meta := &provider.DebMetadata{
RepoName: repoName,
FilePath: storagePath,
ContentHash: contentHash,
Name: fields["Package"],
Version: fields["Version"],
Architecture: fields["Architecture"],
Control: strings.TrimRight(control, "\n"),
Size: blobSize,
MD5: hex.EncodeToString(sum[:]),
SHA256: strings.TrimPrefix(contentHash, "sha256:"),
}
if meta.Name == "" {
slog.Error("deb metadata: control missing Package field", "repo", repoName, "path", storagePath)
return
}
if err := db.InsertDebMetadata(ctx, meta); err != nil {
slog.Error("deb metadata: insert failed", "repo", repoName, "path", storagePath, "error", err)
return
}
slog.Info("deb metadata: parsed", "repo", repoName, "name", meta.Name, "version", meta.Version, "arch", meta.Architecture)
}
func (p *Provider) AfterDelete(ctx context.Context, repoName, storagePath string, db provider.MetadataDeleter) error {
if err := db.DeleteDebMetadata(ctx, repoName, storagePath); err != nil {
slog.Error("deb metadata: delete failed", "repo", repoName, "path", storagePath, "error", err)
return err
}
slog.Info("deb metadata: deleted", "repo", repoName, "path", storagePath)
return nil
}
// extractControl reads a .deb (an ar archive), locates the control.tar.* member,
// decompresses it, and returns the raw ./control paragraph. Pure Go: no dpkg.
func extractControl(deb []byte) (string, error) {
members, err := readAr(deb)
if err != nil {
return "", err
}
var name string
var data []byte
for _, m := range members {
if strings.HasPrefix(m.name, "control.tar") {
name = m.name
data = m.data
break
}
}
if data == nil {
return "", errors.New("no control.tar member in .deb")
}
tarBytes, err := decompress(name, data)
if err != nil {
return "", err
}
return readControlParagraph(tarBytes)
}
// readControlParagraph scans a decompressed control.tar and returns the raw
// ./control paragraph. Shared by the local upload path (extractControl) and the
// github_deb ranged-prefix parser.
func readControlParagraph(controlTar []byte) (string, error) {
tr := tar.NewReader(bytes.NewReader(controlTar))
for {
hdr, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return "", fmt.Errorf("read control.tar: %w", err)
}
clean := strings.TrimPrefix(hdr.Name, "./")
if clean == "control" {
b, err := io.ReadAll(tr)
if err != nil {
return "", fmt.Errorf("read control file: %w", err)
}
return string(b), nil
}
}
return "", errors.New("no ./control in control.tar")
}
func decompress(name string, data []byte) ([]byte, error) {
switch {
case strings.HasSuffix(name, ".gz"):
zr, err := gzip.NewReader(bytes.NewReader(data))
if err != nil {
return nil, err
}
defer zr.Close()
return io.ReadAll(zr)
case strings.HasSuffix(name, ".xz"):
xr, err := xz.NewReader(bytes.NewReader(data))
if err != nil {
return nil, err
}
return io.ReadAll(xr)
case strings.HasSuffix(name, ".zst"):
zr, err := zstd.NewReader(bytes.NewReader(data))
if err != nil {
return nil, err
}
defer zr.Close()
return io.ReadAll(zr)
case strings.HasSuffix(name, ".tar"):
return data, nil
}
return nil, fmt.Errorf("unsupported control.tar compression: %s", name)
}
type arMember struct {
name string
data []byte
}
// readAr parses the (trivial) Unix ar archive that wraps a .deb. Each member has
// a 60-byte header; the size field is decimal ASCII and data is padded to an
// even offset.
func readAr(data []byte) ([]arMember, error) {
const magic = "!<arch>\n"
if len(data) < len(magic) || string(data[:len(magic)]) != magic {
return nil, errors.New("not an ar archive")
}
off := len(magic)
var members []arMember
for off+60 <= len(data) {
hdr := data[off : off+60]
off += 60
name := strings.TrimRight(string(hdr[0:16]), " ")
name = strings.TrimSuffix(name, "/")
size, err := strconv.ParseInt(strings.TrimSpace(string(hdr[48:58])), 10, 64)
if err != nil {
return nil, fmt.Errorf("bad ar size for %q: %w", name, err)
}
if off+int(size) > len(data) {
return nil, fmt.Errorf("truncated ar member %q", name)
}
members = append(members, arMember{name: name, data: data[off : off+int(size)]})
off += int(size)
if size%2 == 1 {
off++
}
}
return members, nil
}
// parseControlFields reads the single-line fields of an RFC822-style control
// paragraph. Continuation lines (leading whitespace) belong to the previous
// field and are ignored here since only Package/Version/Architecture are read.
func parseControlFields(control string) map[string]string {
fields := map[string]string{}
sc := bufio.NewScanner(strings.NewReader(control))
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for sc.Scan() {
line := sc.Text()
if line == "" || line[0] == ' ' || line[0] == '\t' {
continue
}
idx := strings.IndexByte(line, ':')
if idx < 0 {
continue
}
key := strings.TrimSpace(line[:idx])
if _, seen := fields[key]; seen {
continue
}
fields[key] = strings.TrimSpace(line[idx+1:])
}
return fields
}
// normalizeIndexPath collapses apt's verbatim dist prefix from a flat-repo
// request. For `deb ... <repo>/ ./`, apt appends the "./" dist literally and asks
// for "./Packages" (and "./Release", "./InRelease"); dot-segments must be
// collapsed so the index matcher sees "Packages". A no-op for pool/*.deb paths.
func normalizeIndexPath(p string) string {
return strings.TrimPrefix(path.Clean("/"+p), "/")
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, reqPath string) bool {
path := normalizeIndexPath(reqPath)
switch path {
case "Packages", "Packages.gz", "Release":
default:
return false
}
reader, ok := files.(provider.DebMetadataReader)
if !ok {
http.Error(w, "deb metadata not available", http.StatusInternalServerError)
return true
}
metas, err := reader.ListDebMetadataEntries(r.Context(), repoName)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
slog.Warn("deb: metadata read canceled", "repo", repoName, "error", err)
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
return true
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return true
}
switch path {
case "Packages":
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
w.Write(generatePackages(metas))
case "Packages.gz":
w.Header().Set("Content-Type", "application/gzip")
w.WriteHeader(http.StatusOK)
w.Write(gzipBytes(generatePackages(metas)))
case "Release":
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
w.Write(generateRelease(metas))
}
return true
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
return nil, fmt.Errorf("deb local index generation for virtual repos not supported")
}
// generatePackages emits the flat-repo Packages file: each package's raw control
// stanza followed by the apt-required Filename/Size/MD5sum/SHA256 fields, blank
// line separated.
func generatePackages(metas []provider.DebMetadata) []byte {
var b bytes.Buffer
for _, m := range metas {
b.WriteString(strings.TrimRight(m.Control, "\n"))
b.WriteString("\n")
fmt.Fprintf(&b, "Filename: %s\n", m.FilePath)
fmt.Fprintf(&b, "Size: %d\n", m.Size)
if m.MD5 != "" {
fmt.Fprintf(&b, "MD5sum: %s\n", m.MD5)
}
if m.SHA256 != "" {
fmt.Fprintf(&b, "SHA256: %s\n", m.SHA256)
}
b.WriteString("\n")
}
return b.Bytes()
}
func generateRelease(metas []provider.DebMetadata) []byte {
packages := generatePackages(metas)
packagesGz := gzipBytes(packages)
arches := uniqueArches(metas)
var b bytes.Buffer
fmt.Fprintf(&b, "Date: %s\n", time.Now().UTC().Format(time.RFC1123Z))
fmt.Fprintf(&b, "Architectures: %s\n", strings.Join(arches, " "))
b.WriteString("Acquire-By-Hash: no\n")
b.WriteString("MD5Sum:\n")
writeReleaseEntry(&b, md5Hex(packages), len(packages), "Packages")
writeReleaseEntry(&b, md5Hex(packagesGz), len(packagesGz), "Packages.gz")
b.WriteString("SHA256:\n")
writeReleaseEntry(&b, sha256Hex(packages), len(packages), "Packages")
writeReleaseEntry(&b, sha256Hex(packagesGz), len(packagesGz), "Packages.gz")
return b.Bytes()
}
func writeReleaseEntry(b *bytes.Buffer, hash string, size int, name string) {
fmt.Fprintf(b, " %s %d %s\n", hash, size, name)
}
func uniqueArches(metas []provider.DebMetadata) []string {
seen := map[string]bool{}
var out []string
for _, m := range metas {
a := m.Architecture
if a == "" || seen[a] {
continue
}
seen[a] = true
out = append(out, a)
}
return out
}
func gzipBytes(data []byte) []byte {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
gz.Write(data)
gz.Close()
return buf.Bytes()
}
func md5Hex(data []byte) string {
h := md5.Sum(data)
return hex.EncodeToString(h[:])
}
func sha256Hex(data []byte) string {
h := sha256.Sum256(data)
return hex.EncodeToString(h[:])
}
+408
View File
@@ -0,0 +1,408 @@
package deb
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/klauspost/compress/zstd"
"github.com/ulikunitz/xz"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type fakeBlobReader struct{ data []byte }
func (f fakeBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
return io.NopCloser(bytes.NewReader(f.data)), int64(len(f.data)), nil
}
type errBlobReader struct{}
func (errBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
return nil, 0, io.ErrUnexpectedEOF
}
// fakeDebStore satisfies provider.MetadataStore (both insert methods) and
// records the deb row that AfterUpload writes.
type fakeDebStore struct{ inserted *provider.DebMetadata }
func (f *fakeDebStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
func (f *fakeDebStore) InsertDebMetadata(_ context.Context, m *provider.DebMetadata) error {
f.inserted = m
return nil
}
type fakeDebReader struct{ metas []provider.DebMetadata }
func (f fakeDebReader) ListDebMetadataEntries(context.Context, string) ([]provider.DebMetadata, error) {
return f.metas, nil
}
func (f fakeDebReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
return nil, nil
}
func (f fakeDebReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
type errDebReader struct{}
func (errDebReader) ListDebMetadataEntries(context.Context, string) ([]provider.DebMetadata, error) {
return nil, io.ErrUnexpectedEOF
}
func (errDebReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
return nil, nil
}
func (errDebReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
func TestDebPureFuncs(t *testing.T) {
p := &Provider{}
if p.Type() != models.PackageDeb {
t.Errorf("type = %q", p.Type())
}
if out, _ := p.RewriteResponse(nil, models.Remote{}, "http://p"); out != nil {
t.Error("deb never rewrites")
}
if got := p.UpstreamURL(models.Remote{BaseURL: "https://mirror/"}, "/dists/bookworm/Release"); got != "https://mirror/dists/bookworm/Release" {
t.Errorf("upstream url %q", got)
}
h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
if h.Get("Authorization") == "" {
t.Error("auth header")
}
}
func TestDebClassify(t *testing.T) {
p := &Provider{}
tests := []struct {
path string
want provider.Mutability
}{
{"pool/foo_1.0_amd64.deb", provider.Immutable},
{"Packages", provider.Mutable},
{"Packages.gz", provider.Mutable},
{"Release", provider.Mutable},
{"InRelease", provider.Mutable},
{"Release.gpg", provider.Mutable},
{"dists/bookworm/main/binary-amd64/Packages", provider.Mutable},
{"dists/bookworm/Release", provider.Mutable},
{"dists/bookworm/main/by-hash/SHA256/abc", provider.Mutable},
{"dists/bookworm/main/Contents-amd64.gz", provider.Mutable},
}
for _, tt := range tests {
if got := p.Classify(tt.path); got != tt.want {
t.Errorf("Classify(%q) = %v, want %v", tt.path, got, tt.want)
}
}
}
func TestDebContentType(t *testing.T) {
p := &Provider{}
for path, want := range map[string]string{
"pool/foo_1.0_amd64.deb": "application/vnd.debian.binary-package",
"dists/bookworm/main/bin/Packages.gz": "application/gzip",
"dists/bookworm/main/bin/Packages.xz": "application/x-xz",
"Packages": "text/plain",
"Release": "text/plain",
"InRelease": "text/plain",
"pool/other": "application/octet-stream",
} {
if got := p.ContentType(path); got != want {
t.Errorf("ContentType(%q) = %q, want %q", path, got, want)
}
}
}
func TestDebValidateUpload(t *testing.T) {
p := &Provider{}
sp, ct, err := p.ValidateUpload("dir/foo_1.0_amd64.deb")
if err != nil || sp != "pool/foo_1.0_amd64.deb" || ct != "application/vnd.debian.binary-package" {
t.Errorf("sp=%q ct=%q err=%v", sp, ct, err)
}
if _, _, err := p.ValidateUpload("foo.rpm"); err == nil {
t.Error("expected error for non-deb")
}
resp := p.UploadResponse("pool/foo_1.0_amd64.deb", "sha256:abc", 42)
if resp["filename"] != "foo_1.0_amd64.deb" || resp["content_hash"] != "sha256:abc" || resp["size_bytes"] != int64(42) {
t.Errorf("upload response %v", resp)
}
}
func TestDebAfterUpload(t *testing.T) {
data := testsupport.MinimalDeb("e2e-testpkg", "1.2.3", "amd64")
store := &fakeDebStore{}
(&Provider{}).AfterUpload(context.Background(), "myrepo", "pool/e2e-testpkg_1.2.3_amd64.deb",
"sha256:deadbeef", fakeBlobReader{data: data}, store)
m := store.inserted
if m == nil {
t.Fatal("no metadata inserted")
}
if m.Name != "e2e-testpkg" || m.Version != "1.2.3" || m.Architecture != "amd64" {
t.Errorf("unexpected metadata: %+v", m)
}
if m.Size != int64(len(data)) {
t.Errorf("Size = %d, want %d", m.Size, len(data))
}
if m.SHA256 != "deadbeef" {
t.Errorf("SHA256 = %q, want deadbeef", m.SHA256)
}
if m.MD5 == "" {
t.Error("MD5 not computed")
}
if !strings.Contains(m.Control, "Package: e2e-testpkg") {
t.Errorf("raw control not stored: %q", m.Control)
}
// The raw stanza is stored verbatim (no trailing newline) so Packages can
// reproduce it faithfully.
if strings.HasSuffix(m.Control, "\n") {
t.Error("control should be trimmed of trailing newline")
}
}
func TestDebAfterUploadErrors(t *testing.T) {
// Download failure: no insert, no panic.
store := &fakeDebStore{}
(&Provider{}).AfterUpload(context.Background(), "r", "p", "sha256:x", errBlobReader{}, store)
if store.inserted != nil {
t.Error("no metadata should be inserted on download error")
}
// Not a .deb (ar) archive.
store2 := &fakeDebStore{}
(&Provider{}).AfterUpload(context.Background(), "r", "p", "sha256:x", fakeBlobReader{data: []byte("not a deb")}, store2)
if store2.inserted != nil {
t.Error("no metadata should be inserted on parse error")
}
}
func TestDebControlDecompression(t *testing.T) {
// The control tarball may be gzip, xz, or zstd (goreleaser/nfpm emit gzip or
// xz); each must round-trip to the same control stanza.
for _, tc := range []struct {
name string
member string
comp func([]byte) []byte
}{
{"gzip", "control.tar.gz", gzipBytes},
{"xz", "control.tar.xz", xzBytes},
{"zstd", "control.tar.zst", zstdBytes},
} {
t.Run(tc.name, func(t *testing.T) {
deb := buildDeb("pkg", "9.9", "arm64", tc.member, tc.comp)
control, err := extractControl(deb)
if err != nil {
t.Fatalf("extractControl: %v", err)
}
fields := parseControlFields(control)
if fields["Package"] != "pkg" || fields["Version"] != "9.9" || fields["Architecture"] != "arm64" {
t.Errorf("fields = %v", fields)
}
})
}
}
func TestDebParseControlContinuationLines(t *testing.T) {
control := "Package: p\nVersion: 1\n" +
"Description: short\n very long\n .\n more\n" +
"Architecture: all\n"
f := parseControlFields(control)
if f["Package"] != "p" || f["Version"] != "1" || f["Architecture"] != "all" {
t.Errorf("continuation lines corrupted parse: %v", f)
}
if f["Description"] != "short" {
t.Errorf("Description folded continuation into value: %q", f["Description"])
}
}
func TestDebServeLocalIndex(t *testing.T) {
p := &Provider{}
reader := fakeDebReader{metas: []provider.DebMetadata{
{Name: "aaa", Version: "1.0", Architecture: "amd64", FilePath: "pool/aaa_1.0_amd64.deb",
Control: "Package: aaa\nVersion: 1.0\nArchitecture: amd64", Size: 100, MD5: "md5aaa", SHA256: "sha256aaa"},
{Name: "bbb", Version: "2.0", Architecture: "arm64", FilePath: "pool/bbb_2.0_arm64.deb",
Control: "Package: bbb\nVersion: 2.0\nArchitecture: arm64", Size: 200, MD5: "md5bbb", SHA256: "sha256bbb"},
}}
serve := func(path string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if !p.ServeLocalIndex(w, r, reader, "myrepo", path) {
t.Fatalf("ServeLocalIndex returned false for %q", path)
}
return w
}
// Packages lists both packages with their apt fields.
w := serve("Packages")
body := w.Body.String()
if w.Code != 200 {
t.Fatalf("Packages code %d", w.Code)
}
for _, want := range []string{
"Package: aaa", "Package: bbb",
"Filename: pool/aaa_1.0_amd64.deb", "Size: 100", "MD5sum: md5aaa", "SHA256: sha256aaa",
"Filename: pool/bbb_2.0_arm64.deb", "Size: 200",
} {
if !strings.Contains(body, want) {
t.Errorf("Packages missing %q:\n%s", want, body)
}
}
// Stanzas are blank-line separated.
if !strings.Contains(body, "SHA256: sha256aaa\n\n") {
t.Errorf("stanzas not blank-line separated:\n%s", body)
}
// Packages.gz decompresses to exactly the plain Packages bytes.
w = serve("Packages.gz")
if w.Code != 200 {
t.Fatalf("Packages.gz code %d", w.Code)
}
zr, err := gzip.NewReader(bytes.NewReader(w.Body.Bytes()))
if err != nil {
t.Fatalf("Packages.gz not gzip: %v", err)
}
plain, _ := io.ReadAll(zr)
if !bytes.Equal(plain, []byte(body)) {
t.Error("Packages.gz does not decompress to Packages")
}
// Release lists arches and both index files under MD5Sum/SHA256.
w = serve("Release")
rel := w.Body.String()
if w.Code != 200 {
t.Fatalf("Release code %d", w.Code)
}
for _, want := range []string{"Date:", "Architectures: amd64 arm64", "Acquire-By-Hash: no", "MD5Sum:", "SHA256:", " Packages\n", " Packages.gz\n"} {
if !strings.Contains(rel, want) {
t.Errorf("Release missing %q:\n%s", want, rel)
}
}
// Unsigned trust model: no InRelease / Release.gpg served here.
for _, path := range []string{"InRelease", "Release.gpg", "pool/aaa_1.0_amd64.deb"} {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if p.ServeLocalIndex(w, r, reader, "myrepo", path) {
t.Errorf("ServeLocalIndex should return false for %q", path)
}
}
}
// Real apt appends the flat-repo dist "./" verbatim, so it requests "./Packages"
// / "./Release" (curl pre-normalizes /./ which masks this). The handler must
// collapse the dot-segment and return the same bytes as the un-prefixed request.
func TestDebServeLocalIndexAptDotSegment(t *testing.T) {
p := &Provider{}
reader := fakeDebReader{metas: []provider.DebMetadata{
{Name: "aaa", Version: "1.0", Architecture: "amd64", FilePath: "pool/aaa_1.0_amd64.deb",
Control: "Package: aaa\nVersion: 1.0\nArchitecture: amd64", Size: 100, MD5: "md5aaa", SHA256: "sha256aaa"},
}}
serve := func(path string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if !p.ServeLocalIndex(w, r, reader, "myrepo", path) {
t.Fatalf("ServeLocalIndex returned false for %q", path)
}
return w
}
// Packages is deterministic: require exact byte identity.
if plain, dotted := serve("Packages"), serve("./Packages"); plain.Code != 200 || dotted.Code != 200 {
t.Fatalf("Packages: plain=%d dotted=%d, want 200/200", plain.Code, dotted.Code)
} else if !bytes.Equal(plain.Body.Bytes(), dotted.Body.Bytes()) {
t.Error("./Packages body differs from Packages body")
}
// Release carries a Date: header stamped from time.Now(); compare the rest.
plain, dotted := serve("Release"), serve("./Release")
if plain.Code != 200 || dotted.Code != 200 {
t.Fatalf("Release: plain=%d dotted=%d, want 200/200", plain.Code, dotted.Code)
}
if stripDate(plain.Body.String()) != stripDate(dotted.Body.String()) {
t.Error("./Release body differs from Release body (ignoring Date)")
}
}
func stripDate(s string) string {
var out []string
for _, line := range strings.Split(s, "\n") {
if strings.HasPrefix(line, "Date:") {
continue
}
out = append(out, line)
}
return strings.Join(out, "\n")
}
func TestDebServeMetadataError(t *testing.T) {
p := &Provider{}
for _, path := range []string{"Packages", "Packages.gz", "Release"} {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
p.ServeLocalIndex(w, r, errDebReader{}, "repo", path)
if w.Code != 500 {
t.Errorf("%s with failing reader = %d, want 500", path, w.Code)
}
}
}
func TestDebGenerateLocalIndexUnsupported(t *testing.T) {
if _, err := (&Provider{}).GenerateLocalIndex(context.Background(), fakeDebReader{}, "r", "Packages"); err == nil {
t.Error("expected unsupported error")
}
}
// buildDeb assembles an ar .deb whose control member uses the given name and
// compressor, so the decompression branches can be exercised directly.
func buildDeb(name, version, arch, member string, comp func([]byte) []byte) []byte {
control := "Package: " + name + "\nVersion: " + version + "\nArchitecture: " + arch + "\n"
controlTar := comp(tarSingle("./control", []byte(control)))
var buf bytes.Buffer
buf.WriteString("!<arch>\n")
arWrite(&buf, "debian-binary", []byte("2.0\n"))
arWrite(&buf, member, controlTar)
arWrite(&buf, "data.tar.gz", gzipBytes(tarSingle("./x", []byte("x"))))
return buf.Bytes()
}
func tarSingle(name string, data []byte) []byte {
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(data)), Typeflag: tar.TypeReg})
tw.Write(data)
tw.Close()
return buf.Bytes()
}
func arWrite(buf *bytes.Buffer, name string, data []byte) {
fmt.Fprintf(buf, "%-16s%-12s%-6s%-6s%-8s%-10d`\n", name, "0", "0", "0", "100644", len(data))
buf.Write(data)
if len(data)%2 == 1 {
buf.WriteByte('\n')
}
}
func xzBytes(data []byte) []byte {
var buf bytes.Buffer
w, _ := xz.NewWriter(&buf)
w.Write(data)
w.Close()
return buf.Bytes()
}
func zstdBytes(data []byte) []byte {
var buf bytes.Buffer
w, _ := zstd.NewWriter(&buf)
w.Write(data)
w.Close()
return buf.Bytes()
}
+724
View File
@@ -0,0 +1,724 @@
package deb
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/url"
"regexp"
"strconv"
"strings"
"sync"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/githubauth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// gitHubProvider is the process-wide singleton for github_deb. The background
// Syncer binds its shared rate limiter and work queue onto this instance so the
// request path and the syncer drive the same derive machinery.
var gitHubProvider = newGitHubProvider()
func init() {
provider.Register(gitHubProvider)
}
// Tuning knobs for the no-precache control fetch. A .deb is an ar archive whose
// control.tar member sits right after the tiny debian-binary member, so a small
// front prefix reliably covers it.
const (
defaultHeaderRangeInitial = 32 << 10 // 32 KiB — covers control.tar of almost every .deb
defaultHeaderRangeMax = 16 << 20 // 16 MiB — give up past this and skip the asset
defaultReleasePageCap = 10 // 100 releases/page * 10 pages
defaultScanTimeout = 10 * time.Minute
defaultServeTimeout = 30 * time.Second
defaultColdWait = 8 * time.Second
)
// GitHubProvider is a metadata-only remote: it scans a GitHub repo's releases
// for .deb assets, derives per-asset control metadata via a ranged prefix fetch
// (never downloading whole packages), synthesizes a flat apt repository from that
// cached metadata, and redirects package downloads to a backend "releases_remote"
// (the generic github.com remote) that serves the actual bytes.
type GitHubProvider struct {
client *http.Client
headerInitial int64
headerMax int64
pageCap int
scanTimeout time.Duration
serveTimeout time.Duration
coldWait time.Duration
limiter *rate.Limiter
syncer *Syncer
serverCred githubauth.Credential
mu sync.Mutex
scanning map[string]bool
lastScan map[string]time.Time
}
func newGitHubProvider() *GitHubProvider {
return &GitHubProvider{
client: &http.Client{},
headerInitial: defaultHeaderRangeInitial,
headerMax: defaultHeaderRangeMax,
pageCap: defaultReleasePageCap,
scanTimeout: defaultScanTimeout,
serveTimeout: defaultServeTimeout,
coldWait: defaultColdWait,
scanning: map[string]bool{},
lastScan: map[string]time.Time{},
}
}
func (p *GitHubProvider) limiterWait(ctx context.Context) error {
if p.limiter == nil {
return nil
}
return p.limiter.Wait(ctx)
}
func (p *GitHubProvider) Type() models.PackageType { return models.PackageGitHubDeb }
func (p *GitHubProvider) Classify(path string) provider.Mutability {
switch path {
case "Packages", "Packages.gz", "Release", "InRelease", "Release.gpg":
return provider.Mutable
}
return provider.Immutable
}
func (p *GitHubProvider) ContentType(path string) string {
switch {
case strings.HasSuffix(path, ".deb"):
return "application/vnd.debian.binary-package"
case strings.HasSuffix(path, ".gz"):
return "application/gzip"
case path == "Packages" || path == "Release" || path == "InRelease":
return "text/plain"
}
return "application/octet-stream"
}
func (p *GitHubProvider) UpstreamURL(remote models.Remote, path string) string {
return strings.TrimRight(remote.BaseURL, "/") + "/" + strings.TrimLeft(path, "/")
}
func (p *GitHubProvider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
return nil, nil
}
func (p *GitHubProvider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
return p.githubHeaders(ctx, remote, false)
}
// ServeRemote answers a request against a github_deb remote. It refreshes the
// derived metadata (bounded by mutable_ttl), serves a synthesized flat apt repo
// (Packages/Packages.gz/Release), 404s the signed index variants (the repo is
// consumed via [trusted=yes]), and 302-redirects .deb downloads to the backend
// releases_remote. Returns false only for paths it does not own.
func (p *GitHubProvider) ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, reqPath, proxyBaseURL string, store provider.RemoteMetadataStore) bool {
p.onRequest(remote, store)
// apt appends the flat-repo dist "./" verbatim, so it asks for "./Packages"
// etc.; collapse the dot-segment before matching the synthesized index.
path := normalizeIndexPath(reqPath)
switch path {
case "Packages", "Packages.gz", "Release":
p.serveIndex(w, r, remote, path, store)
return true
case "InRelease", "Release.gpg":
// Unsigned flat repo: apt consumes it with [trusted=yes]. Signal absence
// so apt falls back to the plain Release without waiting on a signature.
http.Error(w, "not found", http.StatusNotFound)
return true
}
if strings.HasSuffix(path, ".deb") {
if remote.ReleasesRemote == "" {
http.Error(w, "github_deb remote has no releases_remote configured for downloads", http.StatusInternalServerError)
return true
}
loc := strings.TrimRight(proxyBaseURL, "/") + "/api/v1/remote/" + remote.ReleasesRemote + "/" + strings.TrimLeft(path, "/")
http.Redirect(w, r, loc, http.StatusFound)
return true
}
return false
}
func (p *GitHubProvider) serveIndex(w http.ResponseWriter, r *http.Request, remote models.Remote, path string, store provider.RemoteMetadataStore) {
// Serve on a context detached from the inbound request so a client disconnect
// never cancels the metadata DB read and surfaces as a 500.
sctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), p.serveTimeout)
defer cancel()
if p.syncer != nil && !p.ensurePrimed(sctx, remote, store) {
w.Header().Set("Retry-After", "5")
http.Error(w, "metadata is being prepared, retry shortly", http.StatusServiceUnavailable)
return
}
reader, ok := store.(provider.DebMetadataReader)
if !ok {
http.Error(w, "deb metadata not available", http.StatusInternalServerError)
return
}
metas, err := reader.ListDebMetadataEntries(sctx, remote.Name)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
return
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
switch path {
case "Packages":
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
w.Write(generatePackages(metas))
case "Packages.gz":
w.Header().Set("Content-Type", "application/gzip")
w.WriteHeader(http.StatusOK)
w.Write(gzipBytes(generatePackages(metas)))
case "Release":
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
w.Write(generateRelease(metas))
}
}
// onRequest keeps a remote's derived metadata fresh off the request path.
func (p *GitHubProvider) onRequest(remote models.Remote, store provider.RemoteMetadataStore) {
if p.syncer != nil {
p.syncer.enqueue(remote, false)
return
}
p.refresh(remote, store)
}
// ensurePrimed returns true once the remote has at least one cached row. On an
// empty cache it enqueues a prime and polls briefly for it to land.
func (p *GitHubProvider) ensurePrimed(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) bool {
if !p.cacheEmpty(ctx, store, remote.Name) {
return true
}
if p.syncer != nil {
p.syncer.enqueue(remote, true)
}
deadline := time.Now().Add(p.coldWait)
for time.Now().Before(deadline) {
select {
case <-ctx.Done():
return false
case <-time.After(400 * time.Millisecond):
}
if !p.cacheEmpty(ctx, store, remote.Name) {
return true
}
}
return false
}
func (p *GitHubProvider) cacheEmpty(ctx context.Context, store provider.RemoteMetadataStore, name string) bool {
reader, ok := store.(provider.DebMetadataReader)
if !ok {
return false
}
rows, err := reader.ListDebMetadataEntries(ctx, name)
if err != nil {
return false
}
return len(rows) == 0
}
// refresh brings the derived metadata up to date without coupling the scan to
// the inbound request (legacy inline path used without a syncer / in unit tests).
func (p *GitHubProvider) refresh(remote models.Remote, store provider.RemoteMetadataStore) {
ttl := time.Duration(remote.MutableTTL) * time.Second
if ttl <= 0 {
ttl = 5 * time.Minute
}
p.mu.Lock()
last, ok := p.lastScan[remote.Name]
fresh := ok && time.Since(last) < ttl
if fresh || p.scanning[remote.Name] {
p.mu.Unlock()
return
}
p.scanning[remote.Name] = true
p.mu.Unlock()
if p.cacheEmpty(context.Background(), store, remote.Name) {
p.runScan(remote, store)
return
}
go p.runScan(remote, store)
}
func (p *GitHubProvider) runScan(remote models.Remote, store provider.RemoteMetadataStore) {
defer func() {
p.mu.Lock()
delete(p.scanning, remote.Name)
p.mu.Unlock()
}()
ctx, cancel := context.WithTimeout(context.Background(), p.scanTimeout)
defer cancel()
if err := p.scan(ctx, remote, store); err != nil {
slog.Error("github_deb: release scan failed", "remote", remote.Name, "error", err)
return
}
p.mu.Lock()
p.lastScan[remote.Name] = time.Now()
p.mu.Unlock()
}
// scan runs a full unconditional derive. Retained for the legacy inline refresh
// path and existing tests; the syncer uses scanWithState.
func (p *GitHubProvider) scan(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) error {
_, _, err := p.scanWithState(ctx, remote, store, "")
return err
}
// scanWithState derives metadata incrementally. It sends the prior releases-list
// ETag as a conditional request: a 304 means nothing changed. On a 200 it diffs
// the release assets against the cache, derives only new/changed assets, prunes
// assets that disappeared, and returns the new ETag.
func (p *GitHubProvider) scanWithState(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore, etag string) (newEtag string, changed bool, err error) {
releases, newEtag, notModified, err := p.fetchReleases(ctx, remote, etag)
if err != nil {
return etag, false, err
}
if notModified {
return etag, false, nil
}
reader, ok := store.(provider.DebMetadataReader)
if !ok {
return newEtag, false, errors.New("store does not support deb metadata reads")
}
existing, err := reader.ListDebMetadataEntries(ctx, remote.Name)
if err != nil {
return newEtag, false, err
}
existingByPath := make(map[string]provider.DebMetadata, len(existing))
for _, m := range existing {
existingByPath[m.FilePath] = m
}
allow, err := compilePatterns(remote.Patterns)
if err != nil {
return newEtag, false, err
}
seen := map[string]bool{}
for _, rel := range releases {
if rel.Draft {
continue
}
for _, asset := range rel.Assets {
if !strings.HasSuffix(strings.ToLower(asset.Name), ".deb") {
continue
}
if !matchesAny(allow, asset.Name) {
continue
}
fp := assetPath(asset)
if fp == "" {
continue
}
seen[fp] = true
if cur, ok := existingByPath[fp]; ok {
if asset.Digest == "" || cur.ContentHash == asset.Digest {
continue
}
_ = store.DeleteDebMetadata(ctx, remote.Name, fp)
}
meta, err := p.deriveAsset(ctx, remote, asset, fp)
if err != nil {
slog.Warn("github_deb: derive asset failed", "remote", remote.Name, "asset", asset.Name, "error", err)
continue
}
if err := store.InsertDebMetadata(ctx, meta); err != nil {
slog.Error("github_deb: insert metadata failed", "remote", remote.Name, "asset", asset.Name, "error", err)
continue
}
slog.Info("github_deb: derived asset", "remote", remote.Name, "name", meta.Name, "version", meta.Version, "arch", meta.Architecture)
}
}
for fp := range existingByPath {
if !seen[fp] {
_ = store.DeleteDebMetadata(ctx, remote.Name, fp)
}
}
return newEtag, true, nil
}
type ghRelease struct {
TagName string `json:"tag_name"`
Draft bool `json:"draft"`
Assets []ghAsset `json:"assets"`
}
type ghAsset struct {
Name string `json:"name"`
Size int64 `json:"size"`
BrowserDownloadURL string `json:"browser_download_url"`
Digest string `json:"digest"`
}
// fetchReleases lists a repo's releases, sending the prior ETag as If-None-Match
// on page 1 so an unchanged repo short-circuits to notModified. Every call waits
// on the shared limiter first.
func (p *GitHubProvider) fetchReleases(ctx context.Context, remote models.Remote, etag string) (all []ghRelease, newEtag string, notModified bool, err error) {
base := strings.TrimRight(remote.BaseURL, "/") + "/releases"
for page := 1; page <= p.pageCap; page++ {
u := fmt.Sprintf("%s?per_page=100&page=%d", base, page)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return nil, "", false, err
}
hdr, err := p.githubHeaders(ctx, remote, true)
if err != nil {
return nil, "", false, err
}
copyHeaders(req, hdr)
if page == 1 && etag != "" {
req.Header.Set("If-None-Match", etag)
}
if err := p.limiterWait(ctx); err != nil {
return nil, "", false, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, "", false, err
}
if page == 1 && resp.StatusCode == http.StatusNotModified {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
return nil, etag, true, nil
}
body, err := io.ReadAll(resp.Body)
respEtag := resp.Header.Get("ETag")
resp.Body.Close()
if err != nil {
return nil, "", false, err
}
if resp.StatusCode != http.StatusOK {
return nil, "", false, fmt.Errorf("github releases API %s: status %d", u, resp.StatusCode)
}
if page == 1 {
newEtag = respEtag
}
var releases []ghRelease
if err := json.Unmarshal(body, &releases); err != nil {
return nil, "", false, fmt.Errorf("decode releases: %w", err)
}
if len(releases) == 0 {
break
}
all = append(all, releases...)
if len(releases) < 100 {
break
}
}
return all, newEtag, false, nil
}
func (p *GitHubProvider) deriveAsset(ctx context.Context, remote models.Remote, asset ghAsset, fp string) (*provider.DebMetadata, error) {
control, err := p.fetchControl(ctx, remote, asset.BrowserDownloadURL)
if err != nil {
return nil, err
}
fields := parseControlFields(control)
meta := &provider.DebMetadata{
RepoName: remote.Name,
FilePath: fp,
Name: fields["Package"],
Version: fields["Version"],
Architecture: fields["Architecture"],
Control: strings.TrimRight(control, "\n"),
Size: asset.Size,
}
if meta.Name == "" {
return nil, errors.New("control missing Package field")
}
// The Packages SHA256 must be the sha256 of the whole .deb. Prefer GitHub's
// asset digest so we never download the body; only when it is absent (or not
// sha256) do we stream the asset once. MD5sum is left unset — apt verifies the
// download against SHA256 alone under [trusted=yes].
if h, ok := sha256FromDigest(asset.Digest); ok {
meta.ContentHash = "sha256:" + h
meta.SHA256 = h
} else {
h, err := p.computeSHA256(ctx, remote, asset.BrowserDownloadURL)
if err != nil {
return nil, fmt.Errorf("compute sha256: %w", err)
}
meta.ContentHash = "sha256:" + h
meta.SHA256 = h
}
return meta, nil
}
// fetchControl pulls only the front of the .deb with a ranged GET and extracts
// the control paragraph from it. control.tar sits right after the tiny
// debian-binary member, so a small prefix suffices; a prefix that truncates the
// control member doubles the range and retries.
func (p *GitHubProvider) fetchControl(ctx context.Context, remote models.Remote, downloadURL string) (string, error) {
n := p.headerInitial
for {
body, full, err := p.rangeGet(ctx, remote, downloadURL, n)
if err != nil {
return "", err
}
control, complete, perr := controlFromPrefix(body)
if perr != nil {
return "", fmt.Errorf("parse deb control: %w", perr)
}
if complete {
return control, nil
}
if full || n >= p.headerMax {
return "", fmt.Errorf("control.tar not found within %d bytes of %s", n, downloadURL)
}
n *= 2
if n > p.headerMax {
n = p.headerMax
}
}
}
// controlFromPrefix parses the ar members present in a front prefix of a .deb.
// It returns the ./control paragraph once control.tar.* is fully covered
// (complete=true); a prefix too short to cover it returns complete=false so the
// caller can widen the range. Later members (data.tar.*) are ignored.
func controlFromPrefix(prefix []byte) (control string, complete bool, err error) {
const magic = "!<arch>\n"
if len(prefix) < len(magic) {
return "", false, nil
}
if string(prefix[:len(magic)]) != magic {
return "", false, errors.New("not an ar archive")
}
off := len(magic)
for {
if off+60 > len(prefix) {
return "", false, nil
}
hdr := prefix[off : off+60]
off += 60
name := strings.TrimSuffix(strings.TrimRight(string(hdr[0:16]), " "), "/")
size, err := strconv.ParseInt(strings.TrimSpace(string(hdr[48:58])), 10, 64)
if err != nil {
return "", false, fmt.Errorf("bad ar size for %q: %w", name, err)
}
if strings.HasPrefix(name, "control.tar") {
if off+int(size) > len(prefix) {
return "", false, nil
}
tarBytes, err := decompress(name, prefix[off:off+int(size)])
if err != nil {
return "", false, err
}
c, err := readControlParagraph(tarBytes)
if err != nil {
return "", false, err
}
return c, true, nil
}
if off+int(size) > len(prefix) {
return "", false, nil
}
off += int(size)
if size%2 == 1 {
off++
}
}
}
// rangeGet returns the first n bytes of downloadURL. full is true when the
// response body was shorter than n (i.e. we already have the whole object).
func (p *GitHubProvider) rangeGet(ctx context.Context, remote models.Remote, downloadURL string, n int64) ([]byte, bool, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
if err != nil {
return nil, false, err
}
hdr, err := p.githubHeaders(ctx, remote, false)
if err != nil {
return nil, false, err
}
copyHeaders(req, hdr)
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", n-1))
if err := p.limiterWait(ctx); err != nil {
return nil, false, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, false, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
return nil, false, fmt.Errorf("range GET %s: status %d", downloadURL, resp.StatusCode)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, n))
if err != nil {
return nil, false, err
}
full := int64(len(body)) < n
return body, full, nil
}
func (p *GitHubProvider) computeSHA256(ctx context.Context, remote models.Remote, downloadURL string) (string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
if err != nil {
return "", err
}
hdr, err := p.githubHeaders(ctx, remote, false)
if err != nil {
return "", err
}
copyHeaders(req, hdr)
if err := p.limiterWait(ctx); err != nil {
return "", err
}
resp, err := p.client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("GET %s: status %d", downloadURL, resp.StatusCode)
}
h := sha256.New()
if _, err := io.Copy(h, resp.Body); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
// assetPath is the package's location relative to github.com — the path the
// backend releases_remote (base https://github.com) proxies. It doubles as the
// deb_metadata key and the Filename field in the Packages index, so a .deb
// download resolves back to this remote and redirects to the backend.
func assetPath(asset ghAsset) string {
u, err := url.Parse(asset.BrowserDownloadURL)
if err != nil {
return ""
}
return strings.TrimPrefix(u.Path, "/")
}
func sha256FromDigest(digest string) (string, bool) {
if strings.HasPrefix(digest, "sha256:") {
return strings.TrimPrefix(digest, "sha256:"), true
}
return "", false
}
// githubHeaders builds the outbound headers for a GitHub request, attaching a
// bearer credential when one is available. A per-remote credential wins; absent
// that, the process-wide server credential is used; absent both, the request is
// unauthenticated.
func (p *GitHubProvider) githubHeaders(ctx context.Context, remote models.Remote, api bool) (http.Header, error) {
h := http.Header{}
if api {
h.Set("Accept", "application/vnd.github+json")
h.Set("X-GitHub-Api-Version", "2022-11-28")
}
tok, err := p.githubToken(ctx, remote)
if err != nil {
return nil, err
}
if tok != "" {
h.Set("Authorization", "Bearer "+tok)
}
return h, nil
}
// githubToken resolves the bearer token for a remote. Precedence: a per-remote
// credential (password, then username) overrides the server credential.
func (p *GitHubProvider) githubToken(ctx context.Context, remote models.Remote) (string, error) {
if remote.Password != "" {
return remote.Password, nil
}
if remote.Username != "" {
return remote.Username, nil
}
if c := p.serverCredential(); c != nil {
return c.Token(ctx)
}
return "", nil
}
func (p *GitHubProvider) serverCredential() githubauth.Credential {
if p.serverCred != nil {
return p.serverCred
}
return githubauth.Server()
}
func copyHeaders(req *http.Request, h http.Header) {
for k, vals := range h {
for _, v := range vals {
req.Header.Add(k, v)
}
}
}
func compilePatterns(patterns []string) ([]*regexp.Regexp, error) {
var out []*regexp.Regexp
for _, p := range patterns {
re, err := regexp.Compile(p)
if err != nil {
return nil, fmt.Errorf("invalid pattern %q: %w", p, err)
}
out = append(out, re)
}
return out, nil
}
func matchesAny(res []*regexp.Regexp, s string) bool {
if len(res) == 0 {
return true
}
for _, re := range res {
if re.MatchString(s) {
return true
}
}
return false
}
+462
View File
@@ -0,0 +1,462 @@
package deb
import (
"bytes"
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeStore is an in-memory provider.RemoteMetadataStore + DebMetadataReader
// keyed by file_path, mirroring the (repo_name, file_path) uniqueness of the
// real deb_metadata table.
type fakeStore struct {
mu sync.Mutex
rows map[string]provider.DebMetadata
}
func newFakeStore() *fakeStore { return &fakeStore{rows: map[string]provider.DebMetadata{}} }
func (f *fakeStore) InsertDebMetadata(_ context.Context, m *provider.DebMetadata) error {
f.mu.Lock()
defer f.mu.Unlock()
if _, ok := f.rows[m.FilePath]; ok {
return nil // ON CONFLICT DO NOTHING
}
f.rows[m.FilePath] = *m
return nil
}
func (f *fakeStore) DeleteDebMetadata(_ context.Context, _, filePath string) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.rows, filePath)
return nil
}
func (f *fakeStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
func (f *fakeStore) DeleteRPMMetadata(context.Context, string, string) error { return nil }
func (f *fakeStore) ListRPMMetadataEntries(context.Context, string) ([]provider.RPMMetadata, error) {
return nil, nil
}
func (f *fakeStore) ListDebMetadataEntries(ctx context.Context, _ string) ([]provider.DebMetadata, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
out := make([]provider.DebMetadata, 0, len(f.rows))
for _, m := range f.rows {
out = append(out, m)
}
return out, nil
}
// githubFixture serves the releases API and the .deb asset downloads (with Range
// support) for a set of packages. digest controls whether the asset carries a
// sha256 digest (no-download path) or not (compute path).
type githubFixture struct {
srv *httptest.Server
debBytes map[string][]byte
rangeHit map[string]int
fullHit map[string]int
etag string
releasesHit int
notModHit int
releaseAuth string
assetAuth string
mu sync.Mutex
}
func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
t.Helper()
f := &githubFixture{
debBytes: map[string][]byte{},
rangeHit: map[string]int{},
fullHit: map[string]int{},
}
f.debBytes["demo_1.2-3_amd64.deb"] = testsupport.MinimalDeb("demo", "1.2-3", "amd64")
mux := http.NewServeMux()
mux.HandleFunc("/repos/acme/tools/releases", func(w http.ResponseWriter, r *http.Request) {
page := r.URL.Query().Get("page")
if page != "" && page != "1" {
w.Write([]byte("[]"))
return
}
f.mu.Lock()
f.releasesHit++
f.releaseAuth = r.Header.Get("Authorization")
etag := f.etag
if etag != "" && r.Header.Get("If-None-Match") == etag {
f.notModHit++
f.mu.Unlock()
w.WriteHeader(http.StatusNotModified)
return
}
f.mu.Unlock()
if etag != "" {
w.Header().Set("ETag", etag)
}
var assets []map[string]any
for name := range f.debBytes {
a := map[string]any{
"name": name,
"size": len(f.debBytes[name]),
"browser_download_url": f.srv.URL + "/acme/tools/releases/download/v1.2-3/" + name,
}
if withDigest {
sum := sha256.Sum256(f.debBytes[name])
a["digest"] = "sha256:" + hex.EncodeToString(sum[:])
}
assets = append(assets, a)
}
rel := []map[string]any{{"tag_name": "v1.2-3", "draft": false, "assets": assets}}
json.NewEncoder(w).Encode(rel)
})
mux.HandleFunc("/acme/tools/releases/download/", func(w http.ResponseWriter, r *http.Request) {
name := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
body, ok := f.debBytes[name]
if !ok {
http.Error(w, "not found", 404)
return
}
rng := r.Header.Get("Range")
f.mu.Lock()
f.assetAuth = r.Header.Get("Authorization")
if rng != "" {
f.rangeHit[name]++
} else {
f.fullHit[name]++
}
f.mu.Unlock()
if rng == "" {
w.WriteHeader(200)
w.Write(body)
return
}
var end int
fmt.Sscanf(rng, "bytes=0-%d", &end)
if end >= len(body)-1 {
end = len(body) - 1
}
w.Header().Set("Content-Range", fmt.Sprintf("bytes 0-%d/%d", end, len(body)))
w.Header().Set("Content-Length", strconv.Itoa(end+1))
w.WriteHeader(http.StatusPartialContent)
w.Write(body[:end+1])
})
f.srv = httptest.NewServer(mux)
t.Cleanup(f.srv.Close)
return f
}
func (f *githubFixture) remote() models.Remote {
return models.Remote{
Name: "acme-deb",
PackageType: models.PackageGitHubDeb,
BaseURL: f.srv.URL + "/repos/acme/tools",
ReleasesRemote: "github",
MutableTTL: 3600,
}
}
func newTestProvider() *GitHubProvider {
p := newGitHubProvider()
p.headerInitial = 32 // force the ranged-fetch retry loop against the tiny fixture
p.headerMax = 1 << 20
return p
}
const demoPath = "acme/tools/releases/download/v1.2-3/demo_1.2-3_amd64.deb"
func TestGitHubScanDerivesControlFromPrefixAndDigest(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
metas, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb")
if len(metas) != 1 {
t.Fatalf("want 1 metadata row, got %d", len(metas))
}
m := metas[0]
if m.Name != "demo" || m.Version != "1.2-3" || m.Architecture != "amd64" {
t.Fatalf("bad control fields: %+v", m)
}
if m.FilePath != demoPath {
t.Fatalf("FilePath = %q, want %q", m.FilePath, demoPath)
}
if int(m.Size) != len(fx.debBytes["demo_1.2-3_amd64.deb"]) {
t.Fatalf("Size = %d, want %d", m.Size, len(fx.debBytes["demo_1.2-3_amd64.deb"]))
}
sum := sha256.Sum256(fx.debBytes["demo_1.2-3_amd64.deb"])
if m.SHA256 != hex.EncodeToString(sum[:]) {
t.Fatalf("SHA256 = %q, want digest", m.SHA256)
}
if m.ContentHash != "sha256:"+hex.EncodeToString(sum[:]) {
t.Fatalf("ContentHash = %q", m.ContentHash)
}
if m.MD5 != "" {
t.Fatalf("MD5 should be unset for metadata-only derive, got %q", m.MD5)
}
if fx.fullHit["demo_1.2-3_amd64.deb"] != 0 {
t.Fatalf("expected no full download when digest present, got %d", fx.fullHit["demo_1.2-3_amd64.deb"])
}
if fx.rangeHit["demo_1.2-3_amd64.deb"] == 0 {
t.Fatalf("expected ranged control fetch")
}
if !strings.Contains(m.Control, "Package: demo") {
t.Fatalf("raw control not captured: %q", m.Control)
}
}
func TestGitHubChecksumComputedWhenDigestAbsent(t *testing.T) {
fx := newGitHubFixture(t, false)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
metas, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb")
if len(metas) != 1 {
t.Fatalf("want 1 row, got %d", len(metas))
}
sum := sha256.Sum256(fx.debBytes["demo_1.2-3_amd64.deb"])
if metas[0].SHA256 != hex.EncodeToString(sum[:]) {
t.Fatalf("computed checksum mismatch: %q", metas[0].SHA256)
}
if fx.fullHit["demo_1.2-3_amd64.deb"] == 0 {
t.Fatalf("expected a full download to compute sha256 when digest absent")
}
}
func TestGitHubServeRemoteIndexAndRedirect(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
const proxyBase = "https://artifactapi.example"
// Release is served and triggers the initial scan.
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Release", nil)
if !p.ServeRemote(rec, req, remote, "Release", proxyBase, store) {
t.Fatal("ServeRemote did not handle Release")
}
if rec.Code != 200 || !strings.Contains(rec.Body.String(), "Architectures:") {
t.Fatalf("Release bad: code=%d body=%s", rec.Code, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "amd64") {
t.Fatalf("Release missing arch: %s", rec.Body.String())
}
// Packages carries the package with a Filename that is the github-relative
// download path (so it resolves back to this remote and redirects).
rec = httptest.NewRecorder()
req = httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, "Packages", proxyBase, store) {
t.Fatal("ServeRemote did not handle Packages")
}
pkgs := rec.Body.String()
if !strings.Contains(pkgs, "Package: demo") {
t.Fatalf("Packages missing package: %s", pkgs)
}
if !strings.Contains(pkgs, "Filename: "+demoPath) {
t.Fatalf("Packages missing/incorrect Filename: %s", pkgs)
}
if !strings.Contains(pkgs, "SHA256: ") {
t.Fatalf("Packages missing SHA256: %s", pkgs)
}
if strings.Contains(pkgs, "MD5sum:") {
t.Fatalf("Packages should omit empty MD5sum: %s", pkgs)
}
// Packages.gz decompresses to the same content.
rec = httptest.NewRecorder()
req = httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, "Packages.gz", proxyBase, store) {
t.Fatal("ServeRemote did not handle Packages.gz")
}
gz, err := gzip.NewReader(rec.Body)
if err != nil {
t.Fatalf("gzip: %v", err)
}
unz, _ := io.ReadAll(gz)
if !strings.Contains(string(unz), "Package: demo") {
t.Fatalf("Packages.gz missing package: %s", unz)
}
// InRelease/Release.gpg 404 (unsigned, consumed via [trusted=yes]).
for _, sp := range []string{"InRelease", "Release.gpg"} {
rec = httptest.NewRecorder()
req = httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, sp, proxyBase, store) {
t.Fatalf("ServeRemote did not handle %s", sp)
}
if rec.Code != http.StatusNotFound {
t.Fatalf("%s want 404, got %d", sp, rec.Code)
}
}
// A .deb request redirects to the backend releases_remote.
rec = httptest.NewRecorder()
req = httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/"+demoPath, nil)
if !p.ServeRemote(rec, req, remote, demoPath, proxyBase, store) {
t.Fatal("ServeRemote did not handle .deb")
}
if rec.Code != http.StatusFound {
t.Fatalf("want 302, got %d", rec.Code)
}
wantLoc := proxyBase + "/api/v1/remote/github/" + demoPath
if got := rec.Header().Get("Location"); got != wantLoc {
t.Fatalf("Location = %q, want %q", got, wantLoc)
}
}
// Real apt appends the flat-repo dist "./" verbatim, so the metadata-only remote
// receives "./Packages" / "./Release"; ServeRemote must collapse the dot-segment
// and synthesize the same index as the un-prefixed request.
func TestGitHubServeRemoteAptDotSegment(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
const proxyBase = "https://artifactapi.example"
serve := func(path string) *httptest.ResponseRecorder {
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/"+path, nil)
if !p.ServeRemote(rec, req, remote, path, proxyBase, store) {
t.Fatalf("ServeRemote did not handle %q", path)
}
return rec
}
// Packages is deterministic: byte-identical to the un-prefixed request.
plain, dotted := serve("Packages"), serve("./Packages")
if plain.Code != 200 || dotted.Code != 200 {
t.Fatalf("Packages: plain=%d dotted=%d, want 200/200", plain.Code, dotted.Code)
}
if !strings.Contains(dotted.Body.String(), "Package: demo") {
t.Fatalf("./Packages missing synthesized body: %s", dotted.Body.String())
}
if !bytes.Equal(plain.Body.Bytes(), dotted.Body.Bytes()) {
t.Error("./Packages body differs from Packages body")
}
// Release carries a time.Now() Date: header; compare the rest.
rPlain, rDotted := serve("Release"), serve("./Release")
if rPlain.Code != 200 || rDotted.Code != 200 {
t.Fatalf("Release: plain=%d dotted=%d, want 200/200", rPlain.Code, rDotted.Code)
}
if stripDate(rPlain.Body.String()) != stripDate(rDotted.Body.String()) {
t.Error("./Release body differs from Release body (ignoring Date)")
}
}
// A canceled inbound request must still serve the warm cache (detached context),
// not turn the metadata read into a 500.
func TestGitHubServeRemoteCanceledRequestServesCache(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("warm scan: %v", err)
}
p.mu.Lock()
p.lastScan[remote.Name] = time.Now()
p.mu.Unlock()
ctx, cancel := context.WithCancel(context.Background())
cancel()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Packages", nil).WithContext(ctx)
if !p.ServeRemote(rec, req, remote, "Packages", "https://x", store) {
t.Fatal("ServeRemote did not handle Packages")
}
if rec.Code != http.StatusOK {
t.Fatalf("canceled request must serve cache, not error; got code=%d body=%s", rec.Code, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "Package: demo") {
t.Fatalf("expected Packages served from cache, got %s", rec.Body.String())
}
}
func TestGitHubServeRemoteRedirectRequiresReleasesRemote(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
remote.ReleasesRemote = ""
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, demoPath, "https://x", store) {
t.Fatal("expected handled")
}
if rec.Code != http.StatusInternalServerError {
t.Fatalf("want 500 when releases_remote unset, got %d", rec.Code)
}
}
func TestGitHubScanPrunesRemovedAssets(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
if rows, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb"); len(rows) != 1 {
t.Fatalf("want 1 row after first scan, got %d", len(rows))
}
delete(fx.debBytes, "demo_1.2-3_amd64.deb")
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("rescan: %v", err)
}
if rows, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb"); len(rows) != 0 {
t.Fatalf("want 0 rows after prune, got %d", len(rows))
}
}
func TestGitHubAssetPatternFilter(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.debBytes["other_9_arm64.deb"] = testsupport.MinimalDeb("other", "9", "arm64")
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
remote.Patterns = []string{`^demo_.*_amd64\.deb$`}
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("scan: %v", err)
}
rows, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb")
if len(rows) != 1 || rows[0].Name != "demo" {
t.Fatalf("pattern filter failed, rows=%+v", rows)
}
}
+238
View File
@@ -0,0 +1,238 @@
package deb
import (
"context"
"crypto/rand"
"encoding/hex"
"log/slog"
"os"
"sync"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
const (
syncLeaseDuration = 15 * time.Minute
defaultSyncFreshness = 5 * time.Minute
jobQueueDepth = 256
)
// SyncStore is the persistence surface the deb syncer needs: the metadata cache
// it primes plus the shared sync-state coordination (remote enumeration and the
// per-remote lease). *database.DB satisfies it.
type SyncStore interface {
provider.RemoteMetadataStore
ListGitHubDebRemotes(ctx context.Context) ([]models.Remote, error)
ClaimGitHubDebSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (claimed bool, etag string, err error)
ReleaseGitHubDebSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error
}
// SyncConfig tunes the shared syncer. Zero values fall back to safe defaults.
type SyncConfig struct {
RatePerSec float64
Burst int
Workers int
PollInterval time.Duration
}
type syncJob struct {
remote models.Remote
prime bool
}
// Syncer is the single per-process background worker that keeps every github_deb
// remote's derived metadata fresh. It owns a deduped work queue, a pool of
// workers, and a global token-bucket rate limiter shared across all remotes and
// bound onto the github_deb provider. Periodic checks are gated by a shared DB
// lease so, across replicas, only one performs each scan.
type Syncer struct {
store SyncStore
prov *GitHubProvider
limiter *rate.Limiter
cfg SyncConfig
owner string
jobs chan syncJob
mu sync.Mutex
active map[string]bool
}
// NewSyncer builds the syncer bound to the process-wide github_deb provider
// singleton. Call Run to start it.
func NewSyncer(store SyncStore, cfg SyncConfig) *Syncer {
return newSyncer(store, gitHubProvider, cfg)
}
func newSyncer(store SyncStore, prov *GitHubProvider, cfg SyncConfig) *Syncer {
if cfg.RatePerSec <= 0 {
cfg.RatePerSec = 1
}
if cfg.Burst <= 0 {
cfg.Burst = 5
}
if cfg.Workers <= 0 {
cfg.Workers = 3
}
if cfg.PollInterval <= 0 {
cfg.PollInterval = 60 * time.Second
}
lim := rate.NewLimiter(rate.Limit(cfg.RatePerSec), cfg.Burst)
s := &Syncer{
store: store,
prov: prov,
limiter: lim,
cfg: cfg,
owner: leaseOwner(),
jobs: make(chan syncJob, jobQueueDepth),
active: map[string]bool{},
}
prov.limiter = lim
prov.syncer = s
return s
}
// Run starts the worker pool and the periodic scheduler and blocks until ctx is
// canceled, at which point it drains in-flight scans and returns.
func (s *Syncer) Run(ctx context.Context) {
slog.Info("github_deb syncer started",
"rate_per_sec", s.cfg.RatePerSec, "burst", s.cfg.Burst,
"workers", s.cfg.Workers, "poll_interval", s.cfg.PollInterval, "owner", s.owner)
var wg sync.WaitGroup
for i := 0; i < s.cfg.Workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
s.worker(ctx)
}()
}
ticker := time.NewTicker(s.cfg.PollInterval)
defer ticker.Stop()
s.schedule(ctx)
for {
select {
case <-ctx.Done():
wg.Wait()
slog.Info("github_deb syncer stopped")
return
case <-ticker.C:
s.schedule(ctx)
}
}
}
// schedule enqueues a periodic check for every github_deb remote. The DB lease
// enforces the per-remote mutable_ttl cadence and cross-replica coordination.
func (s *Syncer) schedule(ctx context.Context) {
remotes, err := s.store.ListGitHubDebRemotes(ctx)
if err != nil {
slog.Error("github_deb syncer: list remotes", "error", err)
return
}
for _, r := range remotes {
s.enqueue(r, false)
}
}
// EnqueuePrime queues an immediate background prime for a freshly created remote.
func (s *Syncer) EnqueuePrime(remote models.Remote) {
if s == nil {
return
}
s.enqueue(remote, true)
}
// enqueue adds a job unless the remote is already queued or in-flight, coalescing
// duplicate requests down to one scan. It never blocks.
func (s *Syncer) enqueue(remote models.Remote, prime bool) {
s.mu.Lock()
if s.active[remote.Name] {
s.mu.Unlock()
return
}
s.active[remote.Name] = true
s.mu.Unlock()
select {
case s.jobs <- syncJob{remote: remote, prime: prime}:
default:
s.mu.Lock()
delete(s.active, remote.Name)
s.mu.Unlock()
}
}
func (s *Syncer) worker(ctx context.Context) {
for {
select {
case <-ctx.Done():
return
case job := <-s.jobs:
s.process(ctx, job)
}
}
}
// process claims the shared lease and, if won, runs an incremental scan. Losing
// the claim (another replica scanning, or not yet due) is a no-op.
func (s *Syncer) process(ctx context.Context, job syncJob) {
defer func() {
s.mu.Lock()
delete(s.active, job.remote.Name)
s.mu.Unlock()
}()
freshness := time.Duration(job.remote.MutableTTL) * time.Second
if freshness <= 0 {
freshness = defaultSyncFreshness
}
if job.prime {
freshness = 0
}
claimed, etag, err := s.store.ClaimGitHubDebSyncLease(ctx, job.remote.Name, s.owner, freshness, syncLeaseDuration)
if err != nil {
slog.Error("github_deb syncer: claim lease", "remote", job.remote.Name, "error", err)
return
}
if !claimed {
return
}
scanCtx, cancel := context.WithTimeout(ctx, s.prov.scanTimeout)
defer cancel()
newEtag, changed, scanErr := s.prov.scanWithState(scanCtx, job.remote, s.store, etag)
releaseEtag := etag
if scanErr == nil {
releaseEtag = newEtag
} else {
slog.Error("github_deb syncer: scan failed", "remote", job.remote.Name, "error", scanErr)
}
relCtx, relCancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer relCancel()
if err := s.store.ReleaseGitHubDebSyncLease(relCtx, job.remote.Name, s.owner, releaseEtag, time.Now()); err != nil {
slog.Warn("github_deb syncer: release lease", "remote", job.remote.Name, "error", err)
}
if scanErr == nil && changed {
slog.Info("github_deb syncer: refreshed", "remote", job.remote.Name, "prime", job.prime)
}
}
// leaseOwner is a per-replica identity for the lease: hostname plus a random
// suffix so restarts and colocated replicas never collide.
func leaseOwner() string {
host, _ := os.Hostname()
var b [6]byte
_, _ = rand.Read(b[:])
return host + "-" + hex.EncodeToString(b[:])
}
+300
View File
@@ -0,0 +1,300 @@
package deb
import (
"context"
"net/http"
"net/http/httptest"
"sync"
"testing"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeSyncStore is an in-memory SyncStore: the metadata cache (via the embedded
// fakeStore) plus the shared sync-state lease, whose claim mirrors the atomic
// semantics of the real SQL (recency gate AND no live lease).
type fakeSyncStore struct {
*fakeStore
mu sync.Mutex
remotes []models.Remote
leaseOwner map[string]string
leaseExp map[string]time.Time
lastSynced map[string]time.Time
etags map[string]string
}
func newFakeSyncStore() *fakeSyncStore {
return &fakeSyncStore{
fakeStore: newFakeStore(),
leaseOwner: map[string]string{},
leaseExp: map[string]time.Time{},
lastSynced: map[string]time.Time{},
etags: map[string]string{},
}
}
func (f *fakeSyncStore) ListGitHubDebRemotes(_ context.Context) ([]models.Remote, error) {
f.mu.Lock()
defer f.mu.Unlock()
return append([]models.Remote(nil), f.remotes...), nil
}
func (f *fakeSyncStore) ClaimGitHubDebSyncLease(_ context.Context, name, owner string, freshness, lease time.Duration) (bool, string, error) {
f.mu.Lock()
defer f.mu.Unlock()
now := time.Now()
ls, hasLS := f.lastSynced[name]
exp, hasExp := f.leaseExp[name]
freshOK := !hasLS || now.Sub(ls) >= freshness
leaseOK := !hasExp || exp.Before(now)
if freshOK && leaseOK {
f.leaseOwner[name] = owner
f.leaseExp[name] = now.Add(lease)
return true, f.etags[name], nil
}
return false, "", nil
}
func (f *fakeSyncStore) ReleaseGitHubDebSyncLease(_ context.Context, name, owner, etag string, syncedAt time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.leaseOwner[name] != owner {
return nil
}
f.lastSynced[name] = syncedAt
f.etags[name] = etag
delete(f.leaseOwner, name)
delete(f.leaseExp, name)
return nil
}
func testSyncConfig() SyncConfig {
return SyncConfig{RatePerSec: 1000, Burst: 100, Workers: 1, PollInterval: time.Hour}
}
// (a) A 304 conditional response must derive nothing: no asset fetches and
// changed=false, so an unchanged repo is nearly free.
func TestSyncerConditionalNotModifiedSkipsDerive(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
etag1, changed, err := p.scanWithState(context.Background(), remote, store, "")
if err != nil {
t.Fatalf("first scan: %v", err)
}
if !changed || etag1 != `"v1"` {
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag1)
}
priorRange := fx.rangeHit["demo_1.2-3_amd64.deb"]
if priorRange == 0 {
t.Fatal("first scan should have fetched the asset control")
}
etag2, changed2, err := p.scanWithState(context.Background(), remote, store, etag1)
if err != nil {
t.Fatalf("second scan: %v", err)
}
if changed2 {
t.Fatal("304 scan must report changed=false")
}
if etag2 != etag1 {
t.Fatalf("etag changed across 304: %q -> %q", etag1, etag2)
}
if fx.notModHit != 1 {
t.Fatalf("want exactly one 304 releases response, got %d", fx.notModHit)
}
if got := fx.rangeHit["demo_1.2-3_amd64.deb"]; got != priorRange {
t.Fatalf("304 scan re-fetched asset control: %d -> %d", priorRange, got)
}
}
// (b) On a real change, only the newly added asset is derived.
func TestSyncerIncrementalDerivesOnlyNewAsset(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
if _, _, err := p.scanWithState(context.Background(), remote, store, ""); err != nil {
t.Fatalf("first scan: %v", err)
}
demoRange := fx.rangeHit["demo_1.2-3_amd64.deb"]
fx.debBytes["other_9_arm64.deb"] = testsupport.MinimalDeb("other", "9", "arm64")
fx.etag = `"v2"`
if _, changed, err := p.scanWithState(context.Background(), remote, store, `"v1"`); err != nil || !changed {
t.Fatalf("second scan changed=%v err=%v", changed, err)
}
rows, _ := store.ListDebMetadataEntries(context.Background(), remote.Name)
if len(rows) != 2 {
t.Fatalf("want 2 cached rows after incremental derive, got %d", len(rows))
}
if got := fx.rangeHit["demo_1.2-3_amd64.deb"]; got != demoRange {
t.Fatalf("already-cached asset was re-fetched: %d -> %d", demoRange, got)
}
if fx.rangeHit["other_9_arm64.deb"] == 0 {
t.Fatal("newly added asset was not derived")
}
}
// (c) The shared limiter caps the request rate.
func TestRateLimiterCapsRequestRate(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
p.limiter = rate.NewLimiter(rate.Every(120*time.Millisecond), 1)
remote := fx.remote()
start := time.Now()
for i := 0; i < 3; i++ {
if _, _, _, err := p.fetchReleases(context.Background(), remote, ""); err != nil {
t.Fatalf("fetchReleases %d: %v", i, err)
}
}
if elapsed := time.Since(start); elapsed < 200*time.Millisecond {
t.Fatalf("rate limiter did not throttle: 3 calls took %v, want >= 200ms", elapsed)
}
}
// (d) Concurrent enqueues for the same remote coalesce to a single queued job.
func TestSyncerEnqueueDedup(t *testing.T) {
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := models.Remote{Name: "acme-deb", PackageType: models.PackageGitHubDeb, MutableTTL: 3600}
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() { defer wg.Done(); s.enqueue(remote, false) }()
}
wg.Wait()
if got := len(s.jobs); got != 1 {
t.Fatalf("want exactly 1 coalesced job, got %d", got)
}
}
// (e) Prime-on-create enqueues a prime job.
func TestSyncerEnqueuePrime(t *testing.T) {
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := models.Remote{Name: "acme-deb", PackageType: models.PackageGitHubDeb, MutableTTL: 3600}
s.EnqueuePrime(remote)
select {
case job := <-s.jobs:
if !job.prime || job.remote.Name != "acme-deb" {
t.Fatalf("bad prime job: %+v", job)
}
default:
t.Fatal("EnqueuePrime did not enqueue a job")
}
}
// (f) A held lease prevents a second replica from scanning.
func TestSyncerLeasePreventsSecondReplica(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := fx.remote()
claimed, _, err := store.ClaimGitHubDebSyncLease(context.Background(), remote.Name, "replica-1", time.Duration(remote.MutableTTL)*time.Second, syncLeaseDuration)
if err != nil || !claimed {
t.Fatalf("replica-1 claim: claimed=%v err=%v", claimed, err)
}
s.process(context.Background(), syncJob{remote: remote})
if fx.releasesHit != 0 {
t.Fatalf("second replica scanned while lease held: %d releases calls", fx.releasesHit)
}
if rows, _ := store.ListDebMetadataEntries(context.Background(), remote.Name); len(rows) != 0 {
t.Fatalf("second replica derived metadata while lease held: %d rows", len(rows))
}
}
// With the syncer wired and the cache empty, an index request enqueues a prime
// and returns a retryable 503 when it has not landed within the cold wait.
func TestServeRemoteColdStartReturns503(t *testing.T) {
fx := newGitHubFixture(t, true)
store := newFakeSyncStore()
p := newTestProvider()
p.coldWait = 300 * time.Millisecond
_ = newSyncer(store, p, testSyncConfig()) // binds p.syncer, but no workers running
remote := fx.remote()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Packages", nil)
if !p.ServeRemote(rec, req, remote, "Packages", "https://x", store) {
t.Fatal("ServeRemote did not handle Packages")
}
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("cold empty cache must return 503, got %d", rec.Code)
}
if rec.Header().Get("Retry-After") == "" {
t.Fatal("503 should carry Retry-After")
}
if got := len(p.syncer.jobs); got != 1 {
t.Fatalf("cold start did not enqueue a prime, jobs=%d", got)
}
}
// With the cache warm, the same request serves the index immediately (no 503).
func TestServeRemoteWarmCacheServesImmediately(t *testing.T) {
fx := newGitHubFixture(t, true)
store := newFakeSyncStore()
p := newTestProvider()
_ = newSyncer(store, p, testSyncConfig())
remote := fx.remote()
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("warm scan: %v", err)
}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Packages", nil)
if !p.ServeRemote(rec, req, remote, "Packages", "https://x", store) {
t.Fatal("ServeRemote did not handle Packages")
}
if rec.Code != http.StatusOK {
t.Fatalf("warm cache must serve 200, got %d body=%s", rec.Code, rec.Body.String())
}
}
// A prime job (freshness 0) runs even right after a sync; a periodic job at the
// same moment is gated by the recency window.
func TestSyncerPrimeBypassesRecencyPeriodicDoesNot(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := fx.remote()
var _ provider.RemoteMetadataStore = store
s.process(context.Background(), syncJob{remote: remote, prime: true})
if rows, _ := store.ListDebMetadataEntries(context.Background(), remote.Name); len(rows) != 1 {
t.Fatalf("prime did not derive: %d rows", len(rows))
}
releasesAfterPrime := fx.releasesHit
s.process(context.Background(), syncJob{remote: remote, prime: false})
if fx.releasesHit != releasesAfterPrime {
t.Fatalf("periodic scan ran inside recency window: %d -> %d releases calls", releasesAfterPrime, fx.releasesHit)
}
}
+35 -1
View File
@@ -4,9 +4,11 @@ import (
"context"
"encoding/base64"
"net/http"
"net/url"
"path"
"strings"
"git.unkin.net/unkin/artifactapi/internal/githubauth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -59,10 +61,42 @@ func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte,
return nil, nil
}
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
// AuthHeaders authenticates outbound requests. A per-remote username/password
// (Basic auth) takes precedence. Otherwise, when the remote points at a GitHub
// host (e.g. a releases_remote proxying private release assets), the process-wide
// GitHub credential is attached as a bearer token so private downloads work.
func (p *Provider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
h := http.Header{}
if remote.Username != "" {
h.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(remote.Username+":"+remote.Password)))
return h, nil
}
if isGitHubHost(remote.BaseURL) {
if c := githubauth.Server(); c != nil {
tok, err := c.Token(ctx)
if err != nil {
return nil, err
}
if tok != "" {
h.Set("Authorization", "Bearer "+tok)
}
}
}
return h, nil
}
// isGitHubHost reports whether rawURL targets a GitHub API/download host that
// accepts the server credential. objects.githubusercontent.com is deliberately
// excluded: release-asset downloads 302-redirect there with a pre-signed URL
// that must not carry an Authorization header.
func isGitHubHost(rawURL string) bool {
u, err := url.Parse(rawURL)
if err != nil {
return false
}
switch strings.ToLower(u.Hostname()) {
case "github.com", "www.github.com", "api.github.com", "codeload.github.com", "uploads.github.com":
return true
}
return false
}
+45
View File
@@ -4,11 +4,56 @@ import (
"context"
"testing"
"git.unkin.net/unkin/artifactapi/internal/githubauth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/provider/generic"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type staticCred string
func (s staticCred) Token(context.Context) (string, error) { return string(s), nil }
func TestProvider_AuthHeaders_GitHubServerCredential(t *testing.T) {
githubauth.SetServer(staticCred("ghs_server"))
t.Cleanup(func() { githubauth.SetServer(nil) })
p := &generic.Provider{}
h, err := p.AuthHeaders(context.Background(), models.Remote{BaseURL: "https://github.com"})
if err != nil {
t.Fatalf("auth headers: %v", err)
}
if h.Get("Authorization") != "Bearer ghs_server" {
t.Fatalf("Authorization = %q, want Bearer ghs_server", h.Get("Authorization"))
}
}
func TestProvider_AuthHeaders_NonGitHubHostNoServerCredential(t *testing.T) {
githubauth.SetServer(staticCred("ghs_server"))
t.Cleanup(func() { githubauth.SetServer(nil) })
p := &generic.Provider{}
h, _ := p.AuthHeaders(context.Background(), models.Remote{BaseURL: "https://example.com/downloads"})
if h.Get("Authorization") != "" {
t.Fatalf("server credential must not be sent to non-github host, got %q", h.Get("Authorization"))
}
}
func TestProvider_AuthHeaders_PerRemoteOverridesServerCredential(t *testing.T) {
githubauth.SetServer(staticCred("ghs_server"))
t.Cleanup(func() { githubauth.SetServer(nil) })
p := &generic.Provider{}
h, _ := p.AuthHeaders(context.Background(), models.Remote{
BaseURL: "https://github.com",
Username: "user",
Password: "pass",
})
if got := h.Get("Authorization"); got != "Basic dXNlcjpwYXNz" {
t.Fatalf("per-remote Basic auth must win, got %q", got)
}
}
func TestProvider_Type(t *testing.T) {
p := &generic.Provider{}
if p.Type() != models.PackageGeneric {
+46
View File
@@ -61,16 +61,60 @@ type PostDeleteHook interface {
type MetadataStore interface {
InsertRPMMetadata(ctx context.Context, meta *RPMMetadata) error
InsertDebMetadata(ctx context.Context, meta *DebMetadata) error
}
// RemoteServer lets a remote provider fully answer a request itself instead of
// going through the byte-proxy engine. It is the remote-side analog of
// LocalIndexer: a metadata-only remote (e.g. github_rpm) uses it to synthesize
// repodata from derived per-asset metadata and to redirect package downloads to
// a backend remote, without ever precaching the packages. Returning false lets
// the normal proxy path take over.
type RemoteServer interface {
ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store RemoteMetadataStore) bool
}
// RemoteMetadataStore is the persistence surface a RemoteServer needs to cache
// and read the metadata it derives per upstream asset. *database.DB satisfies it.
type RemoteMetadataStore interface {
RPMMetadataReader
MetadataStore
MetadataDeleter
}
type MetadataDeleter interface {
DeleteRPMMetadata(ctx context.Context, repoName, filePath string) error
DeleteDebMetadata(ctx context.Context, repoName, filePath string) error
}
type RPMMetadataReader interface {
ListRPMMetadataEntries(ctx context.Context, repoName string) ([]RPMMetadata, error)
}
// DebMetadataReader is the read surface the deb LocalIndexer needs to
// regenerate a flat apt repository (Packages/Release) from stored rows.
// *database.DB satisfies it; ServeLocalIndex type-asserts the FileStore to it,
// mirroring how the rpm provider reaches its RPMMetadataReader.
type DebMetadataReader interface {
ListDebMetadataEntries(ctx context.Context, repoName string) ([]DebMetadata, error)
}
// DebMetadata is the derived per-package metadata for a Debian .deb, carrying
// the full raw control stanza so the Packages index can be regenerated
// faithfully alongside the computed size/md5/sha256 apt requires.
type DebMetadata struct {
RepoName string
FilePath string
ContentHash string
Name string
Version string
Architecture string
Control string
Size int64
MD5 string
SHA256 string
}
type RPMMetadata struct {
RepoName string
FilePath string
@@ -93,6 +137,8 @@ type RPMMetadata struct {
Packager string
Requires []RPMDep
Provides []RPMDep
Conflicts []RPMDep
Obsoletes []RPMDep
Files []RPMFile
Changelogs []RPMChangelog
}
+732
View File
@@ -0,0 +1,732 @@
package rpm
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/url"
"regexp"
"strings"
"sync"
"time"
rpmlib "github.com/cavaliergopher/rpm"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/githubauth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// gitHubProvider is the process-wide singleton. The background Syncer binds its
// shared rate limiter and work queue onto this instance so the request path and
// the syncer drive the same derive machinery.
var gitHubProvider = newGitHubProvider()
func init() {
provider.Register(gitHubProvider)
}
// Tuning knobs for the no-precache header fetch. Fields (not consts) so tests
// can shrink them against small fixtures.
const (
defaultHeaderRangeInitial = 1 << 20 // 1 MiB — covers the header of almost every RPM
defaultHeaderRangeMax = 16 << 20 // 16 MiB — give up past this and skip the asset
defaultReleasePageCap = 10 // 100 releases/page * 10 pages
// defaultScanTimeout bounds a detached background scan (which may do one
// ranged fetch per asset across every release) so it can never run forever.
defaultScanTimeout = 10 * time.Minute
// defaultServeTimeout bounds a repodata DB read served on a detached context.
defaultServeTimeout = 30 * time.Second
// defaultColdWait bounds how long a repodata request blocks waiting for a
// just-enqueued prime to populate an empty cache before returning a
// retryable 503. Kept short so a client never hangs on a rate-limited derive
// of a large repo; small repos usually prime within this window.
defaultColdWait = 8 * time.Second
)
// GitHubProvider is a metadata-only remote: it scans a GitHub repo's releases
// for .rpm assets, derives per-asset RPM metadata via a ranged header fetch
// (never downloading whole packages), synthesizes yum repodata from that cached
// metadata, and redirects package downloads to a backend "releases_remote"
// (the generic github.com remote) that serves the actual bytes.
type GitHubProvider struct {
client *http.Client
headerInitial int64
headerMax int64
pageCap int
scanTimeout time.Duration
serveTimeout time.Duration
coldWait time.Duration
// limiter, when set by the Syncer, gates every GitHub HTTP call (releases
// list + each ranged asset fetch) through a single process-wide token bucket.
// nil means unlimited (direct provider use / unit tests).
limiter *rate.Limiter
// syncer, when set, routes freshness refresh and cold-start priming through
// the shared background work queue instead of an inline per-replica scan.
syncer *Syncer
// serverCred overrides the process-wide GitHub credential for this provider
// instance. nil falls back to githubauth.Server(); set directly in tests.
serverCred githubauth.Credential
mu sync.Mutex
scanning map[string]bool
lastScan map[string]time.Time
}
func newGitHubProvider() *GitHubProvider {
return &GitHubProvider{
client: &http.Client{},
headerInitial: defaultHeaderRangeInitial,
headerMax: defaultHeaderRangeMax,
pageCap: defaultReleasePageCap,
scanTimeout: defaultScanTimeout,
serveTimeout: defaultServeTimeout,
coldWait: defaultColdWait,
scanning: map[string]bool{},
lastScan: map[string]time.Time{},
}
}
// limiterWait blocks until the shared rate limiter grants a token, or returns
// the context error if it is canceled first. A nil limiter is a no-op.
func (p *GitHubProvider) limiterWait(ctx context.Context) error {
if p.limiter == nil {
return nil
}
return p.limiter.Wait(ctx)
}
func (p *GitHubProvider) Type() models.PackageType { return models.PackageGitHubRPM }
// Classify/ContentType/UpstreamURL/RewriteResponse/AuthHeaders satisfy the
// Provider interface. The proxy engine never reaches them for this type because
// ServeRemote handles every request, but they must exist for registry lookup.
func (p *GitHubProvider) Classify(path string) provider.Mutability {
if strings.HasPrefix(path, "repodata/") {
return provider.Mutable
}
return provider.Immutable
}
func (p *GitHubProvider) ContentType(path string) string {
switch {
case strings.HasSuffix(path, ".rpm"):
return "application/x-rpm"
case strings.HasSuffix(path, ".xml.gz"):
return "application/gzip"
case strings.HasSuffix(path, ".xml"):
return "application/xml"
}
return "application/octet-stream"
}
func (p *GitHubProvider) UpstreamURL(remote models.Remote, path string) string {
return strings.TrimRight(remote.BaseURL, "/") + "/" + strings.TrimLeft(path, "/")
}
func (p *GitHubProvider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
return nil, nil
}
func (p *GitHubProvider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
return p.githubHeaders(ctx, remote, false)
}
// ServeRemote answers a request against a github_rpm remote. It refreshes the
// derived metadata (bounded by mutable_ttl), serves synthesized repodata, and
// 302-redirects .rpm downloads to the backend releases_remote. Returns false
// only for paths it does not own, letting the normal proxy path take over.
func (p *GitHubProvider) ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store provider.RemoteMetadataStore) bool {
p.onRequest(remote, store)
if strings.HasPrefix(path, "repodata/") {
// Serve repodata on a context detached from the inbound request: a
// client disconnect (e.g. dnf makecache timing out) must never cancel
// the metadata DB read and surface as a 500.
sctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), p.serveTimeout)
defer cancel()
sr := r.WithContext(sctx)
// Cold start: with the syncer wired, an empty cache means the prime has
// not landed yet. Enqueue it and wait briefly rather than serving empty
// repodata; if it still has not primed, return a retryable 503.
if p.syncer != nil && !p.ensurePrimed(sctx, remote, store) {
w.Header().Set("Retry-After", "5")
http.Error(w, "metadata is being prepared, retry shortly", http.StatusServiceUnavailable)
return true
}
tail := strings.TrimPrefix(path, "repodata/")
lp := &Provider{}
switch {
case tail == "repomd.xml":
lp.serveRepomd(w, sr, store, remote.Name)
case strings.HasSuffix(tail, "-primary.xml.gz"):
lp.servePrimary(w, sr, store, remote.Name)
case strings.HasSuffix(tail, "-filelists.xml.gz"):
lp.serveFilelists(w, sr, store, remote.Name)
case strings.HasSuffix(tail, "-other.xml.gz"):
lp.serveOther(w, sr, store, remote.Name)
default:
http.Error(w, "not found", http.StatusNotFound)
}
return true
}
if strings.HasSuffix(path, ".rpm") {
if remote.ReleasesRemote == "" {
http.Error(w, "github_rpm remote has no releases_remote configured for downloads", http.StatusInternalServerError)
return true
}
loc := strings.TrimRight(proxyBaseURL, "/") + "/api/v1/remote/" + remote.ReleasesRemote + "/" + strings.TrimLeft(path, "/")
http.Redirect(w, r, loc, http.StatusFound)
return true
}
return false
}
// onRequest keeps a remote's derived metadata fresh off the request path. With
// the background syncer wired it enqueues a deduped, rate-limited, lease-gated
// refresh and returns immediately; the request always serves the current cache.
// Without a syncer (direct provider use / unit tests) it falls back to the
// legacy inline single-flight scan.
func (p *GitHubProvider) onRequest(remote models.Remote, store provider.RemoteMetadataStore) {
if p.syncer != nil {
p.syncer.enqueue(remote, false)
return
}
p.refresh(remote, store)
}
// ensurePrimed returns true once the remote has at least one cached metadata
// row. On an empty cache it enqueues a prime and polls briefly for it to land,
// so the very first client after a remote is created gets real repodata instead
// of an empty index or a blocking multi-minute derive. Returns false if the
// cache is still empty after the bounded wait.
func (p *GitHubProvider) ensurePrimed(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) bool {
if !p.cacheEmpty(ctx, store, remote.Name) {
return true
}
if p.syncer != nil {
p.syncer.enqueue(remote, true)
}
deadline := time.Now().Add(p.coldWait)
for time.Now().Before(deadline) {
select {
case <-ctx.Done():
return false
case <-time.After(400 * time.Millisecond):
}
if !p.cacheEmpty(ctx, store, remote.Name) {
return true
}
}
return false
}
func (p *GitHubProvider) cacheEmpty(ctx context.Context, store provider.RemoteMetadataStore, name string) bool {
rows, err := store.ListRPMMetadataEntries(ctx, name)
if err != nil {
// Treat a failed read as "not empty" so a transient DB error becomes a
// normal serve attempt (which reports its own error) rather than a 503.
return false
}
return len(rows) == 0
}
// refresh brings the derived metadata up to date without coupling the scan to
// the inbound request. When the cache is stale it single-flights a scan: if the
// cache already holds rows the scan runs in the background and the caller serves
// the current cache immediately; only a completely empty cache blocks on a
// bounded first scan (so the first client sees packages rather than an empty or
// 500 repodata).
func (p *GitHubProvider) refresh(remote models.Remote, store provider.RemoteMetadataStore) {
ttl := time.Duration(remote.MutableTTL) * time.Second
if ttl <= 0 {
ttl = 5 * time.Minute
}
p.mu.Lock()
last, ok := p.lastScan[remote.Name]
fresh := ok && time.Since(last) < ttl
if fresh || p.scanning[remote.Name] {
p.mu.Unlock()
return
}
p.scanning[remote.Name] = true
p.mu.Unlock()
empty := true
if rows, err := store.ListRPMMetadataEntries(context.Background(), remote.Name); err == nil {
empty = len(rows) == 0
}
if empty {
p.runScan(remote, store)
return
}
go p.runScan(remote, store)
}
// runScan derives metadata on a detached, bounded context so a client cancel
// can neither abort the shared derive nor poison the metadata read. The caller
// must have already claimed the single-flight slot (scanning[name] = true).
func (p *GitHubProvider) runScan(remote models.Remote, store provider.RemoteMetadataStore) {
defer func() {
p.mu.Lock()
delete(p.scanning, remote.Name)
p.mu.Unlock()
}()
ctx, cancel := context.WithTimeout(context.Background(), p.scanTimeout)
defer cancel()
if err := p.scan(ctx, remote, store); err != nil {
// Keep serving whatever metadata is already cached rather than 500ing.
slog.Error("github_rpm: release scan failed", "remote", remote.Name, "error", err)
return
}
p.mu.Lock()
p.lastScan[remote.Name] = time.Now()
p.mu.Unlock()
}
// scan runs a full unconditional derive. Retained for the legacy inline refresh
// path and existing tests; the syncer uses scanWithState to pass and receive the
// releases-list ETag.
func (p *GitHubProvider) scan(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) error {
_, _, err := p.scanWithState(ctx, remote, store, "")
return err
}
// scanWithState derives metadata incrementally. It sends the prior releases-list
// ETag as a conditional request: a 304 means nothing changed, so it returns
// (etag, changed=false) without a single asset fetch. On a 200 it diffs the
// release assets against the cache, derives only new/changed assets, prunes
// assets that disappeared, and returns the new ETag.
func (p *GitHubProvider) scanWithState(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore, etag string) (newEtag string, changed bool, err error) {
releases, newEtag, notModified, err := p.fetchReleases(ctx, remote, etag)
if err != nil {
return etag, false, err
}
if notModified {
return etag, false, nil
}
existing, err := store.ListRPMMetadataEntries(ctx, remote.Name)
if err != nil {
return newEtag, false, err
}
existingByPath := make(map[string]provider.RPMMetadata, len(existing))
for _, m := range existing {
existingByPath[m.FilePath] = m
}
allow, err := compilePatterns(remote.Patterns)
if err != nil {
return newEtag, false, err
}
seen := map[string]bool{}
for _, rel := range releases {
if rel.Draft {
continue
}
for _, asset := range rel.Assets {
if !strings.HasSuffix(strings.ToLower(asset.Name), ".rpm") {
continue
}
if !matchesAny(allow, asset.Name) {
continue
}
fp := assetPath(asset)
if fp == "" {
continue
}
seen[fp] = true
if cur, ok := existingByPath[fp]; ok {
// Assets are effectively immutable; only re-derive when the
// upstream digest is known and no longer matches what we cached.
if asset.Digest == "" || cur.ContentHash == asset.Digest {
continue
}
_ = store.DeleteRPMMetadata(ctx, remote.Name, fp)
}
meta, err := p.deriveAsset(ctx, remote, asset, fp)
if err != nil {
slog.Warn("github_rpm: derive asset failed", "remote", remote.Name, "asset", asset.Name, "error", err)
continue
}
if err := store.InsertRPMMetadata(ctx, meta); err != nil {
slog.Error("github_rpm: insert metadata failed", "remote", remote.Name, "asset", asset.Name, "error", err)
continue
}
slog.Info("github_rpm: derived asset", "remote", remote.Name, "name", meta.Name, "version", meta.Version, "arch", meta.Arch)
}
}
for fp := range existingByPath {
if !seen[fp] {
_ = store.DeleteRPMMetadata(ctx, remote.Name, fp)
}
}
return newEtag, true, nil
}
type ghRelease struct {
TagName string `json:"tag_name"`
Draft bool `json:"draft"`
Assets []ghAsset `json:"assets"`
}
type ghAsset struct {
Name string `json:"name"`
Size int64 `json:"size"`
BrowserDownloadURL string `json:"browser_download_url"`
Digest string `json:"digest"`
}
// fetchReleases lists a repo's releases. It sends the prior ETag as
// If-None-Match on page 1 (the newest releases, where a new one first appears):
// a 304 there means the repo is unchanged, so it returns notModified without
// paging further — GitHub does not count 304 conditional responses against the
// rate limit, making an unchanged repo nearly free. On a 200 it captures the
// page-1 ETag and pages through the rest normally. Every call waits on the
// shared limiter first.
func (p *GitHubProvider) fetchReleases(ctx context.Context, remote models.Remote, etag string) (all []ghRelease, newEtag string, notModified bool, err error) {
base := strings.TrimRight(remote.BaseURL, "/") + "/releases"
for page := 1; page <= p.pageCap; page++ {
u := fmt.Sprintf("%s?per_page=100&page=%d", base, page)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
if err != nil {
return nil, "", false, err
}
hdr, err := p.githubHeaders(ctx, remote, true)
if err != nil {
return nil, "", false, err
}
copyHeaders(req, hdr)
if page == 1 && etag != "" {
req.Header.Set("If-None-Match", etag)
}
if err := p.limiterWait(ctx); err != nil {
return nil, "", false, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, "", false, err
}
if page == 1 && resp.StatusCode == http.StatusNotModified {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
return nil, etag, true, nil
}
body, err := io.ReadAll(resp.Body)
respEtag := resp.Header.Get("ETag")
resp.Body.Close()
if err != nil {
return nil, "", false, err
}
if resp.StatusCode != http.StatusOK {
return nil, "", false, fmt.Errorf("github releases API %s: status %d", u, resp.StatusCode)
}
if page == 1 {
newEtag = respEtag
}
var releases []ghRelease
if err := json.Unmarshal(body, &releases); err != nil {
return nil, "", false, fmt.Errorf("decode releases: %w", err)
}
if len(releases) == 0 {
break
}
all = append(all, releases...)
if len(releases) < 100 {
break
}
}
return all, newEtag, false, nil
}
func (p *GitHubProvider) deriveAsset(ctx context.Context, remote models.Remote, asset ghAsset, fp string) (*provider.RPMMetadata, error) {
pkg, err := p.fetchHeader(ctx, remote, asset.BrowserDownloadURL)
if err != nil {
return nil, err
}
meta := &provider.RPMMetadata{
RepoName: remote.Name,
FilePath: fp,
Name: pkg.Name(),
Epoch: pkg.Epoch(),
Version: pkg.Version(),
Release: pkg.Release(),
Arch: pkg.Architecture(),
Summary: pkg.Summary(),
Description: pkg.Description(),
RPMSize: asset.Size,
InstalledSize: int64(pkg.Size()),
License: pkg.License(),
Vendor: pkg.Vendor(),
Group: firstGroup(pkg.Groups()),
BuildHost: pkg.BuildHost(),
SourceRPM: pkg.SourceRPM(),
URL: pkg.URL(),
Packager: pkg.Packager(),
}
for _, d := range pkg.Requires() {
meta.Requires = append(meta.Requires, rpmDepFromEntry(d))
}
for _, d := range pkg.Provides() {
meta.Provides = append(meta.Provides, rpmDepFromEntry(d))
}
for _, d := range pkg.Conflicts() {
meta.Conflicts = append(meta.Conflicts, rpmDepFromEntry(d))
}
for _, d := range pkg.Obsoletes() {
meta.Obsoletes = append(meta.Obsoletes, rpmDepFromEntry(d))
}
for _, f := range pkg.Files() {
rf := provider.RPMFile{Path: f.Name()}
if f.IsDir() {
rf.Type = "dir"
}
meta.Files = append(meta.Files, rf)
}
if meta.Requires == nil {
meta.Requires = []provider.RPMDep{}
}
if meta.Provides == nil {
meta.Provides = []provider.RPMDep{}
}
if meta.Conflicts == nil {
meta.Conflicts = []provider.RPMDep{}
}
if meta.Obsoletes == nil {
meta.Obsoletes = []provider.RPMDep{}
}
if meta.Files == nil {
meta.Files = []provider.RPMFile{}
}
meta.Changelogs = []provider.RPMChangelog{}
// The primary.xml pkgid checksum must be the sha256 of the whole package.
// Prefer GitHub's asset digest so we never download the body; only when it
// is absent (or not sha256) do we stream the asset once to compute it.
if h, ok := sha256FromDigest(asset.Digest); ok {
meta.ContentHash = "sha256:" + h
} else {
h, err := p.computeSHA256(ctx, remote, asset.BrowserDownloadURL)
if err != nil {
return nil, fmt.Errorf("compute sha256: %w", err)
}
meta.ContentHash = "sha256:" + h
}
return meta, nil
}
// fetchHeader pulls only the front of the package with a ranged GET and parses
// the RPM header from it. The header sits before the payload, so a small prefix
// is enough; on a truncated-header parse error it doubles the range and retries.
func (p *GitHubProvider) fetchHeader(ctx context.Context, remote models.Remote, downloadURL string) (*rpmlib.Package, error) {
n := p.headerInitial
for {
body, full, err := p.rangeGet(ctx, remote, downloadURL, n)
if err != nil {
return nil, err
}
pkg, perr := rpmlib.Read(bytes.NewReader(body))
if perr == nil {
return pkg, nil
}
truncated := errors.Is(perr, io.ErrUnexpectedEOF) || errors.Is(perr, io.EOF)
if truncated && !full && n < p.headerMax {
n *= 2
if n > p.headerMax {
n = p.headerMax
}
continue
}
return nil, fmt.Errorf("parse rpm header: %w", perr)
}
}
// rangeGet returns the first n bytes of downloadURL. full is true when the
// response body was shorter than n (i.e. we already have the whole object).
func (p *GitHubProvider) rangeGet(ctx context.Context, remote models.Remote, downloadURL string, n int64) ([]byte, bool, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
if err != nil {
return nil, false, err
}
hdr, err := p.githubHeaders(ctx, remote, false)
if err != nil {
return nil, false, err
}
copyHeaders(req, hdr)
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", n-1))
if err := p.limiterWait(ctx); err != nil {
return nil, false, err
}
resp, err := p.client.Do(req)
if err != nil {
return nil, false, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
return nil, false, fmt.Errorf("range GET %s: status %d", downloadURL, resp.StatusCode)
}
body, err := io.ReadAll(io.LimitReader(resp.Body, n))
if err != nil {
return nil, false, err
}
full := int64(len(body)) < n
return body, full, nil
}
func (p *GitHubProvider) computeSHA256(ctx context.Context, remote models.Remote, downloadURL string) (string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
if err != nil {
return "", err
}
hdr, err := p.githubHeaders(ctx, remote, false)
if err != nil {
return "", err
}
copyHeaders(req, hdr)
if err := p.limiterWait(ctx); err != nil {
return "", err
}
resp, err := p.client.Do(req)
if err != nil {
return "", err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("GET %s: status %d", downloadURL, resp.StatusCode)
}
h := sha256.New()
if _, err := io.Copy(h, resp.Body); err != nil {
return "", err
}
return hex.EncodeToString(h.Sum(nil)), nil
}
// assetPath is the package's location relative to github.com — the path the
// backend releases_remote (base https://github.com) proxies. It doubles as the
// rpm_metadata key and the <location href> in primary.xml.
func assetPath(asset ghAsset) string {
u, err := url.Parse(asset.BrowserDownloadURL)
if err != nil {
return ""
}
return strings.TrimPrefix(u.Path, "/")
}
func sha256FromDigest(digest string) (string, bool) {
if strings.HasPrefix(digest, "sha256:") {
return strings.TrimPrefix(digest, "sha256:"), true
}
return "", false
}
// githubHeaders builds the outbound headers for a GitHub request, attaching a
// bearer credential when one is available. A per-remote credential wins; absent
// that, the process-wide server credential is used; absent both, the request is
// unauthenticated (anonymous, subject to the 60/hr cap).
func (p *GitHubProvider) githubHeaders(ctx context.Context, remote models.Remote, api bool) (http.Header, error) {
h := http.Header{}
if api {
h.Set("Accept", "application/vnd.github+json")
h.Set("X-GitHub-Api-Version", "2022-11-28")
}
tok, err := p.githubToken(ctx, remote)
if err != nil {
return nil, err
}
if tok != "" {
h.Set("Authorization", "Bearer "+tok)
}
return h, nil
}
// githubToken resolves the bearer token for a remote. Precedence: a per-remote
// credential (password, then username) overrides the server credential.
func (p *GitHubProvider) githubToken(ctx context.Context, remote models.Remote) (string, error) {
if remote.Password != "" {
return remote.Password, nil
}
if remote.Username != "" {
return remote.Username, nil
}
if c := p.serverCredential(); c != nil {
return c.Token(ctx)
}
return "", nil
}
// serverCredential returns this provider's server credential, defaulting to the
// process-wide one installed at startup.
func (p *GitHubProvider) serverCredential() githubauth.Credential {
if p.serverCred != nil {
return p.serverCred
}
return githubauth.Server()
}
func copyHeaders(req *http.Request, h http.Header) {
for k, vals := range h {
for _, v := range vals {
req.Header.Add(k, v)
}
}
}
func compilePatterns(patterns []string) ([]*regexp.Regexp, error) {
var out []*regexp.Regexp
for _, p := range patterns {
re, err := regexp.Compile(p)
if err != nil {
return nil, fmt.Errorf("invalid pattern %q: %w", p, err)
}
out = append(out, re)
}
return out, nil
}
func matchesAny(res []*regexp.Regexp, s string) bool {
if len(res) == 0 {
return true
}
for _, re := range res {
if re.MatchString(s) {
return true
}
}
return false
}
+134
View File
@@ -0,0 +1,134 @@
package rpm
import (
"context"
"encoding/json"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/githubauth"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// staticCred is a test Credential yielding a fixed token.
type staticCred string
func (s staticCred) Token(context.Context) (string, error) { return string(s), nil }
func TestGitHubServerCredentialAttachedToReleasesAndAssets(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
p.serverCred = staticCred("ghp_server_secret")
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
if got := fx.releaseAuth; got != "Bearer ghp_server_secret" {
t.Fatalf("releases Authorization = %q, want Bearer ghp_server_secret", got)
}
if got := fx.assetAuth; got != "Bearer ghp_server_secret" {
t.Fatalf("asset Authorization = %q, want Bearer ghp_server_secret", got)
}
}
func TestGitHubPerRemoteCredentialOverridesServer(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
p.serverCred = staticCred("ghp_server_secret")
store := newFakeStore()
remote := fx.remote()
remote.Password = "ghp_remote_wins"
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("scan: %v", err)
}
if got := fx.releaseAuth; got != "Bearer ghp_remote_wins" {
t.Fatalf("releases Authorization = %q, want per-remote token to win", got)
}
if got := fx.assetAuth; got != "Bearer ghp_remote_wins" {
t.Fatalf("asset Authorization = %q, want per-remote token to win", got)
}
}
func TestGitHubNoCredentialSendsNoAuthHeader(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider() // serverCred nil, package Server() unset in unit tests
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
if fx.releaseAuth != "" {
t.Fatalf("expected no Authorization header, got %q", fx.releaseAuth)
}
if fx.assetAuth != "" {
t.Fatalf("expected no asset Authorization header, got %q", fx.assetAuth)
}
// Requests still succeed anonymously.
if rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm"); len(rows) != 1 {
t.Fatalf("anonymous scan should still derive metadata, got %d rows", len(rows))
}
}
func TestGitHubETag304FlowWithAuth(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
p := newTestProvider()
p.serverCred = staticCred("ghp_server_secret")
store := newFakeStore()
etag, changed, err := p.scanWithState(context.Background(), fx.remote(), store, "")
if err != nil {
t.Fatalf("first scan: %v", err)
}
if !changed || etag != `"v1"` {
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag)
}
// Re-scan with the captured ETag: a 304 means no change and no asset fetch.
etag2, changed2, err := p.scanWithState(context.Background(), fx.remote(), store, etag)
if err != nil {
t.Fatalf("second scan: %v", err)
}
if changed2 {
t.Fatal("expected no change on 304")
}
if etag2 != `"v1"` {
t.Fatalf("etag = %q, want preserved \"v1\"", etag2)
}
if fx.notModHit != 1 {
t.Fatalf("expected exactly one 304 response, got %d", fx.notModHit)
}
// The conditional request still carried the credential.
if fx.releaseAuth != "Bearer ghp_server_secret" {
t.Fatalf("conditional request Authorization = %q, want the server credential", fx.releaseAuth)
}
}
// TestGitHubCredentialAbsentFromRemoteJSON asserts the server credential never
// appears in a remote's serialized API representation, and per-remote secrets
// stay redacted by the models.Remote json:"-" tags.
func TestGitHubCredentialAbsentFromRemoteJSON(t *testing.T) {
githubauth.SetServer(staticCred("ghp_super_secret_server_token"))
t.Cleanup(func() { githubauth.SetServer(nil) })
remote := models.Remote{
Name: "acme-rpm",
PackageType: models.PackageGitHubRPM,
BaseURL: "https://api.github.com/repos/acme/tools",
Username: "per_remote_user",
Password: "per_remote_secret",
}
b, err := json.Marshal(remote)
if err != nil {
t.Fatalf("marshal remote: %v", err)
}
js := string(b)
for _, secret := range []string{"ghp_super_secret_server_token", "per_remote_secret", "per_remote_user"} {
if strings.Contains(js, secret) {
t.Fatalf("credential %q leaked into remote JSON: %s", secret, js)
}
}
}
+390
View File
@@ -0,0 +1,390 @@
package rpm
import (
"compress/gzip"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeStore is an in-memory provider.RemoteMetadataStore keyed by file_path,
// mirroring the (repo_name, file_path) uniqueness of the real table.
type fakeStore struct {
mu sync.Mutex
rows map[string]provider.RPMMetadata
}
func newFakeStore() *fakeStore { return &fakeStore{rows: map[string]provider.RPMMetadata{}} }
func (f *fakeStore) InsertRPMMetadata(_ context.Context, m *provider.RPMMetadata) error {
f.mu.Lock()
defer f.mu.Unlock()
if _, ok := f.rows[m.FilePath]; ok {
return nil // ON CONFLICT DO NOTHING
}
f.rows[m.FilePath] = *m
return nil
}
func (f *fakeStore) DeleteRPMMetadata(_ context.Context, _, filePath string) error {
f.mu.Lock()
defer f.mu.Unlock()
delete(f.rows, filePath)
return nil
}
func (f *fakeStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
func (f *fakeStore) DeleteDebMetadata(context.Context, string, string) error { return nil }
func (f *fakeStore) ListRPMMetadataEntries(ctx context.Context, _ string) ([]provider.RPMMetadata, error) {
// Mirror pgx: a canceled/expired context fails the read. This is what
// poisons the repodata response if the read runs on the inbound request.
if err := ctx.Err(); err != nil {
return nil, err
}
f.mu.Lock()
defer f.mu.Unlock()
out := make([]provider.RPMMetadata, 0, len(f.rows))
for _, m := range f.rows {
out = append(out, m)
}
return out, nil
}
// githubFixture serves the releases API and the .rpm asset downloads (with
// Range support) for a set of packages. digest controls whether the asset
// carries a sha256 digest (no-download path) or not (compute path).
type githubFixture struct {
srv *httptest.Server
rpmBytes map[string][]byte // asset filename -> bytes
rangeHit map[string]int // asset filename -> number of ranged GETs
fullHit map[string]int // asset filename -> number of full GETs
etag string // when set, served as ETag; matching If-None-Match yields 304
releasesHit int // total releases-list requests (200 + 304)
notModHit int // releases-list requests answered 304
releaseAuth string // Authorization header seen on the last releases request
assetAuth string // Authorization header seen on the last asset request
mu sync.Mutex
}
func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
t.Helper()
f := &githubFixture{
rpmBytes: map[string][]byte{},
rangeHit: map[string]int{},
fullHit: map[string]int{},
}
f.rpmBytes["demo-1.2-3.x86_64.rpm"] = testsupport.MinimalRPM("demo", "1.2", "3", "x86_64")
mux := http.NewServeMux()
mux.HandleFunc("/repos/acme/tools/releases", func(w http.ResponseWriter, r *http.Request) {
page := r.URL.Query().Get("page")
if page != "" && page != "1" {
w.Write([]byte("[]"))
return
}
f.mu.Lock()
f.releasesHit++
f.releaseAuth = r.Header.Get("Authorization")
etag := f.etag
if etag != "" && r.Header.Get("If-None-Match") == etag {
f.notModHit++
f.mu.Unlock()
w.WriteHeader(http.StatusNotModified)
return
}
f.mu.Unlock()
if etag != "" {
w.Header().Set("ETag", etag)
}
var assets []map[string]any
for name := range f.rpmBytes {
a := map[string]any{
"name": name,
"size": len(f.rpmBytes[name]),
"browser_download_url": f.srv.URL + "/acme/tools/releases/download/v1.2-3/" + name,
}
if withDigest {
sum := sha256.Sum256(f.rpmBytes[name])
a["digest"] = "sha256:" + hex.EncodeToString(sum[:])
}
assets = append(assets, a)
}
rel := []map[string]any{{"tag_name": "v1.2-3", "draft": false, "assets": assets}}
json.NewEncoder(w).Encode(rel)
})
mux.HandleFunc("/acme/tools/releases/download/", func(w http.ResponseWriter, r *http.Request) {
name := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
body, ok := f.rpmBytes[name]
if !ok {
http.Error(w, "not found", 404)
return
}
rng := r.Header.Get("Range")
f.mu.Lock()
f.assetAuth = r.Header.Get("Authorization")
if rng != "" {
f.rangeHit[name]++
} else {
f.fullHit[name]++
}
f.mu.Unlock()
if rng == "" {
w.WriteHeader(200)
w.Write(body)
return
}
// Parse "bytes=0-N".
var end int
fmt.Sscanf(rng, "bytes=0-%d", &end)
if end >= len(body)-1 {
end = len(body) - 1
}
w.Header().Set("Content-Range", fmt.Sprintf("bytes 0-%d/%d", end, len(body)))
w.Header().Set("Content-Length", strconv.Itoa(end+1))
w.WriteHeader(http.StatusPartialContent)
w.Write(body[:end+1])
})
f.srv = httptest.NewServer(mux)
t.Cleanup(f.srv.Close)
return f
}
func (f *githubFixture) remote() models.Remote {
return models.Remote{
Name: "acme-rpm",
PackageType: models.PackageGitHubRPM,
BaseURL: f.srv.URL + "/repos/acme/tools",
ReleasesRemote: "github",
MutableTTL: 3600,
}
}
func newTestProvider() *GitHubProvider {
p := newGitHubProvider()
p.headerInitial = 32 // force the ranged-fetch retry loop against the tiny fixture
p.headerMax = 1 << 20
return p
}
func TestGitHubScanDerivesMetadataFromHeaderAndDigest(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
metas, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm")
if len(metas) != 1 {
t.Fatalf("want 1 metadata row, got %d", len(metas))
}
m := metas[0]
if m.Name != "demo" || m.Version != "1.2" || m.Release != "3" || m.Arch != "x86_64" {
t.Fatalf("bad NEVRA: %+v", m)
}
// location href / redirect key must be the github-relative download path.
wantPath := "acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"
if m.FilePath != wantPath {
t.Fatalf("FilePath = %q, want %q", m.FilePath, wantPath)
}
if int(m.RPMSize) != len(fx.rpmBytes["demo-1.2-3.x86_64.rpm"]) {
t.Fatalf("RPMSize = %d, want %d", m.RPMSize, len(fx.rpmBytes["demo-1.2-3.x86_64.rpm"]))
}
// Digest present => checksum from digest, no full download.
sum := sha256.Sum256(fx.rpmBytes["demo-1.2-3.x86_64.rpm"])
if m.ContentHash != "sha256:"+hex.EncodeToString(sum[:]) {
t.Fatalf("ContentHash = %q, want digest", m.ContentHash)
}
if fx.fullHit["demo-1.2-3.x86_64.rpm"] != 0 {
t.Fatalf("expected no full download when digest present, got %d", fx.fullHit["demo-1.2-3.x86_64.rpm"])
}
if fx.rangeHit["demo-1.2-3.x86_64.rpm"] == 0 {
t.Fatalf("expected ranged header fetch")
}
}
func TestGitHubChecksumComputedWhenDigestAbsent(t *testing.T) {
fx := newGitHubFixture(t, false)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
metas, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm")
if len(metas) != 1 {
t.Fatalf("want 1 row, got %d", len(metas))
}
sum := sha256.Sum256(fx.rpmBytes["demo-1.2-3.x86_64.rpm"])
if metas[0].ContentHash != "sha256:"+hex.EncodeToString(sum[:]) {
t.Fatalf("computed checksum mismatch: %q", metas[0].ContentHash)
}
if fx.fullHit["demo-1.2-3.x86_64.rpm"] == 0 {
t.Fatalf("expected a full download to compute sha256 when digest absent")
}
}
func TestGitHubServeRemoteRepodataAndRedirect(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
const proxyBase = "https://artifactapi.example"
// repomd.xml is served and triggers the initial scan.
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil)
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", proxyBase, store) {
t.Fatal("ServeRemote did not handle repomd.xml")
}
if rec.Code != 200 || !strings.Contains(rec.Body.String(), "<repomd") {
t.Fatalf("repomd bad: code=%d body=%s", rec.Code, rec.Body.String())
}
// primary.xml.gz must carry the package with a location href that is the
// github-relative download path (so it resolves back to this remote and
// redirects to the backend).
rec = httptest.NewRecorder()
req = httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, "repodata/abc-primary.xml.gz", proxyBase, store) {
t.Fatal("ServeRemote did not handle primary")
}
gz, err := gzip.NewReader(rec.Body)
if err != nil {
t.Fatalf("gzip: %v", err)
}
xmlBytes, _ := io.ReadAll(gz)
primary := string(xmlBytes)
if !strings.Contains(primary, `<name>demo</name>`) {
t.Fatalf("primary missing package: %s", primary)
}
if !strings.Contains(primary, `<location href="acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"/>`) {
t.Fatalf("primary missing/incorrect location href: %s", primary)
}
// A .rpm request redirects to the backend releases_remote.
rec = httptest.NewRecorder()
pkgPath := "acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"
req = httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/"+pkgPath, nil)
if !p.ServeRemote(rec, req, remote, pkgPath, proxyBase, store) {
t.Fatal("ServeRemote did not handle .rpm")
}
if rec.Code != http.StatusFound {
t.Fatalf("want 302, got %d", rec.Code)
}
wantLoc := proxyBase + "/api/v1/remote/github/" + pkgPath
if got := rec.Header().Get("Location"); got != wantLoc {
t.Fatalf("Location = %q, want %q", got, wantLoc)
}
}
// TestGitHubServeRemoteCanceledRequestServesCache reproduces the cold-makecache
// 500: when the inbound request context is already canceled (dnf timed out and
// disconnected), the repodata read must not be run on that context and turned
// into a 500. With the cache already warm, the handler serves it as 200.
// Before the fix the read used r.Context() and returned 500; after the fix it
// runs on a detached context and serves the cached repomd.
func TestGitHubServeRemoteCanceledRequestServesCache(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
// Warm the cache and mark the scan fresh so ServeRemote does not re-derive.
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("warm scan: %v", err)
}
p.mu.Lock()
p.lastScan[remote.Name] = time.Now()
p.mu.Unlock()
// Inbound request whose context is already canceled (client went away).
ctx, cancel := context.WithCancel(context.Background())
cancel()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil).WithContext(ctx)
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", "https://x", store) {
t.Fatal("ServeRemote did not handle repomd.xml")
}
if rec.Code != http.StatusOK {
t.Fatalf("canceled request must serve cache, not error; got code=%d body=%s", rec.Code, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "<repomd") {
t.Fatalf("expected repomd served from cache, got %s", rec.Body.String())
}
}
func TestGitHubServeRemoteRedirectRequiresReleasesRemote(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
remote.ReleasesRemote = ""
rec := httptest.NewRecorder()
pkgPath := "acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"
req := httptest.NewRequest(http.MethodGet, "/x", nil)
if !p.ServeRemote(rec, req, remote, pkgPath, "https://x", store) {
t.Fatal("expected handled")
}
if rec.Code != http.StatusInternalServerError {
t.Fatalf("want 500 when releases_remote unset, got %d", rec.Code)
}
}
func TestGitHubScanPrunesRemovedAssets(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
store := newFakeStore()
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("scan: %v", err)
}
if rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm"); len(rows) != 1 {
t.Fatalf("want 1 row after first scan, got %d", len(rows))
}
// Remove the asset upstream; a rescan must prune the stale metadata row.
delete(fx.rpmBytes, "demo-1.2-3.x86_64.rpm")
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
t.Fatalf("rescan: %v", err)
}
if rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm"); len(rows) != 0 {
t.Fatalf("want 0 rows after prune, got %d", len(rows))
}
}
func TestGitHubAssetPatternFilter(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.rpmBytes["other-9-9.aarch64.rpm"] = testsupport.MinimalRPM("other", "9", "9", "aarch64")
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
remote.Patterns = []string{`^demo-.*\.x86_64\.rpm$`}
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("scan: %v", err)
}
rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm")
if len(rows) != 1 || rows[0].Name != "demo" {
t.Fatalf("pattern filter failed, rows=%+v", rows)
}
}
+52 -10
View File
@@ -7,6 +7,7 @@ import (
"crypto/sha256"
"encoding/hex"
"encoding/xml"
"errors"
"fmt"
"log/slog"
"net/http"
@@ -133,6 +134,12 @@ func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, conte
for _, prov := range pkg.Provides() {
meta.Provides = append(meta.Provides, rpmDepFromEntry(prov))
}
for _, con := range pkg.Conflicts() {
meta.Conflicts = append(meta.Conflicts, rpmDepFromEntry(con))
}
for _, obs := range pkg.Obsoletes() {
meta.Obsoletes = append(meta.Obsoletes, rpmDepFromEntry(obs))
}
if meta.Requires == nil {
meta.Requires = []provider.RPMDep{}
@@ -140,6 +147,12 @@ func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, conte
if meta.Provides == nil {
meta.Provides = []provider.RPMDep{}
}
if meta.Conflicts == nil {
meta.Conflicts = []provider.RPMDep{}
}
if meta.Obsoletes == nil {
meta.Obsoletes = []provider.RPMDep{}
}
meta.Files = []provider.RPMFile{}
meta.Changelogs = []provider.RPMChangelog{}
@@ -229,10 +242,28 @@ func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileSt
return nil, fmt.Errorf("rpm local index generation for virtual repos not supported")
}
func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
// readMetadataEntries loads the repo's derived metadata, translating the read
// error into an HTTP response. A canceled/deadline-exceeded context (typically a
// client that went away) becomes a retryable 503 rather than a hard 500, so a
// dnf disconnect never looks like a server fault. ok is false when a response
// has already been written.
func readMetadataEntries(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) ([]provider.RPMMetadata, bool) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
slog.Warn("rpm: metadata read canceled", "repo", repoName, "error", err)
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
return nil, false
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return nil, false
}
return metas, true
}
func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, ok := readMetadataEntries(w, r, reader, repoName)
if !ok {
return
}
@@ -252,9 +283,8 @@ func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader pr
}
func (p *Provider) servePrimary(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
metas, ok := readMetadataEntries(w, r, reader, repoName)
if !ok {
return
}
@@ -264,9 +294,8 @@ func (p *Provider) servePrimary(w http.ResponseWriter, r *http.Request, reader p
}
func (p *Provider) serveFilelists(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
metas, ok := readMetadataEntries(w, r, reader, repoName)
if !ok {
return
}
@@ -276,9 +305,8 @@ func (p *Provider) serveFilelists(w http.ResponseWriter, r *http.Request, reader
}
func (p *Provider) serveOther(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
metas, ok := readMetadataEntries(w, r, reader, repoName)
if !ok {
return
}
@@ -363,6 +391,20 @@ func generatePrimaryXMLGZ(metas []provider.RPMMetadata) []byte {
}
xmlBuf.WriteString(" </rpm:requires>\n")
}
if len(m.Conflicts) > 0 {
xmlBuf.WriteString(" <rpm:conflicts>\n")
for _, d := range m.Conflicts {
writeRPMEntry(&xmlBuf, d)
}
xmlBuf.WriteString(" </rpm:conflicts>\n")
}
if len(m.Obsoletes) > 0 {
xmlBuf.WriteString(" <rpm:obsoletes>\n")
for _, d := range m.Obsoletes {
writeRPMEntry(&xmlBuf, d)
}
xmlBuf.WriteString(" </rpm:obsoletes>\n")
}
fmt.Fprintf(&xmlBuf, " </format>\n")
fmt.Fprintf(&xmlBuf, "</package>\n")
+2
View File
@@ -28,6 +28,8 @@ func (f *fakeMetaStore) InsertRPMMetadata(_ context.Context, m *provider.RPMMeta
return nil
}
func (f *fakeMetaStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
type fakeRPMReader struct{ metas []provider.RPMMetadata }
func (f fakeRPMReader) ListRPMMetadataEntries(_ context.Context, _ string) ([]provider.RPMMetadata, error) {
+256
View File
@@ -0,0 +1,256 @@
package rpm
import (
"context"
"crypto/rand"
"encoding/hex"
"log/slog"
"os"
"sync"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
const (
// syncLeaseDuration is how long a claimed sync lease is held before it is
// considered abandoned. It comfortably exceeds a scan's own timeout so a live
// scan never loses its lease, while a crashed replica's lease still expires.
syncLeaseDuration = 15 * time.Minute
// defaultSyncFreshness is the periodic re-check interval used when a remote's
// mutable_ttl is unset.
defaultSyncFreshness = 5 * time.Minute
// jobQueueDepth bounds the pending work queue; enqueues past it are dropped
// (a later poll re-enqueues), never blocking the caller.
jobQueueDepth = 256
)
// SyncStore is the persistence surface the syncer needs: the metadata cache it
// primes plus the shared sync-state coordination (remote enumeration and the
// per-remote lease). *database.DB satisfies it.
type SyncStore interface {
provider.RemoteMetadataStore
ListGitHubRPMRemotes(ctx context.Context) ([]models.Remote, error)
ClaimGitHubSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (claimed bool, etag string, err error)
ReleaseGitHubSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error
}
// SyncConfig tunes the shared syncer. Zero values fall back to safe defaults.
type SyncConfig struct {
RatePerSec float64 // global GitHub request rate (req/s)
Burst int // token-bucket burst
Workers int // concurrent scan workers
PollInterval time.Duration // base scheduler tick; per-remote cadence is mutable_ttl
}
type syncJob struct {
remote models.Remote
prime bool
}
// Syncer is the single per-process background worker that keeps every
// github_rpm remote's derived metadata fresh. It owns a deduped work queue, a
// pool of workers, and a global token-bucket rate limiter shared across all
// remotes and bound onto the github provider so every GitHub call it makes
// passes through the same bucket. Periodic checks are gated by a shared DB lease
// so, across replicas, only one performs each scan.
type Syncer struct {
store SyncStore
prov *GitHubProvider
limiter *rate.Limiter
cfg SyncConfig
owner string
jobs chan syncJob
mu sync.Mutex
active map[string]bool // remotes queued or in-flight, for dedup/coalescing
}
// NewSyncer builds the syncer bound to the process-wide github provider
// singleton. Call Run to start it.
func NewSyncer(store SyncStore, cfg SyncConfig) *Syncer {
return newSyncer(store, gitHubProvider, cfg)
}
func newSyncer(store SyncStore, prov *GitHubProvider, cfg SyncConfig) *Syncer {
if cfg.RatePerSec <= 0 {
cfg.RatePerSec = 1
}
if cfg.Burst <= 0 {
cfg.Burst = 5
}
if cfg.Workers <= 0 {
cfg.Workers = 3
}
if cfg.PollInterval <= 0 {
cfg.PollInterval = 60 * time.Second
}
lim := rate.NewLimiter(rate.Limit(cfg.RatePerSec), cfg.Burst)
s := &Syncer{
store: store,
prov: prov,
limiter: lim,
cfg: cfg,
owner: leaseOwner(),
jobs: make(chan syncJob, jobQueueDepth),
active: map[string]bool{},
}
// Bind the shared limiter and back-reference so the request path routes
// through this syncer and every derive HTTP call is rate limited.
prov.limiter = lim
prov.syncer = s
return s
}
// Run starts the worker pool and the periodic scheduler and blocks until ctx is
// canceled, at which point it drains in-flight scans and returns.
func (s *Syncer) Run(ctx context.Context) {
slog.Info("github_rpm syncer started",
"rate_per_sec", s.cfg.RatePerSec, "burst", s.cfg.Burst,
"workers", s.cfg.Workers, "poll_interval", s.cfg.PollInterval, "owner", s.owner)
var wg sync.WaitGroup
for i := 0; i < s.cfg.Workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
s.worker(ctx)
}()
}
ticker := time.NewTicker(s.cfg.PollInterval)
defer ticker.Stop()
s.schedule(ctx) // sweep at boot so existing remotes are checked immediately
for {
select {
case <-ctx.Done():
wg.Wait()
slog.Info("github_rpm syncer stopped")
return
case <-ticker.C:
s.schedule(ctx)
}
}
}
// schedule enqueues a periodic check for every github_rpm remote. The DB lease
// (claimed in the worker) enforces the per-remote mutable_ttl cadence and cross
// replica coordination, so enqueuing every tick is cheap: a not-yet-due remote
// simply fails to claim and is skipped.
func (s *Syncer) schedule(ctx context.Context) {
remotes, err := s.store.ListGitHubRPMRemotes(ctx)
if err != nil {
slog.Error("github_rpm syncer: list remotes", "error", err)
return
}
for _, r := range remotes {
s.enqueue(r, false)
}
}
// EnqueuePrime queues an immediate background prime for a freshly created
// remote so its metadata is derived without blocking the create call.
func (s *Syncer) EnqueuePrime(remote models.Remote) {
if s == nil {
return
}
s.enqueue(remote, true)
}
// enqueue adds a job unless the remote is already queued or in-flight, coalescing
// duplicate requests down to one scan. It never blocks: a full queue drops the
// job (a later poll re-enqueues it) after clearing the dedup slot.
func (s *Syncer) enqueue(remote models.Remote, prime bool) {
s.mu.Lock()
if s.active[remote.Name] {
s.mu.Unlock()
return
}
s.active[remote.Name] = true
s.mu.Unlock()
select {
case s.jobs <- syncJob{remote: remote, prime: prime}:
default:
s.mu.Lock()
delete(s.active, remote.Name)
s.mu.Unlock()
}
}
func (s *Syncer) worker(ctx context.Context) {
for {
select {
case <-ctx.Done():
return
case job := <-s.jobs:
s.process(ctx, job)
}
}
}
// process claims the shared lease and, if won, runs an incremental scan. The
// lease bounds total GitHub load to one scan per freshness window across all
// replicas; losing the claim (another replica scanning, or not yet due) is a
// no-op.
func (s *Syncer) process(ctx context.Context, job syncJob) {
defer func() {
s.mu.Lock()
delete(s.active, job.remote.Name)
s.mu.Unlock()
}()
freshness := time.Duration(job.remote.MutableTTL) * time.Second
if freshness <= 0 {
freshness = defaultSyncFreshness
}
if job.prime {
freshness = 0 // prime ignores the recency gate but still respects a live lease
}
claimed, etag, err := s.store.ClaimGitHubSyncLease(ctx, job.remote.Name, s.owner, freshness, syncLeaseDuration)
if err != nil {
slog.Error("github_rpm syncer: claim lease", "remote", job.remote.Name, "error", err)
return
}
if !claimed {
return
}
scanCtx, cancel := context.WithTimeout(ctx, s.prov.scanTimeout)
defer cancel()
newEtag, changed, scanErr := s.prov.scanWithState(scanCtx, job.remote, s.store, etag)
releaseEtag := etag
if scanErr == nil {
releaseEtag = newEtag
} else {
slog.Error("github_rpm syncer: scan failed", "remote", job.remote.Name, "error", scanErr)
}
// Release on a detached context so a clean shutdown mid-scan still frees the
// lease and advances last_synced_at (otherwise it simply expires).
relCtx, relCancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer relCancel()
if err := s.store.ReleaseGitHubSyncLease(relCtx, job.remote.Name, s.owner, releaseEtag, time.Now()); err != nil {
slog.Warn("github_rpm syncer: release lease", "remote", job.remote.Name, "error", err)
}
if scanErr == nil && changed {
slog.Info("github_rpm syncer: refreshed", "remote", job.remote.Name, "prime", job.prime)
}
}
// leaseOwner is a per-replica identity for the lease: hostname plus a random
// suffix so restarts and colocated replicas never collide.
func leaseOwner() string {
host, _ := os.Hostname()
var b [6]byte
_, _ = rand.Read(b[:])
return host + "-" + hex.EncodeToString(b[:])
}
+312
View File
@@ -0,0 +1,312 @@
package rpm
import (
"context"
"net/http"
"net/http/httptest"
"sync"
"testing"
"time"
"golang.org/x/time/rate"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeSyncStore is an in-memory SyncStore: the metadata cache (via the embedded
// fakeStore) plus the shared sync-state lease, whose claim mirrors the atomic
// semantics of the real SQL (recency gate AND no live lease).
type fakeSyncStore struct {
*fakeStore
mu sync.Mutex
remotes []models.Remote
leaseOwner map[string]string
leaseExp map[string]time.Time
lastSynced map[string]time.Time
etags map[string]string
}
func newFakeSyncStore() *fakeSyncStore {
return &fakeSyncStore{
fakeStore: newFakeStore(),
leaseOwner: map[string]string{},
leaseExp: map[string]time.Time{},
lastSynced: map[string]time.Time{},
etags: map[string]string{},
}
}
func (f *fakeSyncStore) ListGitHubRPMRemotes(_ context.Context) ([]models.Remote, error) {
f.mu.Lock()
defer f.mu.Unlock()
return append([]models.Remote(nil), f.remotes...), nil
}
func (f *fakeSyncStore) ClaimGitHubSyncLease(_ context.Context, name, owner string, freshness, lease time.Duration) (bool, string, error) {
f.mu.Lock()
defer f.mu.Unlock()
now := time.Now()
ls, hasLS := f.lastSynced[name]
exp, hasExp := f.leaseExp[name]
freshOK := !hasLS || now.Sub(ls) >= freshness
leaseOK := !hasExp || exp.Before(now)
if freshOK && leaseOK {
f.leaseOwner[name] = owner
f.leaseExp[name] = now.Add(lease)
return true, f.etags[name], nil
}
return false, "", nil
}
func (f *fakeSyncStore) ReleaseGitHubSyncLease(_ context.Context, name, owner, etag string, syncedAt time.Time) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.leaseOwner[name] != owner {
return nil
}
f.lastSynced[name] = syncedAt
f.etags[name] = etag
delete(f.leaseOwner, name)
delete(f.leaseExp, name)
return nil
}
func testSyncConfig() SyncConfig {
return SyncConfig{RatePerSec: 1000, Burst: 100, Workers: 1, PollInterval: time.Hour}
}
// (a) A 304 conditional response must derive nothing: no asset header GETs and
// changed=false, so an unchanged repo is nearly free.
func TestSyncerConditionalNotModifiedSkipsDerive(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
etag1, changed, err := p.scanWithState(context.Background(), remote, store, "")
if err != nil {
t.Fatalf("first scan: %v", err)
}
if !changed || etag1 != `"v1"` {
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag1)
}
priorRange := fx.rangeHit["demo-1.2-3.x86_64.rpm"]
if priorRange == 0 {
t.Fatal("first scan should have fetched the asset header")
}
etag2, changed2, err := p.scanWithState(context.Background(), remote, store, etag1)
if err != nil {
t.Fatalf("second scan: %v", err)
}
if changed2 {
t.Fatal("304 scan must report changed=false")
}
if etag2 != etag1 {
t.Fatalf("etag changed across 304: %q -> %q", etag1, etag2)
}
if fx.notModHit != 1 {
t.Fatalf("want exactly one 304 releases response, got %d", fx.notModHit)
}
if got := fx.rangeHit["demo-1.2-3.x86_64.rpm"]; got != priorRange {
t.Fatalf("304 scan re-fetched asset header: %d -> %d", priorRange, got)
}
}
// (b) On a real change, only the newly added asset is derived; assets already
// cached are never re-fetched.
func TestSyncerIncrementalDerivesOnlyNewAsset(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
p := newTestProvider()
store := newFakeStore()
remote := fx.remote()
if _, _, err := p.scanWithState(context.Background(), remote, store, ""); err != nil {
t.Fatalf("first scan: %v", err)
}
demoRange := fx.rangeHit["demo-1.2-3.x86_64.rpm"]
// Add a new asset and bump the ETag so the conditional request returns 200.
fx.rpmBytes["other-9-9.aarch64.rpm"] = testsupport.MinimalRPM("other", "9", "9", "aarch64")
fx.etag = `"v2"`
if _, changed, err := p.scanWithState(context.Background(), remote, store, `"v1"`); err != nil || !changed {
t.Fatalf("second scan changed=%v err=%v", changed, err)
}
rows, _ := store.ListRPMMetadataEntries(context.Background(), remote.Name)
if len(rows) != 2 {
t.Fatalf("want 2 cached rows after incremental derive, got %d", len(rows))
}
if got := fx.rangeHit["demo-1.2-3.x86_64.rpm"]; got != demoRange {
t.Fatalf("already-cached asset was re-fetched: %d -> %d", demoRange, got)
}
if fx.rangeHit["other-9-9.aarch64.rpm"] == 0 {
t.Fatal("newly added asset was not derived")
}
}
// (c) The shared limiter caps the request rate: three gated releases calls at
// one token per 120ms cannot complete faster than ~2 gaps.
func TestRateLimiterCapsRequestRate(t *testing.T) {
fx := newGitHubFixture(t, true)
p := newTestProvider()
p.limiter = rate.NewLimiter(rate.Every(120*time.Millisecond), 1)
remote := fx.remote()
start := time.Now()
for i := 0; i < 3; i++ {
if _, _, _, err := p.fetchReleases(context.Background(), remote, ""); err != nil {
t.Fatalf("fetchReleases %d: %v", i, err)
}
}
if elapsed := time.Since(start); elapsed < 200*time.Millisecond {
t.Fatalf("rate limiter did not throttle: 3 calls took %v, want >= 200ms", elapsed)
}
}
// (d) Concurrent enqueues for the same remote coalesce to a single queued job.
func TestSyncerEnqueueDedup(t *testing.T) {
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := models.Remote{Name: "acme-rpm", PackageType: models.PackageGitHubRPM, MutableTTL: 3600}
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() { defer wg.Done(); s.enqueue(remote, false) }()
}
wg.Wait()
if got := len(s.jobs); got != 1 {
t.Fatalf("want exactly 1 coalesced job, got %d", got)
}
}
// (e) Prime-on-create enqueues a prime job.
func TestSyncerEnqueuePrime(t *testing.T) {
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := models.Remote{Name: "acme-rpm", PackageType: models.PackageGitHubRPM, MutableTTL: 3600}
s.EnqueuePrime(remote)
select {
case job := <-s.jobs:
if !job.prime || job.remote.Name != "acme-rpm" {
t.Fatalf("bad prime job: %+v", job)
}
default:
t.Fatal("EnqueuePrime did not enqueue a job")
}
}
// (f) A held lease prevents a second replica from scanning: with the lease owned
// by another replica, process claims nothing and makes zero GitHub calls.
func TestSyncerLeasePreventsSecondReplica(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := fx.remote()
// Replica 1 holds the lease.
claimed, _, err := store.ClaimGitHubSyncLease(context.Background(), remote.Name, "replica-1", time.Duration(remote.MutableTTL)*time.Second, syncLeaseDuration)
if err != nil || !claimed {
t.Fatalf("replica-1 claim: claimed=%v err=%v", claimed, err)
}
// Replica 2 (this syncer) tries to process the same remote; it must skip.
s.process(context.Background(), syncJob{remote: remote})
if fx.releasesHit != 0 {
t.Fatalf("second replica scanned while lease held: %d releases calls", fx.releasesHit)
}
if rows, _ := store.ListRPMMetadataEntries(context.Background(), remote.Name); len(rows) != 0 {
t.Fatalf("second replica derived metadata while lease held: %d rows", len(rows))
}
}
// With the syncer wired and the cache empty, a repodata request enqueues a
// prime and, when it has not landed within the bounded cold wait, returns a
// retryable 503 rather than serving empty repodata (and without regressing the
// detached-context serve).
func TestServeRemoteColdStartReturns503(t *testing.T) {
fx := newGitHubFixture(t, true)
store := newFakeSyncStore()
p := newTestProvider()
p.coldWait = 300 * time.Millisecond
_ = newSyncer(store, p, testSyncConfig()) // binds p.syncer, but no workers running
remote := fx.remote()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil)
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", "https://x", store) {
t.Fatal("ServeRemote did not handle repomd.xml")
}
if rec.Code != http.StatusServiceUnavailable {
t.Fatalf("cold empty cache must return 503, got %d", rec.Code)
}
if rec.Header().Get("Retry-After") == "" {
t.Fatal("503 should carry Retry-After")
}
// The prime was enqueued.
if got := len(p.syncer.jobs); got != 1 {
t.Fatalf("cold start did not enqueue a prime, jobs=%d", got)
}
}
// With the cache warm, the same request serves repodata immediately (no 503).
func TestServeRemoteWarmCacheServesImmediately(t *testing.T) {
fx := newGitHubFixture(t, true)
store := newFakeSyncStore()
p := newTestProvider()
_ = newSyncer(store, p, testSyncConfig())
remote := fx.remote()
if err := p.scan(context.Background(), remote, store); err != nil {
t.Fatalf("warm scan: %v", err)
}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil)
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", "https://x", store) {
t.Fatal("ServeRemote did not handle repomd.xml")
}
if rec.Code != http.StatusOK {
t.Fatalf("warm cache must serve 200, got %d body=%s", rec.Code, rec.Body.String())
}
}
// A prime job (freshness 0) runs even right after a sync, deriving metadata,
// while a periodic job at the same moment is gated by the recency window.
func TestSyncerPrimeBypassesRecencyPeriodicDoesNot(t *testing.T) {
fx := newGitHubFixture(t, true)
fx.etag = `"v1"`
store := newFakeSyncStore()
p := newTestProvider()
s := newSyncer(store, p, testSyncConfig())
remote := fx.remote()
var _ provider.RemoteMetadataStore = store
// Prime derives despite no prior sync.
s.process(context.Background(), syncJob{remote: remote, prime: true})
if rows, _ := store.ListRPMMetadataEntries(context.Background(), remote.Name); len(rows) != 1 {
t.Fatalf("prime did not derive: %d rows", len(rows))
}
releasesAfterPrime := fx.releasesHit
// A periodic job immediately after is gated by mutable_ttl recency: no new
// releases call.
s.process(context.Background(), syncJob{remote: remote, prime: false})
if fx.releasesHit != releasesAfterPrime {
t.Fatalf("periodic scan ran inside recency window: %d -> %d releases calls", releasesAfterPrime, fx.releasesHit)
}
}
+21
View File
@@ -26,6 +26,27 @@ var providerZipRe = regexp.MustCompile(
var semverRe = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(?:-[a-zA-Z0-9.]+)?$`)
// ParsedProviderZip describes a terraform-provider-{type}_{version}_{os}_{arch}.zip
// filename. Ok is false when the name doesn't match that convention.
type ParsedProviderZip struct {
Type string
Version string
OS string
Arch string
Ok bool
}
// ParseProviderZip extracts the type, version and platform from a provider zip
// filename (the base name, not a full path). It's the canonical parser shared by
// the network-mirror index and the provider registry handler.
func ParseProviderZip(filename string) ParsedProviderZip {
m := providerZipRe.FindStringSubmatch(filename)
if m == nil {
return ParsedProviderZip{}
}
return ParsedProviderZip{Type: m[1], Version: m[2], OS: m[3], Arch: m[4], Ok: true}
}
type Provider struct{}
func (p *Provider) Type() models.PackageType { return models.PackageTerraform }
@@ -0,0 +1,171 @@
package terraform
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type fakeFileStore struct{ entries []provider.FileEntry }
func (f fakeFileStore) ListFilesByPrefix(_ context.Context, _, prefix string) ([]provider.FileEntry, error) {
var out []provider.FileEntry
for _, e := range f.entries {
if strings.HasPrefix(e.FilePath, prefix) {
out = append(out, e)
}
}
return out, nil
}
func (f fakeFileStore) ListPackages(_ context.Context, _ string) ([]string, error) { return nil, nil }
func TestTFPureFuncs(t *testing.T) {
p := &Provider{}
if p.Classify("hashicorp/aws/versions") != provider.Mutable {
t.Error("versions should be mutable")
}
if p.Classify("hashicorp/aws/terraform-provider-aws_1.0.0_linux_amd64.zip") != provider.Immutable {
t.Error("zip should be immutable")
}
if got := p.UpstreamURL(models.Remote{BaseURL: "https://registry.terraform.io"}, "hashicorp/aws/versions"); got != "https://registry.terraform.io/v1/providers/hashicorp/aws/versions" {
t.Errorf("upstream url %q", got)
}
h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
if h.Get("Authorization") == "" {
t.Error("auth header")
}
_ = p.ContentType("x.json")
}
func TestTFValidateUpload(t *testing.T) {
p := &Provider{}
sp, ct, err := p.ValidateUpload("hashicorp/aws/terraform-provider-aws_1.2.3_linux_amd64.zip")
if err != nil || sp != "hashicorp/aws/terraform-provider-aws_1.2.3_linux_amd64.zip" || ct != "application/zip" {
t.Errorf("valid: sp=%q ct=%q err=%v", sp, ct, err)
}
if _, _, err := p.ValidateUpload("too/few"); err == nil {
t.Error("expected error for wrong path depth")
}
if _, _, err := p.ValidateUpload("ns/aws/not-a-provider.zip"); err == nil {
t.Error("expected error for bad filename")
}
if _, _, err := p.ValidateUpload("ns/gcp/terraform-provider-aws_1.0.0_linux_amd64.zip"); err == nil {
t.Error("expected error for type mismatch")
}
}
func TestTFUploadResponse(t *testing.T) {
p := &Provider{}
resp := p.UploadResponse("hashicorp/aws/terraform-provider-aws_1.2.3_linux_amd64.zip", "sha256:abc", 100)
if resp["namespace"] != "hashicorp" || resp["type"] != "aws" || resp["version"] != "1.2.3" || resp["os"] != "linux" || resp["arch"] != "amd64" {
t.Errorf("structured response wrong: %v", resp)
}
fallback := p.UploadResponse("weird/path", "sha256:x", 1)
if fallback["path"] != "weird/path" {
t.Errorf("fallback response wrong: %v", fallback)
}
}
func TestTFRewriteResponse(t *testing.T) {
p := &Provider{}
remote := models.Remote{Name: "tf", ReleasesRemote: "hashicorp-releases"}
if out, _ := p.RewriteResponse([]byte(`{"download_url":"x"}`), models.Remote{}, "http://proxy"); out != nil {
t.Error("no ReleasesRemote should be a no-op")
}
if out, _ := p.RewriteResponse([]byte("not json"), remote, "http://proxy"); out != nil {
t.Error("invalid json should be a no-op")
}
body := []byte(`{"download_url":"https://releases.hashicorp.com/terraform-provider-aws/1.0/aws.zip"}`)
out, err := p.RewriteResponse(body, remote, "http://proxy")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(out), "http://proxy/api/v1/remote/hashicorp-releases/") {
t.Errorf("download_url not rewritten: %s", out)
}
}
func TestTFServeLocalIndex(t *testing.T) {
p := &Provider{}
fs := fakeFileStore{entries: []provider.FileEntry{
{FilePath: "hashicorp/aws/terraform-provider-aws_1.0.0_linux_amd64.zip", ContentHash: "sha256:deadbeef"},
{FilePath: "hashicorp/aws/terraform-provider-aws_1.0.0_darwin_arm64.zip", ContentHash: "sha256:cafe"},
}}
serve := func(path string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
p.ServeLocalIndex(w, r, fs, "repo", path)
return w
}
if w := serve("hashicorp/aws/index.json"); w.Code != 200 || !strings.Contains(w.Body.String(), "1.0.0") {
t.Errorf("index.json: code=%d body=%s", w.Code, w.Body.String())
}
if w := serve("hashicorp/aws/1.0.0.json"); w.Code != 200 || !strings.Contains(w.Body.String(), "linux_amd64") {
t.Errorf("version doc: code=%d body=%s", w.Code, w.Body.String())
}
// Not a terraform index path.
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/x", nil)
if p.ServeLocalIndex(w, r, fs, "repo", "hashicorp/aws/other.txt") {
t.Error("non-index path should return false")
}
if p.ServeLocalIndex(httptest.NewRecorder(), r, fs, "repo", "too/short") {
t.Error("short path should return false")
}
}
func TestTFContentTypeAndEmptyIndex(t *testing.T) {
p := &Provider{}
for path, want := range map[string]string{
"x.zip": "application/zip",
"x.sig": "application/octet-stream",
"index.json": "application/json",
} {
if got := p.ContentType(path); got != want {
t.Errorf("ContentType(%q)=%q want %q", path, got, want)
}
}
// index / version doc with no matching files -> 404.
empty := fakeFileStore{}
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/hashicorp/aws/index.json", nil)
p.ServeLocalIndex(w, r, empty, "repo", "hashicorp/aws/index.json")
if w.Code != http.StatusNotFound {
t.Errorf("empty index should be 404, got %d", w.Code)
}
w = httptest.NewRecorder()
p.ServeLocalIndex(w, r, empty, "repo", "hashicorp/aws/1.0.0.json")
if w.Code != http.StatusNotFound {
t.Errorf("empty version doc should be 404, got %d", w.Code)
}
}
func TestRewriteDownloadURL(t *testing.T) {
// Empty proxy base -> unchanged.
if got := rewriteDownloadURL("https://x/a.zip", "rel", ""); got != "https://x/a.zip" {
t.Errorf("empty base: %q", got)
}
// Unparseable URL -> unchanged.
if got := rewriteDownloadURL("://bad", "rel", "http://p"); got != "://bad" {
t.Errorf("bad url: %q", got)
}
// Normal rewrite.
if got := rewriteDownloadURL("https://cdn/path/a.zip", "rel", "http://p"); got != "http://p/api/v1/remote/rel/path/a.zip" {
t.Errorf("rewrite: %q", got)
}
}
func TestTFGenerateLocalIndexUnsupported(t *testing.T) {
if _, err := (&Provider{}).GenerateLocalIndex(context.Background(), fakeFileStore{}, "r", "x"); err == nil {
t.Error("expected unsupported error")
}
}
+75 -2
View File
@@ -12,13 +12,16 @@ import (
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
tfregistry "git.unkin.net/unkin/artifactapi/internal/api/terraform"
v1 "git.unkin.net/unkin/artifactapi/internal/api/v1"
v2 "git.unkin.net/unkin/artifactapi/internal/api/v2"
"git.unkin.net/unkin/artifactapi/internal/cache"
"git.unkin.net/unkin/artifactapi/internal/config"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/gc"
"git.unkin.net/unkin/artifactapi/internal/githubauth"
_ "git.unkin.net/unkin/artifactapi/internal/provider/alpine"
"git.unkin.net/unkin/artifactapi/internal/provider/deb"
_ "git.unkin.net/unkin/artifactapi/internal/provider/docker"
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
_ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
@@ -26,11 +29,13 @@ import (
_ "git.unkin.net/unkin/artifactapi/internal/provider/npm"
_ "git.unkin.net/unkin/artifactapi/internal/provider/puppet"
_ "git.unkin.net/unkin/artifactapi/internal/provider/pypi"
_ "git.unkin.net/unkin/artifactapi/internal/provider/rpm"
"git.unkin.net/unkin/artifactapi/internal/provider/rpm"
_ "git.unkin.net/unkin/artifactapi/internal/provider/terraform"
"git.unkin.net/unkin/artifactapi/internal/proxy"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/internal/tfsign"
"git.unkin.net/unkin/artifactapi/internal/virtual"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type Server struct {
@@ -43,7 +48,10 @@ type Server struct {
engine *proxy.Engine
virtEngine *virtual.Engine
localHandler *v2.LocalHandler
tfRegistry *tfregistry.Handler
gc *gc.Collector
syncer *rpm.Syncer
debSyncer *deb.Syncer
}
func New(cfg *config.Config, version string) (*Server, error) {
@@ -62,10 +70,60 @@ func New(cfg *config.Config, version string) (*Server, error) {
return nil, fmt.Errorf("s3: %w", err)
}
// Install the process-wide GitHub credential before any provider makes an
// outbound call. A misconfiguration (e.g. App id without a private key) fails
// closed here rather than silently falling back to anonymous. No credential
// configured is fine — requests stay anonymous.
ghCred, err := githubauth.New(githubauth.Options{
Token: cfg.GitHubToken,
AppID: cfg.GitHubAppID,
InstallationID: cfg.GitHubAppInstallationID,
PrivateKeyPEM: cfg.GitHubAppPrivateKey,
PrivateKeyPath: cfg.GitHubAppPrivateKeyPath,
})
if err != nil {
return nil, fmt.Errorf("github auth: %w", err)
}
githubauth.SetServer(ghCred)
if ghCred != nil {
slog.Info("github machine credential configured")
}
engine := proxy.NewEngine(db, redis, s3)
localHandler := v2.NewLocalHandler(db, s3)
virtEngine := virtual.NewEngine(db, engine)
collector := gc.New(db, s3, 1*time.Hour)
syncer := rpm.NewSyncer(db, rpm.SyncConfig{
RatePerSec: cfg.GitHubSyncRatePerSec,
Burst: cfg.GitHubSyncBurst,
Workers: cfg.GitHubSyncWorkers,
PollInterval: time.Duration(cfg.GitHubSyncPollInterval) * time.Second,
})
debSyncer := deb.NewSyncer(db, deb.SyncConfig{
RatePerSec: cfg.GitHubSyncRatePerSec,
Burst: cfg.GitHubSyncBurst,
Workers: cfg.GitHubSyncWorkers,
PollInterval: time.Duration(cfg.GitHubSyncPollInterval) * time.Second,
})
// The terraform registry signs with a GPG key. A configured file wins (BYO
// key); otherwise artifactapi generates one on first start and persists it in
// the database so every replica shares it. A failure here must not take the
// server down — the registry just stays disabled.
var signer *tfsign.Signer
if cfg.TFSigningKeyPath != "" {
signer, err = tfsign.Load(cfg.TFSigningKeyPath, cfg.TFSigningKeyPassphrase)
} else {
signer, err = tfsign.LoadOrCreate(context.Background(), db, "terraform-provider")
}
if err != nil {
slog.Warn("terraform provider registry disabled", "error", err)
signer = nil
}
tfRegistry := tfregistry.NewHandler(db, signer, cfg.TFProviderProtocols)
if tfRegistry.Enabled() {
slog.Info("terraform provider registry enabled", "key_id", signer.KeyID())
}
s := &Server{
cfg: cfg,
@@ -76,7 +134,10 @@ func New(cfg *config.Config, version string) (*Server, error) {
engine: engine,
virtEngine: virtEngine,
localHandler: localHandler,
tfRegistry: tfRegistry,
gc: collector,
syncer: syncer,
debSyncer: debSyncer,
}
s.router = s.routes()
@@ -97,11 +158,19 @@ func (s *Server) routes() chi.Router {
r.Get("/", s.handleRoot)
r.Get("/version", s.handleVersion)
// Terraform provider registry: service discovery at the well-known path,
// providers.v1 protocol under /terraform/v1/providers.
r.Get("/.well-known/terraform.json", s.tfRegistry.ServiceDiscovery)
r.Mount(tfregistry.MountPath, s.tfRegistry.Routes())
proxyHandler := v1.NewProxyHandler(s.engine, s.virtEngine, s.db, s.store, s.localHandler)
r.Mount("/api/v1", proxyHandler.Routes())
r.Mount("/v2", proxyHandler.DockerV2Routes())
remotesHandler := v2.NewRemotesHandler(s.db)
remotesHandler := v2.NewRemotesHandler(s.db, map[models.PackageType]v2.Primer{
models.PackageGitHubRPM: s.syncer,
models.PackageGitHubDeb: s.debSyncer,
})
virtualsHandler := v2.NewVirtualsHandler(s.db)
healthHandler := v2.NewHealthHandler(s.db, s.cache, s.store)
statsHandler := v2.NewStatsHandler(s.db)
@@ -168,6 +237,8 @@ func (s *Server) newHTTPServer() *http.Server {
func (s *Server) Run(ctx context.Context) error {
go s.gc.Run(ctx)
go s.syncer.Run(ctx)
go s.debSyncer.Run(ctx)
httpServer := s.newHTTPServer()
@@ -188,6 +259,8 @@ func (s *Server) Run(ctx context.Context) error {
func (s *Server) RunOnListener(ctx context.Context, ln net.Listener) error {
go s.gc.Run(ctx)
go s.syncer.Run(ctx)
go s.debSyncer.Run(ctx)
httpServer := s.newHTTPServer()
+16
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"io"
"log/slog"
"time"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
@@ -97,3 +98,18 @@ func (s *S3) Stat(ctx context.Context, key string) (*minio.ObjectInfo, error) {
}
return &info, nil
}
// ListStaleObjects returns keys under prefix last modified before cutoff. Used
// by the GC to reap abandoned staging objects (e.g. cancelled docker pushes).
func (s *S3) ListStaleObjects(ctx context.Context, prefix string, cutoff time.Time) ([]string, error) {
var keys []string
for obj := range s.client.ListObjects(ctx, s.bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: true}) {
if obj.Err != nil {
return nil, obj.Err
}
if obj.LastModified.Before(cutoff) {
keys = append(keys, obj.Key)
}
}
return keys, nil
}
+61
View File
@@ -0,0 +1,61 @@
package testsupport
import (
"archive/tar"
"bytes"
"compress/gzip"
"fmt"
)
// MinimalDeb builds a valid-enough Debian package in pure Go (no committed
// binary fixture, no dpkg-deb): an ar archive of debian-binary, a gzip
// control.tar.gz carrying ./control, and an (empty) gzip data.tar.gz. It is the
// deb analog of MinimalRPM and is parseable by the deb provider.
func MinimalDeb(name, version, arch string) []byte {
control := fmt.Sprintf(
"Package: %s\nVersion: %s\nArchitecture: %s\nMaintainer: e2e <e2e@example.com>\n"+
"Section: utils\nPriority: optional\nDescription: minimal test package\n",
name, version, arch)
controlTarGz := gzipBytes(tarSingle("./control", []byte(control)))
dataTarGz := gzipBytes(tarEmpty())
var buf bytes.Buffer
buf.WriteString("!<arch>\n")
arWrite(&buf, "debian-binary", []byte("2.0\n"))
arWrite(&buf, "control.tar.gz", controlTarGz)
arWrite(&buf, "data.tar.gz", dataTarGz)
return buf.Bytes()
}
func arWrite(buf *bytes.Buffer, name string, data []byte) {
fmt.Fprintf(buf, "%-16s%-12s%-6s%-6s%-8s%-10d`\n", name, "0", "0", "0", "100644", len(data))
buf.Write(data)
if len(data)%2 == 1 {
buf.WriteByte('\n')
}
}
func tarSingle(name string, data []byte) []byte {
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(data)), Typeflag: tar.TypeReg})
tw.Write(data)
tw.Close()
return buf.Bytes()
}
func tarEmpty() []byte {
var buf bytes.Buffer
tw := tar.NewWriter(&buf)
tw.Close()
return buf.Bytes()
}
func gzipBytes(data []byte) []byte {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
gz.Write(data)
gz.Close()
return buf.Bytes()
}
+173
View File
@@ -0,0 +1,173 @@
// Package tfsign loads a GPG signing key and produces the detached signatures
// the Terraform provider registry protocol requires over SHA256SUMS files.
package tfsign
import (
"bytes"
"context"
"fmt"
"os"
"strings"
"golang.org/x/crypto/openpgp"
"golang.org/x/crypto/openpgp/armor"
)
// KeyStore persists a generated signing key. *database.DB satisfies it.
type KeyStore interface {
GetSigningKey(ctx context.Context, purpose string) (armor, keyID string, found bool, err error)
InsertSigningKeyIfAbsent(ctx context.Context, purpose, armor, keyID string) error
}
// LoadOrCreate returns a signer for purpose, generating and persisting a new key
// the first time it is needed. It is safe across replicas: a lost insert race
// just re-reads whichever key won.
func LoadOrCreate(ctx context.Context, store KeyStore, purpose string) (*Signer, error) {
armored, _, found, err := store.GetSigningKey(ctx, purpose)
if err != nil {
return nil, err
}
if !found {
newArmor, keyID, err := Generate()
if err != nil {
return nil, err
}
if err := store.InsertSigningKeyIfAbsent(ctx, purpose, newArmor, keyID); err != nil {
return nil, err
}
if armored, _, _, err = store.GetSigningKey(ctx, purpose); err != nil {
return nil, err
}
}
return LoadArmored(armored, "")
}
// Signer holds a decrypted GPG entity and exposes what the registry download
// response needs: a detached signature, the armored public key, and the key ID.
type Signer struct {
entity *openpgp.Entity
publicASCII string
keyID string
}
// Load reads an armored private key from path, decrypting it with passphrase if
// the key is protected. A blank path returns (nil, nil): a nil *Signer means the
// caller should fall back to another source (e.g. a DB-stored key).
func Load(path, passphrase string) (*Signer, error) {
if path == "" {
return nil, nil
}
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("open signing key: %w", err)
}
return fromArmor(string(data), passphrase, path)
}
// LoadArmored builds a signer from an in-memory armored private key, e.g. one
// read from the database. A blank key returns (nil, nil).
func LoadArmored(armored, passphrase string) (*Signer, error) {
if armored == "" {
return nil, nil
}
return fromArmor(armored, passphrase, "stored key")
}
// Generate creates a fresh signing keypair and returns the armored private key
// (to persist) and its uppercase key id.
func Generate() (armoredPrivateKey, keyID string, err error) {
entity, err := openpgp.NewEntity("artifactapi terraform registry", "provider signing", "artifactapi@localhost", nil)
if err != nil {
return "", "", err
}
var buf bytes.Buffer
w, err := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
if err != nil {
return "", "", err
}
if err := entity.SerializePrivate(w, nil); err != nil {
return "", "", err
}
if err := w.Close(); err != nil {
return "", "", err
}
return buf.String(), strings.ToUpper(entity.PrimaryKey.KeyIdString()), nil
}
func fromArmor(armored, passphrase, src string) (*Signer, error) {
keyring, err := openpgp.ReadArmoredKeyRing(strings.NewReader(armored))
if err != nil {
return nil, fmt.Errorf("read signing key: %w", err)
}
if len(keyring) == 0 {
return nil, fmt.Errorf("signing key (%s) contains no entities", src)
}
entity := keyring[0]
if entity.PrivateKey == nil {
return nil, fmt.Errorf("signing key (%s) has no private key material", src)
}
if entity.PrivateKey.Encrypted {
if err := decrypt(entity, passphrase); err != nil {
return nil, err
}
}
pub, err := armorPublicKey(entity)
if err != nil {
return nil, err
}
return &Signer{
entity: entity,
publicASCII: pub,
keyID: entity.PrimaryKey.KeyIdString(),
}, nil
}
// decrypt unlocks the entity's private key and all subkeys with the passphrase.
func decrypt(entity *openpgp.Entity, passphrase string) error {
pw := []byte(passphrase)
if err := entity.PrivateKey.Decrypt(pw); err != nil {
return fmt.Errorf("decrypt signing key: %w", err)
}
for _, sub := range entity.Subkeys {
if sub.PrivateKey != nil && sub.PrivateKey.Encrypted {
_ = sub.PrivateKey.Decrypt(pw)
}
}
return nil
}
func armorPublicKey(entity *openpgp.Entity) (string, error) {
var buf bytes.Buffer
w, err := armor.Encode(&buf, openpgp.PublicKeyType, nil)
if err != nil {
return "", err
}
if err := entity.Serialize(w); err != nil {
return "", err
}
if err := w.Close(); err != nil {
return "", err
}
return buf.String(), nil
}
// Sign returns a binary detached signature over message, matching the
// SHA256SUMS.sig format Terraform verifies.
func (s *Signer) Sign(message []byte) ([]byte, error) {
var buf bytes.Buffer
if err := openpgp.DetachSign(&buf, s.entity, bytes.NewReader(message), nil); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// PublicKeyArmor returns the ASCII-armored public key for the registry's
// signing_keys response.
func (s *Signer) PublicKeyArmor() string { return s.publicASCII }
// KeyID returns the 16-hex-char uppercase key ID Terraform matches against the
// signature's issuer.
func (s *Signer) KeyID() string { return strings.ToUpper(s.keyID) }
+155
View File
@@ -0,0 +1,155 @@
package tfsign
import (
"bytes"
"context"
"os"
"path/filepath"
"regexp"
"testing"
"golang.org/x/crypto/openpgp"
"golang.org/x/crypto/openpgp/armor"
)
// armoredPrivateKey generates a throwaway armored private key for tests.
func armoredPrivateKey(t *testing.T) string {
t.Helper()
e, err := openpgp.NewEntity("artifactapi test", "tf registry", "tf@example.com", nil)
if err != nil {
t.Fatal(err)
}
var buf bytes.Buffer
w, err := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
if err != nil {
t.Fatal(err)
}
if err := e.SerializePrivate(w, nil); err != nil {
t.Fatal(err)
}
w.Close()
return buf.String()
}
func writeKey(t *testing.T, contents string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "private-key.asc")
if err := os.WriteFile(p, []byte(contents), 0o600); err != nil {
t.Fatal(err)
}
return p
}
func TestLoadSignAndVerify(t *testing.T) {
path := writeKey(t, armoredPrivateKey(t))
s, err := Load(path, "")
if err != nil {
t.Fatal(err)
}
if s == nil {
t.Fatal("expected a signer")
}
if !regexp.MustCompile(`^[0-9A-F]{16}$`).MatchString(s.KeyID()) {
t.Errorf("key id %q is not 16 uppercase hex chars", s.KeyID())
}
msg := []byte("deadbeef terraform-provider-x_1.0.0_linux_amd64.zip\n")
sig, err := s.Sign(msg)
if err != nil {
t.Fatal(err)
}
// The advertised public key must verify the signature over the same bytes.
keyring, err := openpgp.ReadArmoredKeyRing(bytes.NewReader([]byte(s.PublicKeyArmor())))
if err != nil {
t.Fatal(err)
}
if _, err := openpgp.CheckDetachedSignature(keyring, bytes.NewReader(msg), bytes.NewReader(sig)); err != nil {
t.Errorf("signature did not verify: %v", err)
}
}
func TestGenerateAndLoadArmored(t *testing.T) {
priv, keyID, err := Generate()
if err != nil {
t.Fatal(err)
}
if !regexp.MustCompile(`^[0-9A-F]{16}$`).MatchString(keyID) {
t.Errorf("generated key id %q malformed", keyID)
}
s, err := LoadArmored(priv, "")
if err != nil {
t.Fatal(err)
}
if s.KeyID() != keyID {
t.Errorf("loaded key id %q != generated %q", s.KeyID(), keyID)
}
msg := []byte("abc terraform-provider-x_1.0.0_linux_amd64.zip\n")
sig, err := s.Sign(msg)
if err != nil {
t.Fatal(err)
}
keyring, _ := openpgp.ReadArmoredKeyRing(bytes.NewReader([]byte(s.PublicKeyArmor())))
if _, err := openpgp.CheckDetachedSignature(keyring, bytes.NewReader(msg), bytes.NewReader(sig)); err != nil {
t.Errorf("signature did not verify: %v", err)
}
}
// memStore is an in-memory KeyStore that records how many keys it accepted.
type memStore struct {
armor, keyID string
found bool
inserts int
}
func (m *memStore) GetSigningKey(_ context.Context, _ string) (string, string, bool, error) {
return m.armor, m.keyID, m.found, nil
}
func (m *memStore) InsertSigningKeyIfAbsent(_ context.Context, _, armor, keyID string) error {
if !m.found { // ON CONFLICT DO NOTHING
m.armor, m.keyID, m.found = armor, keyID, true
m.inserts++
}
return nil
}
func TestLoadOrCreateGeneratesOnceThenReuses(t *testing.T) {
store := &memStore{}
first, err := LoadOrCreate(context.Background(), store, "terraform-provider")
if err != nil || first == nil {
t.Fatalf("first LoadOrCreate: signer=%v err=%v", first, err)
}
second, err := LoadOrCreate(context.Background(), store, "terraform-provider")
if err != nil || second == nil {
t.Fatalf("second LoadOrCreate: signer=%v err=%v", second, err)
}
if store.inserts != 1 {
t.Errorf("expected exactly one key generated, got %d", store.inserts)
}
if first.KeyID() != second.KeyID() {
t.Errorf("key id changed between loads: %q vs %q", first.KeyID(), second.KeyID())
}
}
func TestLoadEmptyPathDisabled(t *testing.T) {
s, err := Load("", "")
if err != nil {
t.Fatal(err)
}
if s != nil {
t.Error("empty path should yield a nil (disabled) signer")
}
}
func TestLoadMissingFile(t *testing.T) {
if _, err := Load(filepath.Join(t.TempDir(), "nope.asc"), ""); err == nil {
t.Error("expected an error for a missing key file")
}
}
+6
View File
@@ -11,10 +11,13 @@ const (
PackagePyPI PackageType = "pypi"
PackageNPM PackageType = "npm"
PackageRPM PackageType = "rpm"
PackageDeb PackageType = "deb"
PackageAlpine PackageType = "alpine"
PackagePuppet PackageType = "puppet"
PackageTerraform PackageType = "terraform"
PackageGoProxy PackageType = "goproxy"
PackageGitHubRPM PackageType = "github_rpm"
PackageGitHubDeb PackageType = "github_deb"
)
var validPackageTypes = map[PackageType]bool{
@@ -24,10 +27,13 @@ var validPackageTypes = map[PackageType]bool{
PackagePyPI: true,
PackageNPM: true,
PackageRPM: true,
PackageDeb: true,
PackageAlpine: true,
PackagePuppet: true,
PackageTerraform: true,
PackageGoProxy: true,
PackageGitHubRPM: true,
PackageGitHubDeb: true,
}
func (p PackageType) Valid() bool {
+1
View File
@@ -18,6 +18,7 @@ func TestPackageTypeValid(t *testing.T) {
models.PackagePuppet,
models.PackageTerraform,
models.PackageGoProxy,
models.PackageGitHubRPM,
}
for _, pt := range valid {
if !pt.Valid() {
+99
View File
@@ -0,0 +1,99 @@
.usage-panel {
margin: 24px 0;
background: var(--bg-surface);
border: 1px solid var(--border);
border-radius: var(--radius);
overflow: hidden;
}
.usage-toggle {
display: flex;
align-items: center;
gap: 8px;
width: 100%;
padding: 14px 18px;
background: transparent;
border: none;
color: var(--text-bright);
font-size: 0.95em;
font-weight: 600;
cursor: pointer;
text-align: left;
}
.usage-toggle:hover {
background: var(--bg-elevated);
}
.usage-caret {
display: inline-block;
transition: transform 0.15s;
color: var(--text-muted);
font-size: 0.9em;
}
.usage-caret.open {
transform: rotate(90deg);
}
.usage-body {
padding: 4px 18px 18px;
border-top: 1px solid var(--border);
display: flex;
flex-direction: column;
gap: 18px;
}
.usage-snippet-title {
font-size: 0.85em;
font-weight: 600;
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.03em;
margin: 14px 0 8px;
}
.usage-codebox {
position: relative;
background: var(--bg);
border: 1px solid var(--border);
border-radius: var(--radius);
}
.usage-codebox pre {
margin: 0;
padding: 14px 16px;
overflow-x: auto;
font-family: var(--font-mono);
font-size: 0.85em;
line-height: 1.5;
color: var(--text-bright);
white-space: pre;
}
.usage-copy-btn {
position: absolute;
top: 8px;
right: 8px;
padding: 3px 10px;
font-size: 0.75em;
font-family: var(--font-sans);
color: var(--text-muted);
background: var(--bg-elevated);
border: 1px solid var(--border);
border-radius: 4px;
cursor: pointer;
transition: all 0.15s;
}
.usage-copy-btn:hover {
color: var(--text-bright);
border-color: var(--accent);
}
.usage-note {
margin-top: 8px;
font-size: 0.82em;
color: var(--text-muted);
line-height: 1.45;
}
+331
View File
@@ -0,0 +1,331 @@
import { useState } from 'react';
import './UsageInstructions.css';
// repoClass distinguishes the three ways a repository is consumed. remotes are
// caching proxies, locals are real registries you also publish to, virtuals are
// merged read-only indexes.
type RepoClass = 'remote' | 'local' | 'virtual';
interface Snippet {
title: string;
language: string;
code: string;
note?: string;
}
// baseURL resolves the externally reachable origin of this artifactapi instance.
// The UI is served on the same origin as the API (client BASE is ''), so
// window.location.origin is the address a host would actually curl/pull against
// — no hardcoded hostname, works in prod and in `npm run dev` behind a proxy.
function baseURL(): string {
if (typeof window !== 'undefined' && window.location?.origin) {
return window.location.origin.replace(/\/$/, '');
}
return 'https://artifactapi.k8s.syd1.au.unkin.net';
}
// hostOnly is the bare host[:port] with no scheme, for docker/terraform source
// addresses which are scheme-less.
function hostOnly(): string {
try {
return new URL(baseURL()).host;
} catch {
return 'artifactapi.k8s.syd1.au.unkin.net';
}
}
// remoteProxyBase is where a remote (or virtual) repo's proxied artifacts live.
function remoteProxyBase(cls: RepoClass, name: string): string {
const seg = cls === 'virtual' ? 'virtual' : 'remote';
return `${baseURL()}/api/v1/${seg}/${name}`;
}
export function buildSnippets(packageType: string, repoClass: RepoClass, name: string): Snippet[] {
const url = baseURL();
const host = hostOnly();
const proxy = remoteProxyBase(repoClass, name);
const isLocal = repoClass === 'local';
switch (packageType) {
case 'rpm':
return [
{
title: isLocal
? 'Add the yum repo (real yum repo, repodata auto-regenerated)'
: 'Add the yum repo (caching proxy)',
language: 'bash',
code: `sudo tee /etc/yum.repos.d/${name}.repo >/dev/null <<'EOF'
[${name}]
name=${name} (artifactapi)
baseurl=${isLocal ? `${url}/api/v2/remotes/${name}/files/` : `${proxy}/`}
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
sudo dnf install <package>`,
note: isLocal
? 'gpgcheck=0: artifactapi serves the repo unsigned. If you sign your RPMs, import your key and set gpgcheck=1.'
: 'gpgcheck=0 trusts upstream over the proxy. To verify package signatures, import the upstream GPG key and set gpgcheck=1.',
},
...(isLocal
? [
{
title: 'Publish an RPM (repodata regenerates automatically)',
language: 'bash',
code: `curl -fsSL --upload-file ./my-package-1.0-1.el9.x86_64.rpm \\
${url}/api/v2/remotes/${name}/files/my-package-1.0-1.el9.x86_64.rpm`,
},
]
: []),
];
case 'pypi':
return [
{
title: 'Install a package (one-off)',
language: 'bash',
code: `pip install --index-url ${proxy}/simple/ <package>`,
},
{
title: 'Configure pip persistently',
language: 'bash',
code: `mkdir -p ~/.config/pip
cat > ~/.config/pip/pip.conf <<'EOF'
[global]
index-url = ${proxy}/simple/
EOF
pip install <package>`,
},
];
case 'npm':
return [
{
title: 'Point npm at this registry',
language: 'bash',
code: `npm config set registry ${proxy}/
npm install <package>`,
},
{
title: 'Per-project (.npmrc)',
language: 'bash',
code: `echo 'registry=${proxy}/' >> .npmrc
npm install`,
},
];
case 'docker':
return [
{
title: 'Pull an image',
language: 'bash',
code: `docker pull ${host}/${name}/<image>:<tag>`,
note: 'The first path segment after the host is the artifactapi repo name; the rest is the image name.',
},
...(isLocal
? [
{
title: 'Push an image (this is a real Registry V2)',
language: 'bash',
code: `docker tag myapp:latest ${host}/${name}/myapp:latest
docker push ${host}/${name}/myapp:latest`,
note: 'Works with docker, podman, skopeo and buildah. If the registry requires auth, run `docker login ' + host + '` first.',
},
]
: []),
];
case 'terraform':
return [
{
title: 'Use as a provider source (bare address, no mirror config)',
language: 'hcl',
code: `terraform {
required_providers {
${name} = {
source = "${host}/${name}/<type>"
version = ">= 0.1.0"
}
}
}`,
note: 'The namespace segment is this repo name; <type> is the provider type. artifactapi signs SHA256SUMS server-side with its GPG key, so `terraform init` installs with no .terraformrc.',
},
...(isLocal
? [
{
title: 'Publish a provider build',
language: 'bash',
code: `curl -fsSL --upload-file terraform-provider-<type>_0.1.0_linux_amd64.zip \\
${url}/api/v2/remotes/${name}/files/${name}/<type>/terraform-provider-<type>_0.1.0_linux_amd64.zip`,
},
]
: []),
];
case 'helm':
return [
{
title: 'Add the Helm repo',
language: 'bash',
code: `helm repo add ${name} ${proxy}/
helm repo update
helm install <release> ${name}/<chart>`,
},
];
case 'alpine':
return [
{
title: 'Add the APK repository',
language: 'bash',
code: `echo '${proxy}/' | sudo tee -a /etc/apk/repositories
sudo apk update
sudo apk add <package>`,
note: 'If the index is unsigned over the proxy, add --allow-untrusted or install the signing key into /etc/apk/keys.',
},
];
case 'goproxy':
return [
{
title: 'Point the Go module proxy here',
language: 'bash',
code: `export GOPROXY=${proxy}
go mod download`,
note: 'Append ,direct to fall back to VCS for modules this proxy does not cover.',
},
];
case 'puppet':
return [
{
title: 'Install a module from the Forge proxy',
language: 'bash',
code: `puppet module install <author>-<module> \\
--module_repository ${proxy}`,
},
];
case 'deb':
return isLocal
? [
{
title: 'Add the apt repo (real apt repo, flat — Packages/Release auto-generated)',
language: 'bash',
code: `echo 'deb [trusted=yes] ${url}/api/v1/local/${name}/ ./' | sudo tee /etc/apt/sources.list.d/${name}.list
sudo apt-get update
sudo apt-get install <package>`,
note: '[trusted=yes]: artifactapi serves the flat repo unsigned (matches the rpm repo\'s gpgcheck=0). The `./` is the flat-repo suite — apt fetches Packages/Release from the repo root.',
},
{
title: 'Publish a .deb (index regenerates automatically)',
language: 'bash',
code: `curl -fsSL --upload-file ./my-package_1.0_amd64.deb \\
${url}/api/v2/remotes/${name}/files/my-package_1.0_amd64.deb`,
},
]
: [
{
title: 'Add the apt repo (caching proxy)',
language: 'bash',
code: `echo 'deb ${proxy} <suite> <component>' | sudo tee /etc/apt/sources.list.d/${name}.list
sudo apt-get update
sudo apt-get install <package>`,
note: "Signatures are verified against the upstream mirror's real signed Release through the proxy (no [trusted=yes] needed). Example suite/component: bookworm main.",
},
];
case 'github_deb':
return [
{
title: 'Add the apt repo (metadata-only, from GitHub releases)',
language: 'bash',
code: `echo 'deb [trusted=yes] ${proxy}/ ./' | sudo tee /etc/apt/sources.list.d/${name}.list
sudo apt-get update
sudo apt-get install <package>`,
note: "The apt index is synthesized from the configured GitHub repo's release .deb assets; package downloads are redirected to the backing releases remote. Served unsigned, so [trusted=yes].",
},
];
case 'generic':
default:
return [
{
title: 'Download a file',
language: 'bash',
code: `curl -fsSLO ${proxy}/<path>`,
note:
packageType === 'generic'
? 'Generic repos are fetched as plain files at their upstream path.'
: `No tailored client instructions for "${packageType}" yet — fetch artifacts directly by path.`,
},
...(isLocal
? [
{
title: 'Publish a file',
language: 'bash',
code: `curl -fsSL --upload-file ./myfile \\
${url}/api/v2/remotes/${name}/files/<path>/myfile`,
},
]
: []),
];
}
}
function CodeBox({ snippet }: { snippet: Snippet }) {
const [copied, setCopied] = useState(false);
async function copy() {
try {
await navigator.clipboard.writeText(snippet.code);
setCopied(true);
setTimeout(() => setCopied(false), 1500);
} catch {
// Clipboard API unavailable (e.g. non-secure context); silently ignore.
}
}
return (
<div className="usage-snippet">
<div className="usage-snippet-title">{snippet.title}</div>
<div className="usage-codebox">
<button className="usage-copy-btn" onClick={copy} type="button">
{copied ? 'copied' : 'copy'}
</button>
<pre className="mono">{snippet.code}</pre>
</div>
{snippet.note && <div className="usage-note">{snippet.note}</div>}
</div>
);
}
interface UsageInstructionsProps {
packageType: string;
repoClass: RepoClass;
name: string;
defaultOpen?: boolean;
}
export function UsageInstructions({ packageType, repoClass, name, defaultOpen = false }: UsageInstructionsProps) {
const [open, setOpen] = useState(defaultOpen);
const snippets = buildSnippets(packageType, repoClass, name);
return (
<div className="usage-panel">
<button className="usage-toggle" onClick={() => setOpen(o => !o)} type="button" aria-expanded={open}>
<span className={`usage-caret ${open ? 'open' : ''}`}>&#9656;</span>
How do I use this?
</button>
{open && (
<div className="usage-body">
{snippets.map((s, i) => (
<CodeBox key={i} snippet={s} />
))}
</div>
)}
</div>
);
}
+36
View File
@@ -0,0 +1,36 @@
// Per-repo-type "downloadable" capability map.
//
// When a repo's package_type has an entry here, file entries in the object
// browser render as direct-download links pointing at the URL the entry
// builds. Types absent from the map render as plain text. Enabling a new
// type is a one-line addition below.
//
// Download routes are same-origin, unauthenticated GETs (the API serves local
// repos as real registries with no token on reads), so a bare <a href download>
// works and carries no credentials.
// buildUrl receives the repo name and the artifact's full path (may contain
// slashes) and returns the direct-download URL for that file.
type DownloadUrlBuilder = (repo: string, path: string) => string;
export const downloadableTypes: Record<string, DownloadUrlBuilder> = {
// rpm locals are real yum repos; files are served at
// /api/v2/remotes/<repo>/files/<path>.
rpm: (repo, path) =>
`/api/v2/remotes/${encodeURIComponent(repo)}/files/${path
.split('/')
.map(encodeURIComponent)
.join('/')}`,
};
// downloadUrlFor returns the direct-download URL for a file when its repo type
// is downloadable, or null otherwise (render as plain text).
export function downloadUrlFor(
packageType: string | undefined,
repo: string,
path: string,
): string | null {
if (!packageType) return null;
const build = downloadableTypes[packageType];
return build ? build(repo, path) : null;
}
+3
View File
@@ -3,6 +3,7 @@ import { useParams, Link } from 'react-router-dom';
import { api } from '../api/client';
import type { Remote } from '../api/types';
import { Badge } from '../components/Badge';
import { UsageInstructions } from '../components/UsageInstructions';
import './RemoteDetail.css';
export function LocalDetail() {
@@ -36,6 +37,8 @@ export function LocalDetail() {
<p className="detail-description">{remote.description}</p>
)}
<UsageInstructions packageType={remote.package_type} repoClass="local" name={remote.name} />
<div className="detail-actions">
<Link to={`/locals/${remote.name}/objects`} className="btn btn-primary">
Browse Files
+9
View File
@@ -37,6 +37,15 @@
word-break: break-all;
}
.tree-file-link {
color: var(--accent, #4c9aff);
text-decoration: none;
}
.tree-file-link:hover {
text-decoration: underline;
}
.tree-dir {
cursor: pointer;
}
+35 -4
View File
@@ -3,6 +3,7 @@ import { useParams, useLocation, Link } from 'react-router-dom';
import { api } from '../api/client';
import type { Artifact } from '../api/types';
import { formatBytes, timeAgo, truncateHash } from '../components/format';
import { downloadUrlFor } from '../components/downloads';
import './Objects.css';
interface TreeNode {
@@ -100,11 +101,16 @@ interface TreeRowProps {
expanded: Set<string>;
onToggle: (path: string) => void;
onEvict: (path: string) => void;
repo: string;
packageType?: string;
}
function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
function TreeRow({ node, depth, expanded, onToggle, onEvict, repo, packageType }: TreeRowProps) {
const isDir = node.children.size > 0 && !node.artifact;
const isExpanded = expanded.has(node.path);
const downloadUrl = node.artifact
? downloadUrlFor(packageType, repo, node.artifact.path)
: null;
const sortedChildren = useMemo(() => {
if (!isDir) return [];
@@ -124,9 +130,20 @@ function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
{isDir && (
<span className="tree-toggle">{isExpanded ? '▾' : '▸'}</span>
)}
<span className={isDir ? 'tree-dir-name' : 'mono tree-file-name'}>
{node.name}{isDir ? '/' : ''}
</span>
{downloadUrl ? (
<a
className="mono tree-file-name tree-file-link"
href={downloadUrl}
download
onClick={(e) => e.stopPropagation()}
>
{node.name}
</a>
) : (
<span className={isDir ? 'tree-dir-name' : 'mono tree-file-name'}>
{node.name}{isDir ? '/' : ''}
</span>
)}
</span>
</td>
<td className="num-cell">{formatBytes(node.totalSize)}</td>
@@ -163,6 +180,8 @@ function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
expanded={expanded}
onToggle={onToggle}
onEvict={onEvict}
repo={repo}
packageType={packageType}
/>
))}
</>
@@ -175,6 +194,7 @@ export function Objects() {
const isLocal = location.pathname.startsWith('/locals/');
const backLink = isLocal ? `/locals/${name}` : `/remotes/${name}`;
const [artifacts, setArtifacts] = useState<Artifact[]>([]);
const [packageType, setPackageType] = useState<string | undefined>(undefined);
const [loading, setLoading] = useState(true);
const [filter, setFilter] = useState('');
const [expanded, setExpanded] = useState<Set<string>>(new Set());
@@ -190,6 +210,15 @@ export function Objects() {
useEffect(() => { load(); }, [load]);
// The repo's package_type is the modularity hook: it decides whether file
// names render as direct-download links (see downloadableTypes).
useEffect(() => {
if (!name) return;
api.getRemote(name)
.then(r => setPackageType(r.package_type))
.catch(() => setPackageType(undefined));
}, [name]);
const handleEvict = async (path: string) => {
if (!name || !confirm(`Evict ${path}?`)) return;
await (isLocal ? api.evictLocalObject(name, path) : api.evictObject(name, path));
@@ -287,6 +316,8 @@ export function Objects() {
expanded={expanded}
onToggle={toggleExpand}
onEvict={handleEvict}
repo={name!}
packageType={packageType}
/>
))
)}
+3
View File
@@ -3,6 +3,7 @@ import { useParams, Link } from 'react-router-dom';
import { api } from '../api/client';
import type { Remote } from '../api/types';
import { Badge } from '../components/Badge';
import { UsageInstructions } from '../components/UsageInstructions';
import './RemoteDetail.css';
export function RemoteDetail() {
@@ -109,6 +110,8 @@ export function RemoteDetail() {
)}
</div>
<UsageInstructions packageType={remote.package_type} repoClass="remote" name={remote.name} />
<div className="detail-actions">
<Link to={`/remotes/${remote.name}/objects`} className="btn btn-primary">
Browse Objects
+7
View File
@@ -4,6 +4,7 @@ import { api } from '../api/client';
import type { Remote, Virtual } from '../api/types';
import { Badge } from '../components/Badge';
import { DataTable } from '../components/DataTable';
import { UsageInstructions } from '../components/UsageInstructions';
import './Virtuals.css';
export function Virtuals() {
@@ -98,6 +99,12 @@ export function Virtuals() {
);
})}
</ul>
{(() => {
const v = virtuals.find(x => x.name === expanded);
return v ? (
<UsageInstructions packageType={v.package_type} repoClass="virtual" name={v.name} defaultOpen />
) : null;
})()}
</div>
)}
</div>