artifactapi

unkin/artifactapi

Fork 0

Author	SHA1	Message	Date
unkinben	109ba2ce27	feat: server-level GitHub machine credential for authenticated requests (#109 ) ci/woodpecker/tag/docker Pipeline was successful Details ## Why Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories. A machine credential usable by a free (non-enterprise) account is needed to lift the request budget to ~5000/hr and to read private-repo release assets. Builds on the background syncer (#108, now merged to `master`); this diff is the auth changes only. ## How - Add `internal/githubauth`: a process-wide GitHub credential delivered via env/secret, applied by default to every outbound GitHub request (releases scan, ranged asset-header GETs, and the generic-github byte proxy for private assets). - Support two modes: - PAT — `GITHUB_TOKEN` sent as `Authorization: Bearer <token>`. - GitHub App — `GITHUB_APP_ID` + `GITHUB_APP_INSTALLATION_ID` + private key (`GITHUB_APP_PRIVATE_KEY` inline PEM or `GITHUB_APP_PRIVATE_KEY_PATH`). Mint a short-lived RS256 JWT with stdlib `crypto/rsa` (no new dependency), exchange it at `POST /app/installations/{id}/access_tokens` for a ~1h installation token, cache it, and single-flight a refresh a few minutes before expiry. - Inject at the two GitHub call paths: the rpm github provider header builder (releases + ranged fetches) and the generic provider `AuthHeaders` (byte proxy, github.com hosts only; the pre-signed `objects.githubusercontent.com` redirect deliberately gets no Authorization). - Honor precedence: a remote's own `username`/`password` overrides the server credential; no credential configured stays anonymous (current behavior). - Fail closed at startup on partial App configuration (e.g. App id without a private key); a token-and-App conflict is also rejected. - Never persist the credential to the DB, return it from an API, or log it (token-exchange failures never echo the response body). - Read config via the existing `getenv` convention; document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication. ## Rate limit Authenticated requests share the syncer's single global limiter — no second limiter is added. A token raises the effective GitHub ceiling (~5000/hr vs ~60/hr), so the limiter defaults stay safe. ## Tests `internal/githubauth` and `internal/provider/{rpm,generic}`: - PAT attaches the correct `Authorization` header to releases + asset-header requests. - App mints a valid RS256 JWT (verified against the app public key), exchanges it at a mocked endpoint, reuses the cached token without re-exchanging, refreshes near expiry, and single-flights concurrent callers. - Per-remote credential overrides the server credential (rpm + generic). - No credential → no `Authorization` header, requests still succeed anonymously. - ETag/304 flow still works with auth attached. - The credential does not appear in a remote's serialized JSON. - Config validation: no-config is anonymous; partial App config and token/App conflict both error. Verified fail-before/pass-after for the injection tests. `gofmt -l`, `go build ./...`, `go vet ./...`, `go test ./...` all clean (26 packages). Reviewed-on: #109 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>	2026-08-10 21:42:39 +10:00
unkinben	e24c35f534	feat: background syncer for github_rpm remotes (#108 ) ## Why Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the client request path, single-flighted per replica. Two problems remain: the derive still happens per replica, so across a multi-replica deployment the same releases are scanned and re-derived N times, multiplying GitHub queries; and a cold cache blocks the first request on a full derive. GitHub's rate limits are low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single coordinated syncer with a shared rate limit and conditional requests. ## How - Add a single per-process background syncer (started at boot, cleanly stopped on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the github provider so every GitHub call (releases list + each ranged asset GET) acquires a token first. - Re-check each `github_rpm` remote for new/changed releases on its existing `mutable_ttl` cadence; derive only new/changed assets incrementally and prune assets that disappear upstream. Repodata is served from primed DB rows. - Prime metadata in the background on remote creation; the create call returns immediately. - Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives nothing and is not counted against GitHub's rate limit, so an unchanged repo is nearly free. - Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`, `etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only for the replica that atomically claims the lease, bounding total GitHub load to ~once per `mutable_ttl` regardless of replica count; the ETag is shared through the same row. - Keep the request path fast: serve current cache, enqueue a prime on an empty cache, and return a bounded wait then a retryable `503` rather than blocking on a cold derive. - Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` / `GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst 5, 3 workers, 60s tick) and document the syncer in the README. ## Tests - Unit (httptest, Range/ETag-aware fixture): `304` releases response derives nothing; incremental derive fetches only the newly added asset; the shared limiter caps request rate; work-queue enqueues coalesce to one job; prime enqueues a job; a held lease stops a second replica from scanning; cold-start serves `503` while warm cache serves `200`. - DB integration (testcontainers postgres): the real lease SQL — one holder at a time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0) bypasses recency but respects a live lease. - Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the background at ~1 req/s (global limiter); `dnf makecache` served fast from the priming cache (no cold block); clean shutdown mid-scan, no panics. ## Notes - Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief. Reviewed-on: #108 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>	2026-08-10 21:31:24 +10:00

Author

SHA1

Message

Date

unkinben

109ba2ce27

feat: server-level GitHub machine credential for authenticated requests (#109 )

ci/woodpecker/tag/docker Pipeline was successful

Details

## Why

Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories. A machine credential usable by a free (non-enterprise) account is needed to lift the request budget to ~5000/hr and to read private-repo release assets.

Builds on the background syncer (#108, now merged to `master`); this diff is the auth changes only.

## How

- Add `internal/githubauth`: a process-wide GitHub credential delivered via env/secret, applied by default to every outbound GitHub request (releases scan, ranged asset-header GETs, and the generic-github byte proxy for private assets).
- Support two modes:
  - **PAT** — `GITHUB_TOKEN` sent as `Authorization: Bearer <token>`.
  - **GitHub App** — `GITHUB_APP_ID` + `GITHUB_APP_INSTALLATION_ID` + private key (`GITHUB_APP_PRIVATE_KEY` inline PEM or `GITHUB_APP_PRIVATE_KEY_PATH`). Mint a short-lived RS256 JWT with stdlib `crypto/rsa` (no new dependency), exchange it at `POST /app/installations/{id}/access_tokens` for a ~1h installation token, cache it, and single-flight a refresh a few minutes before expiry.
- Inject at the two GitHub call paths: the rpm github provider header builder (releases + ranged fetches) and the generic provider `AuthHeaders` (byte proxy, github.com hosts only; the pre-signed `objects.githubusercontent.com` redirect deliberately gets no Authorization).
- Honor precedence: a remote's own `username`/`password` overrides the server credential; no credential configured stays anonymous (current behavior).
- Fail closed at startup on partial App configuration (e.g. App id without a private key); a token-and-App conflict is also rejected.
- Never persist the credential to the DB, return it from an API, or log it (token-exchange failures never echo the response body).
- Read config via the existing `getenv` convention; document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication.

## Rate limit

Authenticated requests share the syncer's single global limiter — no second limiter is added. A token raises the effective GitHub ceiling (~5000/hr vs ~60/hr), so the limiter defaults stay safe.

## Tests

`internal/githubauth` and `internal/provider/{rpm,generic}`:
- PAT attaches the correct `Authorization` header to releases + asset-header requests.
- App mints a valid RS256 JWT (verified against the app public key), exchanges it at a mocked endpoint, reuses the cached token without re-exchanging, refreshes near expiry, and single-flights concurrent callers.
- Per-remote credential overrides the server credential (rpm + generic).
- No credential → no `Authorization` header, requests still succeed anonymously.
- ETag/304 flow still works with auth attached.
- The credential does not appear in a remote's serialized JSON.
- Config validation: no-config is anonymous; partial App config and token/App conflict both error.

Verified fail-before/pass-after for the injection tests. `gofmt -l`, `go build ./...`, `go vet ./...`, `go test ./...` all clean (26 packages).

Reviewed-on: #109
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>

2026-08-10 21:42:39 +10:00

unkinben

e24c35f534

feat: background syncer for github_rpm remotes (#108 )

## Why

Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.

## How

- Add a single per-process background syncer (started at boot, cleanly stopped
  on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
  global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the
  github provider so every GitHub call (releases list + each ranged asset GET)
  acquires a token first.
- Re-check each `github_rpm` remote for new/changed releases on its existing
  `mutable_ttl` cadence; derive only new/changed assets incrementally and prune
  assets that disappear upstream. Repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call returns
  immediately.
- Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives
  nothing and is not counted against GitHub's rate limit, so an unchanged repo
  is nearly free.
- Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`,
  `etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only
  for the replica that atomically claims the lease, bounding total GitHub load
  to ~once per `mutable_ttl` regardless of replica count; the ETag is shared
  through the same row.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
  cache, and return a bounded wait then a retryable `503` rather than blocking
  on a cold derive.
- Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` /
  `GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst
  5, 3 workers, 60s tick) and document the syncer in the README.

## Tests

- Unit (httptest, Range/ETag-aware fixture): `304` releases response derives
  nothing; incremental derive fetches only the newly added asset; the shared
  limiter caps request rate; work-queue enqueues coalesce to one job; prime
  enqueues a job; a held lease stops a second replica from scanning; cold-start
  serves `503` while warm cache serves `200`.
- DB integration (testcontainers postgres): the real lease SQL — one holder at a
  time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
  bypasses recency but respects a live lease.
- Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the
  background at ~1 req/s (global limiter); `dnf makecache` served fast from the
  priming cache (no cold block); clean shutdown mid-scan, no panics.

## Notes

- Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.

Reviewed-on: #108
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>

2026-08-10 21:31:24 +10:00

Compare commits

2 Commits

9b94f18d7a .. v3.8.0

Diff Content Not Available

Compare commits

2 Commits 9b94f18d7a .. v3.8.0

Diff Content Not Available

2 Commits

9b94f18d7a .. v3.8.0