artifactapi

unkin/artifactapi

Fork 0

Author	SHA1	Message	Date
unkinben	9b94f18d7a	feat: server-level GitHub machine credential for authenticated requests ci/woodpecker/pr/test Pipeline was successful Details ci/woodpecker/pr/build Pipeline was successful Details ci/woodpecker/pr/pre-commit Pipeline was successful Details Anonymous GitHub is capped at 60 requests/hour and cannot read private repositories, so a machine credential usable by a free (non-enterprise) account is needed to lift the request budget and reach private release assets. - Add internal/githubauth: a process-wide credential delivered via env/secret, applied by default to every outbound GitHub request. - Support two modes: a Personal Access Token sent as `Authorization: Bearer`, and a GitHub App that mints a short-lived RS256 JWT (stdlib crypto, no new dependency), exchanges it for a ~1h installation token, caches it, and single-flights a refresh a few minutes before expiry. - Inject the credential at the two GitHub call paths: the rpm github provider (releases scan + ranged asset-header GETs) and the generic byte proxy (private release-asset downloads for github.com hosts). - Honor precedence: a remote's own username/password overrides the server credential; no credential configured stays anonymous. - Fail closed at startup on partial App configuration; never persist the credential to the DB, return it from an API, or log it. - Read GITHUB_TOKEN / GITHUB_APP_ID / GITHUB_APP_INSTALLATION_ID / GITHUB_APP_PRIVATE_KEY[_PATH] via the existing getenv convention. - Document PAT vs App setup, the free-account fine-grained PAT scopes (Contents:read + Metadata:read), precedence, and the rate-limit implication.	2026-08-10 21:28:24 +10:00
unkinben	6dc72920da	feat: background syncer for github_rpm remotes ci/woodpecker/pr/build Pipeline was successful Details ci/woodpecker/pr/test Pipeline was successful Details ci/woodpecker/pr/pre-commit Pipeline was successful Details Lazy per-replica scans re-derived RPM metadata on the client request path and, run independently on every replica, multiplied GitHub queries by the replica count. A single background syncer with a shared rate limit, ETag conditional checks, and a DB lease keeps metadata fresh off the request path while bounding GitHub load to ~once per mutable_ttl across the fleet. - Add a single per-process syncer (started at boot, stopped on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one global token-bucket rate limiter bound onto the github provider so every GitHub call (releases list + each ranged asset GET) acquires a token first. - Check each github_rpm remote for new/changed releases on its mutable_ttl cadence; derive only new/changed assets incrementally and prune assets that disappear upstream, so repodata is served from primed DB rows. - Prime metadata in the background on remote creation; the create call never blocks on a derive. - Send the stored releases-list ETag as If-None-Match; a 304 derives nothing (and does not count against GitHub's rate limit), making an unchanged repo nearly free. - Coordinate replicas through a github_rpm_sync_state row (last_synced_at, etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only for the replica that atomically claims the lease, bounding total GitHub load to ~once per mutable_ttl regardless of replica count. - Keep the request path fast: serve current cache, enqueue a prime on an empty cache, and return a bounded wait then a retryable 503 rather than blocking on a cold derive. - Add GITHUB_SYNC_RATE/BURST/WORKERS/POLL_INTERVAL config (conservative defaults) and document the syncer in the README.	2026-08-10 21:08:24 +10:00

Author

SHA1

Message

Date

unkinben

9b94f18d7a

feat: server-level GitHub machine credential for authenticated requests

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

Anonymous GitHub is capped at 60 requests/hour and cannot read private
repositories, so a machine credential usable by a free (non-enterprise)
account is needed to lift the request budget and reach private release
assets.

- Add internal/githubauth: a process-wide credential delivered via env/secret,
  applied by default to every outbound GitHub request.
- Support two modes: a Personal Access Token sent as `Authorization: Bearer`,
  and a GitHub App that mints a short-lived RS256 JWT (stdlib crypto, no new
  dependency), exchanges it for a ~1h installation token, caches it, and
  single-flights a refresh a few minutes before expiry.
- Inject the credential at the two GitHub call paths: the rpm github provider
  (releases scan + ranged asset-header GETs) and the generic byte proxy
  (private release-asset downloads for github.com hosts).
- Honor precedence: a remote's own username/password overrides the server
  credential; no credential configured stays anonymous.
- Fail closed at startup on partial App configuration; never persist the
  credential to the DB, return it from an API, or log it.
- Read GITHUB_TOKEN / GITHUB_APP_ID / GITHUB_APP_INSTALLATION_ID /
  GITHUB_APP_PRIVATE_KEY[_PATH] via the existing getenv convention.
- Document PAT vs App setup, the free-account fine-grained PAT scopes
  (Contents:read + Metadata:read), precedence, and the rate-limit implication.

2026-08-10 21:28:24 +10:00

unkinben

6dc72920da

feat: background syncer for github_rpm remotes

ci/woodpecker/pr/build Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

Lazy per-replica scans re-derived RPM metadata on the client request path
and, run independently on every replica, multiplied GitHub queries by the
replica count. A single background syncer with a shared rate limit, ETag
conditional checks, and a DB lease keeps metadata fresh off the request path
while bounding GitHub load to ~once per mutable_ttl across the fleet.

- Add a single per-process syncer (started at boot, stopped on shutdown) that
  owns a deduped/coalescing work queue, a worker pool, and one global
  token-bucket rate limiter bound onto the github provider so every GitHub call
  (releases list + each ranged asset GET) acquires a token first.
- Check each github_rpm remote for new/changed releases on its mutable_ttl
  cadence; derive only new/changed assets incrementally and prune assets that
  disappear upstream, so repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call never
  blocks on a derive.
- Send the stored releases-list ETag as If-None-Match; a 304 derives nothing
  (and does not count against GitHub's rate limit), making an unchanged repo
  nearly free.
- Coordinate replicas through a github_rpm_sync_state row (last_synced_at,
  etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only for
  the replica that atomically claims the lease, bounding total GitHub load to
  ~once per mutable_ttl regardless of replica count.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
  cache, and return a bounded wait then a retryable 503 rather than blocking on
  a cold derive.
- Add GITHUB_SYNC_RATE/BURST/WORKERS/POLL_INTERVAL config (conservative
  defaults) and document the syncer in the README.

2026-08-10 21:08:24 +10:00

Compare commits

2 Commits

v3.8.0 .. 9b94f18d7a

Diff Content Not Available

Compare commits

2 Commits v3.8.0 .. 9b94f18d7a

Diff Content Not Available

2 Commits

v3.8.0 .. 9b94f18d7a