unkin/artifactapi
3
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

feat: background syncer for github_rpm remotes #108

Merged
benvin merged 1 commits from benvin/github-rpm-syncer into master 2026-08-10 21:31:25 +10:00
Conversation 0 Commits 1 Files Changed 14
+1015 -28
unkinben commented 2026-08-10 21:12:17 +10:00
Owner
Copy Link
Copy Source

Why

Builds on #107 (merged), which derives github_rpm RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.

How

  • Add a single per-process background syncer (started at boot, cleanly stopped
    on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
    global token-bucket rate limiter (golang.org/x/time/rate) bound onto the
    github provider so every GitHub call (releases list + each ranged asset GET)
    acquires a token first.
  • Re-check each github_rpm remote for new/changed releases on its existing
    mutable_ttl cadence; derive only new/changed assets incrementally and prune
    assets that disappear upstream. Repodata is served from primed DB rows.
  • Prime metadata in the background on remote creation; the create call returns
    immediately.
  • Send the stored releases-list ETag as If-None-Match; a 304 derives
    nothing and is not counted against GitHub's rate limit, so an unchanged repo
    is nearly free.
  • Coordinate replicas through a github_rpm_sync_state row (last_synced_at,
    etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only
    for the replica that atomically claims the lease, bounding total GitHub load
    to ~once per mutable_ttl regardless of replica count; the ETag is shared
    through the same row.
  • Keep the request path fast: serve current cache, enqueue a prime on an empty
    cache, and return a bounded wait then a retryable 503 rather than blocking
    on a cold derive.
  • Add GITHUB_SYNC_RATE / GITHUB_SYNC_BURST / GITHUB_SYNC_WORKERS /
    GITHUB_SYNC_POLL_INTERVAL config with conservative defaults (1 req/s, burst
    5, 3 workers, 60s tick) and document the syncer in the README.

Tests

  • Unit (httptest, Range/ETag-aware fixture): 304 releases response derives
    nothing; incremental derive fetches only the newly added asset; the shared
    limiter caps request rate; work-queue enqueues coalesce to one job; prime
    enqueues a job; a held lease stops a second replica from scanning; cold-start
    serves 503 while warm cache serves 200.
  • DB integration (testcontainers postgres): the real lease SQL — one holder at a
    time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
    bypasses recency but respects a live lease.
  • Docker e2e re-run: dnf install dotvault works; prime-on-create derives in the
    background at ~1 req/s (global limiter); dnf makecache served fast from the
    priming cache (no cold block); clean shutdown mid-scan, no panics.

Notes

  • Reuses mutable_ttl as the check interval (no new per-remote field), per brief.
## Why Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the client request path, single-flighted per replica. Two problems remain: the derive still happens per replica, so across a multi-replica deployment the same releases are scanned and re-derived N times, multiplying GitHub queries; and a cold cache blocks the first request on a full derive. GitHub's rate limits are low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single coordinated syncer with a shared rate limit and conditional requests. ## How - Add a single per-process background syncer (started at boot, cleanly stopped on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the github provider so every GitHub call (releases list + each ranged asset GET) acquires a token first. - Re-check each `github_rpm` remote for new/changed releases on its existing `mutable_ttl` cadence; derive only new/changed assets incrementally and prune assets that disappear upstream. Repodata is served from primed DB rows. - Prime metadata in the background on remote creation; the create call returns immediately. - Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives nothing and is not counted against GitHub's rate limit, so an unchanged repo is nearly free. - Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`, `etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only for the replica that atomically claims the lease, bounding total GitHub load to ~once per `mutable_ttl` regardless of replica count; the ETag is shared through the same row. - Keep the request path fast: serve current cache, enqueue a prime on an empty cache, and return a bounded wait then a retryable `503` rather than blocking on a cold derive. - Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` / `GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst 5, 3 workers, 60s tick) and document the syncer in the README. ## Tests - Unit (httptest, Range/ETag-aware fixture): `304` releases response derives nothing; incremental derive fetches only the newly added asset; the shared limiter caps request rate; work-queue enqueues coalesce to one job; prime enqueues a job; a held lease stops a second replica from scanning; cold-start serves `503` while warm cache serves `200`. - DB integration (testcontainers postgres): the real lease SQL — one holder at a time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0) bypasses recency but respects a live lease. - Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the background at ~1 req/s (global limiter); `dnf makecache` served fast from the priming cache (no cold block); clean shutdown mid-scan, no panics. ## Notes - Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.
unkinben added 1 commit 2026-08-10 21:12:17 +10:00
feat: background syncer for github_rpm remotes
ci/woodpecker/pr/build Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
6dc72920da 
Lazy per-replica scans re-derived RPM metadata on the client request path
and, run independently on every replica, multiplied GitHub queries by the
replica count. A single background syncer with a shared rate limit, ETag
conditional checks, and a DB lease keeps metadata fresh off the request path
while bounding GitHub load to ~once per mutable_ttl across the fleet.

- Add a single per-process syncer (started at boot, stopped on shutdown) that
  owns a deduped/coalescing work queue, a worker pool, and one global
  token-bucket rate limiter bound onto the github provider so every GitHub call
  (releases list + each ranged asset GET) acquires a token first.
- Check each github_rpm remote for new/changed releases on its mutable_ttl
  cadence; derive only new/changed assets incrementally and prune assets that
  disappear upstream, so repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call never
  blocks on a derive.
- Send the stored releases-list ETag as If-None-Match; a 304 derives nothing
  (and does not count against GitHub's rate limit), making an unchanged repo
  nearly free.
- Coordinate replicas through a github_rpm_sync_state row (last_synced_at,
  etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only for
  the replica that atomically claims the lease, bounding total GitHub load to
  ~once per mutable_ttl regardless of replica count.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
  cache, and return a bounded wait then a retryable 503 rather than blocking on
  a cold derive.
- Add GITHUB_SYNC_RATE/BURST/WORKERS/POLL_INTERVAL config (conservative
  defaults) and document the syncer in the README.
benvin merged commit e24c35f534 into master 2026-08-10 21:31:25 +10:00
benvin deleted branch benvin/github-rpm-syncer 2026-08-10 21:31:25 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#108
Delete Branch "benvin/github-rpm-syncer"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?