Lazy per-replica scans re-derived RPM metadata on the client request path
and, run independently on every replica, multiplied GitHub queries by the
replica count. A single background syncer with a shared rate limit, ETag
conditional checks, and a DB lease keeps metadata fresh off the request path
while bounding GitHub load to ~once per mutable_ttl across the fleet.
- Add a single per-process syncer (started at boot, stopped on shutdown) that
owns a deduped/coalescing work queue, a worker pool, and one global
token-bucket rate limiter bound onto the github provider so every GitHub call
(releases list + each ranged asset GET) acquires a token first.
- Check each github_rpm remote for new/changed releases on its mutable_ttl
cadence; derive only new/changed assets incrementally and prune assets that
disappear upstream, so repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call never
blocks on a derive.
- Send the stored releases-list ETag as If-None-Match; a 304 derives nothing
(and does not count against GitHub's rate limit), making an unchanged repo
nearly free.
- Coordinate replicas through a github_rpm_sync_state row (last_synced_at,
etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only for
the replica that atomically claims the lease, bounding total GitHub load to
~once per mutable_ttl regardless of replica count.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
cache, and return a bounded wait then a retryable 503 rather than blocking on
a cold derive.
- Add GITHUB_SYNC_RATE/BURST/WORKERS/POLL_INTERVAL config (conservative
defaults) and document the syncer in the README.
unkinben