feat: background syncer for github_rpm remotes (#108)

## Why

Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.

## How

- Add a single per-process background syncer (started at boot, cleanly stopped
  on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
  global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the
  github provider so every GitHub call (releases list + each ranged asset GET)
  acquires a token first.
- Re-check each `github_rpm` remote for new/changed releases on its existing
  `mutable_ttl` cadence; derive only new/changed assets incrementally and prune
  assets that disappear upstream. Repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call returns
  immediately.
- Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives
  nothing and is not counted against GitHub's rate limit, so an unchanged repo
  is nearly free.
- Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`,
  `etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only
  for the replica that atomically claims the lease, bounding total GitHub load
  to ~once per `mutable_ttl` regardless of replica count; the ETag is shared
  through the same row.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
  cache, and return a bounded wait then a retryable `503` rather than blocking
  on a cold derive.
- Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` /
  `GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst
  5, 3 workers, 60s tick) and document the syncer in the README.

## Tests

- Unit (httptest, Range/ETag-aware fixture): `304` releases response derives
  nothing; incremental derive fetches only the newly added asset; the shared
  limiter caps request rate; work-queue enqueues coalesce to one job; prime
  enqueues a job; a held lease stops a second replica from scanning; cold-start
  serves `503` while warm cache serves `200`.
- DB integration (testcontainers postgres): the real lease SQL — one holder at a
  time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
  bypasses recency but respects a live lease.
- Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the
  background at ~1 req/s (global limiter); `dnf makecache` served fast from the
  priming cache (no cold block); clean shutdown mid-scan, no panics.

## Notes

- Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.

Reviewed-on: #108
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #108.
This commit is contained in:
2026-08-10 21:31:24 +10:00
committed by BenVincent
parent d154fbf3f3
commit e24c35f534
14 changed files with 1015 additions and 28 deletions
+1 -1
View File
@@ -57,7 +57,7 @@ func do(t *testing.T, h http.Handler, method, path, body string) int {
}
func TestRemotesErrorPaths(t *testing.T) {
h := NewRemotesHandler(closedDB(t)).Routes()
h := NewRemotesHandler(closedDB(t), nil).Routes()
if c := do(t, h, "GET", "/", ""); c != 500 {
t.Errorf("list with dead db = %d, want 500", c)
}
+16 -4
View File
@@ -11,12 +11,19 @@ import (
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type RemotesHandler struct {
db *database.DB
// Primer enqueues a background metadata prime for a newly created remote so the
// create call never blocks on a derive. *rpm.Syncer satisfies it.
type Primer interface {
EnqueuePrime(remote models.Remote)
}
func NewRemotesHandler(db *database.DB) *RemotesHandler {
return &RemotesHandler{db: db}
type RemotesHandler struct {
db *database.DB
primer Primer
}
func NewRemotesHandler(db *database.DB, primer Primer) *RemotesHandler {
return &RemotesHandler{db: db, primer: primer}
}
func (h *RemotesHandler) Routes() chi.Router {
@@ -77,6 +84,11 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Prime a github_rpm remote's metadata in the background so its first
// repodata request is served from cache instead of a cold on-demand derive.
if h.primer != nil && remote.PackageType == models.PackageGitHubRPM {
h.primer.EnqueuePrime(remote)
}
writeJSON(w, http.StatusCreated, remote)
}