Lazy per-replica scans re-derived RPM metadata on the client request path
and, run independently on every replica, multiplied GitHub queries by the
replica count. A single background syncer with a shared rate limit, ETag
conditional checks, and a DB lease keeps metadata fresh off the request path
while bounding GitHub load to ~once per mutable_ttl across the fleet.
- Add a single per-process syncer (started at boot, stopped on shutdown) that
owns a deduped/coalescing work queue, a worker pool, and one global
token-bucket rate limiter bound onto the github provider so every GitHub call
(releases list + each ranged asset GET) acquires a token first.
- Check each github_rpm remote for new/changed releases on its mutable_ttl
cadence; derive only new/changed assets incrementally and prune assets that
disappear upstream, so repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call never
blocks on a derive.
- Send the stored releases-list ETag as If-None-Match; a 304 derives nothing
(and does not count against GitHub's rate limit), making an unchanged repo
nearly free.
- Coordinate replicas through a github_rpm_sync_state row (last_synced_at,
etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only for
the replica that atomically claims the lease, bounding total GitHub load to
~once per mutable_ttl regardless of replica count.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
cache, and return a bounded wait then a retryable 503 rather than blocking on
a cold derive.
- Add GITHUB_SYNC_RATE/BURST/WORKERS/POLL_INTERVAL config (conservative
defaults) and document the syncer in the README.
Raises statement coverage of the core packages (all of `internal/` except the interactive `tui/`, plus `pkg/`) from **8.7% to 90.1%**.
## Approach
- **Pure-go unit tests** for all providers, virtual mergers, classifier, config, auth, models, and the API client (httptest).
- **Testcontainers-backed** tests (new `internal/testsupport` helper: Postgres/Redis/MinIO, Ryuk disabled) for database, storage, cache, the proxy engine, the GC, and a full-stack `server` test that drives the whole HTTP API. These `t.Skip` when Docker is absent so `go test` still runs locally without it.
## Measuring
```
go test -coverpkg=./internal/...,./pkg/... -coverprofile=cover.out ./internal/... ./pkg/...
grep -v /internal/tui/ cover.out | go tool cover -func=/dev/stdin | tail -1 # 90.1%
```
Run with `-p 1` (containers are heavy).
## Notes
- The interactive `tui/` package and `cmd/main` are excluded from the target per the agreed scope.
- Some defensive error branches are covered via fault injection (closed DB pool, killing MinIO mid-upload).
Reviewed-on: #98
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkinben