Switching a remote's backend (base_url) left the previously-cached mutable
metadata (repodata / Release / APKINDEX) served until TTL expiry, pointing
at the old upstream. cache.FlushRemote existed but was wired to nothing.
- Inject a MetadataFlusher (satisfied by *cache.Redis) into RemotesHandler.
- On update, read the existing remote first, then after a successful DB
update flush the remote's cached metadata when base_url changed so the
next request re-fetches fresh from the new upstream.
- A flush failure is logged as a warning and does not fail the request; the
DB update already landed.
- Add tests: base_url change flushes exactly once, an unchanged base_url
does not flush, and a flush error still returns 200.
## Why
Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.
## How
- Add a single per-process background syncer (started at boot, cleanly stopped
on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the
github provider so every GitHub call (releases list + each ranged asset GET)
acquires a token first.
- Re-check each `github_rpm` remote for new/changed releases on its existing
`mutable_ttl` cadence; derive only new/changed assets incrementally and prune
assets that disappear upstream. Repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call returns
immediately.
- Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives
nothing and is not counted against GitHub's rate limit, so an unchanged repo
is nearly free.
- Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`,
`etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only
for the replica that atomically claims the lease, bounding total GitHub load
to ~once per `mutable_ttl` regardless of replica count; the ETag is shared
through the same row.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
cache, and return a bounded wait then a retryable `503` rather than blocking
on a cold derive.
- Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` /
`GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst
5, 3 workers, 60s tick) and document the syncer in the README.
## Tests
- Unit (httptest, Range/ETag-aware fixture): `304` releases response derives
nothing; incremental derive fetches only the newly added asset; the shared
limiter caps request rate; work-queue enqueues coalesce to one job; prime
enqueues a job; a held lease stops a second replica from scanning; cold-start
serves `503` while warm cache serves `200`.
- DB integration (testcontainers postgres): the real lease SQL — one holder at a
time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
bypasses recency but respects a live lease.
- Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the
background at ~1 req/s (global limiter); `dnf makecache` served fast from the
priming cache (no cold block); clean shutdown mid-scan, no panics.
## Notes
- Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.
Reviewed-on: #108
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Raises statement coverage of the core packages (all of `internal/` except the interactive `tui/`, plus `pkg/`) from **8.7% to 90.1%**.
## Approach
- **Pure-go unit tests** for all providers, virtual mergers, classifier, config, auth, models, and the API client (httptest).
- **Testcontainers-backed** tests (new `internal/testsupport` helper: Postgres/Redis/MinIO, Ryuk disabled) for database, storage, cache, the proxy engine, the GC, and a full-stack `server` test that drives the whole HTTP API. These `t.Skip` when Docker is absent so `go test` still runs locally without it.
## Measuring
```
go test -coverpkg=./internal/...,./pkg/... -coverprofile=cover.out ./internal/... ./pkg/...
grep -v /internal/tui/ cover.out | go tool cover -func=/dev/stdin | tail -1 # 90.1%
```
Run with `-p 1` (containers are heavy).
## Notes
- The interactive `tui/` package and `cmd/main` are excluded from the target per the agreed scope.
- Some defensive error branches are covered via fault injection (closed DB pool, killing MinIO mid-upload).
Reviewed-on: #98
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben