feat: background syncer for github_rpm remotes (#108)

## Why

Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the
client request path, single-flighted per replica. Two problems remain: the
derive still happens per replica, so across a multi-replica deployment the same
releases are scanned and re-derived N times, multiplying GitHub queries; and a
cold cache blocks the first request on a full derive. GitHub's rate limits are
low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single
coordinated syncer with a shared rate limit and conditional requests.

## How

- Add a single per-process background syncer (started at boot, cleanly stopped
  on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one
  global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the
  github provider so every GitHub call (releases list + each ranged asset GET)
  acquires a token first.
- Re-check each `github_rpm` remote for new/changed releases on its existing
  `mutable_ttl` cadence; derive only new/changed assets incrementally and prune
  assets that disappear upstream. Repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call returns
  immediately.
- Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives
  nothing and is not counted against GitHub's rate limit, so an unchanged repo
  is nearly free.
- Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`,
  `etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only
  for the replica that atomically claims the lease, bounding total GitHub load
  to ~once per `mutable_ttl` regardless of replica count; the ETag is shared
  through the same row.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
  cache, and return a bounded wait then a retryable `503` rather than blocking
  on a cold derive.
- Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` /
  `GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst
  5, 3 workers, 60s tick) and document the syncer in the README.

## Tests

- Unit (httptest, Range/ETag-aware fixture): `304` releases response derives
  nothing; incremental derive fetches only the newly added asset; the shared
  limiter caps request rate; work-queue enqueues coalesce to one job; prime
  enqueues a job; a held lease stops a second replica from scanning; cold-start
  serves `503` while warm cache serves `200`.
- DB integration (testcontainers postgres): the real lease SQL — one holder at a
  time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0)
  bypasses recency but respects a live lease.
- Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the
  background at ~1 req/s (global limiter); `dnf makecache` served fast from the
  priming cache (no cold block); clean shutdown mid-scan, no panics.

## Notes

- Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief.

Reviewed-on: #108
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #108.
This commit is contained in:
2026-08-10 21:31:24 +10:00
committed by BenVincent
parent d154fbf3f3
commit e24c35f534
14 changed files with 1015 additions and 28 deletions
+34
View File
@@ -32,6 +32,18 @@ type Config struct {
TFSigningKeyPath string
TFSigningKeyPassphrase string
TFProviderProtocols string
// github_rpm background syncer. The syncer keeps derived RPM metadata for
// every github_rpm remote fresh off the client request path, sharing a
// single global token-bucket limiter across all remotes so GitHub is never
// hammered. Defaults are conservative: 1 req/s (3600/hr) sits well under an
// authenticated token's 5000/hr. Unauthenticated remotes (60/hr) lean on
// ETag/304 — an unchanged repo costs nothing — so keep those repos small or
// configure a token.
GitHubSyncRatePerSec float64
GitHubSyncBurst int
GitHubSyncWorkers int
GitHubSyncPollInterval int
}
func (c *Config) DatabaseDSN() string {
@@ -49,6 +61,23 @@ func Load() (*Config, error) {
s3Secure, _ := strconv.ParseBool(getenv("MINIO_SECURE", "false"))
syncRate, err := strconv.ParseFloat(getenv("GITHUB_SYNC_RATE", "1"), 64)
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_RATE: %w", err)
}
syncBurst, err := strconv.Atoi(getenv("GITHUB_SYNC_BURST", "5"))
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_BURST: %w", err)
}
syncWorkers, err := strconv.Atoi(getenv("GITHUB_SYNC_WORKERS", "3"))
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_WORKERS: %w", err)
}
syncPoll, err := strconv.Atoi(getenv("GITHUB_SYNC_POLL_INTERVAL", "60"))
if err != nil {
return nil, fmt.Errorf("invalid GITHUB_SYNC_POLL_INTERVAL: %w", err)
}
cfg := &Config{
ListenAddr: getenv("LISTEN_ADDR", ":8000"),
@@ -71,6 +100,11 @@ func Load() (*Config, error) {
TFSigningKeyPath: getenv("TF_SIGNING_KEY_PATH", ""),
TFSigningKeyPassphrase: getenv("TF_SIGNING_KEY_PASSPHRASE", ""),
TFProviderProtocols: getenv("TF_PROVIDER_PROTOCOLS", "5.0,6.0"),
GitHubSyncRatePerSec: syncRate,
GitHubSyncBurst: syncBurst,
GitHubSyncWorkers: syncWorkers,
GitHubSyncPollInterval: syncPoll,
}
return cfg, nil