feat: background syncer for github_rpm remotes
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful

Lazy per-replica scans re-derived RPM metadata on the client request path
and, run independently on every replica, multiplied GitHub queries by the
replica count. A single background syncer with a shared rate limit, ETag
conditional checks, and a DB lease keeps metadata fresh off the request path
while bounding GitHub load to ~once per mutable_ttl across the fleet.

- Add a single per-process syncer (started at boot, stopped on shutdown) that
  owns a deduped/coalescing work queue, a worker pool, and one global
  token-bucket rate limiter bound onto the github provider so every GitHub call
  (releases list + each ranged asset GET) acquires a token first.
- Check each github_rpm remote for new/changed releases on its mutable_ttl
  cadence; derive only new/changed assets incrementally and prune assets that
  disappear upstream, so repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call never
  blocks on a derive.
- Send the stored releases-list ETag as If-None-Match; a 304 derives nothing
  (and does not count against GitHub's rate limit), making an unchanged repo
  nearly free.
- Coordinate replicas through a github_rpm_sync_state row (last_synced_at,
  etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only for
  the replica that atomically claims the lease, bounding total GitHub load to
  ~once per mutable_ttl regardless of replica count.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
  cache, and return a bounded wait then a retryable 503 rather than blocking on
  a cold derive.
- Add GITHUB_SYNC_RATE/BURST/WORKERS/POLL_INTERVAL config (conservative
  defaults) and document the syncer in the README.
This commit is contained in:
2026-08-10 21:08:15 +10:00
parent d154fbf3f3
commit 6dc72920da
14 changed files with 1015 additions and 28 deletions
+21 -5
View File
@@ -66,11 +66,14 @@ func (f *fakeStore) ListRPMMetadataEntries(ctx context.Context, _ string) ([]pro
// Range support) for a set of packages. digest controls whether the asset
// carries a sha256 digest (no-download path) or not (compute path).
type githubFixture struct {
srv *httptest.Server
rpmBytes map[string][]byte // asset filename -> bytes
rangeHit map[string]int // asset filename -> number of ranged GETs
fullHit map[string]int // asset filename -> number of full GETs
mu sync.Mutex
srv *httptest.Server
rpmBytes map[string][]byte // asset filename -> bytes
rangeHit map[string]int // asset filename -> number of ranged GETs
fullHit map[string]int // asset filename -> number of full GETs
etag string // when set, served as ETag; matching If-None-Match yields 304
releasesHit int // total releases-list requests (200 + 304)
notModHit int // releases-list requests answered 304
mu sync.Mutex
}
func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
@@ -89,6 +92,19 @@ func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
w.Write([]byte("[]"))
return
}
f.mu.Lock()
f.releasesHit++
etag := f.etag
if etag != "" && r.Header.Get("If-None-Match") == etag {
f.notModHit++
f.mu.Unlock()
w.WriteHeader(http.StatusNotModified)
return
}
f.mu.Unlock()
if etag != "" {
w.Header().Set("ETag", etag)
}
var assets []map[string]any
for name := range f.rpmBytes {
a := map[string]any{