Add per-remote sync_interval and spread GitHub syncs (#166)
ci/woodpecker/tag/docker Pipeline was successful

github_* remotes sharing a mutable_ttl all become due at the same moment, so many remotes created or synced together burst GitHub at once.

- add per-remote `sync_interval` (seconds) for github_rpm/github_deb/github_alpine remotes
- offset each remote's slots by a stable name-hashed phase so remotes on one interval spread out
- compute slots and the first-sync gate in the lease claim from the DB clock so replicas agree
- wait for the first slot on a scheduled first sync and skip the create-time prime; cold-cache primes stay immediate
- keep the mutable_ttl freshness predicate unchanged when `sync_interval` is unset
- document `sync_interval` in README, rpm docs and UI types

Reviewed-on: #166
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
This commit was merged in pull request #166.
This commit is contained in:
2026-10-25 18:58:22 +11:00
committed by BenVincent
parent e5313342ba
commit ee55ef3fcd
20 changed files with 503 additions and 70 deletions
+7 -3
View File
@@ -32,8 +32,12 @@ derived metadata current off the client request path:
The first `dnf` request is served from cache. If a request arrives before the
prime lands, it returns a retryable `503` (with `Retry-After`) rather than
serving an empty repo or blocking on a multi-minute derive.
- **Periodic re-check, driven by `mutable_ttl`.** Each remote is re-checked for
new or changed releases no more often than its `mutable_ttl`. New/changed
- **Periodic re-check, driven by `sync_interval` or `mutable_ttl`.** With
`sync_interval` (seconds) set, a remote re-checks once per interval at a
stable per-remote offset, so remotes on the same interval spread out instead
of syncing together; its first scheduled sync waits for that slot (a
cold-cache request still primes at once). Unset, it re-checks no more often
than its `mutable_ttl`. New/changed
assets are derived incrementally; assets already cached are never re-fetched,
and assets that disappear upstream are pruned.
- **ETag / 304 conditional requests.** The releases-list `ETag` is stored per
@@ -50,7 +54,7 @@ derived metadata current off the client request path:
periodic scan a replica must atomically claim a per-remote lease
(`github_rpm_sync_state`: `last_synced_at`, `etag`, `sync_lease_owner`,
`sync_lease_expires`); only the winner scans. This bounds total GitHub load to
~once per `mutable_ttl` regardless of replica count, and the shared `etag`
~once per `sync_interval` (else `mutable_ttl`) regardless of replica count, and the shared `etag`
lets any replica issue the conditional request.
```hcl