Add per-remote sync_interval and spread GitHub syncs (#166)
ci/woodpecker/tag/docker Pipeline was successful
ci/woodpecker/tag/docker Pipeline was successful
github_* remotes sharing a mutable_ttl all become due at the same moment, so many remotes created or synced together burst GitHub at once. - add per-remote `sync_interval` (seconds) for github_rpm/github_deb/github_alpine remotes - offset each remote's slots by a stable name-hashed phase so remotes on one interval spread out - compute slots and the first-sync gate in the lease claim from the DB clock so replicas agree - wait for the first slot on a scheduled first sync and skip the create-time prime; cold-cache primes stay immediate - keep the mutable_ttl freshness predicate unchanged when `sync_interval` is unset - document `sync_interval` in README, rpm docs and UI types Reviewed-on: #166 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net>
This commit was merged in pull request #166.
This commit is contained in:
@@ -32,8 +32,12 @@ derived metadata current off the client request path:
|
||||
The first `dnf` request is served from cache. If a request arrives before the
|
||||
prime lands, it returns a retryable `503` (with `Retry-After`) rather than
|
||||
serving an empty repo or blocking on a multi-minute derive.
|
||||
- **Periodic re-check, driven by `mutable_ttl`.** Each remote is re-checked for
|
||||
new or changed releases no more often than its `mutable_ttl`. New/changed
|
||||
- **Periodic re-check, driven by `sync_interval` or `mutable_ttl`.** With
|
||||
`sync_interval` (seconds) set, a remote re-checks once per interval at a
|
||||
stable per-remote offset, so remotes on the same interval spread out instead
|
||||
of syncing together; its first scheduled sync waits for that slot (a
|
||||
cold-cache request still primes at once). Unset, it re-checks no more often
|
||||
than its `mutable_ttl`. New/changed
|
||||
assets are derived incrementally; assets already cached are never re-fetched,
|
||||
and assets that disappear upstream are pruned.
|
||||
- **ETag / 304 conditional requests.** The releases-list `ETag` is stored per
|
||||
@@ -50,7 +54,7 @@ derived metadata current off the client request path:
|
||||
periodic scan a replica must atomically claim a per-remote lease
|
||||
(`github_rpm_sync_state`: `last_synced_at`, `etag`, `sync_lease_owner`,
|
||||
`sync_lease_expires`); only the winner scans. This bounds total GitHub load to
|
||||
~once per `mutable_ttl` regardless of replica count, and the shared `etag`
|
||||
~once per `sync_interval` (else `mutable_ttl`) regardless of replica count, and the shared `etag`
|
||||
lets any replica issue the conditional request.
|
||||
|
||||
```hcl
|
||||
|
||||
Reference in New Issue
Block a user