Add per-remote sync_interval and spread GitHub syncs #166

Merged

benvin merged 5 commits from benvin/github-sync-spread into master 2026-10-25 18:58:23 +11:00

Conversation 3 Commits 5 Files Changed 20

+503 -70

unkin-agent commented 2026-10-25 13:13:33 +11:00

Member

Copy Link

Copy Source

github_* remotes sharing a mutable_ttl all become due at the same moment, so many remotes created or synced together burst GitHub at once.

• add per-remote sync_interval (seconds) for github_rpm/github_deb/github_alpine remotes
• offset each remote's slots by a stable name-hashed phase so remotes on one interval spread out
• compute slots and the first-sync gate in the lease claim from the DB clock so replicas agree
• wait for the first slot on a scheduled first sync and skip the create-time prime; cold-cache primes stay immediate
• keep the mutable_ttl freshness predicate unchanged when sync_interval is unset
• document sync_interval in README, rpm docs and UI types

github_* remotes sharing a mutable_ttl all become due at the same moment, so many remotes created or synced together burst GitHub at once. - add per-remote `sync_interval` (seconds) for github_rpm/github_deb/github_alpine remotes - offset each remote's slots by a stable name-hashed phase so remotes on one interval spread out - compute slots and the first-sync gate in the lease claim from the DB clock so replicas agree - wait for the first slot on a scheduled first sync and skip the create-time prime; cold-cache primes stay immediate - keep the mutable_ttl freshness predicate unchanged when `sync_interval` is unset - document `sync_interval` in README, rpm docs and UI types

unkin-agent added 1 commit 2026-10-25 13:13:34 +11:00

Add per-remote sync_interval and spread GitHub syncs across it 707e380c98

unkin-agent commented 2026-10-25 13:17:16 +11:00

Author

Member

Copy Link

Copy Source

• internal/database/github_sync.go:66 — slot boundaries apply to every caller, so remotes with sync_interval unset (mutable_ttl only) also move from "ttl since last sync" to phased boundaries; this contradicts "0 keeps mutable_ttl" prior behaviour, and a sync landing just before a boundary (retry success, prime) re-syncs seconds later → gate the phased maths on sync_interval being set (e.g. pass phase 0 / keep old predicate when unset), or state and test the behaviour change.
• internal/database/github_sync.go:66 — first sync (last_synced_at IS NULL) and prime (freshness 0) stay unspread, so remotes created together still burst on first sync → state as out of scope or stagger the first sync by phase.
• internal/database/github_sync_test.go:330 — no test for interval change, unset-interval equivalence to the old predicate, or the alpine/deb tables; the same-slot assertion is skipped when within 10s of a boundary and depends on wall-clock → inject/seed times relative to a fixed boundary and cover the unset case.
• internal/database/github_sync.go:66 — scope: new field/API/validation and the claim-query behaviour change are independent concerns → split into (1) sync_interval column+API+syncer wiring, (2) phased lease.
• nit: docs/rpm/remote-github.md:35, README.md:205, ui/src/api/types.ts:8 — still describe cadence as mutable_ttl only and omit sync_interval → update.

- internal/database/github_sync.go:66 — slot boundaries apply to every caller, so remotes with `sync_interval` unset (mutable_ttl only) also move from "ttl since last sync" to phased boundaries; this contradicts "0 keeps mutable_ttl" prior behaviour, and a sync landing just before a boundary (retry success, prime) re-syncs seconds later → gate the phased maths on sync_interval being set (e.g. pass phase 0 / keep old predicate when unset), or state and test the behaviour change. - internal/database/github_sync.go:66 — first sync (`last_synced_at IS NULL`) and prime (freshness 0) stay unspread, so remotes created together still burst on first sync → state as out of scope or stagger the first sync by phase. - internal/database/github_sync_test.go:330 — no test for interval change, unset-interval equivalence to the old predicate, or the alpine/deb tables; the same-slot assertion is skipped when within 10s of a boundary and depends on wall-clock → inject/seed times relative to a fixed boundary and cover the unset case. - internal/database/github_sync.go:66 — scope: new field/API/validation and the claim-query behaviour change are independent concerns → split into (1) sync_interval column+API+syncer wiring, (2) phased lease. - nit: docs/rpm/remote-github.md:35, README.md:205, ui/src/api/types.ts:8 — still describe cadence as mutable_ttl only and omit `sync_interval` → update.

unkin-agent added 1 commit 2026-10-25 13:27:59 +11:00

Confine phased sync slots to sync_interval remotes ... 1582533e5a

Unset remotes keep the mutable_ttl freshness predicate. Scheduled remotes
compute their slot in Go, skip the create-time prime and wait for their
first slot; a cold-cache prime still runs at once. Document sync_interval.

unkin-agent commented 2026-10-25 13:53:21 +11:00

Author

Member

Copy Link

Copy Source

• internal/provider/{rpm,deb,alpine}/syncer.go process() (slot = job.remote.SyncSlot(time.Now())) and internal/database/github_sync.go claimSyncLease (last_synced_at < $5) — the slot boundary is computed on the replica's clock but compared to last_synced_at, which the release stamps with the DB now(). A replica whose clock is ahead by d claims at its own slot start while DB time is still before it, so the release writes last_synced_at < slot and the remote stays due: it re-claims and rescans every poll tick until the DB clock passes the slot. Replicas that disagree also pick different slots → pass interval and phase to the claim and compute the slot in SQL from now(); keep SyncSlot only for the unit-testable maths.
• internal/provider/{rpm,deb,alpine}/syncer.go process() (job.remote.CreatedAt.Before(slot)) — same mix: created_at is a DB timestamp, slot is replica-clock. Skew can release the first-slot wait early or hold it a slot too long → do the created_at gate in the same SQL claim against the DB-computed slot.
• nit: internal/database/github_sync_test.go seedSynced — remote name built from time.Now().Format(...), and seeded last_synced_at uses time.Now().Add(...); use a counter for names.

- internal/provider/{rpm,deb,alpine}/syncer.go process() (`slot = job.remote.SyncSlot(time.Now())`) and internal/database/github_sync.go claimSyncLease (`last_synced_at < $5`) — the slot boundary is computed on the replica's clock but compared to last_synced_at, which the release stamps with the DB `now()`. A replica whose clock is ahead by d claims at its own slot start while DB time is still before it, so the release writes last_synced_at < slot and the remote stays due: it re-claims and rescans every poll tick until the DB clock passes the slot. Replicas that disagree also pick different slots → pass interval and phase to the claim and compute the slot in SQL from `now()`; keep `SyncSlot` only for the unit-testable maths. - internal/provider/{rpm,deb,alpine}/syncer.go process() (`job.remote.CreatedAt.Before(slot)`) — same mix: created_at is a DB timestamp, slot is replica-clock. Skew can release the first-slot wait early or hold it a slot too long → do the created_at gate in the same SQL claim against the DB-computed slot. - nit: internal/database/github_sync_test.go seedSynced — remote name built from `time.Now().Format(...)`, and seeded last_synced_at uses `time.Now().Add(...)`; use a counter for names.

unkin-agent added 1 commit 2026-10-25 13:58:02 +11:00

Compute sync slots and the first-sync gate on the DB clock 82b1852bc4

unkin-agent commented 2026-10-25 14:02:10 +11:00

Author

Member

Copy Link

Copy Source

• pkg/models/remote.go:190 — ValidateSyncInterval has no upper bound; values above 2147483647 pass validation then fail the INTEGER column as a 500, and large values overflow time.Duration(r.SyncInterval)*time.Second → reject > MaxInt32 (or a sane cap) with a 400, add a case to TestValidateSyncInterval.
• docs/deb/remote-github.md, docs/alpine/remote-github.md — sync_interval undocumented (only docs/rpm/remote-github.md updated, though the field applies to all three) → add the same re-check/phase/first-slot paragraph to both; also add a sync_interval row to the remote fields table in docs/common/caching.md.
• internal/database/github_sync_test.go:350 — no test for an interval change on a remote with an existing state row (0→N and N→0), nor that the created_at gate applies only on the insert path (existing row with last_synced_at NULL claims immediately inside the slot) → add cases pinning both behaviours.
• nit: docs/rpm/remote-github.md:57 — line left unwrapped after the edit → rewrap.

- pkg/models/remote.go:190 — ValidateSyncInterval has no upper bound; values above 2147483647 pass validation then fail the INTEGER column as a 500, and large values overflow time.Duration(r.SyncInterval)*time.Second → reject > MaxInt32 (or a sane cap) with a 400, add a case to TestValidateSyncInterval. - docs/deb/remote-github.md, docs/alpine/remote-github.md — sync_interval undocumented (only docs/rpm/remote-github.md updated, though the field applies to all three) → add the same re-check/phase/first-slot paragraph to both; also add a `sync_interval` row to the remote fields table in docs/common/caching.md. - internal/database/github_sync_test.go:350 — no test for an interval change on a remote with an existing state row (0→N and N→0), nor that the created_at gate applies only on the insert path (existing row with last_synced_at NULL claims immediately inside the slot) → add cases pinning both behaviours. - nit: docs/rpm/remote-github.md:57 — line left unwrapped after the edit → rewrap.

unkin-agent added 1 commit 2026-10-25 15:21:59 +11:00

Merge master b7338fd261

unkin-agent added 1 commit 2026-10-25 15:47:54 +11:00

Merge master 8257349548

benvin merged commit ee55ef3fcd into master 2026-10-25 18:58:23 +11:00

benvin deleted branch benvin/github-sync-spread 2026-10-25 18:58:23 +11:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#166