unkinben
e24c35f534
## Why Builds on #107 (merged), which derives `github_rpm` RPM metadata lazily on the client request path, single-flighted per replica. Two problems remain: the derive still happens per replica, so across a multi-replica deployment the same releases are scanned and re-derived N times, multiplying GitHub queries; and a cold cache blocks the first request on a full derive. GitHub's rate limits are low (~60/hr unauthenticated, ~5000/hr authenticated), so this needs a single coordinated syncer with a shared rate limit and conditional requests. ## How - Add a single per-process background syncer (started at boot, cleanly stopped on shutdown) that owns a deduped/coalescing work queue, a worker pool, and one global token-bucket rate limiter (`golang.org/x/time/rate`) bound onto the github provider so every GitHub call (releases list + each ranged asset GET) acquires a token first. - Re-check each `github_rpm` remote for new/changed releases on its existing `mutable_ttl` cadence; derive only new/changed assets incrementally and prune assets that disappear upstream. Repodata is served from primed DB rows. - Prime metadata in the background on remote creation; the create call returns immediately. - Send the stored releases-list `ETag` as `If-None-Match`; a `304` derives nothing and is not counted against GitHub's rate limit, so an unchanged repo is nearly free. - Coordinate replicas through a `github_rpm_sync_state` row (`last_synced_at`, `etag`, `sync_lease_owner`, `sync_lease_expires`): a periodic scan runs only for the replica that atomically claims the lease, bounding total GitHub load to ~once per `mutable_ttl` regardless of replica count; the ETag is shared through the same row. - Keep the request path fast: serve current cache, enqueue a prime on an empty cache, and return a bounded wait then a retryable `503` rather than blocking on a cold derive. - Add `GITHUB_SYNC_RATE` / `GITHUB_SYNC_BURST` / `GITHUB_SYNC_WORKERS` / `GITHUB_SYNC_POLL_INTERVAL` config with conservative defaults (1 req/s, burst 5, 3 workers, 60s tick) and document the syncer in the README. ## Tests - Unit (httptest, Range/ETag-aware fixture): `304` releases response derives nothing; incremental derive fetches only the newly added asset; the shared limiter caps request rate; work-queue enqueues coalesce to one job; prime enqueues a job; a held lease stops a second replica from scanning; cold-start serves `503` while warm cache serves `200`. - DB integration (testcontainers postgres): the real lease SQL — one holder at a time, recency gate blocks a too-soon periodic re-claim, prime (freshness 0) bypasses recency but respects a live lease. - Docker e2e re-run: `dnf install dotvault` works; prime-on-create derives in the background at ~1 req/s (global limiter); `dnf makecache` served fast from the priming cache (no cold block); clean shutdown mid-scan, no panics. ## Notes - Reuses `mutable_ttl` as the check interval (no new per-remote field), per brief. Reviewed-on: #108 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
107 lines
4.8 KiB
Modula-2
107 lines
4.8 KiB
Modula-2
module git.unkin.net/unkin/artifactapi
|
|
|
|
go 1.25.9
|
|
|
|
require (
|
|
github.com/cavaliergopher/rpm v1.3.0
|
|
github.com/charmbracelet/bubbletea v1.3.10
|
|
github.com/charmbracelet/lipgloss v1.1.0
|
|
github.com/go-chi/chi/v5 v5.3.0
|
|
github.com/google/uuid v1.6.0
|
|
github.com/jackc/pgx/v5 v5.10.0
|
|
github.com/minio/minio-go/v7 v7.2.0
|
|
github.com/redis/go-redis/v9 v9.20.0
|
|
github.com/testcontainers/testcontainers-go v0.42.0
|
|
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
|
|
github.com/testcontainers/testcontainers-go/modules/redis v0.42.0
|
|
golang.org/x/crypto v0.51.0
|
|
golang.org/x/time v0.15.0
|
|
gopkg.in/yaml.v3 v3.0.1
|
|
)
|
|
|
|
require (
|
|
dario.cat/mergo v1.0.2 // indirect
|
|
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
|
|
github.com/Microsoft/go-winio v0.6.2 // indirect
|
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
|
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
|
|
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
|
github.com/charmbracelet/colorprofile v0.4.3 // indirect
|
|
github.com/charmbracelet/x/ansi v0.11.7 // indirect
|
|
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
|
|
github.com/charmbracelet/x/term v0.2.2 // indirect
|
|
github.com/clipperhouse/displaywidth v0.11.0 // indirect
|
|
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
|
|
github.com/containerd/errdefs v1.0.0 // indirect
|
|
github.com/containerd/errdefs/pkg v0.3.0 // indirect
|
|
github.com/containerd/log v0.1.0 // indirect
|
|
github.com/containerd/platforms v0.2.1 // indirect
|
|
github.com/cpuguy83/dockercfg v0.3.2 // indirect
|
|
github.com/davecgh/go-spew v1.1.1 // indirect
|
|
github.com/distribution/reference v0.6.0 // indirect
|
|
github.com/docker/go-connections v0.6.0 // indirect
|
|
github.com/docker/go-units v0.5.0 // indirect
|
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
|
github.com/ebitengine/purego v0.10.0 // indirect
|
|
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
|
github.com/felixge/httpsnoop v1.0.4 // indirect
|
|
github.com/go-logr/logr v1.4.3 // indirect
|
|
github.com/go-logr/stdr v1.2.2 // indirect
|
|
github.com/go-ole/go-ole v1.2.6 // indirect
|
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
|
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
|
github.com/klauspost/compress v1.18.6 // indirect
|
|
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
|
|
github.com/klauspost/crc32 v1.3.0 // indirect
|
|
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
|
|
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
|
|
github.com/magiconair/properties v1.8.10 // indirect
|
|
github.com/mattn/go-isatty v0.0.20 // indirect
|
|
github.com/mattn/go-localereader v0.0.1 // indirect
|
|
github.com/mattn/go-runewidth v0.0.23 // indirect
|
|
github.com/mdelapenya/tlscert v0.2.0 // indirect
|
|
github.com/minio/crc64nvme v1.1.1 // indirect
|
|
github.com/minio/md5-simd v1.1.2 // indirect
|
|
github.com/moby/docker-image-spec v1.3.1 // indirect
|
|
github.com/moby/go-archive v0.2.0 // indirect
|
|
github.com/moby/moby/api v1.54.1 // indirect
|
|
github.com/moby/moby/client v0.4.0 // indirect
|
|
github.com/moby/patternmatcher v0.6.1 // indirect
|
|
github.com/moby/sys/sequential v0.6.0 // indirect
|
|
github.com/moby/sys/user v0.4.0 // indirect
|
|
github.com/moby/sys/userns v0.1.0 // indirect
|
|
github.com/moby/term v0.5.2 // indirect
|
|
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
|
|
github.com/muesli/cancelreader v0.2.2 // indirect
|
|
github.com/muesli/termenv v0.16.0 // indirect
|
|
github.com/opencontainers/go-digest v1.0.0 // indirect
|
|
github.com/opencontainers/image-spec v1.1.1 // indirect
|
|
github.com/philhofer/fwd v1.2.0 // indirect
|
|
github.com/pmezard/go-difflib v1.0.0 // indirect
|
|
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
|
|
github.com/rivo/uniseg v0.4.7 // indirect
|
|
github.com/rs/xid v1.6.0 // indirect
|
|
github.com/shirou/gopsutil/v4 v4.26.3 // indirect
|
|
github.com/sirupsen/logrus v1.9.4 // indirect
|
|
github.com/stretchr/testify v1.11.1 // indirect
|
|
github.com/tinylib/msgp v1.6.1 // indirect
|
|
github.com/tklauser/go-sysconf v0.3.16 // indirect
|
|
github.com/tklauser/numcpus v0.11.0 // indirect
|
|
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
|
|
github.com/yusufpapurcu/wmi v1.2.4 // indirect
|
|
github.com/zeebo/xxh3 v1.1.0 // indirect
|
|
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
|
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 // indirect
|
|
go.opentelemetry.io/otel v1.41.0 // indirect
|
|
go.opentelemetry.io/otel/metric v1.41.0 // indirect
|
|
go.opentelemetry.io/otel/trace v1.41.0 // indirect
|
|
go.uber.org/atomic v1.11.0 // indirect
|
|
go.yaml.in/yaml/v3 v3.0.4 // indirect
|
|
golang.org/x/net v0.53.0 // indirect
|
|
golang.org/x/sync v0.20.0 // indirect
|
|
golang.org/x/sys v0.44.0 // indirect
|
|
golang.org/x/text v0.37.0 // indirect
|
|
gopkg.in/ini.v1 v1.67.2 // indirect
|
|
)
|