Lazy per-replica scans re-derived RPM metadata on the client request path
and, run independently on every replica, multiplied GitHub queries by the
replica count. A single background syncer with a shared rate limit, ETag
conditional checks, and a DB lease keeps metadata fresh off the request path
while bounding GitHub load to ~once per mutable_ttl across the fleet.
- Add a single per-process syncer (started at boot, stopped on shutdown) that
owns a deduped/coalescing work queue, a worker pool, and one global
token-bucket rate limiter bound onto the github provider so every GitHub call
(releases list + each ranged asset GET) acquires a token first.
- Check each github_rpm remote for new/changed releases on its mutable_ttl
cadence; derive only new/changed assets incrementally and prune assets that
disappear upstream, so repodata is served from primed DB rows.
- Prime metadata in the background on remote creation; the create call never
blocks on a derive.
- Send the stored releases-list ETag as If-None-Match; a 304 derives nothing
(and does not count against GitHub's rate limit), making an unchanged repo
nearly free.
- Coordinate replicas through a github_rpm_sync_state row (last_synced_at,
etag, sync_lease_owner, sync_lease_expires): a periodic scan runs only for
the replica that atomically claims the lease, bounding total GitHub load to
~once per mutable_ttl regardless of replica count.
- Keep the request path fast: serve current cache, enqueue a prime on an empty
cache, and return a bounded wait then a retryable 503 rather than blocking on
a cold derive.
- Add GITHUB_SYNC_RATE/BURST/WORKERS/POLL_INTERVAL config (conservative
defaults) and document the syncer in the README.
Fixes#72
## Why
`compilePatterns` silently discards any pattern that fails to compile. A typo in a blocklist entry therefore turns a deny rule into a no-op — a fail-open with security impact.
## Changes
- Add `Remote.ValidatePatterns`, which compiles every pattern list (patterns, blocklist, mutable/immutable patterns, ban_tags) and returns an error on the first invalid regex.
- Reject invalid patterns with 400 at remote create and update time.
- Unit test for valid and invalid patterns.
## Validation
- `go test ./pkg/models/` and `make e2e` pass.
Reviewed-on: #87
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Introduces repo_type (remote/local) as a separate axis from package_type
so that any package type can be hosted locally. A terraform local repo
is package_type=terraform + repo_type=local.
- Remote model gains RepoType field (defaults to "remote")
- Database schema adds repo_type column with migration for existing DBs
- V1 proxy adds /api/v1/local/{name}/* route for serving local files
- V2 upload via PUT /api/v2/remotes/{name}/files/{ns}/{type}/{file}.zip
validates filename matches terraform-provider-{type}_{ver}_{os}_{arch}.zip
and returns 409 on duplicate (no overwrites)
- index.json and {version}.json are computed on-the-fly from uploaded zips
rather than stored as separate files
- V2 create validates repo_type and requires base_url only for remotes
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #49
unkinben
benvin