unkin-agent
ac8be0a105
OS package remotes (rpm/deb/apk) fetch many small files and benefit from
spreading upstream load across mirrors and surviving a mirror outage. A
remote may now set a `mirrorlist` of additional upstream base URLs; the
effective upstream pool is [base_url] + mirrorlist, which the shared proxy
engine load-balances round-robin and, on a network error/timeout/5xx,
fails over to the next mirror before returning an error. Because selection
happens in the engine, it works for every provider that reaches upstream.
Backward compatible: `base_url` stays a plain string that providers read
unchanged, and a remote with no mirrorlist behaves exactly as today
(single attempt, same error path).
- add models.Remote.Mirrorlist ([]string, json "mirrorlist,omitempty") and
UpstreamPool() = [base_url] + mirrorlist; ValidateMirrorlist enforces
remote repo_type + package_type in {rpm, deb, alpine} and http/https URLs
- v2 create/update: reject a mirrorlist on any other repo (400); base_url
remains required for remotes
- persist mirrorlist in a new additive `mirrorlist TEXT[]` column
(remoteCols/scanRemote/CreateRemote/UpdateRemote); base_url column
unchanged
- engine: per-remote round-robin cursor over the pool; wrap the
fetch/head/revalidate upstream calls in a failover loop that narrows the
remote to one selected mirror per attempt; only network errors and 5xx
fail over (404/403/... return as-is); circuit breaker stays keyed per
remote and trips only after all mirrors fail
- tests: model JSON round-trip + validation gating, engine
round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip,
and a docker acceptance suite (round-robin across two mock upstreams,
failover past a dead primary, no-mirrorlist regression, and a real dnf
makecache+install through a two-mirror rpm remote whose base_url is dead)
Least-connections and a per-remote strategy selector are a follow-up PR.
2.2 KiB
Dockerised end-to-end suite
Black-box tests that run against a fully containerised artifactapi stack
(built image + Postgres + Redis + MinIO) plus a static mock upstream. Unlike the
in-process e2e/ suite (testcontainers, server run in-process), these only speak
HTTP to the running product, so they exercise the shipped container image.
Run
make docker-e2e # build image, compose up, run suite, compose down
scripts/docker-e2e.sh builds and starts docker-compose.yml +
docker-compose.e2e.yml, waits for /health, then runs
go test -tags=dockere2e ./e2e-docker/... and tears everything down.
The stack publishes artifactapi on host port 8001 (to avoid colliding with a
local instance on 8000). Override with ARTIFACTAPI_URL to point the tests at an
already-running stack.
Coverage
- Repository lifecycle — add / change / delete for remote, local and virtual repos.
- Caching — one immutable artifact per remote package type (generic, docker,
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the
mock upstream: first fetch
X-Artifact-Source: remote, secondcache, bytes verified against the origin fixture. - Local uploads — generic (upload/download), pypi (wheel + generated
simple/index), rpm (real package + automatic repodata generation). - Virtual repositories — pypi simple-index merge and helm
index.yamlmerge across two members. - Mirrorlist — an rpm remote with a
mirrorlistof extra upstream mirrors (pool =base_url+mirrorlist): round-robin distribution across both mirrors (constant-bodymockupstreama/mockupstreamb), failover past a dead primary, no-mirrorlist regression, and a realdnf(stockrockylinux:9container)makecache+installthrough a two-mirror rpm remote whosebase_urlis dead — a dead mirror must not break the client.
Fixtures
fixtures/ is served by the mock upstream at its web root. Paths mirror each
provider's upstream URL layout (e.g. v2/... for docker, v1/providers/... for
terraform). The RPM under fixtures/rpmrepo/Packages/ is a real package so the
rpm provider can parse its metadata for repodata generation.