unkin/artifactapi

Fork 0

Files

T

History

unkin-agent cd7c2c4383

ci/woodpecker/tag/docker Pipeline was successful

Details

remotes: add least-connections mirror strategy (round-robin remains default) (#122 )

## Why

The mirrorlist (PR #121) always load-balances round-robin. Round-robin is oblivious to how busy each mirror is, so a slow or saturated mirror keeps getting its fair share of new requests. This adds an opt-in **least-connections** strategy that favors the mirror currently handling the fewest in-flight requests, steering new work toward idle mirrors. Round-robin stays the default, so existing remotes are unchanged.

## How

- **Model**: `models.Remote` gains `mirror_strategy` (`round_robin` default/empty, or `least_conn`). `ValidateMirrorStrategy` checks the enum and requires a non-empty mirrorlist for `least_conn`. Empty behaves as `round_robin` for back-compat.
- **DB**: additive `mirror_strategy TEXT NOT NULL DEFAULT 'round_robin'` column (CREATE TABLE + `ADD COLUMN IF NOT EXISTS`), wired through remoteCols/scanRemote/CreateRemote/UpdateRemote; empty normalized to `round_robin` on write.
- **Engine**: a per-remote, per-pool-URL atomic in-flight gauge is incremented around each upstream call (head/fetch/checkUpstream). For a `least_conn` remote the attempt order starts with the least-loaded pool URL (ties broken by the existing round-robin rotation). Round-robin path and failover order are unchanged; single-URL pools are a no-op.
- **Tests**: unit tests for least-loaded selection, round-robin default, gauge inc/dec, single-URL no-op, and validation; DB round-trip covers the new column; docker e2e adds a `least_conn` distribution test plus a real `dnf` install through a `least_conn` remote.

Back-compat: unset/empty `mirror_strategy` is `round_robin`, so all existing remotes keep their current behavior.
Reviewed-on: #122
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>

2026-08-13 17:23:59 +10:00

fixtures

remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk) (#121 )

2026-08-13 16:08:55 +10:00

mirror-conf

remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk) (#121 )

2026-08-13 16:08:55 +10:00

caching_test.go

test: comprehensive dockerised end-to-end suite (#97 )

2026-07-03 14:34:45 +10:00

docker_local_test.go

feat: serve local docker repos as a real registry (#103 )

2026-07-05 16:55:53 +10:00

helpers_test.go

test: comprehensive dockerised end-to-end suite (#97 )

2026-07-03 14:34:45 +10:00

leastconn_test.go

remotes: add least-connections mirror strategy (round-robin remains default) (#122 )

2026-08-13 17:23:59 +10:00

local_test.go

Add Alpine/apk local repository support (#114 )

2026-08-12 20:39:27 +10:00

multibaseurl_test.go

remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk) (#121 )

2026-08-13 16:08:55 +10:00

README.md

remotes: add least-connections mirror strategy (round-robin remains default) (#122 )

2026-08-13 17:23:59 +10:00

repos_test.go

test: comprehensive dockerised end-to-end suite (#97 )

2026-07-03 14:34:45 +10:00

virtual_test.go

test: comprehensive dockerised end-to-end suite (#97 )

2026-07-03 14:34:45 +10:00

README.md

Dockerised end-to-end suite

Black-box tests that run against a fully containerised artifactapi stack (built image + Postgres + Redis + MinIO) plus a static mock upstream. Unlike the in-process e2e/ suite (testcontainers, server run in-process), these only speak HTTP to the running product, so they exercise the shipped container image.

Run

make docker-e2e          # build image, compose up, run suite, compose down

scripts/docker-e2e.sh builds and starts docker-compose.yml + docker-compose.e2e.yml, waits for /health, then runs go test -tags=dockere2e ./e2e-docker/... and tears everything down.

The stack publishes artifactapi on host port 8001 (to avoid colliding with a local instance on 8000). Override with ARTIFACTAPI_URL to point the tests at an already-running stack.

Coverage

Repository lifecycle — add / change / delete for remote, local and virtual repos.
Caching — one immutable artifact per remote package type (generic, docker, helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the mock upstream: first fetch X-Artifact-Source: remote, second cache, bytes verified against the origin fixture.
Local uploads — generic (upload/download), pypi (wheel + generated simple/ index), rpm (real package + automatic repodata generation).
Virtual repositories — pypi simple-index merge and helm index.yaml merge across two members.
Mirrorlist — an rpm remote with a mirrorlist of extra upstream mirrors (pool = base_url + mirrorlist): round-robin distribution across both mirrors (constant-body mockupstreama / mockupstreamb), failover past a dead primary, no-mirrorlist regression, and a real dnf (stock rockylinux:9 container) makecache + install through a two-mirror rpm remote whose base_url is dead — a dead mirror must not break the client.
Mirror strategy (least_conn) — a mirror_strategy: least_conn rpm remote over the two constant-body mirrors exercises the least-connections selection path end-to-end (both mirrors serve, all requests succeed), plus a real dnf install through a least_conn remote with a dead primary (failover unchanged). The precise least-loaded pick is asserted deterministically in the proxy unit test, since an in-flight-skew assertion over HTTP is timing-sensitive.

Fixtures

fixtures/ is served by the mock upstream at its web root. Paths mirror each provider's upstream URL layout (e.g. v2/... for docker, v1/providers/... for terraform). The RPM under fixtures/rpmrepo/Packages/ is a real package so the rpm provider can parse its metadata for repodata generation.