remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk) #121

2026-08-13T08:26:56+10:00

unkin-agent commented

2026-08-13 08:26:56 +10:00

Why

OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a mirrorlist of additional upstream base URLs. The effective upstream pool is [base_url] + mirrorlist, which the shared proxy engine load-balances round-robin and, on a network error/timeout/5xx, fails over to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.

Backward compatible: base_url stays a plain string (providers read it unchanged), and a remote with no mirrorlist behaves exactly as today (single attempt, same error path).

How

models.Remote.Mirrorlist ([]string, json:"mirrorlist,omitempty") + UpstreamPool() = [base_url] + mirrorlist.
ValidateMirrorlist: a non-empty mirrorlist is allowed only when repo_type==remote and package_type ∈ {rpm, deb, alpine}; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); base_url stays required for remotes.
Persist the mirrorlist in a new additive mirrorlist TEXT[] column (remoteCols/scanRemote/CreateRemote/UpdateRemote); the base_url column is unchanged.
Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.

Scope

Round-robin + failover only, restricted to remote rpm/deb/apk repos. Least-connections and a per-remote strategy selector are a follow-up PR.

Tests

Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. make test (go test -race) green.
Docker acceptance (e2e-docker, dockere2e tag, wired into docker-e2e.sh): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a real dnf makecache + install through a two-mirror rpm remote whose base_url is dead. All four pass locally.

## Why OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream. **Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path). ## How - `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`. - `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes. - Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged. - Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail. ## Scope Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**. ## Tests - Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green. - Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.

unkin-agent added 1 commit 2026-08-13 08:37:28 +10:00

remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk)

ci/woodpecker/pr/build Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ac8be0a105

OS package remotes (rpm/deb/apk) fetch many small files and benefit from
spreading upstream load across mirrors and surviving a mirror outage. A
remote may now set a `mirrorlist` of additional upstream base URLs; the
effective upstream pool is [base_url] + mirrorlist, which the shared proxy
engine load-balances round-robin and, on a network error/timeout/5xx,
fails over to the next mirror before returning an error. Because selection
happens in the engine, it works for every provider that reaches upstream.

Backward compatible: `base_url` stays a plain string that providers read
unchanged, and a remote with no mirrorlist behaves exactly as today
(single attempt, same error path).

- add models.Remote.Mirrorlist ([]string, json "mirrorlist,omitempty") and
  UpstreamPool() = [base_url] + mirrorlist; ValidateMirrorlist enforces
  remote repo_type + package_type in {rpm, deb, alpine} and http/https URLs
- v2 create/update: reject a mirrorlist on any other repo (400); base_url
  remains required for remotes
- persist mirrorlist in a new additive `mirrorlist TEXT[]` column
  (remoteCols/scanRemote/CreateRemote/UpdateRemote); base_url column
  unchanged
- engine: per-remote round-robin cursor over the pool; wrap the
  fetch/head/revalidate upstream calls in a failover loop that narrows the
  remote to one selected mirror per attempt; only network errors and 5xx
  fail over (404/403/... return as-is); circuit breaker stays keyed per
  remote and trips only after all mirrors fail
- tests: model JSON round-trip + validation gating, engine
  round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip,
  and a docker acceptance suite (round-robin across two mock upstreams,
  failover past a dead primary, no-mirrorlist regression, and a real dnf
  makecache+install through a two-mirror rpm remote whose base_url is dead)

Least-connections and a per-remote strategy selector are a follow-up PR.

unkin-agent force-pushed benvin/multi-baseurl-core from 68a1f14e17 to ac8be0a105

2026-08-13 08:37:28 +10:00

Compare

unkin-agent changed title from ~~remotes: support multiple base_urls with round-robin + failover~~ to remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk)

2026-08-13 08:37:46 +10:00

benvin merged commit f1820fd104 into master

2026-08-13 16:08:56 +10:00

benvin deleted branch benvin/multi-baseurl-core

2026-08-13 16:08:56 +10:00

unkin-agent referenced this pull request

2026-08-13 16:20:42 +10:00

remotes: add least-connections mirror strategy (round-robin remains default) #122

benvin referenced this issue from a commit

2026-08-13 17:24:00 +10:00

remotes: add least-connections mirror strategy (round-robin remains default) (#122)

Sign in to join this conversation.