remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk)
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful

OS package remotes (rpm/deb/apk) fetch many small files and benefit from
spreading upstream load across mirrors and surviving a mirror outage. A
remote may now set a `mirrorlist` of additional upstream base URLs; the
effective upstream pool is [base_url] + mirrorlist, which the shared proxy
engine load-balances round-robin and, on a network error/timeout/5xx,
fails over to the next mirror before returning an error. Because selection
happens in the engine, it works for every provider that reaches upstream.

Backward compatible: `base_url` stays a plain string that providers read
unchanged, and a remote with no mirrorlist behaves exactly as today
(single attempt, same error path).

- add models.Remote.Mirrorlist ([]string, json "mirrorlist,omitempty") and
  UpstreamPool() = [base_url] + mirrorlist; ValidateMirrorlist enforces
  remote repo_type + package_type in {rpm, deb, alpine} and http/https URLs
- v2 create/update: reject a mirrorlist on any other repo (400); base_url
  remains required for remotes
- persist mirrorlist in a new additive `mirrorlist TEXT[]` column
  (remoteCols/scanRemote/CreateRemote/UpdateRemote); base_url column
  unchanged
- engine: per-remote round-robin cursor over the pool; wrap the
  fetch/head/revalidate upstream calls in a failover loop that narrows the
  remote to one selected mirror per attempt; only network errors and 5xx
  fail over (404/403/... return as-is); circuit breaker stays keyed per
  remote and trips only after all mirrors fail
- tests: model JSON round-trip + validation gating, engine
  round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip,
  and a docker acceptance suite (round-robin across two mock upstreams,
  failover past a dead primary, no-mirrorlist regression, and a real dnf
  makecache+install through a two-mirror rpm remote whose base_url is dead)

Least-connections and a per-remote strategy selector are a follow-up PR.
This commit is contained in:
2026-08-13 08:26:17 +10:00
parent 73c0bfc670
commit ac8be0a105
22 changed files with 763 additions and 13 deletions
+33
View File
@@ -99,6 +99,39 @@ func TestRemotesCRUD(t *testing.T) {
}
}
func TestRemoteMirrorlistRoundTrip(t *testing.T) {
requireDB(t)
mirrors := []string{"https://b.example", "https://c.example"}
if err := testDB.CreateRemote(ctx(), &models.Remote{
Name: "r-mirror", PackageType: models.PackageRPM, RepoType: models.RepoTypeRemote,
BaseURL: "https://a.example", Mirrorlist: mirrors, MutableTTL: 3600,
}); err != nil {
t.Fatalf("create mirrorlist remote: %v", err)
}
defer testDB.DeleteRemote(ctx(), "r-mirror")
got, err := testDB.GetRemote(ctx(), "r-mirror")
if err != nil {
t.Fatalf("get: %v", err)
}
if got.BaseURL != "https://a.example" {
t.Fatalf("BaseURL = %q, want https://a.example", got.BaseURL)
}
if len(got.Mirrorlist) != 2 || got.Mirrorlist[0] != mirrors[0] || got.Mirrorlist[1] != mirrors[1] {
t.Fatalf("Mirrorlist round-trip = %v, want %v", got.Mirrorlist, mirrors)
}
// Clearing the mirrorlist on update persists an empty list.
got.Mirrorlist = nil
if err := testDB.UpdateRemote(ctx(), got); err != nil {
t.Fatalf("update clearing mirrorlist: %v", err)
}
got, _ = testDB.GetRemote(ctx(), "r-mirror")
if len(got.Mirrorlist) != 0 {
t.Fatalf("mirrorlist after clear = %v, want empty", got.Mirrorlist)
}
}
func TestArtifactsAndBlobs(t *testing.T) {
requireDB(t)
seedRemote(t, "r-art")