remotes: add mirrorlist for round-robin + failover across mirrors (rpm/deb/apk) (#121)
## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.
**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).
## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.
## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.
## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.
Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
This commit was merged in pull request #121.
This commit is contained in:
+53
-3
@@ -2,6 +2,7 @@ package models
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"time"
|
||||
)
|
||||
@@ -39,9 +40,13 @@ type Remote struct {
|
||||
PackageType PackageType `json:"package_type"`
|
||||
RepoType RepoType `json:"repo_type"`
|
||||
BaseURL string `json:"base_url"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Username string `json:"-"`
|
||||
Password string `json:"-"`
|
||||
// Mirrorlist holds additional upstream mirror base URLs. The effective
|
||||
// upstream pool is [base_url] + mirrorlist, load-balanced round-robin with
|
||||
// failover by the proxy engine. Only valid on remote rpm/deb/apk repos.
|
||||
Mirrorlist []string `json:"mirrorlist,omitempty"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Username string `json:"-"`
|
||||
Password string `json:"-"`
|
||||
|
||||
ImmutableTTL int `json:"immutable_ttl"`
|
||||
MutableTTL int `json:"mutable_ttl"`
|
||||
@@ -72,6 +77,51 @@ type Remote struct {
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
// mirrorlistPackageTypes are the package types for which a mirrorlist is
|
||||
// allowed: OS package repos (rpm, deb, apk/alpine) that fetch many small files
|
||||
// and benefit most from mirror load-balancing and failover.
|
||||
var mirrorlistPackageTypes = map[PackageType]bool{
|
||||
PackageRPM: true,
|
||||
PackageDeb: true,
|
||||
PackageAlpine: true,
|
||||
}
|
||||
|
||||
// UpstreamPool returns the ordered upstream base URLs for this remote: the
|
||||
// primary base_url first, followed by any mirrorlist entries. The proxy engine
|
||||
// load-balances round-robin across the pool and fails over between them.
|
||||
func (r Remote) UpstreamPool() []string {
|
||||
pool := make([]string, 0, 1+len(r.Mirrorlist))
|
||||
if r.BaseURL != "" {
|
||||
pool = append(pool, r.BaseURL)
|
||||
}
|
||||
pool = append(pool, r.Mirrorlist...)
|
||||
return pool
|
||||
}
|
||||
|
||||
// ValidateMirrorlist enforces that a mirrorlist is only configured on remote
|
||||
// rpm/deb/apk repositories and that every entry is a parseable http/https URL.
|
||||
func (r *Remote) ValidateMirrorlist() error {
|
||||
if len(r.Mirrorlist) == 0 {
|
||||
return nil
|
||||
}
|
||||
if r.RepoType != RepoTypeRemote {
|
||||
return fmt.Errorf("mirrorlist is only allowed on remote repositories")
|
||||
}
|
||||
if !mirrorlistPackageTypes[r.PackageType] {
|
||||
return fmt.Errorf("mirrorlist is only allowed for rpm, deb and alpine package types, not %q", r.PackageType)
|
||||
}
|
||||
for _, u := range r.Mirrorlist {
|
||||
parsed, err := url.ParseRequestURI(u)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid mirrorlist url %q: %w", u, err)
|
||||
}
|
||||
if parsed.Scheme != "http" && parsed.Scheme != "https" {
|
||||
return fmt.Errorf("mirrorlist url %q must be http or https", u)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidatePatterns ensures every configured regex compiles. Storing an
|
||||
// invalid pattern would otherwise be silently dropped at match time, which
|
||||
// for the blocklist is a fail-open: a mistyped deny rule becomes a no-op.
|
||||
|
||||
Reference in New Issue
Block a user