unkin-agent
f1820fd104
## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.
**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).
## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.
## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.
## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.
Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
35 lines
1.3 KiB
YAML
35 lines
1.3 KiB
YAML
# Overlay for the dockerised end-to-end suite (scripts/docker-e2e.sh).
|
|
# Adds a static mock upstream that the artifactapi container proxies, so the
|
|
# caching tests are hermetic and need no internet access.
|
|
services:
|
|
mockupstream:
|
|
image: nginx:alpine
|
|
volumes:
|
|
- ./e2e-docker/fixtures:/usr/share/nginx/html:ro,z
|
|
# No host port needed: only the artifactapi container talks to it, and the
|
|
# tests compare served bytes against the on-disk fixtures.
|
|
|
|
# Two constant-body upstreams for the multi-base_url suite: each returns a
|
|
# distinct, upstream-identifying body for any path, so round-robin
|
|
# distribution across a two-mirror remote is directly observable.
|
|
mockupstreama:
|
|
image: nginx:alpine
|
|
volumes:
|
|
- ./e2e-docker/mirror-conf/a.conf:/etc/nginx/conf.d/default.conf:ro,z
|
|
mockupstreamb:
|
|
image: nginx:alpine
|
|
volumes:
|
|
- ./e2e-docker/mirror-conf/b.conf:/etc/nginx/conf.d/default.conf:ro,z
|
|
|
|
artifactapi:
|
|
# The host port is set via ARTIFACTAPI_PORT (see scripts/docker-e2e.sh),
|
|
# defaulting to 8000; the e2e run uses 8001 to avoid colliding with a
|
|
# locally-running instance.
|
|
depends_on:
|
|
mockupstream:
|
|
condition: service_started
|
|
mockupstreama:
|
|
condition: service_started
|
|
mockupstreamb:
|
|
condition: service_started
|