unkin-agent
f1820fd104
## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.
**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).
## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.
## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.
## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.
Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
46 lines
2.2 KiB
Markdown
46 lines
2.2 KiB
Markdown
# Dockerised end-to-end suite
|
|
|
|
Black-box tests that run against a fully **containerised** artifactapi stack
|
|
(built image + Postgres + Redis + MinIO) plus a static mock upstream. Unlike the
|
|
in-process `e2e/` suite (testcontainers, server run in-process), these only speak
|
|
HTTP to the running product, so they exercise the shipped container image.
|
|
|
|
## Run
|
|
|
|
```bash
|
|
make docker-e2e # build image, compose up, run suite, compose down
|
|
```
|
|
|
|
`scripts/docker-e2e.sh` builds and starts `docker-compose.yml` +
|
|
`docker-compose.e2e.yml`, waits for `/health`, then runs
|
|
`go test -tags=dockere2e ./e2e-docker/...` and tears everything down.
|
|
|
|
The stack publishes artifactapi on host port **8001** (to avoid colliding with a
|
|
local instance on 8000). Override with `ARTIFACTAPI_URL` to point the tests at an
|
|
already-running stack.
|
|
|
|
## Coverage
|
|
|
|
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
|
|
- **Caching** — one immutable artifact per remote package type (generic, docker,
|
|
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the
|
|
mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes
|
|
verified against the origin fixture.
|
|
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/`
|
|
index), rpm (real package + **automatic repodata** generation).
|
|
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge
|
|
across two members.
|
|
- **Mirrorlist** — an rpm remote with a `mirrorlist` of extra upstream mirrors
|
|
(pool = `base_url` + `mirrorlist`): round-robin distribution across both mirrors
|
|
(constant-body `mockupstreama` / `mockupstreamb`), failover past a dead primary,
|
|
no-mirrorlist regression, and a real `dnf` (stock `rockylinux:9` container)
|
|
`makecache` + `install` through a two-mirror rpm remote whose `base_url` is dead
|
|
— a dead mirror must not break the client.
|
|
|
|
## Fixtures
|
|
|
|
`fixtures/` is served by the mock upstream at its web root. Paths mirror each
|
|
provider's upstream URL layout (e.g. `v2/...` for docker, `v1/providers/...` for
|
|
terraform). The RPM under `fixtures/rpmrepo/Packages/` is a real package so the
|
|
rpm provider can parse its metadata for repodata generation.
|