unkin-agent
f1820fd104
## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.
**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).
## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.
## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.
## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.
Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
51 lines
2.1 KiB
Bash
Executable File
51 lines
2.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Build the artifactapi container, bring up the full stack (postgres, redis,
|
|
# minio, artifactapi) plus a static mock upstream, and run the dockerised e2e
|
|
# suite against the running product over HTTP. Tears everything down on exit.
|
|
set -euo pipefail
|
|
|
|
cd "$(dirname "$0")/.."
|
|
|
|
# Publish artifactapi on 8001 to avoid colliding with a local instance on 8000.
|
|
export ARTIFACTAPI_PORT="${ARTIFACTAPI_PORT:-8001}"
|
|
COMPOSE=(docker compose -f docker-compose.yml -f docker-compose.e2e.yml)
|
|
API_URL="${ARTIFACTAPI_URL:-http://localhost:${ARTIFACTAPI_PORT}}"
|
|
|
|
cleanup() {
|
|
echo "==> tearing down stack"
|
|
"${COMPOSE[@]}" down -v --remove-orphans >/dev/null 2>&1 || true
|
|
}
|
|
trap cleanup EXIT
|
|
|
|
echo "==> building and starting stack (postgres, redis, minio, mockupstream(s), artifactapi)"
|
|
"${COMPOSE[@]}" up -d --build postgres redis minio mockupstream mockupstreama mockupstreamb artifactapi
|
|
|
|
echo "==> waiting for artifactapi health at ${API_URL}"
|
|
for i in $(seq 1 60); do
|
|
if curl -fsS "${API_URL}/health" >/dev/null 2>&1; then
|
|
echo " healthy after ${i}s"
|
|
break
|
|
fi
|
|
if [ "$i" -eq 60 ]; then
|
|
echo "!!! artifactapi did not become healthy in time; recent logs:"
|
|
"${COMPOSE[@]}" logs --tail=50 artifactapi
|
|
exit 1
|
|
fi
|
|
sleep 1
|
|
done
|
|
|
|
# Resolve the compose network the artifactapi container is attached to, so the
|
|
# real-package-manager test can launch a stock distro container on the same
|
|
# network and reach artifactapi by service name.
|
|
API_CID="$("${COMPOSE[@]}" ps -q artifactapi)"
|
|
COMPOSE_NETWORK="$(docker inspect -f '{{range $k,$_ := .NetworkSettings.Networks}}{{$k}}{{end}}' "${API_CID}" 2>/dev/null || true)"
|
|
|
|
echo "==> running dockerised e2e suite (compose network: ${COMPOSE_NETWORK:-unknown})"
|
|
ARTIFACTAPI_URL="${API_URL}" \
|
|
MOCK_UPSTREAM_INTERNAL="${MOCK_UPSTREAM_INTERNAL:-http://mockupstream}" \
|
|
MOCK_UPSTREAM_A_INTERNAL="${MOCK_UPSTREAM_A_INTERNAL:-http://mockupstreama}" \
|
|
MOCK_UPSTREAM_B_INTERNAL="${MOCK_UPSTREAM_B_INTERNAL:-http://mockupstreamb}" \
|
|
ARTIFACTAPI_INTERNAL="${ARTIFACTAPI_INTERNAL:-http://artifactapi:8000}" \
|
|
COMPOSE_NETWORK="${COMPOSE_NETWORK}" \
|
|
go test -tags=dockere2e -count=1 -timeout=10m -v ./e2e-docker/...
|