Yank hard-deleted a local gem version, so the same name-version-platform could be re-pushed with different bytes and break lockfiles pinning the old checksum. rubygems.org forbids re-pushing a yanked version.
- write a `_yanked/<name>/<full>` tombstone before deleting the gem, quick spec and index line
- reject a push of a tombstoned version with 409
- serialise push and yank of one version on the engine Redis lock; 503 when it stays busy
- drop partial push writes on failure
- return 404 for `_index/` and `_yanked/` paths on gems locals
- encode out-of-range gemspec dates (before 1900) as the epoch instead of wrapping the Time#_dump year field
- extend the ruby:3.3 e2e to yank, re-push different bytes and fail to install the yanked version
Reviewed-on: #155
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Bundler needs one source for internal and public gems, but gems virtuals have no merger, so a local gems repo and rubygems.org cannot be served together.
- merge `info/<gem>` across members (member order wins a duplicate version-platform); pass a single-member gem through verbatim
- regenerate `versions` lines and md5s for gems several members list; union `names`
- 302 `gems/*.gem` and `quick/Marshal.4.8/*` to the member owning that version, via the shared member-path validation
- return 502 on a member error; cache merges in the shared virtual cache; route HEAD on virtuals
- add `docs/gems/virtual.md` and a stock `ruby:3.3` bundler e2e-docker case
Reviewed-on: #154
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
In-house gems have nowhere to publish: gems locals only take raw file uploads, which `gem push`, `gem install` and bundler cannot use.
- accept `gem push` (`POST <repo>/api/v1/gems`, 256MiB cap, 8MiB metadata cap) and `gem yank` (`DELETE <repo>/api/v1/gems/yank`); any API key works
- parse the gemspec from the `.gem`; reject an existing name-version-platform with 409
- serve the compact index (`versions`, `info/<gem>`, `names`), `.gem` files and generated `quick/Marshal.4.8` specs; no legacy `specs.4.8.gz`
- route HEAD on locals (gem probes `versions` with HEAD)
- add `docs/gems/local.md`, a UI snippet and a stock `ruby:3.3` e2e-docker case
Reviewed-on: #153
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
npm clients need one registry for internal and public packages, but npm virtuals have no merger, so a local npm repo and npmjs cannot be served together.
- merge packuments across members: union versions (member order wins), first-defined dist-tags minus ones naming missing versions
- serve full and abbreviated (`install-v1`) packuments, scoped names included, via the shared virtual merge cache
- point tarballs at `<virtual>/<member>/<path>` and 302 them to the owning member
- reject an invalid Host with 400; serve virtual and local packuments `no-store` with `Vary: Host, X-Forwarded-Proto, Accept`
- return 502 when a member errors; skip members without the package
- add `docs/npm/virtual.md` and a stock `node:22` e2e-docker case
Reviewed-on: #152
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
In-house npm packages have nowhere to publish: npm locals only accept raw file uploads, which `npm publish` and `npm install` cannot use.
- accept `npm publish` (`PUT /api/v1/local/<repo>/<pkg>`, scoped included) on local npm repos, capping the body at 256MiB and verifying the tarball against dist integrity
- write tarball, then manifest, then dist-tags; publishing an existing version returns 409 and leaves tags alone, while a matching leftover tarball from a failed publish is reused so retries complete
- serve packuments with tarball URLs pointing back at the local repo, dropping dist-tags that name unpublished versions
- document the `.npmrc` registry + `_authToken` setup in the UI and `docs/npm/local.md`
- add a stock `node:22` publish/install e2e-docker case
Reviewed-on: #139
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Parallel package-type PRs conflict because every type edits the same shared lines (README tables, provider.go, server.go imports, the package type list, UI maps, the e2e caching table). A new type now only adds files.
- move per-type README sections into docs/<type>/{remote,local,virtual}.md
- split provider.go capability and metadata types into cap_*.go and metadata_*.go
- import providers via internal/provider/all, one file per type
- declare each package type in package_type_<type>.go, registered via init()
- split the e2e caching cases and UI colours/snippets into per-type files
Reviewed-on: #150
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Puppet modules can only be proxied from a remote Forge; there is nowhere to publish internal modules for puppet module install, r10k and g10k.
- add puppet local uploads at <owner>-<module>-<version>.tar.gz, rejecting existing versions
- record each release's metadata.json, md5 and sha256 in a puppet_metadata table, streaming the tarball under size caps
- serve Forge v3 /v3/modules, paginated /v3/releases, /v3/releases/<slug> and /v3/files
- add an e2e-docker case driving real r10k and puppet module install
- document local puppet in README and UI usage snippets
Reviewed-on: #145
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
.NET builds restore from nuget.org directly because artifactapi has no NuGet remote type.
- add `nuget` package type proxying a NuGet v3 feed; `index.json` maps to the upstream service index
- add a `ServeRewriter` provider hook so cached JSON has upstream-origin URLs pointed back at the requesting host per request
- treat `.nupkg`/`.nuspec` as immutable and service, registration and flat container indexes as mutable
- add a `nuget.config` usage snippet to the UI and a nuget e2e caching case
Reviewed-on: #141
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Rust crate builds (rpmbuilder CI) fetch from crates.io directly because artifactapi has no Cargo remote type.
- add `cargo` package type proxying the sparse registry protocol (RFC 2789)
- synthesize `config.json` so `dl` points crate downloads back at the remote
- treat index files as mutable and `crates/*/*.crate` as immutable, fetched from static.crates.io for crates.io
- add a `~/.cargo/config.toml` usage snippet to the UI and a cargo e2e caching case
Reviewed-on: #132
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The mirrorlist (PR #121) always load-balances round-robin. Round-robin is oblivious to how busy each mirror is, so a slow or saturated mirror keeps getting its fair share of new requests. This adds an opt-in **least-connections** strategy that favors the mirror currently handling the fewest in-flight requests, steering new work toward idle mirrors. Round-robin stays the default, so existing remotes are unchanged.
## How
- **Model**: `models.Remote` gains `mirror_strategy` (`round_robin` default/empty, or `least_conn`). `ValidateMirrorStrategy` checks the enum and requires a non-empty mirrorlist for `least_conn`. Empty behaves as `round_robin` for back-compat.
- **DB**: additive `mirror_strategy TEXT NOT NULL DEFAULT 'round_robin'` column (CREATE TABLE + `ADD COLUMN IF NOT EXISTS`), wired through remoteCols/scanRemote/CreateRemote/UpdateRemote; empty normalized to `round_robin` on write.
- **Engine**: a per-remote, per-pool-URL atomic in-flight gauge is incremented around each upstream call (head/fetch/checkUpstream). For a `least_conn` remote the attempt order starts with the least-loaded pool URL (ties broken by the existing round-robin rotation). Round-robin path and failover order are unchanged; single-URL pools are a no-op.
- **Tests**: unit tests for least-loaded selection, round-robin default, gauge inc/dec, single-URL no-op, and validation; DB round-trip covers the new column; docker e2e adds a `least_conn` distribution test plus a real `dnf` install through a `least_conn` remote.
Back-compat: unset/empty `mirror_strategy` is `round_robin`, so all existing remotes keep their current behavior.
Reviewed-on: #122
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
OS package remotes (rpm/deb/apk) fetch many small files and benefit from spreading upstream load across mirrors and surviving a mirror outage. A remote may now set a **`mirrorlist`** of additional upstream base URLs. The effective upstream pool is **`[base_url] + mirrorlist`**, which the shared proxy engine load-balances **round-robin** and, on a network error/timeout/5xx, **fails over** to the next mirror before returning an error. Selection happens in the engine, so it works for every provider that reaches upstream.
**Backward compatible:** `base_url` stays a plain string (providers read it unchanged), and a remote with **no mirrorlist behaves exactly as today** (single attempt, same error path).
## How
- `models.Remote.Mirrorlist` (`[]string`, `json:"mirrorlist,omitempty"`) + `UpstreamPool()` = `[base_url] + mirrorlist`.
- `ValidateMirrorlist`: a non-empty mirrorlist is allowed **only** when `repo_type==remote` **and** `package_type ∈ {rpm, deb, alpine}`; each entry must be an http/https URL. Enforced in the v2 create/update handlers (400 otherwise); `base_url` stays required for remotes.
- Persist the mirrorlist in a new additive `mirrorlist TEXT[]` column (`remoteCols`/`scanRemote`/`CreateRemote`/`UpdateRemote`); the `base_url` column is unchanged.
- Engine keeps a per-remote round-robin cursor over the pool; the fetch/head/revalidate upstream calls run in a failover loop that narrows the remote to one selected mirror per attempt. Only network errors and 5xx fail over (404/403/… return as-is). The circuit breaker stays keyed per remote and trips only after all mirrors fail.
## Scope
Round-robin + failover only, restricted to **remote rpm/deb/apk** repos. Least-connections and a per-remote strategy selector are a **follow-up PR**.
## Tests
- Unit: model JSON round-trip + validation gating (rejected on non-rpm/deb/apk and on local, accepted on rpm/deb/apk, bad URL rejected), engine round-robin/failover/no-mirrorlist-unchanged, DB mirrorlist round-trip. `make test` (`go test -race`) green.
- Docker acceptance (`e2e-docker`, `dockere2e` tag, wired into `docker-e2e.sh`): round-robin distribution across two mock upstreams, failover past a dead primary, no-mirrorlist regression, and a **real `dnf` makecache + install** through a two-mirror rpm remote whose `base_url` is dead. All four pass locally.
Reviewed-on: #121
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Adds a black-box e2e suite that runs against the **built container image** via docker-compose (complementing the in-process `e2e/` testcontainers suite).
## What it does
`make docker-e2e` → `scripts/docker-e2e.sh`: builds the image, brings up the full stack (postgres, redis, minio, artifactapi) plus a static nginx **mock upstream** for hermetic caching, waits for `/health`, runs `go test -tags=dockere2e ./e2e-docker/...`, and tears everything down.
## Coverage
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact for **each of the 10 remote package types** (generic, docker, helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/` index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge across two members.
## Notes
- The artifactapi host port is parameterised (`ARTIFACTAPI_PORT`, default `8000`; the e2e run uses `8001`) so it does not collide with a locally-running instance. This is the only change to the production `docker-compose.yml`.
- Fixtures under `e2e-docker/fixtures/` are real package files (incl. a real RPM so repodata parsing works); a `.gitignore` negation tracks them over the global ignore of those extensions.
## Validation
Ran `make docker-e2e` locally: **all suites pass** against the containerised product.
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #97
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben