Merge benvin/remote-query-forwarder into benvin/remote-ansible
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful

This commit is contained in:
2026-10-18 03:16:10 +11:00
190 changed files with 8593 additions and 1107 deletions
+7
View File
@@ -0,0 +1,7 @@
# `github_alpine` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `github_alpine` | `APKINDEX.tar.gz` (synthesized) | `.apk` (redirected) |
+7
View File
@@ -0,0 +1,7 @@
# `alpine` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `alpine` | `APKINDEX.tar.gz` | `.apk` |
+11
View File
@@ -0,0 +1,11 @@
# `ansible` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `ansible` | Galaxy `api/` JSON (links rewritten, `limit`/`offset` forwarded) | `artifacts/*.tar.gz` |
```bash
ansible-galaxy collection install community.general -s https://artifactapi.example/api/v1/remote/galaxy/api/
```
+7
View File
@@ -0,0 +1,7 @@
# `cargo` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `cargo` | sparse index files (`config.json` synthesized) | `crates/*/*.crate` |
+101
View File
@@ -0,0 +1,101 @@
# Caching and classification (all remotes)
Every proxied request on a remote is classified as **denied**, **immutable** or
**mutable**. The class picks the storage path and the freshness TTL.
## Fields
| Field | Meaning |
|---|---|
| `patterns` | Allowlist of regexes. Empty allows every path; otherwise a non-matching path is denied (403). |
| `blocklist` | Regexes; a match is denied (403). |
| `immutable_patterns` | Regexes forcing a path immutable. |
| `mutable_patterns` | Regexes forcing a path mutable. |
| `immutable_ttl` | Seconds an immutable path stays fresh. `0` = forever. |
| `mutable_ttl` | Seconds a mutable path stays fresh. `0` also means never expires. Omitted on create, it defaults to `3600`; omitted on update, the stored value is kept. |
| `check_mutable` | When a mutable path's `mutable_ttl` expires, sends a conditional `HEAD` (`If-None-Match`) with the stored upstream ETag. A `304` re-arms the TTL and serves the stored copy; anything else refetches in full. The ETag is kept for `max(10 × mutable_ttl, 7 days)`, re-armed on each `304`, so idle paths age out. It is only stored when `check_mutable` is set and `mutable_ttl` is non-zero, and never for immutable paths. |
| `stale_on_error` | On an upstream network error, serve the stored copy instead of failing. |
| `quarantine_enabled` / `quarantine_days` | Stored on the remote but not enforced by the proxy yet. |
| `mirrorlist` / `mirror_strategy` | rpm/deb/alpine only. Upstream pool is `[base_url] + mirrorlist`; `round_robin` (default) rotates the starting mirror, `least_conn` starts at the one with the fewest in-flight requests. Network errors and 5xx fail over to the next mirror; other statuses return as-is. |
## Precedence
The first rule that matches wins:
1. `blocklist` matches: **denied**
2. `patterns` is non-empty and nothing matches: **denied**
3. `immutable_patterns` matches: **immutable**
4. `mutable_patterns` matches: **mutable**
5. the provider's built-in `Classify` says mutable: **mutable**
6. anything else: **immutable**
So `blocklist` beats the allowlist, and `immutable_patterns` beats
`mutable_patterns`. Either user list overrides the built-in rules. `GET` and
`HEAD` both apply this order.
## Matching rules
- Regexes are Go `regexp`, matched **unanchored** (`MatchString`). Add `^` or `$` yourself when you need them.
- They match the path after `/api/v1/remote/<name>/`, without a leading slash, e.g. `9/BaseOS/x86_64/os/repodata/repomd.xml`.
- The query string is not part of the path. It is dropped and not forwarded upstream.
- `create` and `update` reject an invalid regex in `patterns`, `blocklist`, `mutable_patterns`, `immutable_patterns` or `ban_tags` with a 400. Only invalid regexes already stored before that check are skipped at match time.
## Lifecycles
| | Immutable | Mutable |
|---|---|---|
| Stored as | content-addressed blob (sha256), streamed | per-path index object, buffered and rewritten by the provider |
| Fresh for | `immutable_ttl` (`0` = forever) | `mutable_ttl` |
| After expiry | refetched from upstream | revalidated with `If-None-Match` when `check_mutable` is set and an ETag is stored, else refetched |
| Upstream down | `stale_on_error` serves the stored copy and re-arms the TTL; an open circuit breaker always serves stale if a copy exists, else 503 | same |
Concurrent misses on one path share a fetch lock. Waiters poll the store for
up to 5s instead of all going upstream.
## Eviction
```sh
curl -X DELETE https://artifactapi.example.com/api/v2/remotes/<name>/objects/<path>
curl -X DELETE https://artifactapi.example.com/api/v2/remotes/<name>/objects/<dir>/*
```
- Evict drops the artifact row, the index object and the Redis TTL and ETag keys, so the next request goes upstream whatever the class or TTL.
- Deleting a remote drops its Redis TTL and ETag keys, so a re-created remote of the same name starts cold.
- A single-path evict takes the fetch lock first, so an in-flight fetch can't re-cache the path. It returns 503 if the lock stays busy.
- A wildcard must end in `<dir>/*` and takes no lock.
- Returns 204, or 404 for an unknown remote.
## Examples
Bodies for `POST https://artifactapi.example.com/api/v2/remotes`.
GitHub release assets through a `generic` remote, allowlisted:
```json
{"name": "github", "package_type": "generic", "base_url": "https://github.com",
"immutable_ttl": 0, "mutable_ttl": 7200,
"patterns": ["^mikefarah/yq/releases/download/.*/yq_linux_amd64$"]}
```
rpm remote: `repodata/*` and `repomd.xml` are mutable by default, so only the TTLs are needed:
```json
{"name": "almalinux", "package_type": "rpm", "base_url": "https://repo.almalinux.org/almalinux",
"immutable_ttl": 0, "mutable_ttl": 3600, "check_mutable": true, "stale_on_error": true}
```
Force a moving path mutable, e.g. branch tarballs on `generic`:
```json
{"mutable_patterns": ["/archive/refs/heads/.*\\.tar\\.gz$"], "mutable_ttl": 600}
```
## Built-in classification per type
Each type's page lists its auto-detected mutable and immutable paths:
[alpine](../alpine/remote.md), [alpine github](../alpine/remote-github.md),
[cargo](../cargo/remote.md), [deb](../deb/remote.md), [deb github](../deb/remote-github.md),
[docker](../docker/remote.md), [generic](../generic/remote.md), [gems](../gems/remote.md), [goproxy](../goproxy/remote.md),
[helm](../helm/remote.md), [npm](../npm/remote.md), [nuget](../nuget/remote.md),
[puppet](../puppet/remote.md), [pypi](../pypi/remote.md), [rpm](../rpm/remote.md),
[rpm github](../rpm/remote-github.md), [terraform](../terraform/remote.md).
+7
View File
@@ -0,0 +1,7 @@
# `github_deb` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `github_deb` | `Packages*`, `Release*` (synthesized) | `.deb` (redirected) |
+7
View File
@@ -0,0 +1,7 @@
# `deb` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `deb` | `Packages*`, `Sources*`, `Release`, `InRelease`, `Release.gpg`, `Contents-*` | `.deb` |
+17
View File
@@ -0,0 +1,17 @@
# Local docker registry
A local `docker` repo is a real container registry, not a mirror: it serves the
Docker Registry HTTP API V2 for both push and pull, so any client (`docker`,
`podman`, `skopeo`, `buildah`) can use it directly.
```sh
docker tag myapp:latest artifactapi.example.com/docker-internal/myapp:latest
docker push artifactapi.example.com/docker-internal/myapp:latest
docker pull artifactapi.example.com/docker-internal/myapp:latest
```
The first path segment after `/v2/` is the artifactapi repo name; the remainder
is the image name. Blobs and manifests are stored through the shared
content-addressable store (deduplicated by digest, reaped by GC once
unreferenced); tags are mutable references and re-pushing a tag moves it. Blob
uploads support both the monolithic and chunked (`POST`/`PATCH`/`PUT`) flows.
+26
View File
@@ -0,0 +1,26 @@
# `docker` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `docker` | tag manifests, `/tags/list` | blobs, digest manifests |
## Terraform
```hcl
resource "artifactapi_remote_docker" "dockerhub" {
name = "dockerhub"
base_url = "https://registry-1.docker.io"
immutable_ttl = 0
mutable_ttl = 300
ban_tags_enabled = true
ban_tags = ["latest"]
patterns = [
"^library/postgres",
"^library/redis",
]
}
```
+24
View File
@@ -0,0 +1,24 @@
# Local gems repository
A local `gems` repo is a RubyGems source: `gem push`, `gem yank`, `gem install`
and `bundle install` work against `/api/v1/local/<repo>`. `gem push` needs an
API key; any value works.
```bash
GEM_HOST_API_KEY=unused gem push --host https://artifactapi.example.com/api/v1/local/<repo> mygem-1.0.0.gem
gem install mygem --source https://artifactapi.example.com/api/v1/local/<repo>/
gem yank mygem -v 1.0.0 --host https://artifactapi.example.com/api/v1/local/<repo>
```
```ruby
# Gemfile
source "https://artifactapi.example.com/api/v1/local/<repo>/"
gem "mygem"
```
The repo serves the compact index (`versions`, `info/<gem>`, `names`), `gems/*.gem`
and `quick/Marshal.4.8/*.gemspec.rz`. The legacy `specs.4.8.gz` indexes are not
served; current gem and bundler clients use the compact index.
Versions are immutable: pushing an existing name-version-platform is rejected
with 409. A yanked version disappears from the index and its files are removed.
+7
View File
@@ -0,0 +1,7 @@
# `gems` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `gems` | compact index (`versions`, `info/*`, `names`), `*specs.4.8.gz` | `gems/*.gem`, `quick/Marshal.4.8/*.gemspec.rz` |
+28
View File
@@ -0,0 +1,28 @@
# `gems` virtual
A gems virtual serves one RubyGems source over local and remote gems members:
`gem install` and `bundle install` work against `/api/v1/virtual/<name>/`.
- `info/<gem>` is the union of member lines; on a duplicate version-platform
the earlier member wins. A gem only one member has is served verbatim.
- `versions` regenerates the line (and md5) of every gem several members list
from its merged `info`; other gems' lines pass through. `names` is the union.
- `gems/*.gem` and `quick/Marshal.4.8/*.gemspec.rz` 302 to the member whose
`info` line won.
- A member error fails the request with 502; a member without the gem is
skipped. Merges are cached for 60s.
```hcl
resource "artifactapi_virtual" "gems" {
name = "gems"
package_type = "gems"
members = [artifactapi_local_gems.internal.name, artifactapi_remote_gems.rubygems.name]
}
```
```ruby
# Gemfile
source "https://artifactapi.example.com/api/v1/virtual/gems/"
```
Push to the local member directly; the virtual is read-only.
+28
View File
@@ -0,0 +1,28 @@
# `generic` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `generic` | nothing | everything |
## Terraform
```hcl
resource "artifactapi_remote_generic" "github" {
name = "github"
base_url = "https://github.com"
immutable_ttl = 0
mutable_ttl = 7200
patterns = [
"ducaale/xh/.*/xh-.*-x86_64-unknown-linux-musl.tar.gz$",
"mikefarah/yq/.*/yq_linux_amd64$",
]
mutable_patterns = [
".*/archive/refs/heads/.*\\.tar\\.gz$",
]
}
```
+7
View File
@@ -0,0 +1,7 @@
# `goproxy` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
+15
View File
@@ -0,0 +1,15 @@
# Local helm repository
A local `helm` repo is a real chart repository: `PUT` a packaged chart to
`/api/v2/remotes/<repo>/files/<chart>-<version>.tgz` and `index.yaml` is
generated from each chart's `Chart.yaml` (with sha256 digests) at
`/api/v1/local/<repo>/index.yaml`; a file not named after its `Chart.yaml`
name and version is stored but left out of the index. Add the local as a
member of a helm virtual to serve it alongside remotes.
```bash
helm package ./mychart
curl -fsSL --upload-file mychart-0.1.0.tgz \
https://artifactapi.example.com/api/v2/remotes/<repo>/files/mychart-0.1.0.tgz
helm repo add <repo> https://artifactapi.example.com/api/v1/local/<repo>
```
+19
View File
@@ -0,0 +1,19 @@
# `helm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `helm` | `index.yaml` | `.tgz` charts |
## Terraform
```hcl
resource "artifactapi_remote_helm" "jetstack" {
name = "jetstack"
base_url = "https://charts.jetstack.io"
immutable_ttl = 0
mutable_ttl = 3600
}
```
+9
View File
@@ -0,0 +1,9 @@
# `helm` virtual
```hcl
resource "artifactapi_virtual" "helm" {
name = "helm"
package_type = "helm"
members = [artifactapi_remote_helm.jetstack.name]
}
```
+14
View File
@@ -0,0 +1,14 @@
# Local npm registry
A local `npm` repo is a real npm registry: `npm publish` (scoped and unscoped)
and `npm install` work against `/api/v1/local/<repo>/`. npm requires an
`_authToken` for the registry before it will publish; any value works.
```ini
# .npmrc
registry=https://artifactapi.example.com/api/v1/local/<repo>/
//artifactapi.example.com/api/v1/local/<repo>/:_authToken=unused
```
Versions are immutable: republishing an existing version is rejected. The
dist-tags sent with a publish (`latest`, or `--tag`) move to that version.
+7
View File
@@ -0,0 +1,7 @@
# `npm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `npm` | package metadata | `.tgz` tarballs |
+30
View File
@@ -0,0 +1,30 @@
# `npm` virtual
An npm virtual serves one registry over local and remote npm members:
`npm install` and `npm view` work against `/api/v1/virtual/<name>/`.
- Versions are the union across members; on a duplicate version the earlier
member wins.
- Each dist-tag comes from the first member defining it, and is dropped if it
names a version missing from the merged set.
- Tarball URLs point at the virtual (`<member>/<path>`), which 302s to the
owning member.
- Full and abbreviated (`application/vnd.npm.install-v1+json`) packuments are
both served.
- A member error fails the request with 502; a member without the package is
skipped. Merges are cached for 60s per package.
```hcl
resource "artifactapi_virtual" "npm" {
name = "npm"
package_type = "npm"
members = [artifactapi_local_npm.internal.name, artifactapi_remote_npm.npmjs.name]
}
```
```ini
# .npmrc
registry=https://artifactapi.example.com/api/v1/virtual/npm/
```
Publish to the local member directly; the virtual is read-only.
+7
View File
@@ -0,0 +1,7 @@
# `nuget` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `nuget` | `index.json` (service index), registration and flat container indexes | `.nupkg`, `.snupkg`, `.nuspec`, `readme`, `icon` |
+14
View File
@@ -0,0 +1,14 @@
# Local puppet forge
A local `puppet` repo is a Forge: `PUT` a module tarball to
`/api/v2/remotes/<repo>/files/<owner>-<module>-<version>.tar.gz` and the Forge
v3 API (`/v3/modules`, `/v3/releases`, `/v3/files`) is served from each
release's `metadata.json` at `/api/v1/local/<repo>`. Versions are immutable.
```bash
puppet module build ./hello
curl -fsSL --upload-file pkg/unkin-hello-0.1.0.tar.gz \
https://artifactapi.example.com/api/v2/remotes/<repo>/files/unkin-hello-0.1.0.tar.gz
puppet module install unkin-hello \
--module_repository https://artifactapi.example.com/api/v1/local/<repo>
```
+7
View File
@@ -0,0 +1,7 @@
# `puppet` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `puppet` | `v3/modules/*`, `v3/releases*` | `.tar.gz` |
+7
View File
@@ -0,0 +1,7 @@
# `pypi` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `pypi` | `simple/*` index pages | `.whl`, `.tar.gz` |
+82
View File
@@ -0,0 +1,82 @@
# `github_rpm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
## GitHub releases as a yum repo (metadata-only, no precache)
A `github_rpm` remote turns a GitHub repo's **releases** into a real `dnf`/`yum`
repository without ever caching the packages. It scans releases for `.rpm`
assets, derives each package's metadata (NEVRA, requires/provides/conflicts/
obsoletes, files, checksum) and **synthesizes `repodata/` on the fly**. Package
metadata comes from a **ranged GET of just the RPM header** (the header sits at
the front of the file, so the whole package is never downloaded); the sha256
checksum comes from the GitHub asset `digest` when present, else a one-time
lazy stream. Derived metadata is cached (keyed by asset) so repodata generation
is served from primed DB rows, never a cold on-demand derive.
Each package's `<location>` points back at the remote, which **302-redirects**
the download to the `releases_remote` — an existing generic `github.com` remote
that streams the actual bytes. `dnf` follows the redirect transparently.
### Background syncer
A single process-wide **background syncer** keeps every `github_rpm` remote's
derived metadata current off the client request path:
- **Prime on create.** Creating a `github_rpm` remote enqueues a background prime
scan, so its metadata is derived right away without blocking the create call.
The first `dnf` request is served from cache. If a request arrives before the
prime lands, it returns a retryable `503` (with `Retry-After`) rather than
serving an empty repo or blocking on a multi-minute derive.
- **Periodic re-check, driven by `mutable_ttl`.** Each remote is re-checked for
new or changed releases no more often than its `mutable_ttl`. New/changed
assets are derived incrementally; assets already cached are never re-fetched,
and assets that disappear upstream are pruned.
- **ETag / 304 conditional requests.** The releases-list `ETag` is stored per
remote and sent as `If-None-Match`; a `304 Not Modified` means nothing changed
and the syncer derives nothing. GitHub does not count `304` conditional
responses against the rate limit, so an unchanged repo is nearly free — this is
the main lever keeping GitHub traffic low.
- **Global rate limit.** Every GitHub call (releases list + each ranged asset
header GET) passes through a single token-bucket limiter **shared across all
remotes**, so GitHub is never hammered. Configure a token (`password`) on the
remote for the higher authenticated rate limit (~5000/hr vs ~60/hr
unauthenticated).
- **Multi-replica coordination.** State is shared through the database. Before a
periodic scan a replica must atomically claim a per-remote lease
(`github_rpm_sync_state`: `last_synced_at`, `etag`, `sync_lease_owner`,
`sync_lease_expires`); only the winner scans. This bounds total GitHub load to
~once per `mutable_ttl` regardless of replica count, and the shared `etag`
lets any replica issue the conditional request.
```hcl
# Backend that serves the actual .rpm bytes from github.com.
resource "artifactapi_remote_generic" "github" {
name = "github"
base_url = "https://github.com"
patterns = [
"acme/tools/releases/download/.*\\.rpm$", # allowlist the repo's release assets
]
}
resource "artifactapi_remote_github_rpm" "acme-tools" {
name = "acme-tools"
base_url = "https://api.github.com/repos/acme/tools" # the releases API root
releases_remote = "github" # backend for downloads
mutable_ttl = 3600 # release re-scan interval
# Optional: restrict which release assets become packages (regex on filename).
patterns = [".*\\.x86_64\\.rpm$", ".*\\.noarch\\.rpm$"]
# Optional: a token for private repos / higher API rate limits.
# password = "ghp_..."
}
```
`dnf` config: `baseurl=https://artifactapi.example/api/v1/remote/acme-tools`.
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
from the synthesized metadata.
+7
View File
@@ -0,0 +1,7 @@
# `rpm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `rpm` | `repomd.xml`, `repodata/*` | `.rpm` |
+29
View File
@@ -0,0 +1,29 @@
# Serving providers as a registry
A local `terraform` repo is a real provider registry: upload
`terraform-provider-{type}_{version}_{os}_{arch}.zip` files under
`{namespace}/{type}/`, and Terraform installs them from a bare source address —
no `.terraformrc` mirror config:
```hcl
terraform {
required_providers {
artifactapi = {
source = "artifactapi.example.com/<repo>/<type>"
version = "0.1.2"
}
}
}
```
The Terraform *namespace* segment is the artifactapi repo name; the provider is
matched by *type*. The registry serves service discovery
(`/.well-known/terraform.json`), the `providers.v1` version/download endpoints,
and a GPG-signed `SHA256SUMS` per the provider registry protocol.
Signing needs a GPG key. By default artifactapi generates one on first start and
stores it in the database (`signing_keys` table), so every replica shares it and
there's nothing to provision. To bring your own key instead, point
`TF_SIGNING_KEY_PATH` at an armored private key (optionally
`TF_SIGNING_KEY_PASSPHRASE`), which takes precedence over the generated one.
`TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
+7
View File
@@ -0,0 +1,7 @@
# `terraform` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `terraform` | `*/versions` | `*/download/*/*` |