Compare commits

..

13 Commits

Author SHA1 Message Date
unkin-agent 2c671a1be7 Add npm virtual repositories (#152)
ci/woodpecker/tag/docker Pipeline was successful
npm clients need one registry for internal and public packages, but npm virtuals have no merger, so a local npm repo and npmjs cannot be served together.

- merge packuments across members: union versions (member order wins), first-defined dist-tags minus ones naming missing versions
- serve full and abbreviated (`install-v1`) packuments, scoped names included, via the shared virtual merge cache
- point tarballs at `<virtual>/<member>/<path>` and 302 them to the owning member
- reject an invalid Host with 400; serve virtual and local packuments `no-store` with `Vary: Host, X-Forwarded-Proto, Accept`
- return 502 when a member errors; skip members without the package
- add `docs/npm/virtual.md` and a stock `node:22` e2e-docker case

Reviewed-on: #152
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 21:52:55 +11:00
unkin-agent bf8228e9f2 Harden nuget per-request index rewrite (#149)
NuGet index rewrites embed the client Host and X-Forwarded-Proto into per-request JSON, so a shared cache could store one host's URLs, a forged header could inject content, HEAD skipped the rewrite, and immutable flat-container blobs were re-validated as mutable.

- set `no-store`, `Vary: Host, X-Forwarded-Proto` and the rewritten Content-Length on GET and HEAD
- share `v1.Scheme()` (first X-Forwarded-Proto value, `http`/`https` only) across nuget, cargo, virtual redirects and the terraform registry
- reject non host[:port] Host headers and JSON-escape the injected base URL
- classify `.snupkg`, `readme` and `icon` flat-container blobs as immutable

requires unkin/artifactapi#148 merged first

Reviewed-on: #149
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 21:39:42 +11:00
unkin-agent ad72c3d630 Add HEAD route for remotes and gems remote type (#148)
Remotes answer only GET, so HEAD probes get 405, and there is no RubyGems remote. Synthesized (github_*) and rewritten (npm, helm, pypi, puppet, terraform, nuget) mutable responses need the GET path for HEAD lengths to match.

- add a HEAD route for /api/v1/remote/{name}/*; rewritten and synthesized mutable HEADs go through the GET path
- mark rewriting providers with a ResponseRewriter capability; set Content-Length on proxied GETs
- use the remote's upstream client for upstream HEAD
- add a gems remote type: compact index and specs mutable, .gem and gemspec immutable
- add gems docs, UI snippet and e2e caching case; add HEAD-vs-GET e2e checks

Reviewed-on: #148
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 21:23:29 +11:00
unkin-agent 42998a80f4 Add local npm registry publish and install (#139)
In-house npm packages have nowhere to publish: npm locals only accept raw file uploads, which `npm publish` and `npm install` cannot use.

- accept `npm publish` (`PUT /api/v1/local/<repo>/<pkg>`, scoped included) on local npm repos, capping the body at 256MiB and verifying the tarball against dist integrity
- write tarball, then manifest, then dist-tags; publishing an existing version returns 409 and leaves tags alone, while a matching leftover tarball from a failed publish is reused so retries complete
- serve packuments with tarball URLs pointing back at the local repo, dropping dist-tags that name unpublished versions
- document the `.npmrc` registry + `_authToken` setup in the UI and `docs/npm/local.md`
- add a stock `node:22` publish/install e2e-docker case

Reviewed-on: #139
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 17:18:05 +11:00
unkin-agent d69a58e54a Fix mutable path refresh: default mutable_ttl, revalidate after expiry (#151)
A remote created through the API without `mutable_ttl` was stored as 0, so its mutable paths never refreshed. With `check_mutable`, the ETag expired together with the freshness key, so an expired mutable path was always refetched in full and never revalidated.

- default an omitted `mutable_ttl` to 3600 on create; keep the stored value on update; explicit 0 still means never expire
- keep mutable ETags for `max(10 × mutable_ttl, 7d)`, re-armed on each 304, only when `check_mutable` is set and `mutable_ttl` is non-zero
- drop the stored ETag when a refetch returns none; never store one for immutable paths
- flush a remote's Redis keys on delete
- update docs/common/caching.md

Reviewed-on: #151
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 17:17:41 +11:00
unkin-agent 6e6481e09a Split per-type registration points into per-type files (#150)
Parallel package-type PRs conflict because every type edits the same shared lines (README tables, provider.go, server.go imports, the package type list, UI maps, the e2e caching table). A new type now only adds files.

- move per-type README sections into docs/<type>/{remote,local,virtual}.md
- split provider.go capability and metadata types into cap_*.go and metadata_*.go
- import providers via internal/provider/all, one file per type
- declare each package type in package_type_<type>.go, registered via init()
- split the e2e caching cases and UI colours/snippets into per-type files

Reviewed-on: #150
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 15:54:47 +11:00
unkin-agent aaa923ffd1 Add helm local repositories (#140)
ci/woodpecker/tag/docker Pipeline was successful
In-house charts have nowhere to publish: a helm local stores uploads but serves no index.yaml, and the helm virtual merger drops every chart field except name/version/urls.

- add helm local uploads (`PUT .../files/<name>-<version>.tgz`) recording each Chart.yaml in a new `helm_metadata` table; mismatched filenames stay out of the index
- bound chart parsing (50MiB decompressed, 10k entries, 1MiB Chart.yaml)
- generate a deterministic `index.yaml` with sha256 digests and path-escaped relative urls
- keep all chart fields when merging helm virtual indexes, expanding YAML aliases
- add a real-helm e2e-docker test, UI snippet and README section

Reviewed-on: #140
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 15:14:36 +11:00
unkin-agent 0150b5ebb4 Share the virtual merge cache across index types (#147)
The rpm virtual's merge cache, singleflight and two-generation retention are hard-wired to RPMRepo, so other merged index types (deb next) can't reuse them.

- add a mergedRepo interface (index + referenced files) implemented by RPMRepo
- key the merge cache by type-scoped string and serve files from both retained generations via cachedFile
- extract parallel fail-closed member fetching into a generic fetchMembers
- add tests for generation retention and per-key isolation

Reviewed-on: #147
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 13:28:15 +11:00
unkin-agent 9b91aa39a6 Add local puppet forge (#145)
Puppet modules can only be proxied from a remote Forge; there is nowhere to publish internal modules for puppet module install, r10k and g10k.

- add puppet local uploads at <owner>-<module>-<version>.tar.gz, rejecting existing versions
- record each release's metadata.json, md5 and sha256 in a puppet_metadata table, streaming the tarball under size caps
- serve Forge v3 /v3/modules, paginated /v3/releases, /v3/releases/<slug> and /v3/files
- add an e2e-docker case driving real r10k and puppet module install
- document local puppet in README and UI usage snippets

Reviewed-on: #145
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 13:17:51 +11:00
unkin-agent bc68eb75fc Add nuget v3 remote type (#141)
.NET builds restore from nuget.org directly because artifactapi has no NuGet remote type.

- add `nuget` package type proxying a NuGet v3 feed; `index.json` maps to the upstream service index
- add a `ServeRewriter` provider hook so cached JSON has upstream-origin URLs pointed back at the requesting host per request
- treat `.nupkg`/`.nuspec` as immutable and service, registration and flat container indexes as mutable
- add a `nuget.config` usage snippet to the UI and a nuget e2e caching case

Reviewed-on: #141
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 12:19:51 +11:00
unkin-agent 9161558334 Add optional OIDC bearer auth on remote/virtual config writes (#136)
ci/woodpecker/tag/docker Pipeline was successful
Remote and virtual configuration writes are unauthenticated, so anything that can reach the API can repoint or delete repos. Vault identity OIDC tokens give callers a verifiable identity.

- add `ARTIFACTAPI_OIDC_ISSUER`/`ARTIFACTAPI_OIDC_AUDIENCE`; unset issuer keeps auth off
- require a JWKS-verified bearer token on POST/PUT/DELETE of `/api/v2/remotes` and `/api/v2/virtuals`, 401 otherwise
- leave uploads, object deletes, probe, reads and `/v2` open
- discover the issuer lazily with a 10s client timeout; fail fast with 503 for 15s after a failed discovery
- log the verified `sub`; allow `Authorization` in CORS

Reviewed-on: #136
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-16 22:08:20 +11:00
unkin-agent 492607a164 Retry failed GitHub release scans with backoff (#133)
ci/woodpecker/tag/docker Pipeline was successful
Releasing a GitHub sync lease always advanced `last_synced_at`, even after a failed scan (e.g. a rate-limit 403). A remote that failed once waited a full `mutable_ttl` before retrying, and a cold remote kept returning 503 until then.

- record scan outcomes in one shared lease helper for github_rpm/deb/alpine
- keep `last_synced_at` and the ETag on failure; retry from 60s with exponential backoff, capped at min(10m, ttl/4)
- honour `Retry-After` / `X-RateLimit-Reset`, clamped to `mutable_ttl`
- add `sync_failures` / `next_retry_at` columns (migration 0002)

Reviewed-on: #133
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-09 23:15:55 +11:00
unkin-agent 1780b3d77c Push release images to artifactapi docker-internal (#134)
Gitea's registry rejects blob HEADs with 401, so the v3.13.0 tag pipeline cannot push. Release images move to the in-cluster artifactapi docker-internal registry, which trusts the runner without credentials.

- push docker-api to docker-internal/artifactapi and docker-web to docker-internal/artifactapi-ui
- use the CA-baked docker-internal/plugin-docker-buildx image
- drop the droneci credentials
- set kubernetes resources and service account on both steps

Reviewed-on: #134
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-09 23:08:22 +11:00
189 changed files with 7197 additions and 1103 deletions
+26 -12
View File
@@ -4,31 +4,45 @@ when:
steps:
- name: docker-api
image: woodpeckerci/plugin-docker-buildx
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx:latest
settings:
registry: git.unkin.net
repo: git.unkin.net/unkin/artifactapi
registry: artifactapi.k8s.syd1.au.unkin.net
repo: artifactapi.k8s.syd1.au.unkin.net/docker-internal/artifactapi
build_args:
VERSION: ${CI_COMMIT_TAG}
username: droneci
password:
from_secret: DRONECI_PASSWORD
tags:
- ${CI_COMMIT_TAG}
- latest
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 1Gi
cpu: 1
limits:
memory: 4Gi
cpu: 2
- name: docker-web
image: woodpeckerci/plugin-docker-buildx
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx:latest
settings:
registry: git.unkin.net
repo: git.unkin.net/unkin/artifactapi-ui
registry: artifactapi.k8s.syd1.au.unkin.net
repo: artifactapi.k8s.syd1.au.unkin.net/docker-internal/artifactapi-ui
dockerfile: ui/Dockerfile.ui
context: ui
build_args:
BASE_PATH: /ui
username: droneci
password:
from_secret: DRONECI_PASSWORD
tags:
- ${CI_COMMIT_TAG}
- latest
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 1Gi
cpu: 1
limits:
memory: 4Gi
cpu: 2
+1 -1
View File
@@ -672,7 +672,7 @@ func TestMain(m *testing.M) {
```hcl
provider "artifactapi" {
endpoint = "https://artifactapi.k8s.syd1.au.unkin.net"
endpoint = "https://artifactapi.example.com"
}
resource "artifactapi_remote" "terraform_registry" {
+22 -186
View File
@@ -1,6 +1,6 @@
# ArtifactAPI
Caching proxy for package repositories. Single Go binary, 11 package types, content-addressable storage, managed by Terraform.
Caching proxy for package repositories. Single Go binary, 12 package types, content-addressable storage, managed by Terraform.
## Quick Start
@@ -20,97 +20,26 @@ API: `http://localhost:8000` | Frontend: `http://localhost:5173`
## Package Types
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `generic` | nothing | everything |
| `docker` | tag manifests, `/tags/list` | blobs, digest manifests |
| `helm` | `index.yaml` | `.tgz` charts |
| `pypi` | `simple/*` index pages | `.whl`, `.tar.gz` |
| `npm` | package metadata | `.tgz` tarballs |
| `rpm` | `repomd.xml`, `repodata/*` | `.rpm` |
| `alpine` | `APKINDEX.tar.gz` | `.apk` |
| `puppet` | `v3/modules/*`, `v3/releases*` | `.tar.gz` |
| `terraform` | `*/versions` | `*/download/*/*` |
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
| `cargo` | sparse index files (`config.json` synthesized) | `crates/*/*.crate` |
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
Providers classify paths automatically. Users only configure what to proxy and TTLs.
### `github_rpm` — GitHub releases as a yum repo (metadata-only, no precache)
Classification precedence, TTLs and eviction for every remote: [docs/common/caching.md](docs/common/caching.md).
A `github_rpm` remote turns a GitHub repo's **releases** into a real `dnf`/`yum`
repository without ever caching the packages. It scans releases for `.rpm`
assets, derives each package's metadata (NEVRA, requires/provides/conflicts/
obsoletes, files, checksum) and **synthesizes `repodata/` on the fly**. Package
metadata comes from a **ranged GET of just the RPM header** (the header sits at
the front of the file, so the whole package is never downloaded); the sha256
checksum comes from the GitHub asset `digest` when present, else a one-time
lazy stream. Derived metadata is cached (keyed by asset) so repodata generation
is served from primed DB rows, never a cold on-demand derive.
Each type's paths, Terraform examples and local/virtual behaviour are in `docs/<type>/`:
Each package's `<location>` points back at the remote, which **302-redirects**
the download to the `releases_remote` — an existing generic `github.com` remote
that streams the actual bytes. `dnf` follows the redirect transparently.
#### Background syncer
A single process-wide **background syncer** keeps every `github_rpm` remote's
derived metadata current off the client request path:
- **Prime on create.** Creating a `github_rpm` remote enqueues a background prime
scan, so its metadata is derived right away without blocking the create call.
The first `dnf` request is served from cache. If a request arrives before the
prime lands, it returns a retryable `503` (with `Retry-After`) rather than
serving an empty repo or blocking on a multi-minute derive.
- **Periodic re-check, driven by `mutable_ttl`.** Each remote is re-checked for
new or changed releases no more often than its `mutable_ttl`. New/changed
assets are derived incrementally; assets already cached are never re-fetched,
and assets that disappear upstream are pruned.
- **ETag / 304 conditional requests.** The releases-list `ETag` is stored per
remote and sent as `If-None-Match`; a `304 Not Modified` means nothing changed
and the syncer derives nothing. GitHub does not count `304` conditional
responses against the rate limit, so an unchanged repo is nearly free — this is
the main lever keeping GitHub traffic low.
- **Global rate limit.** Every GitHub call (releases list + each ranged asset
header GET) passes through a single token-bucket limiter **shared across all
remotes**, so GitHub is never hammered. Configure a token (`password`) on the
remote for the higher authenticated rate limit (~5000/hr vs ~60/hr
unauthenticated).
- **Multi-replica coordination.** State is shared through the database. Before a
periodic scan a replica must atomically claim a per-remote lease
(`github_rpm_sync_state`: `last_synced_at`, `etag`, `sync_lease_owner`,
`sync_lease_expires`); only the winner scans. This bounds total GitHub load to
~once per `mutable_ttl` regardless of replica count, and the shared `etag`
lets any replica issue the conditional request.
```hcl
# Backend that serves the actual .rpm bytes from github.com.
resource "artifactapi_remote_generic" "github" {
name = "github"
base_url = "https://github.com"
patterns = [
"acme/tools/releases/download/.*\\.rpm$", # allowlist the repo's release assets
]
}
resource "artifactapi_remote_github_rpm" "acme-tools" {
name = "acme-tools"
base_url = "https://api.github.com/repos/acme/tools" # the releases API root
releases_remote = "github" # backend for downloads
mutable_ttl = 3600 # release re-scan interval
# Optional: restrict which release assets become packages (regex on filename).
patterns = [".*\\.x86_64\\.rpm$", ".*\\.noarch\\.rpm$"]
# Optional: a token for private repos / higher API rate limits.
# password = "ghp_..."
}
```
`dnf` config: `baseurl=https://artifactapi.example/api/v1/remote/acme-tools`.
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
from the synthesized metadata.
- `alpine`: [remote](docs/alpine/remote.md), [remote-github](docs/alpine/remote-github.md)
- `cargo`: [remote](docs/cargo/remote.md)
- `deb`: [remote](docs/deb/remote.md), [remote-github](docs/deb/remote-github.md)
- `docker`: [remote](docs/docker/remote.md), [local](docs/docker/local.md)
- `generic`: [remote](docs/generic/remote.md)
- `gems`: [remote](docs/gems/remote.md)
- `goproxy`: [remote](docs/goproxy/remote.md)
- `helm`: [remote](docs/helm/remote.md), [local](docs/helm/local.md), [virtual](docs/helm/virtual.md)
- `npm`: [remote](docs/npm/remote.md), [local](docs/npm/local.md), [virtual](docs/npm/virtual.md)
- `nuget`: [remote](docs/nuget/remote.md)
- `puppet`: [remote](docs/puppet/remote.md), [local](docs/puppet/local.md)
- `pypi`: [remote](docs/pypi/remote.md)
- `rpm`: [remote](docs/rpm/remote.md), [remote-github](docs/rpm/remote-github.md)
- `terraform`: [remote](docs/terraform/remote.md), [local](docs/terraform/local.md)
### GitHub authentication
@@ -179,105 +108,9 @@ GITHUB_APP_PRIVATE_KEY_PATH=/etc/artifactapi/github-app.pem
## Terraform
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
Remotes and virtuals are managed by Terraform. Each package type has its own resource; see `docs/<type>/` for examples.
```hcl
resource "artifactapi_remote_generic" "github" {
name = "github"
base_url = "https://github.com"
immutable_ttl = 0
mutable_ttl = 7200
patterns = [
"ducaale/xh/.*/xh-.*-x86_64-unknown-linux-musl.tar.gz$",
"mikefarah/yq/.*/yq_linux_amd64$",
]
mutable_patterns = [
".*/archive/refs/heads/.*\\.tar\\.gz$",
]
}
resource "artifactapi_remote_docker" "dockerhub" {
name = "dockerhub"
base_url = "https://registry-1.docker.io"
immutable_ttl = 0
mutable_ttl = 300
ban_tags_enabled = true
ban_tags = ["latest"]
patterns = [
"^library/postgres",
"^library/redis",
]
}
resource "artifactapi_remote_helm" "jetstack" {
name = "jetstack"
base_url = "https://charts.jetstack.io"
immutable_ttl = 0
mutable_ttl = 3600
}
resource "artifactapi_virtual" "helm" {
name = "helm"
package_type = "helm"
members = [artifactapi_remote_helm.jetstack.name]
}
```
Provider: [terraform-provider-artifactapi](../terraform-provider-artifactapi)
### Serving providers as a registry
A local `terraform` repo is a real provider registry: upload
`terraform-provider-{type}_{version}_{os}_{arch}.zip` files under
`{namespace}/{type}/`, and Terraform installs them from a bare source address —
no `.terraformrc` mirror config:
```hcl
terraform {
required_providers {
artifactapi = {
source = "artifactapi.k8s.syd1.au.unkin.net/<repo>/<type>"
version = "0.1.2"
}
}
}
```
The Terraform *namespace* segment is the artifactapi repo name; the provider is
matched by *type*. The registry serves service discovery
(`/.well-known/terraform.json`), the `providers.v1` version/download endpoints,
and a GPG-signed `SHA256SUMS` per the provider registry protocol.
Signing needs a GPG key. By default artifactapi generates one on first start and
stores it in the database (`signing_keys` table), so every replica shares it and
there's nothing to provision. To bring your own key instead, point
`TF_SIGNING_KEY_PATH` at an armored private key (optionally
`TF_SIGNING_KEY_PASSPHRASE`), which takes precedence over the generated one.
`TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
### Local docker registry
A local `docker` repo is a real container registry, not a mirror: it serves the
Docker Registry HTTP API V2 for both push and pull, so any client (`docker`,
`podman`, `skopeo`, `buildah`) can use it directly.
```sh
docker tag myapp:latest artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
docker push artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
docker pull artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
```
The first path segment after `/v2/` is the artifactapi repo name; the remainder
is the image name. Blobs and manifests are stored through the shared
content-addressable store (deduplicated by digest, reaped by GC once
unreferenced); tags are mutable references and re-pushing a tag moves it. Blob
uploads support both the monolithic and chunked (`POST`/`PATCH`/`PUT`) flows.
Provider: [terraform-provider-artifactapi](https://git.unkin.net/unkin/terraform-provider-artifactapi)
## Access Control
@@ -297,6 +130,7 @@ No patterns + no blocklist = open proxy. Provider handles mutability classificat
```
GET /api/v1/remote/{name}/{path} Proxy/cache artifact
GET /api/v1/virtual/{name}/{path} Virtual repo (merged index)
PUT /api/v1/local/{name}/{package} npm publish (local npm repos)
GET /v2/{name}/{path} Docker Registry v2
```
@@ -371,6 +205,8 @@ additive and idempotent for the same reason; a test enforces it.
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
| `ARTIFACTAPI_OIDC_ISSUER` | | OIDC issuer URL (e.g. Vault `https://vault.service.consul:8200/v1/identity/oidc`). When set, `POST /api/v2/remotes`, `POST /api/v2/virtuals` and `PUT`/`DELETE` on `/api/v2/{remotes,virtuals}/{name}` require `Authorization: Bearer <jwt>` verified against the issuer's JWKS. Uploads, object/file deletes, probe, reads and `/v2` stay open. Unset = no auth |
| `ARTIFACTAPI_OIDC_AUDIENCE` | | Comma-separated allowed `aud` values (e.g. the Vault OIDC role's client_id). Required when `ARTIFACTAPI_OIDC_ISSUER` is set |
## Development
+7
View File
@@ -0,0 +1,7 @@
# `github_alpine` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `github_alpine` | `APKINDEX.tar.gz` (synthesized) | `.apk` (redirected) |
+7
View File
@@ -0,0 +1,7 @@
# `alpine` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `alpine` | `APKINDEX.tar.gz` | `.apk` |
+7
View File
@@ -0,0 +1,7 @@
# `cargo` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `cargo` | sparse index files (`config.json` synthesized) | `crates/*/*.crate` |
+101
View File
@@ -0,0 +1,101 @@
# Caching and classification (all remotes)
Every proxied request on a remote is classified as **denied**, **immutable** or
**mutable**. The class picks the storage path and the freshness TTL.
## Fields
| Field | Meaning |
|---|---|
| `patterns` | Allowlist of regexes. Empty allows every path; otherwise a non-matching path is denied (403). |
| `blocklist` | Regexes; a match is denied (403). |
| `immutable_patterns` | Regexes forcing a path immutable. |
| `mutable_patterns` | Regexes forcing a path mutable. |
| `immutable_ttl` | Seconds an immutable path stays fresh. `0` = forever. |
| `mutable_ttl` | Seconds a mutable path stays fresh. `0` also means never expires. Omitted on create, it defaults to `3600`; omitted on update, the stored value is kept. |
| `check_mutable` | When a mutable path's `mutable_ttl` expires, sends a conditional `HEAD` (`If-None-Match`) with the stored upstream ETag. A `304` re-arms the TTL and serves the stored copy; anything else refetches in full. The ETag is kept for `max(10 × mutable_ttl, 7 days)`, re-armed on each `304`, so idle paths age out. It is only stored when `check_mutable` is set and `mutable_ttl` is non-zero, and never for immutable paths. |
| `stale_on_error` | On an upstream network error, serve the stored copy instead of failing. |
| `quarantine_enabled` / `quarantine_days` | Stored on the remote but not enforced by the proxy yet. |
| `mirrorlist` / `mirror_strategy` | rpm/deb/alpine only. Upstream pool is `[base_url] + mirrorlist`; `round_robin` (default) rotates the starting mirror, `least_conn` starts at the one with the fewest in-flight requests. Network errors and 5xx fail over to the next mirror; other statuses return as-is. |
## Precedence
The first rule that matches wins:
1. `blocklist` matches: **denied**
2. `patterns` is non-empty and nothing matches: **denied**
3. `immutable_patterns` matches: **immutable**
4. `mutable_patterns` matches: **mutable**
5. the provider's built-in `Classify` says mutable: **mutable**
6. anything else: **immutable**
So `blocklist` beats the allowlist, and `immutable_patterns` beats
`mutable_patterns`. Either user list overrides the built-in rules. `GET` and
`HEAD` both apply this order.
## Matching rules
- Regexes are Go `regexp`, matched **unanchored** (`MatchString`). Add `^` or `$` yourself when you need them.
- They match the path after `/api/v1/remote/<name>/`, without a leading slash, e.g. `9/BaseOS/x86_64/os/repodata/repomd.xml`.
- The query string is not part of the path. It is dropped and not forwarded upstream.
- `create` and `update` reject an invalid regex in `patterns`, `blocklist`, `mutable_patterns`, `immutable_patterns` or `ban_tags` with a 400. Only invalid regexes already stored before that check are skipped at match time.
## Lifecycles
| | Immutable | Mutable |
|---|---|---|
| Stored as | content-addressed blob (sha256), streamed | per-path index object, buffered and rewritten by the provider |
| Fresh for | `immutable_ttl` (`0` = forever) | `mutable_ttl` |
| After expiry | refetched from upstream | revalidated with `If-None-Match` when `check_mutable` is set and an ETag is stored, else refetched |
| Upstream down | `stale_on_error` serves the stored copy and re-arms the TTL; an open circuit breaker always serves stale if a copy exists, else 503 | same |
Concurrent misses on one path share a fetch lock. Waiters poll the store for
up to 5s instead of all going upstream.
## Eviction
```sh
curl -X DELETE https://artifactapi.example.com/api/v2/remotes/<name>/objects/<path>
curl -X DELETE https://artifactapi.example.com/api/v2/remotes/<name>/objects/<dir>/*
```
- Evict drops the artifact row, the index object and the Redis TTL and ETag keys, so the next request goes upstream whatever the class or TTL.
- Deleting a remote drops its Redis TTL and ETag keys, so a re-created remote of the same name starts cold.
- A single-path evict takes the fetch lock first, so an in-flight fetch can't re-cache the path. It returns 503 if the lock stays busy.
- A wildcard must end in `<dir>/*` and takes no lock.
- Returns 204, or 404 for an unknown remote.
## Examples
Bodies for `POST https://artifactapi.example.com/api/v2/remotes`.
GitHub release assets through a `generic` remote, allowlisted:
```json
{"name": "github", "package_type": "generic", "base_url": "https://github.com",
"immutable_ttl": 0, "mutable_ttl": 7200,
"patterns": ["^mikefarah/yq/releases/download/.*/yq_linux_amd64$"]}
```
rpm remote: `repodata/*` and `repomd.xml` are mutable by default, so only the TTLs are needed:
```json
{"name": "almalinux", "package_type": "rpm", "base_url": "https://repo.almalinux.org/almalinux",
"immutable_ttl": 0, "mutable_ttl": 3600, "check_mutable": true, "stale_on_error": true}
```
Force a moving path mutable, e.g. branch tarballs on `generic`:
```json
{"mutable_patterns": ["/archive/refs/heads/.*\\.tar\\.gz$"], "mutable_ttl": 600}
```
## Built-in classification per type
Each type's page lists its auto-detected mutable and immutable paths:
[alpine](../alpine/remote.md), [alpine github](../alpine/remote-github.md),
[cargo](../cargo/remote.md), [deb](../deb/remote.md), [deb github](../deb/remote-github.md),
[docker](../docker/remote.md), [generic](../generic/remote.md), [gems](../gems/remote.md), [goproxy](../goproxy/remote.md),
[helm](../helm/remote.md), [npm](../npm/remote.md), [nuget](../nuget/remote.md),
[puppet](../puppet/remote.md), [pypi](../pypi/remote.md), [rpm](../rpm/remote.md),
[rpm github](../rpm/remote-github.md), [terraform](../terraform/remote.md).
+7
View File
@@ -0,0 +1,7 @@
# `github_deb` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `github_deb` | `Packages*`, `Release*` (synthesized) | `.deb` (redirected) |
+7
View File
@@ -0,0 +1,7 @@
# `deb` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `deb` | `Packages*`, `Sources*`, `Release`, `InRelease`, `Release.gpg`, `Contents-*` | `.deb` |
+17
View File
@@ -0,0 +1,17 @@
# Local docker registry
A local `docker` repo is a real container registry, not a mirror: it serves the
Docker Registry HTTP API V2 for both push and pull, so any client (`docker`,
`podman`, `skopeo`, `buildah`) can use it directly.
```sh
docker tag myapp:latest artifactapi.example.com/docker-internal/myapp:latest
docker push artifactapi.example.com/docker-internal/myapp:latest
docker pull artifactapi.example.com/docker-internal/myapp:latest
```
The first path segment after `/v2/` is the artifactapi repo name; the remainder
is the image name. Blobs and manifests are stored through the shared
content-addressable store (deduplicated by digest, reaped by GC once
unreferenced); tags are mutable references and re-pushing a tag moves it. Blob
uploads support both the monolithic and chunked (`POST`/`PATCH`/`PUT`) flows.
+26
View File
@@ -0,0 +1,26 @@
# `docker` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `docker` | tag manifests, `/tags/list` | blobs, digest manifests |
## Terraform
```hcl
resource "artifactapi_remote_docker" "dockerhub" {
name = "dockerhub"
base_url = "https://registry-1.docker.io"
immutable_ttl = 0
mutable_ttl = 300
ban_tags_enabled = true
ban_tags = ["latest"]
patterns = [
"^library/postgres",
"^library/redis",
]
}
```
+7
View File
@@ -0,0 +1,7 @@
# `gems` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `gems` | compact index (`versions`, `info/*`, `names`), `*specs.4.8.gz` | `gems/*.gem`, `quick/Marshal.4.8/*.gemspec.rz` |
+28
View File
@@ -0,0 +1,28 @@
# `generic` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `generic` | nothing | everything |
## Terraform
```hcl
resource "artifactapi_remote_generic" "github" {
name = "github"
base_url = "https://github.com"
immutable_ttl = 0
mutable_ttl = 7200
patterns = [
"ducaale/xh/.*/xh-.*-x86_64-unknown-linux-musl.tar.gz$",
"mikefarah/yq/.*/yq_linux_amd64$",
]
mutable_patterns = [
".*/archive/refs/heads/.*\\.tar\\.gz$",
]
}
```
+7
View File
@@ -0,0 +1,7 @@
# `goproxy` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
+15
View File
@@ -0,0 +1,15 @@
# Local helm repository
A local `helm` repo is a real chart repository: `PUT` a packaged chart to
`/api/v2/remotes/<repo>/files/<chart>-<version>.tgz` and `index.yaml` is
generated from each chart's `Chart.yaml` (with sha256 digests) at
`/api/v1/local/<repo>/index.yaml`; a file not named after its `Chart.yaml`
name and version is stored but left out of the index. Add the local as a
member of a helm virtual to serve it alongside remotes.
```bash
helm package ./mychart
curl -fsSL --upload-file mychart-0.1.0.tgz \
https://artifactapi.example.com/api/v2/remotes/<repo>/files/mychart-0.1.0.tgz
helm repo add <repo> https://artifactapi.example.com/api/v1/local/<repo>
```
+19
View File
@@ -0,0 +1,19 @@
# `helm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `helm` | `index.yaml` | `.tgz` charts |
## Terraform
```hcl
resource "artifactapi_remote_helm" "jetstack" {
name = "jetstack"
base_url = "https://charts.jetstack.io"
immutable_ttl = 0
mutable_ttl = 3600
}
```
+9
View File
@@ -0,0 +1,9 @@
# `helm` virtual
```hcl
resource "artifactapi_virtual" "helm" {
name = "helm"
package_type = "helm"
members = [artifactapi_remote_helm.jetstack.name]
}
```
+14
View File
@@ -0,0 +1,14 @@
# Local npm registry
A local `npm` repo is a real npm registry: `npm publish` (scoped and unscoped)
and `npm install` work against `/api/v1/local/<repo>/`. npm requires an
`_authToken` for the registry before it will publish; any value works.
```ini
# .npmrc
registry=https://artifactapi.example.com/api/v1/local/<repo>/
//artifactapi.example.com/api/v1/local/<repo>/:_authToken=unused
```
Versions are immutable: republishing an existing version is rejected. The
dist-tags sent with a publish (`latest`, or `--tag`) move to that version.
+7
View File
@@ -0,0 +1,7 @@
# `npm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `npm` | package metadata | `.tgz` tarballs |
+30
View File
@@ -0,0 +1,30 @@
# `npm` virtual
An npm virtual serves one registry over local and remote npm members:
`npm install` and `npm view` work against `/api/v1/virtual/<name>/`.
- Versions are the union across members; on a duplicate version the earlier
member wins.
- Each dist-tag comes from the first member defining it, and is dropped if it
names a version missing from the merged set.
- Tarball URLs point at the virtual (`<member>/<path>`), which 302s to the
owning member.
- Full and abbreviated (`application/vnd.npm.install-v1+json`) packuments are
both served.
- A member error fails the request with 502; a member without the package is
skipped. Merges are cached for 60s per package.
```hcl
resource "artifactapi_virtual" "npm" {
name = "npm"
package_type = "npm"
members = [artifactapi_local_npm.internal.name, artifactapi_remote_npm.npmjs.name]
}
```
```ini
# .npmrc
registry=https://artifactapi.example.com/api/v1/virtual/npm/
```
Publish to the local member directly; the virtual is read-only.
+7
View File
@@ -0,0 +1,7 @@
# `nuget` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `nuget` | `index.json` (service index), registration and flat container indexes | `.nupkg`, `.snupkg`, `.nuspec`, `readme`, `icon` |
+14
View File
@@ -0,0 +1,14 @@
# Local puppet forge
A local `puppet` repo is a Forge: `PUT` a module tarball to
`/api/v2/remotes/<repo>/files/<owner>-<module>-<version>.tar.gz` and the Forge
v3 API (`/v3/modules`, `/v3/releases`, `/v3/files`) is served from each
release's `metadata.json` at `/api/v1/local/<repo>`. Versions are immutable.
```bash
puppet module build ./hello
curl -fsSL --upload-file pkg/unkin-hello-0.1.0.tar.gz \
https://artifactapi.example.com/api/v2/remotes/<repo>/files/unkin-hello-0.1.0.tar.gz
puppet module install unkin-hello \
--module_repository https://artifactapi.example.com/api/v1/local/<repo>
```
+7
View File
@@ -0,0 +1,7 @@
# `puppet` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `puppet` | `v3/modules/*`, `v3/releases*` | `.tar.gz` |
+7
View File
@@ -0,0 +1,7 @@
# `pypi` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `pypi` | `simple/*` index pages | `.whl`, `.tar.gz` |
+82
View File
@@ -0,0 +1,82 @@
# `github_rpm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
## GitHub releases as a yum repo (metadata-only, no precache)
A `github_rpm` remote turns a GitHub repo's **releases** into a real `dnf`/`yum`
repository without ever caching the packages. It scans releases for `.rpm`
assets, derives each package's metadata (NEVRA, requires/provides/conflicts/
obsoletes, files, checksum) and **synthesizes `repodata/` on the fly**. Package
metadata comes from a **ranged GET of just the RPM header** (the header sits at
the front of the file, so the whole package is never downloaded); the sha256
checksum comes from the GitHub asset `digest` when present, else a one-time
lazy stream. Derived metadata is cached (keyed by asset) so repodata generation
is served from primed DB rows, never a cold on-demand derive.
Each package's `<location>` points back at the remote, which **302-redirects**
the download to the `releases_remote` — an existing generic `github.com` remote
that streams the actual bytes. `dnf` follows the redirect transparently.
### Background syncer
A single process-wide **background syncer** keeps every `github_rpm` remote's
derived metadata current off the client request path:
- **Prime on create.** Creating a `github_rpm` remote enqueues a background prime
scan, so its metadata is derived right away without blocking the create call.
The first `dnf` request is served from cache. If a request arrives before the
prime lands, it returns a retryable `503` (with `Retry-After`) rather than
serving an empty repo or blocking on a multi-minute derive.
- **Periodic re-check, driven by `mutable_ttl`.** Each remote is re-checked for
new or changed releases no more often than its `mutable_ttl`. New/changed
assets are derived incrementally; assets already cached are never re-fetched,
and assets that disappear upstream are pruned.
- **ETag / 304 conditional requests.** The releases-list `ETag` is stored per
remote and sent as `If-None-Match`; a `304 Not Modified` means nothing changed
and the syncer derives nothing. GitHub does not count `304` conditional
responses against the rate limit, so an unchanged repo is nearly free — this is
the main lever keeping GitHub traffic low.
- **Global rate limit.** Every GitHub call (releases list + each ranged asset
header GET) passes through a single token-bucket limiter **shared across all
remotes**, so GitHub is never hammered. Configure a token (`password`) on the
remote for the higher authenticated rate limit (~5000/hr vs ~60/hr
unauthenticated).
- **Multi-replica coordination.** State is shared through the database. Before a
periodic scan a replica must atomically claim a per-remote lease
(`github_rpm_sync_state`: `last_synced_at`, `etag`, `sync_lease_owner`,
`sync_lease_expires`); only the winner scans. This bounds total GitHub load to
~once per `mutable_ttl` regardless of replica count, and the shared `etag`
lets any replica issue the conditional request.
```hcl
# Backend that serves the actual .rpm bytes from github.com.
resource "artifactapi_remote_generic" "github" {
name = "github"
base_url = "https://github.com"
patterns = [
"acme/tools/releases/download/.*\\.rpm$", # allowlist the repo's release assets
]
}
resource "artifactapi_remote_github_rpm" "acme-tools" {
name = "acme-tools"
base_url = "https://api.github.com/repos/acme/tools" # the releases API root
releases_remote = "github" # backend for downloads
mutable_ttl = 3600 # release re-scan interval
# Optional: restrict which release assets become packages (regex on filename).
patterns = [".*\\.x86_64\\.rpm$", ".*\\.noarch\\.rpm$"]
# Optional: a token for private repos / higher API rate limits.
# password = "ghp_..."
}
```
`dnf` config: `baseurl=https://artifactapi.example/api/v1/remote/acme-tools`.
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
from the synthesized metadata.
+7
View File
@@ -0,0 +1,7 @@
# `rpm` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `rpm` | `repomd.xml`, `repodata/*` | `.rpm` |
+29
View File
@@ -0,0 +1,29 @@
# Serving providers as a registry
A local `terraform` repo is a real provider registry: upload
`terraform-provider-{type}_{version}_{os}_{arch}.zip` files under
`{namespace}/{type}/`, and Terraform installs them from a bare source address —
no `.terraformrc` mirror config:
```hcl
terraform {
required_providers {
artifactapi = {
source = "artifactapi.example.com/<repo>/<type>"
version = "0.1.2"
}
}
}
```
The Terraform *namespace* segment is the artifactapi repo name; the provider is
matched by *type*. The registry serves service discovery
(`/.well-known/terraform.json`), the `providers.v1` version/download endpoints,
and a GPG-signed `SHA256SUMS` per the provider registry protocol.
Signing needs a GPG key. By default artifactapi generates one on first start and
stores it in the database (`signing_keys` table), so every replica shares it and
there's nothing to provision. To bring your own key instead, point
`TF_SIGNING_KEY_PATH` at an armored private key (optionally
`TF_SIGNING_KEY_PASSPHRASE`), which takes precedence over the generated one.
`TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
+7
View File
@@ -0,0 +1,7 @@
# `terraform` remote
Shared classification, TTL and eviction rules: [caching](../common/caching.md).
| Type | Mutable (auto-detected) | Immutable (auto-detected) |
|---|---|---|
| `terraform` | `*/versions` | `*/download/*/*` |
+9 -6
View File
@@ -22,14 +22,17 @@ already-running stack.
## Coverage
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact per remote package type (generic, docker,
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy, cargo) proxied through the
mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes
- **Caching** — one immutable artifact per remote package type (one
`caching_<type>_test.go` case each) proxied through the mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes
verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/`
index), rpm (real package + **automatic repodata** generation).
- **Local uploads** — puppet (Forge v3 API via real `r10k` and `puppet module install`),
generic (upload/download), pypi (wheel + generated `simple/`
index), rpm (real package + **automatic repodata** generation), npm (stock
`node:22` client: `npm publish` unscoped + scoped, `npm install` + `require()`,
second version, republish rejected).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge
across two members.
across two members; npm packument merge over two locals via a stock `node:22`
client (merged versions, install through the member redirect).
- **Mirrorlist** — an rpm remote with a `mirrorlist` of extra upstream mirrors
(pool = `base_url` + `mirrorlist`): round-robin distribution across both mirrors
(constant-body `mockupstreama` / `mockupstreamb`), failover past a dead primary,
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"alpine", "alpine/x86_64/testpkg-1.0-r0.apk", "alpine/x86_64/testpkg-1.0-r0.apk"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"cargo", "crates/mycrate/mycrate-1.0.0.crate", "crates/mycrate/mycrate-1.0.0.crate"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"docker", "library/testimg/blobs/blobdata", "v2/library/testimg/blobs/blobdata"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"gems", "gems/mygem-1.0.0.gem", "gems/mygem-1.0.0.gem"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"generic", "blobs/hello.bin", "blobs/hello.bin"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"goproxy", "goproxy/example.com/mod/@v/v1.0.0.zip", "goproxy/example.com/mod/@v/v1.0.0.zip"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"helm", "charts/mychart-1.0.0.tgz", "charts/mychart-1.0.0.tgz"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"npm", "mypkg/-/mypkg-1.0.0.tgz", "mypkg/-/mypkg-1.0.0.tgz"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"nuget", "nuget/e2epkg/1.0.0/e2epkg.1.0.0.nupkg", "nuget/e2epkg/1.0.0/e2epkg.1.0.0.nupkg"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"puppet", "puppet-releases/author-mod-1.0.0.tar.gz", "puppet-releases/author-mod-1.0.0.tar.gz"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"pypi", "packages/foo-1.0-py3-none-any.whl", "packages/foo-1.0-py3-none-any.whl"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"rpm", "rpmrepo/Packages/e2e-testpkg-1.0-1.noarch.rpm", "rpmrepo/Packages/e2e-testpkg-1.0-1.noarch.rpm"})
}
+7
View File
@@ -0,0 +1,7 @@
//go:build dockere2e
package e2edocker
func init() {
cachingCases = append(cachingCases, cachingCase{"terraform", "hashicorp/aws/download/pkg.zip", "v1/providers/hashicorp/aws/download/pkg.zip"})
}
+13 -22
View File
@@ -9,32 +9,23 @@ import (
"testing"
)
type cachingCase struct {
pkgType string
// path is the request path under /api/v1/remote/<name>/. The provider
// derives the upstream URL from it (docker prepends /v2/, terraform
// prepends /v1/providers/), and the fixture lives at that resolved path.
path string
fixture string
}
// cachingCases is filled by one caching_<type>_test.go file per package type.
var cachingCases []cachingCase
// TestCachingPerProvider proxies one immutable artifact for every remote
// package type through the mock upstream and asserts: first fetch is served
// from the remote, the second from cache, and the bytes match the origin.
func TestCachingPerProvider(t *testing.T) {
cases := []struct {
pkgType string
// path is the request path under /api/v1/remote/<name>/. The provider
// derives the upstream URL from it (docker prepends /v2/, terraform
// prepends /v1/providers/), and the fixture lives at that resolved path.
path string
fixture string
}{
{"generic", "blobs/hello.bin", "blobs/hello.bin"},
{"npm", "mypkg/-/mypkg-1.0.0.tgz", "mypkg/-/mypkg-1.0.0.tgz"},
{"helm", "charts/mychart-1.0.0.tgz", "charts/mychart-1.0.0.tgz"},
{"pypi", "packages/foo-1.0-py3-none-any.whl", "packages/foo-1.0-py3-none-any.whl"},
{"rpm", "rpmrepo/Packages/e2e-testpkg-1.0-1.noarch.rpm", "rpmrepo/Packages/e2e-testpkg-1.0-1.noarch.rpm"},
{"alpine", "alpine/x86_64/testpkg-1.0-r0.apk", "alpine/x86_64/testpkg-1.0-r0.apk"},
{"puppet", "puppet-releases/author-mod-1.0.0.tar.gz", "puppet-releases/author-mod-1.0.0.tar.gz"},
{"goproxy", "goproxy/example.com/mod/@v/v1.0.0.zip", "goproxy/example.com/mod/@v/v1.0.0.zip"},
{"cargo", "crates/mycrate/mycrate-1.0.0.crate", "crates/mycrate/mycrate-1.0.0.crate"},
{"terraform", "hashicorp/aws/download/pkg.zip", "v1/providers/hashicorp/aws/download/pkg.zip"},
{"docker", "library/testimg/blobs/blobdata", "v2/library/testimg/blobs/blobdata"},
}
for _, tc := range cases {
for _, tc := range cachingCases {
t.Run(tc.pkgType, func(t *testing.T) {
name := "cache-" + tc.pkgType
createRepo(t, fmt.Sprintf(`{
+1
View File
@@ -0,0 +1 @@
mygem-1.0.0 e2e fixture
+71
View File
@@ -0,0 +1,71 @@
//go:build dockere2e
package e2edocker
import (
"net/http"
"os"
"os/exec"
"strings"
"testing"
"time"
)
// TestLocalHelmRealClient packages a chart with the real helm CLI, publishes it
// to a helm local, then drives helm repo add/search/pull/template against the
// local and against a helm virtual that has the local as a member.
func TestLocalHelmRealClient(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := strings.TrimRight(os.Getenv("ARTIFACTAPI_INTERNAL"), "/")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
createRepo(t, `{"name":"local-helm","package_type":"helm","repo_type":"local"}`)
defer deleteRepo(t, "local-helm")
createVirtual(t, `{"name":"vhelm","package_type":"helm","members":["local-helm"]}`)
defer deleteVirtual(t, "vhelm")
helm := func(script string) string {
t.Helper()
out, err := exec.Command("docker", "run", "--rm", "--network", network, "--entrypoint", "sh",
"alpine/helm:3.16.4", "-c", "set -eu; "+script).Output()
if err != nil {
stderr := ""
if ee, ok := err.(*exec.ExitError); ok {
stderr = string(ee.Stderr)
}
t.Fatalf("helm script failed: %v\n%s%s", err, out, stderr)
}
return string(out)
}
chart := helm("cd /tmp && helm create e2echart >/dev/null && helm package e2echart >/dev/null && cat e2echart-0.1.0.tgz")
uploadFile(t, "local-helm", "e2echart-0.1.0.tgz", []byte(chart), "application/gzip")
// Chart metadata is recorded asynchronously after upload; poll the index.
var body []byte
for deadline := time.Now().Add(15 * time.Second); time.Now().Before(deadline); time.Sleep(500 * time.Millisecond) {
var resp *http.Response
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-helm/index.yaml"), nil, "")
if resp.StatusCode == http.StatusOK && strings.Contains(string(body), "e2echart-0.1.0.tgz") {
break
}
}
if !strings.Contains(string(body), "e2echart-0.1.0.tgz") || !strings.Contains(string(body), "digest: ") {
t.Fatalf("index.yaml never listed the chart:\n%s", body)
}
for _, repoURL := range []string{internal + "/api/v1/local/local-helm", internal + "/api/v1/virtual/vhelm"} {
out := helm("helm repo add e2e " + repoURL + " >/dev/null && " +
"helm search repo e2e/e2echart && " +
"cd /tmp && helm pull e2e/e2echart --version 0.1.0 && " +
"helm template rel e2echart-0.1.0.tgz | grep -c 'kind: Deployment'")
if !strings.Contains(out, "e2e/e2echart") || !strings.Contains(out, "0.1.0") || !strings.HasSuffix(strings.TrimSpace(out), "1") {
t.Fatalf("helm via %s:\n%s", repoURL, out)
}
}
}
+62
View File
@@ -0,0 +1,62 @@
//go:build dockere2e
package e2edocker
import (
"fmt"
"os"
"os/exec"
"strings"
"testing"
"time"
)
// TestLocalNPMPublishInstall drives a stock npm client against a local npm
// repo: publish unscoped and scoped packages, install and require() them in a
// fresh project, publish a second version, and reject a republish.
func TestLocalNPMPublishInstall(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := os.Getenv("ARTIFACTAPI_INTERNAL")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
name := fmt.Sprintf("e2e-npm-local-%d", time.Now().UnixNano())
createRepo(t, fmt.Sprintf(`{"name":%q,"package_type":"npm","repo_type":"local"}`, name))
defer deleteRepo(t, name)
reg := strings.TrimRight(internal, "/") + "/api/v1/local/" + name + "/"
script := `set -euo pipefail
printf 'registry=%s\n%s:_authToken=unused\n' "$REG" "${REG#http:}" > ~/.npmrc
mk() { d="/w/src/$1-$2"; mkdir -p "$d"; cd "$d"
printf '{"name":"%s","version":"%s","main":"index.js"}' "$1" "$2" > package.json
echo "module.exports = '$1@$2';" > index.js; }
mk e2e-npm-plain 1.0.0 && npm publish
mk @e2e/npm-scoped 1.0.0 && npm publish
mk e2e-npm-plain 1.0.0
if npm publish 2>/dev/null; then echo "republish accepted"; exit 1; fi
mkdir -p /w/app && cd /w/app && npm init -y >/dev/null
npm install --no-audit --cache /tmp/fresh-cache e2e-npm-plain @e2e/npm-scoped
node -e "console.log('REQ', require('e2e-npm-plain'), require('@e2e/npm-scoped'))"
mk e2e-npm-plain 1.1.0 && npm publish
echo "VERSIONS $(npm view e2e-npm-plain versions --json | tr -d ' \n')"
echo "LATEST $(npm view e2e-npm-plain dist-tags.latest)"`
out, err := exec.Command("docker", "run", "--rm", "--network", network,
"-e", "REG="+reg, "node:22", "bash", "-c", script).CombinedOutput()
if err != nil {
t.Fatalf("npm client run failed: %v\n%s", err, out)
}
for _, want := range []string{
"REQ e2e-npm-plain@1.0.0 @e2e/npm-scoped@1.0.0",
`VERSIONS ["1.0.0","1.1.0"]`,
"LATEST 1.1.0",
} {
if !strings.Contains(string(out), want) {
t.Fatalf("missing %q in npm output:\n%s", want, out)
}
}
}
+68
View File
@@ -0,0 +1,68 @@
//go:build dockere2e
package e2edocker
import (
"fmt"
"os"
"os/exec"
"strings"
"testing"
"time"
)
// TestVirtualNPMInstall publishes to two local npm repos and drives a stock
// npm client through a virtual over both: merged versions and dist-tags, and
// an install whose tarballs resolve via the virtual's member redirect.
func TestVirtualNPMInstall(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := os.Getenv("ARTIFACTAPI_INTERNAL")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
id := time.Now().UnixNano()
a, b, virt := fmt.Sprintf("e2e-npm-a-%d", id), fmt.Sprintf("e2e-npm-b-%d", id), fmt.Sprintf("e2e-npm-v-%d", id)
for _, name := range []string{a, b} {
createRepo(t, fmt.Sprintf(`{"name":%q,"package_type":"npm","repo_type":"local"}`, name))
defer deleteRepo(t, name)
}
createVirtual(t, fmt.Sprintf(`{"name":%q,"package_type":"npm","members":[%q,%q]}`, virt, a, b))
defer deleteVirtual(t, virt)
base := strings.TrimRight(internal, "/")
script := `set -euo pipefail
pub() { d="/w/src/$2-$3-$1"; mkdir -p "$d"; cd "$d"
printf '{"name":"%s","version":"%s","main":"index.js"}' "$2" "$3" > package.json
echo "module.exports = '$1:$2@$3';" > index.js
printf 'registry=%s\n%s:_authToken=unused\n' "$BASE/api/v1/local/$1/" "${BASE#http:}/api/v1/local/$1/" > .npmrc
npm publish; }
pub "$A" @e2e/virt 1.0.0
pub "$B" @e2e/virt 1.0.0
pub "$B" @e2e/virt 1.1.0
mkdir -p /w/app && cd /w/app && npm init -y >/dev/null
echo "registry=$BASE/api/v1/virtual/$V/" > .npmrc
echo "VERSIONS $(npm view @e2e/virt versions --json | tr -d ' \n')"
npm install --no-audit --cache /tmp/fresh-cache @e2e/virt@1.0.0
node -e "console.log('REQ', require('@e2e/virt'))"
echo "RESOLVED $(node -p "require('./package-lock.json').packages['node_modules/@e2e/virt'].resolved")"`
out, err := exec.Command("docker", "run", "--rm", "--network", network,
"-e", "BASE="+base, "-e", "A="+a, "-e", "B="+b, "-e", "V="+virt,
"node:22", "bash", "-c", script).CombinedOutput()
if err != nil {
t.Fatalf("npm client run failed: %v\n%s", err, out)
}
for _, want := range []string{
`VERSIONS ["1.0.0","1.1.0"]`,
"REQ " + a + ":@e2e/virt@1.0.0",
"RESOLVED " + base + "/api/v1/virtual/" + virt + "/" + a + "/@e2e/virt/-/virt-1.0.0.tgz",
} {
if !strings.Contains(string(out), want) {
t.Fatalf("missing %q in npm output:\n%s", want, out)
}
}
}
+88
View File
@@ -0,0 +1,88 @@
//go:build dockere2e
package e2edocker
import (
"archive/tar"
"bytes"
"compress/gzip"
"net/http"
"os"
"os/exec"
"strings"
"testing"
"time"
)
const puppetHelloMetadata = `{"name":"unkin-hello","version":"0.1.0","author":"unkin","summary":"e2e","license":"Apache-2.0","source":"https://example.invalid","dependencies":[{"name":"puppetlabs/stdlib","version_requirement":">= 4.0.0 < 10.0.0"}]}`
func puppetModuleTarball(t *testing.T) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for name, body := range map[string]string{
"unkin-hello-0.1.0/metadata.json": puppetHelloMetadata,
"unkin-hello-0.1.0/manifests/init.pp": "class hello {}\n",
} {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body)), ModTime: time.Now()}); err != nil {
t.Fatal(err)
}
_, _ = tw.Write([]byte(body))
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
// TestLocalPuppetRealClient publishes a module to a puppet local and installs
// it with the real r10k and puppet module tool. The stdlib dependency is not on
// a local-only forge, so puppet runs with --ignore-dependencies.
func TestLocalPuppetRealClient(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := strings.TrimRight(os.Getenv("ARTIFACTAPI_INTERNAL"), "/")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
createRepo(t, `{"name":"local-puppet","package_type":"puppet","repo_type":"local"}`)
defer deleteRepo(t, "local-puppet")
upload := api("/api/v2/remotes/local-puppet/files/unkin-hello-0.1.0.tar.gz")
defer doRequest(t, http.MethodDelete, api("/api/v2/remotes/local-puppet/files/v3/files/unkin-hello-0.1.0.tar.gz"), nil, "")
tarball := puppetModuleTarball(t)
if resp, body := doRequest(t, http.MethodPut, upload, tarball, "application/gzip"); resp.StatusCode != http.StatusCreated {
t.Fatalf("upload: %d %s", resp.StatusCode, body)
}
if resp, _ := doRequest(t, http.MethodPut, upload, tarball, "application/gzip"); resp.StatusCode != http.StatusConflict {
t.Fatalf("re-upload of an existing version: %d, want 409", resp.StatusCode)
}
if resp, body := getEventually(t, api("/api/v1/local/local-puppet/v3/releases/unkin-hello-0.1.0"), 15*time.Second); resp.StatusCode != http.StatusOK || !strings.Contains(string(body), `"file_md5"`) {
t.Fatalf("release never indexed: %d %s", resp.StatusCode, body)
}
forge := internal + "/api/v1/local/local-puppet"
run := func(image, script string) string {
t.Helper()
out, err := exec.Command("docker", "run", "--rm", "--network", network, "--entrypoint", "sh", image, "-c", "set -eu; "+script).CombinedOutput()
if err != nil {
t.Fatalf("%s failed: %v\n%s", image, err, out)
}
return string(out)
}
out := run("ghcr.io/voxpupuli/r10k:latest", "cd /tmp && printf 'forge:\\n baseurl: "+forge+"\\n' > r10k.yaml && "+
"printf \"forge '"+forge+"'\\nmod 'unkin-hello', '0.1.0'\\n\" > Puppetfile && "+
"r10k puppetfile install -c r10k.yaml --verbose && cat modules/hello/metadata.json")
if !strings.Contains(out, `"unkin-hello"`) {
t.Fatalf("r10k did not install unkin-hello:\n%s", out)
}
out = run("ghcr.io/openvoxproject/openvoxagent:latest", "export PATH=/opt/puppetlabs/bin:$PATH; puppet module install unkin-hello --module_repository "+forge+" --ignore-dependencies --target-dir /tmp/mods && puppet module list --color=false --modulepath /tmp/mods")
if !strings.Contains(out, "unkin-hello (v0.1.0)") {
t.Fatalf("puppet module install did not install unkin-hello:\n%s", out)
}
}
+182
View File
@@ -0,0 +1,182 @@
//go:build e2e
package e2e
import (
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
// assertHeadMatchesGet checks HEAD returns GET's status, type and length with no body.
func assertHeadMatchesGet(t *testing.T, url string) {
t.Helper()
get, err := http.Get(url)
if err != nil {
t.Fatalf("GET: %v", err)
}
body, _ := io.ReadAll(get.Body)
get.Body.Close()
if get.StatusCode != http.StatusOK {
t.Fatalf("GET %s: %d %s", url, get.StatusCode, body)
}
head, err := http.Head(url)
if err != nil {
t.Fatalf("HEAD: %v", err)
}
hb, _ := io.ReadAll(head.Body)
head.Body.Close()
if head.StatusCode != http.StatusOK {
t.Fatalf("HEAD %s: got %d, want 200", url, head.StatusCode)
}
if len(hb) != 0 {
t.Errorf("HEAD returned a %d-byte body", len(hb))
}
for _, h := range []string{"Content-Type", "Cache-Control", "Vary"} {
if head.Header.Get(h) != get.Header.Get(h) {
t.Errorf("HEAD %s %q, GET %q", h, head.Header.Get(h), get.Header.Get(h))
}
}
if head.ContentLength != int64(len(body)) {
t.Errorf("HEAD Content-Length %d, GET body %d bytes", head.ContentLength, len(body))
}
}
func TestProxyHeadNuGetRewritten(t *testing.T) {
var upstream *httptest.Server
upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(map[string]any{"version": "3.0.0", "resources": []map[string]string{
{"@id": upstream.URL + "/v3-flatcontainer/", "@type": "PackageBaseAddress/3.0.0"},
}})
}))
defer upstream.Close()
createRemote(t, mustFormat(`{"name": "head-nuget", "package_type": "nuget", "base_url": %q}`, upstream.URL+"/v3/index.json"))
defer deleteRemote(t, "head-nuget")
url := apiURL("/api/v1/remote/head-nuget/index.json")
assertHeadMatchesGet(t, url)
req, _ := http.NewRequest(http.MethodHead, url, nil)
req.Host = "evil.example:"
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusBadRequest {
t.Errorf("HEAD with invalid Host: got %d, want 400", resp.StatusCode)
}
}
func TestProxyHeadGitHubRPMRepodata(t *testing.T) {
rpm := testsupport.MinimalRPM("demo", "1.2", "3", "x86_64")
var gh *httptest.Server
mux := http.NewServeMux()
mux.HandleFunc("/repos/acme/tools/releases", func(w http.ResponseWriter, r *http.Request) {
if p := r.URL.Query().Get("page"); p != "" && p != "1" {
w.Write([]byte("[]"))
return
}
json.NewEncoder(w).Encode([]map[string]any{{"tag_name": "v1.2-3", "assets": []map[string]any{{
"name": "demo-1.2-3.x86_64.rpm",
"size": len(rpm),
"browser_download_url": gh.URL + "/acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm",
}}}})
})
mux.HandleFunc("/acme/tools/releases/download/", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Length", strconv.Itoa(len(rpm)))
w.Write(rpm)
})
gh = httptest.NewServer(mux)
defer gh.Close()
createRemote(t, mustFormat(`{"name": "head-ghrpm", "package_type": "github_rpm", "base_url": %q, "releases_remote": "github"}`, gh.URL+"/repos/acme/tools"))
defer deleteRemote(t, "head-ghrpm")
url := apiURL("/api/v1/remote/head-ghrpm/repodata/repomd.xml")
deadline := time.Now().Add(30 * time.Second)
for {
if _, status := getBody(t, url); status == http.StatusOK || time.Now().After(deadline) {
break
}
time.Sleep(500 * time.Millisecond)
}
assertHeadMatchesGet(t, url)
}
func TestProxyHeadNpmUncachedPackument(t *testing.T) {
const abbreviated = "application/vnd.npm.install-v1+json"
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
doc := map[string]any{"name": "left-pad", "versions": map[string]any{"1.3.0": map[string]any{}}}
if r.Header.Get("Accept") != abbreviated {
doc["readme"] = strings.Repeat("full packument ", 200)
}
json.NewEncoder(w).Encode(doc)
}))
defer upstream.Close()
createRemote(t, mustFormat(`{"name": "head-npm", "package_type": "npm", "base_url": %q}`, upstream.URL))
defer deleteRemote(t, "head-npm")
url := apiURL("/api/v1/remote/head-npm/left-pad")
do := func(method string) (*http.Response, []byte) {
req, _ := http.NewRequest(method, url, nil)
req.Header.Set("Accept", abbreviated)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("%s: %v", method, err)
}
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
return resp, b
}
head, _ := do(http.MethodHead)
get, body := do(http.MethodGet)
if head.StatusCode != http.StatusOK || get.StatusCode != http.StatusOK {
t.Fatalf("HEAD %d, GET %d", head.StatusCode, get.StatusCode)
}
if head.ContentLength != int64(len(body)) {
t.Errorf("uncached HEAD Content-Length %d, GET body %d bytes", head.ContentLength, len(body))
}
}
func TestProxyHeadNonRewritingUsesUpstreamHead(t *testing.T) {
methods := make(chan string, 4)
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
methods <- r.Method
w.Header().Set("Content-Type", "text/xml")
w.Write([]byte("<repomd/>"))
}))
defer upstream.Close()
createRemote(t, mustFormat(`{"name": "head-rpm", "package_type": "rpm", "base_url": %q}`, upstream.URL))
defer deleteRemote(t, "head-rpm")
resp, err := http.Head(apiURL("/api/v1/remote/head-rpm/repodata/repomd.xml"))
if err != nil {
t.Fatalf("HEAD: %v", err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK || resp.Header.Get("X-Artifact-Source") != "remote" {
t.Fatalf("HEAD %d source %q", resp.StatusCode, resp.Header.Get("X-Artifact-Source"))
}
close(methods)
var seen []string
for m := range methods {
seen = append(seen, m)
}
if len(seen) != 1 || seen[0] != http.MethodHead {
t.Errorf("upstream saw %v, want a single HEAD", seen)
}
}
+59
View File
@@ -0,0 +1,59 @@
//go:build e2e
package e2e
import (
"encoding/json"
"net/http"
"strings"
"testing"
)
func getForwardedHTTPS(t *testing.T, url string) *http.Response {
t.Helper()
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header.Set("X-Forwarded-Proto", "https")
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
resp, err := client.Do(req)
if err != nil {
t.Fatalf("GET %s: %v", url, err)
}
return resp
}
func TestForwardedProtoCargoConfig(t *testing.T) {
createRemote(t, `{"name": "scheme-cargo", "package_type": "cargo", "base_url": "https://index.crates.io"}`)
defer deleteRemote(t, "scheme-cargo")
resp := getForwardedHTTPS(t, apiURL("/api/v1/remote/scheme-cargo/config.json"))
defer resp.Body.Close()
var cfg map[string]string
if err := json.NewDecoder(resp.Body).Decode(&cfg); err != nil {
t.Fatal(err)
}
if want := "https://" + strings.TrimPrefix(baseURL, "http://") + "/api/v1/remote/scheme-cargo/"; !strings.HasPrefix(cfg["dl"], want) {
t.Errorf("dl = %q, want prefix %q", cfg["dl"], want)
}
}
func TestForwardedProtoVirtualRedirect(t *testing.T) {
createRemote(t, `{"name": "scheme-rpm", "package_type": "rpm", "base_url": "https://rpm.example.com"}`)
defer deleteRemote(t, "scheme-rpm")
resp, err := http.Post(apiURL("/api/v2/virtuals"), "application/json",
strings.NewReader(`{"name": "scheme-virt", "package_type": "rpm", "members": ["scheme-rpm"]}`))
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
t.Fatalf("create virtual: status %d", resp.StatusCode)
}
defer deleteRequest(t, apiURL("/api/v2/virtuals/scheme-virt"))
resp = getForwardedHTTPS(t, apiURL("/api/v1/virtual/scheme-virt/scheme-rpm/Packages/a.rpm"))
resp.Body.Close()
loc := resp.Header.Get("Location")
if want := "https://" + strings.TrimPrefix(baseURL, "http://") + "/api/v1/remote/scheme-rpm/Packages/a.rpm"; resp.StatusCode != http.StatusFound || loc != want {
t.Errorf("redirect = %d %q, want 302 %q", resp.StatusCode, loc, want)
}
}
+3
View File
@@ -7,7 +7,9 @@ require (
github.com/cavaliergopher/rpm v1.3.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/coreos/go-oidc/v3 v3.21.0
github.com/go-chi/chi/v5 v5.3.0
github.com/go-jose/go-jose/v4 v4.1.4
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.10.0
github.com/klauspost/compress v1.19.2
@@ -102,6 +104,7 @@ require (
go.uber.org/atomic v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.40.0 // indirect
gopkg.in/ini.v1 v1.67.2 // indirect
+6
View File
@@ -44,6 +44,8 @@ github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/coreos/go-oidc/v3 v3.21.0 h1:wZo4Q9Pum8dYEj0eMUPrqR+kvuGkeUplbLpNCkBqoWM=
github.com/coreos/go-oidc/v3 v3.21.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26KVTCI9SrY4=
github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
@@ -67,6 +69,8 @@ github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeO
github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE=
github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -225,6 +229,8 @@ golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM
golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+2 -8
View File
@@ -13,6 +13,7 @@ import (
"github.com/go-chi/chi/v5"
v1 "git.unkin.net/unkin/artifactapi/internal/api/v1"
"git.unkin.net/unkin/artifactapi/internal/database"
tfprov "git.unkin.net/unkin/artifactapi/internal/provider/terraform"
"git.unkin.net/unkin/artifactapi/internal/tfsign"
@@ -290,12 +291,5 @@ func writeJSON(w http.ResponseWriter, v any) {
}
func baseURL(r *http.Request) string {
scheme := "http"
if r.TLS != nil {
scheme = "https"
}
if fwd := r.Header.Get("X-Forwarded-Proto"); fwd != "" {
scheme = fwd
}
return scheme + "://" + r.Host
return v1.Scheme(r) + "://" + r.Host
}
+334
View File
@@ -0,0 +1,334 @@
package v1
import (
"bytes"
"context"
"crypto/sha1"
"crypto/sha512"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/url"
"regexp"
"strings"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// Local npm repos speak the npm registry publish/read protocol. Per package:
// <pkg>/-/<base>-<version>.tgz tarball (immutable)
// <pkg>/-/<base>-<version>.json version manifest (immutable)
// <pkg>/-/dist-tags/<tag> version string (mutable)
var (
npmNameRe = regexp.MustCompile(`^(@[a-z0-9][a-z0-9._~-]*/)?[a-z0-9][a-z0-9._~-]*$`)
npmRefRe = regexp.MustCompile(`^[0-9A-Za-z][0-9A-Za-z.+_-]*$`)
npmSemverRe = regexp.MustCompile(`^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$`)
npmMaxPublishBytes int64 = 256 << 20
)
func (h *ProxyHandler) localNPMRemote(r *http.Request, name string) (*models.Remote, bool) {
remote, err := h.db.GetRemote(r.Context(), name)
if err != nil {
return nil, false
}
return remote, remote.RepoType == models.RepoTypeLocal && remote.PackageType == models.PackageNPM
}
// npmPackageName returns the package a packument/publish path addresses, or ""
// when the path is something else (e.g. a tarball under <pkg>/-/).
func npmPackageName(raw string) string {
p, err := url.PathUnescape(raw)
if err != nil {
return ""
}
p = strings.TrimSuffix(p, "/")
if !npmNameRe.MatchString(p) {
return ""
}
return p
}
type npmRelease struct {
version string
manifest []byte
tarball []byte
tags []string
}
func parseNPMPublish(body io.Reader, pkg string) (*npmRelease, error) {
var doc struct {
Name string `json:"name"`
DistTags map[string]string `json:"dist-tags"`
Versions map[string]json.RawMessage `json:"versions"`
Attachments map[string]struct {
Data string `json:"data"`
} `json:"_attachments"`
}
if err := json.NewDecoder(body).Decode(&doc); err != nil {
return nil, fmt.Errorf("invalid publish body: %w", err)
}
if doc.Name != pkg {
return nil, fmt.Errorf("body name %q does not match %q", doc.Name, pkg)
}
if len(doc.Versions) != 1 || len(doc.Attachments) != 1 {
return nil, errors.New("publish must carry exactly one version and one tarball")
}
rel := &npmRelease{}
for v, m := range doc.Versions {
rel.version, rel.manifest = v, m
}
if !npmSemverRe.MatchString(rel.version) {
return nil, fmt.Errorf("invalid version %q", rel.version)
}
var meta struct {
Name string `json:"name"`
Version string `json:"version"`
Dist struct {
Integrity string `json:"integrity"`
Shasum string `json:"shasum"`
} `json:"dist"`
}
if err := json.Unmarshal(rel.manifest, &meta); err != nil {
return nil, fmt.Errorf("invalid version manifest: %w", err)
}
if meta.Name != pkg || meta.Version != rel.version {
return nil, errors.New("version manifest name/version mismatch")
}
for _, a := range doc.Attachments {
tgz, err := base64.StdEncoding.DecodeString(a.Data)
if err != nil {
return nil, fmt.Errorf("invalid tarball encoding: %w", err)
}
rel.tarball = tgz
}
if err := verifyNPMDist(rel.tarball, meta.Dist.Integrity, meta.Dist.Shasum); err != nil {
return nil, err
}
for tag, v := range doc.DistTags {
if !npmRefRe.MatchString(tag) || v != rel.version {
return nil, fmt.Errorf("dist-tag %q must point at the published version", tag)
}
rel.tags = append(rel.tags, tag)
}
return rel, nil
}
func verifyNPMDist(tgz []byte, integrity, shasum string) error {
if b64, ok := strings.CutPrefix(integrity, "sha512-"); ok {
sum := sha512.Sum512(tgz)
if base64.StdEncoding.EncodeToString(sum[:]) != b64 {
return errors.New("tarball does not match dist.integrity")
}
return nil
}
if shasum != "" {
sum := sha1.Sum(tgz)
if hex.EncodeToString(sum[:]) != shasum {
return errors.New("tarball does not match dist.shasum")
}
return nil
}
return errors.New("version manifest carries no sha512 integrity or shasum")
}
func npmError(w http.ResponseWriter, status int, msg string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(map[string]string{"error": msg})
}
func (h *ProxyHandler) handleLocalPut(w http.ResponseWriter, r *http.Request) {
remote, ok := h.localNPMRemote(r, chi.URLParam(r, "localName"))
if !ok {
npmError(w, http.StatusMethodNotAllowed, "publish is only supported on local npm repositories")
return
}
pkg := npmPackageName(chi.URLParam(r, "*"))
if pkg == "" {
npmError(w, http.StatusBadRequest, "invalid package name")
return
}
rel, err := parseNPMPublish(http.MaxBytesReader(w, r.Body, npmMaxPublishBytes), pkg)
if err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) {
npmError(w, http.StatusRequestEntityTooLarge, fmt.Sprintf("publish body exceeds %d bytes", npmMaxPublishBytes))
return
}
npmError(w, http.StatusBadRequest, err.Error())
return
}
base := pkg[strings.LastIndex(pkg, "/")+1:]
stem := pkg + "/-/" + base + "-" + rel.version
// The manifest gates visibility and tags may only name visible versions, so
// the order is tarball, manifest, tags. A tarball left by a failed publish is
// accepted again only with identical bytes; an existing manifest always conflicts.
conflict := func() {
npmError(w, http.StatusConflict, fmt.Sprintf("%s@%s already exists; overwrites are not allowed", pkg, rel.version))
}
ok, err = h.npmStoreImmutable(r.Context(), remote.Name, stem+".tgz", rel.tarball, "application/octet-stream")
if err != nil {
npmError(w, http.StatusInternalServerError, err.Error())
return
}
if !ok {
conflict()
return
}
if _, err := h.npmStore(r.Context(), remote.Name, stem+".json", rel.manifest, "application/json", false); err != nil {
if errors.Is(err, database.ErrAlreadyExists) {
conflict()
return
}
npmError(w, http.StatusInternalServerError, err.Error())
return
}
for _, tag := range rel.tags {
if _, err := h.npmStore(r.Context(), remote.Name, pkg+"/-/dist-tags/"+tag, []byte(rel.version), "text/plain", true); err != nil {
npmError(w, http.StatusInternalServerError, err.Error())
return
}
}
slog.Info("local npm package published", "repo", remote.Name, "package", pkg, "version", rel.version)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusCreated)
_ = json.NewEncoder(w).Encode(map[string]any{"ok": true, "id": pkg})
}
// npmStoreImmutable reports false when path already holds different bytes.
func (h *ProxyHandler) npmStoreImmutable(ctx context.Context, repo, path string, data []byte, contentType string) (bool, error) {
hash, err := h.npmStore(ctx, repo, path, data, contentType, false)
if !errors.Is(err, database.ErrAlreadyExists) {
return err == nil, err
}
existing, err := h.db.GetLocalFile(ctx, repo, path)
if err != nil || existing == nil {
return false, fmt.Errorf("read existing %s: %v", path, err)
}
return existing.ContentHash == hash, nil
}
func (h *ProxyHandler) npmStore(ctx context.Context, repo, path string, data []byte, contentType string, mutable bool) (string, error) {
result, err := h.cas.Store(ctx, bytes.NewReader(data), contentType)
if err != nil {
return "", err
}
if err := h.db.UpsertBlob(ctx, result.ContentHash, result.S3Key, result.SizeBytes, contentType); err != nil {
return "", err
}
if mutable {
return result.ContentHash, h.db.UpsertLocalFile(ctx, repo, path, result.ContentHash)
}
return result.ContentHash, h.db.CreateLocalFile(ctx, repo, path, result.ContentHash)
}
func (h *ProxyHandler) readLocalBlob(ctx context.Context, contentHash string) ([]byte, error) {
reader, _, err := h.store.Download(ctx, storage.BlobKey(strings.TrimPrefix(contentHash, "sha256:")))
if err != nil {
return nil, err
}
defer reader.Close()
return io.ReadAll(reader)
}
func (h *ProxyHandler) serveNPMPackument(w http.ResponseWriter, r *http.Request, remote *models.Remote, pkg string) {
files, err := h.db.ListLocalFilesByPrefix(r.Context(), remote.Name, pkg+"/-/")
if err != nil {
npmError(w, http.StatusInternalServerError, err.Error())
return
}
base, ok := requestBaseURL(w, r)
if !ok {
return
}
read := func(hash string) ([]byte, error) { return h.readLocalBlob(r.Context(), hash) }
tarballBase := base + "/api/v1/local/" + remote.Name + "/"
doc, err := buildNPMPackument(pkg, files, read, tarballBase)
if err != nil {
npmError(w, http.StatusInternalServerError, err.Error())
return
}
if doc == nil {
npmError(w, http.StatusNotFound, "not found")
return
}
setNPMPerHostHeaders(w)
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Artifact-Source", "local")
w.WriteHeader(http.StatusOK)
_ = json.NewEncoder(w).Encode(doc)
}
// setNPMPerHostHeaders keeps shared caches off packuments whose tarball URLs
// are built from the request host and Accept (abbreviated vs full).
func setNPMPerHostHeaders(w http.ResponseWriter) {
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Vary", "Host, X-Forwarded-Proto, Accept")
}
// buildNPMPackument returns nil when pkg has no published versions.
func buildNPMPackument(pkg string, files []database.LocalFile, read func(hash string) ([]byte, error), tarballBase string) (map[string]any, error) {
prefix := pkg + "/-/"
versions := map[string]any{}
tags := map[string]string{}
for _, f := range files {
// LIKE treats '_' as a wildcard, so the prefix match can over-select.
rest, ok := strings.CutPrefix(f.FilePath, prefix)
if !ok {
continue
}
if tag, isTag := strings.CutPrefix(rest, "dist-tags/"); isTag {
v, err := read(f.ContentHash)
if err != nil {
return nil, err
}
tags[tag] = string(v)
continue
}
stem, isManifest := strings.CutSuffix(rest, ".json")
if !isManifest {
continue
}
raw, err := read(f.ContentHash)
if err != nil {
return nil, err
}
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
return nil, fmt.Errorf("manifest %s: %w", f.FilePath, err)
}
dist, _ := m["dist"].(map[string]any)
if dist == nil {
dist = map[string]any{}
m["dist"] = dist
}
dist["tarball"] = tarballBase + prefix + stem + ".tgz"
v, _ := m["version"].(string)
versions[v] = m
}
if len(versions) == 0 {
return nil, nil
}
for tag, v := range tags {
if _, ok := versions[v]; !ok {
delete(tags, tag)
}
}
return map[string]any{"_id": pkg, "name": pkg, "dist-tags": tags, "versions": versions}, nil
}
+333
View File
@@ -0,0 +1,333 @@
package v1
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"sync"
"testing"
"time"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/internal/virtual"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
var (
testDSN string
testMinio testsupport.MinioConn
)
func TestMain(m *testing.M) {
ctx := context.Background()
dsn, termPG, err := testsupport.StartPostgres(ctx)
if err != nil {
os.Exit(m.Run())
}
conn, termMinio, err := testsupport.StartMinio(ctx)
if err != nil {
termPG()
os.Exit(m.Run())
}
testDSN, testMinio = dsn, conn
code := m.Run()
termMinio()
termPG()
os.Exit(code)
}
func npmTestServer(t *testing.T) (*ProxyHandler, *httptest.Server) {
t.Helper()
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
t.Cleanup(db.Close)
var store *storage.S3
for i := 0; i < 20; i++ {
if store, err = storage.NewS3(testMinio.Endpoint, testMinio.AccessKey, testMinio.SecretKey, "npm", false, ""); err == nil {
break
}
time.Sleep(500 * time.Millisecond)
}
if err != nil {
t.Fatal(err)
}
h := NewProxyHandler(nil, nil, db, store, nil)
r := chi.NewRouter()
r.Mount("/api/v1", h.Routes())
srv := httptest.NewServer(r)
t.Cleanup(srv.Close)
return h, srv
}
func npmRepo(t *testing.T, h *ProxyHandler, pt models.PackageType) string {
t.Helper()
name := fmt.Sprintf("npm-%s-%d", pt, time.Now().UnixNano())
if err := h.db.CreateRemote(context.Background(), &models.Remote{Name: name, PackageType: pt, RepoType: models.RepoTypeLocal}); err != nil {
t.Fatal(err)
}
return name
}
func npmDo(t *testing.T, method, url, body string) (int, string) {
t.Helper()
req, _ := http.NewRequest(method, url, strings.NewReader(body))
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
defer func() { _ = resp.Body.Close() }()
b, _ := io.ReadAll(resp.Body)
return resp.StatusCode, string(b)
}
func packumentVersions(t *testing.T, url string) map[string]any {
t.Helper()
code, body := npmDo(t, http.MethodGet, url, "")
if code != http.StatusOK {
t.Fatalf("GET %s = %d: %s", url, code, body)
}
var doc struct {
Versions map[string]any `json:"versions"`
}
if err := json.Unmarshal([]byte(body), &doc); err != nil {
t.Fatal(err)
}
return doc.Versions
}
func packumentTags(t *testing.T, url string) map[string]string {
t.Helper()
_, body := npmDo(t, http.MethodGet, url, "")
var doc struct {
Tags map[string]string `json:"dist-tags"`
}
if err := json.Unmarshal([]byte(body), &doc); err != nil {
t.Fatal(err)
}
return doc.Tags
}
func concurrentPuts(t *testing.T, url string, bodies ...string) []int {
t.Helper()
var wg sync.WaitGroup
codes := make([]int, len(bodies))
for i, b := range bodies {
wg.Add(1)
go func() {
defer wg.Done()
codes[i], _ = npmDo(t, http.MethodPut, url, b)
}()
}
wg.Wait()
return codes
}
func TestNPMPublishHandler(t *testing.T) {
h, srv := npmTestServer(t)
repo := npmRepo(t, h, models.PackageNPM)
base := srv.URL + "/api/v1/local/" + repo + "/"
t.Run("scoped name via %2f", func(t *testing.T) {
body := publishBody(t, "@acme/widget", []byte("scoped tgz"), publishOpts{})
if code, b := npmDo(t, http.MethodPut, base+"@acme%2fwidget", body); code != http.StatusCreated {
t.Fatalf("publish = %d: %s", code, b)
}
v := packumentVersions(t, base+"@acme%2fwidget")
dist := v["1.0.0"].(map[string]any)["dist"].(map[string]any)
tarball := dist["tarball"].(string)
if !strings.HasPrefix(tarball, base+"@acme/widget/-/widget-1.0.0.tgz") {
t.Fatalf("tarball = %s", tarball)
}
if code, b := npmDo(t, http.MethodGet, tarball, ""); code != http.StatusOK || b != "scoped tgz" {
t.Fatalf("tarball fetch = %d %q", code, b)
}
})
t.Run("republish", func(t *testing.T) {
if code, b := npmDo(t, http.MethodPut, base+"repub", publishBody(t, "repub", []byte("v1"), publishOpts{})); code != http.StatusCreated {
t.Fatalf("first publish = %d: %s", code, b)
}
if code, _ := npmDo(t, http.MethodPut, base+"repub", publishBody(t, "repub", []byte("different"), publishOpts{})); code != http.StatusConflict {
t.Errorf("different bytes = %d, want 409", code)
}
if code, b := npmDo(t, http.MethodPut, base+"repub", publishBody(t, "repub", []byte("v2"), publishOpts{version: "1.1.0"})); code != http.StatusCreated {
t.Fatalf("publish 1.1.0 = %d: %s", code, b)
}
if code, _ := npmDo(t, http.MethodPut, base+"repub", publishBody(t, "repub", []byte("v1"), publishOpts{})); code != http.StatusConflict {
t.Errorf("identical republish = %d, want 409", code)
}
if tags := packumentTags(t, base+"repub"); tags["latest"] != "1.1.0" {
t.Errorf("latest = %q, want 1.1.0", tags["latest"])
}
})
t.Run("rejected republish leaves tags", func(t *testing.T) {
for _, v := range []string{"1.0.0", "1.1.0"} {
if code, b := npmDo(t, http.MethodPut, base+"rej", publishBody(t, "rej", []byte("tgz "+v), publishOpts{version: v})); code != http.StatusCreated {
t.Fatalf("publish %s = %d: %s", v, code, b)
}
}
if code, _ := npmDo(t, http.MethodPut, base+"rej", publishBody(t, "rej", []byte("tgz 1.0.0"), publishOpts{shasum: "x"})); code != http.StatusConflict {
t.Fatalf("re-PUT 1.0.0 = %d, want 409", code)
}
if tags := packumentTags(t, base+"rej"); tags["latest"] != "1.1.0" {
t.Errorf("latest = %q, want 1.1.0", tags["latest"])
}
})
t.Run("mid-publish window", func(t *testing.T) {
if code, b := npmDo(t, http.MethodPut, base+"mid", publishBody(t, "mid", []byte("v1"), publishOpts{})); code != http.StatusCreated {
t.Fatalf("publish = %d: %s", code, b)
}
ctx := context.Background()
if _, err := h.npmStore(ctx, repo, "mid/-/mid-1.0.1.tgz", []byte("v2"), "application/octet-stream", false); err != nil {
t.Fatal(err)
}
if tags := packumentTags(t, base+"mid"); tags["latest"] != "1.0.0" {
t.Errorf("tarball-only 1.0.1: latest = %q", tags["latest"])
}
if _, err := h.npmStore(ctx, repo, "mid/-/dist-tags/latest", []byte("1.0.1"), "text/plain", true); err != nil {
t.Fatal(err)
}
if tags := packumentTags(t, base+"mid"); tags["latest"] == "1.0.1" {
t.Error("latest points at invisible 1.0.1")
}
})
t.Run("concurrent identical", func(t *testing.T) {
body := publishBody(t, "same", []byte("same tgz"), publishOpts{})
codes := concurrentPuts(t, base+"same", body, body)
if codes[0]+codes[1] != http.StatusCreated+http.StatusConflict || codes[0] == codes[1] {
t.Fatalf("codes = %v, want one 201 and one 409", codes)
}
if tags := packumentTags(t, base+"same"); len(tags) != 1 || tags["latest"] != "1.0.0" {
t.Errorf("tags = %v", tags)
}
})
t.Run("concurrent different manifests", func(t *testing.T) {
tgz := []byte("race tgz")
codes := concurrentPuts(t, base+"race",
publishBody(t, "race", tgz, publishOpts{shasum: "a", tags: map[string]string{"a": "1.0.0"}}),
publishBody(t, "race", tgz, publishOpts{shasum: "b", tags: map[string]string{"b": "1.0.0"}}))
won := map[int]string{}
for i, c := range codes {
won[c] = []string{"a", "b"}[i]
}
if len(won) != 2 || won[http.StatusCreated] == "" || won[http.StatusConflict] == "" {
t.Fatalf("codes = %v, want one 201 and one 409", codes)
}
v := packumentVersions(t, base+"race")["1.0.0"].(map[string]any)
if got := v["dist"].(map[string]any)["shasum"]; got != won[http.StatusCreated] {
t.Errorf("manifest shasum = %v, winner %s", got, won[http.StatusCreated])
}
tags := packumentTags(t, base+"race")
if len(tags) != 1 || tags[won[http.StatusCreated]] != "1.0.0" {
t.Errorf("tags = %v, winner %s", tags, won[http.StatusCreated])
}
})
t.Run("retry after tarball-only write", func(t *testing.T) {
tgz := []byte("half published")
if _, err := h.npmStore(context.Background(), repo, "retry/-/retry-1.0.0.tgz", tgz, "application/octet-stream", false); err != nil {
t.Fatal(err)
}
if code, _ := npmDo(t, http.MethodGet, base+"retry", ""); code != http.StatusNotFound {
t.Fatalf("tarball-only version visible: %d", code)
}
if code, _ := npmDo(t, http.MethodPut, base+"retry", publishBody(t, "retry", []byte("other bytes"), publishOpts{})); code != http.StatusConflict {
t.Errorf("retry with different bytes = %d, want 409", code)
}
if code, b := npmDo(t, http.MethodPut, base+"retry", publishBody(t, "retry", tgz, publishOpts{})); code != http.StatusCreated {
t.Fatalf("retry with identical bytes = %d: %s", code, b)
}
if _, ok := packumentVersions(t, base+"retry")["1.0.0"]; !ok {
t.Error("retried version missing from packument")
}
})
t.Run("concurrent versions", func(t *testing.T) {
codes := concurrentPuts(t, base+"conc",
publishBody(t, "conc", []byte("tgz 1.0.0"), publishOpts{version: "1.0.0"}),
publishBody(t, "conc", []byte("tgz 2.0.0"), publishOpts{version: "2.0.0"}))
if codes[0] != http.StatusCreated || codes[1] != http.StatusCreated {
t.Fatalf("codes = %v", codes)
}
v := packumentVersions(t, base+"conc")
if _, ok := v["1.0.0"]; !ok {
t.Error("1.0.0 missing")
}
if _, ok := v["2.0.0"]; !ok {
t.Error("2.0.0 missing")
}
})
t.Run("body too large", func(t *testing.T) {
old := npmMaxPublishBytes
npmMaxPublishBytes = 64
defer func() { npmMaxPublishBytes = old }()
if code, _ := npmDo(t, http.MethodPut, base+"big", publishBody(t, "big", []byte("x"), publishOpts{})); code != http.StatusRequestEntityTooLarge {
t.Errorf("oversized body = %d, want 413", code)
}
})
t.Run("non-npm local", func(t *testing.T) {
generic := npmRepo(t, h, models.PackageGeneric)
if code, _ := npmDo(t, http.MethodPut, srv.URL+"/api/v1/local/"+generic+"/pkg", publishBody(t, "pkg", []byte("x"), publishOpts{})); code != http.StatusMethodNotAllowed {
t.Errorf("publish to generic local = %d, want 405", code)
}
})
}
func TestNPMPackumentPerHost(t *testing.T) {
h, srv := npmTestServer(t)
h.virtualEngine = virtual.NewEngine(h.db, nil)
h.virtualEngine.NPMLocal = h.LocalNPMPackument
repo := npmRepo(t, h, models.PackageNPM)
virt := "v" + repo
if err := h.db.CreateVirtual(context.Background(), &models.Virtual{Name: virt, PackageType: models.PackageNPM, Members: []string{repo}}); err != nil {
t.Fatal(err)
}
if code, b := npmDo(t, http.MethodPut, srv.URL+"/api/v1/local/"+repo+"/hostpkg", publishBody(t, "hostpkg", []byte("tgz"), publishOpts{})); code != http.StatusCreated {
t.Fatalf("publish = %d: %s", code, b)
}
routes := h.Routes()
get := func(path, host, proto string) *httptest.ResponseRecorder {
r := httptest.NewRequest(http.MethodGet, path, nil)
r.Host = host
if proto != "" {
r.Header.Set("X-Forwarded-Proto", proto)
}
w := httptest.NewRecorder()
routes.ServeHTTP(w, r)
return w
}
for _, path := range []string{"/local/" + repo + "/hostpkg", "/virtual/" + virt + "/hostpkg"} {
for _, host := range []string{"x@evil.com", "evil.example:"} {
if w := get(path, host, ""); w.Code != http.StatusBadRequest {
t.Errorf("%s Host %q = %d, want 400", path, host, w.Code)
}
}
w := get(path, "aa.example", "https")
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `"tarball":"https://aa.example/api/v1/`) {
t.Fatalf("%s = %d: %s", path, w.Code, w.Body)
}
if cc, vary := w.Header().Get("Cache-Control"), w.Header().Get("Vary"); cc != "no-store" || vary != "Host, X-Forwarded-Proto, Accept" {
t.Errorf("%s Cache-Control %q Vary %q", path, cc, vary)
}
}
}
+166
View File
@@ -0,0 +1,166 @@
package v1
import (
"crypto/sha1"
"crypto/sha512"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/database"
)
func TestNPMPackageName(t *testing.T) {
tests := map[string]string{
"left-pad": "left-pad",
"left-pad/": "left-pad",
"@acme%2fwidget": "@acme/widget",
"@acme%2Fwidget": "@acme/widget",
"@acme/widget": "@acme/widget",
"left-pad/-/left-pad.tgz": "",
"@acme/widget/-/w-1.tgz": "",
"a/b": "",
"../etc": "",
"Upper": "",
"": "",
}
for in, want := range tests {
if got := npmPackageName(in); got != want {
t.Errorf("npmPackageName(%q) = %q, want %q", in, got, want)
}
}
}
type publishOpts struct {
bodyName, manifestName, version, integrity, shasum string
tags map[string]string
extraVersion bool
}
func publishBody(t *testing.T, pkg string, tgz []byte, o publishOpts) string {
t.Helper()
sha := sha512.Sum512(tgz)
sh1 := sha1.Sum(tgz)
if o.bodyName == "" {
o.bodyName = pkg
}
if o.manifestName == "" {
o.manifestName = pkg
}
if o.version == "" {
o.version = "1.0.0"
}
if o.integrity == "" {
o.integrity = "sha512-" + base64.StdEncoding.EncodeToString(sha[:])
}
if o.shasum == "" {
o.shasum = hex.EncodeToString(sh1[:])
}
if o.tags == nil {
o.tags = map[string]string{"latest": o.version}
}
versions := map[string]any{o.version: map[string]any{
"name": o.manifestName, "version": o.version, "main": "index.js",
"dist": map[string]any{"integrity": o.integrity, "shasum": o.shasum, "tarball": "http://elsewhere/x.tgz"},
}}
if o.extraVersion {
versions["2.0.0"] = map[string]any{"name": pkg, "version": "2.0.0"}
}
b, _ := json.Marshal(map[string]any{
"_id": pkg, "name": o.bodyName, "dist-tags": o.tags, "versions": versions,
"_attachments": map[string]any{pkg + "-" + o.version + ".tgz": map[string]any{
"content_type": "application/octet-stream", "data": base64.StdEncoding.EncodeToString(tgz), "length": len(tgz),
}},
})
return string(b)
}
func TestParseNPMPublish(t *testing.T) {
tgz := []byte("fake tarball bytes")
rel, err := parseNPMPublish(strings.NewReader(publishBody(t, "@acme/widget", tgz, publishOpts{tags: map[string]string{"latest": "1.0.0", "beta": "1.0.0"}})), "@acme/widget")
if err != nil {
t.Fatalf("valid scoped publish rejected: %v", err)
}
if rel.version != "1.0.0" || string(rel.tarball) != string(tgz) || len(rel.tags) != 2 {
t.Fatalf("unexpected release: %+v", rel)
}
if _, err := parseNPMPublish(strings.NewReader(publishBody(t, "left-pad", tgz, publishOpts{integrity: "none", shasum: hex.EncodeToString(make([]byte, 20))})), "left-pad"); err == nil {
t.Fatal("shasum fallback should reject a mismatched tarball")
}
if _, err := parseNPMPublish(strings.NewReader(publishBody(t, "left-pad", tgz, publishOpts{integrity: "none"})), "left-pad"); err != nil {
t.Fatalf("matching shasum fallback rejected: %v", err)
}
bad := map[string]publishOpts{
"body name mismatch": {bodyName: "other"},
"manifest name mismatch": {manifestName: "other"},
"integrity mismatch": {integrity: "sha512-AAAA"},
"path-like version": {version: "../1.0.0"},
"non-semver version": {version: "1..2"},
"leading-zero version": {version: "01.0.0"},
"tag at another version": {tags: map[string]string{"latest": "9.9.9"}},
"path-like tag": {tags: map[string]string{"../x": "1.0.0"}},
"multiple versions at once": {extraVersion: true},
}
for name, o := range bad {
if _, err := parseNPMPublish(strings.NewReader(publishBody(t, "left-pad", tgz, o)), "left-pad"); err == nil {
t.Errorf("%s: expected rejection", name)
}
}
if _, err := parseNPMPublish(strings.NewReader(`{"name":"left-pad","versions":{}}`), "left-pad"); err == nil {
t.Error("deprecate-style body without attachments should be rejected")
}
}
func TestBuildNPMPackument(t *testing.T) {
blobs := map[string]string{
"h1": `{"name":"my_pkg","version":"1.0.0","dist":{"shasum":"aa","tarball":"http://elsewhere/x.tgz"}}`,
"h2": `{"name":"my_pkg","version":"1.1.0"}`,
"h3": "1.1.0",
"h4": "2.0.0",
"hx": `{"name":"myXpkg","version":"6.6.6"}`,
}
read := func(h string) ([]byte, error) {
b, ok := blobs[h]
if !ok {
return nil, errors.New("missing blob")
}
return []byte(b), nil
}
files := []database.LocalFile{
{FilePath: "my_pkg/-/dist-tags/latest", ContentHash: "h3"},
{FilePath: "my_pkg/-/dist-tags/next", ContentHash: "h4"},
{FilePath: "my_pkg/-/my_pkg-1.0.0.json", ContentHash: "h1"},
{FilePath: "my_pkg/-/my_pkg-1.0.0.tgz", ContentHash: "nope"},
{FilePath: "my_pkg/-/my_pkg-1.1.0.json", ContentHash: "h2"},
{FilePath: "myXpkg/-/myXpkg-6.6.6.json", ContentHash: "hx"},
}
doc, err := buildNPMPackument("my_pkg", files, read, "http://reg/api/v1/local/npm/")
if err != nil {
t.Fatal(err)
}
if tags := doc["dist-tags"].(map[string]string); tags["latest"] != "1.1.0" || len(tags) != 1 {
t.Errorf("dist-tags = %v", tags)
}
versions := doc["versions"].(map[string]any)
if len(versions) != 2 {
t.Fatalf("versions = %v (LIKE over-match must be dropped)", versions)
}
v1 := versions["1.0.0"].(map[string]any)["dist"].(map[string]any)
if v1["tarball"] != "http://reg/api/v1/local/npm/my_pkg/-/my_pkg-1.0.0.tgz" || v1["shasum"] != "aa" {
t.Errorf("1.0.0 dist = %v", v1)
}
if versions["1.1.0"].(map[string]any)["dist"].(map[string]any)["tarball"] != "http://reg/api/v1/local/npm/my_pkg/-/my_pkg-1.1.0.tgz" {
t.Error("1.1.0 missing rewritten tarball")
}
if doc, err := buildNPMPackument("absent", files, read, "x/"); err != nil || doc != nil {
t.Errorf("absent package: doc=%v err=%v", doc, err)
}
}
+26
View File
@@ -0,0 +1,26 @@
package v1
import (
"context"
"encoding/json"
"git.unkin.net/unkin/artifactapi/internal/virtual"
)
// LocalNPMPackument serves a local npm repo's packument to npm virtuals, with
// tarball paths relative to the repo.
func (h *ProxyHandler) LocalNPMPackument(ctx context.Context, repo, pkg string) ([]byte, error) {
files, err := h.db.ListLocalFilesByPrefix(ctx, repo, pkg+"/-/")
if err != nil {
return nil, err
}
read := func(hash string) ([]byte, error) { return h.readLocalBlob(ctx, hash) }
doc, err := buildNPMPackument(pkg, files, read, "")
if err != nil {
return nil, err
}
if doc == nil {
return nil, virtual.ErrNotFound
}
return json.Marshal(doc)
}
+152 -30
View File
@@ -1,11 +1,15 @@
package v1
import (
"bytes"
"errors"
"fmt"
"io"
"log/slog"
"net"
"net/http"
"strconv"
"strings"
"github.com/go-chi/chi/v5"
@@ -15,6 +19,7 @@ import (
"git.unkin.net/unkin/artifactapi/internal/proxy"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/internal/virtual"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type ProxyHandler struct {
@@ -40,7 +45,9 @@ func NewProxyHandler(engine *proxy.Engine, virtualEngine *virtual.Engine, db *da
func (h *ProxyHandler) Routes() chi.Router {
r := chi.NewRouter()
r.Get("/remote/{remoteName}/*", h.handleProxy)
r.Head("/remote/{remoteName}/*", h.handleProxyHead)
r.Get("/local/{localName}/*", h.handleLocal)
r.Put("/local/{localName}/*", h.handleLocalPut)
r.Get("/virtual/{virtualName}/*", h.handleVirtual)
return r
}
@@ -67,31 +74,43 @@ func (h *ProxyHandler) handleDockerPing(w http.ResponseWriter, r *http.Request)
w.WriteHeader(http.StatusOK)
}
func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
// remoteFor resolves the remote and provider named in the request, writing the
// error response itself when either is missing.
func (h *ProxyHandler) remoteFor(w http.ResponseWriter, r *http.Request) (*models.Remote, provider.Provider, bool) {
remoteName := chi.URLParam(r, "remoteName")
path := chi.URLParam(r, "*")
remote, err := h.db.GetRemote(r.Context(), remoteName)
if err != nil {
http.Error(w, fmt.Sprintf("remote %q not found", remoteName), http.StatusNotFound)
return
return nil, nil, false
}
prov, err := provider.Get(remote.PackageType)
if err != nil {
http.Error(w, fmt.Sprintf("no provider for %q", remote.PackageType), http.StatusInternalServerError)
return nil, nil, false
}
return remote, prov, true
}
// serveRemote lets metadata-only remotes (e.g. github_rpm) synthesize their own
// responses and redirect package downloads instead of proxying bytes.
func (h *ProxyHandler) serveRemote(w http.ResponseWriter, r *http.Request, remote *models.Remote, prov provider.Provider, path string) bool {
rs, ok := prov.(provider.RemoteServer)
return ok && rs.ServeRemote(w, r, *remote, path, fmt.Sprintf("%s://%s", Scheme(r), r.Host), h.db)
}
func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
remote, prov, ok := h.remoteFor(w, r)
if !ok {
return
}
// Metadata-only remotes (e.g. github_rpm) synthesize their own responses and
// redirect package downloads to a backend remote instead of proxying bytes.
if rs, ok := prov.(provider.RemoteServer); ok {
proxyBaseURL := fmt.Sprintf("%s://%s", scheme(r), r.Host)
if rs.ServeRemote(w, r, *remote, path, proxyBaseURL, h.db) {
return
}
path := chi.URLParam(r, "*")
if h.serveRemote(w, r, remote, prov, path) {
return
}
h.serveFetch(w, r, remote, prov, path)
}
func (h *ProxyHandler) serveFetch(w http.ResponseWriter, r *http.Request, remote *models.Remote, prov provider.Provider, path string) {
result, err := h.engine.Fetch(r.Context(), *remote, path, prov, r.Header)
if err != nil {
var proxyErr *proxy.ProxyError
@@ -99,34 +118,51 @@ func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
http.Error(w, proxyErr.Message, proxyErr.Status)
return
}
slog.Error("proxy fetch failed", "remote", remoteName, "path", path, "error", err)
slog.Error("proxy fetch failed", "remote", remote.Name, "path", path, "error", err)
http.Error(w, "bad gateway", http.StatusBadGateway)
return
}
defer result.Reader.Close()
var body io.Reader = result.Reader
if sr, ok := prov.(provider.ServeRewriter); ok && prov.Classify(path) == provider.Mutable {
raw, ok := rewriteServed(w, r, sr, *remote, result.Reader)
if !ok {
return
}
body, result.Size = bytes.NewReader(raw), int64(len(raw))
}
w.Header().Set("Content-Type", result.ContentType)
w.Header().Set("X-Artifact-Source", result.Source)
if result.Size > 0 {
w.Header().Set("Content-Length", fmt.Sprintf("%d", result.Size))
w.Header().Set("X-Artifact-Size", fmt.Sprintf("%d", result.Size))
}
w.WriteHeader(http.StatusOK)
io.Copy(w, result.Reader)
io.Copy(w, body)
}
func (h *ProxyHandler) handleProxyHead(w http.ResponseWriter, r *http.Request) {
remoteName := chi.URLParam(r, "remoteName")
path := chi.URLParam(r, "*")
func rewritesMutable(prov provider.Provider, path string) bool {
_, sr := prov.(provider.ServeRewriter)
_, rr := prov.(provider.ResponseRewriter)
return (sr || rr) && prov.Classify(path) == provider.Mutable
}
remote, err := h.db.GetRemote(r.Context(), remoteName)
if err != nil {
http.Error(w, fmt.Sprintf("remote %q not found", remoteName), http.StatusNotFound)
// handleProxyHead answers HEAD like GET without a body. Responses a provider
// synthesizes or rewrites per request go through the GET path so their headers
// match; everything else is answered from store metadata or an upstream HEAD.
func (h *ProxyHandler) handleProxyHead(w http.ResponseWriter, r *http.Request) {
remote, prov, ok := h.remoteFor(w, r)
if !ok {
return
}
prov, err := provider.Get(remote.PackageType)
if err != nil {
http.Error(w, fmt.Sprintf("no provider for %q", remote.PackageType), http.StatusInternalServerError)
path := chi.URLParam(r, "*")
if h.serveRemote(w, r, remote, prov, path) {
return
}
if rewritesMutable(prov, path) {
h.serveFetch(w, r, remote, prov, path)
return
}
@@ -137,7 +173,7 @@ func (h *ProxyHandler) handleProxyHead(w http.ResponseWriter, r *http.Request) {
http.Error(w, proxyErr.Message, proxyErr.Status)
return
}
slog.Error("proxy head failed", "remote", remoteName, "path", path, "error", err)
slog.Error("proxy head failed", "remote", remote.Name, "path", path, "error", err)
http.Error(w, "bad gateway", http.StatusBadGateway)
return
}
@@ -160,7 +196,10 @@ func (h *ProxyHandler) handleVirtual(w http.ResponseWriter, r *http.Request) {
return
}
proxyBaseURL := fmt.Sprintf("%s://%s", scheme(r), r.Host)
proxyBaseURL, ok := requestBaseURL(w, r)
if !ok {
return
}
loc, ok, err := h.virtualEngine.MemberRedirect(r.Context(), *virt, path, r.URL.RawQuery, proxyBaseURL)
if err != nil {
@@ -172,7 +211,14 @@ func (h *ProxyHandler) handleVirtual(w http.ResponseWriter, r *http.Request) {
return
}
body, contentType, err := h.virtualEngine.Fetch(r.Context(), *virt, path, proxyBaseURL)
var body []byte
var contentType string
if virt.PackageType == models.PackageNPM {
abbreviated := strings.Contains(r.Header.Get("Accept"), virtual.NPMAbbreviatedType)
body, contentType, err = h.virtualEngine.FetchNPM(r.Context(), *virt, path, proxyBaseURL, abbreviated)
} else {
body, contentType, err = h.virtualEngine.Fetch(r.Context(), *virt, path, proxyBaseURL)
}
if errors.Is(err, virtual.ErrNotFound) {
http.Error(w, "not found", http.StatusNotFound)
return
@@ -183,6 +229,9 @@ func (h *ProxyHandler) handleVirtual(w http.ResponseWriter, r *http.Request) {
return
}
if virt.PackageType == models.PackageNPM {
setNPMPerHostHeaders(w)
}
w.Header().Set("Content-Type", contentType)
w.Header().Set("X-Artifact-Source", "virtual")
w.WriteHeader(http.StatusOK)
@@ -199,6 +248,13 @@ func (h *ProxyHandler) handleLocal(w http.ResponseWriter, r *http.Request) {
return
}
if remote.PackageType == models.PackageNPM {
if pkg := npmPackageName(path); pkg != "" {
h.serveNPMPackument(w, r, remote, pkg)
return
}
}
prov, _ := provider.Get(remote.PackageType)
if indexer, ok := prov.(provider.LocalIndexer); ok {
if indexer.ServeLocalIndex(w, r, h.db, remote.Name, path) {
@@ -237,12 +293,78 @@ func (h *ProxyHandler) serveLocalFile(w http.ResponseWriter, r *http.Request, re
io.Copy(w, reader)
}
func scheme(r *http.Request) string {
// Scheme returns the client-facing scheme: TLS, else the first
// X-Forwarded-Proto value when it is http or https, else http.
func Scheme(r *http.Request) string {
if r.TLS != nil {
return "https"
}
if fwd := r.Header.Get("X-Forwarded-Proto"); fwd != "" {
fwd, _, _ := strings.Cut(r.Header.Get("X-Forwarded-Proto"), ",")
if fwd = strings.ToLower(strings.TrimSpace(fwd)); fwd == "https" {
return fwd
}
return "http"
}
// rewriteServed rewrites a cached body for the requesting host and marks the
// response uncacheable by shared caches, since it varies per Host.
func rewriteServed(w http.ResponseWriter, r *http.Request, sr provider.ServeRewriter, remote models.Remote, src io.Reader) ([]byte, bool) {
base, ok := requestBaseURL(w, r)
if !ok {
return nil, false
}
raw, err := io.ReadAll(src)
if err != nil {
http.Error(w, "bad gateway", http.StatusBadGateway)
return nil, false
}
raw = sr.RewriteServed(raw, remote, base)
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Vary", "Host, X-Forwarded-Proto")
w.Header().Set("Content-Length", strconv.Itoa(len(raw)))
return raw, true
}
// requestBaseURL returns the client-facing scheme://host, answering 400 on an
// invalid Host.
func requestBaseURL(w http.ResponseWriter, r *http.Request) (string, bool) {
if !validHost(r.Host) {
http.Error(w, "invalid host header", http.StatusBadRequest)
return "", false
}
return Scheme(r) + "://" + r.Host, true
}
// validHost accepts host[:port] where host is a DNS name or IP literal.
func validHost(hostport string) bool {
host, port := hostport, ""
if h, p, err := net.SplitHostPort(hostport); err == nil {
if p == "" {
return false
}
host, port = h, p
} else if strings.HasPrefix(hostport, "[") {
return strings.HasSuffix(hostport, "]") && net.ParseIP(hostport[1:len(hostport)-1]) != nil
}
if port != "" {
if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 {
return false
}
}
if net.ParseIP(host) != nil {
return true
}
if strings.HasPrefix(hostport, "[") {
return false
}
if host == "" || len(host) > 253 {
return false
}
for _, label := range strings.Split(strings.TrimSuffix(host, "."), ".") {
if label == "" || len(label) > 63 || label[0] == '-' || label[len(label)-1] == '-' ||
strings.Trim(strings.ToLower(label), "abcdefghijklmnopqrstuvwxyz0123456789-_") != "" {
return false
}
}
return true
}
+74
View File
@@ -0,0 +1,74 @@
package v1
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider/nuget"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestValidHost(t *testing.T) {
for _, h := range []string{"aa.example", "aa.example:8443", "artifactapi.k8s.syd1.au.unkin.net", "10.0.0.1:80", "[::1]:8080", "[::1]", "::1", "localhost", "my_host.local"} {
if !validHost(h) {
t.Errorf("validHost(%q) = false, want true", h)
}
}
for _, h := range []string{"", "evil\".example", "a b", "aa.example:0", "aa.example:99999", "aa.example:x", "aa.example:", ":8080", "-bad.example", "a..b", "[::1", "[nope]:80", "a/b", "a<script>", strings.Repeat("a", 64) + ".example"} {
if validHost(h) {
t.Errorf("validHost(%q) = true, want false", h)
}
}
}
var nugetOrg = models.Remote{Name: "nuget-org", BaseURL: "https://api.nuget.org/v3/index.json"}
const serviceIndex = `{"resources":[{"@id":"https://api.nuget.org/v3-flatcontainer/"}]}`
func rewrite(t *testing.T, host, proto string) (*httptest.ResponseRecorder, string, bool) {
t.Helper()
r := httptest.NewRequest(http.MethodGet, "/api/v1/remote/nuget-org/index.json", nil)
r.Host = host
if proto != "" {
r.Header.Set("X-Forwarded-Proto", proto)
}
w := httptest.NewRecorder()
raw, ok := rewriteServed(w, r, &nuget.Provider{}, nugetOrg, strings.NewReader(serviceIndex))
return w, string(raw), ok
}
func TestRewriteServedHeaders(t *testing.T) {
w, body, ok := rewrite(t, "aa.example:8443", "https")
if !ok {
t.Fatal("valid request rejected")
}
if want := `{"resources":[{"@id":"https://aa.example:8443/api/v1/remote/nuget-org/v3-flatcontainer/"}]}`; body != want {
t.Errorf("body = %s, want %s", body, want)
}
if got := w.Header().Get("Cache-Control"); got != "no-store" {
t.Errorf("Cache-Control = %q, want no-store", got)
}
if got := w.Header().Get("Vary"); got != "Host, X-Forwarded-Proto" {
t.Errorf("Vary = %q", got)
}
if got := w.Header().Get("Content-Length"); got != "91" || len(body) != 91 {
t.Errorf("Content-Length = %q, body len %d", got, len(body))
}
}
func TestRewriteServedIgnoresBogusProto(t *testing.T) {
_, plain, _ := rewrite(t, "aa.example", "")
_, bogus, _ := rewrite(t, "aa.example", "javascript")
if bogus != plain || !strings.Contains(bogus, `"http://aa.example/`) {
t.Errorf("bogus X-Forwarded-Proto changed the body: %s", bogus)
}
}
func TestRewriteServedRejectsInvalidHost(t *testing.T) {
w, _, ok := rewrite(t, `evil"},{"x":"`, "")
if ok || w.Code != http.StatusBadRequest {
t.Errorf("invalid host: ok=%v code=%d, want rejected with 400", ok, w.Code)
}
}
+41
View File
@@ -0,0 +1,41 @@
package v1
import (
"net/http"
"testing"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/provider/npm"
"git.unkin.net/unkin/artifactapi/internal/provider/nuget"
"git.unkin.net/unkin/artifactapi/internal/provider/rpm"
)
func TestRoutesRemoteHead(t *testing.T) {
r := (&ProxyHandler{}).Routes()
for _, m := range []string{http.MethodGet, http.MethodHead} {
if !r.Match(chi.NewRouteContext(), m, "/remote/myremote/some/path") {
t.Errorf("%s /remote/myremote/some/path not routed", m)
}
}
}
func TestRewritesMutable(t *testing.T) {
tests := []struct {
name string
prov provider.Provider
path string
want bool
}{
{"npm packument", &npm.Provider{}, "left-pad", true},
{"npm tarball", &npm.Provider{}, "left-pad/-/left-pad-1.3.0.tgz", false},
{"nuget index", &nuget.Provider{}, "index.json", true},
{"rpm repodata", &rpm.Provider{}, "repodata/repomd.xml", false},
}
for _, tc := range tests {
if got := rewritesMutable(tc.prov, tc.path); got != tc.want {
t.Errorf("%s: rewritesMutable = %v, want %v", tc.name, got, tc.want)
}
}
}
+17 -3
View File
@@ -7,14 +7,28 @@ import (
)
func TestScheme(t *testing.T) {
if got := scheme(&http.Request{TLS: &tls.ConnectionState{}}); got != "https" {
if got := Scheme(&http.Request{TLS: &tls.ConnectionState{}}); got != "https" {
t.Errorf("TLS request scheme = %q, want https", got)
}
r := &http.Request{Header: http.Header{"X-Forwarded-Proto": {"https"}}}
if got := scheme(r); got != "https" {
if got := Scheme(r); got != "https" {
t.Errorf("X-Forwarded-Proto scheme = %q, want https", got)
}
if got := scheme(&http.Request{Header: http.Header{}}); got != "http" {
for fwd, want := range map[string]string{
"https, http": "https",
"http, https": "http",
"HTTPS": "https",
" https ": "https",
"javascript": "http",
"ftp": "http",
"https;evil": "http",
} {
r := &http.Request{Header: http.Header{"X-Forwarded-Proto": {fwd}}}
if got := Scheme(r); got != want {
t.Errorf("X-Forwarded-Proto %q scheme = %q, want %s", fwd, got, want)
}
}
if got := Scheme(&http.Request{Header: http.Header{}}); got != "http" {
t.Errorf("default scheme = %q, want http", got)
}
}
+20 -10
View File
@@ -67,8 +67,12 @@ func (h *RemotesHandler) get(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, remote)
}
// defaultMutableTTL matches the remotes.mutable_ttl column default; a create
// that omits mutable_ttl gets it, an explicit 0 still means never expire.
const defaultMutableTTL = 3600
func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
var remote models.Remote
remote := models.Remote{MutableTTL: defaultMutableTTL}
if err := json.NewDecoder(r.Body).Decode(&remote); err != nil {
http.Error(w, "invalid json", http.StatusBadRequest)
return
@@ -114,7 +118,15 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
func (h *RemotesHandler) update(w http.ResponseWriter, r *http.Request) {
name := chi.URLParam(r, "name")
// Capture the current backend before the update so we can tell whether the
// remote's base_url (its upstream) changed. A read failure just means we
// skip the freshness flush; it must not block the update.
existing, err := h.db.GetRemote(r.Context(), name)
oldKnown := err == nil
var remote models.Remote
if oldKnown {
remote.MutableTTL = existing.MutableTTL
}
if err := json.NewDecoder(r.Body).Decode(&remote); err != nil {
http.Error(w, "invalid json", http.StatusBadRequest)
return
@@ -132,13 +144,6 @@ func (h *RemotesHandler) update(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// Capture the current backend before the update so we can tell whether the
// remote's base_url (its upstream) changed. A read failure just means we
// skip the freshness flush; it must not block the update.
oldBaseURL, oldKnown := "", false
if existing, err := h.db.GetRemote(r.Context(), name); err == nil {
oldBaseURL, oldKnown = existing.BaseURL, true
}
if err := h.db.UpdateRemote(r.Context(), &remote); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
@@ -147,13 +152,13 @@ func (h *RemotesHandler) update(w http.ResponseWriter, r *http.Request) {
// Release / APKINDEX): purge it so the next request re-fetches from the new
// upstream instead of serving stale data until TTL expiry. A flush failure
// is logged but does not fail the request — the DB update already landed.
if oldKnown && oldBaseURL != remote.BaseURL && h.cache != nil {
if oldKnown && existing.BaseURL != remote.BaseURL && h.cache != nil {
if err := h.cache.FlushRemote(r.Context(), name); err != nil {
slog.Warn("flush cached metadata after base_url change failed",
"remote", name, "error", err)
} else {
slog.Info("flushed cached metadata after base_url change",
"remote", name, "old_base_url", oldBaseURL, "new_base_url", remote.BaseURL)
"remote", name, "old_base_url", existing.BaseURL, "new_base_url", remote.BaseURL)
}
}
writeJSON(w, http.StatusOK, remote)
@@ -165,6 +170,11 @@ func (h *RemotesHandler) del(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if h.cache != nil {
if err := h.cache.FlushRemote(r.Context(), name); err != nil {
slog.Warn("flush cached metadata after delete failed", "remote", name, "error", err)
}
}
w.WriteHeader(http.StatusNoContent)
}
+25
View File
@@ -94,3 +94,28 @@ func TestUpdateFlushFailureStillSucceeds(t *testing.T) {
t.Fatalf("flush calls = %v, want exactly one attempted flush", ff.calls)
}
}
// Deleting a remote flushes its cached state so a re-created remote of the
// same name starts cold; a flush error does not fail the delete.
func TestDeleteFlushesCache(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
for _, ff := range []*fakeFlusher{{}, {err: errors.New("redis down")}} {
const name = "rpm-flush-delete"
seedRemote(t, db, name, "https://example.com/repo")
h := NewRemotesHandler(db, ff, nil).Routes()
if c := do(t, h, "DELETE", "/"+name, ""); c != 204 {
t.Fatalf("delete = %d, want 204", c)
}
if len(ff.calls) != 1 || ff.calls[0] != name {
t.Fatalf("flush calls = %v, want exactly one flush of %q", ff.calls, name)
}
}
}
+57
View File
@@ -0,0 +1,57 @@
package v2
import (
"context"
"testing"
"git.unkin.net/unkin/artifactapi/internal/database"
)
// An omitted mutable_ttl defaults to 3600 on create and keeps the stored value
// on update; an explicit 0 is stored as 0 in both.
func TestMutableTTLOmittedVsZero(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
h := NewRemotesHandler(db, nil, nil).Routes()
ttl := func(name string) int {
t.Helper()
r, err := db.GetRemote(context.Background(), name)
if err != nil {
t.Fatalf("get %s: %v", name, err)
}
return r.MutableTTL
}
const body = `"package_type":"rpm","base_url":"https://example.com/repo"`
steps := []struct {
method, path, json string
want int
}{
{"POST", "/", `{"name":"ttl-omit",` + body + `}`, 3600},
{"PUT", "/ttl-omit", `{` + body + `,"mutable_ttl":120}`, 120},
{"PUT", "/ttl-omit", `{` + body + `}`, 120},
{"PUT", "/ttl-omit", `{` + body + `,"mutable_ttl":0}`, 0},
{"PUT", "/ttl-omit", `{` + body + `}`, 0},
}
for i, s := range steps {
if c := do(t, h, s.method, s.path, s.json); c/100 != 2 {
t.Fatalf("step %d %s %s = %d", i, s.method, s.path, c)
}
if got := ttl("ttl-omit"); got != s.want {
t.Errorf("step %d: mutable_ttl = %d, want %d", i, got, s.want)
}
}
if c := do(t, h, "POST", "/", `{"name":"ttl-zero",`+body+`,"mutable_ttl":0}`); c != 201 {
t.Fatalf("create explicit 0 = %d", c)
}
if got := ttl("ttl-zero"); got != 0 {
t.Errorf("explicit 0 stored as %d", got)
}
}
+158
View File
@@ -0,0 +1,158 @@
package auth
import (
"context"
"errors"
"log/slog"
"net/http"
"slices"
"strings"
"sync"
"time"
"github.com/coreos/go-oidc/v3/oidc"
)
// BearerOIDC verifies `Authorization: Bearer <jwt>` against an OIDC issuer's
// JWKS. Discovery is lazy so startup never depends on the issuer being
// reachable; a failed discovery is not retried until the backoff elapses.
type BearerOIDC struct {
issuer string
audiences []string
timeout time.Duration
backoff time.Duration
discover func(context.Context, string) (*oidc.Provider, error)
mu sync.Mutex
verifier *oidc.IDTokenVerifier
inflight chan struct{}
failedAt time.Time
}
var errIssuerUnavailable = errors.New("oidc issuer unavailable")
// NewBearerOIDC returns nil when issuer is empty, meaning auth is disabled.
func NewBearerOIDC(issuer string, audiences []string) *BearerOIDC {
if issuer == "" {
return nil
}
return &BearerOIDC{issuer: issuer, audiences: audiences, timeout: 10 * time.Second, backoff: 15 * time.Second, discover: oidc.NewProvider}
}
// getVerifier runs at most one discovery at a time without holding the lock
// across network I/O; concurrent callers wait for it, and only callers within
// the post-failure backoff fail fast.
func (b *BearerOIDC) getVerifier(ctx context.Context) (*oidc.IDTokenVerifier, error) {
b.mu.Lock()
if v := b.verifier; v != nil {
b.mu.Unlock()
return v, nil
}
if done := b.inflight; done != nil {
b.mu.Unlock()
select {
case <-done:
case <-ctx.Done():
return nil, ctx.Err()
}
b.mu.Lock()
defer b.mu.Unlock()
if b.verifier == nil {
return nil, errIssuerUnavailable
}
return b.verifier, nil
}
if time.Since(b.failedAt) < b.backoff {
b.mu.Unlock()
return nil, errIssuerUnavailable
}
done := make(chan struct{})
b.inflight = done
b.mu.Unlock()
v, err := b.discoverOnce(done)
if err != nil {
slog.Error("oidc discovery failed", "issuer", b.issuer, "error", err)
return nil, err
}
return v, nil
}
// discoverOnce releases inflight and waiters on every exit, a panic included,
// recording failedAt unless discovery succeeded.
func (b *BearerOIDC) discoverOnce(done chan struct{}) (v *oidc.IDTokenVerifier, err error) {
defer func() {
b.mu.Lock()
b.inflight = nil
if v == nil {
b.failedAt = time.Now()
}
b.verifier = v
b.mu.Unlock()
close(done)
}()
// The provider keeps this context and client for JWKS refreshes, so the
// deadline lives on the client rather than the context.
client := &http.Client{Timeout: b.timeout}
p, err := b.discover(oidc.ClientContext(context.Background(), client), b.issuer)
if err != nil {
return nil, err
}
return p.Verifier(&oidc.Config{SkipClientIDCheck: true}), nil
}
// Require gates every request through next on a valid token. A nil receiver
// passes everything through.
func (b *BearerOIDC) Require(next http.Handler) http.Handler {
if b == nil {
return next
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
scheme, raw, _ := strings.Cut(strings.TrimSpace(r.Header.Get("Authorization")), " ")
raw = strings.TrimSpace(raw)
if !strings.EqualFold(scheme, "Bearer") || raw == "" {
unauthorized(w, "missing bearer token")
return
}
v, err := b.getVerifier(r.Context())
if err != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
_, _ = w.Write([]byte(`{"error":"token issuer unavailable"}`))
return
}
tok, err := v.Verify(r.Context(), raw)
if err != nil {
slog.Info("bearer token rejected", "error", err)
unauthorized(w, "invalid token")
return
}
if len(b.audiences) > 0 && !slices.ContainsFunc(tok.Audience, func(a string) bool { return slices.Contains(b.audiences, a) }) {
slog.Info("bearer token rejected", "error", "audience not allowed", "aud", tok.Audience)
unauthorized(w, "invalid token")
return
}
slog.Info("bearer token accepted", "sub", tok.Subject, "method", r.Method, "path", r.URL.Path)
next.ServeHTTP(w, r)
})
}
// RequireWrites gates only mutating methods.
func (b *BearerOIDC) RequireWrites(next http.Handler) http.Handler {
gated := b.Require(next)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet, http.MethodHead, http.MethodOptions:
next.ServeHTTP(w, r)
default:
gated.ServeHTTP(w, r)
}
})
}
func unauthorized(w http.ResponseWriter, msg string) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("WWW-Authenticate", "Bearer")
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte(`{"error":"` + msg + `"}`))
}
+255
View File
@@ -0,0 +1,255 @@
package auth
import (
"context"
"crypto/rand"
"crypto/rsa"
"encoding/base64"
"net/http"
"net/http/httptest"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/coreos/go-oidc/v3/oidc"
"github.com/go-jose/go-jose/v4"
"github.com/go-jose/go-jose/v4/jwt"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
func serve(h http.Handler, method, token string) int {
if token == "" {
return serveAuth(h, method, "")
}
return serveAuth(h, method, "Bearer "+token)
}
func serveAuth(h http.Handler, method, authz string) int {
r := httptest.NewRequest(method, "/x", nil)
if authz != "" {
r.Header.Set("Authorization", authz)
}
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
return w.Code
}
func claims(iss string) jwt.Claims {
now := time.Now()
return jwt.Claims{
Issuer: iss,
Subject: "ci",
Audience: jwt.Audience{"artifactapi"},
IssuedAt: jwt.NewNumericDate(now.Add(-time.Minute)),
Expiry: jwt.NewNumericDate(now.Add(time.Hour)),
}
}
func hs256Token(t *testing.T, c jwt.Claims) string {
sig, err := jose.NewSigner(jose.SigningKey{Algorithm: jose.HS256, Key: []byte("0123456789abcdef0123456789abcdef")},
(&jose.SignerOptions{}).WithType("JWT").WithHeader("kid", "k1"))
if err != nil {
t.Fatal(err)
}
tok, err := jwt.Signed(sig).Claims(c).Serialize()
if err != nil {
t.Fatal(err)
}
return tok
}
func noneToken(iss string) string {
enc := base64.RawURLEncoding.EncodeToString
return enc([]byte(`{"alg":"none","typ":"JWT","kid":"k1"}`)) + "." +
enc([]byte(`{"iss":"`+iss+`","sub":"ci","aud":"artifactapi","exp":`+strconv.FormatInt(time.Now().Add(time.Hour).Unix(), 10)+`}`)) + "."
}
var ok = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) })
func TestBearerOIDC(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
h := NewBearerOIDC(iss.URL, []string{"other", "artifactapi"}).Require(ok)
otherKey, _ := rsa.GenerateKey(rand.Reader, 2048)
cases := []struct {
name string
token string
want int
}{
{"valid", iss.Token(nil, "artifactapi", "ci", time.Hour), http.StatusNoContent},
{"missing", "", http.StatusUnauthorized},
{"garbage", "not.a.jwt", http.StatusUnauthorized},
{"bad signature", iss.Token(otherKey, "artifactapi", "ci", time.Hour), http.StatusUnauthorized},
{"wrong aud", iss.Token(nil, "nope", "ci", time.Hour), http.StatusUnauthorized},
{"expired", iss.Token(nil, "artifactapi", "ci", -time.Hour), http.StatusUnauthorized},
{"wrong iss", iss.Sign(nil, claims("https://evil.example")), http.StatusUnauthorized},
{"hs256", hs256Token(t, claims(iss.URL)), http.StatusUnauthorized},
{"alg none", noneToken(iss.URL), http.StatusUnauthorized},
}
for _, c := range cases {
if got := serve(h, http.MethodPost, c.token); got != c.want {
t.Errorf("%s: status %d, want %d", c.name, got, c.want)
}
}
valid := iss.Token(nil, "artifactapi", "ci", time.Hour)
headers := []struct {
name string
authz string
want int
}{
{"lowercase scheme", "bearer " + valid, http.StatusNoContent},
{"extra whitespace", " Bearer " + valid + " ", http.StatusNoContent},
{"empty value", "Bearer ", http.StatusUnauthorized},
{"scheme only", "Bearer", http.StatusUnauthorized},
{"basic scheme", "Basic " + valid, http.StatusUnauthorized},
}
for _, c := range headers {
if got := serveAuth(h, http.MethodPost, c.authz); got != c.want {
t.Errorf("%s: status %d, want %d", c.name, got, c.want)
}
}
}
func TestBearerOIDCHangingIssuerBacksOff(t *testing.T) {
var hits atomic.Int32
release := make(chan struct{})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits.Add(1)
select {
case <-r.Context().Done():
case <-release:
}
}))
defer srv.Close()
defer close(release)
b := NewBearerOIDC(srv.URL, nil)
b.timeout = 100 * time.Millisecond
b.backoff = time.Hour
h := b.Require(ok)
start := time.Now()
if got := serve(h, http.MethodPost, "x.y.z"); got != http.StatusServiceUnavailable {
t.Fatalf("hanging issuer: status %d, want 503", got)
}
if d := time.Since(start); d > 2*time.Second {
t.Fatalf("hanging issuer: took %s, timeout not applied", d)
}
if got := serve(h, http.MethodPost, "x.y.z"); got != http.StatusServiceUnavailable {
t.Fatalf("during backoff: status %d, want 503", got)
}
if n := hits.Load(); n != 1 {
t.Fatalf("issuer hit %d times, want 1", n)
}
}
func TestBearerOIDCPanickingDiscoveryBacksOff(t *testing.T) {
b := NewBearerOIDC("http://issuer.invalid", nil)
b.backoff = time.Hour
b.discover = func(context.Context, string) (*oidc.Provider, error) { panic("boom") }
h := b.Require(ok)
func() {
defer func() { _ = recover() }()
serve(h, http.MethodPost, "x.y.z")
}()
got := make(chan int, 1)
go func() { got <- serve(h, http.MethodPost, "x.y.z") }()
select {
case code := <-got:
if code != http.StatusServiceUnavailable {
t.Fatalf("after panic: status %d, want 503", code)
}
case <-time.After(2 * time.Second):
t.Fatal("after panic: request hung on stale inflight discovery")
}
}
func TestBearerOIDCConcurrentFirstRequestsWaitForDiscovery(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
var hits atomic.Int32
inner := iss.Config.Handler
iss.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/.well-known/openid-configuration" {
hits.Add(1)
time.Sleep(200 * time.Millisecond)
}
inner.ServeHTTP(w, r)
})
h := NewBearerOIDC(iss.URL, nil).Require(ok)
tok := iss.Token(nil, "artifactapi", "ci", time.Hour)
var wg sync.WaitGroup
codes := make([]int, 20)
for i := range codes {
wg.Add(1)
go func() {
defer wg.Done()
codes[i] = serve(h, http.MethodPost, tok)
}()
}
wg.Wait()
for i, c := range codes {
if c != http.StatusNoContent {
t.Errorf("request %d: status %d, want 204", i, c)
}
}
if n := hits.Load(); n != 1 {
t.Fatalf("discovery hit %d times, want 1", n)
}
}
func TestBearerOIDCRetriesAfterBackoff(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
b := NewBearerOIDC(iss.URL, nil)
b.failedAt = time.Now()
b.backoff = 50 * time.Millisecond
tok := iss.Token(nil, "artifactapi", "ci", time.Hour)
if got := serve(b.Require(ok), http.MethodPost, tok); got != http.StatusServiceUnavailable {
t.Fatalf("within backoff: status %d, want 503", got)
}
time.Sleep(60 * time.Millisecond)
if got := serve(b.Require(ok), http.MethodPost, tok); got != http.StatusNoContent {
t.Fatalf("after backoff: status %d, want 204", got)
}
}
func TestBearerOIDCIssuerDown(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
b := NewBearerOIDC(iss.URL, []string{"artifactapi"})
tok := iss.Token(nil, "artifactapi", "ci", time.Hour)
iss.Close()
if got := serve(b.Require(ok), http.MethodPost, tok); got != http.StatusServiceUnavailable {
t.Errorf("issuer down: status %d, want 503", got)
}
}
func TestBearerOIDCDisabled(t *testing.T) {
if got := serve(NewBearerOIDC("", nil).RequireWrites(ok), http.MethodDelete, ""); got != http.StatusNoContent {
t.Errorf("disabled: status %d, want 204", got)
}
}
func TestRequireWritesLeavesReadsOpen(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
h := NewBearerOIDC(iss.URL, []string{"artifactapi"}).RequireWrites(ok)
for _, m := range []string{http.MethodGet, http.MethodHead, http.MethodOptions} {
if got := serve(h, m, ""); got != http.StatusNoContent {
t.Errorf("%s: status %d, want 204", m, got)
}
}
for _, m := range []string{http.MethodPost, http.MethodPut, http.MethodDelete} {
if got := serve(h, m, ""); got != http.StatusUnauthorized {
t.Errorf("%s: status %d, want 401", m, got)
}
}
}
+6
View File
@@ -86,6 +86,12 @@ func TestETagAndToken(t *testing.T) {
if v, _ := testRedis.GetETag(ctx, "r", "p"); v != `"abc"` {
t.Errorf("etag = %q", v)
}
if err := testRedis.SetETag(ctx, "r", "p", "", 0); err != nil {
t.Fatal(err)
}
if v, _ := testRedis.GetETag(ctx, "r", "p"); v != "" {
t.Errorf("empty etag should drop the stored one, got %q", v)
}
if v, _ := testRedis.GetToken(ctx, "missing"); v != "" {
t.Error("missing token should be empty")
+4
View File
@@ -57,8 +57,12 @@ func (r *Redis) ReleaseLock(ctx context.Context, remote, path string) error {
return r.client.Del(ctx, key).Err()
}
// SetETag stores etag for path, or drops the stored one when etag is empty.
func (r *Redis) SetETag(ctx context.Context, remote, path, etag string, ttl time.Duration) error {
key := fmt.Sprintf("etag:%s:%s", remote, path)
if etag == "" {
return r.client.Del(ctx, key).Err()
}
return r.client.Set(ctx, key, etag, ttl).Err()
}
+19
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"os"
"strconv"
"strings"
)
type Config struct {
@@ -56,6 +57,10 @@ type Config struct {
GitHubAppInstallationID string
GitHubAppPrivateKey string
GitHubAppPrivateKeyPath string
// Bearer-JWT auth on configuration writes; empty issuer disables it.
OIDCIssuer string
OIDCAudiences []string
}
func (c *Config) DatabaseDSN() string {
@@ -90,6 +95,17 @@ func Load() (*Config, error) {
return nil, fmt.Errorf("invalid GITHUB_SYNC_POLL_INTERVAL: %w", err)
}
oidcIssuer := getenv("ARTIFACTAPI_OIDC_ISSUER", "")
var oidcAudiences []string
for _, a := range strings.Split(getenv("ARTIFACTAPI_OIDC_AUDIENCE", ""), ",") {
if a = strings.TrimSpace(a); a != "" {
oidcAudiences = append(oidcAudiences, a)
}
}
if oidcIssuer != "" && len(oidcAudiences) == 0 {
return nil, fmt.Errorf("ARTIFACTAPI_OIDC_AUDIENCE is required when ARTIFACTAPI_OIDC_ISSUER is set")
}
cfg := &Config{
ListenAddr: getenv("LISTEN_ADDR", ":8000"),
@@ -123,6 +139,9 @@ func Load() (*Config, error) {
GitHubAppInstallationID: getenv("GITHUB_APP_INSTALLATION_ID", ""),
GitHubAppPrivateKey: getenv("GITHUB_APP_PRIVATE_KEY", ""),
GitHubAppPrivateKeyPath: getenv("GITHUB_APP_PRIVATE_KEY_PATH", ""),
OIDCIssuer: oidcIssuer,
OIDCAudiences: oidcAudiences,
}
return cfg, nil
+17
View File
@@ -64,3 +64,20 @@ func TestLoadInvalidPort(t *testing.T) {
t.Error("expected error for invalid DBPORT")
}
}
func TestLoadOIDC(t *testing.T) {
t.Setenv("ARTIFACTAPI_OIDC_ISSUER", "https://vault.example/v1/identity/oidc")
t.Setenv("ARTIFACTAPI_OIDC_AUDIENCE", " a, ,b ")
cfg, err := Load()
if err != nil {
t.Fatalf("load: %v", err)
}
if len(cfg.OIDCAudiences) != 2 || cfg.OIDCAudiences[0] != "a" || cfg.OIDCAudiences[1] != "b" {
t.Errorf("audiences = %q", cfg.OIDCAudiences)
}
t.Setenv("ARTIFACTAPI_OIDC_AUDIENCE", "")
if _, err := Load(); err == nil {
t.Error("expected error for issuer without audience")
}
}
+8 -38
View File
@@ -2,11 +2,9 @@ package database
import (
"context"
"errors"
"time"
"github.com/jackc/pgx/v5"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -30,42 +28,14 @@ func (db *DB) ListGitHubAlpineRemotes(ctx context.Context) ([]models.Remote, err
return remotes, rows.Err()
}
// ClaimGitHubAlpineSyncLease atomically claims the per-remote sync lease. It
// succeeds only when the remote is due (never synced, or synced longer than
// freshness ago) and no live lease is held by another replica. A zero freshness
// (prime scans) ignores the recency gate. The returned etag is the stored
// releases-list ETag, shared across replicas.
// ClaimGitHubAlpineSyncLease atomically claims the per-remote github_alpine sync
// lease. See claimSyncLease.
func (db *DB) ClaimGitHubAlpineSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
row := db.Pool.QueryRow(ctx, `
INSERT INTO github_alpine_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
VALUES ($1, $2, now() + make_interval(secs => $4))
ON CONFLICT (remote_name) DO UPDATE
SET sync_lease_owner = $2,
sync_lease_expires = now() + make_interval(secs => $4)
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
RETURNING s.etag
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
var etag string
if err := row.Scan(&etag); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return false, "", nil
}
return false, "", err
}
return true, etag, nil
return db.claimSyncLease(ctx, "github_alpine_sync_state", remoteName, owner, freshness, lease)
}
// ReleaseGitHubAlpineSyncLease records the completed scan and frees the lease.
// Only the owning replica may release; last_synced_at advances so the next poll
// waits a full freshness window, and etag is persisted for the next conditional
// request.
func (db *DB) ReleaseGitHubAlpineSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
_, err := db.Pool.Exec(ctx, `
UPDATE github_alpine_sync_state
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
WHERE remote_name = $1 AND sync_lease_owner = $2
`, remoteName, owner, syncedAt, etag)
return err
// ReleaseGitHubAlpineSyncLease records a github_alpine scan outcome and frees the
// lease. See releaseSyncLease.
func (db *DB) ReleaseGitHubAlpineSyncLease(ctx context.Context, remoteName, owner string, res provider.SyncResult) error {
return db.releaseSyncLease(ctx, "github_alpine_sync_state", remoteName, owner, res)
}
+8 -37
View File
@@ -2,11 +2,9 @@ package database
import (
"context"
"errors"
"time"
"github.com/jackc/pgx/v5"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -30,41 +28,14 @@ func (db *DB) ListGitHubDebRemotes(ctx context.Context) ([]models.Remote, error)
return remotes, rows.Err()
}
// ClaimGitHubDebSyncLease atomically claims the per-remote sync lease. It
// succeeds only when the remote is due (never synced, or synced longer than
// freshness ago) and no live lease is held by another replica. A zero freshness
// (prime scans) ignores the recency gate. The returned etag is the stored
// releases-list ETag, shared across replicas.
// ClaimGitHubDebSyncLease atomically claims the per-remote github_deb sync
// lease. See claimSyncLease.
func (db *DB) ClaimGitHubDebSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
row := db.Pool.QueryRow(ctx, `
INSERT INTO github_deb_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
VALUES ($1, $2, now() + make_interval(secs => $4))
ON CONFLICT (remote_name) DO UPDATE
SET sync_lease_owner = $2,
sync_lease_expires = now() + make_interval(secs => $4)
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
RETURNING s.etag
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
var etag string
if err := row.Scan(&etag); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return false, "", nil
}
return false, "", err
}
return true, etag, nil
return db.claimSyncLease(ctx, "github_deb_sync_state", remoteName, owner, freshness, lease)
}
// ReleaseGitHubDebSyncLease records the completed scan and frees the lease. Only
// the owning replica may release; last_synced_at advances so the next poll waits
// a full freshness window, and etag is persisted for the next conditional request.
func (db *DB) ReleaseGitHubDebSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
_, err := db.Pool.Exec(ctx, `
UPDATE github_deb_sync_state
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
WHERE remote_name = $1 AND sync_lease_owner = $2
`, remoteName, owner, syncedAt, etag)
return err
// ReleaseGitHubDebSyncLease records a github_deb scan outcome and frees the
// lease. See releaseSyncLease.
func (db *DB) ReleaseGitHubDebSyncLease(ctx context.Context, remoteName, owner string, res provider.SyncResult) error {
return db.releaseSyncLease(ctx, "github_deb_sync_state", remoteName, owner, res)
}
+2 -1
View File
@@ -4,6 +4,7 @@ import (
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -44,7 +45,7 @@ func TestGitHubDebSyncLease(t *testing.T) {
t.Fatal("replica-2 claimed while replica-1 holds the lease")
}
if err := testDB.ReleaseGitHubDebSyncLease(ctx(), name, "replica-1", `"etag-1"`, time.Now()); err != nil {
if err := testDB.ReleaseGitHubDebSyncLease(ctx(), name, "replica-1", provider.SyncResult{Etag: `"etag-1"`}); err != nil {
t.Fatalf("release: %v", err)
}
+45 -17
View File
@@ -7,6 +7,8 @@ import (
"github.com/jackc/pgx/v5"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -30,22 +32,35 @@ func (db *DB) ListGitHubRPMRemotes(ctx context.Context) ([]models.Remote, error)
return remotes, rows.Err()
}
// ClaimGitHubSyncLease atomically claims the per-remote sync lease. It succeeds
// (claimed=true) only when the remote is due — never synced, or synced longer
// than freshness ago — and no live lease is held by another replica. This bounds
// total GitHub load to roughly one scan per freshness window regardless of how
// many replicas poll. The returned etag is the stored releases-list ETag, shared
// across replicas so a conditional request can short-circuit an unchanged repo.
// A zero freshness (used for prime scans) ignores the recency gate and claims
// whenever no live lease is held.
// ClaimGitHubSyncLease atomically claims the per-remote github_rpm sync lease.
// See claimSyncLease.
func (db *DB) ClaimGitHubSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
return db.claimSyncLease(ctx, "github_rpm_sync_state", remoteName, owner, freshness, lease)
}
// ReleaseGitHubSyncLease records a github_rpm scan outcome and frees the lease.
// See releaseSyncLease.
func (db *DB) ReleaseGitHubSyncLease(ctx context.Context, remoteName, owner string, res provider.SyncResult) error {
return db.releaseSyncLease(ctx, "github_rpm_sync_state", remoteName, owner, res)
}
// claimSyncLease atomically claims a per-remote sync lease in table. It succeeds
// (claimed=true) only when the remote is due and no live lease is held by
// another replica. Due means: a pending retry after a failed scan has reached
// next_retry_at, or, with no retry pending, the remote was never synced or was
// synced longer than freshness ago. A zero freshness (prime scans) ignores the
// recency gate but still honours a pending retry's backoff. The returned etag is
// the stored releases-list ETag, shared across replicas so a conditional request
// can short-circuit an unchanged repo.
func (db *DB) claimSyncLease(ctx context.Context, table, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
row := db.Pool.QueryRow(ctx, `
INSERT INTO github_rpm_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
INSERT INTO `+table+` AS s (remote_name, sync_lease_owner, sync_lease_expires)
VALUES ($1, $2, now() + make_interval(secs => $4))
ON CONFLICT (remote_name) DO UPDATE
SET sync_lease_owner = $2,
sync_lease_expires = now() + make_interval(secs => $4)
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
WHERE (CASE WHEN s.next_retry_at IS NOT NULL THEN s.next_retry_at <= now()
ELSE s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3) END)
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
RETURNING s.etag
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
@@ -60,14 +75,27 @@ func (db *DB) ClaimGitHubSyncLease(ctx context.Context, remoteName, owner string
return true, etag, nil
}
// ReleaseGitHubSyncLease records the completed scan and frees the lease. Only the
// owning replica may release; last_synced_at advances so the next poll waits a
// full freshness window, and etag is persisted for the next conditional request.
func (db *DB) ReleaseGitHubSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
// releaseSyncLease records a scan outcome and frees the lease; only the owning
// replica may release. Success advances last_synced_at, persists the etag and
// clears any retry. Failure leaves last_synced_at and etag untouched (the last
// good metadata keeps serving) and schedules next_retry_at with exponential
// backoff from the consecutive-failure count, never before res.RetryAt.
func (db *DB) releaseSyncLease(ctx context.Context, table, remoteName, owner string, res provider.SyncResult) error {
var retryAt *time.Time
if !res.RetryAt.IsZero() {
retryAt = &res.RetryAt
}
_, err := db.Pool.Exec(ctx, `
UPDATE github_rpm_sync_state
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
UPDATE `+table+`
SET last_synced_at = CASE WHEN $3 THEN last_synced_at ELSE now() END,
etag = CASE WHEN $3 THEN etag ELSE $4 END,
sync_failures = CASE WHEN $3 THEN sync_failures + 1 ELSE 0 END,
next_retry_at = CASE WHEN $3 THEN GREATEST(
now() + make_interval(secs => LEAST($5 * power(2, LEAST(sync_failures, 20)), $6)),
$7::timestamptz)
END,
sync_lease_owner = '', sync_lease_expires = NULL
WHERE remote_name = $1 AND sync_lease_owner = $2
`, remoteName, owner, syncedAt, etag)
`, remoteName, owner, res.Failed, res.Etag, res.Backoff.Seconds(), res.MaxBackoff.Seconds(), retryAt)
return err
}
+115 -1
View File
@@ -4,6 +4,7 @@ import (
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -47,7 +48,7 @@ func TestGitHubSyncLease(t *testing.T) {
}
// Replica 1 finishes: record the sync and persist an etag.
if err := testDB.ReleaseGitHubSyncLease(ctx(), name, "replica-1", `"etag-1"`, time.Now()); err != nil {
if err := testDB.ReleaseGitHubSyncLease(ctx(), name, "replica-1", provider.SyncResult{Etag: `"etag-1"`}); err != nil {
t.Fatalf("release: %v", err)
}
@@ -93,3 +94,116 @@ func TestListGitHubRPMRemotes(t *testing.T) {
t.Fatalf("seeded remote %q not returned", name)
}
}
// A failed scan must not count as a sync: the etag and last_synced_at stay put,
// every claim (prime included) is held off until the backoff elapses, the delay
// doubles per consecutive failure up to the cap, an upstream retry hint pushes
// it later, and a success clears the retry state.
func TestGitHubSyncLeaseFailureBackoff(t *testing.T) {
requireDB(t)
name := "gh-retry-" + time.Now().Format("150405.000000")
seedGitHubRPMRemote(t, name)
const lease = 15 * time.Minute
freshness := time.Hour
claim := func(f time.Duration) (bool, string) {
t.Helper()
ok, etag, err := testDB.ClaimGitHubSyncLease(ctx(), name, "r1", f, lease)
if err != nil {
t.Fatalf("claim: %v", err)
}
return ok, etag
}
release := func(res provider.SyncResult) {
t.Helper()
if err := testDB.ReleaseGitHubSyncLease(ctx(), name, "r1", res); err != nil {
t.Fatalf("release: %v", err)
}
}
state := func() (failures int, retryIn time.Duration, synced *time.Time, etag string) {
t.Helper()
var next *time.Time
var dbNow time.Time
if err := testDB.Pool.QueryRow(ctx(), `SELECT sync_failures, next_retry_at, last_synced_at, etag, now() FROM github_rpm_sync_state WHERE remote_name = $1`, name).
Scan(&failures, &next, &synced, &etag, &dbNow); err != nil {
t.Fatalf("read state: %v", err)
}
if next != nil {
retryIn = next.Sub(dbNow)
}
return
}
elapse := func() {
t.Helper()
if _, err := testDB.Pool.Exec(ctx(), `UPDATE github_rpm_sync_state SET next_retry_at = now() - interval '1 second' WHERE remote_name = $1`, name); err != nil {
t.Fatalf("elapse backoff: %v", err)
}
}
failed := provider.SyncResult{Failed: true, Etag: `"ignored"`, Backoff: time.Minute, MaxBackoff: 3 * time.Minute}
// Last good sync.
if ok, _ := claim(freshness); !ok {
t.Fatal("first claim failed")
}
release(provider.SyncResult{Etag: `"good"`})
_, _, goodSynced, _ := state()
// First failure: 60s backoff, sync time and etag untouched.
if ok, _ := claim(0); !ok {
t.Fatal("prime claim failed")
}
release(failed)
n, in, synced, etag := state()
if n != 1 || in < 55*time.Second || in > 65*time.Second {
t.Fatalf("after 1st failure: failures=%d retry in %v, want 1 and ~60s", n, in)
}
if etag != `"good"` || synced == nil || !synced.Equal(*goodSynced) {
t.Fatalf("failed scan overwrote sync state: etag=%q synced=%v", etag, synced)
}
if ok, _ := claim(0); ok {
t.Fatal("prime claimed inside the retry backoff")
}
// Backoff elapsed: claimable despite being inside the freshness window,
// and the second failure doubles the delay.
elapse()
if ok, _ := claim(freshness); !ok {
t.Fatal("periodic claim refused after backoff elapsed")
}
release(failed)
if n, in, _, _ := state(); n != 2 || in < 115*time.Second || in > 125*time.Second {
t.Fatalf("after 2nd failure: failures=%d retry in %v, want 2 and ~120s", n, in)
}
// Third failure caps at MaxBackoff (3m, not 4m).
elapse()
claim(freshness)
release(failed)
if n, in, _, _ := state(); n != 3 || in < 175*time.Second || in > 185*time.Second {
t.Fatalf("after 3rd failure: failures=%d retry in %v, want 3 and ~180s", n, in)
}
// An upstream rate-limit reset later than the backoff wins.
elapse()
claim(freshness)
hinted := failed
hinted.RetryAt = time.Now().Add(20 * time.Minute)
release(hinted)
if _, in, _, _ := state(); in < 19*time.Minute || in > 21*time.Minute {
t.Fatalf("retry hint ignored: retry in %v, want ~20m", in)
}
// Recovery: success persists the new etag and clears the retry, so the
// normal freshness window applies again.
elapse()
if ok, etag := claim(freshness); !ok || etag != `"good"` {
t.Fatalf("recovery claim ok=%v etag=%q, want last good etag", ok, etag)
}
release(provider.SyncResult{Etag: `"new"`})
if n, in, _, etag := state(); n != 0 || in != 0 || etag != `"new"` {
t.Fatalf("after success: failures=%d retry in %v etag=%q", n, in, etag)
}
if ok, _ := claim(freshness); ok {
t.Fatal("claimed inside freshness window after a successful sync")
}
}
+44
View File
@@ -0,0 +1,44 @@
package database
import (
"context"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func (db *DB) InsertHelmMetadata(ctx context.Context, meta *provider.HelmMetadata) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO helm_metadata (repo_name, file_path, content_hash, name, version, chart_yaml)
VALUES ($1,$2,$3,$4,$5,$6)
ON CONFLICT (repo_name, file_path) DO NOTHING
`, meta.RepoName, meta.FilePath, meta.ContentHash, meta.Name, meta.Version, meta.ChartYAML)
return err
}
func (db *DB) DeleteHelmMetadata(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM helm_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
func (db *DB) ListHelmMetadataEntries(ctx context.Context, repoName string) ([]provider.HelmMetadata, error) {
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash, name, version, chart_yaml, created_at
FROM helm_metadata
WHERE repo_name = $1
ORDER BY name, version, file_path
`, repoName)
if err != nil {
return nil, err
}
defer rows.Close()
var result []provider.HelmMetadata
for rows.Next() {
var m provider.HelmMetadata
if err := rows.Scan(&m.RepoName, &m.FilePath, &m.ContentHash, &m.Name, &m.Version, &m.ChartYAML, &m.CreatedAt); err != nil {
return nil, err
}
result = append(result, m)
}
return result, rows.Err()
}
+46
View File
@@ -0,0 +1,46 @@
package database
import (
"context"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func (db *DB) InsertPuppetMetadata(ctx context.Context, meta *provider.PuppetMetadata) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO puppet_metadata (repo_name, file_path, content_hash, slug, version, metadata_json, file_size, file_md5)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8)
ON CONFLICT (repo_name, file_path) DO NOTHING
`, meta.RepoName, meta.FilePath, meta.ContentHash, meta.Slug, meta.Version, meta.MetadataJSON, meta.Size, meta.MD5)
return err
}
func (db *DB) DeletePuppetMetadata(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM puppet_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
// ListPuppetMetadataEntries returns a repo's releases, optionally narrowed to
// one module slug (empty slug lists every module).
func (db *DB) ListPuppetMetadataEntries(ctx context.Context, repoName, slug string) ([]provider.PuppetMetadata, error) {
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash, slug, version, metadata_json, file_size, file_md5, created_at
FROM puppet_metadata
WHERE repo_name = $1 AND ($2 = '' OR slug = $2)
ORDER BY slug, file_path
`, repoName, slug)
if err != nil {
return nil, err
}
defer rows.Close()
var result []provider.PuppetMetadata
for rows.Next() {
var m provider.PuppetMetadata
if err := rows.Scan(&m.RepoName, &m.FilePath, &m.ContentHash, &m.Slug, &m.Version, &m.MetadataJSON, &m.Size, &m.MD5, &m.CreatedAt); err != nil {
return nil, err
}
result = append(result, m)
}
return result, rows.Err()
}
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/alpine"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/cargo"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/deb"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/docker"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/gems"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/helm"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/npm"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/nuget"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/puppet"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/pypi"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/rpm"
+3
View File
@@ -0,0 +1,3 @@
package all
import _ "git.unkin.net/unkin/artifactapi/internal/provider/terraform"
+1 -1
View File
@@ -470,7 +470,7 @@ func (p *GitHubProvider) fetchReleases(ctx context.Context, remote models.Remote
return nil, "", false, err
}
if resp.StatusCode != http.StatusOK {
return nil, "", false, fmt.Errorf("github releases API %s: status %d", u, resp.StatusCode)
return nil, "", false, provider.NewUpstreamStatusError("github releases API "+u, resp)
}
if page == 1 {
newEtag = respEtag
+7 -9
View File
@@ -28,7 +28,7 @@ type SyncStore interface {
provider.RemoteMetadataStore
ListGitHubAlpineRemotes(ctx context.Context) ([]models.Remote, error)
ClaimGitHubAlpineSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (claimed bool, etag string, err error)
ReleaseGitHubAlpineSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error
ReleaseGitHubAlpineSyncLease(ctx context.Context, remoteName, owner string, res provider.SyncResult) error
}
// SyncConfig tunes the shared syncer. Zero values fall back to safe defaults.
@@ -189,10 +189,11 @@ func (s *Syncer) process(ctx context.Context, job syncJob) {
s.mu.Unlock()
}()
freshness := time.Duration(job.remote.MutableTTL) * time.Second
if freshness <= 0 {
freshness = defaultSyncFreshness
ttl := time.Duration(job.remote.MutableTTL) * time.Second
if ttl <= 0 {
ttl = defaultSyncFreshness
}
freshness := ttl
if job.prime {
freshness = 0
}
@@ -210,16 +211,13 @@ func (s *Syncer) process(ctx context.Context, job syncJob) {
defer cancel()
newEtag, changed, scanErr := s.prov.scanWithState(scanCtx, job.remote, s.store, etag)
releaseEtag := etag
if scanErr == nil {
releaseEtag = newEtag
} else {
if scanErr != nil {
slog.Error("github_alpine syncer: scan failed", "remote", job.remote.Name, "error", scanErr)
}
relCtx, relCancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer relCancel()
if err := s.store.ReleaseGitHubAlpineSyncLease(relCtx, job.remote.Name, s.owner, releaseEtag, time.Now()); err != nil {
if err := s.store.ReleaseGitHubAlpineSyncLease(relCtx, job.remote.Name, s.owner, provider.NewSyncResult(newEtag, scanErr, ttl)); err != nil {
slog.Warn("github_alpine syncer: release lease", "remote", job.remote.Name, "error", err)
}
+13 -3
View File
@@ -27,6 +27,7 @@ type fakeSyncStore struct {
leaseExp map[string]time.Time
lastSynced map[string]time.Time
etags map[string]string
retryAt map[string]time.Time
}
func newFakeSyncStore() *fakeSyncStore {
@@ -36,6 +37,7 @@ func newFakeSyncStore() *fakeSyncStore {
leaseExp: map[string]time.Time{},
lastSynced: map[string]time.Time{},
etags: map[string]string{},
retryAt: map[string]time.Time{},
}
}
@@ -52,6 +54,9 @@ func (f *fakeSyncStore) ClaimGitHubAlpineSyncLease(_ context.Context, name, owne
ls, hasLS := f.lastSynced[name]
exp, hasExp := f.leaseExp[name]
freshOK := !hasLS || now.Sub(ls) >= freshness
if ra, pending := f.retryAt[name]; pending {
freshOK = !now.Before(ra)
}
leaseOK := !hasExp || exp.Before(now)
if freshOK && leaseOK {
f.leaseOwner[name] = owner
@@ -61,14 +66,19 @@ func (f *fakeSyncStore) ClaimGitHubAlpineSyncLease(_ context.Context, name, owne
return false, "", nil
}
func (f *fakeSyncStore) ReleaseGitHubAlpineSyncLease(_ context.Context, name, owner, etag string, syncedAt time.Time) error {
func (f *fakeSyncStore) ReleaseGitHubAlpineSyncLease(_ context.Context, name, owner string, res provider.SyncResult) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.leaseOwner[name] != owner {
return nil
}
f.lastSynced[name] = syncedAt
f.etags[name] = etag
if res.Failed {
f.retryAt[name] = time.Now().Add(res.Backoff)
} else {
f.lastSynced[name] = time.Now()
f.etags[name] = res.Etag
delete(f.retryAt, name)
}
delete(f.leaseOwner, name)
delete(f.leaseExp, name)
return nil
+15
View File
@@ -0,0 +1,15 @@
package provider
import (
"context"
"io"
)
type FileStore interface {
ListFilesByPrefix(ctx context.Context, repoName, prefix string) ([]FileEntry, error)
ListPackages(ctx context.Context, repoName string) ([]string, error)
}
type BlobReader interface {
Download(ctx context.Context, key string) (io.ReadCloser, int64, error)
}
+10
View File
@@ -0,0 +1,10 @@
package provider
type IndexMerger interface {
MergeIndexes(members []MemberIndex, proxyBaseURL string) ([]byte, error)
}
type MemberIndex struct {
RemoteName string
Body []byte
}
+11
View File
@@ -0,0 +1,11 @@
package provider
import (
"context"
"net/http"
)
type LocalIndexer interface {
ServeLocalIndex(w http.ResponseWriter, r *http.Request, files FileStore, repoName, path string) bool
GenerateLocalIndex(ctx context.Context, files FileStore, repoName, path string) ([]byte, error)
}
+6
View File
@@ -0,0 +1,6 @@
package provider
type LocalUploader interface {
ValidateUpload(filePath string) (storagePath, contentType string, err error)
UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any
}
+11
View File
@@ -0,0 +1,11 @@
package provider
import (
"context"
)
// PostDeleteHook lets a provider clean up derived state (e.g. RPM metadata that
// feeds generated repodata) after a local file is removed.
type PostDeleteHook interface {
AfterDelete(ctx context.Context, repoName, storagePath string, db MetadataDeleter) error
}

Some files were not shown because too many files have changed in this diff Show More