244 lines
11 KiB
Markdown
244 lines
11 KiB
Markdown
# ArtifactAPI
|
|
|
|
Caching proxy for package repositories. Single Go binary, 12 package types, content-addressable storage, managed by Terraform.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Start backing services
|
|
docker compose up -d postgres redis minio
|
|
|
|
# Build and run
|
|
make build
|
|
./bin/artifactapi
|
|
|
|
# Frontend (separate container or dev server)
|
|
cd ui && npm install && npm run dev
|
|
```
|
|
|
|
API: `http://localhost:8000` | Frontend: `http://localhost:5173`
|
|
|
|
## Package Types
|
|
|
|
Providers classify paths automatically. Users only configure what to proxy and TTLs.
|
|
|
|
Classification precedence, TTLs and eviction for every remote: [docs/common/caching.md](docs/common/caching.md).
|
|
|
|
Each type's paths, Terraform examples and local/virtual behaviour are in `docs/<type>/`:
|
|
|
|
- `alpine`: [remote](docs/alpine/remote.md), [remote-github](docs/alpine/remote-github.md)
|
|
- `ansible`: [remote](docs/ansible/remote.md), [local](docs/ansible/local.md)
|
|
- `cargo`: [remote](docs/cargo/remote.md)
|
|
- `deb`: [remote](docs/deb/remote.md), [remote-github](docs/deb/remote-github.md)
|
|
- `docker`: [remote](docs/docker/remote.md), [local](docs/docker/local.md)
|
|
- `generic`: [remote](docs/generic/remote.md)
|
|
- `gems`: [remote](docs/gems/remote.md), [local](docs/gems/local.md), [virtual](docs/gems/virtual.md)
|
|
- `goproxy`: [remote](docs/goproxy/remote.md)
|
|
- `helm`: [remote](docs/helm/remote.md), [local](docs/helm/local.md), [virtual](docs/helm/virtual.md)
|
|
- `npm`: [remote](docs/npm/remote.md), [local](docs/npm/local.md), [virtual](docs/npm/virtual.md)
|
|
- `nuget`: [remote](docs/nuget/remote.md)
|
|
- `puppet`: [remote](docs/puppet/remote.md), [local](docs/puppet/local.md)
|
|
- `pypi`: [remote](docs/pypi/remote.md)
|
|
- `rpm`: [remote](docs/rpm/remote.md), [remote-github](docs/rpm/remote-github.md)
|
|
- `terraform`: [remote](docs/terraform/remote.md), [local](docs/terraform/local.md)
|
|
|
|
### GitHub authentication
|
|
|
|
Anonymous GitHub is capped at **60 requests/hour** and cannot read private
|
|
repositories. Configure a **server-level GitHub credential** to raise the ceiling
|
|
to roughly **5000 requests/hour** and to read private-repo release assets. The
|
|
credential is a process-wide machine identity applied by default to *every*
|
|
outbound GitHub request — the releases scan, the ranged asset-header fetches, and
|
|
the generic-github byte proxy that streams private release assets.
|
|
|
|
The credential is read from the environment (deliver it from a Vault or
|
|
Kubernetes secret). It is **never** stored per-remote in the database, **never**
|
|
returned by any API, and **never** logged. Configure **exactly one** mode.
|
|
|
|
**Precedence.** A remote's own `username`/`password` credential still wins for
|
|
that remote's requests; the server credential is the default for everything else.
|
|
With no credential configured at all, requests stay anonymous (current behavior).
|
|
Partial configuration (e.g. an App id with no private key) is a **startup error**
|
|
— artifactapi fails closed rather than silently falling back to anonymous.
|
|
|
|
Both modes share the syncer's single global rate limiter, so a token simply
|
|
raises the effective GitHub ceiling; the default limiter settings stay safe.
|
|
|
|
#### Mode 1 — Personal Access Token (minimum viable, recommended for free accounts)
|
|
|
|
Set `GITHUB_TOKEN`. It is sent as `Authorization: Bearer <token>`.
|
|
|
|
Recommended free-account setup — a **fine-grained PAT** scoped to just the target
|
|
repositories:
|
|
|
|
1. GitHub → *Settings → Developer settings → Personal access tokens →
|
|
Fine-grained tokens → Generate new token*.
|
|
2. Limit *Repository access* to the specific repo(s) serving releases.
|
|
3. Grant repository permissions **Contents: Read-only** and **Metadata:
|
|
Read-only** (Metadata is mandatory and auto-selected).
|
|
|
|
A classic PAT with the `repo` scope also works but is broader than necessary.
|
|
|
|
```bash
|
|
GITHUB_TOKEN=github_pat_xxxxxxxx
|
|
```
|
|
|
|
#### Mode 2 — GitHub App installation token (proper machine identity)
|
|
|
|
A GitHub App is not tied to a personal account and can be created and installed on
|
|
free personal repos. artifactapi mints a short-lived RS256 **JWT** from the app
|
|
private key, exchanges it at `POST /app/installations/{id}/access_tokens` for a
|
|
~1-hour **installation access token**, caches that token, and refreshes it a few
|
|
minutes before expiry (thread-safe, single-flighted).
|
|
|
|
1. GitHub → *Settings → Developer settings → GitHub Apps → New GitHub App*.
|
|
2. Under *Permissions → Repository permissions* grant **Contents: Read-only**
|
|
(Metadata: Read-only is implied).
|
|
3. Generate a **private key** (downloads a PEM) and note the **App ID**.
|
|
4. *Install* the App on the account and select the target repositories, then read
|
|
the **Installation ID** from the installation URL
|
|
(`.../settings/installations/<installation-id>`).
|
|
|
|
```bash
|
|
GITHUB_APP_ID=123456
|
|
GITHUB_APP_INSTALLATION_ID=7654321
|
|
GITHUB_APP_PRIVATE_KEY_PATH=/etc/artifactapi/github-app.pem
|
|
# or inline PEM (e.g. mounted from a secret):
|
|
# GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
|
|
```
|
|
|
|
## Terraform
|
|
|
|
Remotes and virtuals are managed by Terraform. Each package type has its own resource; see `docs/<type>/` for examples.
|
|
|
|
Provider: [terraform-provider-artifactapi](https://git.unkin.net/unkin/terraform-provider-artifactapi)
|
|
|
|
## Access Control
|
|
|
|
| Field | Default | Behaviour |
|
|
|---|---|---|
|
|
| `patterns` | empty (proxy all) | If set, only matching paths are proxied. Acts as allowlist. |
|
|
| `blocklist` | empty | Matching paths always denied. Checked first. |
|
|
| `mutable_patterns` | empty | Override: force paths to mutable TTL. |
|
|
| `immutable_patterns` | empty | Override: force paths to immutable TTL. |
|
|
|
|
No patterns + no blocklist = open proxy. Provider handles mutability classification automatically.
|
|
|
|
## API
|
|
|
|
### Proxy (v1)
|
|
|
|
```
|
|
GET /api/v1/remote/{name}/{path} Proxy/cache artifact
|
|
GET /api/v1/virtual/{name}/{path} Virtual repo (merged index)
|
|
PUT /api/v1/local/{name}/{package} npm publish (local npm repos)
|
|
POST/DELETE /api/v1/local/{name}/api/v1/gems[/yank] gem push / yank (local gems repos)
|
|
GET /v2/{name}/{path} Docker Registry v2
|
|
```
|
|
|
|
### Management (v2)
|
|
|
|
```
|
|
GET/POST /api/v2/remotes List / create remotes
|
|
GET/PUT/DELETE /api/v2/remotes/{name} Read / update / delete remote
|
|
GET/DELETE /api/v2/remotes/{name}/objects Browse / evict cached objects
|
|
GET /api/v2/stats Overview stats
|
|
GET /api/v2/health Service health
|
|
POST /api/v2/probe Test a remote (fetch without streaming to client)
|
|
GET /api/v2/events SSE event stream
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
PostgreSQL ─── config (remotes, virtuals), artifact metadata, access log
|
|
Redis ─── TTL keys, fetch locks, circuit breaker state
|
|
S3/MinIO ─── content-addressable blob storage (blobs/sha256/{hash})
|
|
```
|
|
|
|
S3 client supports MinIO, Ceph RGW, and AWS S3 (via minio-go).
|
|
|
|
### Schema migrations
|
|
|
|
The SQL schema lives in `migrations/` as numbered `.sql` files, embedded into
|
|
the binary and applied at startup before the server listens, so there is no
|
|
mirrored copy of the schema in the deployment to drift out of sync.
|
|
|
|
The runner is [`golib/pg`](https://git.unkin.net/unkin/golib)'s `pg.NewMigrated`
|
|
— artifactapi owns the SQL, the shared library owns the mechanics.
|
|
|
|
Each start takes `pg_advisory_lock` on a fixed key (FNV-1a/64 of the lock name
|
|
`artifactapi-migrations`), creates `schema_migrations` (`version`, `applied_at`)
|
|
if missing, and applies every embedded file whose filename is not yet recorded —
|
|
in lexical (version) order, each file's SQL and its tracking row in one
|
|
transaction — then unlocks. Replicas starting together queue on the lock and
|
|
then find nothing to do.
|
|
|
|
A file absent from `schema_migrations` is re-run even when the database already
|
|
has the schema, which is how a database migrated by the old untracked inline DDL
|
|
picks `0001_init.sql` up: the statements are `IF NOT EXISTS`-guarded, so the
|
|
re-run is a no-op that only lands the tracking row. New migrations must stay
|
|
additive and idempotent for the same reason; a test enforces it.
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `LISTEN_ADDR` | `:8000` | Server listen address |
|
|
| `DBHOST` | `localhost` | PostgreSQL host |
|
|
| `DBPORT` | `5432` | PostgreSQL port |
|
|
| `DBUSER` | `artifacts` | PostgreSQL user |
|
|
| `DBPASS` | | PostgreSQL password |
|
|
| `DBNAME` | `artifacts` | PostgreSQL database |
|
|
| `DBSSL` | `disable` | PostgreSQL `sslmode` |
|
|
| `REDIS_URL` | `redis://localhost:6379` | Redis URL |
|
|
| `MINIO_ENDPOINT` | `localhost:9000` | S3 endpoint |
|
|
| `MINIO_ACCESS_KEY` | | S3 access key |
|
|
| `MINIO_SECRET_KEY` | | S3 secret key |
|
|
| `MINIO_BUCKET` | `artifacts` | S3 bucket |
|
|
| `MINIO_SECURE` | `false` | Use HTTPS for S3 |
|
|
| `MINIO_REGION` | | S3 region (AWS) |
|
|
| `GITHUB_SYNC_RATE` | `1` | `github_rpm` syncer global GitHub request rate (req/s), shared across all remotes. `1`/s = 3600/hr, under an authenticated token's ~5000/hr; unauthenticated (~60/hr) relies on ETag/304 |
|
|
| `GITHUB_SYNC_BURST` | `5` | Token-bucket burst for the shared limiter |
|
|
| `GITHUB_SYNC_WORKERS` | `3` | Concurrent `github_rpm` scan workers |
|
|
| `GITHUB_SYNC_POLL_INTERVAL` | `60` | Base scheduler tick in seconds; per-remote cadence is its `mutable_ttl`, enforced by the DB lease |
|
|
| `GITHUB_TOKEN` | | Server-level GitHub PAT (fine-grained or classic), sent as `Authorization: Bearer`. Applies to every GitHub request; per-remote creds override it. See [GitHub authentication](#github-authentication) |
|
|
| `GITHUB_APP_ID` | | GitHub App id (App auth mode; mutually exclusive with `GITHUB_TOKEN`) |
|
|
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
|
|
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
|
|
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
|
|
| `ARTIFACTAPI_OIDC_ISSUER` | | OIDC issuer URL (e.g. Vault `https://vault.service.consul:8200/v1/identity/oidc`). When set, `POST /api/v2/remotes`, `POST /api/v2/virtuals` and `PUT`/`DELETE` on `/api/v2/{remotes,virtuals}/{name}` require `Authorization: Bearer <jwt>` verified against the issuer's JWKS. Uploads, object/file deletes, probe, reads and `/v2` stay open. Unset = no auth |
|
|
| `ARTIFACTAPI_OIDC_AUDIENCE` | | Comma-separated allowed `aud` values (e.g. the Vault OIDC role's client_id). Required when `ARTIFACTAPI_OIDC_ISSUER` is set |
|
|
|
|
## Development
|
|
|
|
```bash
|
|
make build # Build binary
|
|
make test # Unit tests
|
|
make e2e # E2E tests (needs Docker)
|
|
make lint # golangci-lint + go vet
|
|
make fmt # gofmt + goimports
|
|
```
|
|
|
|
### `GOPRIVATE`
|
|
|
|
artifactapi depends on `git.unkin.net/unkin/golib`, which is served by Gitea and
|
|
is unknown to `proxy.golang.org` / `sum.golang.org`. Module resolution therefore
|
|
needs:
|
|
|
|
```
|
|
export GOPRIVATE=git.unkin.net
|
|
```
|
|
|
|
The `Makefile` exports it for every target, and the `Dockerfile` and the
|
|
woodpecker Go steps set it themselves, so `make build|test|lint` and CI work on
|
|
a clean checkout. Only bare `go` commands run outside `make` need it in your
|
|
shell — set it there (or in your shell profile) rather than with `go env -w`,
|
|
which is machine state this repo cannot carry.
|
|
|
|
### TUI
|
|
|
|
```bash
|
|
./bin/artifactapi tui --endpoint http://localhost:8000
|
|
```
|