9ba96ace417f2e5e1985b154ddc3b80144c56155
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 unkinben
|
9ba96ace41 |
fix: decouple github_rpm scan and repodata read from the client request
A cold `dnf makecache` against a github_rpm remote with many release assets 500s on the first request: ServeRemote derives metadata for every asset synchronously on the inbound request context, so once dnf hits its makecache timeout and disconnects the canceled request context both aborts the in-flight derive and poisons the subsequent repodata DB read, which surfaces as HTTP 500. It only "works" on a lucky client retry that finds the partially-populated cache fresh. - detach the release scan to a background, timeout-bounded context so a client cancel can neither abort the shared derive nor cancel the read - single-flight the scan per remote so concurrent requests never launch duplicate derives - serve the current cache immediately when it is non-empty and derive in the background; only a completely empty cache blocks on a bounded first scan - serve repodata on a context detached from the request, and treat a canceled/deadline-exceeded metadata read as a retryable 503 instead of a hard 500 |
||
|
unkinben
|
b727b990a2 |
feat: add github_rpm metadata-only remote serving GitHub releases as a yum repo
Publishing RPMs to GitHub releases is common, but consuming them with dnf requires repodata that GitHub does not provide, and mirroring every package into a local repo wastes storage and staleness-tracking on artifacts that already have a durable home. Expose GitHub releases as a first-class RPM source that synthesizes repodata on the fly and never precaches the packages. Add a `github_rpm` remote package type backed by a metadata-only provider: - Introduce a `RemoteServer` interception hook (the remote-side analog of `LocalIndexer`): `handleProxy` lets a provider fully answer a request before the byte-proxy engine, passing the request-derived proxy base URL and the DB as a `RemoteMetadataStore`. - Scan a repo's releases via the GitHub API (`base_url` = the releases API root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset filename) and reuse the existing local-rpm repodata generators to emit `repomd.xml`/`primary`/`filelists`/`other`. - Derive per-asset metadata without precaching: fetch only the RPM header via a ranged GET (retrying with a larger range on a truncated-header parse) for NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from the GitHub asset `digest` when present, else compute it once by streaming. - Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more often than `mutable_ttl`, pruning assets that disappear upstream. - Serve each package's `<location>` as the github-relative download path so the client comes back to this remote, which 302-redirects to the `releases_remote` (an existing generic github.com remote) that streams the actual bytes. Reuse the existing `releases_remote` field as the redirect target — it already carries exactly this "downloads served by remote X" semantic end to end. Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns, added idempotently) so both local and github_rpm repodata resolve upgrades and conflicts; the local upload path now records them too. Tests cover header-range parsing with the retry loop, digest-vs-computed checksum selection, repodata synthesis with the redirect-able location href, the 302 redirect path, asset pattern filtering, and stale-asset pruning. |
||
|
unkinben
|
936cf8846a |
feat: serve local terraform repos as a provider registry (#102)
ci/woodpecker/tag/docker Pipeline was successful
## Why
Local terraform repos already served the Terraform **network mirror** protocol, but consuming that requires every user to add a `provider_installation { network_mirror }` block to `~/.terraformrc`. A `source = "artifactapi.k8s.../ns/type"` address instead triggers the **provider registry** protocol (service discovery at `/.well-known/terraform.json` + GPG-signed SHA256SUMS), which returned 404 — hence *"does not offer a provider registry."*
Local repos are meant to be the real thing, so this makes a terraform local repo a first-class provider registry: `terraform init` installs from a bare source address with no client config.
## What
- Serve `/.well-known/terraform.json` service discovery and the `providers.v1` endpoints under `/terraform/v1/providers`: `versions`, `download/{os}/{arch}`, `sha256sums`, `sha256sums.sig`.
- Map the Terraform **namespace** segment to the artifactapi **repo name**; locate the provider by **type**. `download_url` points back at the existing `/api/v1/local/...` path.
- Generate `SHA256SUMS` per version and sign it with a GPG key loaded from `TF_SIGNING_KEY_PATH` (optional `TF_SIGNING_KEY_PASSPHRASE`); advertise the public key + key id in the download response. **No key → registry stays disabled (endpoints 404)**, so behaviour is unchanged until the signing secret is present.
- New `internal/tfsign` (key load + detached signing, via `x/crypto/openpgp`) and `internal/api/terraform` (registry handler). Export `ParseProviderZip` for reuse.
- `TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
- README section documenting usage.
## Consumer
```hcl
terraform {
required_providers {
artifactapi = {
source = "artifactapi.k8s.syd1.au.unkin.net/terraform-unkin/artifactapi"
version = "0.1.2"
}
}
}
```
## Tests
- `internal/tfsign`: sign + verify round-trip, disabled/missing-key paths.
- `internal/api/terraform`: dockerised full flow (discovery → versions → download → sha256sums → sig), verifying the signature against the advertised public key.
## Follow-ups (separate PRs)
- **argocd-apps**: mount the signing K8s secret into the api deployment + set `TF_SIGNING_KEY_PATH`. The `/` HTTPRoute already routes `/.well-known` and `/terraform` to the API, so no gateway change is needed.
- Image/version bump once tagged.
## Note
Anchored the `terraform/` gitignore to the repo root (`/terraform/`) so it stops matching `internal/*/terraform/`. This surfaced `internal/provider/terraform/terraform_extra_test.go`, which had been silently untracked — now committed.
Reviewed-on: #102
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
|
||
|
unkinben
|
0ec28660ba |
fix: prune RPM metadata when a local file is evicted (#100)
ci/woodpecker/tag/docker Pipeline was successful
Follow-up to #99. ## Why Evicting or deleting a local RPM removed the \`local_files\` row but left its \`rpm_metadata\` behind. Since generated repodata is built from \`rpm_metadata\`, \`primary.xml\` kept advertising a package that no longer exists, producing 404s for clients that tried to fetch it. ## Changes - Add \`PostDeleteHook\` and \`MetadataDeleter\` provider interfaces (symmetric to the existing \`PostUploadHook\`/\`MetadataStore\`), plus a \`DeleteRPMMetadata\` DB method. - Implement \`AfterDelete\` in the RPM provider to drop the metadata row for the deleted file. - Route both local delete paths — the new \`evictLocal\` and the existing files handler's \`remove\` — through a shared \`deleteLocalFile\` helper that removes the file then runs the provider's post-delete hook. Non-RPM providers have no hook, so nothing changes for them. - Cover the cleanup with a dockerised test. Reviewed-on: #100 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
a1ba86e76b |
test: raise core-package unit coverage to 90% (#98)
Raises statement coverage of the core packages (all of `internal/` except the interactive `tui/`, plus `pkg/`) from **8.7% to 90.1%**. ## Approach - **Pure-go unit tests** for all providers, virtual mergers, classifier, config, auth, models, and the API client (httptest). - **Testcontainers-backed** tests (new `internal/testsupport` helper: Postgres/Redis/MinIO, Ryuk disabled) for database, storage, cache, the proxy engine, the GC, and a full-stack `server` test that drives the whole HTTP API. These `t.Skip` when Docker is absent so `go test` still runs locally without it. ## Measuring ``` go test -coverpkg=./internal/...,./pkg/... -coverprofile=cover.out ./internal/... ./pkg/... grep -v /internal/tui/ cover.out | go tool cover -func=/dev/stdin | tail -1 # 90.1% ``` Run with `-p 1` (containers are heavy). ## Notes - The interactive `tui/` package and `cmd/main` are excluded from the target per the agreed scope. - Some defensive error branches are covered via fault injection (closed DB pool, killing MinIO mid-upload). Reviewed-on: #98 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
6f8e70c27a |
feat: add local RPM repository with on-demand repodata (#53)
## Summary - Upload RPMs to local repos, metadata parsed async via cavaliergopher/rpm - Repodata (repomd.xml, primary/filelists/other.xml.gz) generated on-demand from DB — nothing stored in S3 - RPM provider implements LocalUploader, PostUploadHook, and LocalIndexer - New rpm_metadata table for parsed RPM header data (name, version, deps, etc.) - New provider interfaces: PostUploadHook, BlobReader, MetadataStore, RPMMetadataReader ## Test plan - [x] Upload cowsay RPM from epel → async metadata parse confirmed in logs - [x] repomd.xml generated with correct hashes → primary.xml.gz has correct metadata - [x] `dnf install` from local repo: download + install successful - [x] Bad file rejection (.txt → 400), overwrite rejection (409) Reviewed-on: #53 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
3a6721c2a7 |
refactor: modular local provider interfaces (#52)
ci/woodpecker/tag/docker Pipeline was successful
## Summary Move package-type-specific local repo logic out of centralized handlers into provider packages via optional Go interfaces. **New interfaces in `provider` package:** - \`LocalUploader\`: \`ValidateUpload(filePath) → (storagePath, contentType, error)\` + \`UploadResponse(...)\` - \`LocalIndexer\`: \`ServeLocalIndex(w, r, files, repoName, path) → bool\` + \`GenerateLocalIndex(ctx, files, repoName, path) → ([]byte, error)\` - \`FileStore\`: \`ListFilesByPrefix\` + \`ListPackages\` (implemented by database.DB) **Providers implement these interfaces:** - PyPI: upload validation (wheel/sdist naming), simple index serving + generation - Terraform: upload validation (provider zip naming), mirror protocol serving **Handlers simplified to generic dispatch:** - \`local.go\`: type-asserts to \`LocalUploader\`, falls back to generic upload - \`proxy.go\`: type-asserts to \`LocalIndexer\`, falls back to raw file serving - \`engine.go\`: type-asserts to \`LocalIndexer\` for local virtual members Adding a new local repo type (e.g. RPM) = implement the interfaces in its provider package. Zero handler changes. ## Test plan - [x] Build + unit tests pass - [x] E2E: PyPI local upload → simple index → uv pip install (smoke test after refactor) Reviewed-on: #52 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
 benvin
|
b46c116f6b |
Feat/v3 go rewrite (#47)
ci/woodpecker/tag/docker Pipeline was successful
Complete rewrite of ArtifactAPI from Python/FastAPI to Go as a single binary. Core engine: - 10 package providers: generic, docker, helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy — each with built-in mutable patterns - Content-addressable storage (SHA256 dedup across all remotes) - Three-tier caching: Redis (TTL/locks) → S3/MinIO (blobs) → upstream - Classifier with allowlist/blocklist per-remote (empty = allow all) - Circuit breaker, conditional revalidation, stale-on-error - Background garbage collection for orphaned blobs - Access logging to PostgreSQL API: - v1 proxy endpoints (backwards compatible) - v2 management API: CRUD remotes/virtuals, object browser, stats, health, SSE events, probe/test endpoint - Virtual repos with index merging (Helm YAML + PyPI HTML) Frontend (React + Vite, separate Dockerfile): - Dashboard with stats, health indicators, top remotes - Remotes list with type filter, remote detail with config/patterns - Object browser with pagination and evict - Test Remote page: probe any remote path, see headers/size/timing - Virtuals page with expandable member lists TUI (Bubble Tea): - Dashboard, remotes list/detail, object browser, virtuals - Vim-style navigation, artifactapi tui --endpoint <url> Infrastructure: - S3 client supports MinIO, Ceph RGW, AWS S3 (minio-go) - PostgreSQL schema with migrations - Docker Compose: API + UI + Postgres 17 + Redis 7 + MinIO - Makefile with Go version check, build/test/lint/fmt/e2e targets - Distroless Docker image (~15MB) Testing: - Unit tests for models, classifier, providers, mergers - E2E tests with testcontainers-go (real Postgres/Redis/MinIO) Terraform config: - All 40 production remotes + helm virtual as HCL - Provider repo: terraform-provider-artifactapi v0.0.1 (separate) --------- Co-authored-by: Ben Vincent <ben@unkin.net> Reviewed-on: #47 |