Compare commits

..

33 Commits

Author SHA1 Message Date
unkinben 3a3b7fe7b7 feat: redirect / to the web UI (#101)
ci/woodpecker/tag/docker Pipeline was successful
## Why

The web UI ships as a separate image served under \`/ui\` (built with \`BASE_PATH=/ui\`). Hitting the bare domain (e.g. \`https://artifactapi.k8s.syd1.au.unkin.net/\`) returned the API's JSON identity blob instead of the app, so browsers never landed on the UI.

## Changes

- Redirect \`GET /\` to \`/ui/\` (302 Found).
- Preserve the former root JSON (\`{"name","version"}\`) at \`/version\`, so health/monitoring can still read the running version.
- Update the server integration test to assert the redirect and the \`/version\` payload.

Reviewed-on: #101
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 15:00:19 +10:00
unkinben 0ec28660ba fix: prune RPM metadata when a local file is evicted (#100)
ci/woodpecker/tag/docker Pipeline was successful
Follow-up to #99.

## Why

Evicting or deleting a local RPM removed the \`local_files\` row but left its \`rpm_metadata\` behind. Since generated repodata is built from \`rpm_metadata\`, \`primary.xml\` kept advertising a package that no longer exists, producing 404s for clients that tried to fetch it.

## Changes

- Add \`PostDeleteHook\` and \`MetadataDeleter\` provider interfaces (symmetric to the existing \`PostUploadHook\`/\`MetadataStore\`), plus a \`DeleteRPMMetadata\` DB method.
- Implement \`AfterDelete\` in the RPM provider to drop the metadata row for the deleted file.
- Route both local delete paths — the new \`evictLocal\` and the existing files handler's \`remove\` — through a shared \`deleteLocalFile\` helper that removes the file then runs the provider's post-delete hook. Non-RPM providers have no hook, so nothing changes for them.
- Cover the cleanup with a dockerised test.

Reviewed-on: #100
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 14:54:28 +10:00
unkinben 787de74b3d fix: show local-repo files in the cached-objects UI (#99)
ci/woodpecker/tag/docker Pipeline was successful
## Why

Local repos store uploaded files in the \`local_files\` table, whereas remote/proxy repos cache into the \`artifacts\` table. The shared **Cached Objects** page always queried the artifacts table via \`/api/v2/remotes/{name}/objects\`, so files uploaded to a local repo (e.g. an internal RPM) were fully stored and servable but showed as **0 objects** in the UI.

## Changes

- Add \`ListLocalArtifacts\`, joining \`local_files\` with \`blobs\` and returning \`models.Artifact\`-shaped rows (size from the blob; access/fetch counters zero and timestamps derived from \`created_at\`, since local files track no access).
- Add \`LocalRoutes\` to the objects handler: \`listLocal\` reads \`local_files\`, \`evictLocal\` deletes via \`DeleteLocalFile\`. Extract shared page/per_page parsing into \`pageBounds\`.
- Mount \`/api/v2/locals/{name}/objects\` (GET + DELETE) in the server.
- Add \`listLocalObjects\`/\`evictLocalObject\` to the UI client and route the Objects page to them when viewing a local repo.
- Cover the listing and eviction paths with a dockerised test.

## Notes

Generated \`repodata/*\` files are not listed — they are produced on the fly from \`rpm_metadata\` and never stored in \`local_files\`, which matches how the repo serves them.

Reviewed-on: #99
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 14:46:41 +10:00
unkinben 30acc32174 test: comprehensive dockerised end-to-end suite (#97)
Adds a black-box e2e suite that runs against the **built container image** via docker-compose (complementing the in-process `e2e/` testcontainers suite).

## What it does
`make docker-e2e` → `scripts/docker-e2e.sh`: builds the image, brings up the full stack (postgres, redis, minio, artifactapi) plus a static nginx **mock upstream** for hermetic caching, waits for `/health`, runs `go test -tags=dockere2e ./e2e-docker/...`, and tears everything down.

## Coverage
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact for **each of the 10 remote package types** (generic, docker, helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/` index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge across two members.

## Notes
- The artifactapi host port is parameterised (`ARTIFACTAPI_PORT`, default `8000`; the e2e run uses `8001`) so it does not collide with a locally-running instance. This is the only change to the production `docker-compose.yml`.
- Fixtures under `e2e-docker/fixtures/` are real package files (incl. a real RPM so repodata parsing works); a `.gitignore` negation tracks them over the global ignore of those extensions.

## Validation
Ran `make docker-e2e` locally: **all suites pass** against the containerised product.

---------

Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #97
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 14:34:45 +10:00
unkinben a1ba86e76b test: raise core-package unit coverage to 90% (#98)
Raises statement coverage of the core packages (all of `internal/` except the interactive `tui/`, plus `pkg/`) from **8.7% to 90.1%**.

## Approach
- **Pure-go unit tests** for all providers, virtual mergers, classifier, config, auth, models, and the API client (httptest).
- **Testcontainers-backed** tests (new `internal/testsupport` helper: Postgres/Redis/MinIO, Ryuk disabled) for database, storage, cache, the proxy engine, the GC, and a full-stack `server` test that drives the whole HTTP API. These `t.Skip` when Docker is absent so `go test` still runs locally without it.

## Measuring
```
go test -coverpkg=./internal/...,./pkg/... -coverprofile=cover.out ./internal/... ./pkg/...
grep -v /internal/tui/ cover.out | go tool cover -func=/dev/stdin | tail -1   # 90.1%
```
Run with `-p 1` (containers are heavy).

## Notes
- The interactive `tui/` package and `cmd/main` are excluded from the target per the agreed scope.
- Some defensive error branches are covered via fault injection (closed DB pool, killing MinIO mid-upload).

Reviewed-on: #98
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 14:31:24 +10:00
unkinben 1b585af14e feat: wire the circuit breaker into the proxy fetch path (#90)
ci/woodpecker/tag/docker Pipeline was successful
Fixes #74

## Why
`internal/proxy/circuit.go` implemented and tested a circuit breaker, but nothing ever called it — a repeatedly-failing upstream was still hit on every request.

## Changes
- Construct a `CircuitBreaker` in `NewEngine`.
- In `Engine.Fetch`: short-circuit when the breaker is open (serve stale from the store if present, otherwise return 503), `RecordFailure` on each `UpstreamError`, and `RecordSuccess` on a successful fetch.

## Validation
- `go test ./internal/proxy/` and `make e2e` pass.

---------

Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #90
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:43:22 +10:00
unkinben e7c9387bcc fix: GC has no grace period (TOCTOU with dedup uploads) (#86)
Fixes #71

## Why
`FindOrphanedBlobs` returned any blob not currently referenced. Because CAS dedups (the blob row can exist before its artifact/local_files row is written), a concurrent upload reusing an existing hash could have its S3 object deleted mid-flight by the GC.

## Changes
- `FindOrphanedBlobs` now takes a `minAge` and only returns blobs with `created_at < now()-minAge`.
- The collector passes a 1h `blobGracePeriod`.

## Validation
- `go test ./internal/gc/...` and `make e2e` pass.

---------

Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #86
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:43:18 +10:00
unkinben 7e07eaa758 fix: repair master build after conflicting merges (#96)
## Why
`master` does not compile. Three PRs that each built individually combined badly:
- #92 changed `fetchBearerToken` to return `(string, time.Duration, error)` and added `cachedBearerToken` (which hashes the challenge via `sha256Hash`).
- #94 (streaming) removed the now-unused-in-that-PR `sha256Hash` helper and its `crypto/sha256` / `encoding/hex` imports.
- #89 (HEAD) added `headUpstream`, which calls `fetchBearerToken` expecting two return values.

Result on `master`: `internal/proxy/engine.go` fails to build (`assignment mismatch: 2 variables but fetchBearerToken returns 3 values`; `undefined: sha256Hash`).

## Changes
- Re-add the `sha256Hash` helper and its `crypto/sha256` + `encoding/hex` imports.
- Fix the `headUpstream` 401 path to handle `fetchBearerToken`s three return values.

## Validation
- `go build ./...`, `go vet`, and `make e2e` all pass.

Should merge before the other in-flight branches so they rebase onto a compiling `master`.

Reviewed-on: #96
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:36:09 +10:00
unkinben f61ab99ae8 fix: set timeouts on the upstream HTTP client (#83)
Fixes #67

## Why
The proxy used `http.DefaultClient` for all upstream GET/HEAD and bearer-token requests. It has no timeouts, so a slow or hung upstream holds a goroutine and connection indefinitely.

## Changes
- Add a shared `upstreamClient` (`internal/proxy/httpclient.go`) with dial, TLS-handshake, response-header and idle-connection timeouts, plus connection pooling.
- Deliberately no overall `Client.Timeout`, so large artifact bodies can still stream; total time is bounded by the request context.
- Route all four upstream calls in the engine through it.

## Validation
- `make e2e` passes.

Reviewed-on: #83
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:24:49 +10:00
unkinben c39703ed0d fix: getenv treats an explicitly-empty value as unset (#85)
Fixes #69

## Why
`getenv` returned the fallback whenever `os.Getenv` was empty, so an intentionally-empty env var could not override a non-empty default.

## Changes
- Use `os.LookupEnv` to distinguish unset from set-but-empty.

## Validation
- `make e2e` passes.

Reviewed-on: #85
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:09:09 +10:00
unkinben 5261af4c63 fix: coalesce concurrent cache-miss fetches (thundering herd) (#93)
Fixes #75

## Why
On a fetch-lock miss, `Engine.Fetch` slept a flat 500ms once, tried the store, and otherwise fell through to fetch upstream unlocked. A cold-cache stampede therefore still hit upstream once per waiter.

## Changes
- Add `waitForStore`, which polls the store every 100ms for up to 5s (stopping on context cancellation) so waiters pick up the lock leaders populated result.
- Only fall through to an upstream fetch if the leader has not populated the store within the wait budget.

## Validation
- `make e2e` passes.

Reviewed-on: #93
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:08:29 +10:00
unkinben 45d6cdbc64 perf: batch access-log writes instead of goroutine+insert per request (#91)
Fixes #76

## Why
Every proxied request spawned a goroutine running a 5s-timeout single-row INSERT. Under load this is unbounded goroutines and connection-pool pressure.

## Changes
- Add `database.AccessLogEntry` + `InsertAccessLogBatch` (bulk `COPY`).
- The engine starts one background writer that drains a buffered channel and flushes every 128 entries or 2s.
- `logAccess` is now a non-blocking channel send (drops on full buffer), so the request path never blocks on the DB. Best-effort telemetry: a small tail may be lost on abrupt shutdown.

## Validation
- `make e2e` passes.

Reviewed-on: #91
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:07:56 +10:00
unkinben b59cc45765 fix: HEAD requests fetch and stream the full body (#89)
Fixes #70

## Why
Docker `HEAD` routes mapped to `handleProxy`, which ran a full `Fetch` + `io.Copy` — downloading the entire blob (and fetching upstream on a miss) only for net/http to discard the body. HEAD existence checks (manifests, blobs) are common.

## Changes
- Add `Engine.Head`: answers cached artifacts/indexes from store metadata (no blob download); on a miss issues an upstream `HEAD` (with bearer-token handling) and never caches a body.
- Route `HEAD /v2/{remote}/*` to a dedicated `handleProxyHead` that writes headers only.
- Add e2e tests for HEAD on a blocklisted path (403) and an unknown remote (404).

## Note
`headUpstream` uses `http.DefaultClient` to build cleanly on master; it will pick up the shared timeout-configured client from #67 once that merges.

## Validation
- `make e2e` passes (includes new HEAD tests).

Reviewed-on: #89
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 22:06:50 +10:00
unkinben e7027c8ccc feat: cache upstream bearer tokens (#92)
Fixes #77

## Why
Each upstream 401 re-ran the full token-endpoint request, even though a single Docker pull triggers many blob/manifest requests sharing one scope.

## Changes
- Add Redis `GetToken`/`SetToken`.
- `fetchBearerToken` now also parses `expires_in` and returns a TTL.
- New `Engine.cachedBearerToken` reuses a cached token keyed by remote + challenge (hashed), caching for `expires_in` minus a safety margin (default 60s when absent).

## Validation
- `make e2e` passes.

Reviewed-on: #92
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 21:35:46 +10:00
unkinben f3680951b7 perf: stream proxied artifacts instead of buffering the full body in memory (#94)
Fixes #66

## Why
`fetchFromUpstream` read every upstream response with `io.ReadAll`, hashed it in memory, uploaded from memory and served from memory. A single large immutable blob (Docker layer, RPM, tarball, Go module zip) — or several concurrent ones — could OOM the process. The streaming, tempfile-backed CAS already existed but the proxy path bypassed it (and `Engine.cas` was assigned but unused).

## Changes
- Immutable fetches now stream through `CAS.Store` (tempfile -> sha256 -> S3), so memory stays bounded regardless of artifact size, and are served back from the store.
- Mutable indexes stay on the in-memory path (small, and subject to `RewriteResponse`).
- Skipping `RewriteResponse` for immutable content is behaviour-preserving: the proxy path always passes an empty `proxyBaseURL`, under which every providers `RewriteResponse` is a no-op.
- Remove the now-unused in-memory `sha256Hash` helper.

## Validation
- `make e2e` passes.
- Live smoke test against Postgres/Redis/MinIO: proxied a 12 MB blob through a generic remote — fetch #1 `X-Artifact-Source: remote`, fetch #2 `X-Artifact-Source: cache`, both byte-identical (sha256) to the origin.

Reviewed-on: #94
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 21:33:42 +10:00
unkinben 61a1a99112 perf: compile remote match patterns once instead of per-request (#88)
Fixes #73

## Why
`Classifier.Classify` runs on every proxied request and recompiled the Blocklist/Patterns/Immutable/Mutable regex lists each time. Regex compilation is expensive and fully redundant.

## Changes
- Memoise compilation in a `sync.Map` keyed by pattern text (`compileCached`); each distinct pattern compiles once and is reused. Patterns that fail to compile are cached as a typed nil so they are not retried. No invalidation needed since the pattern text is the key.

## Validation
- `go test ./internal/proxy/` and `make e2e` pass.

Reviewed-on: #88
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 20:20:00 +10:00
unkinben f0e44d6810 fix: blocklist fails open when a regex fails to compile (#87)
Fixes #72

## Why
`compilePatterns` silently discards any pattern that fails to compile. A typo in a blocklist entry therefore turns a deny rule into a no-op — a fail-open with security impact.

## Changes
- Add `Remote.ValidatePatterns`, which compiles every pattern list (patterns, blocklist, mutable/immutable patterns, ban_tags) and returns an error on the first invalid regex.
- Reject invalid patterns with 400 at remote create and update time.
- Unit test for valid and invalid patterns.

## Validation
- `go test ./pkg/models/` and `make e2e` pass.

Reviewed-on: #87
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 20:19:27 +10:00
unkinben 0a89b2005c fix: isNetworkError should use errors.As, not a bare type assertion (#84)
Fixes #68

## Why
`isNetworkError` type-asserted `err.(*UpstreamError)` directly. If the error is ever wrapped, stale-on-error handling silently stops triggering.

## Changes
- Use `errors.As` to detect `*UpstreamError` through wrapping.

## Validation
- `make e2e` passes.

Reviewed-on: #84
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 20:18:23 +10:00
unkinben f23bf2a6d9 fix: serveFromStore does a guaranteed-miss S3 lookup on every cache hit (#82)
Fixes #78

## Why
`serveFromStore` first called `store.Download` with the bare content hash as the S3 key, which never matches real object keys (`blobs/sha256/<hash>`). Every cached blob serve therefore paid an extra guaranteed-404 round-trip before retrying with the correct `BlobKey`.

## Changes
- Remove the dead first `Download` attempt; go straight to the `BlobKey` lookup, then fall back to the index key.

## Validation
- `make e2e` passes (proxy cache-hit paths exercised end-to-end).

Reviewed-on: #82
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 20:07:30 +10:00
unkinben b9098bf19c fix: e2e suite fails to build (stale server.New call) (#81)
Fixes #80

## Why
`make e2e` did not compile against master: `e2e/e2e_test.go` called `server.New(cfg)` but the signature is `New(cfg, version string)`. This blocked all end-to-end validation.

## Changes
- Pass a static `"e2e-test"` version to `server.New` in the e2e bootstrap.

## Validation
- `make e2e` builds and passes (testcontainers: postgres/redis/minio).

Reviewed-on: #81
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-02 20:00:24 +10:00
unkinben 8d9bc1c422 feat: add bandwidth saved stat to dashboard (#65)
ci/woodpecker/tag/docker Pipeline was successful
Shows total bytes served from cache (instead of upstream) over the last 30 days. Queries `SUM(size_bytes) WHERE cache_hit = TRUE` from access_log.

Reviewed-on: #65
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 22:18:02 +10:00
unkinben 30b7cef026 fix: strip base URL path prefix from helm chart download URLs (#64)
ci/woodpecker/tag/docker Pipeline was successful
When a helm repo base URL includes a path component (e.g. \`stakater.github.io/stakater-charts\`), the merger was extracting the full URL path (\`stakater-charts/reloader-2.2.8.tgz\`) and the proxy then constructed \`base_url/stakater-charts/reloader-2.2.8.tgz\` = double path = 404.

Fix: \`extractPathRelativeToBase()\` strips the shared base path prefix so only the filename portion is used as the proxy path.
Reviewed-on: #64
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 08:02:52 +10:00
unkinben 603be5b989 fix: report actual version instead of hardcoded 3.0.0-dev (#63)
ci/woodpecker/tag/docker Pipeline was successful
The / endpoint was hardcoded to return 3.0.0-dev. Now uses the git tag version set via ldflags at build time.

Reviewed-on: #63
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 00:51:26 +10:00
unkinben 9eba49500c feat: forward Accept header and fix Content-Type for Docker proxying (#62)
## Problems
1. Docker daemon sends specific Accept headers to negotiate manifest format, but the proxy dropped them — registries defaulted to OCI format, causing "mediaType should be manifest.v2+json not oci.image.index" errors
2. Upstream Content-Type was only used when the provider returned "application/octet-stream" — Docker manifests got the wrong Content-Type

## Fixes
- Forward client Accept header to upstream (both initial request and Bearer token retry)
- Always prefer upstream Content-Type when present
- Fetch signature now accepts variadic clientHeaders for backwards compat

## E2E tested
- DockerHub: redis:7-alpine, alpine:3 — skopeo inspect OK
- GHCR: OCI-only images work with docker pull (GHCR 404s Docker v2 Accept, which is expected)
- Quay: prometheus/node-exporter — skopeo inspect OK

Reviewed-on: #62
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 00:45:23 +10:00
unkinben 0083d67272 fix: nginx config for UI serving under base path (#61)
Vite's \`base: /ui\` makes HTML reference \`/ui/assets/...\` but files are at \`/usr/share/nginx/html/assets/\` (no \`ui/\` subdir). The previous \`location /ui { try_files ... }\` couldn't find the files.

Fix: rewrite strips the base path prefix before try_files, so \`/ui/assets/foo.js\` resolves to \`/usr/share/nginx/html/assets/foo.js\`.
Reviewed-on: #61
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 00:43:45 +10:00
unkinben 8ec7de50e3 feat: handle Docker Bearer token auth for upstream registries (#60)
ci/woodpecker/tag/docker Pipeline was successful
Docker Hub (and other registries) return 401 with a `Www-Authenticate: Bearer realm=...` challenge even for public images. The proxy now:

1. Detects 401 + Bearer challenge
2. Parses realm/service/scope from the header
3. Fetches an anonymous token (or authenticated if username/password configured)
4. Retries the original request with the Bearer token

Fixes: `docker pull artifactapi.../dockerhub/library/redis:latest` returning "unauthorized: upstream returned 401"
Reviewed-on: #60
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 00:18:06 +10:00
unkinben 9c465cbd4c fix: use map format for docker-buildx build_args (#59)
The woodpecker docker-buildx plugin expects build_args as a YAML map (KEY: VALUE), not a list (- KEY=VALUE). The list format was silently ignored, so BASE_PATH was never passed to the Docker build.

Reviewed-on: #59
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 00:12:34 +10:00
unkinben ee6e581b9d feat: configurable UI base path via BASE_PATH build arg (#58)
ci/woodpecker/tag/docker Pipeline was successful
Serves the UI under /ui instead of /. This pairs with the argocd route simplification (argocd-apps#201) where /ui → UI service and everything else → API.

- Vite: `base` set from `BASE_PATH` env var at build time
- React Router: `basename` set from injected `__BASE_PATH__`
- Nginx: location block uses `${BASE_PATH}`, substituted by sed at build
- Dockerfile: `ARG BASE_PATH=/` (default preserves existing behavior)
- Woodpecker: passes `BASE_PATH=/ui` to docker-web build

Tested: assets serve at `/ui/assets/...`, SPA routing works at `/ui/remotes`, etc.
Reviewed-on: #58
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-26 23:50:17 +10:00
unkinben 2a8e544de3 feat: add Docker Registry V2 endpoint at /v2/ (#57)
The v3 Go rewrite removed the /v2/ Docker Registry compatibility endpoint. Docker clients need:
- GET/HEAD /v2/ → 200 (registry ping)
- GET/HEAD /v2/{remoteName}/* → proxy to the docker remote

Usage: `docker pull artifactapi.example.com/{remoteName}/image:tag`
Reviewed-on: #57
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-26 23:37:52 +10:00
unkinben 847eeb839f fix: don't rewrite helm chart URLs pointing to a different host (#56)
## Problem
Helm charts like Intel device plugins have download URLs on `github.com` but the chart index is served from `intel.github.io`. The merger rewrites all URLs through the proxy, constructing:
```
https://artifactapi/api/v1/remote/intel-helm/intel/helm-charts/releases/download/...
```
Which proxies to `https://intel.github.io/helm-charts/intel/helm-charts/releases/download/...` — a 404.

## Fix
Compare the download URL host against the remote's base URL host. If they differ, leave the URL as-is so helm downloads directly from the source. Same-host URLs are still rewritten through the proxy.

Also adds `BaseURL` to `MemberIndex` so the merger has the context it needs, and uses the correct `/local/` vs `/remote/` route prefix.

Reviewed-on: #56
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-26 23:34:00 +10:00
unkinben 74d9c0fa84 chore: add pre-commit config and update CI pipeline (#55)
ci/woodpecker/tag/docker Pipeline was successful
## Summary
- New `.pre-commit-config.yaml` with standard Go hooks (gofmt, go vet, go mod tidy) plus file hygiene checks (trailing whitespace, end-of-file, yaml, large files, merge conflicts)
- go vet runs as a local hook with `./...` since the dnephin per-file hook doesn't work with Go module layouts
- Woodpecker pre-commit pipeline updated to use `almalinux9-gobuilder` image with `uvx pre-commit run --all-files`
- Pre-commit hooks installed into the repo

Reviewed-on: #55
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-23 23:21:09 +10:00
unkinben 097fbf0016 feat: UI separates locals, remotes, and virtuals (#54)
## Summary
- New "Locals" sidebar nav item with list + detail + browse pages
- Remotes page filters out local repos (repo_type=local hidden)
- LocalDetail: simplified view — just name, type, description + "Browse Files" button
- Virtuals: member links resolve to /locals/ or /remotes/ based on repo_type
- Objects page detects context for correct back-navigation

## Test plan
- [ ] Visual check: locals page shows only local repos
- [ ] Remotes page hides local repos
- [ ] Virtual member links point to correct pages
- [ ] Browse files works from local detail page

Reviewed-on: #54
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-23 23:20:18 +10:00
unkinben 6f8e70c27a feat: add local RPM repository with on-demand repodata (#53)
## Summary
- Upload RPMs to local repos, metadata parsed async via cavaliergopher/rpm
- Repodata (repomd.xml, primary/filelists/other.xml.gz) generated on-demand from DB — nothing stored in S3
- RPM provider implements LocalUploader, PostUploadHook, and LocalIndexer
- New rpm_metadata table for parsed RPM header data (name, version, deps, etc.)
- New provider interfaces: PostUploadHook, BlobReader, MetadataStore, RPMMetadataReader

## Test plan
- [x] Upload cowsay RPM from epel → async metadata parse confirmed in logs
- [x] repomd.xml generated with correct hashes → primary.xml.gz has correct metadata
- [x] `dnf install` from local repo: download + install successful
- [x] Bad file rejection (.txt → 400), overwrite rejection (409)

Reviewed-on: #53
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-23 23:20:05 +10:00
95 changed files with 5541 additions and 186 deletions
+4
View File
@@ -1,2 +1,6 @@
bin/ bin/
terraform/ terraform/
# e2e-docker fixtures are real package files (.rpm, .tgz, .whl, .zip, ...) that
# are intentionally tracked, overriding any global ignore of those extensions.
!e2e-docker/fixtures/**
+24
View File
@@ -0,0 +1,24 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- id: check-merge-conflict
- repo: https://github.com/dnephin/pre-commit-golang
rev: v0.5.1
hooks:
- id: go-fmt
- id: go-mod-tidy
- repo: local
hooks:
- id: go-vet
name: go vet
entry: go vet ./...
language: system
types: [go]
pass_filenames: false
+4
View File
@@ -8,6 +8,8 @@ steps:
settings: settings:
registry: git.unkin.net registry: git.unkin.net
repo: git.unkin.net/unkin/artifactapi repo: git.unkin.net/unkin/artifactapi
build_args:
VERSION: ${CI_COMMIT_TAG}
username: droneci username: droneci
password: password:
from_secret: DRONECI_PASSWORD from_secret: DRONECI_PASSWORD
@@ -22,6 +24,8 @@ steps:
repo: git.unkin.net/unkin/artifactapi-ui repo: git.unkin.net/unkin/artifactapi-ui
dockerfile: ui/Dockerfile.ui dockerfile: ui/Dockerfile.ui
context: ui context: ui
build_args:
BASE_PATH: /ui
username: droneci username: droneci
password: password:
from_secret: DRONECI_PASSWORD from_secret: DRONECI_PASSWORD
+11 -3
View File
@@ -3,7 +3,15 @@ when:
steps: steps:
- name: pre-commit - name: pre-commit
image: golang:1.25 image: git.unkin.net/unkin/almalinux9-gobuilder:20260606
commands: commands:
- test -z "$(gofmt -l .)" - uvx pre-commit run --all-files
- go vet ./... backend_options:
kubernetes:
resources:
requests:
memory: 512Mi
cpu: 1
limits:
memory: 2Gi
cpu: 2
+2 -1
View File
@@ -9,7 +9,8 @@ RUN go mod download
COPY . . COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o artifactapi ./cmd/artifactapi ARG VERSION=dev
RUN CGO_ENABLED=0 go build -ldflags="-s -w -X main.version=${VERSION}" -o artifactapi ./cmd/artifactapi
FROM gcr.io/distroless/static-debian12:nonroot FROM gcr.io/distroless/static-debian12:nonroot
+7 -2
View File
@@ -1,4 +1,4 @@
.PHONY: build test lint fmt e2e docker docker-ui compose clean tidy check-go .PHONY: build test lint fmt e2e docker-e2e docker docker-ui compose clean tidy check-go
BINARY := bin/artifactapi BINARY := bin/artifactapi
MODULE := git.unkin.net/unkin/artifactapi MODULE := git.unkin.net/unkin/artifactapi
@@ -12,7 +12,7 @@ check-go:
fi fi
build: check-go tidy build: check-go tidy
go build -ldflags="-s -w" -o $(BINARY) ./cmd/artifactapi go build -ldflags="-s -w -X main.version=$(VERSION)" -o $(BINARY) ./cmd/artifactapi
test: check-go test: check-go
go test -race -count=1 ./pkg/... ./internal/... go test -race -count=1 ./pkg/... ./internal/...
@@ -28,6 +28,11 @@ fmt: check-go
e2e: check-go e2e: check-go
TESTCONTAINERS_RYUK_DISABLED=true go test -tags=e2e -race -count=1 -timeout=5m ./e2e/... TESTCONTAINERS_RYUK_DISABLED=true go test -tags=e2e -race -count=1 -timeout=5m ./e2e/...
# Build the container, bring up the full docker-compose stack + a mock upstream,
# and run the black-box suite against the running product.
docker-e2e: check-go
./scripts/docker-e2e.sh
docker: docker:
docker build -t artifactapi:$(VERSION) . docker build -t artifactapi:$(VERSION) .
+3 -1
View File
@@ -13,6 +13,8 @@ import (
"git.unkin.net/unkin/artifactapi/internal/tui" "git.unkin.net/unkin/artifactapi/internal/tui"
) )
var version = "dev"
func main() { func main() {
if len(os.Args) > 1 && os.Args[1] == "tui" { if len(os.Args) > 1 && os.Args[1] == "tui" {
endpoint := os.Getenv("ARTIFACTAPI_ENDPOINT") endpoint := os.Getenv("ARTIFACTAPI_ENDPOINT")
@@ -42,7 +44,7 @@ func main() {
ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer cancel() defer cancel()
srv, err := server.New(cfg) srv, err := server.New(cfg, version)
if err != nil { if err != nil {
slog.Error("failed to create server", "error", err) slog.Error("failed to create server", "error", err)
os.Exit(1) os.Exit(1)
+18
View File
@@ -0,0 +1,18 @@
# Overlay for the dockerised end-to-end suite (scripts/docker-e2e.sh).
# Adds a static mock upstream that the artifactapi container proxies, so the
# caching tests are hermetic and need no internet access.
services:
mockupstream:
image: nginx:alpine
volumes:
- ./e2e-docker/fixtures:/usr/share/nginx/html:ro,z
# No host port needed: only the artifactapi container talks to it, and the
# tests compare served bytes against the on-disk fixtures.
artifactapi:
# The host port is set via ARTIFACTAPI_PORT (see scripts/docker-e2e.sh),
# defaulting to 8000; the e2e run uses 8001 to avoid colliding with a
# locally-running instance.
depends_on:
mockupstream:
condition: service_started
+1 -1
View File
@@ -2,7 +2,7 @@ services:
artifactapi: artifactapi:
build: . build: .
ports: ports:
- "8000:8000" - "${ARTIFACTAPI_PORT:-8000}:8000"
environment: environment:
LISTEN_ADDR: ":8000" LISTEN_ADDR: ":8000"
DBHOST: postgres DBHOST: postgres
+39
View File
@@ -0,0 +1,39 @@
# Dockerised end-to-end suite
Black-box tests that run against a fully **containerised** artifactapi stack
(built image + Postgres + Redis + MinIO) plus a static mock upstream. Unlike the
in-process `e2e/` suite (testcontainers, server run in-process), these only speak
HTTP to the running product, so they exercise the shipped container image.
## Run
```bash
make docker-e2e # build image, compose up, run suite, compose down
```
`scripts/docker-e2e.sh` builds and starts `docker-compose.yml` +
`docker-compose.e2e.yml`, waits for `/health`, then runs
`go test -tags=dockere2e ./e2e-docker/...` and tears everything down.
The stack publishes artifactapi on host port **8001** (to avoid colliding with a
local instance on 8000). Override with `ARTIFACTAPI_URL` to point the tests at an
already-running stack.
## Coverage
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact per remote package type (generic, docker,
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy) proxied through the
mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes
verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/`
index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge
across two members.
## Fixtures
`fixtures/` is served by the mock upstream at its web root. Paths mirror each
provider's upstream URL layout (e.g. `v2/...` for docker, `v1/providers/...` for
terraform). The RPM under `fixtures/rpmrepo/Packages/` is a real package so the
rpm provider can parse its metadata for repodata generation.
+76
View File
@@ -0,0 +1,76 @@
//go:build dockere2e
package e2edocker
import (
"bytes"
"fmt"
"net/http"
"testing"
)
// TestCachingPerProvider proxies one immutable artifact for every remote
// package type through the mock upstream and asserts: first fetch is served
// from the remote, the second from cache, and the bytes match the origin.
func TestCachingPerProvider(t *testing.T) {
cases := []struct {
pkgType string
// path is the request path under /api/v1/remote/<name>/. The provider
// derives the upstream URL from it (docker prepends /v2/, terraform
// prepends /v1/providers/), and the fixture lives at that resolved path.
path string
fixture string
}{
{"generic", "blobs/hello.bin", "blobs/hello.bin"},
{"npm", "mypkg/-/mypkg-1.0.0.tgz", "mypkg/-/mypkg-1.0.0.tgz"},
{"helm", "charts/mychart-1.0.0.tgz", "charts/mychart-1.0.0.tgz"},
{"pypi", "packages/foo-1.0-py3-none-any.whl", "packages/foo-1.0-py3-none-any.whl"},
{"rpm", "rpmrepo/Packages/e2e-testpkg-1.0-1.noarch.rpm", "rpmrepo/Packages/e2e-testpkg-1.0-1.noarch.rpm"},
{"alpine", "alpine/x86_64/testpkg-1.0-r0.apk", "alpine/x86_64/testpkg-1.0-r0.apk"},
{"puppet", "puppet-releases/author-mod-1.0.0.tar.gz", "puppet-releases/author-mod-1.0.0.tar.gz"},
{"goproxy", "goproxy/example.com/mod/@v/v1.0.0.zip", "goproxy/example.com/mod/@v/v1.0.0.zip"},
{"terraform", "hashicorp/aws/download/pkg.zip", "v1/providers/hashicorp/aws/download/pkg.zip"},
{"docker", "library/testimg/blobs/blobdata", "v2/library/testimg/blobs/blobdata"},
}
for _, tc := range cases {
t.Run(tc.pkgType, func(t *testing.T) {
name := "cache-" + tc.pkgType
createRepo(t, fmt.Sprintf(`{
"name": %q,
"package_type": %q,
"repo_type": "remote",
"base_url": %q,
"stale_on_error": true
}`, name, tc.pkgType, mockUpstream()))
defer deleteRepo(t, name)
want := fixtureBytes(t, tc.fixture)
url := api("/api/v1/remote/" + name + "/" + tc.path)
// First fetch: from remote.
resp, body := doRequest(t, http.MethodGet, url, nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("first fetch: status %d: %s", resp.StatusCode, body)
}
if src := resp.Header.Get("X-Artifact-Source"); src != "remote" {
t.Fatalf("first fetch source = %q, want remote", src)
}
if !bytes.Equal(body, want) {
t.Fatalf("first fetch body mismatch: got %d bytes, want %d", len(body), len(want))
}
// Second fetch: from cache, identical bytes.
resp, body = doRequest(t, http.MethodGet, url, nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("second fetch: status %d: %s", resp.StatusCode, body)
}
if src := resp.Header.Get("X-Artifact-Source"); src != "cache" {
t.Fatalf("second fetch source = %q, want cache", src)
}
if !bytes.Equal(body, want) {
t.Fatalf("cached body mismatch: got %d bytes, want %d", len(body), len(want))
}
})
}
}
Binary file not shown.
+1
View File
@@ -0,0 +1 @@
hello artifactapi generic blob
Binary file not shown.
+8
View File
@@ -0,0 +1,8 @@
apiVersion: v1
entries:
alpha:
- name: alpha
version: 1.0.0
urls:
- charts/alpha-1.0.0.tgz
generated: "2026-01-01T00:00:00Z"
+8
View File
@@ -0,0 +1,8 @@
apiVersion: v1
entries:
beta:
- name: beta
version: 2.0.0
urls:
- charts/beta-2.0.0.tgz
generated: "2026-01-01T00:00:00Z"
Binary file not shown.
+108
View File
@@ -0,0 +1,108 @@
//go:build dockere2e
// Package e2edocker holds the black-box end-to-end suite that runs against a
// fully dockerised artifactapi stack (see scripts/docker-e2e.sh). Unlike the
// in-process e2e suite, these tests only speak HTTP to the running container.
package e2edocker
import (
"bytes"
"io"
"net/http"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func baseURL() string {
if v := os.Getenv("ARTIFACTAPI_URL"); v != "" {
return strings.TrimRight(v, "/")
}
return "http://localhost:8000"
}
// mockUpstream is the base URL the artifactapi *container* uses to reach the
// static mock upstream. It is resolved on the compose network, not the host.
func mockUpstream() string {
if v := os.Getenv("MOCK_UPSTREAM_INTERNAL"); v != "" {
return strings.TrimRight(v, "/")
}
return "http://mockupstream"
}
func api(path string) string { return baseURL() + path }
func fixtureBytes(t *testing.T, rel string) []byte {
t.Helper()
b, err := os.ReadFile(filepath.Join("fixtures", rel))
if err != nil {
t.Fatalf("read fixture %s: %v", rel, err)
}
return b
}
func doRequest(t *testing.T, method, url string, body []byte, contentType string) (*http.Response, []byte) {
t.Helper()
var r io.Reader
if body != nil {
r = bytes.NewReader(body)
}
req, err := http.NewRequest(method, url, r)
if err != nil {
t.Fatalf("%s %s: %v", method, url, err)
}
if contentType != "" {
req.Header.Set("Content-Type", contentType)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("%s %s: %v", method, url, err)
}
defer resp.Body.Close()
respBody, _ := io.ReadAll(resp.Body)
return resp, respBody
}
func createRepo(t *testing.T, jsonBody string) {
t.Helper()
resp, body := doRequest(t, http.MethodPost, api("/api/v2/remotes"), []byte(jsonBody), "application/json")
if resp.StatusCode != http.StatusCreated {
t.Fatalf("create repo: status %d: %s", resp.StatusCode, body)
}
}
func deleteRepo(t *testing.T, name string) {
t.Helper()
doRequest(t, http.MethodDelete, api("/api/v2/remotes/"+name), nil, "")
}
func createVirtual(t *testing.T, jsonBody string) {
t.Helper()
resp, body := doRequest(t, http.MethodPost, api("/api/v2/virtuals"), []byte(jsonBody), "application/json")
if resp.StatusCode != http.StatusCreated {
t.Fatalf("create virtual: status %d: %s", resp.StatusCode, body)
}
}
func deleteVirtual(t *testing.T, name string) {
t.Helper()
doRequest(t, http.MethodDelete, api("/api/v2/virtuals/"+name), nil, "")
}
// getEventually retries a GET until it returns 200 or the deadline passes. Used
// for asynchronously-generated artifacts (e.g. rpm repodata after upload).
func getEventually(t *testing.T, url string, timeout time.Duration) (*http.Response, []byte) {
t.Helper()
deadline := time.Now().Add(timeout)
var resp *http.Response
var body []byte
for {
resp, body = doRequest(t, http.MethodGet, url, nil, "")
if resp.StatusCode == http.StatusOK || time.Now().After(deadline) {
return resp, body
}
time.Sleep(250 * time.Millisecond)
}
}
+93
View File
@@ -0,0 +1,93 @@
//go:build dockere2e
package e2edocker
import (
"bytes"
"net/http"
"strings"
"testing"
"time"
)
func uploadFile(t *testing.T, repo, filePath string, body []byte, contentType string) {
t.Helper()
url := api("/api/v2/remotes/" + repo + "/files/" + filePath)
resp, respBody := doRequest(t, http.MethodPut, url, body, contentType)
if resp.StatusCode != http.StatusCreated {
t.Fatalf("upload %s: status %d: %s", filePath, resp.StatusCode, respBody)
}
}
// TestLocalGenericUpload uploads a generic file and downloads it back.
func TestLocalGenericUpload(t *testing.T) {
createRepo(t, `{"name":"local-generic","package_type":"generic","repo_type":"local"}`)
defer deleteRepo(t, "local-generic")
content := []byte("artifactapi local generic upload payload")
uploadFile(t, "local-generic", "data/hello.bin", content, "application/octet-stream")
resp, body := doRequest(t, http.MethodGet, api("/api/v1/local/local-generic/data/hello.bin"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("download: status %d: %s", resp.StatusCode, body)
}
if !bytes.Equal(body, content) {
t.Fatalf("downloaded content mismatch")
}
}
// TestLocalPyPIUpload uploads a wheel and validates the generated simple index.
func TestLocalPyPIUpload(t *testing.T) {
createRepo(t, `{"name":"local-pypi","package_type":"pypi","repo_type":"local"}`)
defer deleteRepo(t, "local-pypi")
wheel := fixtureBytes(t, "packages/foo-1.0-py3-none-any.whl")
uploadFile(t, "local-pypi", "foo-1.0-py3-none-any.whl", wheel, "application/zip")
// Root index lists the package.
resp, body := doRequest(t, http.MethodGet, api("/api/v1/local/local-pypi/simple/"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("simple index: status %d: %s", resp.StatusCode, body)
}
if !strings.Contains(string(body), "foo") {
t.Fatalf("simple index missing package 'foo': %s", body)
}
// Per-package index lists the wheel file.
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-pypi/simple/foo/"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("package index: status %d: %s", resp.StatusCode, body)
}
if !strings.Contains(string(body), "foo-1.0-py3-none-any.whl") {
t.Fatalf("package index missing wheel: %s", body)
}
// The wheel downloads back byte-identical.
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-pypi/foo/foo-1.0-py3-none-any.whl"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("download wheel: status %d: %s", resp.StatusCode, body)
}
if !bytes.Equal(body, wheel) {
t.Fatalf("wheel content mismatch")
}
}
// TestLocalRPMRepodata uploads a real RPM and validates that repodata is
// generated automatically (the special rpm-local feature).
func TestLocalRPMRepodata(t *testing.T) {
createRepo(t, `{"name":"local-rpm","package_type":"rpm","repo_type":"local"}`)
defer deleteRepo(t, "local-rpm")
rpm := fixtureBytes(t, "rpmrepo/Packages/e2e-testpkg-1.0-1.noarch.rpm")
uploadFile(t, "local-rpm", "e2e-testpkg-1.0-1.noarch.rpm", rpm, "application/x-rpm")
// repodata is generated asynchronously after upload; poll for it.
resp, body := getEventually(t, api("/api/v1/local/local-rpm/repodata/repomd.xml"), 15*time.Second)
if resp.StatusCode != http.StatusOK {
t.Fatalf("repomd.xml: status %d: %s", resp.StatusCode, body)
}
s := string(body)
if !strings.Contains(s, "<repomd") || !strings.Contains(s, "primary") {
t.Fatalf("repomd.xml not a valid repodata document: %s", s)
}
}
+134
View File
@@ -0,0 +1,134 @@
//go:build dockere2e
package e2edocker
import (
"encoding/json"
"net/http"
"testing"
)
func TestHealth(t *testing.T) {
resp, body := doRequest(t, http.MethodGet, api("/health"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("health: status %d: %s", resp.StatusCode, body)
}
}
// TestRemoteLifecycle covers add/change/delete for a remote repository.
func TestRemoteLifecycle(t *testing.T) {
createRepo(t, `{
"name": "crud-remote",
"package_type": "generic",
"repo_type": "remote",
"base_url": "https://example.com",
"mutable_ttl": 600,
"stale_on_error": true
}`)
defer deleteRepo(t, "crud-remote")
got := getRepo(t, "crud-remote")
if got["base_url"] != "https://example.com" || got["mutable_ttl"].(float64) != 600 {
t.Fatalf("unexpected created remote: %v", got)
}
// change
resp, body := doRequest(t, http.MethodPut, api("/api/v2/remotes/crud-remote"), []byte(`{
"package_type": "generic",
"base_url": "https://updated.example.com",
"mutable_ttl": 120,
"stale_on_error": true
}`), "application/json")
if resp.StatusCode != http.StatusOK {
t.Fatalf("update remote: status %d: %s", resp.StatusCode, body)
}
got = getRepo(t, "crud-remote")
if got["base_url"] != "https://updated.example.com" || got["mutable_ttl"].(float64) != 120 {
t.Fatalf("update not applied: %v", got)
}
// delete
resp, _ = doRequest(t, http.MethodDelete, api("/api/v2/remotes/crud-remote"), nil, "")
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("delete remote: status %d", resp.StatusCode)
}
resp, _ = doRequest(t, http.MethodGet, api("/api/v2/remotes/crud-remote"), nil, "")
if resp.StatusCode != http.StatusNotFound {
t.Fatalf("expected 404 after delete, got %d", resp.StatusCode)
}
}
// TestLocalLifecycle covers add/delete for a local repository.
func TestLocalLifecycle(t *testing.T) {
createRepo(t, `{
"name": "crud-local",
"package_type": "generic",
"repo_type": "local"
}`)
defer deleteRepo(t, "crud-local")
got := getRepo(t, "crud-local")
if got["repo_type"] != "local" {
t.Fatalf("expected repo_type local, got %v", got["repo_type"])
}
resp, _ := doRequest(t, http.MethodDelete, api("/api/v2/remotes/crud-local"), nil, "")
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("delete local: status %d", resp.StatusCode)
}
}
// TestVirtualLifecycle covers add/change/delete for a virtual repository.
func TestVirtualLifecycle(t *testing.T) {
createRepo(t, `{"name":"vmem-a","package_type":"helm","repo_type":"remote","base_url":"https://a.example.com","stale_on_error":true}`)
createRepo(t, `{"name":"vmem-b","package_type":"helm","repo_type":"remote","base_url":"https://b.example.com","stale_on_error":true}`)
defer deleteRepo(t, "vmem-a")
defer deleteRepo(t, "vmem-b")
createVirtual(t, `{
"name": "crud-virtual",
"package_type": "helm",
"members": ["vmem-a"]
}`)
defer deleteVirtual(t, "crud-virtual")
// change members
resp, body := doRequest(t, http.MethodPut, api("/api/v2/virtuals/crud-virtual"), []byte(`{
"package_type": "helm",
"members": ["vmem-a", "vmem-b"]
}`), "application/json")
if resp.StatusCode != http.StatusOK {
t.Fatalf("update virtual: status %d: %s", resp.StatusCode, body)
}
resp, body = doRequest(t, http.MethodGet, api("/api/v2/virtuals/crud-virtual"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("get virtual: status %d: %s", resp.StatusCode, body)
}
var v map[string]any
if err := json.Unmarshal(body, &v); err != nil {
t.Fatalf("decode virtual: %v", err)
}
members, _ := v["members"].([]any)
if len(members) != 2 {
t.Fatalf("expected 2 members after update, got %v", v["members"])
}
resp, _ = doRequest(t, http.MethodDelete, api("/api/v2/virtuals/crud-virtual"), nil, "")
if resp.StatusCode != http.StatusNoContent {
t.Fatalf("delete virtual: status %d", resp.StatusCode)
}
}
func getRepo(t *testing.T, name string) map[string]any {
t.Helper()
resp, body := doRequest(t, http.MethodGet, api("/api/v2/remotes/"+name), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("get remote %s: status %d: %s", name, resp.StatusCode, body)
}
var m map[string]any
if err := json.Unmarshal(body, &m); err != nil {
t.Fatalf("decode remote %s: %v", name, err)
}
return m
}
+54
View File
@@ -0,0 +1,54 @@
//go:build dockere2e
package e2edocker
import (
"net/http"
"strings"
"testing"
)
// TestVirtualPyPIMerge uploads different packages to two pypi locals and
// checks that a virtual over them serves a merged simple index.
func TestVirtualPyPIMerge(t *testing.T) {
createRepo(t, `{"name":"pmerge-a","package_type":"pypi","repo_type":"local"}`)
createRepo(t, `{"name":"pmerge-b","package_type":"pypi","repo_type":"local"}`)
defer deleteRepo(t, "pmerge-a")
defer deleteRepo(t, "pmerge-b")
uploadFile(t, "pmerge-a", "foo-1.0-py3-none-any.whl", fixtureBytes(t, "packages/foo-1.0-py3-none-any.whl"), "application/zip")
uploadFile(t, "pmerge-b", "bar-2.0-py3-none-any.whl", []byte("bar wheel payload"), "application/zip")
createVirtual(t, `{"name":"pmerge-v","package_type":"pypi","members":["pmerge-a","pmerge-b"]}`)
defer deleteVirtual(t, "pmerge-v")
resp, body := doRequest(t, http.MethodGet, api("/api/v1/virtual/pmerge-v/simple/"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("virtual simple index: status %d: %s", resp.StatusCode, body)
}
s := string(body)
if !strings.Contains(s, "foo") || !strings.Contains(s, "bar") {
t.Fatalf("merged index missing a member package (want foo and bar): %s", s)
}
}
// TestVirtualHelmMerge points two helm remotes at mock index.yaml documents
// with distinct charts and checks the virtual merges both into one index.
func TestVirtualHelmMerge(t *testing.T) {
createRepo(t, `{"name":"hmerge-a","package_type":"helm","repo_type":"remote","base_url":"`+mockUpstream()+`/helm-a","stale_on_error":true}`)
createRepo(t, `{"name":"hmerge-b","package_type":"helm","repo_type":"remote","base_url":"`+mockUpstream()+`/helm-b","stale_on_error":true}`)
defer deleteRepo(t, "hmerge-a")
defer deleteRepo(t, "hmerge-b")
createVirtual(t, `{"name":"hmerge-v","package_type":"helm","members":["hmerge-a","hmerge-b"]}`)
defer deleteVirtual(t, "hmerge-v")
resp, body := doRequest(t, http.MethodGet, api("/api/v1/virtual/hmerge-v/index.yaml"), nil, "")
if resp.StatusCode != http.StatusOK {
t.Fatalf("virtual index.yaml: status %d: %s", resp.StatusCode, body)
}
s := string(body)
if !strings.Contains(s, "alpha") || !strings.Contains(s, "beta") {
t.Fatalf("merged helm index missing a member chart (want alpha and beta): %s", s)
}
}
+1 -1
View File
@@ -95,7 +95,7 @@ func TestMain(m *testing.M) {
} }
cfg.ListenAddr = "127.0.0.1:0" cfg.ListenAddr = "127.0.0.1:0"
srv, err := server.New(cfg) srv, err := server.New(cfg, "e2e-test")
if err != nil { if err != nil {
log.Fatalf("server: %v", err) log.Fatalf("server: %v", err)
} }
+24
View File
@@ -24,6 +24,30 @@ func TestRoot(t *testing.T) {
} }
} }
func TestRemoteUpstreamTimeouts(t *testing.T) {
createRemote(t, `{
"name": "timeout-test",
"package_type": "generic",
"base_url": "https://example.com",
"stale_on_error": true,
"upstream_dial_timeout": 3,
"upstream_tls_timeout": 4,
"upstream_response_header_timeout": 5
}`)
defer deleteRemote(t, "timeout-test")
remote := getJSON(t, apiURL("/api/v2/remotes/timeout-test"))
for field, want := range map[string]float64{
"upstream_dial_timeout": 3,
"upstream_tls_timeout": 4,
"upstream_response_header_timeout": 5,
} {
if got, _ := remote[field].(float64); got != want {
t.Errorf("%s: got %v, want %v", field, remote[field], want)
}
}
}
func TestRemoteCRUD(t *testing.T) { func TestRemoteCRUD(t *testing.T) {
createRemote(t, `{ createRemote(t, `{
"name": "test-generic", "name": "test-generic",
+33
View File
@@ -24,6 +24,39 @@ func TestProxyBlocklist(t *testing.T) {
assertStatus(t, apiURL("/api/v1/remote/blocklist-test/malware.exe"), http.StatusForbidden) assertStatus(t, apiURL("/api/v1/remote/blocklist-test/malware.exe"), http.StatusForbidden)
} }
func TestProxyHeadBlocklist(t *testing.T) {
createRemote(t, `{
"name": "head-block-test",
"package_type": "generic",
"base_url": "https://example.com",
"blocklist": ["\\.exe$"],
"stale_on_error": true
}`)
defer deleteRemote(t, "head-block-test")
req, _ := http.NewRequest(http.MethodHead, apiURL("/v2/head-block-test/malware.exe"), nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("HEAD: %v", err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusForbidden {
t.Fatalf("HEAD blocklisted path: got %d, want 403", resp.StatusCode)
}
}
func TestProxyHeadUnknownRemote(t *testing.T) {
req, _ := http.NewRequest(http.MethodHead, apiURL("/v2/nonexistent/some/path"), nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("HEAD: %v", err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusNotFound {
t.Fatalf("HEAD unknown remote: got %d, want 404", resp.StatusCode)
}
}
func TestProxyPatterns(t *testing.T) { func TestProxyPatterns(t *testing.T) {
createRemote(t, `{ createRemote(t, `{
"name": "patterns-test", "name": "patterns-test",
+51 -1
View File
@@ -37,6 +37,20 @@ func (h *ProxyHandler) Routes() chi.Router {
return r return r
} }
func (h *ProxyHandler) DockerV2Routes() chi.Router {
r := chi.NewRouter()
r.Get("/", h.handleDockerPing)
r.Head("/", h.handleDockerPing)
r.Get("/{remoteName}/*", h.handleProxy)
r.Head("/{remoteName}/*", h.handleProxyHead)
return r
}
func (h *ProxyHandler) handleDockerPing(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Docker-Distribution-Api-Version", "registry/2.0")
w.WriteHeader(http.StatusOK)
}
func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) { func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
remoteName := chi.URLParam(r, "remoteName") remoteName := chi.URLParam(r, "remoteName")
path := chi.URLParam(r, "*") path := chi.URLParam(r, "*")
@@ -53,7 +67,7 @@ func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
return return
} }
result, err := h.engine.Fetch(r.Context(), *remote, path, prov) result, err := h.engine.Fetch(r.Context(), *remote, path, prov, r.Header)
if err != nil { if err != nil {
var proxyErr *proxy.ProxyError var proxyErr *proxy.ProxyError
if errors.As(err, &proxyErr) { if errors.As(err, &proxyErr) {
@@ -75,6 +89,42 @@ func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
io.Copy(w, result.Reader) io.Copy(w, result.Reader)
} }
func (h *ProxyHandler) handleProxyHead(w http.ResponseWriter, r *http.Request) {
remoteName := chi.URLParam(r, "remoteName")
path := chi.URLParam(r, "*")
remote, err := h.db.GetRemote(r.Context(), remoteName)
if err != nil {
http.Error(w, fmt.Sprintf("remote %q not found", remoteName), http.StatusNotFound)
return
}
prov, err := provider.Get(remote.PackageType)
if err != nil {
http.Error(w, fmt.Sprintf("no provider for %q", remote.PackageType), http.StatusInternalServerError)
return
}
result, err := h.engine.Head(r.Context(), *remote, path, prov)
if err != nil {
var proxyErr *proxy.ProxyError
if errors.As(err, &proxyErr) {
http.Error(w, proxyErr.Message, proxyErr.Status)
return
}
slog.Error("proxy head failed", "remote", remoteName, "path", path, "error", err)
http.Error(w, "bad gateway", http.StatusBadGateway)
return
}
w.Header().Set("Content-Type", result.ContentType)
w.Header().Set("X-Artifact-Source", result.Source)
if result.Size > 0 {
w.Header().Set("Content-Length", fmt.Sprintf("%d", result.Size))
}
w.WriteHeader(http.StatusOK)
}
func (h *ProxyHandler) handleVirtual(w http.ResponseWriter, r *http.Request) { func (h *ProxyHandler) handleVirtual(w http.ResponseWriter, r *http.Request) {
virtualName := chi.URLParam(r, "virtualName") virtualName := chi.URLParam(r, "virtualName")
path := chi.URLParam(r, "*") path := chi.URLParam(r, "*")
+20
View File
@@ -0,0 +1,20 @@
package v1
import (
"crypto/tls"
"net/http"
"testing"
)
func TestScheme(t *testing.T) {
if got := scheme(&http.Request{TLS: &tls.ConnectionState{}}); got != "https" {
t.Errorf("TLS request scheme = %q, want https", got)
}
r := &http.Request{Header: http.Header{"X-Forwarded-Proto": {"https"}}}
if got := scheme(r); got != "https" {
t.Errorf("X-Forwarded-Proto scheme = %q, want https", got)
}
if got := scheme(&http.Request{Header: http.Header{}}); got != "http" {
t.Errorf("default scheme = %q, want http", got)
}
}
+130
View File
@@ -0,0 +1,130 @@
package v2
import (
"context"
"io"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
var testDSN string
func TestMain(m *testing.M) {
ctx := context.Background()
dsn, terminate, err := testsupport.StartPostgres(ctx)
if err != nil {
os.Exit(m.Run())
}
testDSN = dsn
code := m.Run()
terminate()
if code != 0 {
os.Exit(code)
}
}
// closedDB returns a DB whose pool has been closed, so every query fails —
// used to drive the handlers' error branches.
func closedDB(t *testing.T) *database.DB {
t.Helper()
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatalf("new db: %v", err)
}
db.Close()
return db
}
func do(t *testing.T, h http.Handler, method, path, body string) int {
t.Helper()
var r io.Reader
if body != "" {
r = strings.NewReader(body)
}
req := httptest.NewRequest(method, path, r)
w := httptest.NewRecorder()
h.ServeHTTP(w, req)
return w.Code
}
func TestRemotesErrorPaths(t *testing.T) {
h := NewRemotesHandler(closedDB(t)).Routes()
if c := do(t, h, "GET", "/", ""); c != 500 {
t.Errorf("list with dead db = %d, want 500", c)
}
if c := do(t, h, "POST", "/", `{"name":"x","package_type":"generic","repo_type":"remote","base_url":"https://x"}`); c != 500 {
t.Errorf("create with dead db = %d, want 500", c)
}
if c := do(t, h, "PUT", "/x", `{"package_type":"generic","base_url":"https://x"}`); c != 500 {
t.Errorf("update with dead db = %d, want 500", c)
}
if c := do(t, h, "GET", "/x", ""); c != 404 {
t.Errorf("get missing = %d, want 404", c)
}
if c := do(t, h, "DELETE", "/x", ""); c != 500 {
t.Errorf("delete with dead db = %d, want 500", c)
}
// Bad request bodies never reach the db.
if c := do(t, h, "POST", "/", `not json`); c != 400 {
t.Errorf("invalid json = %d, want 400", c)
}
}
func TestVirtualsErrorPaths(t *testing.T) {
h := NewVirtualsHandler(closedDB(t)).Routes()
if c := do(t, h, "GET", "/", ""); c != 500 {
t.Errorf("list = %d, want 500", c)
}
if c := do(t, h, "GET", "/x", ""); c != 404 {
t.Errorf("get missing = %d, want 404", c)
}
if c := do(t, h, "POST", "/", `{"name":"v","package_type":"helm","members":["a"]}`); c != 500 {
t.Errorf("create = %d, want 500", c)
}
if c := do(t, h, "PUT", "/v", `{"package_type":"helm","members":["a"]}`); c != 500 {
t.Errorf("update = %d, want 500", c)
}
if c := do(t, h, "DELETE", "/v", ""); c != 500 {
t.Errorf("delete = %d, want 500", c)
}
}
func TestStatsErrorPaths(t *testing.T) {
h := NewStatsHandler(closedDB(t)).Routes()
for _, p := range []string{"/", "/top-remotes", "/top-files-by-hits", "/top-files-by-bandwidth"} {
if c := do(t, h, "GET", p, ""); c != 500 {
t.Errorf("stats %s = %d, want 500", p, c)
}
}
}
func TestLocalErrorPaths(t *testing.T) {
h := NewLocalHandler(closedDB(t), nil).Routes()
// GetRemote fails on the closed db -> not found.
if c := do(t, h, "PUT", "/x/files/a.bin", "data"); c != 404 {
t.Errorf("upload unknown repo = %d, want 404", c)
}
// download / remove hit the db and 500.
if c := do(t, h, "GET", "/x/files/a.bin", ""); c != 500 {
t.Errorf("download = %d, want 500", c)
}
if c := do(t, h, "DELETE", "/x/files/a.bin", ""); c != 500 {
t.Errorf("remove = %d, want 500", c)
}
}
func TestLocalHandlerDBAccessor(t *testing.T) {
db := closedDB(t)
if NewLocalHandler(db, nil).DB() != db {
t.Error("DB() should return the handler's database")
}
}
+23 -1
View File
@@ -185,13 +185,35 @@ func (h *LocalHandler) remove(w http.ResponseWriter, r *http.Request) {
repoName := chi.URLParam(r, "name") repoName := chi.URLParam(r, "name")
filePath := chi.URLParam(r, "*") filePath := chi.URLParam(r, "*")
if err := h.db.DeleteLocalFile(r.Context(), repoName, filePath); err != nil { if err := deleteLocalFile(r.Context(), h.db, repoName, filePath); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError) http.Error(w, err.Error(), http.StatusInternalServerError)
return return
} }
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
} }
// deleteLocalFile removes a local file and runs the provider's post-delete hook,
// so provider-derived state (e.g. RPM metadata that feeds generated repodata)
// stops referencing a package that no longer exists.
func deleteLocalFile(ctx context.Context, db *database.DB, repoName, filePath string) error {
if err := db.DeleteLocalFile(ctx, repoName, filePath); err != nil {
return err
}
remote, err := db.GetRemote(ctx, repoName)
if err != nil {
return nil // file is gone; no repo left to resolve a cleanup hook from
}
prov, err := provider.Get(remote.PackageType)
if err != nil {
return nil
}
if hook, ok := prov.(provider.PostDeleteHook); ok {
return hook.AfterDelete(ctx, repoName, filePath, db)
}
return nil
}
func (h *LocalHandler) DB() *database.DB { func (h *LocalHandler) DB() *database.DB {
return h.db return h.db
} }
@@ -0,0 +1,75 @@
package v2
import (
"context"
"net/http/httptest"
"testing"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/provider"
_ "git.unkin.net/unkin/artifactapi/internal/provider/rpm" // register the rpm provider so its PostDeleteHook runs
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// TestLocalEvictCleansRPMMetadata verifies that evicting an RPM from a local
// repo also removes the derived rpm_metadata row, so generated repodata stops
// listing the deleted package.
func TestLocalEvictCleansRPMMetadata(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
ctx := context.Background()
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
const repo = "rpm-evict-cleanup"
if err := db.CreateRemote(ctx, &models.Remote{Name: repo, PackageType: models.PackageRPM, RepoType: models.RepoTypeLocal}); err != nil {
t.Fatal(err)
}
const hash = "sha256:bb22"
const path = "Packages/example-0.1.0-1.x86_64.rpm"
if err := db.UpsertBlob(ctx, hash, "blobs/bb/22", 2048, "application/x-rpm"); err != nil {
t.Fatal(err)
}
if err := db.CreateLocalFile(ctx, repo, path, hash); err != nil {
t.Fatal(err)
}
if err := db.InsertRPMMetadata(ctx, &provider.RPMMetadata{
RepoName: repo, FilePath: path, ContentHash: hash,
Name: "example", Version: "0.1.0", Release: "1", Arch: "x86_64",
Requires: []provider.RPMDep{}, Provides: []provider.RPMDep{},
Files: []provider.RPMFile{}, Changelogs: []provider.RPMChangelog{},
}); err != nil {
t.Fatal(err)
}
h := NewObjectsHandler(db)
router := chi.NewRouter()
router.Route("/locals/{name}/objects", func(r chi.Router) {
r.Delete("/*", h.LocalRoutes().ServeHTTP)
})
del := httptest.NewRequest("DELETE", "/locals/"+repo+"/objects/"+path, nil)
dw := httptest.NewRecorder()
router.ServeHTTP(dw, del)
if dw.Code != 204 {
t.Fatalf("evict = %d, want 204", dw.Code)
}
if f, _ := db.GetLocalFile(ctx, repo, path); f != nil {
t.Fatalf("local file still present after evict: %+v", f)
}
entries, err := db.ListRPMMetadataEntries(ctx, repo)
if err != nil {
t.Fatal(err)
}
if len(entries) != 0 {
t.Fatalf("rpm_metadata still present after evict: %+v", entries)
}
}
+88
View File
@@ -0,0 +1,88 @@
package v2
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// TestLocalUploadStoreFailure covers the upload handlers' store-error branches
// by killing the object store after a successful upload.
func TestLocalUploadStoreFailure(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
ctx := context.Background()
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
conn, termMinio, err := testsupport.StartMinio(ctx)
if err != nil {
t.Skip("minio unavailable")
}
var store *storage.S3
for i := 0; i < 20; i++ {
if store, err = storage.NewS3(conn.Endpoint, conn.AccessKey, conn.SecretKey, "fault", false, ""); err == nil {
break
}
time.Sleep(500 * time.Millisecond)
}
if err != nil {
termMinio()
t.Fatal(err)
}
for _, pt := range []models.PackageType{models.PackageGeneric, models.PackagePyPI} {
if err := db.CreateRemote(ctx, &models.Remote{Name: "fault-" + string(pt), PackageType: pt, RepoType: models.RepoTypeLocal}); err != nil {
t.Fatal(err)
}
}
h := NewLocalHandler(db, store)
router := chi.NewRouter()
router.Route("/remotes/{name}/files", func(r chi.Router) {
r.Put("/*", h.Routes().ServeHTTP)
})
srv := httptest.NewServer(router)
defer srv.Close()
put := func(name, path, body string) int {
rq, _ := http.NewRequest("PUT", srv.URL+"/remotes/"+name+"/files/"+path, strings.NewReader(body))
resp, err := http.DefaultClient.Do(rq)
if err != nil {
t.Fatalf("put: %v", err)
}
resp.Body.Close()
return resp.StatusCode
}
// Sanity: uploads succeed while the store is up.
if c := put("fault-generic", "ok.bin", "data"); c != 201 {
t.Fatalf("generic upload while up = %d", c)
}
if c := put("fault-pypi", "foo-1.0-py3-none-any.whl", "wheel"); c != 201 {
t.Fatalf("pypi upload while up = %d", c)
}
// Kill the store; subsequent CAS.Store calls fail -> 500.
termMinio()
if c := put("fault-generic", "after.bin", "data"); c != 500 {
t.Errorf("generic upload after store down = %d, want 500", c)
}
if c := put("fault-pypi", "bar-1.0-py3-none-any.whl", "wheel"); c != 500 {
t.Errorf("pypi upload after store down = %d, want 500", c)
}
}
+78
View File
@@ -0,0 +1,78 @@
package v2
import (
"context"
"encoding/json"
"net/http/httptest"
"testing"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// TestLocalObjectsListing verifies that files uploaded to a local repo (which
// live in local_files, not artifacts) are listed by the local objects endpoint
// and can be evicted through it.
func TestLocalObjectsListing(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
ctx := context.Background()
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
const repo = "rpm-local-objs"
if err := db.CreateRemote(ctx, &models.Remote{Name: repo, PackageType: models.PackageRPM, RepoType: models.RepoTypeLocal}); err != nil {
t.Fatal(err)
}
const hash = "sha256:aa11"
const path = "Packages/example-0.1.0-1.x86_64.rpm"
if err := db.UpsertBlob(ctx, hash, "blobs/aa/11", 1234, "application/x-rpm"); err != nil {
t.Fatal(err)
}
if err := db.CreateLocalFile(ctx, repo, path, hash); err != nil {
t.Fatal(err)
}
h := NewObjectsHandler(db)
router := chi.NewRouter()
router.Route("/locals/{name}/objects", func(r chi.Router) {
r.Get("/", h.LocalRoutes().ServeHTTP)
r.Delete("/*", h.LocalRoutes().ServeHTTP)
})
// The uploaded package must appear in the listing with its blob size.
req := httptest.NewRequest("GET", "/locals/"+repo+"/objects", nil)
w := httptest.NewRecorder()
router.ServeHTTP(w, req)
if w.Code != 200 {
t.Fatalf("list = %d, want 200", w.Code)
}
var got []models.Artifact
if err := json.Unmarshal(w.Body.Bytes(), &got); err != nil {
t.Fatalf("decode: %v", err)
}
if len(got) != 1 {
t.Fatalf("got %d objects, want 1", len(got))
}
if got[0].Path != path || got[0].SizeBytes != 1234 || got[0].ContentHash != hash {
t.Fatalf("unexpected object: %+v", got[0])
}
// Eviction removes it from local_files.
del := httptest.NewRequest("DELETE", "/locals/"+repo+"/objects/"+path, nil)
dw := httptest.NewRecorder()
router.ServeHTTP(dw, del)
if dw.Code != 204 {
t.Fatalf("evict = %d, want 204", dw.Code)
}
if f, _ := db.GetLocalFile(ctx, repo, path); f != nil {
t.Fatalf("file still present after evict: %+v", f)
}
}
+41 -4
View File
@@ -25,9 +25,18 @@ func (h *ObjectsHandler) Routes() chi.Router {
return r return r
} }
func (h *ObjectsHandler) list(w http.ResponseWriter, r *http.Request) { // LocalRoutes lists and evicts objects for local repos, which live in the
remoteName := chi.URLParam(r, "name") // local_files table rather than the artifacts table used by remotes.
limit, _ := strconv.Atoi(r.URL.Query().Get("per_page")) func (h *ObjectsHandler) LocalRoutes() chi.Router {
r := chi.NewRouter()
r.Get("/", h.listLocal)
r.Delete("/*", h.evictLocal)
return r
}
// pageBounds parses the shared page/per_page query params into a SQL limit and offset.
func pageBounds(r *http.Request) (limit, offset int) {
limit, _ = strconv.Atoi(r.URL.Query().Get("per_page"))
if limit <= 0 || limit > 5000 { if limit <= 0 || limit > 5000 {
limit = 50 limit = 50
} }
@@ -35,7 +44,12 @@ func (h *ObjectsHandler) list(w http.ResponseWriter, r *http.Request) {
if page <= 0 { if page <= 0 {
page = 1 page = 1
} }
offset := (page - 1) * limit return limit, (page - 1) * limit
}
func (h *ObjectsHandler) list(w http.ResponseWriter, r *http.Request) {
remoteName := chi.URLParam(r, "name")
limit, offset := pageBounds(r)
artifacts, err := h.db.ListArtifacts(r.Context(), remoteName, limit, offset) artifacts, err := h.db.ListArtifacts(r.Context(), remoteName, limit, offset)
if err != nil { if err != nil {
@@ -45,6 +59,29 @@ func (h *ObjectsHandler) list(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusOK, artifacts) writeJSON(w, http.StatusOK, artifacts)
} }
func (h *ObjectsHandler) listLocal(w http.ResponseWriter, r *http.Request) {
repoName := chi.URLParam(r, "name")
limit, offset := pageBounds(r)
artifacts, err := h.db.ListLocalArtifacts(r.Context(), repoName, limit, offset)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, artifacts)
}
func (h *ObjectsHandler) evictLocal(w http.ResponseWriter, r *http.Request) {
repoName := chi.URLParam(r, "name")
path := chi.URLParam(r, "*")
if err := deleteLocalFile(r.Context(), h.db, repoName, path); err != nil {
http.Error(w, fmt.Sprintf("evict failed: %v", err), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
func (h *ObjectsHandler) evict(w http.ResponseWriter, r *http.Request) { func (h *ObjectsHandler) evict(w http.ResponseWriter, r *http.Request) {
remoteName := chi.URLParam(r, "name") remoteName := chi.URLParam(r, "name")
path := chi.URLParam(r, "*") path := chi.URLParam(r, "*")
+8
View File
@@ -69,6 +69,10 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
http.Error(w, "base_url is required for remote repositories", http.StatusBadRequest) http.Error(w, "base_url is required for remote repositories", http.StatusBadRequest)
return return
} }
if err := remote.ValidatePatterns(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := h.db.CreateRemote(r.Context(), &remote); err != nil { if err := h.db.CreateRemote(r.Context(), &remote); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError) http.Error(w, err.Error(), http.StatusInternalServerError)
return return
@@ -84,6 +88,10 @@ func (h *RemotesHandler) update(w http.ResponseWriter, r *http.Request) {
return return
} }
remote.Name = name remote.Name = name
if err := remote.ValidatePatterns(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := h.db.UpdateRemote(r.Context(), &remote); err != nil { if err := h.db.UpdateRemote(r.Context(), &remote); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError) http.Error(w, err.Error(), http.StatusInternalServerError)
return return
+23
View File
@@ -0,0 +1,23 @@
package auth
import (
"encoding/base64"
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestBasicHeaders(t *testing.T) {
h := BasicHeaders(models.Remote{Username: "alice", Password: "secret"})
got := h.Get("Authorization")
want := "Basic " + base64.StdEncoding.EncodeToString([]byte("alice:secret"))
if got != want {
t.Errorf("Authorization = %q, want %q", got, want)
}
}
func TestBasicHeadersNoUser(t *testing.T) {
if h := BasicHeaders(models.Remote{}); h.Get("Authorization") != "" {
t.Error("expected no Authorization header without a username")
}
}
+133
View File
@@ -0,0 +1,133 @@
package cache
import (
"context"
"os"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
var testRedis *Redis
func TestMain(m *testing.M) {
ctx := context.Background()
url, terminate, err := testsupport.StartRedis(ctx)
if err != nil {
os.Exit(m.Run())
}
r, err := NewRedis(url)
if err != nil {
terminate()
panic(err)
}
testRedis = r
code := m.Run()
r.Close()
terminate()
if code != 0 {
os.Exit(code)
}
}
func requireRedis(t *testing.T) {
t.Helper()
if testRedis == nil {
t.Skip("Docker unavailable; skipping cache integration test")
}
}
func TestNewRedisInvalid(t *testing.T) {
if _, err := NewRedis("://bad-url"); err == nil {
t.Error("expected error for invalid redis URL")
}
}
func TestTTL(t *testing.T) {
requireRedis(t)
ctx := context.Background()
if fresh, _ := testRedis.CheckTTL(ctx, "r", "missing"); fresh {
t.Error("missing key should not be fresh")
}
if err := testRedis.SetTTL(ctx, "r", "p", time.Minute); err != nil {
t.Fatal(err)
}
if fresh, err := testRedis.CheckTTL(ctx, "r", "p"); err != nil || !fresh {
t.Errorf("expected fresh after SetTTL: %v %v", fresh, err)
}
}
func TestLock(t *testing.T) {
requireRedis(t)
ctx := context.Background()
ok, err := testRedis.AcquireLock(ctx, "r", "lockpath", time.Minute)
if err != nil || !ok {
t.Fatalf("first acquire should succeed: %v %v", ok, err)
}
if ok, _ := testRedis.AcquireLock(ctx, "r", "lockpath", time.Minute); ok {
t.Error("second acquire should fail while held")
}
if err := testRedis.ReleaseLock(ctx, "r", "lockpath"); err != nil {
t.Fatal(err)
}
if ok, _ := testRedis.AcquireLock(ctx, "r", "lockpath", time.Minute); !ok {
t.Error("acquire should succeed after release")
}
}
func TestETagAndToken(t *testing.T) {
requireRedis(t)
ctx := context.Background()
if v, _ := testRedis.GetETag(ctx, "r", "missing"); v != "" {
t.Error("missing etag should be empty")
}
testRedis.SetETag(ctx, "r", "p", `"abc"`, time.Minute)
if v, _ := testRedis.GetETag(ctx, "r", "p"); v != `"abc"` {
t.Errorf("etag = %q", v)
}
if v, _ := testRedis.GetToken(ctx, "missing"); v != "" {
t.Error("missing token should be empty")
}
testRedis.SetToken(ctx, "key", "tok", time.Minute)
if v, _ := testRedis.GetToken(ctx, "key"); v != "tok" {
t.Errorf("token = %q", v)
}
}
func TestCircuit(t *testing.T) {
requireRedis(t)
ctx := context.Background()
if n, _ := testRedis.GetCircuitFailures(ctx, "cr"); n != 0 {
t.Errorf("initial failures = %d", n)
}
n1, err := testRedis.IncrCircuitFailure(ctx, "cr", time.Minute)
if err != nil || n1 != 1 {
t.Fatalf("first incr = %d %v", n1, err)
}
n2, _ := testRedis.IncrCircuitFailure(ctx, "cr", time.Minute)
if n2 != 2 {
t.Errorf("second incr = %d", n2)
}
if n, _ := testRedis.GetCircuitFailures(ctx, "cr"); n != 2 {
t.Errorf("get failures = %d", n)
}
testRedis.ResetCircuit(ctx, "cr")
if n, _ := testRedis.GetCircuitFailures(ctx, "cr"); n != 0 {
t.Errorf("failures after reset = %d", n)
}
}
func TestFlushRemote(t *testing.T) {
requireRedis(t)
ctx := context.Background()
testRedis.SetTTL(ctx, "flushme", "a", time.Hour)
testRedis.SetETag(ctx, "flushme", "a", "x", time.Hour)
if err := testRedis.FlushRemote(ctx, "flushme"); err != nil {
t.Fatal(err)
}
if fresh, _ := testRedis.CheckTTL(ctx, "flushme", "a"); fresh {
t.Error("expected keys flushed")
}
}
+12
View File
@@ -70,6 +70,18 @@ func (r *Redis) GetETag(ctx context.Context, remote, path string) (string, error
return val, err return val, err
} }
func (r *Redis) GetToken(ctx context.Context, key string) (string, error) {
val, err := r.client.Get(ctx, "token:"+key).Result()
if err == redis.Nil {
return "", nil
}
return val, err
}
func (r *Redis) SetToken(ctx context.Context, key, token string, ttl time.Duration) error {
return r.client.Set(ctx, "token:"+key, token, ttl).Err()
}
func (r *Redis) IncrCircuitFailure(ctx context.Context, remote string, cooldown time.Duration) (int64, error) { func (r *Redis) IncrCircuitFailure(ctx context.Context, remote string, cooldown time.Duration) (int64, error) {
key := fmt.Sprintf("circuit:%s", remote) key := fmt.Sprintf("circuit:%s", remote)
pipe := r.client.Pipeline() pipe := r.client.Pipeline()
+1 -1
View File
@@ -65,7 +65,7 @@ func Load() (*Config, error) {
} }
func getenv(key, fallback string) string { func getenv(key, fallback string) string {
if v := os.Getenv(key); v != "" { if v, ok := os.LookupEnv(key); ok {
return v return v
} }
return fallback return fallback
+66
View File
@@ -0,0 +1,66 @@
package config
import (
"os"
"testing"
)
func TestLoadDefaults(t *testing.T) {
// Unset the vars Load reads so the fallback defaults are exercised.
for _, k := range []string{
"LISTEN_ADDR", "DBHOST", "DBPORT", "DBUSER", "DBPASS", "DBNAME", "DBSSL",
"REDIS_URL", "MINIO_ENDPOINT", "MINIO_ACCESS_KEY", "MINIO_SECRET_KEY",
"MINIO_BUCKET", "MINIO_SECURE", "MINIO_REGION",
} {
old, ok := os.LookupEnv(k)
os.Unsetenv(k)
if ok {
t.Cleanup(func() { os.Setenv(k, old) })
}
}
cfg, err := Load()
if err != nil {
t.Fatalf("load: %v", err)
}
if cfg.ListenAddr != ":8000" || cfg.DBPort != 5432 || cfg.DBUser != "artifacts" {
t.Errorf("unexpected defaults: %+v", cfg)
}
if cfg.RedisURL != "redis://localhost:6379" || cfg.S3Bucket != "artifacts" || cfg.S3Secure {
t.Errorf("unexpected defaults: %+v", cfg)
}
}
func TestLoadOverrides(t *testing.T) {
t.Setenv("LISTEN_ADDR", ":9999")
t.Setenv("DBHOST", "db.example.com")
t.Setenv("DBPORT", "6000")
t.Setenv("DBUSER", "u")
t.Setenv("DBPASS", "pw")
t.Setenv("DBNAME", "n")
t.Setenv("DBSSL", "require")
t.Setenv("MINIO_SECURE", "true")
t.Setenv("MINIO_REGION", "us-east-1")
cfg, err := Load()
if err != nil {
t.Fatalf("load: %v", err)
}
if cfg.ListenAddr != ":9999" || cfg.DBHost != "db.example.com" || cfg.DBPort != 6000 {
t.Errorf("overrides not applied: %+v", cfg)
}
if !cfg.S3Secure {
t.Error("MINIO_SECURE=true not parsed")
}
want := "postgres://u:pw@db.example.com:6000/n?sslmode=require"
if got := cfg.DatabaseDSN(); got != want {
t.Errorf("DSN = %q, want %q", got, want)
}
}
func TestLoadInvalidPort(t *testing.T) {
t.Setenv("DBPORT", "not-a-number")
if _, err := Load(); err == nil {
t.Error("expected error for invalid DBPORT")
}
}
+38 -3
View File
@@ -4,6 +4,8 @@ import (
"context" "context"
"time" "time"
"github.com/jackc/pgx/v5"
"git.unkin.net/unkin/artifactapi/pkg/models" "git.unkin.net/unkin/artifactapi/pkg/models"
) )
@@ -109,16 +111,49 @@ func (db *DB) InsertAccessLog(ctx context.Context, remoteName, path string, cach
return err return err
} }
func (db *DB) FindOrphanedBlobs(ctx context.Context) ([]models.Blob, error) { // AccessLogEntry is one buffered access-log record.
type AccessLogEntry struct {
RemoteName string
Path string
CacheHit bool
SizeBytes int64
UpstreamMS int
ClientIP string
}
// InsertAccessLogBatch bulk-inserts access-log rows with a single COPY.
func (db *DB) InsertAccessLogBatch(ctx context.Context, entries []AccessLogEntry) error {
if len(entries) == 0 {
return nil
}
rows := make([][]any, len(entries))
for i, e := range entries {
rows[i] = []any{e.RemoteName, e.Path, e.CacheHit, e.SizeBytes, e.UpstreamMS, e.ClientIP}
}
_, err := db.Pool.CopyFrom(ctx,
pgx.Identifier{"access_log"},
[]string{"remote_name", "path", "cache_hit", "size_bytes", "upstream_ms", "client_ip"},
pgx.CopyFromRows(rows),
)
return err
}
// FindOrphanedBlobs returns blobs no longer referenced by any artifact or
// local file, restricted to those created before now()-minAge. The age cutoff
// is a grace period that avoids a TOCTOU race with in-flight dedup uploads,
// which insert the blob row before the referencing artifact/local_files row.
func (db *DB) FindOrphanedBlobs(ctx context.Context, minAge time.Duration) ([]models.Blob, error) {
cutoff := time.Now().Add(-minAge)
rows, err := db.Pool.Query(ctx, ` rows, err := db.Pool.Query(ctx, `
SELECT b.content_hash, b.s3_key, b.size_bytes, b.content_type, b.created_at SELECT b.content_hash, b.s3_key, b.size_bytes, b.content_type, b.created_at
FROM blobs b FROM blobs b
WHERE b.content_hash NOT IN ( WHERE b.created_at < $1
AND b.content_hash NOT IN (
SELECT content_hash FROM artifacts SELECT content_hash FROM artifacts
UNION UNION
SELECT content_hash FROM local_files SELECT content_hash FROM local_files
) )
`) `, cutoff)
if err != nil { if err != nil {
return nil, err return nil, err
} }
+334
View File
@@ -0,0 +1,334 @@
package database
import (
"context"
"os"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
var (
testDB *DB
testDSN string
)
func TestMain(m *testing.M) {
c := context.Background()
dsn, terminate, err := testsupport.StartPostgres(c)
if err != nil {
// Docker unavailable: run anyway so tests self-skip via requireDB.
os.Exit(m.Run())
}
testDSN = dsn
db, err := New(dsn)
if err != nil {
terminate()
panic(err)
}
testDB = db
code := m.Run()
db.Close()
terminate()
// Return normally on success so the coverage profile is flushed; os.Exit
// would truncate it.
if code != 0 {
os.Exit(code)
}
}
func requireDB(t *testing.T) {
t.Helper()
if testDB == nil {
t.Skip("Docker unavailable; skipping database integration test")
}
}
func ctx() context.Context { return context.Background() }
func seedRemote(t *testing.T, name string) {
t.Helper()
if err := testDB.CreateRemote(ctx(), &models.Remote{
Name: name, PackageType: models.PackageGeneric, RepoType: models.RepoTypeRemote,
BaseURL: "https://example.com", MutableTTL: 3600,
}); err != nil {
t.Fatalf("seed remote: %v", err)
}
}
// seedBlob inserts a blob and returns its full content hash (sha256:<hash>),
// matching the reference convention used by artifacts and local files.
func seedBlob(t *testing.T, hash string) string {
t.Helper()
full := "sha256:" + hash
if err := testDB.UpsertBlob(ctx(), full, "blobs/sha256/"+hash, 10, "application/octet-stream"); err != nil {
t.Fatalf("seed blob: %v", err)
}
return full
}
func TestRemotesCRUD(t *testing.T) {
requireDB(t)
seedRemote(t, "r-crud")
got, err := testDB.GetRemote(ctx(), "r-crud")
if err != nil || got.BaseURL != "https://example.com" {
t.Fatalf("get: %v %v", got, err)
}
got.BaseURL = "https://updated.example.com"
if err := testDB.UpdateRemote(ctx(), got); err != nil {
t.Fatalf("update: %v", err)
}
got, _ = testDB.GetRemote(ctx(), "r-crud")
if got.BaseURL != "https://updated.example.com" {
t.Errorf("update not applied: %v", got.BaseURL)
}
list, err := testDB.ListRemotes(ctx())
if err != nil || len(list) == 0 {
t.Fatalf("list: %v %v", len(list), err)
}
if err := testDB.DeleteRemote(ctx(), "r-crud"); err != nil {
t.Fatalf("delete: %v", err)
}
if _, err := testDB.GetRemote(ctx(), "r-crud"); err == nil {
t.Error("expected error after delete")
}
}
func TestArtifactsAndBlobs(t *testing.T) {
requireDB(t)
seedRemote(t, "r-art")
seedBlob(t, "aaaa")
hash := "sha256:aaaa"
if err := testDB.UpsertBlob(ctx(), hash, "blobs/sha256/aaaa", 10, "text/plain"); err != nil {
t.Fatal(err)
}
if err := testDB.UpsertArtifact(ctx(), "r-art", "path/a.txt", hash, "etag1"); err != nil {
t.Fatal(err)
}
// Upsert again to exercise the ON CONFLICT update branch.
if err := testDB.UpsertArtifact(ctx(), "r-art", "path/a.txt", hash, "etag2"); err != nil {
t.Fatal(err)
}
art, err := testDB.GetArtifact(ctx(), "r-art", "path/a.txt")
if err != nil || art.ContentHash != hash {
t.Fatalf("get artifact: %v %v", art, err)
}
if err := testDB.TouchArtifactAccess(ctx(), "r-art", "path/a.txt"); err != nil {
t.Fatal(err)
}
arts, err := testDB.ListArtifacts(ctx(), "r-art", 10, 0)
if err != nil || len(arts) != 1 {
t.Fatalf("list artifacts: %v %v", len(arts), err)
}
if err := testDB.InsertAccessLog(ctx(), "r-art", "path/a.txt", true, 10, 5, "1.2.3.4"); err != nil {
t.Fatal(err)
}
if err := testDB.InsertAccessLogBatch(ctx(), []AccessLogEntry{
{RemoteName: "r-art", Path: "b", CacheHit: false, SizeBytes: 20, UpstreamMS: 3},
}); err != nil {
t.Fatal(err)
}
if err := testDB.InsertAccessLogBatch(ctx(), nil); err != nil {
t.Fatalf("empty batch should be a no-op: %v", err)
}
if err := testDB.DeleteArtifact(ctx(), "r-art", "path/a.txt"); err != nil {
t.Fatal(err)
}
}
func TestOrphanAndColdCleanup(t *testing.T) {
requireDB(t)
seedBlob(t, "orphanhash")
// A blob with no artifact/local_file reference is orphaned, but only past
// the grace period.
if got, _ := testDB.FindOrphanedBlobs(ctx(), time.Hour); containsHash(got, "sha256:orphanhash") {
t.Error("fresh orphan should be excluded by grace period")
}
orphans, err := testDB.FindOrphanedBlobs(ctx(), -time.Hour) // cutoff in the future => include fresh
if err != nil {
t.Fatal(err)
}
if !containsHash(orphans, "sha256:orphanhash") {
t.Error("expected orphan to be found with zero grace")
}
if err := testDB.DeleteBlob(ctx(), "sha256:orphanhash"); err != nil {
t.Fatal(err)
}
seedRemote(t, "r-cold")
seedBlob(t, "coldhash")
testDB.UpsertArtifact(ctx(), "r-cold", "cold.txt", "sha256:coldhash", "")
n, err := testDB.DeleteColdArtifacts(ctx(), "r-cold", -time.Hour) // negative => everything is "cold"
if err != nil || n < 1 {
t.Fatalf("delete cold: n=%d err=%v", n, err)
}
}
func containsHash(blobs []models.Blob, hash string) bool {
for _, b := range blobs {
if b.ContentHash == hash {
return true
}
}
return false
}
func TestLocalFiles(t *testing.T) {
requireDB(t)
seedRemote(t, "r-local")
seedBlob(t, "localhash")
hash := "sha256:localhash"
if err := testDB.CreateLocalFile(ctx(), "r-local", "foo/foo-1.0.whl", hash); err != nil {
t.Fatal(err)
}
// Duplicate create must be rejected.
if err := testDB.CreateLocalFile(ctx(), "r-local", "foo/foo-1.0.whl", hash); err == nil {
t.Error("expected duplicate local file error")
}
f, err := testDB.GetLocalFile(ctx(), "r-local", "foo/foo-1.0.whl")
if err != nil || f == nil {
t.Fatalf("get local file: %v %v", f, err)
}
if files, err := testDB.ListLocalFiles(ctx(), "r-local", 10, 0); err != nil || len(files) != 1 {
t.Fatalf("list: %v %v", len(files), err)
}
if files, err := testDB.ListLocalFilesByPrefix(ctx(), "r-local", "foo/"); err != nil || len(files) != 1 {
t.Fatalf("list by prefix: %v %v", len(files), err)
}
if entries, err := testDB.ListFilesByPrefix(ctx(), "r-local", "foo/"); err != nil || len(entries) != 1 {
t.Fatalf("provider list by prefix: %v %v", len(entries), err)
}
if pkgs, err := testDB.ListLocalFilePackages(ctx(), "r-local"); err != nil || len(pkgs) == 0 {
t.Fatalf("list packages: %v %v", pkgs, err)
}
if pkgs, err := testDB.ListPackages(ctx(), "r-local"); err != nil || len(pkgs) == 0 {
t.Fatalf("provider list packages: %v %v", pkgs, err)
}
if err := testDB.DeleteLocalFile(ctx(), "r-local", "foo/foo-1.0.whl"); err != nil {
t.Fatal(err)
}
}
func TestVirtualsCRUD(t *testing.T) {
requireDB(t)
if err := testDB.CreateVirtual(ctx(), &models.Virtual{
Name: "v-crud", PackageType: models.PackageHelm, Members: []string{"a", "b"},
}); err != nil {
t.Fatal(err)
}
v, err := testDB.GetVirtual(ctx(), "v-crud")
if err != nil || len(v.Members) != 2 {
t.Fatalf("get virtual: %v %v", v, err)
}
v.Members = []string{"a"}
if err := testDB.UpdateVirtual(ctx(), v); err != nil {
t.Fatal(err)
}
if vs, err := testDB.ListVirtuals(ctx()); err != nil || len(vs) == 0 {
t.Fatalf("list virtuals: %v %v", len(vs), err)
}
if err := testDB.DeleteVirtual(ctx(), "v-crud"); err != nil {
t.Fatal(err)
}
}
func TestStats(t *testing.T) {
requireDB(t)
seedRemote(t, "r-stats")
seedBlob(t, "statshash")
testDB.UpsertArtifact(ctx(), "r-stats", "s.txt", "sha256:statshash", "")
testDB.InsertAccessLog(ctx(), "r-stats", "s.txt", true, 100, 2, "")
if _, err := testDB.GetOverviewStats(ctx()); err != nil {
t.Fatalf("overview: %v", err)
}
if _, err := testDB.GetTopRemotes(ctx(), 5); err != nil {
t.Fatalf("top remotes: %v", err)
}
if _, err := testDB.GetTopFilesByHits(ctx(), 5); err != nil {
t.Fatalf("top files by hits: %v", err)
}
if _, err := testDB.GetTopFilesByBandwidth(ctx(), 5); err != nil {
t.Fatalf("top files by bandwidth: %v", err)
}
}
func TestDatabaseErrorPaths(t *testing.T) {
requireDB(t)
bad, err := New(testDSN)
if err != nil {
t.Fatal(err)
}
bad.Close() // every query now fails
ctx := context.Background()
if _, err := bad.ListRemotes(ctx); err == nil {
t.Error("ListRemotes should error on closed db")
}
if _, err := bad.ListVirtuals(ctx); err == nil {
t.Error("ListVirtuals should error")
}
if _, err := bad.ListArtifacts(ctx, "r", 10, 0); err == nil {
t.Error("ListArtifacts should error")
}
if _, err := bad.ListLocalFiles(ctx, "r", 10, 0); err == nil {
t.Error("ListLocalFiles should error")
}
if _, err := bad.ListLocalFilesByPrefix(ctx, "r", "p"); err == nil {
t.Error("ListLocalFilesByPrefix should error")
}
if _, err := bad.ListLocalFilePackages(ctx, "r"); err == nil {
t.Error("ListLocalFilePackages should error")
}
if _, err := bad.ListFilesByPrefix(ctx, "r", "p"); err == nil {
t.Error("ListFilesByPrefix should error")
}
if _, err := bad.ListPackages(ctx, "r"); err == nil {
t.Error("ListPackages should error")
}
if _, err := bad.FindOrphanedBlobs(ctx, 0); err == nil {
t.Error("FindOrphanedBlobs should error")
}
if _, err := bad.GetOverviewStats(ctx); err == nil {
t.Error("GetOverviewStats should error")
}
if _, err := bad.GetTopRemotes(ctx, 5); err == nil {
t.Error("GetTopRemotes should error")
}
if _, err := bad.GetTopFilesByHits(ctx, 5); err == nil {
t.Error("GetTopFilesByHits should error")
}
if _, err := bad.GetTopFilesByBandwidth(ctx, 5); err == nil {
t.Error("GetTopFilesByBandwidth should error")
}
if _, err := bad.ListRPMMetadataEntries(ctx, "r"); err == nil {
t.Error("ListRPMMetadataEntries should error")
}
}
func TestRPMMetadata(t *testing.T) {
requireDB(t)
seedRemote(t, "r-rpm")
meta := &provider.RPMMetadata{
RepoName: "r-rpm", FilePath: "Packages/x.rpm", ContentHash: "sha256:rpm",
Name: "x", Version: "1.0", Release: "1", Arch: "noarch",
Requires: []provider.RPMDep{{Name: "libc"}},
Provides: []provider.RPMDep{{Name: "x"}},
Files: []provider.RPMFile{},
}
if err := testDB.InsertRPMMetadata(ctx(), meta); err != nil {
t.Fatal(err)
}
entries, err := testDB.ListRPMMetadataEntries(ctx(), "r-rpm")
if err != nil || len(entries) != 1 {
t.Fatalf("list rpm entries: %v %v", len(entries), err)
}
if rows, err := testDB.ListRPMMetadata(ctx(), "r-rpm"); err != nil || len(rows) != 1 {
t.Fatalf("list rpm rows: %v %v", len(rows), err)
}
}
+35
View File
@@ -10,6 +10,7 @@ import (
"github.com/jackc/pgx/v5/pgconn" "github.com/jackc/pgx/v5/pgconn"
"git.unkin.net/unkin/artifactapi/internal/provider" "git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
) )
type LocalFile struct { type LocalFile struct {
@@ -78,6 +79,40 @@ func (db *DB) ListLocalFiles(ctx context.Context, repoName string, limit, offset
return files, rows.Err() return files, rows.Err()
} }
// ListLocalArtifacts returns a repo's local files shaped as models.Artifact so
// the UI's cached-objects view can render them the same way as remote artifacts.
// Local files carry no access/fetch counters, so those are left at zero and the
// timestamps are all derived from created_at.
func (db *DB) ListLocalArtifacts(ctx context.Context, repoName string, limit, offset int) ([]models.Artifact, error) {
rows, err := db.Pool.Query(ctx, `
SELECT lf.id, lf.repo_name, lf.file_path, lf.content_hash,
lf.created_at, b.size_bytes, b.content_type
FROM local_files lf
JOIN blobs b ON lf.content_hash = b.content_hash
WHERE lf.repo_name = $1
ORDER BY lf.file_path
LIMIT $2 OFFSET $3
`, repoName, limit, offset)
if err != nil {
return nil, err
}
defer rows.Close()
var artifacts []models.Artifact
for rows.Next() {
var a models.Artifact
var createdAt time.Time
if err := rows.Scan(&a.ID, &a.RemoteName, &a.Path, &a.ContentHash, &createdAt, &a.SizeBytes, &a.ContentType); err != nil {
return nil, err
}
a.FirstSeenAt = createdAt
a.LastFetchedAt = createdAt
a.LastAccessedAt = createdAt
artifacts = append(artifacts, a)
}
return artifacts, rows.Err()
}
func (db *DB) ListLocalFilesByPrefix(ctx context.Context, repoName, prefix string) ([]LocalFile, error) { func (db *DB) ListLocalFilesByPrefix(ctx context.Context, repoName, prefix string) ([]LocalFile, error) {
rows, err := db.Pool.Query(ctx, ` rows, err := db.Pool.Query(ctx, `
SELECT id, repo_name, file_path, content_hash, created_at SELECT id, repo_name, file_path, content_hash, created_at
+3
View File
@@ -124,6 +124,9 @@ func (db *DB) migrate() error {
CREATE INDEX IF NOT EXISTS idx_access_log_remote_time ON access_log(remote_name, created_at); CREATE INDEX IF NOT EXISTS idx_access_log_remote_time ON access_log(remote_name, created_at);
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS repo_type TEXT DEFAULT 'remote'; ALTER TABLE remotes ADD COLUMN IF NOT EXISTS repo_type TEXT DEFAULT 'remote';
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_dial_timeout INTEGER DEFAULT 0;
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_tls_timeout INTEGER DEFAULT 0;
ALTER TABLE remotes ADD COLUMN IF NOT EXISTS upstream_response_header_timeout INTEGER DEFAULT 0;
CREATE TABLE IF NOT EXISTS rpm_metadata ( CREATE TABLE IF NOT EXISTS rpm_metadata (
id BIGSERIAL PRIMARY KEY, id BIGSERIAL PRIMARY KEY,
+14 -5
View File
@@ -11,7 +11,9 @@ const remoteCols = `name, package_type, repo_type, base_url, description, userna
patterns, blocklist, mutable_patterns, immutable_patterns, patterns, blocklist, mutable_patterns, immutable_patterns,
ban_tags_enabled, ban_tags, ban_tags_enabled, ban_tags,
quarantine_enabled, quarantine_days, stale_on_error, quarantine_enabled, quarantine_days, stale_on_error,
releases_remote, managed_by, created_at, updated_at` releases_remote, managed_by,
upstream_dial_timeout, upstream_tls_timeout, upstream_response_header_timeout,
created_at, updated_at`
func scanRemote(scanner interface{ Scan(...any) error }, r *models.Remote) error { func scanRemote(scanner interface{ Scan(...any) error }, r *models.Remote) error {
return scanner.Scan( return scanner.Scan(
@@ -20,7 +22,9 @@ func scanRemote(scanner interface{ Scan(...any) error }, r *models.Remote) error
&r.Patterns, &r.Blocklist, &r.MutablePatterns, &r.ImmutablePatterns, &r.Patterns, &r.Blocklist, &r.MutablePatterns, &r.ImmutablePatterns,
&r.BanTagsEnabled, &r.BanTags, &r.BanTagsEnabled, &r.BanTags,
&r.QuarantineEnabled, &r.QuarantineDays, &r.StaleOnError, &r.QuarantineEnabled, &r.QuarantineDays, &r.StaleOnError,
&r.ReleasesRemote, &r.ManagedBy, &r.CreatedAt, &r.UpdatedAt, &r.ReleasesRemote, &r.ManagedBy,
&r.UpstreamDialTimeout, &r.UpstreamTLSTimeout, &r.UpstreamResponseHeaderTimeout,
&r.CreatedAt, &r.UpdatedAt,
) )
} }
@@ -59,8 +63,9 @@ func (db *DB) CreateRemote(ctx context.Context, r *models.Remote) error {
patterns, blocklist, mutable_patterns, immutable_patterns, patterns, blocklist, mutable_patterns, immutable_patterns,
ban_tags_enabled, ban_tags, ban_tags_enabled, ban_tags,
quarantine_enabled, quarantine_days, stale_on_error, quarantine_enabled, quarantine_days, stale_on_error,
releases_remote, managed_by releases_remote, managed_by,
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21) upstream_dial_timeout, upstream_tls_timeout, upstream_response_header_timeout
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24)
`, `,
r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Description, r.Username, r.Password, r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Description, r.Username, r.Password,
r.ImmutableTTL, r.MutableTTL, r.CheckMutable, r.ImmutableTTL, r.MutableTTL, r.CheckMutable,
@@ -68,6 +73,7 @@ func (db *DB) CreateRemote(ctx context.Context, r *models.Remote) error {
r.BanTagsEnabled, r.BanTags, r.BanTagsEnabled, r.BanTags,
r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError, r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError,
r.ReleasesRemote, r.ManagedBy, r.ReleasesRemote, r.ManagedBy,
r.UpstreamDialTimeout, r.UpstreamTLSTimeout, r.UpstreamResponseHeaderTimeout,
) )
return err return err
} }
@@ -80,7 +86,9 @@ func (db *DB) UpdateRemote(ctx context.Context, r *models.Remote) error {
patterns=$11, blocklist=$12, mutable_patterns=$13, immutable_patterns=$14, patterns=$11, blocklist=$12, mutable_patterns=$13, immutable_patterns=$14,
ban_tags_enabled=$15, ban_tags=$16, ban_tags_enabled=$15, ban_tags=$16,
quarantine_enabled=$17, quarantine_days=$18, stale_on_error=$19, quarantine_enabled=$17, quarantine_days=$18, stale_on_error=$19,
releases_remote=$20, managed_by=$21, updated_at=NOW() releases_remote=$20, managed_by=$21,
upstream_dial_timeout=$22, upstream_tls_timeout=$23, upstream_response_header_timeout=$24,
updated_at=NOW()
WHERE name=$1 WHERE name=$1
`, `,
r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Description, r.Username, r.Password, r.Name, r.PackageType, r.RepoType, r.BaseURL, r.Description, r.Username, r.Password,
@@ -89,6 +97,7 @@ func (db *DB) UpdateRemote(ctx context.Context, r *models.Remote) error {
r.BanTagsEnabled, r.BanTags, r.BanTagsEnabled, r.BanTags,
r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError, r.QuarantineEnabled, r.QuarantineDays, r.StaleOnError,
r.ReleasesRemote, r.ManagedBy, r.ReleasesRemote, r.ManagedBy,
r.UpstreamDialTimeout, r.UpstreamTLSTimeout, r.UpstreamResponseHeaderTimeout,
) )
return err return err
} }
+27 -22
View File
@@ -32,30 +32,35 @@ func (db *DB) InsertRPMMetadata(ctx context.Context, meta *provider.RPMMetadata)
return err return err
} }
func (db *DB) DeleteRPMMetadata(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM rpm_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
type RPMMetadataRow struct { type RPMMetadataRow struct {
RepoName string RepoName string
FilePath string FilePath string
ContentHash string ContentHash string
Name string Name string
Epoch int Epoch int
Version string Version string
Release string Release string
Arch string Arch string
Summary string Summary string
Description string Description string
RPMSize int64 RPMSize int64
InstalledSize int64 InstalledSize int64
License string License string
Vendor string Vendor string
Group string Group string
BuildHost string BuildHost string
SourceRPM string SourceRPM string
URL string URL string
Packager string Packager string
Requires json.RawMessage Requires json.RawMessage
Provides json.RawMessage Provides json.RawMessage
Files json.RawMessage Files json.RawMessage
Changelogs json.RawMessage Changelogs json.RawMessage
} }
func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]provider.RPMMetadata, error) { func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]provider.RPMMetadata, error) {
+9
View File
@@ -30,6 +30,15 @@ func (db *DB) GetOverviewStats(ctx context.Context) (*models.OverviewStats, erro
return nil, err return nil, err
} }
err = db.Pool.QueryRow(ctx, `
SELECT COALESCE(SUM(size_bytes), 0)
FROM access_log
WHERE cache_hit = TRUE AND created_at > NOW() - INTERVAL '30 days'
`).Scan(&stats.BandwidthSaved30d)
if err != nil {
return nil, err
}
return &stats, nil return &stats, nil
} }
+6 -1
View File
@@ -9,6 +9,11 @@ import (
"git.unkin.net/unkin/artifactapi/internal/storage" "git.unkin.net/unkin/artifactapi/internal/storage"
) )
// blobGracePeriod is how old an orphaned blob must be before GC will delete
// it. This avoids racing in-flight dedup uploads that insert the blob row
// before the referencing artifact/local_files row exists.
const blobGracePeriod = 1 * time.Hour
type Collector struct { type Collector struct {
db *database.DB db *database.DB
store *storage.S3 store *storage.S3
@@ -38,7 +43,7 @@ func (c *Collector) Run(ctx context.Context) {
func (c *Collector) sweep(ctx context.Context) { func (c *Collector) sweep(ctx context.Context) {
start := time.Now() start := time.Now()
orphaned, err := c.db.FindOrphanedBlobs(ctx) orphaned, err := c.db.FindOrphanedBlobs(ctx, blobGracePeriod)
if err != nil { if err != nil {
slog.Error("gc: find orphaned blobs", "error", err) slog.Error("gc: find orphaned blobs", "error", err)
return return
+114
View File
@@ -0,0 +1,114 @@
package gc
import (
"bytes"
"context"
"os"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
var (
testDB *database.DB
testStore *storage.S3
)
func TestMain(m *testing.M) {
ctx := context.Background()
dsn, termPG, err := testsupport.StartPostgres(ctx)
if err != nil {
os.Exit(m.Run())
}
minio, termMinio, err := testsupport.StartMinio(ctx)
if err != nil {
termPG()
os.Exit(m.Run())
}
db, err := database.New(dsn)
if err != nil {
panic(err)
}
var s3 *storage.S3
for i := 0; i < 20; i++ {
if s3, err = storage.NewS3(minio.Endpoint, minio.AccessKey, minio.SecretKey, "gc-test", false, ""); err == nil {
break
}
time.Sleep(500 * time.Millisecond)
}
if err != nil {
panic(err)
}
testDB = db
testStore = s3
code := m.Run()
db.Close()
termMinio()
termPG()
if code != 0 {
os.Exit(code)
}
}
func TestSweepDeletesOldOrphan(t *testing.T) {
if testDB == nil {
t.Skip("Docker unavailable")
}
ctx := context.Background()
hash := "sha256:gcorphan"
key := storage.BlobKey("gcorphan")
if err := testStore.Upload(ctx, key, bytes.NewReader([]byte("orphan")), 6, "application/octet-stream"); err != nil {
t.Fatal(err)
}
if err := testDB.UpsertBlob(ctx, hash, key, 6, "application/octet-stream"); err != nil {
t.Fatal(err)
}
// Age the blob past the grace period.
if _, err := testDB.Pool.Exec(ctx, `UPDATE blobs SET created_at = now() - interval '2 hours' WHERE content_hash = $1`, hash); err != nil {
t.Fatal(err)
}
c := New(testDB, testStore, time.Hour)
c.sweep(ctx)
if exists, _ := testStore.Exists(ctx, key); exists {
t.Error("expected orphan object deleted from store")
}
orphans, _ := testDB.FindOrphanedBlobs(ctx, 0)
for _, b := range orphans {
if b.ContentHash == hash {
t.Error("expected orphan blob row deleted")
}
}
}
func TestSweepNoOrphans(t *testing.T) {
if testDB == nil {
t.Skip("Docker unavailable")
}
// A sweep with nothing to collect should be a clean no-op.
New(testDB, testStore, time.Hour).sweep(context.Background())
}
func TestRunStopsOnContextCancel(t *testing.T) {
if testDB == nil {
t.Skip("Docker unavailable")
}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
New(testDB, testStore, time.Hour).Run(ctx)
close(done)
}()
cancel()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("Run did not return after context cancel")
}
}
+60
View File
@@ -0,0 +1,60 @@
package alpine
import (
"context"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestType(t *testing.T) {
if (&Provider{}).Type() != models.PackageAlpine {
t.Fatal("wrong type")
}
}
func TestClassify(t *testing.T) {
p := &Provider{}
if p.Classify("v3.19/main/x86_64/APKINDEX.tar.gz") != provider.Mutable {
t.Error("APKINDEX should be mutable")
}
if p.Classify("v3.19/main/x86_64/curl-8.0-r0.apk") != provider.Immutable {
t.Error("apk should be immutable")
}
}
func TestContentType(t *testing.T) {
p := &Provider{}
cases := map[string]string{
"pkg.apk": "application/vnd.android.package-archive",
"APKINDEX.tar.gz": "application/gzip",
"something.random": "application/octet-stream",
}
for path, want := range cases {
if got := p.ContentType(path); got != want {
t.Errorf("ContentType(%q) = %q, want %q", path, got, want)
}
}
}
func TestUpstreamURL(t *testing.T) {
p := &Provider{}
got := p.UpstreamURL(models.Remote{BaseURL: "https://dl-cdn.alpinelinux.org/alpine/"}, "/v3.19/main/x86_64/curl.apk")
if got != "https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/curl.apk" {
t.Errorf("got %q", got)
}
}
func TestRewriteResponse(t *testing.T) {
if out, err := (&Provider{}).RewriteResponse([]byte("x"), models.Remote{}, "http://proxy"); out != nil || err != nil {
t.Error("alpine never rewrites")
}
}
func TestAuthHeaders(t *testing.T) {
h, _ := (&Provider{}).AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
if h.Get("Authorization") == "" {
t.Error("expected auth header")
}
}
@@ -0,0 +1,53 @@
package docker
import (
"context"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestDockerClassifyBranches(t *testing.T) {
p := &Provider{}
if p.Classify("library/nginx/tags/list") != provider.Mutable {
t.Error("tags/list should be mutable")
}
if p.Classify("library/nginx/manifests/latest") != provider.Mutable {
t.Error("tag manifest should be mutable")
}
if p.Classify("library/nginx/manifests/sha256:abcdef") != provider.Immutable {
t.Error("digest manifest should be immutable")
}
if p.Classify("library/nginx/blobs/sha256:abc") != provider.Immutable {
t.Error("blob should be immutable")
}
}
func TestDockerContentType(t *testing.T) {
p := &Provider{}
if p.ContentType("x/blobs/sha256:abc") != "application/octet-stream" {
t.Error("blob content type")
}
if p.ContentType("x/manifests/latest") != "application/vnd.docker.distribution.manifest.v2+json" {
t.Error("manifest content type")
}
if p.ContentType("x/tags/list") != "application/json" {
t.Error("default content type")
}
}
func TestDockerRewriteAndAuth(t *testing.T) {
p := &Provider{}
if out, err := p.RewriteResponse([]byte("x"), models.Remote{}, "http://p"); out != nil || err != nil {
t.Error("docker never rewrites")
}
h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
if h.Get("Authorization") == "" {
t.Error("expected basic auth header")
}
h, _ = p.AuthHeaders(context.Background(), models.Remote{})
if h.Get("Authorization") != "" {
t.Error("no creds, no header")
}
}
@@ -0,0 +1,13 @@
package generic
import (
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestGenericRewriteResponse(t *testing.T) {
if out, err := (&Provider{}).RewriteResponse([]byte("x"), models.Remote{}, "http://p"); out != nil || err != nil {
t.Error("generic never rewrites")
}
}
@@ -0,0 +1,27 @@
package goproxy
import (
"context"
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestGoProxyURLAuthRewrite(t *testing.T) {
p := &Provider{}
if got := p.UpstreamURL(models.Remote{BaseURL: "https://proxy.golang.org/"}, "/mod/@v/list"); got != "https://proxy.golang.org/mod/@v/list" {
t.Errorf("upstream url %q", got)
}
if out, err := p.RewriteResponse([]byte("x"), models.Remote{}, "http://p"); out != nil || err != nil {
t.Error("goproxy never rewrites")
}
if h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"}); h.Get("Authorization") == "" {
t.Error("expected basic auth header")
}
if got := p.ContentType("mod/@v/v1.0.0.info"); got != "application/json" {
t.Errorf("info content type %q", got)
}
if got := p.ContentType("mod/@v/v1.0.0.mod"); got != "text/plain" {
t.Errorf("mod content type %q", got)
}
}
+18
View File
@@ -0,0 +1,18 @@
package helm
import "testing"
func TestHelmContentTypeBranches(t *testing.T) {
p := &Provider{}
for path, want := range map[string]string{
"charts/x-1.0.0.tgz": "application/gzip",
"x.tar.gz": "application/gzip",
"index.yaml": "text/yaml",
"x.yml": "text/yaml",
"other": "application/octet-stream",
} {
if got := p.ContentType(path); got != want {
t.Errorf("ContentType(%q)=%q want %q", path, got, want)
}
}
}
+78
View File
@@ -0,0 +1,78 @@
package npm
import (
"context"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestType(t *testing.T) {
if (&Provider{}).Type() != models.PackageNPM {
t.Fatal("wrong type")
}
}
func TestClassify(t *testing.T) {
p := &Provider{}
if p.Classify("pkg/-/pkg-1.0.0.tgz") != provider.Immutable {
t.Error("tgz should be immutable")
}
if p.Classify("pkg") != provider.Mutable {
t.Error("metadata should be mutable")
}
}
func TestContentType(t *testing.T) {
p := &Provider{}
if p.ContentType("pkg/-/pkg-1.0.0.tgz") != "application/gzip" {
t.Error("tgz content type")
}
if p.ContentType("pkg") != "application/json" {
t.Error("metadata content type")
}
}
func TestUpstreamURL(t *testing.T) {
p := &Provider{}
got := p.UpstreamURL(models.Remote{BaseURL: "https://registry.npmjs.org/"}, "/pkg")
if got != "https://registry.npmjs.org/pkg" {
t.Errorf("got %q", got)
}
}
func TestRewriteResponse(t *testing.T) {
p := &Provider{}
remote := models.Remote{Name: "npmjs", BaseURL: "https://registry.npmjs.org"}
if out, _ := p.RewriteResponse([]byte(`{"a":1}`), remote, ""); out != nil {
t.Error("empty proxyBaseURL should be a no-op")
}
if out, _ := p.RewriteResponse([]byte("not json"), remote, "http://proxy"); out != nil {
t.Error("invalid json should be a no-op")
}
body := []byte(`{"tarball":"https://registry.npmjs.org/pkg/-/pkg-1.0.0.tgz"}`)
out, err := p.RewriteResponse(body, remote, "http://proxy")
if err != nil {
t.Fatal(err)
}
if string(out) != `{"tarball":"http://proxy/api/v1/remote/npmjs/pkg/-/pkg-1.0.0.tgz"}` {
t.Errorf("rewrite: %s", out)
}
if out, _ := p.RewriteResponse([]byte(`{"x":"unrelated"}`), remote, "http://proxy"); out != nil {
t.Error("no matching base URL should be a no-op")
}
}
func TestAuthHeaders(t *testing.T) {
p := &Provider{}
h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "pw"})
if h.Get("Authorization") == "" {
t.Error("expected auth header when credentials set")
}
h, _ = p.AuthHeaders(context.Background(), models.Remote{})
if h.Get("Authorization") != "" {
t.Error("expected no auth header without credentials")
}
}
+10
View File
@@ -53,10 +53,20 @@ type PostUploadHook interface {
AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs BlobReader, db MetadataStore) AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs BlobReader, db MetadataStore)
} }
// PostDeleteHook lets a provider clean up derived state (e.g. RPM metadata that
// feeds generated repodata) after a local file is removed.
type PostDeleteHook interface {
AfterDelete(ctx context.Context, repoName, storagePath string, db MetadataDeleter) error
}
type MetadataStore interface { type MetadataStore interface {
InsertRPMMetadata(ctx context.Context, meta *RPMMetadata) error InsertRPMMetadata(ctx context.Context, meta *RPMMetadata) error
} }
type MetadataDeleter interface {
DeleteRPMMetadata(ctx context.Context, repoName, filePath string) error
}
type RPMMetadataReader interface { type RPMMetadataReader interface {
ListRPMMetadataEntries(ctx context.Context, repoName string) ([]RPMMetadata, error) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]RPMMetadata, error)
} }
+78
View File
@@ -0,0 +1,78 @@
package puppet
import (
"context"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestType(t *testing.T) {
if (&Provider{}).Type() != models.PackagePuppet {
t.Fatal("wrong type")
}
}
func TestClassify(t *testing.T) {
p := &Provider{}
if p.Classify("v3/modules/puppetlabs-stdlib") != provider.Mutable {
t.Error("modules should be mutable")
}
if p.Classify("v3/releases?module=x") != provider.Mutable {
t.Error("releases should be mutable")
}
if p.Classify("v3/files/puppetlabs-stdlib-1.0.0.tar.gz") != provider.Immutable {
t.Error("files should be immutable")
}
}
func TestContentType(t *testing.T) {
p := &Provider{}
if p.ContentType("x/mod-1.0.0.tar.gz") != "application/gzip" {
t.Error("tar.gz")
}
if p.ContentType("v3/modules/x") != "application/json" {
t.Error("v3 json")
}
if p.ContentType("other") != "application/octet-stream" {
t.Error("default")
}
}
func TestUpstreamURL(t *testing.T) {
got := (&Provider{}).UpstreamURL(models.Remote{BaseURL: "https://forgeapi.puppet.com/"}, "/v3/modules/x")
if got != "https://forgeapi.puppet.com/v3/modules/x" {
t.Errorf("got %q", got)
}
}
func TestRewriteResponse(t *testing.T) {
p := &Provider{}
remote := models.Remote{Name: "forge", BaseURL: "https://forgeapi.puppet.com"}
if out, _ := p.RewriteResponse([]byte("x"), remote, ""); out != nil {
t.Error("empty proxyBaseURL is a no-op")
}
body := []byte(`{"file_uri":"/v3/files/mod.tar.gz","home":"https://forgeapi.puppet.com/x"}`)
out, err := p.RewriteResponse(body, remote, "http://proxy")
if err != nil {
t.Fatal(err)
}
s := string(out)
if !strings.Contains(s, "http://proxy/api/v1/remote/forge/v3/files/mod.tar.gz") {
t.Errorf("v3/files not rewritten: %s", s)
}
if !strings.Contains(s, "http://proxy/api/v1/remote/forge/x") {
t.Errorf("base URL not rewritten: %s", s)
}
}
func TestAuthHeaders(t *testing.T) {
h, _ := (&Provider{}).AuthHeaders(context.Background(), models.Remote{})
if h.Get("Authorization") != "" {
t.Error("no credentials, no header")
}
}
+177
View File
@@ -0,0 +1,177 @@
package pypi
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// fakeFileStore is an in-memory provider.FileStore for exercising local index
// generation without a database.
type fakeFileStore struct {
packages []string
files map[string][]provider.FileEntry
}
func (f *fakeFileStore) ListPackages(_ context.Context, _ string) ([]string, error) {
return f.packages, nil
}
func (f *fakeFileStore) ListFilesByPrefix(_ context.Context, _, prefix string) ([]provider.FileEntry, error) {
return f.files[prefix], nil
}
func TestTypeClassifyContentType(t *testing.T) {
p := &Provider{}
if p.Type() != models.PackagePyPI {
t.Fatal("type")
}
if p.Classify("simple/foo/") != provider.Mutable {
t.Error("simple index should be mutable")
}
if p.Classify("packages/foo-1.0.whl") != provider.Immutable {
t.Error("wheel should be immutable")
}
cases := map[string]string{
"foo-1.0-py3-none-any.whl": "application/zip",
"foo-1.0.zip": "application/zip",
"foo-1.0.tar.gz": "application/gzip",
"simple/foo/": "text/html",
"weird": "application/octet-stream",
}
for path, want := range cases {
if got := p.ContentType(path); got != want {
t.Errorf("ContentType(%q)=%q want %q", path, got, want)
}
}
}
func TestUpstreamURL(t *testing.T) {
p := &Provider{}
if got := p.UpstreamURL(models.Remote{BaseURL: "https://files.example.com"}, "packages/foo.whl"); got != "https://files.example.com/packages/foo.whl" {
t.Errorf("got %q", got)
}
if got := p.UpstreamURL(models.Remote{BaseURL: "https://x"}, "simple/foo/"); got != "https://pypi.org/simple/foo/" {
t.Errorf("simple should hit pypi.org, got %q", got)
}
}
func TestValidateUpload(t *testing.T) {
p := &Provider{}
sp, ct, err := p.ValidateUpload("numpy-1.26.0-cp311-cp311-linux_x86_64.whl")
if err != nil || sp != "numpy/numpy-1.26.0-cp311-cp311-linux_x86_64.whl" || ct != "application/zip" {
t.Errorf("wheel: sp=%q ct=%q err=%v", sp, ct, err)
}
sp, ct, err = p.ValidateUpload("requests-2.31.0.tar.gz")
if err != nil || sp != "requests/requests-2.31.0.tar.gz" || ct != "application/gzip" {
t.Errorf("sdist: sp=%q ct=%q err=%v", sp, ct, err)
}
if _, _, err := p.ValidateUpload("not-a-package.txt"); err == nil {
t.Error("expected error for bad extension")
}
}
func TestPackageNameParsing(t *testing.T) {
if got := packageFromWheel("Foo_Bar-1.0-py3-none-any.whl"); got != "foo-bar" {
t.Errorf("wheel name = %q", got)
}
if got := packageFromWheel("noseparator.whl"); got != "" {
t.Errorf("expected empty for unparseable wheel, got %q", got)
}
if got := packageFromSdist("My.Pkg-2.0.tar.gz"); got != "my-pkg" {
t.Errorf("sdist name = %q", got)
}
if got := packageFromSdist("noseparator.zip"); got != "" {
t.Errorf("expected empty, got %q", got)
}
}
func TestUploadResponse(t *testing.T) {
resp := (&Provider{}).UploadResponse("foo/foo-1.0.whl", "sha256:abc", 123)
if resp["filename"] != "foo-1.0.whl" || resp["package"] != "foo" || resp["content_hash"] != "sha256:abc" {
t.Errorf("unexpected upload response: %v", resp)
}
}
func TestRewriteResponse(t *testing.T) {
p := &Provider{}
if out, _ := p.RewriteResponse([]byte("x"), models.Remote{Name: "pypi"}, ""); out != nil {
t.Error("empty proxyBaseURL is a no-op")
}
body := []byte(`<a href="https://files.pythonhosted.org/packages/foo.whl">foo.whl</a>`)
out, err := p.RewriteResponse(body, models.Remote{Name: "pypi"}, "http://proxy")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(out), "http://proxy/api/v1/remote/pypi/") {
t.Errorf("not rewritten: %s", out)
}
}
func TestGenerateLocalIndex(t *testing.T) {
p := &Provider{}
fs := &fakeFileStore{
packages: []string{"foo", "bar"},
files: map[string][]provider.FileEntry{
"foo/": {{FilePath: "foo/foo-1.0-py3-none-any.whl", ContentHash: "sha256:aaa"}},
},
}
list, err := p.GenerateLocalIndex(context.Background(), fs, "local", "simple/")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(list), "foo") || !strings.Contains(string(list), "bar") {
t.Errorf("package list missing entries: %s", list)
}
files, err := p.GenerateLocalIndex(context.Background(), fs, "local", "simple/foo/")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(files), "foo-1.0-py3-none-any.whl") {
t.Errorf("file list missing wheel: %s", files)
}
if _, err := p.GenerateLocalIndex(context.Background(), fs, "local", "notsimple"); err == nil {
t.Error("expected error for non-simple path")
}
}
func TestServeLocalIndexHTTP(t *testing.T) {
p := &Provider{}
fs := &fakeFileStore{
packages: []string{"foo"},
files: map[string][]provider.FileEntry{
"foo/": {{FilePath: "foo/foo-1.0-py3-none-any.whl", ContentHash: "sha256:aaa"}},
},
}
serve := func(path string) (*httptest.ResponseRecorder, bool) {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
handled := p.ServeLocalIndex(w, r, fs, "local", path)
return w, handled
}
if w, ok := serve("simple/"); !ok || w.Code != 200 || !strings.Contains(w.Body.String(), "foo") {
t.Errorf("simple index: handled=%v code=%d body=%s", ok, w.Code, w.Body.String())
}
if w, ok := serve("simple/foo/"); !ok || w.Code != 200 || !strings.Contains(w.Body.String(), "foo-1.0-py3-none-any.whl") {
t.Errorf("package index: handled=%v code=%d body=%s", ok, w.Code, w.Body.String())
}
// Non-simple paths are not handled.
if _, ok := serve("packages/foo.whl"); ok {
t.Error("non-index path should not be handled")
}
}
func TestAuthHeaders(t *testing.T) {
h, _ := (&Provider{}).AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
if h.Get("Authorization") == "" {
t.Error("expected auth header")
}
}
+9
View File
@@ -151,6 +151,15 @@ func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, conte
slog.Info("rpm metadata: parsed", "repo", repoName, "name", meta.Name, "version", meta.Version, "arch", meta.Arch) slog.Info("rpm metadata: parsed", "repo", repoName, "name", meta.Name, "version", meta.Version, "arch", meta.Arch)
} }
func (p *Provider) AfterDelete(ctx context.Context, repoName, storagePath string, db provider.MetadataDeleter) error {
if err := db.DeleteRPMMetadata(ctx, repoName, storagePath); err != nil {
slog.Error("rpm metadata: delete failed", "repo", repoName, "path", storagePath, "error", err)
return err
}
slog.Info("rpm metadata: deleted", "repo", repoName, "path", storagePath)
return nil
}
func rpmDepFromEntry(e rpmlib.Dependency) provider.RPMDep { func rpmDepFromEntry(e rpmlib.Dependency) provider.RPMDep {
dep := provider.RPMDep{Name: e.Name()} dep := provider.RPMDep{Name: e.Name()}
if e.Flags() != 0 { if e.Flags() != 0 {
+276
View File
@@ -0,0 +1,276 @@
package rpm
import (
"bytes"
"compress/gzip"
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type fakeBlobReader struct{ data []byte }
func (f fakeBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
return io.NopCloser(bytes.NewReader(f.data)), int64(len(f.data)), nil
}
type fakeMetaStore struct{ inserted *provider.RPMMetadata }
func (f *fakeMetaStore) InsertRPMMetadata(_ context.Context, m *provider.RPMMetadata) error {
f.inserted = m
return nil
}
type fakeRPMReader struct{ metas []provider.RPMMetadata }
func (f fakeRPMReader) ListRPMMetadataEntries(_ context.Context, _ string) ([]provider.RPMMetadata, error) {
return f.metas, nil
}
func (f fakeRPMReader) ListFilesByPrefix(_ context.Context, _, _ string) ([]provider.FileEntry, error) {
return nil, nil
}
func (f fakeRPMReader) ListPackages(_ context.Context, _ string) ([]string, error) { return nil, nil }
func TestRPMPureFuncs(t *testing.T) {
p := &Provider{}
if p.Type() != models.PackageRPM {
t.Error("type")
}
if p.Classify("repodata/repomd.xml") != provider.Mutable {
t.Error("repomd should be mutable")
}
if p.Classify("Packages/foo.rpm") != provider.Immutable {
t.Error("rpm should be immutable")
}
if p.ContentType("x.rpm") != "application/x-rpm" {
t.Error("rpm content type")
}
if got := p.UpstreamURL(models.Remote{BaseURL: "https://mirror/"}, "/Packages/x.rpm"); got != "https://mirror/Packages/x.rpm" {
t.Errorf("upstream url %q", got)
}
if out, _ := p.RewriteResponse(nil, models.Remote{}, "http://p"); out != nil {
t.Error("rpm never rewrites")
}
h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
if h.Get("Authorization") == "" {
t.Error("auth header")
}
}
func TestRPMValidateUpload(t *testing.T) {
p := &Provider{}
sp, ct, err := p.ValidateUpload("dir/foo-1.0.noarch.rpm")
if err != nil || sp != "Packages/foo-1.0.noarch.rpm" || ct != "application/x-rpm" {
t.Errorf("sp=%q ct=%q err=%v", sp, ct, err)
}
if _, _, err := p.ValidateUpload("foo.txt"); err == nil {
t.Error("expected error for non-rpm")
}
resp := p.UploadResponse("Packages/foo.rpm", "sha256:abc", 10)
if resp["content_hash"] != "sha256:abc" {
t.Errorf("upload response %v", resp)
}
}
func TestRPMAfterUpload(t *testing.T) {
data := testsupport.MinimalRPM("e2e-testpkg", "1.0", "1", "noarch")
store := &fakeMetaStore{}
(&Provider{}).AfterUpload(context.Background(), "myrepo", "Packages/e2e-testpkg-1.0-1.noarch.rpm",
"sha256:deadbeef", fakeBlobReader{data: data}, store)
m := store.inserted
if m == nil {
t.Fatal("no metadata inserted")
}
if m.Name != "e2e-testpkg" || m.Version != "1.0" || m.Release != "1" || m.Arch != "noarch" {
t.Errorf("unexpected metadata: %+v", m)
}
if m.RPMSize != int64(len(data)) {
t.Errorf("RPMSize = %d, want %d", m.RPMSize, len(data))
}
if len(m.Provides) == 0 {
t.Error("expected the package to provide itself")
}
}
type errBlobReader struct{}
func (errBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
return nil, 0, io.ErrUnexpectedEOF
}
func TestRPMAfterUploadErrors(t *testing.T) {
// Download failure: no metadata inserted, no panic.
store := &fakeMetaStore{}
(&Provider{}).AfterUpload(context.Background(), "r", "p", "sha256:x", errBlobReader{}, store)
if store.inserted != nil {
t.Error("no metadata should be inserted on download error")
}
// Parse failure: garbage bytes are not a valid RPM.
store2 := &fakeMetaStore{}
(&Provider{}).AfterUpload(context.Background(), "r", "p", "sha256:x", fakeBlobReader{data: []byte("not an rpm")}, store2)
if store2.inserted != nil {
t.Error("no metadata should be inserted on parse error")
}
}
func TestRPMServeRepodata(t *testing.T) {
p := &Provider{}
reader := fakeRPMReader{metas: []provider.RPMMetadata{{
Name: "e2e-testpkg", Version: "1.0", Release: "1", Arch: "noarch",
Summary: "test & <special>",
ContentHash: "sha256:abc",
Requires: []provider.RPMDep{{Name: "libc", Flags: "GE", Version: "2.0"}},
Provides: []provider.RPMDep{{Name: "e2e-testpkg"}},
Files: []provider.RPMFile{{Path: "/usr/share/e2e/README", Type: "file"}},
Changelogs: []provider.RPMChangelog{{Author: "e2e", Date: 1, Text: "init"}},
}}}
serve := func(path string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if !p.ServeLocalIndex(w, r, reader, "myrepo", path) {
t.Fatalf("ServeLocalIndex returned false for %q", path)
}
return w
}
if w := serve("repodata/repomd.xml"); w.Code != 200 || !strings.Contains(w.Body.String(), "<repomd") {
t.Errorf("repomd: code=%d body=%s", w.Code, w.Body.String())
}
for _, name := range []string{"repodata/h-primary.xml.gz", "repodata/h-filelists.xml.gz", "repodata/h-other.xml.gz"} {
w := serve(name)
if w.Code != 200 {
t.Errorf("%s: code %d", name, w.Code)
}
if _, err := gzip.NewReader(bytes.NewReader(w.Body.Bytes())); err != nil {
t.Errorf("%s: not gzip: %v", name, err)
}
}
// Unknown repodata file -> 404.
if w := serve("repodata/bogus"); w.Code != http.StatusNotFound {
t.Errorf("bogus repodata: code %d", w.Code)
}
// Non-repodata path -> not handled.
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/Packages/x.rpm", nil)
if p.ServeLocalIndex(w, r, reader, "myrepo", "Packages/x.rpm") {
t.Error("expected ServeLocalIndex false for non-repodata path")
}
}
type errRPMReader struct{}
func (errRPMReader) ListRPMMetadataEntries(context.Context, string) ([]provider.RPMMetadata, error) {
return nil, io.ErrUnexpectedEOF
}
func (errRPMReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
return nil, nil
}
func (errRPMReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
func TestRPMServeMetadataError(t *testing.T) {
p := &Provider{}
for _, path := range []string{"repodata/repomd.xml", "repodata/h-primary.xml.gz", "repodata/h-filelists.xml.gz", "repodata/h-other.xml.gz"} {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
p.ServeLocalIndex(w, r, errRPMReader{}, "repo", path)
if w.Code != 500 {
t.Errorf("%s with failing reader = %d, want 500", path, w.Code)
}
}
}
func TestRPMFullMetadataXML(t *testing.T) {
// A fully-populated entry exercises every optional-field branch in the
// primary/filelists/other XML generators.
metas := []provider.RPMMetadata{{
Name: "full", Epoch: 1, Version: "2.0", Release: "3", Arch: "x86_64",
Summary: "s", Description: "d", License: "MIT", Vendor: "acme",
Group: "System", BuildHost: "build.example.com", SourceRPM: "full-2.0.src.rpm",
URL: "https://example.com", Packager: "pkgr", ContentHash: "sha256:abc",
RPMSize: 100, InstalledSize: 200,
Requires: []provider.RPMDep{{Name: "libc", Flags: "GE", Epoch: "0", Version: "2.0", Release: "1"}},
Provides: []provider.RPMDep{{Name: "full", Flags: "EQ", Version: "2.0"}},
Files: []provider.RPMFile{{Path: "/usr/bin/full", Type: "file"}, {Path: "/etc/full", Type: "dir"}},
Changelogs: []provider.RPMChangelog{{Author: "a", Date: 100, Text: "changed"}},
}}
for _, gen := range []func([]provider.RPMMetadata) []byte{generatePrimaryXMLGZ, generateFilelistsXMLGZ, generateOtherXMLGZ} {
zr, err := gzip.NewReader(bytes.NewReader(gen(metas)))
if err != nil {
t.Fatal(err)
}
if _, err := io.ReadAll(zr); err != nil {
t.Error(err)
}
}
}
func TestRPMPrimaryXMLContents(t *testing.T) {
// Exercise xmlEscape and dependency entry writing through the gzip'd XML.
metas := []provider.RPMMetadata{{
Name: "pkg", Version: "1", Release: "1", Arch: "x86_64", Summary: "a & b",
Requires: []provider.RPMDep{{Name: "dep", Flags: "EQ", Version: "1.0", Epoch: "0"}},
}}
gz := generatePrimaryXMLGZ(metas)
zr, err := gzip.NewReader(bytes.NewReader(gz))
if err != nil {
t.Fatal(err)
}
out, _ := io.ReadAll(zr)
s := string(out)
if !strings.Contains(s, "a &amp; b") {
t.Errorf("summary not xml-escaped: %s", s)
}
if !strings.Contains(s, "<name>pkg</name>") {
t.Errorf("package name missing: %s", s)
}
}
func TestRPMContentTypeAndHelpers(t *testing.T) {
p := &Provider{}
for path, want := range map[string]string{
"x.rpm": "application/x-rpm",
"repodata/repomd.xml": "application/xml",
"repodata/h-primary.xml.gz": "application/xml",
"repodata/h-primary.xml.xz": "application/xml",
"Packages/other": "application/octet-stream",
} {
if got := p.ContentType(path); got != want {
t.Errorf("ContentType(%q)=%q want %q", path, got, want)
}
}
for flag, want := range map[int]string{
0x08 | 0x04: "GE",
0x02 | 0x04: "LE",
0x08: "GT",
0x02: "LT",
0x04: "EQ",
0x00: "",
} {
if got := rpmFlagString(flag); got != want {
t.Errorf("rpmFlagString(%d)=%q want %q", flag, got, want)
}
}
if firstGroup(nil) != "Unspecified" {
t.Error("empty groups should be Unspecified")
}
if firstGroup([]string{"System", "Base"}) != "System" {
t.Error("firstGroup should return the first")
}
}
func TestGenerateLocalIndexUnsupported(t *testing.T) {
if _, err := (&Provider{}).GenerateLocalIndex(context.Background(), fakeRPMReader{}, "r", "simple/"); err == nil {
t.Error("expected unsupported error")
}
}
+21 -1
View File
@@ -2,6 +2,7 @@ package proxy
import ( import (
"regexp" "regexp"
"sync"
"git.unkin.net/unkin/artifactapi/internal/provider" "git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models" "git.unkin.net/unkin/artifactapi/pkg/models"
@@ -60,10 +61,29 @@ func (c *Classifier) Classify(remote models.Remote, path string) Classification
return ClassImmutable return ClassImmutable
} }
// patternCache memoises regex compilation. Classify runs on every proxied
// request and previously recompiled each remote's pattern lists every time;
// keying by the pattern string lets each distinct pattern compile once and
// then be reused, with no invalidation needed (the pattern text is the key).
// A pattern that fails to compile is cached as a typed nil so we don't retry.
var patternCache sync.Map // map[string]*regexp.Regexp
func compileCached(pattern string) *regexp.Regexp {
if v, ok := patternCache.Load(pattern); ok {
return v.(*regexp.Regexp)
}
re, err := regexp.Compile(pattern)
if err != nil {
re = nil
}
patternCache.Store(pattern, re)
return re
}
func compilePatterns(patterns []string) []*regexp.Regexp { func compilePatterns(patterns []string) []*regexp.Regexp {
compiled := make([]*regexp.Regexp, 0, len(patterns)) compiled := make([]*regexp.Regexp, 0, len(patterns))
for _, p := range patterns { for _, p := range patterns {
if re, err := regexp.Compile(p); err == nil { if re := compileCached(p); re != nil {
compiled = append(compiled, re) compiled = append(compiled, re)
} }
} }
+52
View File
@@ -0,0 +1,52 @@
package proxy
import (
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestClassifierBranches(t *testing.T) {
gp, err := provider.Get(models.PackageGeneric)
if err != nil {
t.Fatal(err)
}
c := NewClassifier(gp)
if c.Classify(models.Remote{Blocklist: []string{`\.exe$`}}, "x.exe") != ClassDenied {
t.Error("blocklist match should be denied")
}
// Allowlist present but path doesn't match -> denied.
allow := models.Remote{Patterns: []string{`^allowed/`}}
if c.Classify(allow, "other/x") != ClassDenied {
t.Error("non-allowlisted path should be denied")
}
if c.Classify(allow, "allowed/x") != ClassImmutable {
t.Error("allowlisted generic path should be immutable")
}
if c.Classify(models.Remote{MutablePatterns: []string{`index$`}}, "a/index") != ClassMutable {
t.Error("mutable pattern override failed")
}
if c.Classify(models.Remote{ImmutablePatterns: []string{`\.bin$`}}, "a.bin") != ClassImmutable {
t.Error("immutable pattern failed")
}
// An invalid regex is skipped (not treated as a match) rather than denying.
if c.Classify(models.Remote{Blocklist: []string{`[invalid`}}, "anything") == ClassDenied {
t.Error("invalid blocklist regex should be skipped, not deny everything")
}
}
func TestClassificationString(t *testing.T) {
for c, want := range map[Classification]string{
ClassImmutable: "immutable",
ClassMutable: "mutable",
ClassDenied: "denied",
Classification(99): "unknown",
} {
if c.String() != want {
t.Errorf("Classification(%d).String() = %q, want %q", c, c.String(), want)
}
}
}
+396 -86
View File
@@ -4,10 +4,13 @@ import (
"context" "context"
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"log/slog" "log/slog"
"net/http" "net/http"
"strings"
"time" "time"
"git.unkin.net/unkin/artifactapi/internal/cache" "git.unkin.net/unkin/artifactapi/internal/cache"
@@ -19,19 +22,65 @@ import (
const fetchLockTTL = 30 * time.Second const fetchLockTTL = 30 * time.Second
const (
accessLogBufferSize = 4096
accessLogBatchSize = 128
accessLogFlushEvery = 2 * time.Second
)
type Engine struct { type Engine struct {
db *database.DB db *database.DB
cache *cache.Redis cache *cache.Redis
store *storage.S3 store *storage.S3
cas *storage.CAS cas *storage.CAS
circuit *CircuitBreaker
accessLog chan database.AccessLogEntry
} }
func NewEngine(db *database.DB, c *cache.Redis, s *storage.S3) *Engine { func NewEngine(db *database.DB, c *cache.Redis, s *storage.S3) *Engine {
return &Engine{ e := &Engine{
db: db, db: db,
cache: c, cache: c,
store: s, store: s,
cas: storage.NewCAS(s), cas: storage.NewCAS(s),
circuit: NewCircuitBreaker(c),
accessLog: make(chan database.AccessLogEntry, accessLogBufferSize),
}
go e.runAccessLogWriter()
return e
}
// runAccessLogWriter drains the access-log channel and writes rows in batches,
// replacing a goroutine-per-request insert. It runs for the process lifetime;
// access logs are best-effort telemetry, so a small tail may be lost on abrupt
// shutdown.
func (e *Engine) runAccessLogWriter() {
ticker := time.NewTicker(accessLogFlushEvery)
defer ticker.Stop()
batch := make([]database.AccessLogEntry, 0, accessLogBatchSize)
flush := func() {
if len(batch) == 0 {
return
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
if err := e.db.InsertAccessLogBatch(ctx, batch); err != nil {
slog.Warn("access log batch insert failed", "error", err, "count", len(batch))
}
cancel()
batch = batch[:0]
}
for {
select {
case entry := <-e.accessLog:
batch = append(batch, entry)
if len(batch) >= accessLogBatchSize {
flush()
}
case <-ticker.C:
flush()
}
} }
} }
@@ -42,7 +91,7 @@ type FetchResult struct {
Source string // "cache" or "remote" Source string // "cache" or "remote"
} }
func (e *Engine) Fetch(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*FetchResult, error) { func (e *Engine) Fetch(ctx context.Context, remote models.Remote, path string, prov provider.Provider, clientHeaders ...http.Header) (*FetchResult, error) {
classifier := NewClassifier(prov) classifier := NewClassifier(prov)
class := classifier.Classify(remote, path) class := classifier.Classify(remote, path)
@@ -61,7 +110,7 @@ func (e *Engine) Fetch(ctx context.Context, remote models.Remote, path string, p
result, err := e.serveFromStore(ctx, remote, path) result, err := e.serveFromStore(ctx, remote, path)
if err == nil { if err == nil {
result.Source = "cache" result.Source = "cache"
go e.logAccess(remote.Name, path, true, result.Size, 0) e.logAccess(remote.Name, path, true, result.Size, 0)
return result, nil return result, nil
} }
slog.Warn("cache hit but S3 miss, re-fetching", "remote", remote.Name, "path", path) slog.Warn("cache hit but S3 miss, re-fetching", "remote", remote.Name, "path", path)
@@ -73,11 +122,12 @@ func (e *Engine) Fetch(ctx context.Context, remote models.Remote, path string, p
} }
if !locked { if !locked {
time.Sleep(500 * time.Millisecond) // Another request holds the fetch lock. Poll the store until the leader
result, err := e.serveFromStore(ctx, remote, path) // populates it rather than immediately racing to fetch upstream too; a
if err == nil { // cold-cache stampede otherwise hits upstream once per waiter.
if result := e.waitForStore(ctx, remote, path); result != nil {
result.Source = "cache" result.Source = "cache"
go e.logAccess(remote.Name, path, true, result.Size, 0) e.logAccess(remote.Name, path, true, result.Size, 0)
return result, nil return result, nil
} }
} }
@@ -96,35 +146,138 @@ func (e *Engine) Fetch(ctx context.Context, remote models.Remote, path string, p
result, err := e.serveFromStore(ctx, remote, path) result, err := e.serveFromStore(ctx, remote, path)
if err == nil { if err == nil {
result.Source = "cache" result.Source = "cache"
go e.logAccess(remote.Name, path, true, result.Size, 0) e.logAccess(remote.Name, path, true, result.Size, 0)
return result, nil return result, nil
} }
} }
} }
} }
var fwdHeaders http.Header
if len(clientHeaders) > 0 && clientHeaders[0] != nil {
fwdHeaders = clientHeaders[0]
}
// Short-circuit upstream calls when the remote's breaker is open: serve
// stale from the store if we have it, otherwise fail fast rather than
// hammering a known-bad upstream.
if e.circuit.IsOpen(ctx, remote.Name) {
if stale, serr := e.serveFromStore(ctx, remote, path); serr == nil {
slog.Warn("circuit open, serving stale", "remote", remote.Name, "path", path)
stale.Source = "cache"
e.logAccess(remote.Name, path, true, stale.Size, 0)
return stale, nil
}
return nil, &ProxyError{Status: http.StatusServiceUnavailable, Message: "upstream circuit open"}
}
start := time.Now() start := time.Now()
result, err := e.fetchFromUpstream(ctx, remote, path, prov, class, ttl) result, err := e.fetchFromUpstream(ctx, remote, path, prov, class, ttl, fwdHeaders)
upstreamMS := int(time.Since(start).Milliseconds()) upstreamMS := int(time.Since(start).Milliseconds())
if err != nil { if err != nil {
if isNetworkError(err) {
e.circuit.RecordFailure(ctx, remote.Name)
}
if remote.StaleOnError && isNetworkError(err) { if remote.StaleOnError && isNetworkError(err) {
_ = e.cache.SetTTL(ctx, remote.Name, path, ttl) _ = e.cache.SetTTL(ctx, remote.Name, path, ttl)
stale, serr := e.serveFromStore(ctx, remote, path) stale, serr := e.serveFromStore(ctx, remote, path)
if serr == nil { if serr == nil {
slog.Warn("serving stale on upstream error", "remote", remote.Name, "path", path, "error", err) slog.Warn("serving stale on upstream error", "remote", remote.Name, "path", path, "error", err)
stale.Source = "cache" stale.Source = "cache"
go e.logAccess(remote.Name, path, true, stale.Size, 0) e.logAccess(remote.Name, path, true, stale.Size, 0)
return stale, nil return stale, nil
} }
} }
return nil, err return nil, err
} }
go e.logAccess(remote.Name, path, false, result.Size, upstreamMS) e.circuit.RecordSuccess(ctx, remote.Name)
e.logAccess(remote.Name, path, false, result.Size, upstreamMS)
return result, nil return result, nil
} }
func (e *Engine) fetchFromUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider, class Classification, ttl time.Duration) (*FetchResult, error) { // HeadResult carries artifact metadata for a HEAD request. There is no body.
type HeadResult struct {
ContentType string
Size int64
Source string // "cache" or "remote"
}
// Head resolves artifact metadata without fetching or streaming the body.
// Cached artifacts/indexes are answered from the store metadata; on a miss it
// issues an upstream HEAD. It never downloads or caches the body.
func (e *Engine) Head(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*HeadResult, error) {
class := NewClassifier(prov).Classify(remote, path)
if class == ClassDenied {
return nil, &ProxyError{Status: http.StatusForbidden, Message: "access denied"}
}
if artifact, err := e.db.GetArtifact(ctx, remote.Name, path); err == nil && artifact != nil {
return &HeadResult{ContentType: artifact.ContentType, Size: artifact.SizeBytes, Source: "cache"}, nil
}
if info, err := e.store.Stat(ctx, storage.IndexKey(remote.Name, path)); err == nil {
return &HeadResult{ContentType: info.ContentType, Size: info.Size, Source: "cache"}, nil
}
return e.headUpstream(ctx, remote, path, prov)
}
func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*HeadResult, error) {
url := prov.UpstreamURL(remote, path)
authHeaders, err := prov.AuthHeaders(ctx, remote)
if err != nil {
return nil, fmt.Errorf("auth headers: %w", err)
}
doHead := func(extra http.Header) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return nil, fmt.Errorf("create request: %w", err)
}
for k, vv := range authHeaders {
for _, v := range vv {
req.Header.Add(k, v)
}
}
for k, vv := range extra {
for _, v := range vv {
req.Header.Set(k, v)
}
}
return http.DefaultClient.Do(req)
}
resp, err := doHead(nil)
if err != nil {
return nil, &UpstreamError{Err: err}
}
if resp.StatusCode == http.StatusUnauthorized {
resp.Body.Close()
token, _, terr := fetchBearerToken(ctx, resp.Header.Get("Www-Authenticate"), remote)
if terr == nil && token != "" {
resp, err = doHead(http.Header{"Authorization": []string{"Bearer " + token}})
if err != nil {
return nil, &UpstreamError{Err: err}
}
} else {
return nil, &ProxyError{Status: http.StatusUnauthorized, Message: "upstream returned 401"}
}
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, &ProxyError{Status: resp.StatusCode, Message: fmt.Sprintf("upstream returned %d", resp.StatusCode)}
}
contentType := prov.ContentType(path)
if ct := resp.Header.Get("Content-Type"); ct != "" {
contentType = ct
}
return &HeadResult{ContentType: contentType, Size: resp.ContentLength, Source: "remote"}, nil
}
func (e *Engine) fetchFromUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider, class Classification, ttl time.Duration, clientHeaders http.Header) (*FetchResult, error) {
url := prov.UpstreamURL(remote, path) url := prov.UpstreamURL(remote, path)
authHeaders, err := prov.AuthHeaders(ctx, remote) authHeaders, err := prov.AuthHeaders(ctx, remote)
@@ -141,94 +294,144 @@ func (e *Engine) fetchFromUpstream(ctx context.Context, remote models.Remote, pa
req.Header.Add(k, v) req.Header.Add(k, v)
} }
} }
if clientHeaders != nil {
if accept := clientHeaders.Get("Accept"); accept != "" {
req.Header.Set("Accept", accept)
}
}
resp, err := http.DefaultClient.Do(req) resp, err := clientForRemote(remote).Do(req)
if err != nil { if err != nil {
return nil, &UpstreamError{Err: err} return nil, &UpstreamError{Err: err}
} }
if resp.StatusCode == http.StatusUnauthorized {
resp.Body.Close()
token, err := e.cachedBearerToken(ctx, resp.Header.Get("Www-Authenticate"), remote)
if err == nil && token != "" {
req2, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
req2.Header.Set("Authorization", "Bearer "+token)
if clientHeaders != nil {
if accept := clientHeaders.Get("Accept"); accept != "" {
req2.Header.Set("Accept", accept)
}
}
resp, err = clientForRemote(remote).Do(req2)
if err != nil {
return nil, &UpstreamError{Err: err}
}
} else {
return nil, &ProxyError{Status: http.StatusUnauthorized, Message: "upstream returned 401"}
}
}
if resp.StatusCode != http.StatusOK { if resp.StatusCode != http.StatusOK {
resp.Body.Close() resp.Body.Close()
return nil, &ProxyError{Status: resp.StatusCode, Message: fmt.Sprintf("upstream returned %d", resp.StatusCode)} return nil, &ProxyError{Status: resp.StatusCode, Message: fmt.Sprintf("upstream returned %d", resp.StatusCode)}
} }
body, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return nil, fmt.Errorf("read upstream body: %w", err)
}
rewritten, err := prov.RewriteResponse(body, remote, "")
if err != nil {
return nil, fmt.Errorf("rewrite response: %w", err)
}
if rewritten != nil {
body = rewritten
}
contentType := prov.ContentType(path) contentType := prov.ContentType(path)
if ct := resp.Header.Get("Content-Type"); ct != "" && contentType == "application/octet-stream" { if ct := resp.Header.Get("Content-Type"); ct != "" {
contentType = ct contentType = ct
} }
// Mutable indexes are small and may be rewritten, so buffer them in memory.
if class == ClassMutable { if class == ClassMutable {
body, err := io.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
return nil, fmt.Errorf("read upstream body: %w", err)
}
rewritten, err := prov.RewriteResponse(body, remote, "")
if err != nil {
return nil, fmt.Errorf("rewrite response: %w", err)
}
if rewritten != nil {
body = rewritten
}
s3Key := storage.IndexKey(remote.Name, path) s3Key := storage.IndexKey(remote.Name, path)
if err := e.store.Upload(ctx, s3Key, bytesReader(body), int64(len(body)), contentType); err != nil { if err := e.store.Upload(ctx, s3Key, bytesReader(body), int64(len(body)), contentType); err != nil {
return nil, fmt.Errorf("upload index: %w", err) return nil, fmt.Errorf("upload index: %w", err)
} }
etag := resp.Header.Get("ETag")
_ = e.cache.SetTTL(ctx, remote.Name, path, ttl)
if etag != "" {
_ = e.cache.SetETag(ctx, remote.Name, path, etag, ttl)
}
} else {
hash := sha256Hash(body)
s3Key := storage.BlobKey(hash)
exists, _ := e.store.Exists(ctx, s3Key)
if !exists {
if err := e.store.Upload(ctx, s3Key, bytesReader(body), int64(len(body)), contentType); err != nil {
return nil, fmt.Errorf("upload blob: %w", err)
}
}
contentHash := fmt.Sprintf("sha256:%s", hash)
if err := e.db.UpsertBlob(ctx, contentHash, s3Key, int64(len(body)), contentType); err != nil {
slog.Warn("upsert blob failed", "error", err)
}
if err := e.db.UpsertArtifact(ctx, remote.Name, path, contentHash, resp.Header.Get("ETag")); err != nil {
slog.Warn("upsert artifact failed", "error", err)
}
_ = e.cache.SetTTL(ctx, remote.Name, path, ttl) _ = e.cache.SetTTL(ctx, remote.Name, path, ttl)
if etag := resp.Header.Get("ETag"); etag != "" { if etag := resp.Header.Get("ETag"); etag != "" {
_ = e.cache.SetETag(ctx, remote.Name, path, etag, ttl) _ = e.cache.SetETag(ctx, remote.Name, path, etag, ttl)
} }
return &FetchResult{
Reader: io.NopCloser(bytesReader(body)),
ContentType: contentType,
Size: int64(len(body)),
Source: "remote",
}, nil
} }
// Immutable blobs are streamed through the content-addressable store
// (tempfile -> sha256 -> S3) so arbitrarily large artifacts never sit
// fully in memory. Immutable content is never rewritten in the proxy path.
casResult, err := e.cas.Store(ctx, resp.Body, contentType)
resp.Body.Close()
if err != nil {
return nil, fmt.Errorf("store blob: %w", err)
}
if err := e.db.UpsertBlob(ctx, casResult.ContentHash, casResult.S3Key, casResult.SizeBytes, contentType); err != nil {
slog.Warn("upsert blob failed", "error", err)
}
if err := e.db.UpsertArtifact(ctx, remote.Name, path, casResult.ContentHash, resp.Header.Get("ETag")); err != nil {
slog.Warn("upsert artifact failed", "error", err)
}
_ = e.cache.SetTTL(ctx, remote.Name, path, ttl)
if etag := resp.Header.Get("ETag"); etag != "" {
_ = e.cache.SetETag(ctx, remote.Name, path, etag, ttl)
}
reader, info, err := e.store.Download(ctx, casResult.S3Key)
if err != nil {
return nil, fmt.Errorf("serve stored blob: %w", err)
}
return &FetchResult{ return &FetchResult{
Reader: io.NopCloser(bytesReader(body)), Reader: reader,
ContentType: contentType, ContentType: info.ContentType,
Size: int64(len(body)), Size: casResult.SizeBytes,
Source: "remote", Source: "remote",
}, nil }, nil
} }
// waitForStore polls the store for an artifact populated by the request that
// holds the fetch lock, returning it once available or nil if it does not
// appear within the wait budget (after which the caller fetches upstream
// itself). It stops early if the request context is cancelled.
func (e *Engine) waitForStore(ctx context.Context, remote models.Remote, path string) *FetchResult {
const (
pollInterval = 100 * time.Millisecond
maxWait = 5 * time.Second
)
deadline := time.Now().Add(maxWait)
for {
if result, err := e.serveFromStore(ctx, remote, path); err == nil {
return result
}
if time.Now().After(deadline) {
return nil
}
select {
case <-ctx.Done():
return nil
case <-time.After(pollInterval):
}
}
}
func (e *Engine) serveFromStore(ctx context.Context, remote models.Remote, path string) (*FetchResult, error) { func (e *Engine) serveFromStore(ctx context.Context, remote models.Remote, path string) (*FetchResult, error) {
artifact, err := e.db.GetArtifact(ctx, remote.Name, path) artifact, err := e.db.GetArtifact(ctx, remote.Name, path)
if err == nil && artifact != nil { if err == nil && artifact != nil {
reader, info, err := e.store.Download(ctx, artifact.ContentHash[len("sha256:"):])
if err == nil {
_ = e.db.TouchArtifactAccess(ctx, remote.Name, path)
return &FetchResult{
Reader: reader,
ContentType: info.ContentType,
Size: info.Size,
}, nil
}
s3Key := storage.BlobKey(artifact.ContentHash[len("sha256:"):]) s3Key := storage.BlobKey(artifact.ContentHash[len("sha256:"):])
reader, info, err = e.store.Download(ctx, s3Key) reader, info, err := e.store.Download(ctx, s3Key)
if err == nil { if err == nil {
_ = e.db.TouchArtifactAccess(ctx, remote.Name, path) _ = e.db.TouchArtifactAccess(ctx, remote.Name, path)
return &FetchResult{ return &FetchResult{
@@ -270,7 +473,7 @@ func (e *Engine) checkUpstream(ctx context.Context, remote models.Remote, path,
} }
} }
resp, err := http.DefaultClient.Do(req) resp, err := clientForRemote(remote).Do(req)
if err != nil { if err != nil {
return false, &UpstreamError{Err: err} return false, &UpstreamError{Err: err}
} }
@@ -291,15 +494,20 @@ func (e *Engine) ttlFor(remote models.Remote, class Classification) time.Duratio
} }
} }
// logAccess enqueues an access-log entry for the batch writer. It never blocks
// the request path: if the buffer is full the entry is dropped.
func (e *Engine) logAccess(remoteName, path string, cacheHit bool, size int64, upstreamMS int) { func (e *Engine) logAccess(remoteName, path string, cacheHit bool, size int64, upstreamMS int) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) select {
defer cancel() case e.accessLog <- database.AccessLogEntry{
_ = e.db.InsertAccessLog(ctx, remoteName, path, cacheHit, size, upstreamMS, "") RemoteName: remoteName,
} Path: path,
CacheHit: cacheHit,
func sha256Hash(data []byte) string { SizeBytes: size,
h := sha256.Sum256(data) UpstreamMS: upstreamMS,
return hex.EncodeToString(h[:]) }:
default:
slog.Warn("access log buffer full, dropping entry", "remote", remoteName, "path", path)
}
} }
func bytesReader(data []byte) io.Reader { func bytesReader(data []byte) io.Reader {
@@ -319,6 +527,110 @@ func (r readerAt) ReadAt(p []byte, off int64) (n int, err error) {
return return
} }
// bearerTokenTTLDefault/Margin bound how long a token is cached: the default
// is used when the token endpoint omits expires_in, and the margin is
// subtracted so a cached token is refreshed slightly before it actually expires.
const (
bearerTokenTTLDefault = 60 * time.Second
bearerTokenTTLMargin = 10 * time.Second
)
func sha256Hash(data []byte) string {
h := sha256.Sum256(data)
return hex.EncodeToString(h[:])
}
// cachedBearerToken returns a bearer token for the given challenge, reusing a
// Redis-cached token for the same remote+challenge while it is still valid.
func (e *Engine) cachedBearerToken(ctx context.Context, wwwAuth string, remote models.Remote) (string, error) {
key := remote.Name + ":" + sha256Hash([]byte(wwwAuth))
if tok, err := e.cache.GetToken(ctx, key); err == nil && tok != "" {
return tok, nil
}
tok, ttl, err := fetchBearerToken(ctx, wwwAuth, remote)
if err != nil {
return "", err
}
if tok != "" {
if ttl <= 0 {
ttl = bearerTokenTTLDefault
}
if ttl > bearerTokenTTLMargin {
ttl -= bearerTokenTTLMargin
}
_ = e.cache.SetToken(ctx, key, tok, ttl)
}
return tok, nil
}
func fetchBearerToken(ctx context.Context, wwwAuth string, remote models.Remote) (string, time.Duration, error) {
if !strings.HasPrefix(wwwAuth, "Bearer ") {
return "", 0, fmt.Errorf("not a Bearer challenge")
}
params := map[string]string{}
for _, part := range strings.Split(wwwAuth[7:], ",") {
part = strings.TrimSpace(part)
eq := strings.Index(part, "=")
if eq < 0 {
continue
}
key := part[:eq]
val := strings.Trim(part[eq+1:], `"`)
params[key] = val
}
realm := params["realm"]
if realm == "" {
return "", 0, fmt.Errorf("no realm in Bearer challenge")
}
tokenURL := realm
sep := "?"
if s, ok := params["service"]; ok {
tokenURL += sep + "service=" + s
sep = "&"
}
if s, ok := params["scope"]; ok {
tokenURL += sep + "scope=" + s
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, tokenURL, nil)
if err != nil {
return "", 0, err
}
if remote.Username != "" && remote.Password != "" {
req.SetBasicAuth(remote.Username, remote.Password)
}
resp, err := clientForRemote(remote).Do(req)
if err != nil {
return "", 0, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", 0, fmt.Errorf("token endpoint returned %d", resp.StatusCode)
}
var tokenResp struct {
Token string `json:"token"`
AccessToken string `json:"access_token"`
ExpiresIn int `json:"expires_in"`
}
if err := json.NewDecoder(resp.Body).Decode(&tokenResp); err != nil {
return "", 0, err
}
ttl := time.Duration(tokenResp.ExpiresIn) * time.Second
if tokenResp.Token != "" {
return tokenResp.Token, ttl, nil
}
return tokenResp.AccessToken, ttl, nil
}
type ProxyError struct { type ProxyError struct {
Status int Status int
Message string Message string
@@ -334,8 +646,6 @@ func (e *UpstreamError) Error() string { return fmt.Sprintf("upstream error: %v"
func (e *UpstreamError) Unwrap() error { return e.Err } func (e *UpstreamError) Unwrap() error { return e.Err }
func isNetworkError(err error) bool { func isNetworkError(err error) bool {
if _, ok := err.(*UpstreamError); ok { var ue *UpstreamError
return true return errors.As(err, &ue)
}
return false
} }
+557
View File
@@ -0,0 +1,557 @@
package proxy
import (
"context"
"io"
"net/http"
"net/http/httptest"
"os"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/cache"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/provider"
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
_ "git.unkin.net/unkin/artifactapi/internal/provider/npm"
"git.unkin.net/unkin/artifactapi/internal/storage"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
var (
testEngine *Engine
testCache *cache.Redis
testDB *database.DB
upstream *httptest.Server
)
func TestMain(m *testing.M) {
ctx := context.Background()
dsn, termPG, err := testsupport.StartPostgres(ctx)
if err != nil {
os.Exit(m.Run())
}
redisURL, termRedis, err := testsupport.StartRedis(ctx)
if err != nil {
termPG()
os.Exit(m.Run())
}
minio, termMinio, err := testsupport.StartMinio(ctx)
if err != nil {
termPG()
termRedis()
os.Exit(m.Run())
}
db, err := database.New(dsn)
if err != nil {
panic(err)
}
redis, err := cache.NewRedis(redisURL)
if err != nil {
panic(err)
}
var s3 *storage.S3
for i := 0; i < 20; i++ {
if s3, err = storage.NewS3(minio.Endpoint, minio.AccessKey, minio.SecretKey, "proxy-test", false, ""); err == nil {
break
}
time.Sleep(500 * time.Millisecond)
}
if err != nil {
panic(err)
}
testCache = redis
testDB = db
testEngine = NewEngine(db, redis, s3)
upstream = httptest.NewServer(http.HandlerFunc(mockUpstream))
code := m.Run()
upstream.Close()
db.Close()
termMinio()
termRedis()
termPG()
if code != 0 {
os.Exit(code)
}
}
func mockUpstream(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/blob.bin":
w.Header().Set("Content-Type", "application/octet-stream")
w.Write([]byte("immutable blob"))
case "/pkg": // npm metadata: mutable, supports revalidation
if r.Method == http.MethodHead && r.Header.Get("If-None-Match") == `"v1"` {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("ETag", `"v1"`)
w.Write([]byte(`{"name":"pkg"}`))
case "/protected.bin": // requires a bearer token obtained from /token
if r.Header.Get("Authorization") != "Bearer minted-token" {
w.Header().Set("Www-Authenticate", `Bearer realm="`+upstream.URL+`/token",service="reg",scope="repo:pull"`)
w.WriteHeader(http.StatusUnauthorized)
return
}
w.Write([]byte("protected payload"))
case "/protected2.bin": // same challenge as /protected.bin
if r.Header.Get("Authorization") != "Bearer minted-token" {
w.Header().Set("Www-Authenticate", `Bearer realm="`+upstream.URL+`/token",service="reg",scope="repo:pull"`)
w.WriteHeader(http.StatusUnauthorized)
return
}
w.Write([]byte("protected payload 2"))
case "/token":
w.Write([]byte(`{"token":"minted-token","expires_in":300}`))
case "/token-at":
w.Write([]byte(`{"access_token":"at-token"}`))
case "/token-500":
w.WriteHeader(http.StatusInternalServerError)
case "/err500":
w.WriteHeader(http.StatusInternalServerError)
case "/noauth": // 401 with an unusable challenge (no realm)
w.Header().Set("Www-Authenticate", `Bearer service="reg"`)
w.WriteHeader(http.StatusUnauthorized)
default:
http.NotFound(w, r)
}
}
func requireStack(t *testing.T) {
t.Helper()
if testEngine == nil {
t.Skip("Docker unavailable; skipping proxy engine test")
}
}
func genericRemote(name string) models.Remote {
return models.Remote{Name: name, PackageType: models.PackageGeneric, RepoType: models.RepoTypeRemote, BaseURL: upstream.URL, StaleOnError: true}
}
// seed inserts the remote so artifact rows (FK to remotes) can be stored.
func seed(t *testing.T, r models.Remote) models.Remote {
t.Helper()
rr := r
if err := testDB.CreateRemote(context.Background(), &rr); err != nil {
t.Fatalf("seed remote %s: %v", r.Name, err)
}
return r
}
func prov(t *testing.T, pt models.PackageType) provider.Provider {
p, err := provider.Get(pt)
if err != nil {
t.Fatalf("provider %s: %v", pt, err)
}
return p
}
func readAll(t *testing.T, res *FetchResult) string {
t.Helper()
defer res.Reader.Close()
b, _ := io.ReadAll(res.Reader)
return string(b)
}
func TestFetchImmutableMissThenHit(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, genericRemote("eng-imm"))
p := prov(t, models.PackageGeneric)
res, err := testEngine.Fetch(ctx, r, "blob.bin", p)
if err != nil {
t.Fatalf("fetch: %v", err)
}
if res.Source != "remote" || readAll(t, res) != "immutable blob" {
t.Errorf("miss: source=%s", res.Source)
}
res, err = testEngine.Fetch(ctx, r, "blob.bin", p)
if err != nil {
t.Fatal(err)
}
if res.Source != "cache" || readAll(t, res) != "immutable blob" {
t.Errorf("hit: source=%s", res.Source)
}
}
func TestFetchDenied(t *testing.T) {
requireStack(t)
r := genericRemote("eng-deny")
r.Blocklist = []string{`\.secret$`}
_, err := testEngine.Fetch(context.Background(), r, "x.secret", prov(t, models.PackageGeneric))
var pe *ProxyError
if err == nil || !asProxyError(err, &pe) || pe.Status != http.StatusForbidden {
t.Errorf("expected 403 ProxyError, got %v", err)
}
}
func TestHead(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, genericRemote("eng-head"))
p := prov(t, models.PackageGeneric)
// Uncached HEAD hits upstream.
h, err := testEngine.Head(ctx, r, "blob.bin", p)
if err != nil || h.Source != "remote" {
t.Fatalf("head uncached: %+v %v", h, err)
}
// Populate the cache, then HEAD should be served from metadata.
res, _ := testEngine.Fetch(ctx, r, "blob.bin", p)
res.Reader.Close()
h, err = testEngine.Head(ctx, r, "blob.bin", p)
if err != nil || h.Source != "cache" {
t.Errorf("head cached: %+v %v", h, err)
}
// Denied HEAD.
r.Blocklist = []string{".*"}
if _, err := testEngine.Head(ctx, r, "blob.bin", p); err == nil {
t.Error("expected denied head error")
}
}
func TestStaleOnError(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, genericRemote("eng-stale"))
p := prov(t, models.PackageGeneric)
if _, err := testEngine.Fetch(ctx, r, "blob.bin", p); err != nil {
t.Fatal(err)
}
// Drop cache freshness so the next fetch goes upstream, then point at a
// dead upstream: stale-on-error must serve the stored copy.
testCache.FlushRemote(ctx, "eng-stale")
r.BaseURL = "http://127.0.0.1:1"
res, err := testEngine.Fetch(ctx, r, "blob.bin", p)
if err != nil {
t.Fatalf("expected stale serve, got %v", err)
}
if res.Source != "cache" || readAll(t, res) != "immutable blob" {
t.Errorf("stale: source=%s", res.Source)
}
}
func TestCircuitOpenServesStale(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, genericRemote("eng-circuit"))
p := prov(t, models.PackageGeneric)
if _, err := testEngine.Fetch(ctx, r, "blob.bin", p); err != nil {
t.Fatal(err)
}
testCache.FlushRemote(ctx, "eng-circuit")
for i := 0; i < 6; i++ {
testEngine.circuit.RecordFailure(ctx, "eng-circuit")
}
res, err := testEngine.Fetch(ctx, r, "blob.bin", p)
if err != nil {
t.Fatalf("circuit-open should serve stale: %v", err)
}
if res.Source != "cache" {
t.Errorf("expected stale from open circuit, got %s", res.Source)
}
res.Reader.Close()
}
func TestMutableRevalidation(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, models.Remote{Name: "eng-npm", PackageType: models.PackageNPM, RepoType: models.RepoTypeRemote, BaseURL: upstream.URL, CheckMutable: true, MutableTTL: 3600, StaleOnError: true})
p := prov(t, models.PackageNPM)
res, err := testEngine.Fetch(ctx, r, "pkg", p)
if err != nil {
t.Fatalf("initial mutable fetch: %v", err)
}
res.Reader.Close()
// Expire only the freshness marker; the ETag persists, forcing a
// conditional revalidation that the upstream answers with 304.
testCache.SetTTL(ctx, "eng-npm", "pkg", time.Millisecond)
time.Sleep(10 * time.Millisecond)
res, err = testEngine.Fetch(ctx, r, "pkg", p)
if err != nil {
t.Fatalf("revalidation fetch: %v", err)
}
if res.Source != "cache" {
t.Errorf("revalidated response should come from cache, got %s", res.Source)
}
res.Reader.Close()
}
func TestBearerTokenFlow(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, genericRemote("eng-bearer"))
p := prov(t, models.PackageGeneric)
// GET: 401 challenge -> token endpoint -> retry with bearer -> 200.
res, err := testEngine.Fetch(ctx, r, "protected.bin", p)
if err != nil {
t.Fatalf("bearer fetch: %v", err)
}
if readAll(t, res) != "protected payload" {
t.Error("bearer-protected content mismatch")
}
// A second protected path with the same challenge reuses the cached token.
res2, err := testEngine.Fetch(ctx, r, "protected2.bin", p)
if err != nil {
t.Fatalf("second bearer fetch: %v", err)
}
if readAll(t, res2) != "protected payload 2" {
t.Error("second bearer content mismatch")
}
// HEAD path also negotiates a bearer token (uncached).
testCache.FlushRemote(ctx, "eng-bearer")
testDB.DeleteArtifact(ctx, "eng-bearer", "protected.bin")
if h, err := testEngine.Head(ctx, r, "protected.bin", p); err != nil || h.Source != "cache" && h.Source != "remote" {
t.Fatalf("bearer head: %+v %v", h, err)
}
}
func TestFetchUpstreamError(t *testing.T) {
requireStack(t)
r := seed(t, genericRemote("eng-404"))
// Upstream 404 (no cached copy, stale-on-error can't help) -> ProxyError.
_, err := testEngine.Fetch(context.Background(), r, "missing", prov(t, models.PackageGeneric))
var pe *ProxyError
if err == nil || !asProxyError(err, &pe) || pe.Status != http.StatusNotFound {
t.Errorf("expected 404 ProxyError, got %v", err)
}
// HEAD of a missing upstream path also errors.
if _, err := testEngine.Head(context.Background(), r, "missing", prov(t, models.PackageGeneric)); err == nil {
t.Error("expected head error for missing path")
}
}
func TestFetchUpstreamStatusErrors(t *testing.T) {
requireStack(t)
ctx := context.Background()
p := prov(t, models.PackageGeneric)
r := seed(t, genericRemote("eng-500"))
_, err := testEngine.Fetch(ctx, r, "err500", p)
var pe *ProxyError
if err == nil || !asProxyError(err, &pe) || pe.Status != http.StatusInternalServerError {
t.Errorf("expected 500 ProxyError, got %v", err)
}
r = seed(t, genericRemote("eng-noauth"))
_, err = testEngine.Fetch(ctx, r, "noauth", p)
if err == nil || !asProxyError(err, &pe) || pe.Status != http.StatusUnauthorized {
t.Errorf("expected 401 ProxyError, got %v", err)
}
}
func TestBearerTokenParsing(t *testing.T) {
// Non-Bearer challenges and missing realms are rejected.
if _, _, err := fetchBearerToken(context.Background(), "Basic realm=x", models.Remote{}); err == nil {
t.Error("expected error for non-Bearer challenge")
}
if _, _, err := fetchBearerToken(context.Background(), `Bearer service="reg"`, models.Remote{}); err == nil {
t.Error("expected error for missing realm")
}
}
func TestWaitForStoreCoalesces(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, genericRemote("eng-herd"))
p := prov(t, models.PackageGeneric)
// Fire concurrent cold-cache fetches: only one holds the lock, the others
// wait on the store (waitForStore) and pick up the result.
const n = 4
done := make(chan string, n)
for i := 0; i < n; i++ {
go func() {
res, err := testEngine.Fetch(ctx, r, "blob.bin", p)
if err != nil {
done <- "err:" + err.Error()
return
}
done <- readAll(t, res)
}()
}
for i := 0; i < n; i++ {
if got := <-done; got != "immutable blob" {
t.Errorf("concurrent fetch got %q", got)
}
}
}
func TestRevalidationUpstreamError(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, models.Remote{Name: "eng-reval-err", PackageType: models.PackageNPM, RepoType: models.RepoTypeRemote, BaseURL: upstream.URL, CheckMutable: true, MutableTTL: 3600, StaleOnError: true})
p := prov(t, models.PackageNPM)
res, err := testEngine.Fetch(ctx, r, "pkg", p)
if err != nil {
t.Fatalf("initial fetch: %v", err)
}
res.Reader.Close()
// Expire freshness but keep the ETag, then break the upstream: the
// conditional HEAD (checkUpstream) errors, and stale-on-error serves the
// stored index.
testCache.SetTTL(ctx, "eng-reval-err", "pkg", time.Millisecond)
time.Sleep(10 * time.Millisecond)
r.BaseURL = "http://127.0.0.1:1"
res, err = testEngine.Fetch(ctx, r, "pkg", p)
if err != nil {
t.Fatalf("expected stale serve on revalidation error, got %v", err)
}
if res.Source != "cache" {
t.Errorf("expected stale cache source, got %s", res.Source)
}
res.Reader.Close()
}
func TestTTLFor(t *testing.T) {
e := &Engine{}
if got := e.ttlFor(models.Remote{ImmutableTTL: 100}, ClassImmutable); got != 100*time.Second {
t.Errorf("immutable ttl = %v", got)
}
if got := e.ttlFor(models.Remote{ImmutableTTL: 0}, ClassImmutable); got != 0 {
t.Errorf("immutable ttl=0 (forever) = %v", got)
}
if got := e.ttlFor(models.Remote{MutableTTL: 50}, ClassMutable); got != 50*time.Second {
t.Errorf("mutable ttl = %v", got)
}
}
func TestHeadUpstreamStatusError(t *testing.T) {
requireStack(t)
r := seed(t, genericRemote("eng-head500"))
if _, err := testEngine.Head(context.Background(), r, "err500", prov(t, models.PackageGeneric)); err == nil {
t.Error("expected error for HEAD of 500 upstream")
}
}
func TestHeadCachedIndex(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := seed(t, models.Remote{Name: "eng-headidx", PackageType: models.PackageNPM, RepoType: models.RepoTypeRemote, BaseURL: upstream.URL, CheckMutable: true, MutableTTL: 3600})
p := prov(t, models.PackageNPM)
// Cache the mutable index, then HEAD is answered from the stored index.
res, err := testEngine.Fetch(ctx, r, "pkg", p)
if err != nil {
t.Fatal(err)
}
res.Reader.Close()
h, err := testEngine.Head(ctx, r, "pkg", p)
if err != nil || h.Source != "cache" {
t.Errorf("head of cached index: %+v %v", h, err)
}
}
func TestFetchBearerTokenVariants(t *testing.T) {
requireStack(t)
ctx := context.Background()
// access_token field + service/scope params + basic auth on the token req.
tok, _, err := fetchBearerToken(ctx, `Bearer realm="`+upstream.URL+`/token-at",service="reg",scope="repo:pull"`, models.Remote{Username: "u", Password: "p"})
if err != nil || tok != "at-token" {
t.Errorf("access_token variant: tok=%q err=%v", tok, err)
}
// Token endpoint error status.
if _, _, err := fetchBearerToken(ctx, `Bearer realm="`+upstream.URL+`/token-500"`, models.Remote{}); err == nil {
t.Error("expected error for 500 token endpoint")
}
}
func TestCheckUpstreamChanged(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := genericRemote("eng-check")
// A non-matching ETag yields a normal 200 (not 304): not modified is false.
notModified, err := testEngine.checkUpstream(ctx, r, "pkg", `"stale-etag"`, prov(t, models.PackageNPM))
if err != nil {
t.Fatalf("checkUpstream: %v", err)
}
if notModified {
t.Error("mismatched etag should report modified (notModified=false)")
}
}
func TestUpstreamErrorUnwrap(t *testing.T) {
base := context.DeadlineExceeded
ue := &UpstreamError{Err: base}
if ue.Unwrap() != base {
t.Error("Unwrap should return the wrapped error")
}
if !isNetworkError(ue) {
t.Error("UpstreamError should be a network error")
}
if isNetworkError(context.Canceled) {
t.Error("plain error should not be a network error")
}
}
func TestImmutableBlobDedup(t *testing.T) {
requireStack(t)
ctx := context.Background()
p := prov(t, models.PackageGeneric)
// Two remotes serving identical content: the second store hits the
// already-exists branch (blob content is deduplicated).
for _, name := range []string{"eng-dedup-a", "eng-dedup-b"} {
r := seed(t, genericRemote(name))
res, err := testEngine.Fetch(ctx, r, "blob.bin", p)
if err != nil {
t.Fatalf("%s fetch: %v", name, err)
}
if readAll(t, res) != "immutable blob" {
t.Errorf("%s content mismatch", name)
}
}
}
func TestCircuitBreakerStates(t *testing.T) {
requireStack(t)
ctx := context.Background()
cb := NewCircuitBreaker(testCache)
const key = "cb-states"
testCache.ResetCircuit(ctx, key)
if cb.IsOpen(ctx, key) {
t.Error("fresh breaker should be closed")
}
if cb.Health(ctx, key).Status != "healthy" {
t.Error("fresh breaker should be healthy")
}
cb.RecordFailure(ctx, key)
if s := cb.Health(ctx, key).Status; s != "degraded" {
t.Errorf("one failure should be degraded, got %q", s)
}
for i := 0; i < 6; i++ {
cb.RecordFailure(ctx, key)
}
if !cb.IsOpen(ctx, key) {
t.Error("breaker should be open after threshold failures")
}
if s := cb.Health(ctx, key).Status; s != "down" {
t.Errorf("open breaker should be down, got %q", s)
}
cb.RecordSuccess(ctx, key)
if cb.IsOpen(ctx, key) {
t.Error("breaker should close after success")
}
}
func asProxyError(err error, target **ProxyError) bool {
pe, ok := err.(*ProxyError)
if ok {
*target = pe
}
return ok
}
+83
View File
@@ -0,0 +1,83 @@
package proxy
import (
"net"
"net/http"
"sync"
"time"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// Default upstream timeouts. A remote may override any of these; a zero
// override falls back to the default here. There is deliberately no overall
// Client.Timeout: the proxy streams arbitrarily large artifacts and total time
// is bounded by the request context instead. We only constrain the phases that
// must never hang — connect, TLS handshake, and time-to-first-response-header —
// so a slow or wedged upstream cannot pin a goroutine and connection.
const (
defaultDialTimeout = 10 * time.Second
defaultTLSTimeout = 10 * time.Second
defaultResponseHeaderTimeout = 30 * time.Second
)
type clientKey struct {
dial time.Duration
tls time.Duration
respHeader time.Duration
}
var (
clientCacheMu sync.Mutex
clientCache = map[clientKey]*http.Client{}
)
// upstreamClientFor returns an HTTP client configured with the given timeouts,
// reusing a cached client (and its connection pool) for identical timeout sets.
// Zero values fall back to the defaults.
func upstreamClientFor(dial, tls, respHeader time.Duration) *http.Client {
if dial <= 0 {
dial = defaultDialTimeout
}
if tls <= 0 {
tls = defaultTLSTimeout
}
if respHeader <= 0 {
respHeader = defaultResponseHeaderTimeout
}
key := clientKey{dial: dial, tls: tls, respHeader: respHeader}
clientCacheMu.Lock()
defer clientCacheMu.Unlock()
if c, ok := clientCache[key]; ok {
return c
}
c := &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{
Timeout: dial,
KeepAlive: 30 * time.Second,
}).DialContext,
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: tls,
ExpectContinueTimeout: 1 * time.Second,
ResponseHeaderTimeout: respHeader,
},
}
clientCache[key] = c
return c
}
// clientForRemote returns the upstream client for a remote, applying its
// per-remote timeout overrides (in seconds) on top of the defaults.
func clientForRemote(remote models.Remote) *http.Client {
return upstreamClientFor(
time.Duration(remote.UpstreamDialTimeout)*time.Second,
time.Duration(remote.UpstreamTLSTimeout)*time.Second,
time.Duration(remote.UpstreamResponseHeaderTimeout)*time.Second,
)
}
+18 -2
View File
@@ -35,6 +35,7 @@ import (
type Server struct { type Server struct {
cfg *config.Config cfg *config.Config
version string
router chi.Router router chi.Router
db *database.DB db *database.DB
cache *cache.Redis cache *cache.Redis
@@ -45,7 +46,7 @@ type Server struct {
gc *gc.Collector gc *gc.Collector
} }
func New(cfg *config.Config) (*Server, error) { func New(cfg *config.Config, version string) (*Server, error) {
db, err := database.New(cfg.DatabaseDSN()) db, err := database.New(cfg.DatabaseDSN())
if err != nil { if err != nil {
return nil, fmt.Errorf("database: %w", err) return nil, fmt.Errorf("database: %w", err)
@@ -68,6 +69,7 @@ func New(cfg *config.Config) (*Server, error) {
s := &Server{ s := &Server{
cfg: cfg, cfg: cfg,
version: version,
db: db, db: db,
cache: redis, cache: redis,
store: s3, store: s3,
@@ -93,9 +95,11 @@ func (s *Server) routes() chi.Router {
r.Get("/health", s.handleHealth) r.Get("/health", s.handleHealth)
r.Get("/", s.handleRoot) r.Get("/", s.handleRoot)
r.Get("/version", s.handleVersion)
proxyHandler := v1.NewProxyHandler(s.engine, s.virtEngine, s.db, s.store, s.localHandler) proxyHandler := v1.NewProxyHandler(s.engine, s.virtEngine, s.db, s.store, s.localHandler)
r.Mount("/api/v1", proxyHandler.Routes()) r.Mount("/api/v1", proxyHandler.Routes())
r.Mount("/v2", proxyHandler.DockerV2Routes())
remotesHandler := v2.NewRemotesHandler(s.db) remotesHandler := v2.NewRemotesHandler(s.db)
virtualsHandler := v2.NewVirtualsHandler(s.db) virtualsHandler := v2.NewVirtualsHandler(s.db)
@@ -118,6 +122,12 @@ func (s *Server) routes() chi.Router {
r.Delete("/*", objHandler.Routes().ServeHTTP) r.Delete("/*", objHandler.Routes().ServeHTTP)
}) })
r.Route("/locals/{name}/objects", func(r chi.Router) {
objHandler := v2.NewObjectsHandler(s.db)
r.Get("/", objHandler.LocalRoutes().ServeHTTP)
r.Delete("/*", objHandler.LocalRoutes().ServeHTTP)
})
r.Route("/remotes/{name}/files", func(r chi.Router) { r.Route("/remotes/{name}/files", func(r chi.Router) {
r.Put("/*", s.localHandler.Routes().ServeHTTP) r.Put("/*", s.localHandler.Routes().ServeHTTP)
r.Get("/*", s.localHandler.Routes().ServeHTTP) r.Get("/*", s.localHandler.Routes().ServeHTTP)
@@ -134,10 +144,16 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, `{"status":"ok"}`) fmt.Fprint(w, `{"status":"ok"}`)
} }
// handleRoot sends browsers landing on the bare domain to the web UI, which is
// served under /ui. The service identity that used to live here is at /version.
func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) { func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ui/", http.StatusFound)
}
func (s *Server) handleVersion(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
fmt.Fprint(w, `{"name":"artifactapi","version":"3.0.0-dev"}`) fmt.Fprintf(w, `{"name":"artifactapi","version":"%s"}`, s.version)
} }
func (s *Server) newHTTPServer() *http.Server { func (s *Server) newHTTPServer() *http.Server {
+639
View File
@@ -0,0 +1,639 @@
package server
import (
"bytes"
"context"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"strconv"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/config"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
var (
testTS *httptest.Server // the artifactapi router
upstream *httptest.Server // mock upstream the proxy fetches from
testSrv *Server
)
func TestMain(m *testing.M) {
ctx := context.Background()
dsn, termPG, err := testsupport.StartPostgres(ctx)
if err != nil {
os.Exit(m.Run())
}
defer termPG()
redisURL, termRedis, err := testsupport.StartRedis(ctx)
if err != nil {
termPG()
os.Exit(m.Run())
}
defer termRedis()
minio, termMinio, err := testsupport.StartMinio(ctx)
if err != nil {
termPG()
termRedis()
os.Exit(m.Run())
}
defer termMinio()
u, _ := url.Parse(dsn)
port, _ := strconv.Atoi(u.Port())
cfg := &config.Config{
ListenAddr: ":0",
DBHost: u.Hostname(),
DBPort: port,
DBUser: "artifacts",
DBPass: "artifacts123",
DBName: "artifacts",
DBSSL: "disable",
RedisURL: redisURL,
S3Endpoint: minio.Endpoint,
S3AccessKey: minio.AccessKey,
S3SecretKey: minio.SecretKey,
S3Bucket: "server-test",
}
var srv *Server
for i := 0; i < 20; i++ { // tolerate MinIO reporting ready before bucket ops succeed
if srv, err = New(cfg, "test-version"); err == nil {
break
}
time.Sleep(500 * time.Millisecond)
}
if err != nil {
panic(err)
}
testSrv = srv
testTS = httptest.NewServer(srv.router)
upstream = httptest.NewServer(http.HandlerFunc(mockUpstream))
code := m.Run()
testTS.Close()
upstream.Close()
termMinio()
termRedis()
termPG()
if code != 0 {
os.Exit(code)
}
}
func mockUpstream(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/data/file.bin":
w.Write([]byte("upstream blob payload"))
case "/helm-a/index.yaml":
w.Write([]byte("apiVersion: v1\nentries:\n alpha:\n - name: alpha\n version: 1.0.0\n urls: [charts/alpha-1.0.0.tgz]\n"))
case "/helm-b/index.yaml":
w.Write([]byte("apiVersion: v1\nentries:\n beta:\n - name: beta\n version: 2.0.0\n urls: [charts/beta-2.0.0.tgz]\n"))
default:
http.NotFound(w, r)
}
}
func requireStack(t *testing.T) {
t.Helper()
if testTS == nil {
t.Skip("Docker unavailable; skipping server integration test")
}
}
func req(t *testing.T, method, path string, body string) (*http.Response, []byte) {
t.Helper()
var r io.Reader
if body != "" {
r = strings.NewReader(body)
}
rq, _ := http.NewRequest(method, testTS.URL+path, r)
if body != "" {
rq.Header.Set("Content-Type", "application/json")
}
resp, err := http.DefaultClient.Do(rq)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
return resp, b
}
// reqNoRedirect issues a request without following redirects so the response's
// status and Location header can be asserted directly.
func reqNoRedirect(t *testing.T, method, path string) *http.Response {
t.Helper()
rq, _ := http.NewRequest(method, testTS.URL+path, nil)
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}}
resp, err := client.Do(rq)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
resp.Body.Close()
return resp
}
func TestServerHealthAndRoot(t *testing.T) {
requireStack(t)
if resp, _ := req(t, "GET", "/health", ""); resp.StatusCode != 200 {
t.Errorf("health: %d", resp.StatusCode)
}
if resp := reqNoRedirect(t, "GET", "/"); resp.StatusCode != http.StatusFound || resp.Header.Get("Location") != "/ui/" {
t.Errorf("root redirect: %d %q", resp.StatusCode, resp.Header.Get("Location"))
}
if resp, b := req(t, "GET", "/version", ""); resp.StatusCode != 200 || !strings.Contains(string(b), "test-version") {
t.Errorf("version: %d %s", resp.StatusCode, b)
}
if resp, _ := req(t, "GET", "/api/v2/health", ""); resp.StatusCode != 200 {
t.Errorf("health v2: %d", resp.StatusCode)
}
}
func TestServerRemoteAndProxy(t *testing.T) {
requireStack(t)
create := fmt.Sprintf(`{"name":"srv-remote","package_type":"generic","repo_type":"remote","base_url":%q,"stale_on_error":true}`, upstream.URL)
if resp, b := req(t, "POST", "/api/v2/remotes", create); resp.StatusCode != 201 {
t.Fatalf("create remote: %d %s", resp.StatusCode, b)
}
defer req(t, "DELETE", "/api/v2/remotes/srv-remote", "")
if resp, _ := req(t, "GET", "/api/v2/remotes/srv-remote", ""); resp.StatusCode != 200 {
t.Errorf("get remote: %d", resp.StatusCode)
}
if resp, _ := req(t, "GET", "/api/v2/remotes", ""); resp.StatusCode != 200 {
t.Errorf("list remotes: %d", resp.StatusCode)
}
// Proxy fetch: miss then hit.
resp, b := req(t, "GET", "/api/v1/remote/srv-remote/data/file.bin", "")
if resp.StatusCode != 200 || string(b) != "upstream blob payload" {
t.Fatalf("proxy miss: %d %s", resp.StatusCode, b)
}
if src := resp.Header.Get("X-Artifact-Source"); src != "remote" {
t.Errorf("expected remote source, got %q", src)
}
resp, _ = req(t, "GET", "/api/v1/remote/srv-remote/data/file.bin", "")
if resp.Header.Get("X-Artifact-Source") != "cache" {
t.Errorf("second fetch should be cache: %q", resp.Header.Get("X-Artifact-Source"))
}
// Objects listing + stats now that we have an artifact.
if resp, _ := req(t, "GET", "/api/v2/remotes/srv-remote/objects", ""); resp.StatusCode != 200 {
t.Errorf("objects: %d", resp.StatusCode)
}
if resp, _ := req(t, "GET", "/api/v2/stats", ""); resp.StatusCode != 200 {
t.Errorf("stats: %d", resp.StatusCode)
}
for _, p := range []string{"/api/v2/stats/top-remotes", "/api/v2/stats/top-files-by-hits", "/api/v2/stats/top-files-by-bandwidth"} {
if resp, _ := req(t, "GET", p, ""); resp.StatusCode != 200 {
t.Errorf("%s: %d", p, resp.StatusCode)
}
}
}
func TestServerLocalUpload(t *testing.T) {
requireStack(t)
if resp, b := req(t, "POST", "/api/v2/remotes", `{"name":"srv-local","package_type":"generic","repo_type":"local"}`); resp.StatusCode != 201 {
t.Fatalf("create local: %d %s", resp.StatusCode, b)
}
defer req(t, "DELETE", "/api/v2/remotes/srv-local", "")
rq, _ := http.NewRequest("PUT", testTS.URL+"/api/v2/remotes/srv-local/files/dir/hello.bin", strings.NewReader("local payload"))
rq.Header.Set("Content-Type", "text/plain") // exercise the content-type branch
resp, err := http.DefaultClient.Do(rq)
if err != nil || resp.StatusCode != 201 {
t.Fatalf("upload: %v %d", err, resp.StatusCode)
}
resp.Body.Close()
resp, b := req(t, "GET", "/api/v1/local/srv-local/dir/hello.bin", "")
if resp.StatusCode != 200 || string(b) != "local payload" {
t.Errorf("download local: %d %s", resp.StatusCode, b)
}
// Also download via the v2 files endpoint.
if resp, b := req(t, "GET", "/api/v2/remotes/srv-local/files/dir/hello.bin", ""); resp.StatusCode != 200 || string(b) != "local payload" {
t.Errorf("v2 download: %d %s", resp.StatusCode, b)
}
}
func TestServerVirtualMerge(t *testing.T) {
requireStack(t)
for _, m := range []string{"a", "b"} {
body := fmt.Sprintf(`{"name":"srv-helm-%s","package_type":"helm","repo_type":"remote","base_url":"%s/helm-%s","stale_on_error":true}`, m, upstream.URL, m)
if resp, b := req(t, "POST", "/api/v2/remotes", body); resp.StatusCode != 201 {
t.Fatalf("create helm-%s: %d %s", m, resp.StatusCode, b)
}
defer req(t, "DELETE", "/api/v2/remotes/srv-helm-"+m, "")
}
if resp, b := req(t, "POST", "/api/v2/virtuals", `{"name":"srv-vh","package_type":"helm","members":["srv-helm-a","srv-helm-b"]}`); resp.StatusCode != 201 {
t.Fatalf("create virtual: %d %s", resp.StatusCode, b)
}
defer req(t, "DELETE", "/api/v2/virtuals/srv-vh", "")
resp, b := req(t, "GET", "/api/v1/virtual/srv-vh/index.yaml", "")
if resp.StatusCode != 200 {
t.Fatalf("virtual fetch: %d %s", resp.StatusCode, b)
}
s := string(b)
if !strings.Contains(s, "alpha") || !strings.Contains(s, "beta") {
t.Errorf("merged index missing charts: %s", s)
}
}
func TestServerProbe(t *testing.T) {
requireStack(t)
create := fmt.Sprintf(`{"name":"srv-probe","package_type":"generic","repo_type":"remote","base_url":%q,"stale_on_error":true}`, upstream.URL)
req(t, "POST", "/api/v2/remotes", create)
defer req(t, "DELETE", "/api/v2/remotes/srv-probe", "")
// Reachable path -> status 200 in the probe body.
if resp, b := req(t, "POST", "/api/v2/probe", `{"remote":"srv-probe","path":"data/file.bin"}`); resp.StatusCode != 200 || !strings.Contains(string(b), `"status":200`) {
t.Errorf("probe reachable: %d %s", resp.StatusCode, b)
}
// Missing upstream path -> upstream error reported (502) in the body.
if resp, b := req(t, "POST", "/api/v2/probe", `{"remote":"srv-probe","path":"missing"}`); resp.StatusCode != 200 || !strings.Contains(string(b), `"status":502`) {
t.Errorf("probe missing: %d %s", resp.StatusCode, b)
}
// Unknown remote -> 404 in the body.
if resp, b := req(t, "POST", "/api/v2/probe", `{"remote":"nope","path":"x"}`); resp.StatusCode != 200 || !strings.Contains(string(b), `"status":404`) {
t.Errorf("probe unknown: %d %s", resp.StatusCode, b)
}
// Bad requests.
if resp, _ := req(t, "POST", "/api/v2/probe", `{}`); resp.StatusCode != 400 {
t.Errorf("probe missing fields: %d", resp.StatusCode)
}
if resp, _ := req(t, "POST", "/api/v2/probe", `not json`); resp.StatusCode != 400 {
t.Errorf("probe invalid json: %d", resp.StatusCode)
}
}
func put(t *testing.T, path string, body []byte) (*http.Response, []byte) {
t.Helper()
rq, _ := http.NewRequest("PUT", testTS.URL+path, bytes.NewReader(body))
resp, err := http.DefaultClient.Do(rq)
if err != nil {
t.Fatalf("PUT %s: %v", path, err)
}
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
return resp, b
}
func TestServerLocalPyPI(t *testing.T) {
requireStack(t)
if resp, b := req(t, "POST", "/api/v2/remotes", `{"name":"srv-pypi","package_type":"pypi","repo_type":"local"}`); resp.StatusCode != 201 {
t.Fatalf("create pypi local: %d %s", resp.StatusCode, b)
}
defer req(t, "DELETE", "/api/v2/remotes/srv-pypi", "")
if resp, b := put(t, "/api/v2/remotes/srv-pypi/files/foo-1.0-py3-none-any.whl", []byte("wheel bytes")); resp.StatusCode != 201 {
t.Fatalf("upload wheel: %d %s", resp.StatusCode, b)
}
// Re-uploading the same file is rejected.
if resp, _ := put(t, "/api/v2/remotes/srv-pypi/files/foo-1.0-py3-none-any.whl", []byte("again")); resp.StatusCode != 409 {
t.Errorf("expected 409 on overwrite, got %d", resp.StatusCode)
}
// Invalid pypi filename rejected.
if resp, _ := put(t, "/api/v2/remotes/srv-pypi/files/not-a-package.txt", []byte("x")); resp.StatusCode != 400 {
t.Errorf("expected 400 for bad filename, got %d", resp.StatusCode)
}
if resp, b := req(t, "GET", "/api/v1/local/srv-pypi/simple/", ""); resp.StatusCode != 200 || !strings.Contains(string(b), "foo") {
t.Errorf("simple index: %d %s", resp.StatusCode, b)
}
if resp, b := req(t, "GET", "/api/v1/local/srv-pypi/simple/foo/", ""); resp.StatusCode != 200 || !strings.Contains(string(b), "foo-1.0-py3-none-any.whl") {
t.Errorf("package index: %d %s", resp.StatusCode, b)
}
}
func TestServerLocalRPMRepodata(t *testing.T) {
requireStack(t)
rpm := testsupport.MinimalRPM("e2e-testpkg", "1.0", "1", "noarch")
if resp, b := req(t, "POST", "/api/v2/remotes", `{"name":"srv-rpm","package_type":"rpm","repo_type":"local"}`); resp.StatusCode != 201 {
t.Fatalf("create rpm local: %d %s", resp.StatusCode, b)
}
defer req(t, "DELETE", "/api/v2/remotes/srv-rpm", "")
if resp, b := put(t, "/api/v2/remotes/srv-rpm/files/e2e-testpkg-1.0-1.noarch.rpm", rpm); resp.StatusCode != 201 {
t.Fatalf("upload rpm: %d %s", resp.StatusCode, b)
}
// repodata is generated asynchronously; poll for it.
var body []byte
for i := 0; i < 40; i++ {
var resp *http.Response
resp, body = req(t, "GET", "/api/v1/local/srv-rpm/repodata/repomd.xml", "")
if resp.StatusCode == 200 && strings.Contains(string(body), "<repomd") {
return
}
time.Sleep(250 * time.Millisecond)
}
t.Errorf("repomd.xml not generated: %s", body)
}
func TestServerObjectEviction(t *testing.T) {
requireStack(t)
create := fmt.Sprintf(`{"name":"srv-evict","package_type":"generic","repo_type":"remote","base_url":%q,"stale_on_error":true}`, upstream.URL)
req(t, "POST", "/api/v2/remotes", create)
defer req(t, "DELETE", "/api/v2/remotes/srv-evict", "")
resp, _ := req(t, "GET", "/api/v1/remote/srv-evict/data/file.bin", "")
resp.Body.Close()
if resp, _ := req(t, "DELETE", "/api/v2/remotes/srv-evict/objects/data/file.bin", ""); resp.StatusCode >= 400 {
t.Errorf("evict object: %d", resp.StatusCode)
}
}
func TestServerValidationErrors(t *testing.T) {
requireStack(t)
if resp, _ := req(t, "POST", "/api/v2/remotes", `{"name":"bad","package_type":"bogus","base_url":"https://x"}`); resp.StatusCode != 400 {
t.Errorf("invalid package type: %d", resp.StatusCode)
}
if resp, _ := req(t, "POST", "/api/v2/remotes", `{"name":"bad","package_type":"generic","repo_type":"remote"}`); resp.StatusCode != 400 {
t.Errorf("missing base_url: %d", resp.StatusCode)
}
if resp, _ := req(t, "POST", "/api/v2/remotes", `not json`); resp.StatusCode != 400 {
t.Errorf("invalid json: %d", resp.StatusCode)
}
// Invalid regex pattern -> 400 from ValidatePatterns.
if resp, _ := req(t, "POST", "/api/v2/remotes", `{"name":"badre","package_type":"generic","repo_type":"remote","base_url":"https://x","blocklist":["[unterminated"]}`); resp.StatusCode != 400 {
t.Errorf("invalid regex: %d", resp.StatusCode)
}
}
func TestServerDockerAndHead(t *testing.T) {
requireStack(t)
create := fmt.Sprintf(`{"name":"srv-docker","package_type":"generic","repo_type":"remote","base_url":%q,"stale_on_error":true}`, upstream.URL)
req(t, "POST", "/api/v2/remotes", create)
defer req(t, "DELETE", "/api/v2/remotes/srv-docker", "")
// Docker registry ping.
if resp, _ := req(t, "GET", "/v2/", ""); resp.StatusCode != 200 {
t.Errorf("docker ping: %d", resp.StatusCode)
}
// HEAD through the docker route resolves metadata (uncached -> upstream).
rq, _ := http.NewRequest("HEAD", testTS.URL+"/v2/srv-docker/data/file.bin", nil)
resp, err := http.DefaultClient.Do(rq)
if err != nil {
t.Fatalf("head: %v", err)
}
resp.Body.Close()
if resp.StatusCode != 200 {
t.Errorf("head status: %d", resp.StatusCode)
}
}
func TestServerRemoteUpdateAndVirtualCRUD(t *testing.T) {
requireStack(t)
req(t, "POST", "/api/v2/remotes", `{"name":"srv-upd","package_type":"helm","repo_type":"remote","base_url":"https://a.example.com","stale_on_error":true}`)
defer req(t, "DELETE", "/api/v2/remotes/srv-upd", "")
if resp, b := req(t, "PUT", "/api/v2/remotes/srv-upd", `{"package_type":"helm","base_url":"https://b.example.com","stale_on_error":true}`); resp.StatusCode != 200 {
t.Errorf("update remote: %d %s", resp.StatusCode, b)
}
req(t, "POST", "/api/v2/virtuals", `{"name":"srv-v2","package_type":"helm","members":["srv-upd"]}`)
defer req(t, "DELETE", "/api/v2/virtuals/srv-v2", "")
if resp, _ := req(t, "GET", "/api/v2/virtuals/srv-v2", ""); resp.StatusCode != 200 {
t.Errorf("get virtual: %d", resp.StatusCode)
}
if resp, _ := req(t, "GET", "/api/v2/virtuals", ""); resp.StatusCode != 200 {
t.Errorf("list virtuals: %d", resp.StatusCode)
}
if resp, b := req(t, "PUT", "/api/v2/virtuals/srv-v2", `{"package_type":"helm","members":["srv-upd"]}`); resp.StatusCode != 200 {
t.Errorf("update virtual: %d %s", resp.StatusCode, b)
}
}
func TestServerLocalRemoveAndMissing(t *testing.T) {
requireStack(t)
req(t, "POST", "/api/v2/remotes", `{"name":"srv-rm","package_type":"generic","repo_type":"local"}`)
defer req(t, "DELETE", "/api/v2/remotes/srv-rm", "")
put(t, "/api/v2/remotes/srv-rm/files/a/b.bin", []byte("payload"))
if resp, _ := req(t, "DELETE", "/api/v2/remotes/srv-rm/files/a/b.bin", ""); resp.StatusCode >= 400 {
t.Errorf("delete local file: %d", resp.StatusCode)
}
if resp, _ := req(t, "GET", "/api/v1/local/srv-rm/a/b.bin", ""); resp.StatusCode != 404 {
t.Errorf("expected 404 for removed file, got %d", resp.StatusCode)
}
}
func TestServerLocalUploadErrors(t *testing.T) {
requireStack(t)
// Uploading to a remote-type repo is rejected.
create := fmt.Sprintf(`{"name":"srv-uerr","package_type":"generic","repo_type":"remote","base_url":%q,"stale_on_error":true}`, upstream.URL)
req(t, "POST", "/api/v2/remotes", create)
defer req(t, "DELETE", "/api/v2/remotes/srv-uerr", "")
if resp, _ := put(t, "/api/v2/remotes/srv-uerr/files/x.bin", []byte("x")); resp.StatusCode != 400 {
t.Errorf("upload to remote repo should be 400, got %d", resp.StatusCode)
}
// Duplicate generic upload is a conflict.
req(t, "POST", "/api/v2/remotes", `{"name":"srv-dup","package_type":"generic","repo_type":"local"}`)
defer req(t, "DELETE", "/api/v2/remotes/srv-dup", "")
put(t, "/api/v2/remotes/srv-dup/files/dup.bin", []byte("one"))
if resp, _ := put(t, "/api/v2/remotes/srv-dup/files/dup.bin", []byte("two")); resp.StatusCode != 409 {
t.Errorf("duplicate upload should be 409, got %d", resp.StatusCode)
}
// Download of a missing local file is 404.
if resp, _ := req(t, "GET", "/api/v1/local/srv-dup/does/not/exist", ""); resp.StatusCode != 404 {
t.Errorf("missing local download should be 404, got %d", resp.StatusCode)
}
// Unknown virtual is 404.
if resp, _ := req(t, "GET", "/api/v1/virtual/nope/index.yaml", ""); resp.StatusCode != 404 {
t.Errorf("unknown virtual should be 404, got %d", resp.StatusCode)
}
}
func TestServerEvents(t *testing.T) {
requireStack(t)
client := &http.Client{Timeout: 800 * time.Millisecond}
resp, err := client.Get(testTS.URL + "/api/v2/events")
if err == nil {
resp.Body.Close()
if resp.StatusCode != 200 {
t.Errorf("events status: %d", resp.StatusCode)
}
}
// A timeout is expected for a streaming endpoint; the handler still ran.
}
func TestRunOnListener(t *testing.T) {
requireStack(t)
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
errc := make(chan error, 1)
go func() { errc <- testSrv.RunOnListener(ctx, ln) }()
base := "http://" + ln.Addr().String()
ok := false
for i := 0; i < 50; i++ {
if resp, e := http.Get(base + "/health"); e == nil {
resp.Body.Close()
ok = resp.StatusCode == 200
break
}
time.Sleep(20 * time.Millisecond)
}
if !ok {
t.Error("server did not serve /health")
}
cancel()
select {
case err := <-errc:
if err != nil {
t.Errorf("RunOnListener returned error: %v", err)
}
case <-time.After(12 * time.Second):
t.Fatal("RunOnListener did not shut down")
}
}
func TestRun(t *testing.T) {
requireStack(t)
ctx, cancel := context.WithCancel(context.Background())
errc := make(chan error, 1)
go func() { errc <- testSrv.Run(ctx) }()
time.Sleep(300 * time.Millisecond) // let it bind and start serving
cancel()
select {
case err := <-errc:
if err != nil {
t.Errorf("Run returned error: %v", err)
}
case <-time.After(12 * time.Second):
t.Fatal("Run did not shut down")
}
}
func TestServerVirtualUnreachableMembers(t *testing.T) {
requireStack(t)
// A virtual whose only member does not exist -> no members reachable.
req(t, "POST", "/api/v2/virtuals", `{"name":"srv-vbad","package_type":"helm","members":["nonexistent-member"]}`)
defer req(t, "DELETE", "/api/v2/virtuals/srv-vbad", "")
if resp, _ := req(t, "GET", "/api/v1/virtual/srv-vbad/index.yaml", ""); resp.StatusCode != 502 {
t.Errorf("virtual with dead members = %d, want 502", resp.StatusCode)
}
}
func TestServerVirtualLocalPyPIMerge(t *testing.T) {
requireStack(t)
for _, n := range []string{"a", "b"} {
req(t, "POST", "/api/v2/remotes", `{"name":"srv-pm-`+n+`","package_type":"pypi","repo_type":"local"}`)
defer req(t, "DELETE", "/api/v2/remotes/srv-pm-"+n, "")
}
put(t, "/api/v2/remotes/srv-pm-a/files/foo-1.0-py3-none-any.whl", []byte("foo"))
put(t, "/api/v2/remotes/srv-pm-b/files/bar-2.0-py3-none-any.whl", []byte("bar"))
req(t, "POST", "/api/v2/virtuals", `{"name":"srv-pmv","package_type":"pypi","members":["srv-pm-a","srv-pm-b"]}`)
defer req(t, "DELETE", "/api/v2/virtuals/srv-pmv", "")
resp, b := req(t, "GET", "/api/v1/virtual/srv-pmv/simple/", "")
if resp.StatusCode != 200 {
t.Fatalf("virtual pypi index: %d %s", resp.StatusCode, b)
}
if s := string(b); !strings.Contains(s, "foo") || !strings.Contains(s, "bar") {
t.Errorf("merged local pypi index missing packages: %s", s)
}
}
func TestServerProxyErrors(t *testing.T) {
requireStack(t)
// Blocklisted path -> 403 propagated through handleProxy.
block := fmt.Sprintf(`{"name":"srv-block","package_type":"generic","repo_type":"remote","base_url":%q,"blocklist":["\\.secret$"],"stale_on_error":true}`, upstream.URL)
req(t, "POST", "/api/v2/remotes", block)
defer req(t, "DELETE", "/api/v2/remotes/srv-block", "")
if resp, _ := req(t, "GET", "/api/v1/remote/srv-block/x.secret", ""); resp.StatusCode != 403 {
t.Errorf("blocklisted GET = %d, want 403", resp.StatusCode)
}
rq, _ := http.NewRequest("HEAD", testTS.URL+"/v2/srv-block/x.secret", nil)
if resp, err := http.DefaultClient.Do(rq); err == nil {
resp.Body.Close()
if resp.StatusCode != 403 {
t.Errorf("blocklisted HEAD = %d, want 403", resp.StatusCode)
}
}
// Unreachable upstream, no stale copy -> 502 bad gateway.
dead := `{"name":"srv-dead","package_type":"generic","repo_type":"remote","base_url":"http://127.0.0.1:1","stale_on_error":false}`
req(t, "POST", "/api/v2/remotes", dead)
defer req(t, "DELETE", "/api/v2/remotes/srv-dead", "")
if resp, _ := req(t, "GET", "/api/v1/remote/srv-dead/x", ""); resp.StatusCode != 502 {
t.Errorf("dead upstream GET = %d, want 502", resp.StatusCode)
}
}
func TestServerLocalMissingBlob(t *testing.T) {
requireStack(t)
req(t, "POST", "/api/v2/remotes", `{"name":"srv-ghost","package_type":"generic","repo_type":"local"}`)
defer req(t, "DELETE", "/api/v2/remotes/srv-ghost", "")
ctx := context.Background()
// A local file whose blob object is absent from the store.
testSrv.db.UpsertBlob(ctx, "sha256:ghost", "blobs/sha256/ghost-missing", 5, "text/plain")
if err := testSrv.db.CreateLocalFile(ctx, "srv-ghost", "ghost.bin", "sha256:ghost"); err != nil {
t.Fatalf("create local file: %v", err)
}
if resp, _ := req(t, "GET", "/api/v1/local/srv-ghost/ghost.bin", ""); resp.StatusCode != 500 {
t.Errorf("v1 download missing blob = %d, want 500", resp.StatusCode)
}
if resp, _ := req(t, "GET", "/api/v2/remotes/srv-ghost/files/ghost.bin", ""); resp.StatusCode != 500 {
t.Errorf("v2 download missing blob = %d, want 500", resp.StatusCode)
}
}
func TestServerBogusProviderType(t *testing.T) {
requireStack(t)
// Insert a remote with an unregistered package type directly, bypassing
// validation, to exercise the provider-not-found branches.
_, err := testSrv.db.Pool.Exec(context.Background(),
`INSERT INTO remotes (name, package_type, repo_type, base_url) VALUES ($1,'bogus','remote','https://x')`, "srv-bogus")
if err != nil {
t.Fatalf("insert bogus remote: %v", err)
}
defer testSrv.db.Pool.Exec(context.Background(), `DELETE FROM remotes WHERE name='srv-bogus'`)
if resp, _ := req(t, "GET", "/api/v1/remote/srv-bogus/x", ""); resp.StatusCode != 500 {
t.Errorf("bogus provider GET = %d, want 500", resp.StatusCode)
}
rq, _ := http.NewRequest("HEAD", testTS.URL+"/v2/srv-bogus/x", nil)
if resp, err := http.DefaultClient.Do(rq); err == nil {
resp.Body.Close()
if resp.StatusCode != 500 {
t.Errorf("bogus provider HEAD = %d, want 500", resp.StatusCode)
}
}
if resp, b := req(t, "POST", "/api/v2/probe", `{"remote":"srv-bogus","path":"x"}`); resp.StatusCode != 200 || !strings.Contains(string(b), `"status":500`) {
t.Errorf("bogus provider probe: %d %s", resp.StatusCode, b)
}
}
func TestServerNotFound(t *testing.T) {
requireStack(t)
if resp, _ := req(t, "GET", "/api/v2/remotes/does-not-exist", ""); resp.StatusCode != 404 {
t.Errorf("expected 404, got %d", resp.StatusCode)
}
if resp, _ := req(t, "GET", "/api/v1/remote/nope/x", ""); resp.StatusCode != 404 {
t.Errorf("expected 404 for unknown remote, got %d", resp.StatusCode)
}
// Unknown local repo -> 404 in handleLocal.
if resp, _ := req(t, "GET", "/api/v1/local/nope/x", ""); resp.StatusCode != 404 {
t.Errorf("expected 404 for unknown local repo, got %d", resp.StatusCode)
}
}
+160
View File
@@ -0,0 +1,160 @@
package storage
import (
"bytes"
"context"
"io"
"os"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
var (
testS3 *S3
testEndpoint string
)
func TestMain(m *testing.M) {
ctx := context.Background()
conn, terminate, err := testsupport.StartMinio(ctx)
if err != nil {
os.Exit(m.Run())
}
var s3 *S3
for i := 0; i < 20; i++ { // MinIO can report ready before bucket ops succeed
if s3, err = NewS3(conn.Endpoint, conn.AccessKey, conn.SecretKey, "test-bucket", false, ""); err == nil {
break
}
time.Sleep(500 * time.Millisecond)
}
if err != nil {
terminate()
panic(err)
}
testS3 = s3
testEndpoint = conn.Endpoint
code := m.Run()
terminate()
if code != 0 {
os.Exit(code)
}
}
func requireS3(t *testing.T) {
t.Helper()
if testS3 == nil {
t.Skip("Docker unavailable; skipping storage integration test")
}
}
func TestKeys(t *testing.T) {
if BlobKey("abc") != "blobs/sha256/abc" {
t.Error("BlobKey")
}
if IndexKey("remote", "path/to/x") != "indexes/remote/path/to/x" {
t.Error("IndexKey")
}
}
func TestS3RoundTrip(t *testing.T) {
requireS3(t)
ctx := context.Background()
key := "blobs/sha256/test1"
content := []byte("hello storage")
if err := testS3.Upload(ctx, key, bytes.NewReader(content), int64(len(content)), "text/plain"); err != nil {
t.Fatalf("upload: %v", err)
}
exists, err := testS3.Exists(ctx, key)
if err != nil || !exists {
t.Fatalf("exists after upload: %v %v", exists, err)
}
reader, info, err := testS3.Download(ctx, key)
if err != nil {
t.Fatalf("download: %v", err)
}
got, _ := io.ReadAll(reader)
reader.Close()
if !bytes.Equal(got, content) {
t.Errorf("content mismatch: %q", got)
}
if info.Size != int64(len(content)) || info.ContentType != "text/plain" {
t.Errorf("stat info wrong: size=%d ct=%s", info.Size, info.ContentType)
}
if _, err := testS3.Stat(ctx, key); err != nil {
t.Errorf("stat: %v", err)
}
if err := testS3.Delete(ctx, key); err != nil {
t.Fatalf("delete: %v", err)
}
if exists, _ := testS3.Exists(ctx, key); exists {
t.Error("expected object gone after delete")
}
}
func TestNewS3ExistingBucket(t *testing.T) {
requireS3(t)
// The bucket already exists from TestMain, so ensureBucket takes the
// "already present" path.
if _, err := NewS3(testEndpoint, "minioadmin", "minioadmin", "test-bucket", false, ""); err != nil {
t.Fatalf("second NewS3: %v", err)
}
}
func TestS3DownloadMissing(t *testing.T) {
requireS3(t)
if _, _, err := testS3.Download(context.Background(), "does/not/exist"); err == nil {
t.Error("expected error downloading missing key")
}
if _, err := testS3.Stat(context.Background(), "does/not/exist"); err == nil {
t.Error("expected error stat-ing missing key")
}
if exists, err := testS3.Exists(context.Background(), "does/not/exist"); err != nil || exists {
t.Errorf("Exists(missing) = %v, %v; want false, nil", exists, err)
}
}
func TestCASStore(t *testing.T) {
requireS3(t)
ctx := context.Background()
cas := NewCAS(testS3)
content := "content-addressed payload"
res, err := cas.Store(ctx, strings.NewReader(content), "text/plain")
if err != nil {
t.Fatalf("store: %v", err)
}
if res.AlreadyExists {
t.Error("first store should not report AlreadyExists")
}
if res.SizeBytes != int64(len(content)) || !strings.HasPrefix(res.ContentHash, "sha256:") {
t.Errorf("unexpected result: %+v", res)
}
// Storing identical content again is deduplicated.
res2, err := cas.Store(ctx, strings.NewReader(content), "text/plain")
if err != nil {
t.Fatalf("store again: %v", err)
}
if !res2.AlreadyExists || res2.ContentHash != res.ContentHash {
t.Errorf("second store should dedup: %+v", res2)
}
// The stored blob is retrievable.
reader, _, err := testS3.Download(ctx, res.S3Key)
if err != nil {
t.Fatalf("download stored blob: %v", err)
}
got, _ := io.ReadAll(reader)
reader.Close()
if string(got) != content {
t.Errorf("stored content mismatch: %q", got)
}
}
+101
View File
@@ -0,0 +1,101 @@
// Package testsupport starts throwaway backing containers (Postgres, Redis,
// MinIO) for integration-style unit tests. It is only ever imported from
// *_test.go files, so it never reaches the production binary. Each Start*
// function returns a connection detail plus a terminate func; callers wire
// them up in a TestMain and skip the package's tests when Docker is absent.
package testsupport
import (
"context"
"fmt"
"os"
"time"
"github.com/testcontainers/testcontainers-go"
tcpostgres "github.com/testcontainers/testcontainers-go/modules/postgres"
tcredis "github.com/testcontainers/testcontainers-go/modules/redis"
"github.com/testcontainers/testcontainers-go/wait"
)
func init() {
// The Ryuk reaper container cannot start in this environment; each Start*
// returns an explicit terminate func for cleanup instead.
if _, ok := os.LookupEnv("TESTCONTAINERS_RYUK_DISABLED"); !ok {
os.Setenv("TESTCONTAINERS_RYUK_DISABLED", "true")
}
}
// StartPostgres launches postgres:17-alpine and returns its DSN.
func StartPostgres(ctx context.Context) (dsn string, terminate func(), err error) {
c, err := tcpostgres.Run(ctx,
"postgres:17-alpine",
tcpostgres.WithDatabase("artifacts"),
tcpostgres.WithUsername("artifacts"),
tcpostgres.WithPassword("artifacts123"),
testcontainers.WithWaitStrategy(
// Postgres opens the port, runs init scripts, then restarts, so wait
// for the readiness log to appear twice to avoid connection resets.
wait.ForLog("database system is ready to accept connections").
WithOccurrence(2).
WithStartupTimeout(60*time.Second),
),
)
if err != nil {
return "", nil, err
}
host, _ := c.Host(ctx)
port, _ := c.MappedPort(ctx, "5432/tcp")
dsn = fmt.Sprintf("postgres://artifacts:artifacts123@%s:%s/artifacts?sslmode=disable", host, port.Port())
return dsn, func() { _ = c.Terminate(ctx) }, nil
}
// StartRedis launches redis:7-alpine and returns its URL.
func StartRedis(ctx context.Context) (url string, terminate func(), err error) {
c, err := tcredis.Run(ctx,
"redis:7-alpine",
testcontainers.WithWaitStrategy(
wait.ForListeningPort("6379/tcp").WithStartupTimeout(60*time.Second),
),
)
if err != nil {
return "", nil, err
}
host, _ := c.Host(ctx)
port, _ := c.MappedPort(ctx, "6379/tcp")
url = fmt.Sprintf("redis://%s:%s", host, port.Port())
return url, func() { _ = c.Terminate(ctx) }, nil
}
// MinioConn holds MinIO connection details.
type MinioConn struct {
Endpoint string
AccessKey string
SecretKey string
}
// StartMinio launches minio and returns its connection details.
func StartMinio(ctx context.Context) (conn MinioConn, terminate func(), err error) {
c, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: testcontainers.ContainerRequest{
Image: "minio/minio:latest",
ExposedPorts: []string{"9000/tcp"},
Cmd: []string{"server", "/data"},
Env: map[string]string{
"MINIO_ROOT_USER": "minioadmin",
"MINIO_ROOT_PASSWORD": "minioadmin",
},
WaitingFor: wait.ForHTTP("/minio/health/ready").WithPort("9000/tcp").WithStartupTimeout(60 * time.Second),
},
Started: true,
})
if err != nil {
return MinioConn{}, nil, err
}
host, _ := c.Host(ctx)
port, _ := c.MappedPort(ctx, "9000/tcp")
return MinioConn{
Endpoint: fmt.Sprintf("%s:%s", host, port.Port()),
AccessKey: "minioadmin",
SecretKey: "minioadmin",
}, func() { _ = c.Terminate(ctx) }, nil
}
+59
View File
@@ -0,0 +1,59 @@
package testsupport
import (
"bytes"
"encoding/binary"
)
// MinimalRPM builds a valid-enough RPM package in pure Go (no committed binary
// fixture, no external rpmbuild). It carries just the header tags the provider
// reads: name/version/release/arch plus a single self Provides entry, which is
// enough for cavaliergopher/rpm to parse and for repodata generation.
func MinimalRPM(name, version, release, arch string) []byte {
type tag struct {
id, typ, count uint32
data []byte
}
cstr := func(s string) []byte { return append([]byte(s), 0) }
tags := []tag{
{1000, 6, 1, cstr(name)}, // RPMTAG_NAME (STRING)
{1001, 6, 1, cstr(version)}, // RPMTAG_VERSION
{1002, 6, 1, cstr(release)}, // RPMTAG_RELEASE
{1022, 6, 1, cstr(arch)}, // RPMTAG_ARCH
{1047, 8, 1, cstr(name)}, // RPMTAG_PROVIDENAME (STRING_ARRAY)
{1112, 4, 1, []byte{0, 0, 0, 0}}, // RPMTAG_PROVIDEFLAGS (INT32)
{1113, 8, 1, cstr(version)}, // RPMTAG_PROVIDEVERSION (STRING_ARRAY)
}
buildHeader := func(entries []tag) []byte {
var index, store bytes.Buffer
for _, e := range entries {
off := uint32(store.Len())
for _, v := range []uint32{e.id, e.typ, off, e.count} {
binary.Write(&index, binary.BigEndian, v)
}
store.Write(e.data)
}
var b bytes.Buffer
b.Write([]byte{0x8e, 0xad, 0xe8, 0x01, 0, 0, 0, 0}) // header magic + reserved
binary.Write(&b, binary.BigEndian, uint32(len(entries)))
binary.Write(&b, binary.BigEndian, uint32(store.Len()))
b.Write(index.Bytes())
b.Write(store.Bytes())
return b.Bytes()
}
lead := make([]byte, 96)
copy(lead[0:4], []byte{0xed, 0xab, 0xee, 0xdb}) // lead magic
lead[4] = 3 // major version
binary.BigEndian.PutUint16(lead[8:10], 1) // archnum
copy(lead[10:76], name) // name (66 bytes, null-padded)
binary.BigEndian.PutUint16(lead[76:78], 1) // osnum
binary.BigEndian.PutUint16(lead[78:80], 5) // signature type
var out bytes.Buffer
out.Write(lead)
out.Write(buildHeader(nil)) // empty signature header (16 bytes, 8-aligned)
out.Write(buildHeader(tags))
return out.Bytes()
}
+2 -2
View File
@@ -79,7 +79,7 @@ func (e *Engine) fetchMemberIndexes(ctx context.Context, virt models.Virtual, pa
results[idx] = result{err: fmt.Errorf("local index %q: %w", name, err)} results[idx] = result{err: fmt.Errorf("local index %q: %w", name, err)}
return return
} }
results[idx] = result{index: MemberIndex{RemoteName: name, RepoType: remote.RepoType, Body: body}} results[idx] = result{index: MemberIndex{RemoteName: name, RepoType: remote.RepoType, BaseURL: remote.BaseURL, Body: body}}
return return
} }
@@ -102,7 +102,7 @@ func (e *Engine) fetchMemberIndexes(ctx context.Context, virt models.Virtual, pa
return return
} }
results[idx] = result{index: MemberIndex{RemoteName: name, RepoType: remote.RepoType, Body: body}} results[idx] = result{index: MemberIndex{RemoteName: name, RepoType: remote.RepoType, BaseURL: remote.BaseURL, Body: body}}
}(i, memberName) }(i, memberName)
} }
+40 -3
View File
@@ -54,15 +54,27 @@ func (m *HelmMerger) MergeIndexes(members []MemberIndex, proxyBaseURL string) ([
seen[chart][ver.Version] = true seen[chart][ver.Version] = true
if proxyBaseURL != "" { if proxyBaseURL != "" {
routePrefix := "remote"
if member.RepoType == "local" {
routePrefix = "local"
}
baseHost := extractHost(member.BaseURL)
for i, u := range ver.URLs { for i, u := range ver.URLs {
if strings.HasPrefix(u, "http://") || strings.HasPrefix(u, "https://") { if strings.HasPrefix(u, "http://") || strings.HasPrefix(u, "https://") {
ver.URLs[i] = fmt.Sprintf("%s/api/v1/remote/%s/%s", if baseHost != "" && extractHost(u) != baseHost {
continue
}
relPath := extractPathRelativeToBase(u, member.BaseURL)
ver.URLs[i] = fmt.Sprintf("%s/api/v1/%s/%s/%s",
strings.TrimRight(proxyBaseURL, "/"), strings.TrimRight(proxyBaseURL, "/"),
routePrefix,
member.RemoteName, member.RemoteName,
extractPath(u)) relPath)
} else { } else {
ver.URLs[i] = fmt.Sprintf("%s/api/v1/remote/%s/%s", ver.URLs[i] = fmt.Sprintf("%s/api/v1/%s/%s/%s",
strings.TrimRight(proxyBaseURL, "/"), strings.TrimRight(proxyBaseURL, "/"),
routePrefix,
member.RemoteName, member.RemoteName,
u) u)
} }
@@ -78,6 +90,31 @@ func (m *HelmMerger) MergeIndexes(members []MemberIndex, proxyBaseURL string) ([
return yaml.Marshal(merged) return yaml.Marshal(merged)
} }
func extractHost(rawURL string) string {
idx := strings.Index(rawURL, "://")
if idx == -1 {
return ""
}
rest := rawURL[idx+3:]
slashIdx := strings.Index(rest, "/")
if slashIdx == -1 {
return rest
}
return rest[:slashIdx]
}
func extractPathRelativeToBase(rawURL, baseURL string) string {
fullPath := extractPath(rawURL)
basePath := extractPath(baseURL)
if basePath != "" {
basePath = strings.TrimRight(basePath, "/") + "/"
if strings.HasPrefix(fullPath, basePath) {
return fullPath[len(basePath):]
}
}
return fullPath
}
func extractPath(rawURL string) string { func extractPath(rawURL string) string {
idx := strings.Index(rawURL, "://") idx := strings.Index(rawURL, "://")
if idx == -1 { if idx == -1 {
+1
View File
@@ -9,6 +9,7 @@ import (
type MemberIndex struct { type MemberIndex struct {
RemoteName string RemoteName string
RepoType models.RepoType RepoType models.RepoType
BaseURL string
Body []byte Body []byte
} }
+155
View File
@@ -0,0 +1,155 @@
package virtual
import (
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func TestRegisterGetMerger(t *testing.T) {
if _, err := GetMerger(models.PackageHelm); err != nil {
t.Errorf("helm merger should be registered: %v", err)
}
if _, err := GetMerger(models.PackagePyPI); err != nil {
t.Errorf("pypi merger should be registered: %v", err)
}
if _, err := GetMerger(models.PackageType("nope")); err == nil {
t.Error("expected error for unknown merger")
}
}
func TestPyPIMerge(t *testing.T) {
m := &PyPIMerger{}
members := []MemberIndex{
{RemoteName: "a", RepoType: models.RepoTypeRemote, Body: []byte(`<a href="pkg/foo-1.0.whl">foo-1.0.whl</a>`)},
{RemoteName: "b", RepoType: models.RepoTypeLocal, Body: []byte(`<a href="/bar-2.0.whl">bar-2.0.whl</a>`)},
}
out, err := m.MergeIndexes(members, "http://proxy")
if err != nil {
t.Fatal(err)
}
s := string(out)
if !strings.Contains(s, "foo-1.0.whl") || !strings.Contains(s, "bar-2.0.whl") {
t.Errorf("merged index missing entries: %s", s)
}
if !strings.Contains(s, "http://proxy/api/v1/remote/a/pkg/foo-1.0.whl") {
t.Errorf("remote href not rewritten: %s", s)
}
if !strings.Contains(s, "http://proxy/api/v1/local/b/bar-2.0.whl") {
t.Errorf("local href not rewritten: %s", s)
}
// Sorted output: foo before... entries sorted by link text.
if strings.Index(s, "bar-2.0.whl") > strings.Index(s, "foo-1.0.whl") {
t.Error("entries should be sorted by text")
}
// Duplicate link texts across members are de-duplicated.
dup := []MemberIndex{
{RemoteName: "a", Body: []byte(`<a href="x">dup</a>`)},
{RemoteName: "b", Body: []byte(`<a href="y">dup</a>`)},
}
out, _ = m.MergeIndexes(dup, "")
if strings.Count(string(out), ">dup</a>") != 1 {
t.Errorf("duplicate not de-duplicated: %s", out)
}
}
func TestPyPIMergeNoProxyAndBadLinks(t *testing.T) {
m := &PyPIMerger{}
members := []MemberIndex{{
RemoteName: "a",
Body: []byte("<a href=\"foo.whl\">foo.whl</a>\n<a>no href</a>\n<span>not a link</span>"),
}}
// No proxy base URL: hrefs are left as-is.
out, err := m.MergeIndexes(members, "")
if err != nil {
t.Fatal(err)
}
s := string(out)
if !strings.Contains(s, ">foo.whl</a>") {
t.Errorf("missing link: %s", s)
}
}
func TestHelmMerge(t *testing.T) {
m := &HelmMerger{}
memberA := `apiVersion: v1
entries:
alpha:
- name: alpha
version: 1.0.0
urls:
- charts/alpha-1.0.0.tgz
`
memberB := `apiVersion: v1
entries:
beta:
- name: beta
version: 2.0.0
urls:
- https://charts.example.com/beta-2.0.0.tgz
gamma:
- name: gamma
version: 3.0.0
urls:
- https://other-host.example.net/gamma-3.0.0.tgz
`
members := []MemberIndex{
{RemoteName: "a", RepoType: models.RepoTypeLocal, BaseURL: "https://charts.example.com", Body: []byte(memberA)},
{RemoteName: "b", RepoType: models.RepoTypeRemote, BaseURL: "https://charts.example.com", Body: []byte(memberB)},
}
out, err := m.MergeIndexes(members, "http://proxy")
if err != nil {
t.Fatal(err)
}
s := string(out)
for _, chart := range []string{"alpha", "beta", "gamma"} {
if !strings.Contains(s, chart) {
t.Errorf("merged index missing chart %q: %s", chart, s)
}
}
// Relative URL from a local member is rewritten under /local/.
if !strings.Contains(s, "http://proxy/api/v1/local/a/charts/alpha-1.0.0.tgz") {
t.Errorf("relative local url not rewritten: %s", s)
}
// Same-host absolute URL from a remote member is rewritten under /remote/.
if !strings.Contains(s, "http://proxy/api/v1/remote/b/beta-2.0.0.tgz") {
t.Errorf("same-host absolute url not rewritten: %s", s)
}
// Cross-host absolute URL is left untouched.
if !strings.Contains(s, "https://other-host.example.net/gamma-3.0.0.tgz") {
t.Errorf("cross-host url should be preserved: %s", s)
}
}
func TestHelmMergeDedup(t *testing.T) {
m := &HelmMerger{}
body := `apiVersion: v1
entries:
alpha:
- name: alpha
version: 1.0.0
urls: [charts/alpha-1.0.0.tgz]
`
members := []MemberIndex{
{RemoteName: "a", BaseURL: "https://x", Body: []byte(body)},
{RemoteName: "b", BaseURL: "https://x", Body: []byte(body)},
}
out, _ := m.MergeIndexes(members, "")
if strings.Count(string(out), "version: 1.0.0") != 1 {
t.Errorf("duplicate chart version not de-duplicated: %s", out)
}
}
func TestHelmMergeInvalidYAML(t *testing.T) {
m := &HelmMerger{}
out, err := m.MergeIndexes([]MemberIndex{{RemoteName: "a", Body: []byte("::: not yaml :::")}}, "")
if err != nil {
t.Fatalf("invalid member yaml should be skipped, not error: %v", err)
}
if !strings.Contains(string(out), "apiVersion") {
t.Errorf("expected a valid empty merged index: %s", out)
}
}
+145
View File
@@ -0,0 +1,145 @@
package client
import (
"context"
"net/http"
"net/http/httptest"
"strings"
"testing"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func testServer(t *testing.T, h http.HandlerFunc) *Client {
t.Helper()
srv := httptest.NewServer(h)
t.Cleanup(srv.Close)
return New(srv.URL)
}
func TestRemotesRoundTrip(t *testing.T) {
var gotMethod, gotPath string
c := testServer(t, func(w http.ResponseWriter, r *http.Request) {
gotMethod, gotPath = r.Method, r.URL.Path
switch {
case r.Method == http.MethodGet && r.URL.Path == "/api/v2/remotes":
w.Write([]byte(`[{"name":"a"},{"name":"b"}]`))
case r.Method == http.MethodGet && r.URL.Path == "/api/v2/remotes/a":
w.Write([]byte(`{"name":"a"}`))
case r.Method == http.MethodDelete:
w.WriteHeader(http.StatusNoContent)
default:
w.WriteHeader(http.StatusCreated)
w.Write([]byte(`{"name":"a"}`))
}
})
ctx := context.Background()
remotes, err := c.ListRemotes(ctx)
if err != nil || len(remotes) != 2 {
t.Fatalf("ListRemotes: %v %v", remotes, err)
}
if r, err := c.GetRemote(ctx, "a"); err != nil || r.Name != "a" {
t.Fatalf("GetRemote: %v %v", r, err)
}
if err := c.CreateRemote(ctx, &models.Remote{Name: "a", PackageType: models.PackageGeneric}); err != nil {
t.Fatalf("CreateRemote: %v", err)
}
if err := c.UpdateRemote(ctx, &models.Remote{Name: "a"}); err != nil {
t.Fatalf("UpdateRemote: %v", err)
}
if err := c.DeleteRemote(ctx, "a"); err != nil {
t.Fatalf("DeleteRemote: %v", err)
}
if gotMethod != http.MethodDelete || gotPath != "/api/v2/remotes/a" {
t.Errorf("last call = %s %s", gotMethod, gotPath)
}
}
func TestVirtualsRoundTrip(t *testing.T) {
c := testServer(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/virtuals"):
w.Write([]byte(`[{"name":"v"}]`))
case r.Method == http.MethodGet:
w.Write([]byte(`{"name":"v"}`))
case r.Method == http.MethodDelete:
w.WriteHeader(http.StatusNoContent)
default:
w.WriteHeader(http.StatusCreated)
w.Write([]byte(`{"name":"v"}`))
}
})
ctx := context.Background()
if vs, err := c.ListVirtuals(ctx); err != nil || len(vs) != 1 {
t.Fatalf("ListVirtuals: %v %v", vs, err)
}
if v, err := c.GetVirtual(ctx, "v"); err != nil || v.Name != "v" {
t.Fatalf("GetVirtual: %v %v", v, err)
}
if err := c.CreateVirtual(ctx, &models.Virtual{Name: "v"}); err != nil {
t.Fatalf("CreateVirtual: %v", err)
}
if err := c.UpdateVirtual(ctx, &models.Virtual{Name: "v"}); err != nil {
t.Fatalf("UpdateVirtual: %v", err)
}
if err := c.DeleteVirtual(ctx, "v"); err != nil {
t.Fatalf("DeleteVirtual: %v", err)
}
}
func TestStatsHealthObjects(t *testing.T) {
c := testServer(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/stats"):
w.Write([]byte(`{"total_remotes":3}`))
case strings.HasSuffix(r.URL.Path, "/health"):
w.Write([]byte(`{"status":"ok"}`))
case r.Method == http.MethodDelete:
w.WriteHeader(http.StatusNoContent)
default:
w.Write([]byte(`[{"path":"p"}]`))
}
})
ctx := context.Background()
if _, err := c.Stats(ctx); err != nil {
t.Fatalf("Stats: %v", err)
}
if _, err := c.Health(ctx); err != nil {
t.Fatalf("Health: %v", err)
}
if objs, err := c.ListObjects(ctx, "r", 1, 50); err != nil || len(objs) != 1 {
t.Fatalf("ListObjects: %v %v", objs, err)
}
if err := c.EvictObject(ctx, "r", "some/path"); err != nil {
t.Fatalf("EvictObject: %v", err)
}
}
func TestErrorResponses(t *testing.T) {
c := testServer(t, func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "boom", http.StatusInternalServerError)
})
ctx := context.Background()
_, err := c.GetRemote(ctx, "x")
if err == nil || !strings.Contains(err.Error(), "api error 500") {
t.Errorf("expected api error, got %v", err)
}
}
func TestDecodeError(t *testing.T) {
c := testServer(t, func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(`not json`))
})
if _, err := c.ListRemotes(context.Background()); err == nil || !strings.Contains(err.Error(), "decode") {
t.Errorf("expected decode error, got %v", err)
}
}
func TestRequestError(t *testing.T) {
// Invalid base URL triggers request construction failure.
c := New("http://[::1]:namedport")
if err := c.DeleteRemote(context.Background(), "x"); err == nil {
t.Error("expected request error for invalid URL")
}
}
+30
View File
@@ -2,6 +2,7 @@ package models
import ( import (
"fmt" "fmt"
"regexp"
"time" "time"
) )
@@ -46,6 +47,11 @@ type Remote struct {
MutableTTL int `json:"mutable_ttl"` MutableTTL int `json:"mutable_ttl"`
CheckMutable bool `json:"check_mutable"` CheckMutable bool `json:"check_mutable"`
// Upstream HTTP timeouts in seconds. 0 means use the server default.
UpstreamDialTimeout int `json:"upstream_dial_timeout,omitempty"`
UpstreamTLSTimeout int `json:"upstream_tls_timeout,omitempty"`
UpstreamResponseHeaderTimeout int `json:"upstream_response_header_timeout,omitempty"`
Patterns []string `json:"patterns,omitempty"` Patterns []string `json:"patterns,omitempty"`
Blocklist []string `json:"blocklist,omitempty"` Blocklist []string `json:"blocklist,omitempty"`
MutablePatterns []string `json:"mutable_patterns,omitempty"` MutablePatterns []string `json:"mutable_patterns,omitempty"`
@@ -66,6 +72,30 @@ type Remote struct {
UpdatedAt time.Time `json:"updated_at"` UpdatedAt time.Time `json:"updated_at"`
} }
// ValidatePatterns ensures every configured regex compiles. Storing an
// invalid pattern would otherwise be silently dropped at match time, which
// for the blocklist is a fail-open: a mistyped deny rule becomes a no-op.
func (r *Remote) ValidatePatterns() error {
groups := []struct {
field string
patterns []string
}{
{"patterns", r.Patterns},
{"blocklist", r.Blocklist},
{"mutable_patterns", r.MutablePatterns},
{"immutable_patterns", r.ImmutablePatterns},
{"ban_tags", r.BanTags},
}
for _, g := range groups {
for _, p := range g.patterns {
if _, err := regexp.Compile(p); err != nil {
return fmt.Errorf("invalid regex in %s: %q: %w", g.field, p, err)
}
}
}
return nil
}
type RemoteWithStats struct { type RemoteWithStats struct {
Remote Remote
Stats RemoteStats `json:"stats"` Stats RemoteStats `json:"stats"`
+19
View File
@@ -0,0 +1,19 @@
package models
import "testing"
func TestRemote_ValidatePatterns(t *testing.T) {
valid := &Remote{
Patterns: []string{`.*\.tar\.gz$`},
Blocklist: []string{`^secret/`},
ImmutablePatterns: []string{`\.rpm$`},
}
if err := valid.ValidatePatterns(); err != nil {
t.Fatalf("expected valid patterns, got %v", err)
}
bad := &Remote{Blocklist: []string{`[unterminated`}}
if err := bad.ValidatePatterns(); err == nil {
t.Fatal("expected error for invalid blocklist regex, got nil")
}
}
+18
View File
@@ -0,0 +1,18 @@
package models
import "testing"
func TestRepoType(t *testing.T) {
if RepoTypeRemote.String() != "remote" || RepoTypeLocal.String() != "local" {
t.Error("RepoType.String")
}
if !RepoTypeRemote.Valid() || RepoType("bogus").Valid() {
t.Error("RepoType.Valid")
}
if rt, err := ParseRepoType("local"); err != nil || rt != RepoTypeLocal {
t.Errorf("ParseRepoType(local) = %v %v", rt, err)
}
if _, err := ParseRepoType("nope"); err == nil {
t.Error("ParseRepoType should reject unknown")
}
}
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Build the artifactapi container, bring up the full stack (postgres, redis,
# minio, artifactapi) plus a static mock upstream, and run the dockerised e2e
# suite against the running product over HTTP. Tears everything down on exit.
set -euo pipefail
cd "$(dirname "$0")/.."
# Publish artifactapi on 8001 to avoid colliding with a local instance on 8000.
export ARTIFACTAPI_PORT="${ARTIFACTAPI_PORT:-8001}"
COMPOSE=(docker compose -f docker-compose.yml -f docker-compose.e2e.yml)
API_URL="${ARTIFACTAPI_URL:-http://localhost:${ARTIFACTAPI_PORT}}"
cleanup() {
echo "==> tearing down stack"
"${COMPOSE[@]}" down -v --remove-orphans >/dev/null 2>&1 || true
}
trap cleanup EXIT
echo "==> building and starting stack (postgres, redis, minio, mockupstream, artifactapi)"
"${COMPOSE[@]}" up -d --build postgres redis minio mockupstream artifactapi
echo "==> waiting for artifactapi health at ${API_URL}"
for i in $(seq 1 60); do
if curl -fsS "${API_URL}/health" >/dev/null 2>&1; then
echo " healthy after ${i}s"
break
fi
if [ "$i" -eq 60 ]; then
echo "!!! artifactapi did not become healthy in time; recent logs:"
"${COMPOSE[@]}" logs --tail=50 artifactapi
exit 1
fi
sleep 1
done
echo "==> running dockerised e2e suite"
ARTIFACTAPI_URL="${API_URL}" \
MOCK_UPSTREAM_INTERNAL="${MOCK_UPSTREAM_INTERNAL:-http://mockupstream}" \
go test -tags=dockere2e -count=1 -timeout=10m -v ./e2e-docker/...
+7
View File
@@ -6,13 +6,20 @@ COPY package.json package-lock.json* ./
RUN npm ci RUN npm ci
COPY . . COPY . .
ARG BASE_PATH=/
ENV BASE_PATH=${BASE_PATH}
RUN npm run build RUN npm run build
FROM nginx:alpine FROM nginx:alpine
ARG BASE_PATH=/
COPY --from=builder /app/dist /usr/share/nginx/html COPY --from=builder /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf COPY nginx.conf /etc/nginx/conf.d/default.conf
RUN sed -i "s|\${BASE_PATH}|${BASE_PATH}|g" /etc/nginx/conf.d/default.conf
EXPOSE 80 EXPOSE 80
CMD ["nginx", "-g", "daemon off;"] CMD ["nginx", "-g", "daemon off;"]
+6 -27
View File
@@ -5,33 +5,12 @@ server {
root /usr/share/nginx/html; root /usr/share/nginx/html;
index index.html; index index.html;
location /api/ { location ${BASE_PATH}/ {
proxy_pass http://artifactapi:8000; rewrite ^${BASE_PATH}(/.*)$ $1 break;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
}
location /v2/ {
proxy_pass http://artifactapi:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering off;
}
location /health {
proxy_pass http://artifactapi:8000;
}
location /metrics {
proxy_pass http://artifactapi:8000;
}
location / {
try_files $uri $uri/ /index.html; try_files $uri $uri/ /index.html;
} }
location = ${BASE_PATH} {
return 301 ${BASE_PATH}/;
}
} }
+6
View File
@@ -2,6 +2,8 @@ import { Routes, Route, NavLink } from 'react-router-dom';
import { Dashboard } from './pages/Dashboard'; import { Dashboard } from './pages/Dashboard';
import { Remotes } from './pages/Remotes'; import { Remotes } from './pages/Remotes';
import { RemoteDetail } from './pages/RemoteDetail'; import { RemoteDetail } from './pages/RemoteDetail';
import { Locals } from './pages/Locals';
import { LocalDetail } from './pages/LocalDetail';
import { Virtuals } from './pages/Virtuals'; import { Virtuals } from './pages/Virtuals';
import { Objects } from './pages/Objects'; import { Objects } from './pages/Objects';
import { Probe } from './pages/Probe'; import { Probe } from './pages/Probe';
@@ -18,6 +20,7 @@ export function App() {
<div className="sidebar-nav"> <div className="sidebar-nav">
<NavLink to="/" end>Dashboard</NavLink> <NavLink to="/" end>Dashboard</NavLink>
<NavLink to="/remotes">Remotes</NavLink> <NavLink to="/remotes">Remotes</NavLink>
<NavLink to="/locals">Locals</NavLink>
<NavLink to="/virtuals">Virtuals</NavLink> <NavLink to="/virtuals">Virtuals</NavLink>
<NavLink to="/probe">Test Remote</NavLink> <NavLink to="/probe">Test Remote</NavLink>
</div> </div>
@@ -31,6 +34,9 @@ export function App() {
<Route path="/remotes" element={<Remotes />} /> <Route path="/remotes" element={<Remotes />} />
<Route path="/remotes/:name" element={<RemoteDetail />} /> <Route path="/remotes/:name" element={<RemoteDetail />} />
<Route path="/remotes/:name/objects" element={<Objects />} /> <Route path="/remotes/:name/objects" element={<Objects />} />
<Route path="/locals" element={<Locals />} />
<Route path="/locals/:name" element={<LocalDetail />} />
<Route path="/locals/:name/objects" element={<Objects />} />
<Route path="/virtuals" element={<Virtuals />} /> <Route path="/virtuals" element={<Virtuals />} />
<Route path="/probe" element={<Probe />} /> <Route path="/probe" element={<Probe />} />
</Routes> </Routes>
+6
View File
@@ -34,6 +34,12 @@ export const api = {
evictObject: (remote: string, path: string) => evictObject: (remote: string, path: string) =>
fetchJSON<void>(`/api/v2/remotes/${remote}/objects/${path}`, { method: 'DELETE' }), fetchJSON<void>(`/api/v2/remotes/${remote}/objects/${path}`, { method: 'DELETE' }),
listLocalObjects: (name: string, page = 1, perPage = 50) =>
fetchJSON<Artifact[]>(`/api/v2/locals/${name}/objects?page=${page}&per_page=${perPage}`),
evictLocalObject: (name: string, path: string) =>
fetchJSON<void>(`/api/v2/locals/${name}/objects/${path}`, { method: 'DELETE' }),
flushRemoteCache: (remote: string) => flushRemoteCache: (remote: string) =>
fetchJSON<void>(`/api/v2/remotes/${remote}/cache`, { method: 'DELETE' }), fetchJSON<void>(`/api/v2/remotes/${remote}/cache`, { method: 'DELETE' }),
+5 -1
View File
@@ -4,9 +4,13 @@ import { BrowserRouter } from 'react-router-dom';
import { App } from './App'; import { App } from './App';
import './index.css'; import './index.css';
declare const __BASE_PATH__: string;
const basename = __BASE_PATH__.replace(/\/+$/, '') || '/';
createRoot(document.getElementById('root')!).render( createRoot(document.getElementById('root')!).render(
<StrictMode> <StrictMode>
<BrowserRouter> <BrowserRouter basename={basename}>
<App /> <App />
</BrowserRouter> </BrowserRouter>
</StrictMode>, </StrictMode>,
+5
View File
@@ -50,6 +50,11 @@ export function Dashboard() {
value={formatNumber(stats.total_blobs_deduped)} value={formatNumber(stats.total_blobs_deduped)}
sub="shared blobs" sub="shared blobs"
/> />
<StatsCard
label="Bandwidth Saved"
value={formatBytes(stats.bandwidth_saved_30d)}
sub="last 30 days"
/>
</div> </div>
{health && ( {health && (
+46
View File
@@ -0,0 +1,46 @@
import { useEffect, useState } from 'react';
import { useParams, Link } from 'react-router-dom';
import { api } from '../api/client';
import type { Remote } from '../api/types';
import { Badge } from '../components/Badge';
import './RemoteDetail.css';
export function LocalDetail() {
const { name } = useParams<{ name: string }>();
const [remote, setRemote] = useState<Remote | null>(null);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
if (!name) return;
api.getRemote(name)
.then(setRemote)
.catch(e => setError(e.message));
}, [name]);
if (error) return <div className="error-banner">{error}</div>;
if (!remote) return <div className="loading">Loading...</div>;
return (
<div>
<div className="detail-header">
<Link to="/locals" className="back-link">&larr; Locals</Link>
<h1 className="page-title">{remote.name}</h1>
<div className="detail-badges">
<Badge variant="blue">{remote.package_type}</Badge>
<Badge variant="default">local</Badge>
{remote.managed_by && <Badge variant="green">managed by {remote.managed_by}</Badge>}
</div>
</div>
{remote.description && (
<p className="detail-description">{remote.description}</p>
)}
<div className="detail-actions">
<Link to={`/locals/${remote.name}/objects`} className="btn btn-primary">
Browse Files
</Link>
</div>
</div>
);
}
+93
View File
@@ -0,0 +1,93 @@
import { useEffect, useState } from 'react';
import { useNavigate } from 'react-router-dom';
import { api } from '../api/client';
import type { Remote } from '../api/types';
import { Badge } from '../components/Badge';
import { DataTable } from '../components/DataTable';
import './Remotes.css';
const typeColors: Record<string, 'blue' | 'green' | 'yellow' | 'red' | 'default'> = {
docker: 'blue',
helm: 'green',
rpm: 'yellow',
pypi: 'blue',
npm: 'red',
generic: 'default',
alpine: 'green',
puppet: 'yellow',
terraform: 'blue',
goproxy: 'green',
};
export function Locals() {
const navigate = useNavigate();
const [remotes, setRemotes] = useState<Remote[]>([]);
const [filter, setFilter] = useState('');
const [loading, setLoading] = useState(true);
useEffect(() => {
api.listRemotes()
.then(r => setRemotes((r || []).filter(x => x.repo_type === 'local')))
.finally(() => setLoading(false));
}, []);
const filtered = remotes.filter(r => {
if (filter && !r.name.toLowerCase().includes(filter.toLowerCase())) return false;
return true;
});
return (
<div>
<h1 className="page-title">Local Repositories</h1>
<div className="remotes-toolbar">
<input
className="search-input"
placeholder="Filter by name..."
value={filter}
onChange={e => setFilter(e.target.value)}
/>
<span className="result-count">{filtered.length} locals</span>
</div>
{loading ? (
<div className="loading">Loading...</div>
) : (
<DataTable
columns={[
{
key: 'name',
header: 'Name',
render: (r: Remote) => <span className="mono">{r.name}</span>,
},
{
key: 'type',
header: 'Type',
render: (r: Remote) => (
<Badge variant={typeColors[r.package_type] || 'default'}>
{r.package_type}
</Badge>
),
width: '110px',
},
{
key: 'description',
header: 'Description',
render: (r: Remote) => r.description || <span className="text-muted"></span>,
},
{
key: 'managed',
header: 'Managed',
render: (r: Remote) =>
r.managed_by ? <Badge variant="blue">{r.managed_by}</Badge> : <span className="text-muted"></span>,
width: '100px',
},
]}
data={filtered}
emptyMessage="No local repositories configured"
onRowClick={(r) => navigate(`/locals/${r.name}`)}
/>
)}
</div>
);
}
+9 -5
View File
@@ -1,5 +1,5 @@
import { useEffect, useState, useCallback, useMemo } from 'react'; import { useEffect, useState, useCallback, useMemo } from 'react';
import { useParams, Link } from 'react-router-dom'; import { useParams, useLocation, Link } from 'react-router-dom';
import { api } from '../api/client'; import { api } from '../api/client';
import type { Artifact } from '../api/types'; import type { Artifact } from '../api/types';
import { formatBytes, timeAgo, truncateHash } from '../components/format'; import { formatBytes, timeAgo, truncateHash } from '../components/format';
@@ -171,6 +171,9 @@ function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
export function Objects() { export function Objects() {
const { name } = useParams<{ name: string }>(); const { name } = useParams<{ name: string }>();
const location = useLocation();
const isLocal = location.pathname.startsWith('/locals/');
const backLink = isLocal ? `/locals/${name}` : `/remotes/${name}`;
const [artifacts, setArtifacts] = useState<Artifact[]>([]); const [artifacts, setArtifacts] = useState<Artifact[]>([]);
const [loading, setLoading] = useState(true); const [loading, setLoading] = useState(true);
const [filter, setFilter] = useState(''); const [filter, setFilter] = useState('');
@@ -179,16 +182,17 @@ export function Objects() {
const load = useCallback(() => { const load = useCallback(() => {
if (!name) return; if (!name) return;
setLoading(true); setLoading(true);
api.listObjects(name, 1, 5000) const req = isLocal ? api.listLocalObjects(name, 1, 5000) : api.listObjects(name, 1, 5000);
req
.then(a => setArtifacts(a || [])) .then(a => setArtifacts(a || []))
.finally(() => setLoading(false)); .finally(() => setLoading(false));
}, [name]); }, [name, isLocal]);
useEffect(() => { load(); }, [load]); useEffect(() => { load(); }, [load]);
const handleEvict = async (path: string) => { const handleEvict = async (path: string) => {
if (!name || !confirm(`Evict ${path}?`)) return; if (!name || !confirm(`Evict ${path}?`)) return;
await api.evictObject(name, path); await (isLocal ? api.evictLocalObject(name, path) : api.evictObject(name, path));
load(); load();
}; };
@@ -233,7 +237,7 @@ export function Objects() {
return ( return (
<div> <div>
<div className="detail-header"> <div className="detail-header">
<Link to={`/remotes/${name}`} className="back-link">&larr; {name}</Link> <Link to={backLink} className="back-link">&larr; {name}</Link>
<h1 className="page-title">Cached Objects</h1> <h1 className="page-title">Cached Objects</h1>
</div> </div>
+3 -2
View File
@@ -32,9 +32,10 @@ export function Remotes() {
.finally(() => setLoading(false)); .finally(() => setLoading(false));
}, []); }, []);
const types = [...new Set(remotes.map(r => r.package_type))].sort(); const remoteOnly = remotes.filter(r => r.repo_type !== 'local');
const types = [...new Set(remoteOnly.map(r => r.package_type))].sort();
const filtered = remotes.filter(r => { const filtered = remoteOnly.filter(r => {
if (typeFilter && r.package_type !== typeFilter) return false; if (typeFilter && r.package_type !== typeFilter) return false;
if (filter && !r.name.toLowerCase().includes(filter.toLowerCase())) return false; if (filter && !r.name.toLowerCase().includes(filter.toLowerCase())) return false;
return true; return true;
+32 -10
View File
@@ -1,21 +1,38 @@
import { useEffect, useState } from 'react'; import { useEffect, useState } from 'react';
import { Link } from 'react-router-dom';
import { api } from '../api/client'; import { api } from '../api/client';
import type { Virtual } from '../api/types'; import type { Remote, Virtual } from '../api/types';
import { Badge } from '../components/Badge'; import { Badge } from '../components/Badge';
import { DataTable } from '../components/DataTable'; import { DataTable } from '../components/DataTable';
import './Virtuals.css'; import './Virtuals.css';
export function Virtuals() { export function Virtuals() {
const [virtuals, setVirtuals] = useState<Virtual[]>([]); const [virtuals, setVirtuals] = useState<Virtual[]>([]);
const [remoteMap, setRemoteMap] = useState<Record<string, Remote>>({});
const [loading, setLoading] = useState(true); const [loading, setLoading] = useState(true);
const [expanded, setExpanded] = useState<string | null>(null); const [expanded, setExpanded] = useState<string | null>(null);
useEffect(() => { useEffect(() => {
api.listVirtuals() Promise.all([api.listVirtuals(), api.listRemotes()])
.then(v => setVirtuals(v || [])) .then(([v, r]) => {
setVirtuals(v || []);
const map: Record<string, Remote> = {};
for (const remote of r || []) {
map[remote.name] = remote;
}
setRemoteMap(map);
})
.finally(() => setLoading(false)); .finally(() => setLoading(false));
}, []); }, []);
function memberLink(name: string) {
const remote = remoteMap[name];
if (remote?.repo_type === 'local') {
return `/locals/${name}`;
}
return `/remotes/${name}`;
}
return ( return (
<div> <div>
<h1 className="page-title">Virtual Repositories</h1> <h1 className="page-title">Virtual Repositories</h1>
@@ -40,7 +57,7 @@ export function Virtuals() {
key: 'members', key: 'members',
header: 'Members', header: 'Members',
render: (v: Virtual) => ( render: (v: Virtual) => (
<span className="member-count">{v.members?.length || 0} remotes</span> <span className="member-count">{v.members?.length || 0} repos</span>
), ),
width: '110px', width: '110px',
}, },
@@ -69,12 +86,17 @@ export function Virtuals() {
<ul className="member-list"> <ul className="member-list">
{virtuals {virtuals
.find(v => v.name === expanded) .find(v => v.name === expanded)
?.members?.map((m, i) => ( ?.members?.map((m, i) => {
<li key={m}> const remote = remoteMap[m];
<span className="member-priority">{i + 1}</span> const typeLabel = remote?.repo_type === 'local' ? 'local' : 'remote';
<a href={`/remotes/${m}`} className="mono">{m}</a> return (
</li> <li key={m}>
))} <span className="member-priority">{i + 1}</span>
<Link to={memberLink(m)} className="mono">{m}</Link>
<Badge variant={typeLabel === 'local' ? 'yellow' : 'default'}>{typeLabel}</Badge>
</li>
);
})}
</ul> </ul>
</div> </div>
)} )}
+6
View File
@@ -1,7 +1,10 @@
import { defineConfig } from 'vite' import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react' import react from '@vitejs/plugin-react'
const basePath = process.env.BASE_PATH || '/'
export default defineConfig({ export default defineConfig({
base: basePath,
plugins: [react()], plugins: [react()],
server: { server: {
proxy: { proxy: {
@@ -11,4 +14,7 @@ export default defineConfig({
'/metrics': 'http://localhost:8000', '/metrics': 'http://localhost:8000',
}, },
}, },
define: {
'__BASE_PATH__': JSON.stringify(basePath),
},
}) })