Compare commits

...

6 Commits

Author SHA1 Message Date
unkin-agent 5a2bfdf73a merge master
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-10-17 13:20:00 +11:00
unkin-agent 9b91aa39a6 Add local puppet forge (#145)
Puppet modules can only be proxied from a remote Forge; there is nowhere to publish internal modules for puppet module install, r10k and g10k.

- add puppet local uploads at <owner>-<module>-<version>.tar.gz, rejecting existing versions
- record each release's metadata.json, md5 and sha256 in a puppet_metadata table, streaming the tarball under size caps
- serve Forge v3 /v3/modules, paginated /v3/releases, /v3/releases/<slug> and /v3/files
- add an e2e-docker case driving real r10k and puppet module install
- document local puppet in README and UI usage snippets

Reviewed-on: #145
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 13:17:51 +11:00
unkin-agent 8378701d9a merge master
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-10-17 12:55:45 +11:00
unkin-agent bc68eb75fc Add nuget v3 remote type (#141)
.NET builds restore from nuget.org directly because artifactapi has no NuGet remote type.

- add `nuget` package type proxying a NuGet v3 feed; `index.json` maps to the upstream service index
- add a `ServeRewriter` provider hook so cached JSON has upstream-origin URLs pointed back at the requesting host per request
- treat `.nupkg`/`.nuspec` as immutable and service, registration and flat container indexes as mutable
- add a `nuget.config` usage snippet to the UI and a nuget e2e caching case

Reviewed-on: #141
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 12:19:51 +11:00
unkin-agent a51323c725 Add goproxy local repositories for private Go modules
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-10-16 23:32:48 +11:00
unkin-agent 9161558334 Add optional OIDC bearer auth on remote/virtual config writes (#136)
ci/woodpecker/tag/docker Pipeline was successful
Remote and virtual configuration writes are unauthenticated, so anything that can reach the API can repoint or delete repos. Vault identity OIDC tokens give callers a verifiable identity.

- add `ARTIFACTAPI_OIDC_ISSUER`/`ARTIFACTAPI_OIDC_AUDIENCE`; unset issuer keeps auth off
- require a JWKS-verified bearer token on POST/PUT/DELETE of `/api/v2/remotes` and `/api/v2/virtuals`, 401 otherwise
- leave uploads, object deletes, probe, reads and `/v2` open
- discover the issuer lazily with a 10s client timeout; fail fast with 503 for 15s after a failed discovery
- log the verified `sub`; allow `Authorization` in CORS

Reviewed-on: #136
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-16 22:08:20 +11:00
33 changed files with 2347 additions and 18 deletions
+40 -1
View File
@@ -1,6 +1,6 @@
# ArtifactAPI
Caching proxy for package repositories. Single Go binary, 11 package types, content-addressable storage, managed by Terraform.
Caching proxy for package repositories. Single Go binary, 12 package types, content-addressable storage, managed by Terraform.
## Quick Start
@@ -33,6 +33,7 @@ API: `http://localhost:8000` | Frontend: `http://localhost:5173`
| `terraform` | `*/versions` | `*/download/*/*` |
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
| `cargo` | sparse index files (`config.json` synthesized) | `crates/*/*.crate` |
| `nuget` | `index.json` (service index), registration and flat container indexes | `.nupkg`, `.nuspec` |
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
Providers classify paths automatically. Users only configure what to proxy and TTLs.
@@ -261,6 +262,42 @@ there's nothing to provision. To bring your own key instead, point
`TF_SIGNING_KEY_PASSPHRASE`), which takes precedence over the generated one.
`TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
### Local Go module proxy
A local `goproxy` repo serves private modules over the GOPROXY protocol
(`@v/list`, `@latest`, `@v/<ver>.info|.mod|.zip`). Publish a version by
uploading its files under the case-encoded module path (uppercase as `!lower`):
```sh
for ext in zip mod info; do
curl -fsSL --upload-file v0.1.0.$ext \
https://artifactapi.k8s.syd1.au.unkin.net/api/v2/remotes/go-internal/files/example.unkin.net/hello/@v/v0.1.0.$ext
done
GOPROXY=https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/go-internal,https://proxy.golang.org,direct \
GONOSUMDB=example.unkin.net go get example.unkin.net/hello@latest
```
Build the zip with `golang.org/x/mod/zip` or reuse the files `go mod download`
leaves in `$(go env GOMODCACHE)/cache/download`. Uploads are validated: zips must
pass `zip.CheckZip` (every file under `module@version/`), `.mod` must declare the
module, and `.info` must match the version. `.info` is optional; it is
synthesized from the version when absent. `@v/list` lists versions with a `.mod`.
### Local puppet forge
A local `puppet` repo is a Forge: `PUT` a module tarball to
`/api/v2/remotes/<repo>/files/<owner>-<module>-<version>.tar.gz` and the Forge
v3 API (`/v3/modules`, `/v3/releases`, `/v3/files`) is served from each
release's `metadata.json` at `/api/v1/local/<repo>`. Versions are immutable.
```bash
puppet module build ./hello
curl -fsSL --upload-file pkg/unkin-hello-0.1.0.tar.gz \
https://artifactapi.k8s.syd1.au.unkin.net/api/v2/remotes/<repo>/files/unkin-hello-0.1.0.tar.gz
puppet module install unkin-hello \
--module_repository https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/<repo>
```
### Local docker registry
A local `docker` repo is a real container registry, not a mirror: it serves the
@@ -371,6 +408,8 @@ additive and idempotent for the same reason; a test enforces it.
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
| `ARTIFACTAPI_OIDC_ISSUER` | | OIDC issuer URL (e.g. Vault `https://vault.service.consul:8200/v1/identity/oidc`). When set, `POST /api/v2/remotes`, `POST /api/v2/virtuals` and `PUT`/`DELETE` on `/api/v2/{remotes,virtuals}/{name}` require `Authorization: Bearer <jwt>` verified against the issuer's JWKS. Uploads, object/file deletes, probe, reads and `/v2` stay open. Unset = no auth |
| `ARTIFACTAPI_OIDC_AUDIENCE` | | Comma-separated allowed `aud` values (e.g. the Vault OIDC role's client_id). Required when `ARTIFACTAPI_OIDC_ISSUER` is set |
## Development
+3 -2
View File
@@ -23,10 +23,11 @@ already-running stack.
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact per remote package type (generic, docker,
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy, cargo) proxied through the
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy, cargo, nuget) proxied through the
mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes
verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/`
- **Local uploads** — puppet (Forge v3 API via real `r10k` and `puppet module install`),
generic (upload/download), pypi (wheel + generated `simple/`
index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge
across two members.
+1
View File
@@ -30,6 +30,7 @@ func TestCachingPerProvider(t *testing.T) {
{"puppet", "puppet-releases/author-mod-1.0.0.tar.gz", "puppet-releases/author-mod-1.0.0.tar.gz"},
{"goproxy", "goproxy/example.com/mod/@v/v1.0.0.zip", "goproxy/example.com/mod/@v/v1.0.0.zip"},
{"cargo", "crates/mycrate/mycrate-1.0.0.crate", "crates/mycrate/mycrate-1.0.0.crate"},
{"nuget", "nuget/e2epkg/1.0.0/e2epkg.1.0.0.nupkg", "nuget/e2epkg/1.0.0/e2epkg.1.0.0.nupkg"},
{"terraform", "hashicorp/aws/download/pkg.zip", "v1/providers/hashicorp/aws/download/pkg.zip"},
{"docker", "library/testimg/blobs/blobdata", "v2/library/testimg/blobs/blobdata"},
}
+57
View File
@@ -0,0 +1,57 @@
//go:build dockere2e
package e2edocker
import (
"bytes"
"net/http"
"os"
"path/filepath"
"testing"
"golang.org/x/mod/module"
modzip "golang.org/x/mod/zip"
)
// TestLocalGoProxy publishes two module versions and reads them back over the
// GOPROXY protocol.
func TestLocalGoProxy(t *testing.T) {
createRepo(t, `{"name":"local-go","package_type":"goproxy","repo_type":"local"}`)
defer deleteRepo(t, "local-go")
const mod = "example.unkin.net/!hello"
gomod := []byte("module example.unkin.net/Hello\n\ngo 1.25\n")
var zip010 []byte
for _, ver := range []string{"v0.1.0", "v0.2.0"} {
dir := t.TempDir()
os.WriteFile(filepath.Join(dir, "go.mod"), gomod, 0o600)
os.WriteFile(filepath.Join(dir, "hello.go"), []byte("package hello\n"), 0o600)
var buf bytes.Buffer
if err := modzip.CreateFromDir(&buf, module.Version{Path: "example.unkin.net/Hello", Version: ver}, dir); err != nil {
t.Fatal(err)
}
if ver == "v0.1.0" {
zip010 = buf.Bytes()
}
uploadFile(t, "local-go", mod+"/@v/"+ver+".zip", buf.Bytes(), "application/zip")
uploadFile(t, "local-go", mod+"/@v/"+ver+".mod", gomod, "text/plain")
}
resp, body := doRequest(t, http.MethodPut, api("/api/v2/remotes/local-go/files/"+mod+"/@v/v0.3.0.zip"), []byte("junk"), "application/zip")
if resp.StatusCode != http.StatusBadRequest {
t.Fatalf("invalid zip: status %d: %s", resp.StatusCode, body)
}
for path, want := range map[string]string{
"/@v/list": "v0.1.0\nv0.2.0\n",
"/@latest": `{"Version":"v0.2.0"}`,
"/@v/v0.1.0.info": `{"Version":"v0.1.0"}`,
"/@v/v0.1.0.mod": string(gomod),
"/@v/v0.1.0.zip": string(zip010),
} {
resp, body := doRequest(t, http.MethodGet, api("/api/v1/local/local-go/"+mod+path), nil, "")
if resp.StatusCode != http.StatusOK || string(body) != want {
t.Errorf("%s: status %d body %q", path, resp.StatusCode, body)
}
}
}
+88
View File
@@ -0,0 +1,88 @@
//go:build dockere2e
package e2edocker
import (
"archive/tar"
"bytes"
"compress/gzip"
"net/http"
"os"
"os/exec"
"strings"
"testing"
"time"
)
const puppetHelloMetadata = `{"name":"unkin-hello","version":"0.1.0","author":"unkin","summary":"e2e","license":"Apache-2.0","source":"https://example.invalid","dependencies":[{"name":"puppetlabs/stdlib","version_requirement":">= 4.0.0 < 10.0.0"}]}`
func puppetModuleTarball(t *testing.T) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for name, body := range map[string]string{
"unkin-hello-0.1.0/metadata.json": puppetHelloMetadata,
"unkin-hello-0.1.0/manifests/init.pp": "class hello {}\n",
} {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body)), ModTime: time.Now()}); err != nil {
t.Fatal(err)
}
_, _ = tw.Write([]byte(body))
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
// TestLocalPuppetRealClient publishes a module to a puppet local and installs
// it with the real r10k and puppet module tool. The stdlib dependency is not on
// a local-only forge, so puppet runs with --ignore-dependencies.
func TestLocalPuppetRealClient(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := strings.TrimRight(os.Getenv("ARTIFACTAPI_INTERNAL"), "/")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
createRepo(t, `{"name":"local-puppet","package_type":"puppet","repo_type":"local"}`)
defer deleteRepo(t, "local-puppet")
upload := api("/api/v2/remotes/local-puppet/files/unkin-hello-0.1.0.tar.gz")
defer doRequest(t, http.MethodDelete, api("/api/v2/remotes/local-puppet/files/v3/files/unkin-hello-0.1.0.tar.gz"), nil, "")
tarball := puppetModuleTarball(t)
if resp, body := doRequest(t, http.MethodPut, upload, tarball, "application/gzip"); resp.StatusCode != http.StatusCreated {
t.Fatalf("upload: %d %s", resp.StatusCode, body)
}
if resp, _ := doRequest(t, http.MethodPut, upload, tarball, "application/gzip"); resp.StatusCode != http.StatusConflict {
t.Fatalf("re-upload of an existing version: %d, want 409", resp.StatusCode)
}
if resp, body := getEventually(t, api("/api/v1/local/local-puppet/v3/releases/unkin-hello-0.1.0"), 15*time.Second); resp.StatusCode != http.StatusOK || !strings.Contains(string(body), `"file_md5"`) {
t.Fatalf("release never indexed: %d %s", resp.StatusCode, body)
}
forge := internal + "/api/v1/local/local-puppet"
run := func(image, script string) string {
t.Helper()
out, err := exec.Command("docker", "run", "--rm", "--network", network, "--entrypoint", "sh", image, "-c", "set -eu; "+script).CombinedOutput()
if err != nil {
t.Fatalf("%s failed: %v\n%s", image, err, out)
}
return string(out)
}
out := run("ghcr.io/voxpupuli/r10k:latest", "cd /tmp && printf 'forge:\\n baseurl: "+forge+"\\n' > r10k.yaml && "+
"printf \"forge '"+forge+"'\\nmod 'unkin-hello', '0.1.0'\\n\" > Puppetfile && "+
"r10k puppetfile install -c r10k.yaml --verbose && cat modules/hello/metadata.json")
if !strings.Contains(out, `"unkin-hello"`) {
t.Fatalf("r10k did not install unkin-hello:\n%s", out)
}
out = run("ghcr.io/openvoxproject/openvoxagent:latest", "export PATH=/opt/puppetlabs/bin:$PATH; puppet module install unkin-hello --module_repository "+forge+" --ignore-dependencies --target-dir /tmp/mods && puppet module list --color=false --modulepath /tmp/mods")
if !strings.Contains(out, "unkin-hello (v0.1.0)") {
t.Fatalf("puppet module install did not install unkin-hello:\n%s", out)
}
}
+4
View File
@@ -7,7 +7,9 @@ require (
github.com/cavaliergopher/rpm v1.3.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/coreos/go-oidc/v3 v3.21.0
github.com/go-chi/chi/v5 v5.3.0
github.com/go-jose/go-jose/v4 v4.1.4
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.10.0
github.com/klauspost/compress v1.19.2
@@ -18,6 +20,7 @@ require (
github.com/testcontainers/testcontainers-go/modules/redis v0.42.0
github.com/ulikunitz/xz v0.5.16
golang.org/x/crypto v0.54.0
golang.org/x/mod v0.37.0
golang.org/x/sync v0.22.0
golang.org/x/time v0.15.0
gopkg.in/yaml.v3 v3.0.1
@@ -102,6 +105,7 @@ require (
go.uber.org/atomic v1.11.0 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/text v0.40.0 // indirect
gopkg.in/ini.v1 v1.67.2 // indirect
+10
View File
@@ -44,6 +44,8 @@ github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
github.com/containerd/platforms v0.2.1 h1:zvwtM3rz2YHPQsF2CHYM8+KtB5dvhISiXh5ZpSBQv6A=
github.com/containerd/platforms v0.2.1/go.mod h1:XHCb+2/hzowdiut9rkudds9bE5yJ7npe7dG/wG+uFPw=
github.com/coreos/go-oidc/v3 v3.21.0 h1:wZo4Q9Pum8dYEj0eMUPrqR+kvuGkeUplbLpNCkBqoWM=
github.com/coreos/go-oidc/v3 v3.21.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26KVTCI9SrY4=
github.com/cpuguy83/dockercfg v0.3.2 h1:DlJTyZGBDlXqUZ2Dk2Q3xHs/FtnooJJVaad2S9GKorA=
github.com/cpuguy83/dockercfg v0.3.2/go.mod h1:sugsbF4//dDlL/i+S+rtpIWp+5h0BHJHfjj5/jFyUJc=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
@@ -67,6 +69,8 @@ github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeO
github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE=
github.com/go-chi/chi/v5 v5.3.0 h1:halUjDxhshgXHMrao5bB8eNBXo/rnzwr8m5m36glehM=
github.com/go-chi/chi/v5 v5.3.0/go.mod h1:R+tYY2hNuVUUjxoPtqUdgBqevM9s9njzkTLutVsOCto=
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -223,8 +227,12 @@ golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -241,6 +249,8 @@ golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
+13 -1
View File
@@ -1,6 +1,7 @@
package v1
import (
"bytes"
"errors"
"fmt"
"io"
@@ -105,13 +106,24 @@ func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
}
defer result.Reader.Close()
var body io.Reader = result.Reader
if sr, ok := prov.(provider.ServeRewriter); ok && prov.Classify(path) == provider.Mutable {
raw, err := io.ReadAll(result.Reader)
if err != nil {
http.Error(w, "bad gateway", http.StatusBadGateway)
return
}
raw = sr.RewriteServed(raw, *remote, fmt.Sprintf("%s://%s", scheme(r), r.Host))
body, result.Size = bytes.NewReader(raw), int64(len(raw))
}
w.Header().Set("Content-Type", result.ContentType)
w.Header().Set("X-Artifact-Source", result.Source)
if result.Size > 0 {
w.Header().Set("X-Artifact-Size", fmt.Sprintf("%d", result.Size))
}
w.WriteHeader(http.StatusOK)
io.Copy(w, result.Reader)
io.Copy(w, body)
}
func (h *ProxyHandler) handleProxyHead(w http.ResponseWriter, r *http.Request) {
+38 -1
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"net/http"
"os"
"github.com/go-chi/chi/v5"
@@ -83,7 +84,22 @@ func (h *LocalHandler) uploadValidated(w http.ResponseWriter, r *http.Request, r
return
}
result, err := h.cas.Store(r.Context(), r.Body, contentType)
var body io.Reader = r.Body
if cv, ok := prov.(provider.ContentValidator); ok {
f, err := spoolBody(r.Body)
if err != nil {
http.Error(w, fmt.Sprintf("read upload: %v", err), http.StatusInternalServerError)
return
}
defer discard(f)
if err := cv.ValidateContent(storagePath, f.Name()); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
body = f
}
result, err := h.cas.Store(r.Context(), body, contentType)
if err != nil {
http.Error(w, fmt.Sprintf("store failed: %v", err), http.StatusInternalServerError)
return
@@ -110,6 +126,27 @@ func (h *LocalHandler) uploadValidated(w http.ResponseWriter, r *http.Request, r
writeJSON(w, http.StatusCreated, uploader.UploadResponse(storagePath, result.ContentHash, result.SizeBytes))
}
func spoolBody(r io.Reader) (*os.File, error) {
f, err := os.CreateTemp("", "artifactapi-upload-*")
if err != nil {
return nil, err
}
if _, err := io.Copy(f, r); err != nil {
discard(f)
return nil, err
}
if _, err := f.Seek(0, io.SeekStart); err != nil {
discard(f)
return nil, err
}
return f, nil
}
func discard(f *os.File) {
_ = f.Close()
_ = os.Remove(f.Name())
}
func (h *LocalHandler) uploadGeneric(w http.ResponseWriter, r *http.Request, remote *models.Remote, filePath string) {
existing, err := h.db.GetLocalFile(r.Context(), remote.Name, filePath)
if err != nil {
+55
View File
@@ -0,0 +1,55 @@
package v2
import (
"context"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
_ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// TestLocalUploadContentRejected verifies a ContentValidator provider rejects
// bad bytes with 400 before anything is stored.
func TestLocalUploadContentRejected(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
ctx := context.Background()
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
const repo = "go-local-validate"
if err := db.CreateRemote(ctx, &models.Remote{Name: repo, PackageType: models.PackageGoProxy, RepoType: models.RepoTypeLocal}); err != nil {
t.Fatal(err)
}
router := chi.NewRouter()
router.Route("/remotes/{name}/files", func(r chi.Router) {
r.Put("/*", NewLocalHandler(db, nil).Routes().ServeHTTP)
})
cases := map[string]string{
"example.unkin.net/hello/@v/v0.1.0.zip": "not a zip",
"example.unkin.net/hello/@v/v0.1.0.mod": "module example.unkin.net/other\n",
"example.unkin.net/hello/@v/v0.1.0.info": `{"Version":"v9.9.9"}`,
"example.unkin.net/Hello/@v/v0.1.0.mod": "module example.unkin.net/Hello\n",
}
for path, body := range cases {
w := httptest.NewRecorder()
router.ServeHTTP(w, httptest.NewRequest("PUT", "/remotes/"+repo+"/files/"+path, strings.NewReader(body)))
if w.Code != 400 {
t.Errorf("%s: status %d, want 400 (%s)", path, w.Code, w.Body.String())
}
if f, _ := db.GetLocalFile(ctx, repo, path); f != nil {
t.Errorf("%s: rejected upload was recorded", path)
}
}
}
+158
View File
@@ -0,0 +1,158 @@
package auth
import (
"context"
"errors"
"log/slog"
"net/http"
"slices"
"strings"
"sync"
"time"
"github.com/coreos/go-oidc/v3/oidc"
)
// BearerOIDC verifies `Authorization: Bearer <jwt>` against an OIDC issuer's
// JWKS. Discovery is lazy so startup never depends on the issuer being
// reachable; a failed discovery is not retried until the backoff elapses.
type BearerOIDC struct {
issuer string
audiences []string
timeout time.Duration
backoff time.Duration
discover func(context.Context, string) (*oidc.Provider, error)
mu sync.Mutex
verifier *oidc.IDTokenVerifier
inflight chan struct{}
failedAt time.Time
}
var errIssuerUnavailable = errors.New("oidc issuer unavailable")
// NewBearerOIDC returns nil when issuer is empty, meaning auth is disabled.
func NewBearerOIDC(issuer string, audiences []string) *BearerOIDC {
if issuer == "" {
return nil
}
return &BearerOIDC{issuer: issuer, audiences: audiences, timeout: 10 * time.Second, backoff: 15 * time.Second, discover: oidc.NewProvider}
}
// getVerifier runs at most one discovery at a time without holding the lock
// across network I/O; concurrent callers wait for it, and only callers within
// the post-failure backoff fail fast.
func (b *BearerOIDC) getVerifier(ctx context.Context) (*oidc.IDTokenVerifier, error) {
b.mu.Lock()
if v := b.verifier; v != nil {
b.mu.Unlock()
return v, nil
}
if done := b.inflight; done != nil {
b.mu.Unlock()
select {
case <-done:
case <-ctx.Done():
return nil, ctx.Err()
}
b.mu.Lock()
defer b.mu.Unlock()
if b.verifier == nil {
return nil, errIssuerUnavailable
}
return b.verifier, nil
}
if time.Since(b.failedAt) < b.backoff {
b.mu.Unlock()
return nil, errIssuerUnavailable
}
done := make(chan struct{})
b.inflight = done
b.mu.Unlock()
v, err := b.discoverOnce(done)
if err != nil {
slog.Error("oidc discovery failed", "issuer", b.issuer, "error", err)
return nil, err
}
return v, nil
}
// discoverOnce releases inflight and waiters on every exit, a panic included,
// recording failedAt unless discovery succeeded.
func (b *BearerOIDC) discoverOnce(done chan struct{}) (v *oidc.IDTokenVerifier, err error) {
defer func() {
b.mu.Lock()
b.inflight = nil
if v == nil {
b.failedAt = time.Now()
}
b.verifier = v
b.mu.Unlock()
close(done)
}()
// The provider keeps this context and client for JWKS refreshes, so the
// deadline lives on the client rather than the context.
client := &http.Client{Timeout: b.timeout}
p, err := b.discover(oidc.ClientContext(context.Background(), client), b.issuer)
if err != nil {
return nil, err
}
return p.Verifier(&oidc.Config{SkipClientIDCheck: true}), nil
}
// Require gates every request through next on a valid token. A nil receiver
// passes everything through.
func (b *BearerOIDC) Require(next http.Handler) http.Handler {
if b == nil {
return next
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
scheme, raw, _ := strings.Cut(strings.TrimSpace(r.Header.Get("Authorization")), " ")
raw = strings.TrimSpace(raw)
if !strings.EqualFold(scheme, "Bearer") || raw == "" {
unauthorized(w, "missing bearer token")
return
}
v, err := b.getVerifier(r.Context())
if err != nil {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
_, _ = w.Write([]byte(`{"error":"token issuer unavailable"}`))
return
}
tok, err := v.Verify(r.Context(), raw)
if err != nil {
slog.Info("bearer token rejected", "error", err)
unauthorized(w, "invalid token")
return
}
if len(b.audiences) > 0 && !slices.ContainsFunc(tok.Audience, func(a string) bool { return slices.Contains(b.audiences, a) }) {
slog.Info("bearer token rejected", "error", "audience not allowed", "aud", tok.Audience)
unauthorized(w, "invalid token")
return
}
slog.Info("bearer token accepted", "sub", tok.Subject, "method", r.Method, "path", r.URL.Path)
next.ServeHTTP(w, r)
})
}
// RequireWrites gates only mutating methods.
func (b *BearerOIDC) RequireWrites(next http.Handler) http.Handler {
gated := b.Require(next)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet, http.MethodHead, http.MethodOptions:
next.ServeHTTP(w, r)
default:
gated.ServeHTTP(w, r)
}
})
}
func unauthorized(w http.ResponseWriter, msg string) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("WWW-Authenticate", "Bearer")
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte(`{"error":"` + msg + `"}`))
}
+255
View File
@@ -0,0 +1,255 @@
package auth
import (
"context"
"crypto/rand"
"crypto/rsa"
"encoding/base64"
"net/http"
"net/http/httptest"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/coreos/go-oidc/v3/oidc"
"github.com/go-jose/go-jose/v4"
"github.com/go-jose/go-jose/v4/jwt"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
func serve(h http.Handler, method, token string) int {
if token == "" {
return serveAuth(h, method, "")
}
return serveAuth(h, method, "Bearer "+token)
}
func serveAuth(h http.Handler, method, authz string) int {
r := httptest.NewRequest(method, "/x", nil)
if authz != "" {
r.Header.Set("Authorization", authz)
}
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
return w.Code
}
func claims(iss string) jwt.Claims {
now := time.Now()
return jwt.Claims{
Issuer: iss,
Subject: "ci",
Audience: jwt.Audience{"artifactapi"},
IssuedAt: jwt.NewNumericDate(now.Add(-time.Minute)),
Expiry: jwt.NewNumericDate(now.Add(time.Hour)),
}
}
func hs256Token(t *testing.T, c jwt.Claims) string {
sig, err := jose.NewSigner(jose.SigningKey{Algorithm: jose.HS256, Key: []byte("0123456789abcdef0123456789abcdef")},
(&jose.SignerOptions{}).WithType("JWT").WithHeader("kid", "k1"))
if err != nil {
t.Fatal(err)
}
tok, err := jwt.Signed(sig).Claims(c).Serialize()
if err != nil {
t.Fatal(err)
}
return tok
}
func noneToken(iss string) string {
enc := base64.RawURLEncoding.EncodeToString
return enc([]byte(`{"alg":"none","typ":"JWT","kid":"k1"}`)) + "." +
enc([]byte(`{"iss":"`+iss+`","sub":"ci","aud":"artifactapi","exp":`+strconv.FormatInt(time.Now().Add(time.Hour).Unix(), 10)+`}`)) + "."
}
var ok = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) })
func TestBearerOIDC(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
h := NewBearerOIDC(iss.URL, []string{"other", "artifactapi"}).Require(ok)
otherKey, _ := rsa.GenerateKey(rand.Reader, 2048)
cases := []struct {
name string
token string
want int
}{
{"valid", iss.Token(nil, "artifactapi", "ci", time.Hour), http.StatusNoContent},
{"missing", "", http.StatusUnauthorized},
{"garbage", "not.a.jwt", http.StatusUnauthorized},
{"bad signature", iss.Token(otherKey, "artifactapi", "ci", time.Hour), http.StatusUnauthorized},
{"wrong aud", iss.Token(nil, "nope", "ci", time.Hour), http.StatusUnauthorized},
{"expired", iss.Token(nil, "artifactapi", "ci", -time.Hour), http.StatusUnauthorized},
{"wrong iss", iss.Sign(nil, claims("https://evil.example")), http.StatusUnauthorized},
{"hs256", hs256Token(t, claims(iss.URL)), http.StatusUnauthorized},
{"alg none", noneToken(iss.URL), http.StatusUnauthorized},
}
for _, c := range cases {
if got := serve(h, http.MethodPost, c.token); got != c.want {
t.Errorf("%s: status %d, want %d", c.name, got, c.want)
}
}
valid := iss.Token(nil, "artifactapi", "ci", time.Hour)
headers := []struct {
name string
authz string
want int
}{
{"lowercase scheme", "bearer " + valid, http.StatusNoContent},
{"extra whitespace", " Bearer " + valid + " ", http.StatusNoContent},
{"empty value", "Bearer ", http.StatusUnauthorized},
{"scheme only", "Bearer", http.StatusUnauthorized},
{"basic scheme", "Basic " + valid, http.StatusUnauthorized},
}
for _, c := range headers {
if got := serveAuth(h, http.MethodPost, c.authz); got != c.want {
t.Errorf("%s: status %d, want %d", c.name, got, c.want)
}
}
}
func TestBearerOIDCHangingIssuerBacksOff(t *testing.T) {
var hits atomic.Int32
release := make(chan struct{})
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits.Add(1)
select {
case <-r.Context().Done():
case <-release:
}
}))
defer srv.Close()
defer close(release)
b := NewBearerOIDC(srv.URL, nil)
b.timeout = 100 * time.Millisecond
b.backoff = time.Hour
h := b.Require(ok)
start := time.Now()
if got := serve(h, http.MethodPost, "x.y.z"); got != http.StatusServiceUnavailable {
t.Fatalf("hanging issuer: status %d, want 503", got)
}
if d := time.Since(start); d > 2*time.Second {
t.Fatalf("hanging issuer: took %s, timeout not applied", d)
}
if got := serve(h, http.MethodPost, "x.y.z"); got != http.StatusServiceUnavailable {
t.Fatalf("during backoff: status %d, want 503", got)
}
if n := hits.Load(); n != 1 {
t.Fatalf("issuer hit %d times, want 1", n)
}
}
func TestBearerOIDCPanickingDiscoveryBacksOff(t *testing.T) {
b := NewBearerOIDC("http://issuer.invalid", nil)
b.backoff = time.Hour
b.discover = func(context.Context, string) (*oidc.Provider, error) { panic("boom") }
h := b.Require(ok)
func() {
defer func() { _ = recover() }()
serve(h, http.MethodPost, "x.y.z")
}()
got := make(chan int, 1)
go func() { got <- serve(h, http.MethodPost, "x.y.z") }()
select {
case code := <-got:
if code != http.StatusServiceUnavailable {
t.Fatalf("after panic: status %d, want 503", code)
}
case <-time.After(2 * time.Second):
t.Fatal("after panic: request hung on stale inflight discovery")
}
}
func TestBearerOIDCConcurrentFirstRequestsWaitForDiscovery(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
var hits atomic.Int32
inner := iss.Config.Handler
iss.Config.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/.well-known/openid-configuration" {
hits.Add(1)
time.Sleep(200 * time.Millisecond)
}
inner.ServeHTTP(w, r)
})
h := NewBearerOIDC(iss.URL, nil).Require(ok)
tok := iss.Token(nil, "artifactapi", "ci", time.Hour)
var wg sync.WaitGroup
codes := make([]int, 20)
for i := range codes {
wg.Add(1)
go func() {
defer wg.Done()
codes[i] = serve(h, http.MethodPost, tok)
}()
}
wg.Wait()
for i, c := range codes {
if c != http.StatusNoContent {
t.Errorf("request %d: status %d, want 204", i, c)
}
}
if n := hits.Load(); n != 1 {
t.Fatalf("discovery hit %d times, want 1", n)
}
}
func TestBearerOIDCRetriesAfterBackoff(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
b := NewBearerOIDC(iss.URL, nil)
b.failedAt = time.Now()
b.backoff = 50 * time.Millisecond
tok := iss.Token(nil, "artifactapi", "ci", time.Hour)
if got := serve(b.Require(ok), http.MethodPost, tok); got != http.StatusServiceUnavailable {
t.Fatalf("within backoff: status %d, want 503", got)
}
time.Sleep(60 * time.Millisecond)
if got := serve(b.Require(ok), http.MethodPost, tok); got != http.StatusNoContent {
t.Fatalf("after backoff: status %d, want 204", got)
}
}
func TestBearerOIDCIssuerDown(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
b := NewBearerOIDC(iss.URL, []string{"artifactapi"})
tok := iss.Token(nil, "artifactapi", "ci", time.Hour)
iss.Close()
if got := serve(b.Require(ok), http.MethodPost, tok); got != http.StatusServiceUnavailable {
t.Errorf("issuer down: status %d, want 503", got)
}
}
func TestBearerOIDCDisabled(t *testing.T) {
if got := serve(NewBearerOIDC("", nil).RequireWrites(ok), http.MethodDelete, ""); got != http.StatusNoContent {
t.Errorf("disabled: status %d, want 204", got)
}
}
func TestRequireWritesLeavesReadsOpen(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
h := NewBearerOIDC(iss.URL, []string{"artifactapi"}).RequireWrites(ok)
for _, m := range []string{http.MethodGet, http.MethodHead, http.MethodOptions} {
if got := serve(h, m, ""); got != http.StatusNoContent {
t.Errorf("%s: status %d, want 204", m, got)
}
}
for _, m := range []string{http.MethodPost, http.MethodPut, http.MethodDelete} {
if got := serve(h, m, ""); got != http.StatusUnauthorized {
t.Errorf("%s: status %d, want 401", m, got)
}
}
}
+19
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"os"
"strconv"
"strings"
)
type Config struct {
@@ -56,6 +57,10 @@ type Config struct {
GitHubAppInstallationID string
GitHubAppPrivateKey string
GitHubAppPrivateKeyPath string
// Bearer-JWT auth on configuration writes; empty issuer disables it.
OIDCIssuer string
OIDCAudiences []string
}
func (c *Config) DatabaseDSN() string {
@@ -90,6 +95,17 @@ func Load() (*Config, error) {
return nil, fmt.Errorf("invalid GITHUB_SYNC_POLL_INTERVAL: %w", err)
}
oidcIssuer := getenv("ARTIFACTAPI_OIDC_ISSUER", "")
var oidcAudiences []string
for _, a := range strings.Split(getenv("ARTIFACTAPI_OIDC_AUDIENCE", ""), ",") {
if a = strings.TrimSpace(a); a != "" {
oidcAudiences = append(oidcAudiences, a)
}
}
if oidcIssuer != "" && len(oidcAudiences) == 0 {
return nil, fmt.Errorf("ARTIFACTAPI_OIDC_AUDIENCE is required when ARTIFACTAPI_OIDC_ISSUER is set")
}
cfg := &Config{
ListenAddr: getenv("LISTEN_ADDR", ":8000"),
@@ -123,6 +139,9 @@ func Load() (*Config, error) {
GitHubAppInstallationID: getenv("GITHUB_APP_INSTALLATION_ID", ""),
GitHubAppPrivateKey: getenv("GITHUB_APP_PRIVATE_KEY", ""),
GitHubAppPrivateKeyPath: getenv("GITHUB_APP_PRIVATE_KEY_PATH", ""),
OIDCIssuer: oidcIssuer,
OIDCAudiences: oidcAudiences,
}
return cfg, nil
+17
View File
@@ -64,3 +64,20 @@ func TestLoadInvalidPort(t *testing.T) {
t.Error("expected error for invalid DBPORT")
}
}
func TestLoadOIDC(t *testing.T) {
t.Setenv("ARTIFACTAPI_OIDC_ISSUER", "https://vault.example/v1/identity/oidc")
t.Setenv("ARTIFACTAPI_OIDC_AUDIENCE", " a, ,b ")
cfg, err := Load()
if err != nil {
t.Fatalf("load: %v", err)
}
if len(cfg.OIDCAudiences) != 2 || cfg.OIDCAudiences[0] != "a" || cfg.OIDCAudiences[1] != "b" {
t.Errorf("audiences = %q", cfg.OIDCAudiences)
}
t.Setenv("ARTIFACTAPI_OIDC_AUDIENCE", "")
if _, err := Load(); err == nil {
t.Error("expected error for issuer without audience")
}
}
+46
View File
@@ -0,0 +1,46 @@
package database
import (
"context"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func (db *DB) InsertPuppetMetadata(ctx context.Context, meta *provider.PuppetMetadata) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO puppet_metadata (repo_name, file_path, content_hash, slug, version, metadata_json, file_size, file_md5)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8)
ON CONFLICT (repo_name, file_path) DO NOTHING
`, meta.RepoName, meta.FilePath, meta.ContentHash, meta.Slug, meta.Version, meta.MetadataJSON, meta.Size, meta.MD5)
return err
}
func (db *DB) DeletePuppetMetadata(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM puppet_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
// ListPuppetMetadataEntries returns a repo's releases, optionally narrowed to
// one module slug (empty slug lists every module).
func (db *DB) ListPuppetMetadataEntries(ctx context.Context, repoName, slug string) ([]provider.PuppetMetadata, error) {
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash, slug, version, metadata_json, file_size, file_md5, created_at
FROM puppet_metadata
WHERE repo_name = $1 AND ($2 = '' OR slug = $2)
ORDER BY slug, file_path
`, repoName, slug)
if err != nil {
return nil, err
}
defer rows.Close()
var result []provider.PuppetMetadata
for rows.Next() {
var m provider.PuppetMetadata
if err := rows.Scan(&m.RepoName, &m.FilePath, &m.ContentHash, &m.Slug, &m.Version, &m.MetadataJSON, &m.Size, &m.MD5, &m.CreatedAt); err != nil {
return nil, err
}
result = append(result, m)
}
return result, rows.Err()
}
+222
View File
@@ -0,0 +1,222 @@
package goproxy
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"slices"
"strings"
"golang.org/x/mod/modfile"
"golang.org/x/mod/module"
"golang.org/x/mod/semver"
modzip "golang.org/x/mod/zip"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
type versionFile struct {
module string
version string
ext string
}
// parseVersionFile splits "<escaped module>/@v/<version>.<ext>" into its parts.
func parseVersionFile(path string) (versionFile, error) {
escMod, file, ok := strings.Cut(strings.TrimPrefix(path, "/"), "/@v/")
if !ok || escMod == "" || strings.Contains(file, "/") {
return versionFile{}, fmt.Errorf("path %q must be <module>/@v/<version>.{zip,mod,info}", path)
}
dot := strings.LastIndex(file, ".")
if dot <= 0 {
return versionFile{}, fmt.Errorf("path %q must end in .zip, .mod or .info", path)
}
ext := file[dot+1:]
if ext != "zip" && ext != "mod" && ext != "info" {
return versionFile{}, fmt.Errorf("path %q must end in .zip, .mod or .info", path)
}
mod, err := module.UnescapePath(escMod)
if err != nil {
return versionFile{}, fmt.Errorf("module path must be case-encoded (uppercase as !lower): %w", err)
}
ver, err := module.UnescapeVersion(file[:dot])
if err != nil {
return versionFile{}, err
}
if err := module.Check(mod, ver); err != nil {
return versionFile{}, err
}
if semver.Canonical(ver) != ver {
return versionFile{}, fmt.Errorf("version %q is not canonical semver", ver)
}
return versionFile{module: mod, version: ver, ext: ext}, nil
}
func (v versionFile) storagePath() string {
escMod, _ := module.EscapePath(v.module)
escVer, _ := module.EscapeVersion(v.version)
return escMod + "/@v/" + escVer + "." + v.ext
}
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
vf, err := parseVersionFile(filePath)
if err != nil {
return "", "", err
}
sp := vf.storagePath()
return sp, p.ContentType(sp), nil
}
func (p *Provider) ValidateContent(storagePath, localFile string) error {
vf, err := parseVersionFile(storagePath)
if err != nil {
return err
}
switch vf.ext {
case "zip":
if _, err := modzip.CheckZip(module.Version{Path: vf.module, Version: vf.version}, localFile); err != nil {
return fmt.Errorf("invalid module zip (files must sit under %s@%s/): %w", vf.module, vf.version, err)
}
case "mod":
data, err := os.ReadFile(localFile)
if err != nil {
return err
}
mf, err := modfile.ParseLax("go.mod", data, nil)
if err != nil {
return fmt.Errorf("invalid go.mod: %w", err)
}
if mf.Module == nil || mf.Module.Mod.Path != vf.module {
return fmt.Errorf("go.mod must declare module %s", vf.module)
}
case "info":
data, err := os.ReadFile(localFile)
if err != nil {
return err
}
var info struct{ Version string }
if err := json.Unmarshal(data, &info); err != nil {
return fmt.Errorf("invalid .info JSON: %w", err)
}
if info.Version != vf.version {
return fmt.Errorf(".info Version %q does not match %s", info.Version, vf.version)
}
}
return nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
vf, _ := parseVersionFile(storagePath)
return map[string]any{
"module": vf.module,
"version": vf.version,
"path": storagePath,
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
// versions returns the canonical versions of escMod that have a .mod uploaded, sorted ascending.
func versions(ctx context.Context, files provider.FileStore, repoName, escMod string) (versions []string, infos map[string]bool, err error) {
entries, err := files.ListFilesByPrefix(ctx, repoName, escMod+"/@v/")
if err != nil {
return nil, nil, err
}
infos = map[string]bool{}
for _, e := range entries {
vf, err := parseVersionFile(e.FilePath)
if err != nil || vf.storagePath() != e.FilePath {
continue
}
switch vf.ext {
case "mod":
versions = append(versions, vf.version)
case "info":
infos[vf.version] = true
}
}
semver.Sort(versions)
return versions, infos, nil
}
func latest(vs []string) string {
for i := len(vs) - 1; i >= 0; i-- {
if semver.Prerelease(vs[i]) == "" {
return vs[i]
}
}
if len(vs) == 0 {
return ""
}
return vs[len(vs)-1]
}
func infoJSON(version string) []byte {
b, _ := json.Marshal(map[string]string{"Version": version})
return b
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, path string) bool {
path = strings.TrimPrefix(path, "/")
switch {
case strings.HasSuffix(path, "/@v/list"), strings.HasSuffix(path, "/@latest"):
case strings.HasSuffix(path, ".info"):
vf, err := parseVersionFile(path)
if err != nil {
return false
}
escMod, _ := module.EscapePath(vf.module)
vs, infos, err := versions(r.Context(), files, repoName, escMod)
if err != nil || infos[vf.version] || !slices.Contains(vs, vf.version) {
return false
}
default:
return false
}
body, err := p.GenerateLocalIndex(r.Context(), files, repoName, path)
if err != nil {
http.Error(w, err.Error(), http.StatusNotFound)
return true
}
w.Header().Set("Content-Type", p.ContentType(path))
w.WriteHeader(http.StatusOK)
_, _ = w.Write(body)
return true
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
path = strings.TrimPrefix(path, "/")
if escMod, ok := strings.CutSuffix(path, "/@v/list"); ok {
vs, _, err := versions(ctx, files, repoName, escMod)
if err != nil {
return nil, err
}
var b strings.Builder
for _, v := range vs {
b.WriteString(v + "\n")
}
return []byte(b.String()), nil
}
if escMod, ok := strings.CutSuffix(path, "/@latest"); ok {
vs, _, err := versions(ctx, files, repoName, escMod)
if err != nil {
return nil, err
}
v := latest(vs)
if v == "" {
return nil, fmt.Errorf("no versions of %s", escMod)
}
return infoJSON(v), nil
}
if strings.HasSuffix(path, ".info") {
vf, err := parseVersionFile(path)
if err != nil {
return nil, err
}
return infoJSON(vf.version), nil
}
return nil, fmt.Errorf("unsupported index path: %q", path)
}
+198
View File
@@ -0,0 +1,198 @@
package goproxy
import (
"archive/zip"
"context"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"golang.org/x/mod/module"
modzip "golang.org/x/mod/zip"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
type fakeFileStore struct{ paths []string }
func (f *fakeFileStore) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
func (f *fakeFileStore) ListFilesByPrefix(_ context.Context, _, prefix string) ([]provider.FileEntry, error) {
var out []provider.FileEntry
for _, p := range f.paths {
if strings.HasPrefix(p, prefix) {
out = append(out, provider.FileEntry{FilePath: p, ContentHash: "sha256:x"})
}
}
return out, nil
}
func TestValidateUpload(t *testing.T) {
p := &Provider{}
cases := []struct {
in, want string
ok bool
}{
{"example.unkin.net/hello/@v/v0.1.0.zip", "example.unkin.net/hello/@v/v0.1.0.zip", true},
{"/example.unkin.net/hello/@v/v0.1.0.mod", "example.unkin.net/hello/@v/v0.1.0.mod", true},
{"github.com/!burnt!sushi/toml/@v/v1.2.0.info", "github.com/!burnt!sushi/toml/@v/v1.2.0.info", true},
{"example.unkin.net/hello/v2/@v/v2.0.0-rc.1.zip", "example.unkin.net/hello/v2/@v/v2.0.0-rc.1.zip", true},
{"github.com/BurntSushi/toml/@v/v1.2.0.zip", "", false},
{"example.unkin.net/hello/@v/v0.1.zip", "", false},
{"example.unkin.net/hello/@v/v2.0.0.zip", "", false},
{"example.unkin.net/hello/@v/v0.1.0.tar.gz", "", false},
{"example.unkin.net/hello/v0.1.0.zip", "", false},
{"example.unkin.net/hello/@v/list", "", false},
}
for _, c := range cases {
got, ct, err := p.ValidateUpload(c.in)
if (err == nil) != c.ok {
t.Errorf("%s: err=%v, want ok=%v", c.in, err, c.ok)
continue
}
if c.ok && (got != c.want || ct == "") {
t.Errorf("%s: got %q %q, want %q", c.in, got, ct, c.want)
}
}
}
func writeZip(t *testing.T, files map[string]string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "m.zip")
f, err := os.Create(path)
if err != nil {
t.Fatal(err)
}
zw := zip.NewWriter(f)
for name, body := range files {
w, err := zw.Create(name)
if err != nil {
t.Fatal(err)
}
if _, err := w.Write([]byte(body)); err != nil {
t.Fatal(err)
}
}
if err := zw.Close(); err != nil {
t.Fatal(err)
}
if err := f.Close(); err != nil {
t.Fatal(err)
}
return path
}
func writeFile(t *testing.T, body string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "f")
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
return path
}
func TestValidateContentZip(t *testing.T) {
p := &Provider{}
sp := "example.unkin.net/hello/@v/v0.1.0.zip"
dir := t.TempDir()
os.WriteFile(filepath.Join(dir, "go.mod"), []byte("module example.unkin.net/hello\n"), 0o600)
os.WriteFile(filepath.Join(dir, "main.go"), []byte("package main\n"), 0o600)
good := filepath.Join(t.TempDir(), "good.zip")
f, _ := os.Create(good)
if err := modzip.CreateFromDir(f, module.Version{Path: "example.unkin.net/hello", Version: "v0.1.0"}, dir); err != nil {
t.Fatal(err)
}
f.Close()
if err := p.ValidateContent(sp, good); err != nil {
t.Errorf("valid zip rejected: %v", err)
}
bad := map[string]map[string]string{
"no prefix": {"go.mod": "module example.unkin.net/hello\n"},
"wrong version": {"example.unkin.net/hello@v0.2.0/go.mod": "module example.unkin.net/hello\n"},
"wrong module": {"example.unkin.net/other@v0.1.0/go.mod": "module example.unkin.net/other\n"},
}
for name, files := range bad {
if err := p.ValidateContent(sp, writeZip(t, files)); err == nil {
t.Errorf("%s: expected rejection", name)
}
}
if err := p.ValidateContent(sp, writeFile(t, "not a zip")); err == nil {
t.Error("non-zip accepted")
}
}
func TestValidateContentModInfo(t *testing.T) {
p := &Provider{}
mod := "example.unkin.net/hello/@v/v0.1.0.mod"
info := "example.unkin.net/hello/@v/v0.1.0.info"
cases := []struct {
path, body string
ok bool
}{
{mod, "module example.unkin.net/hello\n\ngo 1.25\n", true},
{mod, "module example.unkin.net/other\n", false},
{mod, "go 1.25\n", false},
{mod, "module (\n", false},
{info, `{"Version":"v0.1.0","Time":"2026-01-01T00:00:00Z"}`, true},
{info, `{"Version":"v0.2.0"}`, false},
{info, `nope`, false},
}
for _, c := range cases {
if err := p.ValidateContent(c.path, writeFile(t, c.body)); (err == nil) != c.ok {
t.Errorf("%s %q: err=%v, want ok=%v", c.path, c.body, err, c.ok)
}
}
}
func serve(t *testing.T, fs provider.FileStore, path string) (bool, int, string) {
t.Helper()
rec := httptest.NewRecorder()
handled := (&Provider{}).ServeLocalIndex(rec, httptest.NewRequest("GET", "/"+path, nil), fs, "go-local", path)
return handled, rec.Code, rec.Body.String()
}
func TestServeLocalIndex(t *testing.T) {
fs := &fakeFileStore{paths: []string{
"example.unkin.net/hello/@v/v0.10.0.mod",
"example.unkin.net/hello/@v/v0.10.0.zip",
"example.unkin.net/hello/@v/v0.2.0.mod",
"example.unkin.net/hello/@v/v0.2.0.zip",
"example.unkin.net/hello/@v/v0.2.0.info",
"example.unkin.net/hello/@v/v0.11.0-rc.1.mod",
"example.unkin.net/hello/@v/v0.3.0.zip",
"example.unkin.net/hello/v2/@v/v2.0.0.mod",
"example.unkin.net/!pre/@v/v1.0.0-beta.mod",
}}
if h, code, body := serve(t, fs, "example.unkin.net/hello/@v/list"); !h || code != 200 || body != "v0.2.0\nv0.10.0\nv0.11.0-rc.1\n" {
t.Errorf("list: %v %d %q", h, code, body)
}
if h, code, body := serve(t, fs, "example.unkin.net/hello/@latest"); !h || code != 200 || body != `{"Version":"v0.10.0"}` {
t.Errorf("latest: %v %d %q", h, code, body)
}
if h, _, body := serve(t, fs, "example.unkin.net/!pre/@latest"); !h || body != `{"Version":"v1.0.0-beta"}` {
t.Errorf("prerelease-only latest: %q", body)
}
if h, code, _ := serve(t, fs, "example.unkin.net/missing/@latest"); !h || code != 404 {
t.Errorf("missing latest: %v %d", h, code)
}
if h, code, body := serve(t, fs, "example.unkin.net/missing/@v/list"); !h || code != 200 || body != "" {
t.Errorf("missing list: %v %d %q", h, code, body)
}
if h, _, body := serve(t, fs, "example.unkin.net/hello/@v/v0.10.0.info"); !h || body != `{"Version":"v0.10.0"}` {
t.Errorf("synthesized info: %v %q", h, body)
}
if h, _, _ := serve(t, fs, "example.unkin.net/hello/@v/v0.2.0.info"); h {
t.Error("uploaded .info must be served from storage")
}
if h, _, _ := serve(t, fs, "example.unkin.net/hello/@v/v0.3.0.info"); h {
t.Error(".info without a .mod must not be synthesized")
}
if h, _, _ := serve(t, fs, "example.unkin.net/hello/@v/v0.10.0.zip"); h {
t.Error(".zip must be served from storage")
}
}
+85
View File
@@ -0,0 +1,85 @@
// Package nuget proxies a NuGet v3 feed: the service index, registration pages
// and flat container indexes are mutable, .nupkg/.nuspec downloads immutable.
package nuget
import (
"context"
"encoding/json"
"net/http"
"net/url"
"strings"
"git.unkin.net/unkin/artifactapi/internal/auth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func init() {
provider.Register(&Provider{})
}
// serviceIndex is the path clients use as their source URL; it maps to base_url.
const serviceIndex = "index.json"
type Provider struct{}
func (p *Provider) Type() models.PackageType { return models.PackageNuGet }
func (p *Provider) Classify(path string) provider.Mutability {
if isPackage(path) {
return provider.Immutable
}
return provider.Mutable
}
func (p *Provider) ContentType(path string) string {
switch {
case strings.HasSuffix(path, ".nupkg"):
return "application/octet-stream"
case strings.HasSuffix(path, ".nuspec"):
return "application/xml"
}
return "application/json"
}
// UpstreamURL maps index.json to base_url (the upstream service index) and any
// other path to the same path on the base_url's origin, where nuget.org serves
// v3-flatcontainer/ and v3/registration*/.
func (p *Provider) UpstreamURL(remote models.Remote, path string) string {
path = strings.TrimLeft(path, "/")
if path == serviceIndex {
return remote.BaseURL
}
return origin(remote.BaseURL) + "/" + path
}
func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
return nil, nil
}
// RewriteServed points upstream-origin URLs in cached JSON back at this remote.
// It runs per request because the URLs must name the requesting host.
func (p *Provider) RewriteServed(body []byte, remote models.Remote, proxyBaseURL string) []byte {
from := origin(remote.BaseURL)
if from == "" || !json.Valid(body) {
return body
}
to := strings.TrimRight(proxyBaseURL, "/") + "/api/v1/remote/" + remote.Name
return []byte(strings.ReplaceAll(string(body), from, to))
}
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
return auth.BasicHeaders(remote), nil
}
func origin(baseURL string) string {
u, err := url.Parse(baseURL)
if err != nil || u.Host == "" {
return ""
}
return u.Scheme + "://" + u.Host
}
func isPackage(path string) bool {
return strings.HasSuffix(path, ".nupkg") || strings.HasSuffix(path, ".nuspec")
}
+68
View File
@@ -0,0 +1,68 @@
package nuget
import (
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
var _ provider.ServeRewriter = (*Provider)(nil)
var nugetOrg = models.Remote{Name: "nuget-org", BaseURL: "https://api.nuget.org/v3/index.json"}
func TestClassify(t *testing.T) {
p := &Provider{}
tests := []struct {
path string
want provider.Mutability
}{
{"v3-flatcontainer/newtonsoft.json/13.0.3/newtonsoft.json.13.0.3.nupkg", provider.Immutable},
{"v3-flatcontainer/newtonsoft.json/13.0.3/newtonsoft.json.nuspec", provider.Immutable},
{"v3-flatcontainer/newtonsoft.json/index.json", provider.Mutable},
{"v3/registration5-gz-semver2/newtonsoft.json/index.json", provider.Mutable},
{"index.json", provider.Mutable},
}
for _, tt := range tests {
if got := p.Classify(tt.path); got != tt.want {
t.Errorf("Classify(%q) = %v, want %v", tt.path, got, tt.want)
}
}
}
func TestUpstreamURL(t *testing.T) {
p := &Provider{}
tests := []struct {
path, want string
}{
{"index.json", "https://api.nuget.org/v3/index.json"},
{"/index.json", "https://api.nuget.org/v3/index.json"},
{"v3-flatcontainer/serilog/index.json", "https://api.nuget.org/v3-flatcontainer/serilog/index.json"},
{"v3/registration5-gz-semver2/serilog/index.json", "https://api.nuget.org/v3/registration5-gz-semver2/serilog/index.json"},
}
for _, tt := range tests {
if got := p.UpstreamURL(nugetOrg, tt.path); got != tt.want {
t.Errorf("UpstreamURL(%q) = %q, want %q", tt.path, got, tt.want)
}
}
}
func TestRewriteServed(t *testing.T) {
p := &Provider{}
body := `{"resources":[` +
`{"@id":"https://api.nuget.org/v3-flatcontainer/","@type":"PackageBaseAddress/3.0.0"},` +
`{"@id":"https://azuresearch-usnc.nuget.org/query","@type":"SearchQueryService"}]}`
want := `{"resources":[` +
`{"@id":"http://aa.example:8000/api/v1/remote/nuget-org/v3-flatcontainer/","@type":"PackageBaseAddress/3.0.0"},` +
`{"@id":"https://azuresearch-usnc.nuget.org/query","@type":"SearchQueryService"}]}`
if got := string(p.RewriteServed([]byte(body), nugetOrg, "http://aa.example:8000/")); got != want {
t.Errorf("RewriteServed =\n%s\nwant\n%s", got, want)
}
if got := string(p.RewriteServed([]byte(body), models.Remote{Name: "x"}, "http://aa.example")); got != body {
t.Error("remote without a parseable base_url must pass the body through")
}
cert := "\x30\x82https://api.nuget.org"
if got := string(p.RewriteServed([]byte(cert), nugetOrg, "http://aa.example")); got != cert {
t.Error("non-JSON bodies must pass through untouched")
}
}
+38
View File
@@ -41,6 +41,11 @@ type LocalUploader interface {
UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any
}
// ContentValidator checks an upload's bytes (spooled to localFile) before it is stored.
type ContentValidator interface {
ValidateContent(storagePath, localFile string) error
}
type LocalIndexer interface {
ServeLocalIndex(w http.ResponseWriter, r *http.Request, files FileStore, repoName, path string) bool
GenerateLocalIndex(ctx context.Context, files FileStore, repoName, path string) ([]byte, error)
@@ -75,6 +80,13 @@ type RemoteServer interface {
ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store RemoteMetadataStore) bool
}
// ServeRewriter rewrites a cached mutable remote response per request, for
// indexes whose URLs must name the requesting host and so cannot be rewritten
// before caching.
type ServeRewriter interface {
RewriteServed(body []byte, remote models.Remote, proxyBaseURL string) []byte
}
// RemoteMetadataStore is the persistence surface a RemoteServer needs to cache
// and read the metadata it derives per upstream asset. *database.DB satisfies it.
type RemoteMetadataStore interface {
@@ -240,3 +252,29 @@ func Get(t models.PackageType) (Provider, error) {
func All() map[models.PackageType]Provider {
return registry
}
type PuppetMetadataStore interface {
InsertPuppetMetadata(ctx context.Context, meta *PuppetMetadata) error
}
type PuppetMetadataDeleter interface {
DeletePuppetMetadata(ctx context.Context, repoName, filePath string) error
}
type PuppetMetadataReader interface {
ListPuppetMetadataEntries(ctx context.Context, repoName, slug string) ([]PuppetMetadata, error)
}
// PuppetMetadata is a local module release; MetadataJSON is the tarball's
// metadata.json verbatim so Forge release objects carry every field.
type PuppetMetadata struct {
RepoName string
FilePath string
ContentHash string
Slug string
Version string
MetadataJSON string
Size int64
MD5 string
CreatedAt time.Time
}
+393
View File
@@ -0,0 +1,393 @@
package puppet
import (
"archive/tar"
"compress/gzip"
"context"
"crypto/md5"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/url"
"path"
"regexp"
"slices"
"strconv"
"strings"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/storage"
)
const filesPrefix = "v3/files/"
// releaseRe matches <owner>-<name>-<semver>, the Forge release slug and the
// tarball basename that `puppet module build` produces.
var releaseRe = regexp.MustCompile(`^([a-z0-9]+)-([a-z][a-z0-9_]*)-(\d+\.\d+\.\d+(?:-[0-9a-z.-]+)?(?:\+[0-9a-z.-]+)?)$`)
func parseRelease(s string) (slug, version string, ok bool) {
m := releaseRe.FindStringSubmatch(strings.ToLower(s))
if m == nil {
return "", "", false
}
return m[1] + "-" + m[2], m[3], true
}
func normalizeSlug(s string) string {
return strings.ToLower(strings.ReplaceAll(s, "/", "-"))
}
// ValidateUpload stores tarballs under v3/files/ so Forge file_uri downloads
// resolve straight to the stored file.
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
filename := strings.ToLower(path.Base(path.Clean("/" + filePath)))
if _, _, ok := parseRelease(strings.TrimSuffix(filename, ".tar.gz")); !ok || !strings.HasSuffix(filename, ".tar.gz") {
return "", "", fmt.Errorf("file must be named <owner>-<module>-<version>.tar.gz")
}
return filesPrefix + filename, "application/gzip", nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
return map[string]any{
"filename": path.Base(storagePath),
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs provider.BlobReader, db provider.MetadataStore) {
reader, size, err := blobs.Download(ctx, storage.BlobKey(strings.TrimPrefix(contentHash, "sha256:")))
if err != nil {
slog.Error("puppet metadata: download failed", "repo", repoName, "path", storagePath, "error", err)
return
}
defer func() { _ = reader.Close() }()
if size > maxModuleTarball {
slog.Error("puppet metadata: tarball too large", "repo", repoName, "path", storagePath, "size", size)
return
}
hash := md5.New()
src := io.TeeReader(io.LimitReader(reader, maxModuleTarball), hash)
meta, err := parseModule(src)
if err != nil {
slog.Error("puppet metadata: parse failed", "repo", repoName, "path", storagePath, "error", err)
return
}
if _, err := io.Copy(io.Discard, src); err != nil {
slog.Error("puppet metadata: read failed", "repo", repoName, "path", storagePath, "error", err)
return
}
slug, version, _ := parseRelease(strings.TrimSuffix(path.Base(storagePath), ".tar.gz"))
if meta.Slug != slug || meta.Version != version {
slog.Error("puppet metadata: metadata.json does not match filename", "repo", repoName, "path", storagePath, "name", meta.Slug, "version", meta.Version)
return
}
meta.RepoName = repoName
meta.FilePath = storagePath
meta.ContentHash = contentHash
meta.Size = size
meta.MD5 = hex.EncodeToString(hash.Sum(nil))
store, ok := db.(provider.PuppetMetadataStore)
if !ok {
slog.Error("puppet metadata: store does not support puppet metadata", "repo", repoName)
return
}
if err := store.InsertPuppetMetadata(ctx, meta); err != nil {
slog.Error("puppet metadata: insert failed", "repo", repoName, "path", storagePath, "error", err)
return
}
slog.Info("puppet metadata: parsed", "repo", repoName, "module", meta.Slug, "version", meta.Version)
}
func (p *Provider) AfterDelete(ctx context.Context, repoName, storagePath string, db provider.MetadataDeleter) error {
deleter, ok := db.(provider.PuppetMetadataDeleter)
if !ok {
return nil
}
return deleter.DeletePuppetMetadata(ctx, repoName, storagePath)
}
const maxMetadataJSON = 1 << 20
var (
maxModuleTarball int64 = 100 << 20
maxModuleScan int64 = 200 << 20
)
// parseModule reads the top-level <dir>/metadata.json from a module tarball,
// scanning at most maxModuleScan decompressed bytes.
func parseModule(r io.Reader) (*provider.PuppetMetadata, error) {
zr, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
scan := &io.LimitedReader{R: zr, N: maxModuleScan}
tr := tar.NewReader(scan)
for {
hdr, err := tr.Next()
if scan.N <= 0 {
return nil, errors.New("module tarball exceeds decompressed scan limit before metadata.json")
}
if err == io.EOF {
return nil, errors.New("no metadata.json in module tarball")
}
if err != nil {
return nil, err
}
dir, file := path.Split(strings.TrimPrefix(path.Clean(hdr.Name), "./"))
if hdr.Typeflag != tar.TypeReg || file != "metadata.json" || strings.Count(dir, "/") != 1 {
continue
}
body, err := io.ReadAll(io.LimitReader(tr, maxMetadataJSON))
if err != nil {
return nil, err
}
var md struct {
Name string `json:"name"`
Version string `json:"version"`
}
if err := json.Unmarshal(body, &md); err != nil {
return nil, fmt.Errorf("parse metadata.json: %w", err)
}
if md.Name == "" || md.Version == "" {
return nil, errors.New("metadata.json missing name or version")
}
return &provider.PuppetMetadata{Slug: normalizeSlug(md.Name), Version: md.Version, MetadataJSON: string(body)}, nil
}
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, reqPath string) bool {
clean := strings.TrimPrefix(path.Clean("/"+reqPath), "/")
var list func(slug string) ([]provider.PuppetMetadata, error)
if reader, ok := files.(provider.PuppetMetadataReader); ok {
list = func(slug string) ([]provider.PuppetMetadata, error) {
metas, err := reader.ListPuppetMetadataEntries(r.Context(), repoName, slug)
sortReleases(metas)
return metas, err
}
} else {
list = func(string) ([]provider.PuppetMetadata, error) {
return nil, errors.New("puppet metadata not available")
}
}
switch {
case clean == "v3/releases":
serveReleaseList(w, r, list)
case strings.HasPrefix(clean, "v3/releases/"):
slug, version, ok := parseRelease(strings.TrimPrefix(clean, "v3/releases/"))
if !ok {
notFound(w)
return true
}
metas, err := list(slug)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return true
}
for _, m := range metas {
if m.Version == version {
writeJSON(w, releaseObject(m))
return true
}
}
notFound(w)
case strings.HasPrefix(clean, "v3/modules/"):
slug := normalizeSlug(strings.TrimPrefix(clean, "v3/modules/"))
metas, err := list(slug)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return true
}
if len(metas) == 0 {
notFound(w)
return true
}
writeJSON(w, moduleObject(metas))
default:
return false
}
return true
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
return nil, fmt.Errorf("puppet local index generation for virtual repos not supported")
}
func serveReleaseList(w http.ResponseWriter, r *http.Request, list func(string) ([]provider.PuppetMetadata, error)) {
q := r.URL.Query()
slug := normalizeSlug(q.Get("module"))
limit, err := strconv.Atoi(q.Get("limit"))
if err != nil || limit <= 0 {
limit = 20
}
limit = min(limit, 100)
offset, err := strconv.Atoi(q.Get("offset"))
if err != nil || offset < 0 {
offset = 0
}
metas, err := list(slug)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
offset = min(offset, len(metas))
results := []map[string]any{}
for _, m := range metas[offset : offset+min(limit, len(metas)-offset)] {
results = append(results, releaseObject(m))
}
page := func(off int) any {
if off < 0 || off >= len(metas) && off != 0 {
return nil
}
v := url.Values{"limit": {strconv.Itoa(limit)}, "offset": {strconv.Itoa(off)}}
if slug != "" {
v.Set("module", slug)
}
return "/v3/releases?" + v.Encode()
}
var previous any
if offset > 0 {
previous = page(max(offset-limit, 0))
}
writeJSON(w, map[string]any{
"pagination": map[string]any{
"limit": limit,
"offset": offset,
"first": page(0),
"previous": previous,
"current": page(offset),
"next": page(offset + limit),
"total": len(metas),
},
"results": results,
})
}
func owner(slug string) map[string]any {
name, _, _ := strings.Cut(slug, "-")
return map[string]any{"uri": "/v3/users/" + name, "slug": name, "username": name}
}
func moduleRef(slug string) map[string]any {
_, name, _ := strings.Cut(slug, "-")
return map[string]any{"uri": "/v3/modules/" + slug, "slug": slug, "name": name, "deprecated_at": nil, "owner": owner(slug)}
}
const forgeTime = "2006-01-02 15:04:05 -0700"
func releaseObject(m provider.PuppetMetadata) map[string]any {
slug := m.Slug + "-" + m.Version
return map[string]any{
"uri": "/v3/releases/" + slug,
"slug": slug,
"module": moduleRef(m.Slug),
"version": m.Version,
"metadata": json.RawMessage(m.MetadataJSON),
"tags": []string{},
"supported": false,
"pdk": false,
"file_uri": "/" + m.FilePath,
"file_size": m.Size,
"file_md5": m.MD5,
"file_sha256": strings.TrimPrefix(m.ContentHash, "sha256:"),
"downloads": 0,
"readme": nil,
"changelog": nil,
"license": nil,
"created_at": m.CreatedAt.Format(forgeTime),
"updated_at": m.CreatedAt.Format(forgeTime),
"deleted_at": nil,
}
}
// moduleObject expects metas sorted newest version first.
func moduleObject(metas []provider.PuppetMetadata) map[string]any {
obj := moduleRef(metas[0].Slug)
releases := make([]map[string]any, 0, len(metas))
for _, m := range metas {
slug := m.Slug + "-" + m.Version
releases = append(releases, map[string]any{
"uri": "/v3/releases/" + slug, "slug": slug, "version": m.Version,
"file_uri": "/" + m.FilePath, "file_size": m.Size, "supported": false,
"created_at": m.CreatedAt.Format(forgeTime), "deleted_at": nil,
})
}
obj["current_release"] = releaseObject(metas[0])
obj["releases"] = releases
obj["downloads"] = 0
obj["created_at"] = metas[len(metas)-1].CreatedAt.Format(forgeTime)
obj["updated_at"] = metas[0].CreatedAt.Format(forgeTime)
return obj
}
func writeJSON(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
enc := json.NewEncoder(w)
enc.SetEscapeHTML(false)
_ = enc.Encode(v)
}
func notFound(w http.ResponseWriter) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusNotFound)
_, _ = io.WriteString(w, `{"message":"404 Not Found","errors":["The requested resource could not be found"]}`)
}
// sortReleases orders by module slug, then newest semver first.
func sortReleases(metas []provider.PuppetMetadata) {
slices.SortStableFunc(metas, func(a, b provider.PuppetMetadata) int {
if c := strings.Compare(a.Slug, b.Slug); c != 0 {
return c
}
return compareSemver(b.Version, a.Version)
})
}
func compareSemver(a, b string) int {
a, _, _ = strings.Cut(a, "+")
b, _, _ = strings.Cut(b, "+")
aCore, aPre, aHasPre := strings.Cut(a, "-")
bCore, bPre, bHasPre := strings.Cut(b, "-")
if c := compareIdents(strings.Split(aCore, "."), strings.Split(bCore, ".")); c != 0 {
return c
}
switch {
case aHasPre && !bHasPre:
return -1
case !aHasPre && bHasPre:
return 1
}
return compareIdents(strings.Split(aPre, "."), strings.Split(bPre, "."))
}
func compareIdents(a, b []string) int {
for i := 0; i < len(a) && i < len(b); i++ {
an, aErr := strconv.Atoi(a[i])
bn, bErr := strconv.Atoi(b[i])
var c int
switch {
case aErr == nil && bErr == nil:
c = an - bn
case aErr == nil:
c = -1
case bErr == nil:
c = 1
default:
c = strings.Compare(a[i], b[i])
}
if c != 0 {
return c
}
}
return len(a) - len(b)
}
+301
View File
@@ -0,0 +1,301 @@
package puppet
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/md5"
"encoding/hex"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func moduleTarball(t *testing.T, dir, metadata string) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for name, body := range map[string]string{dir + "/metadata.json": metadata, dir + "/manifests/init.pp": "class hello {}\n"} {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body))}); err != nil {
t.Fatal(err)
}
_, _ = tw.Write([]byte(body))
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
const helloMetadata = `{"name":"unkin-hello","version":"0.1.0","dependencies":[{"name":"puppetlabs/stdlib","version_requirement":">= 4.0.0"}]}`
type fakeBlobs struct{ data []byte }
func (f fakeBlobs) Download(context.Context, string) (io.ReadCloser, int64, error) {
return io.NopCloser(bytes.NewReader(f.data)), int64(len(f.data)), nil
}
type fakeStore struct{ inserted *provider.PuppetMetadata }
func (f *fakeStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
func (f *fakeStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
func (f *fakeStore) InsertPuppetMetadata(_ context.Context, m *provider.PuppetMetadata) error {
f.inserted = m
return nil
}
type fakeReader struct{ metas []provider.PuppetMetadata }
func (f fakeReader) ListPuppetMetadataEntries(_ context.Context, _, slug string) ([]provider.PuppetMetadata, error) {
var out []provider.PuppetMetadata
for _, m := range f.metas {
if slug == "" || m.Slug == slug {
out = append(out, m)
}
}
return out, nil
}
func (fakeReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
return nil, nil
}
func (fakeReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
func TestValidateUpload(t *testing.T) {
p := &Provider{}
sp, ct, err := p.ValidateUpload("some/dir/Unkin-hello-0.1.0.tar.gz")
if err != nil || sp != "v3/files/unkin-hello-0.1.0.tar.gz" || ct != "application/gzip" {
t.Errorf("sp=%q ct=%q err=%v", sp, ct, err)
}
for _, bad := range []string{"hello-0.1.0.tar.gz", "unkin-hello-0.1.tar.gz", "unkin-hello-0.1.0.zip", "unkin-Hello_x-0.1.0.tgz"} {
if _, _, err := p.ValidateUpload(bad); err == nil {
t.Errorf("%q accepted", bad)
}
}
}
func TestAfterUploadRecordsMetadata(t *testing.T) {
raw := moduleTarball(t, "unkin-hello-0.1.0", helloMetadata)
store := &fakeStore{}
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
m := store.inserted
if m == nil {
t.Fatal("nothing inserted")
}
sum := md5.Sum(raw)
if m.Slug != "unkin-hello" || m.Version != "0.1.0" || m.MD5 != hex.EncodeToString(sum[:]) || m.Size != int64(len(raw)) || m.MetadataJSON != helloMetadata {
t.Errorf("got %+v", m)
}
}
func TestAfterUploadRejectsMismatchedFilename(t *testing.T) {
raw := moduleTarball(t, "unkin-hello-0.2.0", `{"name":"unkin-hello","version":"0.2.0"}`)
store := &fakeStore{}
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
if store.inserted != nil {
t.Error("mismatched version indexed")
}
}
func TestParseModuleRequiresTopLevelMetadata(t *testing.T) {
if _, err := parseModule(bytes.NewReader(moduleTarball(t, "a/b", helloMetadata))); err == nil {
t.Error("nested metadata.json accepted")
}
if _, err := parseModule(bytes.NewReader(moduleTarball(t, "x", `{"name":"unkin-hello"}`))); err == nil {
t.Error("missing version accepted")
}
}
func tarballOf(t *testing.T, entries ...*tar.Header) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for _, h := range entries {
if err := tw.WriteHeader(h); err != nil {
t.Fatal(err)
}
if h.Typeflag == tar.TypeReg {
body := helloMetadata
if !strings.HasSuffix(h.Name, "metadata.json") {
body = strings.Repeat("x", int(h.Size))
}
_, _ = tw.Write([]byte(body))
}
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
func TestParseModuleSkipsNonRegularMetadata(t *testing.T) {
raw := tarballOf(t,
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json", Typeflag: tar.TypeSymlink, Linkname: "/etc/passwd"},
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json/", Typeflag: tar.TypeDir, Mode: 0o755},
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json", Typeflag: tar.TypeReg, Mode: 0o644, Size: int64(len(helloMetadata))},
)
meta, err := parseModule(bytes.NewReader(raw))
if err != nil || meta.Slug != "unkin-hello" || meta.Version != "0.1.0" {
t.Fatalf("got %v, %v", meta, err)
}
}
func TestParseModuleCapsDecompressedScan(t *testing.T) {
defer func(n int64) { maxModuleScan = n }(maxModuleScan)
maxModuleScan = 64 << 10
raw := tarballOf(t,
&tar.Header{Name: "unkin-hello-0.1.0/big", Typeflag: tar.TypeReg, Mode: 0o644, Size: 1 << 20},
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json", Typeflag: tar.TypeReg, Mode: 0o644, Size: int64(len(helloMetadata))},
)
if _, err := parseModule(bytes.NewReader(raw)); err == nil || !strings.Contains(err.Error(), "scan limit") {
t.Fatalf("expected scan limit error, got %v", err)
}
maxModuleScan = 2 << 20
if _, err := parseModule(bytes.NewReader(raw)); err != nil {
t.Fatalf("under cap: %v", err)
}
}
func TestAfterUploadCapsTarballSize(t *testing.T) {
defer func(n int64) { maxModuleTarball = n }(maxModuleTarball)
raw := moduleTarball(t, "unkin-hello-0.1.0", helloMetadata)
maxModuleTarball = int64(len(raw)) - 1
store := &fakeStore{}
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
if store.inserted != nil {
t.Error("oversized tarball indexed")
}
maxModuleTarball = int64(len(raw))
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
if store.inserted == nil {
t.Error("tarball at cap not indexed")
}
}
func TestReleaseListClampsPagination(t *testing.T) {
files := releases("0.1.0", "0.2.0", "0.3.0")
for _, q := range []string{"offset=9223372036854775807", "offset=9223372036854775807&limit=9223372036854775807", "offset=-5&limit=-5", "limit=9223372036854775807"} {
code, body := get(t, files, "v3/releases?module=unkin-hello&"+q)
if code != http.StatusOK {
t.Fatalf("%s: code %d", q, code)
}
pg := body["pagination"].(map[string]any)
if l := pg["limit"].(float64); l < 1 || l > 100 {
t.Errorf("%s: limit %v", q, l)
}
if o := pg["offset"].(float64); o < 0 || o > 3 {
t.Errorf("%s: offset %v", q, o)
}
}
_, body := get(t, files, "v3/releases?module=unkin-hello&offset=9223372036854775807")
if len(body["results"].([]any)) != 0 || body["pagination"].(map[string]any)["next"] != nil {
t.Errorf("past-end page: %v", body)
}
}
func get(t *testing.T, files provider.FileStore, target string) (int, map[string]any) {
t.Helper()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/x/"+target, nil)
path, _, _ := strings.Cut(target, "?")
if !(&Provider{}).ServeLocalIndex(rec, req, files, "forge", path) {
return 0, nil
}
var body map[string]any
_ = json.Unmarshal(rec.Body.Bytes(), &body)
return rec.Code, body
}
func releases(versions ...string) fakeReader {
var f fakeReader
for i, v := range versions {
f.metas = append(f.metas, provider.PuppetMetadata{
FilePath: "v3/files/unkin-hello-" + v + ".tar.gz", ContentHash: "sha256:feed", Slug: "unkin-hello",
Version: v, MetadataJSON: `{"name":"unkin-hello","version":"` + v + `"}`, MD5: "m" + v,
CreatedAt: time.Unix(int64(i), 0),
})
}
f.metas = append(f.metas, provider.PuppetMetadata{Slug: "other-mod", Version: "9.9.9", MetadataJSON: `{}`})
return f
}
func TestReleaseListPaginatesNewestFirst(t *testing.T) {
files := releases("0.1.0", "0.10.0", "0.2.0", "1.0.0-rc.1", "1.0.0")
code, body := get(t, files, "v3/releases?module=unkin/hello&limit=2")
if code != http.StatusOK {
t.Fatalf("code %d", code)
}
results := body["results"].([]any)
pg := body["pagination"].(map[string]any)
if len(results) != 2 || results[0].(map[string]any)["version"] != "1.0.0" || results[1].(map[string]any)["version"] != "1.0.0-rc.1" {
t.Fatalf("page 1: %v", results)
}
if pg["total"] != float64(5) || pg["next"] != "/v3/releases?limit=2&module=unkin-hello&offset=2" || pg["previous"] != nil {
t.Errorf("pagination: %v", pg)
}
_, body = get(t, files, "v3/releases?module=unkin-hello&limit=2&offset=4")
results = body["results"].([]any)
pg = body["pagination"].(map[string]any)
if len(results) != 1 || results[0].(map[string]any)["version"] != "0.1.0" || pg["next"] != nil || pg["previous"] != "/v3/releases?limit=2&module=unkin-hello&offset=2" {
t.Errorf("last page: %v %v", results, pg)
}
_, body = get(t, files, "v3/releases?module=nobody-none")
if len(body["results"].([]any)) != 0 {
t.Error("unknown module returned results")
}
}
func TestReleaseObject(t *testing.T) {
code, body := get(t, releases("0.1.0"), "v3/releases/unkin-hello-0.1.0")
if code != http.StatusOK {
t.Fatalf("code %d", code)
}
if body["file_uri"] != "/v3/files/unkin-hello-0.1.0.tar.gz" || body["file_md5"] != "m0.1.0" || body["file_sha256"] != "feed" {
t.Errorf("release: %v", body)
}
if body["metadata"].(map[string]any)["name"] != "unkin-hello" || body["module"].(map[string]any)["owner"].(map[string]any)["username"] != "unkin" {
t.Errorf("release refs: %v", body)
}
if code, _ := get(t, releases("0.1.0"), "v3/releases/unkin-hello-0.2.0"); code != http.StatusNotFound {
t.Errorf("missing version: %d", code)
}
}
func TestModuleObject(t *testing.T) {
code, body := get(t, releases("0.9.0", "0.10.0"), "v3/modules/unkin-hello")
if code != http.StatusOK {
t.Fatalf("code %d", code)
}
if body["slug"] != "unkin-hello" || body["name"] != "hello" || body["current_release"].(map[string]any)["version"] != "0.10.0" || len(body["releases"].([]any)) != 2 {
t.Errorf("module: %v", body)
}
if code, _ := get(t, releases(), "v3/modules/unkin-missing"); code != http.StatusNotFound {
t.Errorf("missing module: %d", code)
}
}
func TestFilesFallThrough(t *testing.T) {
if code, _ := get(t, releases(), "v3/files/unkin-hello-0.1.0.tar.gz"); code != 0 {
t.Error("file downloads must fall through to the stored file")
}
}
func TestCompareSemver(t *testing.T) {
ordered := []string{"0.1.0", "0.2.0", "0.10.0", "1.0.0-alpha", "1.0.0-alpha.1", "1.0.0-beta", "1.0.0-beta.2", "1.0.0-beta.11", "1.0.0", "1.0.1+build"}
for i := 1; i < len(ordered); i++ {
if compareSemver(ordered[i-1], ordered[i]) >= 0 || compareSemver(ordered[i], ordered[i-1]) <= 0 {
t.Errorf("%s should sort before %s", ordered[i-1], ordered[i])
}
}
if compareSemver("1.0.0+a", "1.0.0+b") != 0 {
t.Error("build metadata must not affect precedence")
}
}
+1 -1
View File
@@ -12,7 +12,7 @@ func cors(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Access-Control-Allow-Origin", "*")
w.Header().Set("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type, Authorization")
if r.Method == http.MethodOptions {
w.WriteHeader(http.StatusNoContent)
return
+80
View File
@@ -0,0 +1,80 @@
package server
import (
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
)
func TestServerOIDCWriteAuth(t *testing.T) {
requireStack(t)
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
cfg := *testSrv.cfg
cfg.OIDCIssuer = iss.URL
cfg.OIDCAudiences = []string{"artifactapi"}
s := *testSrv
s.cfg = &cfg
ts := httptest.NewServer(s.routes())
defer ts.Close()
good := iss.Token(nil, "artifactapi", "ci", time.Hour)
do := func(method, path, body, token string) int {
t.Helper()
rq, _ := http.NewRequest(method, ts.URL+path, strings.NewReader(body))
rq.Header.Set("Content-Type", "application/json")
if token != "" {
rq.Header.Set("Authorization", "Bearer "+token)
}
resp, err := http.DefaultClient.Do(rq)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
return resp.StatusCode
}
expect := func(want int, method, path, body, token string) {
t.Helper()
if got := do(method, path, body, token); got != want {
t.Errorf("%s %s: status %d, want %d", method, path, got, want)
}
}
remote := `{"name":"srv-oidc","package_type":"generic","repo_type":"local"}`
virtual := `{"name":"srv-oidc-v","package_type":"generic","members":["srv-oidc"]}`
expect(401, "POST", "/api/v2/remotes", remote, "")
expect(401, "POST", "/api/v2/remotes", remote, iss.Token(nil, "other", "ci", time.Hour))
expect(201, "POST", "/api/v2/remotes", remote, good)
expect(401, "PUT", "/api/v2/remotes/srv-oidc", `{"package_type":"generic"}`, "")
expect(401, "DELETE", "/api/v2/remotes/srv-oidc", "", "")
expect(401, "POST", "/api/v2/virtuals", virtual, "")
expect(201, "POST", "/api/v2/virtuals", virtual, good)
expect(401, "PUT", "/api/v2/virtuals/srv-oidc-v", virtual, "")
expect(401, "DELETE", "/api/v2/virtuals/srv-oidc-v", "", "")
expect(200, "GET", "/api/v2/remotes", "", "")
expect(200, "GET", "/api/v2/remotes/srv-oidc", "", "")
expect(200, "GET", "/api/v2/virtuals/srv-oidc-v", "", "")
expect(201, "PUT", "/api/v2/remotes/srv-oidc/files/a.bin", "payload", "")
expect(200, "GET", "/api/v2/remotes/srv-oidc/files/a.bin", "", "")
if got := do("DELETE", "/api/v2/remotes/srv-oidc/files/a.bin", "", ""); got >= 400 {
t.Errorf("delete file: status %d", got)
}
if got := do("POST", "/api/v2/probe", `{}`, ""); got == 401 {
t.Error("probe gated")
}
if got := do("GET", "/v2/", "", ""); got == 401 {
t.Error("docker /v2 gated")
}
expect(204, "DELETE", "/api/v2/virtuals/srv-oidc-v", "", good)
expect(204, "DELETE", "/api/v2/remotes/srv-oidc", "", good)
}
+5 -2
View File
@@ -15,6 +15,7 @@ import (
tfregistry "git.unkin.net/unkin/artifactapi/internal/api/terraform"
v1 "git.unkin.net/unkin/artifactapi/internal/api/v1"
v2 "git.unkin.net/unkin/artifactapi/internal/api/v2"
"git.unkin.net/unkin/artifactapi/internal/auth"
"git.unkin.net/unkin/artifactapi/internal/cache"
"git.unkin.net/unkin/artifactapi/internal/config"
"git.unkin.net/unkin/artifactapi/internal/database"
@@ -28,6 +29,7 @@ import (
_ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
_ "git.unkin.net/unkin/artifactapi/internal/provider/helm"
_ "git.unkin.net/unkin/artifactapi/internal/provider/npm"
_ "git.unkin.net/unkin/artifactapi/internal/provider/nuget"
_ "git.unkin.net/unkin/artifactapi/internal/provider/puppet"
_ "git.unkin.net/unkin/artifactapi/internal/provider/pypi"
"git.unkin.net/unkin/artifactapi/internal/provider/rpm"
@@ -186,10 +188,11 @@ func (s *Server) routes() chi.Router {
statsHandler := v2.NewStatsHandler(s.db)
eventsHandler := v2.NewEventsHandler()
probeHandler := v2.NewProbeHandler(s.engine, s.db)
writeAuth := auth.NewBearerOIDC(s.cfg.OIDCIssuer, s.cfg.OIDCAudiences)
r.Route("/api/v2", func(r chi.Router) {
r.Mount("/remotes", remotesHandler.Routes())
r.Mount("/virtuals", virtualsHandler.Routes())
r.Mount("/remotes", writeAuth.RequireWrites(remotesHandler.Routes()))
r.Mount("/virtuals", writeAuth.RequireWrites(virtualsHandler.Routes()))
r.Mount("/health", healthHandler.Routes())
r.Mount("/stats", statsHandler.Routes())
r.Mount("/events", eventsHandler.Routes())
+73
View File
@@ -0,0 +1,73 @@
package testsupport
import (
"crypto/rand"
"crypto/rsa"
"encoding/json"
"net/http"
"net/http/httptest"
"time"
"github.com/go-jose/go-jose/v4"
"github.com/go-jose/go-jose/v4/jwt"
)
// OIDCIssuer is a minimal OIDC issuer serving discovery and JWKS, signing
// RS256 tokens with a generated key.
type OIDCIssuer struct {
*httptest.Server
key *rsa.PrivateKey
}
func NewOIDCIssuer() *OIDCIssuer {
key, err := rsa.GenerateKey(rand.Reader, 2048)
if err != nil {
panic(err)
}
iss := &OIDCIssuer{key: key}
mux := http.NewServeMux()
mux.HandleFunc("/.well-known/openid-configuration", func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{
"issuer": iss.URL,
"jwks_uri": iss.URL + "/keys",
"id_token_signing_alg_values_supported": []string{"RS256"},
})
})
mux.HandleFunc("/keys", func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(jose.JSONWebKeySet{Keys: []jose.JSONWebKey{
{Key: &key.PublicKey, KeyID: "k1", Algorithm: "RS256", Use: "sig"},
}})
})
iss.Server = httptest.NewServer(mux)
return iss
}
// Token signs a token for aud and sub. Pass a nil key to use the issuer's own;
// expiry is relative to now.
func (o *OIDCIssuer) Token(key *rsa.PrivateKey, aud, sub string, expiry time.Duration) string {
now := time.Now()
return o.Sign(key, jwt.Claims{
Issuer: o.URL,
Subject: sub,
Audience: jwt.Audience{aud},
IssuedAt: jwt.NewNumericDate(now.Add(-time.Minute)),
Expiry: jwt.NewNumericDate(now.Add(expiry)),
})
}
// Sign signs arbitrary claims with RS256. Pass a nil key to use the issuer's own.
func (o *OIDCIssuer) Sign(key *rsa.PrivateKey, claims jwt.Claims) string {
if key == nil {
key = o.key
}
sig, err := jose.NewSigner(jose.SigningKey{Algorithm: jose.RS256, Key: key},
(&jose.SignerOptions{}).WithType("JWT").WithHeader("kid", "k1"))
if err != nil {
panic(err)
}
tok, err := jwt.Signed(sig).Claims(claims).Serialize()
if err != nil {
panic(err)
}
return tok
}
+15
View File
@@ -0,0 +1,15 @@
CREATE TABLE IF NOT EXISTS puppet_metadata (
id BIGSERIAL PRIMARY KEY,
repo_name TEXT NOT NULL,
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
slug TEXT NOT NULL,
version TEXT NOT NULL,
metadata_json TEXT NOT NULL,
file_size BIGINT NOT NULL DEFAULT 0,
file_md5 TEXT NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(repo_name, file_path)
);
CREATE INDEX IF NOT EXISTS idx_puppet_metadata_repo_slug ON puppet_metadata(repo_name, slug);
+2
View File
@@ -17,6 +17,7 @@ const (
PackageTerraform PackageType = "terraform"
PackageGoProxy PackageType = "goproxy"
PackageCargo PackageType = "cargo"
PackageNuGet PackageType = "nuget"
PackageGitHubRPM PackageType = "github_rpm"
PackageGitHubDeb PackageType = "github_deb"
PackageGitHubAlpine PackageType = "github_alpine"
@@ -35,6 +36,7 @@ var validPackageTypes = map[PackageType]bool{
PackageTerraform: true,
PackageGoProxy: true,
PackageCargo: true,
PackageNuGet: true,
PackageGitHubRPM: true,
PackageGitHubDeb: true,
PackageGitHubAlpine: true,
+1
View File
@@ -19,6 +19,7 @@ func TestPackageTypeValid(t *testing.T) {
models.PackageTerraform,
models.PackageGoProxy,
models.PackageCargo,
models.PackageNuGet,
models.PackageGitHubRPM,
models.PackageGitHubDeb,
models.PackageGitHubAlpine,
+59 -10
View File
@@ -218,15 +218,38 @@ sudo apk add --allow-untrusted <package>`,
];
case 'goproxy':
return [
{
title: 'Point the Go module proxy here',
language: 'bash',
code: `export GOPROXY=${proxy}
return isLocal
? [
{
title: 'Fetch private modules from this proxy',
language: 'bash',
code: `export GOPROXY=${url}/api/v1/local/${name},https://proxy.golang.org,direct
export GONOSUMDB=example.unkin.net
go get example.unkin.net/hello@latest`,
note: 'Set GONOSUMDB (or GOPRIVATE) to your module prefix: private modules are not in sum.golang.org.',
},
{
title: 'Publish a module version',
language: 'bash',
code: `# build the zip with golang.org/x/mod/zip (CreateFromDir), or reuse
# the .zip/.mod/.info from $(go env GOMODCACHE)/cache/download after go mod download
MOD=example.unkin.net/hello VER=v0.1.0
for ext in zip mod info; do
curl -fsSL --upload-file $VER.$ext \\
${url}/api/v2/remotes/${name}/files/$MOD/@v/$VER.$ext
done`,
note: 'Paths use the case-encoded module path (uppercase as !lower). Zips must hold every file under module@version/ and are rejected otherwise; .info is optional.',
},
]
: [
{
title: 'Point the Go module proxy here',
language: 'bash',
code: `export GOPROXY=${proxy}
go mod download`,
note: 'Append ,direct to fall back to VCS for modules this proxy does not cover.',
},
];
note: 'Append ,direct to fall back to VCS for modules this proxy does not cover.',
},
];
case 'cargo':
return [
@@ -243,14 +266,40 @@ index = "sparse+${proxy}/"`,
},
];
case 'nuget':
return [
{
title: 'Use this feed instead of nuget.org',
language: 'xml',
code: `<!-- nuget.config -->
<configuration>
<packageSources>
<clear />
<add key="${name}" value="${proxy}/index.json" protocolVersion="3" />
</packageSources>
</configuration>`,
note: 'Serve artifactapi over HTTPS: NuGet rejects the repository signatures resource over plain HTTP.',
},
];
case 'puppet':
return [
{
title: 'Install a module from the Forge proxy',
title: isLocal ? 'Install a module from this forge' : 'Install a module from the Forge proxy',
language: 'bash',
code: `puppet module install <author>-<module> \\
--module_repository ${proxy}`,
--module_repository ${isLocal ? `${url}/api/v1/local/${name}` : proxy}`,
},
...(isLocal
? [
{
title: 'Publish a module (versions are immutable)',
language: 'bash',
code: `puppet module build && curl -fsSL --upload-file pkg/<author>-<module>-<version>.tar.gz \\
${url}/api/v2/remotes/${name}/files/<author>-<module>-<version>.tar.gz`,
},
]
: []),
];
case 'deb':
+1
View File
@@ -18,6 +18,7 @@ const typeColors: Record<string, 'blue' | 'green' | 'yellow' | 'red' | 'default'
terraform: 'blue',
goproxy: 'green',
cargo: 'yellow',
nuget: 'blue',
};
export function Locals() {
+1
View File
@@ -18,6 +18,7 @@ const typeColors: Record<string, 'blue' | 'green' | 'yellow' | 'red' | 'default'
terraform: 'blue',
goproxy: 'green',
cargo: 'yellow',
nuget: 'blue',
};
export function Remotes() {