Compare commits

...

5 Commits

Author SHA1 Message Date
unkin-agent 5a2bfdf73a merge master
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-10-17 13:20:00 +11:00
unkin-agent 9b91aa39a6 Add local puppet forge (#145)
Puppet modules can only be proxied from a remote Forge; there is nowhere to publish internal modules for puppet module install, r10k and g10k.

- add puppet local uploads at <owner>-<module>-<version>.tar.gz, rejecting existing versions
- record each release's metadata.json, md5 and sha256 in a puppet_metadata table, streaming the tarball under size caps
- serve Forge v3 /v3/modules, paginated /v3/releases, /v3/releases/<slug> and /v3/files
- add an e2e-docker case driving real r10k and puppet module install
- document local puppet in README and UI usage snippets

Reviewed-on: #145
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 13:17:51 +11:00
unkin-agent 8378701d9a merge master
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-10-17 12:55:45 +11:00
unkin-agent bc68eb75fc Add nuget v3 remote type (#141)
.NET builds restore from nuget.org directly because artifactapi has no NuGet remote type.

- add `nuget` package type proxying a NuGet v3 feed; `index.json` maps to the upstream service index
- add a `ServeRewriter` provider hook so cached JSON has upstream-origin URLs pointed back at the requesting host per request
- treat `.nupkg`/`.nuspec` as immutable and service, registration and flat container indexes as mutable
- add a `nuget.config` usage snippet to the UI and a nuget e2e caching case

Reviewed-on: #141
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-10-17 12:19:51 +11:00
unkin-agent a51323c725 Add goproxy local repositories for private Go modules
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-10-16 23:32:48 +11:00
26 changed files with 1729 additions and 15 deletions
+38 -1
View File
@@ -1,6 +1,6 @@
# ArtifactAPI
Caching proxy for package repositories. Single Go binary, 11 package types, content-addressable storage, managed by Terraform.
Caching proxy for package repositories. Single Go binary, 12 package types, content-addressable storage, managed by Terraform.
## Quick Start
@@ -33,6 +33,7 @@ API: `http://localhost:8000` | Frontend: `http://localhost:5173`
| `terraform` | `*/versions` | `*/download/*/*` |
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
| `cargo` | sparse index files (`config.json` synthesized) | `crates/*/*.crate` |
| `nuget` | `index.json` (service index), registration and flat container indexes | `.nupkg`, `.nuspec` |
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
Providers classify paths automatically. Users only configure what to proxy and TTLs.
@@ -261,6 +262,42 @@ there's nothing to provision. To bring your own key instead, point
`TF_SIGNING_KEY_PASSPHRASE`), which takes precedence over the generated one.
`TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
### Local Go module proxy
A local `goproxy` repo serves private modules over the GOPROXY protocol
(`@v/list`, `@latest`, `@v/<ver>.info|.mod|.zip`). Publish a version by
uploading its files under the case-encoded module path (uppercase as `!lower`):
```sh
for ext in zip mod info; do
curl -fsSL --upload-file v0.1.0.$ext \
https://artifactapi.k8s.syd1.au.unkin.net/api/v2/remotes/go-internal/files/example.unkin.net/hello/@v/v0.1.0.$ext
done
GOPROXY=https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/go-internal,https://proxy.golang.org,direct \
GONOSUMDB=example.unkin.net go get example.unkin.net/hello@latest
```
Build the zip with `golang.org/x/mod/zip` or reuse the files `go mod download`
leaves in `$(go env GOMODCACHE)/cache/download`. Uploads are validated: zips must
pass `zip.CheckZip` (every file under `module@version/`), `.mod` must declare the
module, and `.info` must match the version. `.info` is optional; it is
synthesized from the version when absent. `@v/list` lists versions with a `.mod`.
### Local puppet forge
A local `puppet` repo is a Forge: `PUT` a module tarball to
`/api/v2/remotes/<repo>/files/<owner>-<module>-<version>.tar.gz` and the Forge
v3 API (`/v3/modules`, `/v3/releases`, `/v3/files`) is served from each
release's `metadata.json` at `/api/v1/local/<repo>`. Versions are immutable.
```bash
puppet module build ./hello
curl -fsSL --upload-file pkg/unkin-hello-0.1.0.tar.gz \
https://artifactapi.k8s.syd1.au.unkin.net/api/v2/remotes/<repo>/files/unkin-hello-0.1.0.tar.gz
puppet module install unkin-hello \
--module_repository https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/<repo>
```
### Local docker registry
A local `docker` repo is a real container registry, not a mirror: it serves the
+3 -2
View File
@@ -23,10 +23,11 @@ already-running stack.
- **Repository lifecycle** — add / change / delete for remote, local and virtual repos.
- **Caching** — one immutable artifact per remote package type (generic, docker,
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy, cargo) proxied through the
helm, pypi, npm, rpm, alpine, puppet, terraform, goproxy, cargo, nuget) proxied through the
mock upstream: first fetch `X-Artifact-Source: remote`, second `cache`, bytes
verified against the origin fixture.
- **Local uploads** — generic (upload/download), pypi (wheel + generated `simple/`
- **Local uploads** — puppet (Forge v3 API via real `r10k` and `puppet module install`),
generic (upload/download), pypi (wheel + generated `simple/`
index), rpm (real package + **automatic repodata** generation).
- **Virtual repositories** — pypi simple-index merge and helm `index.yaml` merge
across two members.
+1
View File
@@ -30,6 +30,7 @@ func TestCachingPerProvider(t *testing.T) {
{"puppet", "puppet-releases/author-mod-1.0.0.tar.gz", "puppet-releases/author-mod-1.0.0.tar.gz"},
{"goproxy", "goproxy/example.com/mod/@v/v1.0.0.zip", "goproxy/example.com/mod/@v/v1.0.0.zip"},
{"cargo", "crates/mycrate/mycrate-1.0.0.crate", "crates/mycrate/mycrate-1.0.0.crate"},
{"nuget", "nuget/e2epkg/1.0.0/e2epkg.1.0.0.nupkg", "nuget/e2epkg/1.0.0/e2epkg.1.0.0.nupkg"},
{"terraform", "hashicorp/aws/download/pkg.zip", "v1/providers/hashicorp/aws/download/pkg.zip"},
{"docker", "library/testimg/blobs/blobdata", "v2/library/testimg/blobs/blobdata"},
}
+57
View File
@@ -0,0 +1,57 @@
//go:build dockere2e
package e2edocker
import (
"bytes"
"net/http"
"os"
"path/filepath"
"testing"
"golang.org/x/mod/module"
modzip "golang.org/x/mod/zip"
)
// TestLocalGoProxy publishes two module versions and reads them back over the
// GOPROXY protocol.
func TestLocalGoProxy(t *testing.T) {
createRepo(t, `{"name":"local-go","package_type":"goproxy","repo_type":"local"}`)
defer deleteRepo(t, "local-go")
const mod = "example.unkin.net/!hello"
gomod := []byte("module example.unkin.net/Hello\n\ngo 1.25\n")
var zip010 []byte
for _, ver := range []string{"v0.1.0", "v0.2.0"} {
dir := t.TempDir()
os.WriteFile(filepath.Join(dir, "go.mod"), gomod, 0o600)
os.WriteFile(filepath.Join(dir, "hello.go"), []byte("package hello\n"), 0o600)
var buf bytes.Buffer
if err := modzip.CreateFromDir(&buf, module.Version{Path: "example.unkin.net/Hello", Version: ver}, dir); err != nil {
t.Fatal(err)
}
if ver == "v0.1.0" {
zip010 = buf.Bytes()
}
uploadFile(t, "local-go", mod+"/@v/"+ver+".zip", buf.Bytes(), "application/zip")
uploadFile(t, "local-go", mod+"/@v/"+ver+".mod", gomod, "text/plain")
}
resp, body := doRequest(t, http.MethodPut, api("/api/v2/remotes/local-go/files/"+mod+"/@v/v0.3.0.zip"), []byte("junk"), "application/zip")
if resp.StatusCode != http.StatusBadRequest {
t.Fatalf("invalid zip: status %d: %s", resp.StatusCode, body)
}
for path, want := range map[string]string{
"/@v/list": "v0.1.0\nv0.2.0\n",
"/@latest": `{"Version":"v0.2.0"}`,
"/@v/v0.1.0.info": `{"Version":"v0.1.0"}`,
"/@v/v0.1.0.mod": string(gomod),
"/@v/v0.1.0.zip": string(zip010),
} {
resp, body := doRequest(t, http.MethodGet, api("/api/v1/local/local-go/"+mod+path), nil, "")
if resp.StatusCode != http.StatusOK || string(body) != want {
t.Errorf("%s: status %d body %q", path, resp.StatusCode, body)
}
}
}
+88
View File
@@ -0,0 +1,88 @@
//go:build dockere2e
package e2edocker
import (
"archive/tar"
"bytes"
"compress/gzip"
"net/http"
"os"
"os/exec"
"strings"
"testing"
"time"
)
const puppetHelloMetadata = `{"name":"unkin-hello","version":"0.1.0","author":"unkin","summary":"e2e","license":"Apache-2.0","source":"https://example.invalid","dependencies":[{"name":"puppetlabs/stdlib","version_requirement":">= 4.0.0 < 10.0.0"}]}`
func puppetModuleTarball(t *testing.T) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for name, body := range map[string]string{
"unkin-hello-0.1.0/metadata.json": puppetHelloMetadata,
"unkin-hello-0.1.0/manifests/init.pp": "class hello {}\n",
} {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body)), ModTime: time.Now()}); err != nil {
t.Fatal(err)
}
_, _ = tw.Write([]byte(body))
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
// TestLocalPuppetRealClient publishes a module to a puppet local and installs
// it with the real r10k and puppet module tool. The stdlib dependency is not on
// a local-only forge, so puppet runs with --ignore-dependencies.
func TestLocalPuppetRealClient(t *testing.T) {
network := os.Getenv("COMPOSE_NETWORK")
internal := strings.TrimRight(os.Getenv("ARTIFACTAPI_INTERNAL"), "/")
if network == "" || internal == "" {
t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; run via scripts/docker-e2e.sh")
}
if _, err := exec.LookPath("docker"); err != nil {
t.Skip("docker not available on the test host")
}
createRepo(t, `{"name":"local-puppet","package_type":"puppet","repo_type":"local"}`)
defer deleteRepo(t, "local-puppet")
upload := api("/api/v2/remotes/local-puppet/files/unkin-hello-0.1.0.tar.gz")
defer doRequest(t, http.MethodDelete, api("/api/v2/remotes/local-puppet/files/v3/files/unkin-hello-0.1.0.tar.gz"), nil, "")
tarball := puppetModuleTarball(t)
if resp, body := doRequest(t, http.MethodPut, upload, tarball, "application/gzip"); resp.StatusCode != http.StatusCreated {
t.Fatalf("upload: %d %s", resp.StatusCode, body)
}
if resp, _ := doRequest(t, http.MethodPut, upload, tarball, "application/gzip"); resp.StatusCode != http.StatusConflict {
t.Fatalf("re-upload of an existing version: %d, want 409", resp.StatusCode)
}
if resp, body := getEventually(t, api("/api/v1/local/local-puppet/v3/releases/unkin-hello-0.1.0"), 15*time.Second); resp.StatusCode != http.StatusOK || !strings.Contains(string(body), `"file_md5"`) {
t.Fatalf("release never indexed: %d %s", resp.StatusCode, body)
}
forge := internal + "/api/v1/local/local-puppet"
run := func(image, script string) string {
t.Helper()
out, err := exec.Command("docker", "run", "--rm", "--network", network, "--entrypoint", "sh", image, "-c", "set -eu; "+script).CombinedOutput()
if err != nil {
t.Fatalf("%s failed: %v\n%s", image, err, out)
}
return string(out)
}
out := run("ghcr.io/voxpupuli/r10k:latest", "cd /tmp && printf 'forge:\\n baseurl: "+forge+"\\n' > r10k.yaml && "+
"printf \"forge '"+forge+"'\\nmod 'unkin-hello', '0.1.0'\\n\" > Puppetfile && "+
"r10k puppetfile install -c r10k.yaml --verbose && cat modules/hello/metadata.json")
if !strings.Contains(out, `"unkin-hello"`) {
t.Fatalf("r10k did not install unkin-hello:\n%s", out)
}
out = run("ghcr.io/openvoxproject/openvoxagent:latest", "export PATH=/opt/puppetlabs/bin:$PATH; puppet module install unkin-hello --module_repository "+forge+" --ignore-dependencies --target-dir /tmp/mods && puppet module list --color=false --modulepath /tmp/mods")
if !strings.Contains(out, "unkin-hello (v0.1.0)") {
t.Fatalf("puppet module install did not install unkin-hello:\n%s", out)
}
}
+1
View File
@@ -20,6 +20,7 @@ require (
github.com/testcontainers/testcontainers-go/modules/redis v0.42.0
github.com/ulikunitz/xz v0.5.16
golang.org/x/crypto v0.54.0
golang.org/x/mod v0.37.0
golang.org/x/sync v0.22.0
golang.org/x/time v0.15.0
gopkg.in/yaml.v3 v3.0.1
+4
View File
@@ -227,6 +227,8 @@ golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
@@ -247,6 +249,8 @@ golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
+13 -1
View File
@@ -1,6 +1,7 @@
package v1
import (
"bytes"
"errors"
"fmt"
"io"
@@ -105,13 +106,24 @@ func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
}
defer result.Reader.Close()
var body io.Reader = result.Reader
if sr, ok := prov.(provider.ServeRewriter); ok && prov.Classify(path) == provider.Mutable {
raw, err := io.ReadAll(result.Reader)
if err != nil {
http.Error(w, "bad gateway", http.StatusBadGateway)
return
}
raw = sr.RewriteServed(raw, *remote, fmt.Sprintf("%s://%s", scheme(r), r.Host))
body, result.Size = bytes.NewReader(raw), int64(len(raw))
}
w.Header().Set("Content-Type", result.ContentType)
w.Header().Set("X-Artifact-Source", result.Source)
if result.Size > 0 {
w.Header().Set("X-Artifact-Size", fmt.Sprintf("%d", result.Size))
}
w.WriteHeader(http.StatusOK)
io.Copy(w, result.Reader)
io.Copy(w, body)
}
func (h *ProxyHandler) handleProxyHead(w http.ResponseWriter, r *http.Request) {
+38 -1
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"net/http"
"os"
"github.com/go-chi/chi/v5"
@@ -83,7 +84,22 @@ func (h *LocalHandler) uploadValidated(w http.ResponseWriter, r *http.Request, r
return
}
result, err := h.cas.Store(r.Context(), r.Body, contentType)
var body io.Reader = r.Body
if cv, ok := prov.(provider.ContentValidator); ok {
f, err := spoolBody(r.Body)
if err != nil {
http.Error(w, fmt.Sprintf("read upload: %v", err), http.StatusInternalServerError)
return
}
defer discard(f)
if err := cv.ValidateContent(storagePath, f.Name()); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
body = f
}
result, err := h.cas.Store(r.Context(), body, contentType)
if err != nil {
http.Error(w, fmt.Sprintf("store failed: %v", err), http.StatusInternalServerError)
return
@@ -110,6 +126,27 @@ func (h *LocalHandler) uploadValidated(w http.ResponseWriter, r *http.Request, r
writeJSON(w, http.StatusCreated, uploader.UploadResponse(storagePath, result.ContentHash, result.SizeBytes))
}
func spoolBody(r io.Reader) (*os.File, error) {
f, err := os.CreateTemp("", "artifactapi-upload-*")
if err != nil {
return nil, err
}
if _, err := io.Copy(f, r); err != nil {
discard(f)
return nil, err
}
if _, err := f.Seek(0, io.SeekStart); err != nil {
discard(f)
return nil, err
}
return f, nil
}
func discard(f *os.File) {
_ = f.Close()
_ = os.Remove(f.Name())
}
func (h *LocalHandler) uploadGeneric(w http.ResponseWriter, r *http.Request, remote *models.Remote, filePath string) {
existing, err := h.db.GetLocalFile(r.Context(), remote.Name, filePath)
if err != nil {
+55
View File
@@ -0,0 +1,55 @@
package v2
import (
"context"
"net/http/httptest"
"strings"
"testing"
"github.com/go-chi/chi/v5"
"git.unkin.net/unkin/artifactapi/internal/database"
_ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
// TestLocalUploadContentRejected verifies a ContentValidator provider rejects
// bad bytes with 400 before anything is stored.
func TestLocalUploadContentRejected(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
ctx := context.Background()
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
const repo = "go-local-validate"
if err := db.CreateRemote(ctx, &models.Remote{Name: repo, PackageType: models.PackageGoProxy, RepoType: models.RepoTypeLocal}); err != nil {
t.Fatal(err)
}
router := chi.NewRouter()
router.Route("/remotes/{name}/files", func(r chi.Router) {
r.Put("/*", NewLocalHandler(db, nil).Routes().ServeHTTP)
})
cases := map[string]string{
"example.unkin.net/hello/@v/v0.1.0.zip": "not a zip",
"example.unkin.net/hello/@v/v0.1.0.mod": "module example.unkin.net/other\n",
"example.unkin.net/hello/@v/v0.1.0.info": `{"Version":"v9.9.9"}`,
"example.unkin.net/Hello/@v/v0.1.0.mod": "module example.unkin.net/Hello\n",
}
for path, body := range cases {
w := httptest.NewRecorder()
router.ServeHTTP(w, httptest.NewRequest("PUT", "/remotes/"+repo+"/files/"+path, strings.NewReader(body)))
if w.Code != 400 {
t.Errorf("%s: status %d, want 400 (%s)", path, w.Code, w.Body.String())
}
if f, _ := db.GetLocalFile(ctx, repo, path); f != nil {
t.Errorf("%s: rejected upload was recorded", path)
}
}
}
+46
View File
@@ -0,0 +1,46 @@
package database
import (
"context"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func (db *DB) InsertPuppetMetadata(ctx context.Context, meta *provider.PuppetMetadata) error {
_, err := db.Pool.Exec(ctx, `
INSERT INTO puppet_metadata (repo_name, file_path, content_hash, slug, version, metadata_json, file_size, file_md5)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8)
ON CONFLICT (repo_name, file_path) DO NOTHING
`, meta.RepoName, meta.FilePath, meta.ContentHash, meta.Slug, meta.Version, meta.MetadataJSON, meta.Size, meta.MD5)
return err
}
func (db *DB) DeletePuppetMetadata(ctx context.Context, repoName, filePath string) error {
_, err := db.Pool.Exec(ctx, `DELETE FROM puppet_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
return err
}
// ListPuppetMetadataEntries returns a repo's releases, optionally narrowed to
// one module slug (empty slug lists every module).
func (db *DB) ListPuppetMetadataEntries(ctx context.Context, repoName, slug string) ([]provider.PuppetMetadata, error) {
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash, slug, version, metadata_json, file_size, file_md5, created_at
FROM puppet_metadata
WHERE repo_name = $1 AND ($2 = '' OR slug = $2)
ORDER BY slug, file_path
`, repoName, slug)
if err != nil {
return nil, err
}
defer rows.Close()
var result []provider.PuppetMetadata
for rows.Next() {
var m provider.PuppetMetadata
if err := rows.Scan(&m.RepoName, &m.FilePath, &m.ContentHash, &m.Slug, &m.Version, &m.MetadataJSON, &m.Size, &m.MD5, &m.CreatedAt); err != nil {
return nil, err
}
result = append(result, m)
}
return result, rows.Err()
}
+222
View File
@@ -0,0 +1,222 @@
package goproxy
import (
"context"
"encoding/json"
"fmt"
"net/http"
"os"
"slices"
"strings"
"golang.org/x/mod/modfile"
"golang.org/x/mod/module"
"golang.org/x/mod/semver"
modzip "golang.org/x/mod/zip"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
type versionFile struct {
module string
version string
ext string
}
// parseVersionFile splits "<escaped module>/@v/<version>.<ext>" into its parts.
func parseVersionFile(path string) (versionFile, error) {
escMod, file, ok := strings.Cut(strings.TrimPrefix(path, "/"), "/@v/")
if !ok || escMod == "" || strings.Contains(file, "/") {
return versionFile{}, fmt.Errorf("path %q must be <module>/@v/<version>.{zip,mod,info}", path)
}
dot := strings.LastIndex(file, ".")
if dot <= 0 {
return versionFile{}, fmt.Errorf("path %q must end in .zip, .mod or .info", path)
}
ext := file[dot+1:]
if ext != "zip" && ext != "mod" && ext != "info" {
return versionFile{}, fmt.Errorf("path %q must end in .zip, .mod or .info", path)
}
mod, err := module.UnescapePath(escMod)
if err != nil {
return versionFile{}, fmt.Errorf("module path must be case-encoded (uppercase as !lower): %w", err)
}
ver, err := module.UnescapeVersion(file[:dot])
if err != nil {
return versionFile{}, err
}
if err := module.Check(mod, ver); err != nil {
return versionFile{}, err
}
if semver.Canonical(ver) != ver {
return versionFile{}, fmt.Errorf("version %q is not canonical semver", ver)
}
return versionFile{module: mod, version: ver, ext: ext}, nil
}
func (v versionFile) storagePath() string {
escMod, _ := module.EscapePath(v.module)
escVer, _ := module.EscapeVersion(v.version)
return escMod + "/@v/" + escVer + "." + v.ext
}
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
vf, err := parseVersionFile(filePath)
if err != nil {
return "", "", err
}
sp := vf.storagePath()
return sp, p.ContentType(sp), nil
}
func (p *Provider) ValidateContent(storagePath, localFile string) error {
vf, err := parseVersionFile(storagePath)
if err != nil {
return err
}
switch vf.ext {
case "zip":
if _, err := modzip.CheckZip(module.Version{Path: vf.module, Version: vf.version}, localFile); err != nil {
return fmt.Errorf("invalid module zip (files must sit under %s@%s/): %w", vf.module, vf.version, err)
}
case "mod":
data, err := os.ReadFile(localFile)
if err != nil {
return err
}
mf, err := modfile.ParseLax("go.mod", data, nil)
if err != nil {
return fmt.Errorf("invalid go.mod: %w", err)
}
if mf.Module == nil || mf.Module.Mod.Path != vf.module {
return fmt.Errorf("go.mod must declare module %s", vf.module)
}
case "info":
data, err := os.ReadFile(localFile)
if err != nil {
return err
}
var info struct{ Version string }
if err := json.Unmarshal(data, &info); err != nil {
return fmt.Errorf("invalid .info JSON: %w", err)
}
if info.Version != vf.version {
return fmt.Errorf(".info Version %q does not match %s", info.Version, vf.version)
}
}
return nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
vf, _ := parseVersionFile(storagePath)
return map[string]any{
"module": vf.module,
"version": vf.version,
"path": storagePath,
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
// versions returns the canonical versions of escMod that have a .mod uploaded, sorted ascending.
func versions(ctx context.Context, files provider.FileStore, repoName, escMod string) (versions []string, infos map[string]bool, err error) {
entries, err := files.ListFilesByPrefix(ctx, repoName, escMod+"/@v/")
if err != nil {
return nil, nil, err
}
infos = map[string]bool{}
for _, e := range entries {
vf, err := parseVersionFile(e.FilePath)
if err != nil || vf.storagePath() != e.FilePath {
continue
}
switch vf.ext {
case "mod":
versions = append(versions, vf.version)
case "info":
infos[vf.version] = true
}
}
semver.Sort(versions)
return versions, infos, nil
}
func latest(vs []string) string {
for i := len(vs) - 1; i >= 0; i-- {
if semver.Prerelease(vs[i]) == "" {
return vs[i]
}
}
if len(vs) == 0 {
return ""
}
return vs[len(vs)-1]
}
func infoJSON(version string) []byte {
b, _ := json.Marshal(map[string]string{"Version": version})
return b
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, path string) bool {
path = strings.TrimPrefix(path, "/")
switch {
case strings.HasSuffix(path, "/@v/list"), strings.HasSuffix(path, "/@latest"):
case strings.HasSuffix(path, ".info"):
vf, err := parseVersionFile(path)
if err != nil {
return false
}
escMod, _ := module.EscapePath(vf.module)
vs, infos, err := versions(r.Context(), files, repoName, escMod)
if err != nil || infos[vf.version] || !slices.Contains(vs, vf.version) {
return false
}
default:
return false
}
body, err := p.GenerateLocalIndex(r.Context(), files, repoName, path)
if err != nil {
http.Error(w, err.Error(), http.StatusNotFound)
return true
}
w.Header().Set("Content-Type", p.ContentType(path))
w.WriteHeader(http.StatusOK)
_, _ = w.Write(body)
return true
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
path = strings.TrimPrefix(path, "/")
if escMod, ok := strings.CutSuffix(path, "/@v/list"); ok {
vs, _, err := versions(ctx, files, repoName, escMod)
if err != nil {
return nil, err
}
var b strings.Builder
for _, v := range vs {
b.WriteString(v + "\n")
}
return []byte(b.String()), nil
}
if escMod, ok := strings.CutSuffix(path, "/@latest"); ok {
vs, _, err := versions(ctx, files, repoName, escMod)
if err != nil {
return nil, err
}
v := latest(vs)
if v == "" {
return nil, fmt.Errorf("no versions of %s", escMod)
}
return infoJSON(v), nil
}
if strings.HasSuffix(path, ".info") {
vf, err := parseVersionFile(path)
if err != nil {
return nil, err
}
return infoJSON(vf.version), nil
}
return nil, fmt.Errorf("unsupported index path: %q", path)
}
+198
View File
@@ -0,0 +1,198 @@
package goproxy
import (
"archive/zip"
"context"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"golang.org/x/mod/module"
modzip "golang.org/x/mod/zip"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
type fakeFileStore struct{ paths []string }
func (f *fakeFileStore) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
func (f *fakeFileStore) ListFilesByPrefix(_ context.Context, _, prefix string) ([]provider.FileEntry, error) {
var out []provider.FileEntry
for _, p := range f.paths {
if strings.HasPrefix(p, prefix) {
out = append(out, provider.FileEntry{FilePath: p, ContentHash: "sha256:x"})
}
}
return out, nil
}
func TestValidateUpload(t *testing.T) {
p := &Provider{}
cases := []struct {
in, want string
ok bool
}{
{"example.unkin.net/hello/@v/v0.1.0.zip", "example.unkin.net/hello/@v/v0.1.0.zip", true},
{"/example.unkin.net/hello/@v/v0.1.0.mod", "example.unkin.net/hello/@v/v0.1.0.mod", true},
{"github.com/!burnt!sushi/toml/@v/v1.2.0.info", "github.com/!burnt!sushi/toml/@v/v1.2.0.info", true},
{"example.unkin.net/hello/v2/@v/v2.0.0-rc.1.zip", "example.unkin.net/hello/v2/@v/v2.0.0-rc.1.zip", true},
{"github.com/BurntSushi/toml/@v/v1.2.0.zip", "", false},
{"example.unkin.net/hello/@v/v0.1.zip", "", false},
{"example.unkin.net/hello/@v/v2.0.0.zip", "", false},
{"example.unkin.net/hello/@v/v0.1.0.tar.gz", "", false},
{"example.unkin.net/hello/v0.1.0.zip", "", false},
{"example.unkin.net/hello/@v/list", "", false},
}
for _, c := range cases {
got, ct, err := p.ValidateUpload(c.in)
if (err == nil) != c.ok {
t.Errorf("%s: err=%v, want ok=%v", c.in, err, c.ok)
continue
}
if c.ok && (got != c.want || ct == "") {
t.Errorf("%s: got %q %q, want %q", c.in, got, ct, c.want)
}
}
}
func writeZip(t *testing.T, files map[string]string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "m.zip")
f, err := os.Create(path)
if err != nil {
t.Fatal(err)
}
zw := zip.NewWriter(f)
for name, body := range files {
w, err := zw.Create(name)
if err != nil {
t.Fatal(err)
}
if _, err := w.Write([]byte(body)); err != nil {
t.Fatal(err)
}
}
if err := zw.Close(); err != nil {
t.Fatal(err)
}
if err := f.Close(); err != nil {
t.Fatal(err)
}
return path
}
func writeFile(t *testing.T, body string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "f")
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
return path
}
func TestValidateContentZip(t *testing.T) {
p := &Provider{}
sp := "example.unkin.net/hello/@v/v0.1.0.zip"
dir := t.TempDir()
os.WriteFile(filepath.Join(dir, "go.mod"), []byte("module example.unkin.net/hello\n"), 0o600)
os.WriteFile(filepath.Join(dir, "main.go"), []byte("package main\n"), 0o600)
good := filepath.Join(t.TempDir(), "good.zip")
f, _ := os.Create(good)
if err := modzip.CreateFromDir(f, module.Version{Path: "example.unkin.net/hello", Version: "v0.1.0"}, dir); err != nil {
t.Fatal(err)
}
f.Close()
if err := p.ValidateContent(sp, good); err != nil {
t.Errorf("valid zip rejected: %v", err)
}
bad := map[string]map[string]string{
"no prefix": {"go.mod": "module example.unkin.net/hello\n"},
"wrong version": {"example.unkin.net/hello@v0.2.0/go.mod": "module example.unkin.net/hello\n"},
"wrong module": {"example.unkin.net/other@v0.1.0/go.mod": "module example.unkin.net/other\n"},
}
for name, files := range bad {
if err := p.ValidateContent(sp, writeZip(t, files)); err == nil {
t.Errorf("%s: expected rejection", name)
}
}
if err := p.ValidateContent(sp, writeFile(t, "not a zip")); err == nil {
t.Error("non-zip accepted")
}
}
func TestValidateContentModInfo(t *testing.T) {
p := &Provider{}
mod := "example.unkin.net/hello/@v/v0.1.0.mod"
info := "example.unkin.net/hello/@v/v0.1.0.info"
cases := []struct {
path, body string
ok bool
}{
{mod, "module example.unkin.net/hello\n\ngo 1.25\n", true},
{mod, "module example.unkin.net/other\n", false},
{mod, "go 1.25\n", false},
{mod, "module (\n", false},
{info, `{"Version":"v0.1.0","Time":"2026-01-01T00:00:00Z"}`, true},
{info, `{"Version":"v0.2.0"}`, false},
{info, `nope`, false},
}
for _, c := range cases {
if err := p.ValidateContent(c.path, writeFile(t, c.body)); (err == nil) != c.ok {
t.Errorf("%s %q: err=%v, want ok=%v", c.path, c.body, err, c.ok)
}
}
}
func serve(t *testing.T, fs provider.FileStore, path string) (bool, int, string) {
t.Helper()
rec := httptest.NewRecorder()
handled := (&Provider{}).ServeLocalIndex(rec, httptest.NewRequest("GET", "/"+path, nil), fs, "go-local", path)
return handled, rec.Code, rec.Body.String()
}
func TestServeLocalIndex(t *testing.T) {
fs := &fakeFileStore{paths: []string{
"example.unkin.net/hello/@v/v0.10.0.mod",
"example.unkin.net/hello/@v/v0.10.0.zip",
"example.unkin.net/hello/@v/v0.2.0.mod",
"example.unkin.net/hello/@v/v0.2.0.zip",
"example.unkin.net/hello/@v/v0.2.0.info",
"example.unkin.net/hello/@v/v0.11.0-rc.1.mod",
"example.unkin.net/hello/@v/v0.3.0.zip",
"example.unkin.net/hello/v2/@v/v2.0.0.mod",
"example.unkin.net/!pre/@v/v1.0.0-beta.mod",
}}
if h, code, body := serve(t, fs, "example.unkin.net/hello/@v/list"); !h || code != 200 || body != "v0.2.0\nv0.10.0\nv0.11.0-rc.1\n" {
t.Errorf("list: %v %d %q", h, code, body)
}
if h, code, body := serve(t, fs, "example.unkin.net/hello/@latest"); !h || code != 200 || body != `{"Version":"v0.10.0"}` {
t.Errorf("latest: %v %d %q", h, code, body)
}
if h, _, body := serve(t, fs, "example.unkin.net/!pre/@latest"); !h || body != `{"Version":"v1.0.0-beta"}` {
t.Errorf("prerelease-only latest: %q", body)
}
if h, code, _ := serve(t, fs, "example.unkin.net/missing/@latest"); !h || code != 404 {
t.Errorf("missing latest: %v %d", h, code)
}
if h, code, body := serve(t, fs, "example.unkin.net/missing/@v/list"); !h || code != 200 || body != "" {
t.Errorf("missing list: %v %d %q", h, code, body)
}
if h, _, body := serve(t, fs, "example.unkin.net/hello/@v/v0.10.0.info"); !h || body != `{"Version":"v0.10.0"}` {
t.Errorf("synthesized info: %v %q", h, body)
}
if h, _, _ := serve(t, fs, "example.unkin.net/hello/@v/v0.2.0.info"); h {
t.Error("uploaded .info must be served from storage")
}
if h, _, _ := serve(t, fs, "example.unkin.net/hello/@v/v0.3.0.info"); h {
t.Error(".info without a .mod must not be synthesized")
}
if h, _, _ := serve(t, fs, "example.unkin.net/hello/@v/v0.10.0.zip"); h {
t.Error(".zip must be served from storage")
}
}
+85
View File
@@ -0,0 +1,85 @@
// Package nuget proxies a NuGet v3 feed: the service index, registration pages
// and flat container indexes are mutable, .nupkg/.nuspec downloads immutable.
package nuget
import (
"context"
"encoding/json"
"net/http"
"net/url"
"strings"
"git.unkin.net/unkin/artifactapi/internal/auth"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
func init() {
provider.Register(&Provider{})
}
// serviceIndex is the path clients use as their source URL; it maps to base_url.
const serviceIndex = "index.json"
type Provider struct{}
func (p *Provider) Type() models.PackageType { return models.PackageNuGet }
func (p *Provider) Classify(path string) provider.Mutability {
if isPackage(path) {
return provider.Immutable
}
return provider.Mutable
}
func (p *Provider) ContentType(path string) string {
switch {
case strings.HasSuffix(path, ".nupkg"):
return "application/octet-stream"
case strings.HasSuffix(path, ".nuspec"):
return "application/xml"
}
return "application/json"
}
// UpstreamURL maps index.json to base_url (the upstream service index) and any
// other path to the same path on the base_url's origin, where nuget.org serves
// v3-flatcontainer/ and v3/registration*/.
func (p *Provider) UpstreamURL(remote models.Remote, path string) string {
path = strings.TrimLeft(path, "/")
if path == serviceIndex {
return remote.BaseURL
}
return origin(remote.BaseURL) + "/" + path
}
func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
return nil, nil
}
// RewriteServed points upstream-origin URLs in cached JSON back at this remote.
// It runs per request because the URLs must name the requesting host.
func (p *Provider) RewriteServed(body []byte, remote models.Remote, proxyBaseURL string) []byte {
from := origin(remote.BaseURL)
if from == "" || !json.Valid(body) {
return body
}
to := strings.TrimRight(proxyBaseURL, "/") + "/api/v1/remote/" + remote.Name
return []byte(strings.ReplaceAll(string(body), from, to))
}
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
return auth.BasicHeaders(remote), nil
}
func origin(baseURL string) string {
u, err := url.Parse(baseURL)
if err != nil || u.Host == "" {
return ""
}
return u.Scheme + "://" + u.Host
}
func isPackage(path string) bool {
return strings.HasSuffix(path, ".nupkg") || strings.HasSuffix(path, ".nuspec")
}
+68
View File
@@ -0,0 +1,68 @@
package nuget
import (
"testing"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
var _ provider.ServeRewriter = (*Provider)(nil)
var nugetOrg = models.Remote{Name: "nuget-org", BaseURL: "https://api.nuget.org/v3/index.json"}
func TestClassify(t *testing.T) {
p := &Provider{}
tests := []struct {
path string
want provider.Mutability
}{
{"v3-flatcontainer/newtonsoft.json/13.0.3/newtonsoft.json.13.0.3.nupkg", provider.Immutable},
{"v3-flatcontainer/newtonsoft.json/13.0.3/newtonsoft.json.nuspec", provider.Immutable},
{"v3-flatcontainer/newtonsoft.json/index.json", provider.Mutable},
{"v3/registration5-gz-semver2/newtonsoft.json/index.json", provider.Mutable},
{"index.json", provider.Mutable},
}
for _, tt := range tests {
if got := p.Classify(tt.path); got != tt.want {
t.Errorf("Classify(%q) = %v, want %v", tt.path, got, tt.want)
}
}
}
func TestUpstreamURL(t *testing.T) {
p := &Provider{}
tests := []struct {
path, want string
}{
{"index.json", "https://api.nuget.org/v3/index.json"},
{"/index.json", "https://api.nuget.org/v3/index.json"},
{"v3-flatcontainer/serilog/index.json", "https://api.nuget.org/v3-flatcontainer/serilog/index.json"},
{"v3/registration5-gz-semver2/serilog/index.json", "https://api.nuget.org/v3/registration5-gz-semver2/serilog/index.json"},
}
for _, tt := range tests {
if got := p.UpstreamURL(nugetOrg, tt.path); got != tt.want {
t.Errorf("UpstreamURL(%q) = %q, want %q", tt.path, got, tt.want)
}
}
}
func TestRewriteServed(t *testing.T) {
p := &Provider{}
body := `{"resources":[` +
`{"@id":"https://api.nuget.org/v3-flatcontainer/","@type":"PackageBaseAddress/3.0.0"},` +
`{"@id":"https://azuresearch-usnc.nuget.org/query","@type":"SearchQueryService"}]}`
want := `{"resources":[` +
`{"@id":"http://aa.example:8000/api/v1/remote/nuget-org/v3-flatcontainer/","@type":"PackageBaseAddress/3.0.0"},` +
`{"@id":"https://azuresearch-usnc.nuget.org/query","@type":"SearchQueryService"}]}`
if got := string(p.RewriteServed([]byte(body), nugetOrg, "http://aa.example:8000/")); got != want {
t.Errorf("RewriteServed =\n%s\nwant\n%s", got, want)
}
if got := string(p.RewriteServed([]byte(body), models.Remote{Name: "x"}, "http://aa.example")); got != body {
t.Error("remote without a parseable base_url must pass the body through")
}
cert := "\x30\x82https://api.nuget.org"
if got := string(p.RewriteServed([]byte(cert), nugetOrg, "http://aa.example")); got != cert {
t.Error("non-JSON bodies must pass through untouched")
}
}
+38
View File
@@ -41,6 +41,11 @@ type LocalUploader interface {
UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any
}
// ContentValidator checks an upload's bytes (spooled to localFile) before it is stored.
type ContentValidator interface {
ValidateContent(storagePath, localFile string) error
}
type LocalIndexer interface {
ServeLocalIndex(w http.ResponseWriter, r *http.Request, files FileStore, repoName, path string) bool
GenerateLocalIndex(ctx context.Context, files FileStore, repoName, path string) ([]byte, error)
@@ -75,6 +80,13 @@ type RemoteServer interface {
ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store RemoteMetadataStore) bool
}
// ServeRewriter rewrites a cached mutable remote response per request, for
// indexes whose URLs must name the requesting host and so cannot be rewritten
// before caching.
type ServeRewriter interface {
RewriteServed(body []byte, remote models.Remote, proxyBaseURL string) []byte
}
// RemoteMetadataStore is the persistence surface a RemoteServer needs to cache
// and read the metadata it derives per upstream asset. *database.DB satisfies it.
type RemoteMetadataStore interface {
@@ -240,3 +252,29 @@ func Get(t models.PackageType) (Provider, error) {
func All() map[models.PackageType]Provider {
return registry
}
type PuppetMetadataStore interface {
InsertPuppetMetadata(ctx context.Context, meta *PuppetMetadata) error
}
type PuppetMetadataDeleter interface {
DeletePuppetMetadata(ctx context.Context, repoName, filePath string) error
}
type PuppetMetadataReader interface {
ListPuppetMetadataEntries(ctx context.Context, repoName, slug string) ([]PuppetMetadata, error)
}
// PuppetMetadata is a local module release; MetadataJSON is the tarball's
// metadata.json verbatim so Forge release objects carry every field.
type PuppetMetadata struct {
RepoName string
FilePath string
ContentHash string
Slug string
Version string
MetadataJSON string
Size int64
MD5 string
CreatedAt time.Time
}
+393
View File
@@ -0,0 +1,393 @@
package puppet
import (
"archive/tar"
"compress/gzip"
"context"
"crypto/md5"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/url"
"path"
"regexp"
"slices"
"strconv"
"strings"
"git.unkin.net/unkin/artifactapi/internal/provider"
"git.unkin.net/unkin/artifactapi/internal/storage"
)
const filesPrefix = "v3/files/"
// releaseRe matches <owner>-<name>-<semver>, the Forge release slug and the
// tarball basename that `puppet module build` produces.
var releaseRe = regexp.MustCompile(`^([a-z0-9]+)-([a-z][a-z0-9_]*)-(\d+\.\d+\.\d+(?:-[0-9a-z.-]+)?(?:\+[0-9a-z.-]+)?)$`)
func parseRelease(s string) (slug, version string, ok bool) {
m := releaseRe.FindStringSubmatch(strings.ToLower(s))
if m == nil {
return "", "", false
}
return m[1] + "-" + m[2], m[3], true
}
func normalizeSlug(s string) string {
return strings.ToLower(strings.ReplaceAll(s, "/", "-"))
}
// ValidateUpload stores tarballs under v3/files/ so Forge file_uri downloads
// resolve straight to the stored file.
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
filename := strings.ToLower(path.Base(path.Clean("/" + filePath)))
if _, _, ok := parseRelease(strings.TrimSuffix(filename, ".tar.gz")); !ok || !strings.HasSuffix(filename, ".tar.gz") {
return "", "", fmt.Errorf("file must be named <owner>-<module>-<version>.tar.gz")
}
return filesPrefix + filename, "application/gzip", nil
}
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
return map[string]any{
"filename": path.Base(storagePath),
"content_hash": contentHash,
"size_bytes": sizeBytes,
}
}
func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs provider.BlobReader, db provider.MetadataStore) {
reader, size, err := blobs.Download(ctx, storage.BlobKey(strings.TrimPrefix(contentHash, "sha256:")))
if err != nil {
slog.Error("puppet metadata: download failed", "repo", repoName, "path", storagePath, "error", err)
return
}
defer func() { _ = reader.Close() }()
if size > maxModuleTarball {
slog.Error("puppet metadata: tarball too large", "repo", repoName, "path", storagePath, "size", size)
return
}
hash := md5.New()
src := io.TeeReader(io.LimitReader(reader, maxModuleTarball), hash)
meta, err := parseModule(src)
if err != nil {
slog.Error("puppet metadata: parse failed", "repo", repoName, "path", storagePath, "error", err)
return
}
if _, err := io.Copy(io.Discard, src); err != nil {
slog.Error("puppet metadata: read failed", "repo", repoName, "path", storagePath, "error", err)
return
}
slug, version, _ := parseRelease(strings.TrimSuffix(path.Base(storagePath), ".tar.gz"))
if meta.Slug != slug || meta.Version != version {
slog.Error("puppet metadata: metadata.json does not match filename", "repo", repoName, "path", storagePath, "name", meta.Slug, "version", meta.Version)
return
}
meta.RepoName = repoName
meta.FilePath = storagePath
meta.ContentHash = contentHash
meta.Size = size
meta.MD5 = hex.EncodeToString(hash.Sum(nil))
store, ok := db.(provider.PuppetMetadataStore)
if !ok {
slog.Error("puppet metadata: store does not support puppet metadata", "repo", repoName)
return
}
if err := store.InsertPuppetMetadata(ctx, meta); err != nil {
slog.Error("puppet metadata: insert failed", "repo", repoName, "path", storagePath, "error", err)
return
}
slog.Info("puppet metadata: parsed", "repo", repoName, "module", meta.Slug, "version", meta.Version)
}
func (p *Provider) AfterDelete(ctx context.Context, repoName, storagePath string, db provider.MetadataDeleter) error {
deleter, ok := db.(provider.PuppetMetadataDeleter)
if !ok {
return nil
}
return deleter.DeletePuppetMetadata(ctx, repoName, storagePath)
}
const maxMetadataJSON = 1 << 20
var (
maxModuleTarball int64 = 100 << 20
maxModuleScan int64 = 200 << 20
)
// parseModule reads the top-level <dir>/metadata.json from a module tarball,
// scanning at most maxModuleScan decompressed bytes.
func parseModule(r io.Reader) (*provider.PuppetMetadata, error) {
zr, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
scan := &io.LimitedReader{R: zr, N: maxModuleScan}
tr := tar.NewReader(scan)
for {
hdr, err := tr.Next()
if scan.N <= 0 {
return nil, errors.New("module tarball exceeds decompressed scan limit before metadata.json")
}
if err == io.EOF {
return nil, errors.New("no metadata.json in module tarball")
}
if err != nil {
return nil, err
}
dir, file := path.Split(strings.TrimPrefix(path.Clean(hdr.Name), "./"))
if hdr.Typeflag != tar.TypeReg || file != "metadata.json" || strings.Count(dir, "/") != 1 {
continue
}
body, err := io.ReadAll(io.LimitReader(tr, maxMetadataJSON))
if err != nil {
return nil, err
}
var md struct {
Name string `json:"name"`
Version string `json:"version"`
}
if err := json.Unmarshal(body, &md); err != nil {
return nil, fmt.Errorf("parse metadata.json: %w", err)
}
if md.Name == "" || md.Version == "" {
return nil, errors.New("metadata.json missing name or version")
}
return &provider.PuppetMetadata{Slug: normalizeSlug(md.Name), Version: md.Version, MetadataJSON: string(body)}, nil
}
}
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, reqPath string) bool {
clean := strings.TrimPrefix(path.Clean("/"+reqPath), "/")
var list func(slug string) ([]provider.PuppetMetadata, error)
if reader, ok := files.(provider.PuppetMetadataReader); ok {
list = func(slug string) ([]provider.PuppetMetadata, error) {
metas, err := reader.ListPuppetMetadataEntries(r.Context(), repoName, slug)
sortReleases(metas)
return metas, err
}
} else {
list = func(string) ([]provider.PuppetMetadata, error) {
return nil, errors.New("puppet metadata not available")
}
}
switch {
case clean == "v3/releases":
serveReleaseList(w, r, list)
case strings.HasPrefix(clean, "v3/releases/"):
slug, version, ok := parseRelease(strings.TrimPrefix(clean, "v3/releases/"))
if !ok {
notFound(w)
return true
}
metas, err := list(slug)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return true
}
for _, m := range metas {
if m.Version == version {
writeJSON(w, releaseObject(m))
return true
}
}
notFound(w)
case strings.HasPrefix(clean, "v3/modules/"):
slug := normalizeSlug(strings.TrimPrefix(clean, "v3/modules/"))
metas, err := list(slug)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return true
}
if len(metas) == 0 {
notFound(w)
return true
}
writeJSON(w, moduleObject(metas))
default:
return false
}
return true
}
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
return nil, fmt.Errorf("puppet local index generation for virtual repos not supported")
}
func serveReleaseList(w http.ResponseWriter, r *http.Request, list func(string) ([]provider.PuppetMetadata, error)) {
q := r.URL.Query()
slug := normalizeSlug(q.Get("module"))
limit, err := strconv.Atoi(q.Get("limit"))
if err != nil || limit <= 0 {
limit = 20
}
limit = min(limit, 100)
offset, err := strconv.Atoi(q.Get("offset"))
if err != nil || offset < 0 {
offset = 0
}
metas, err := list(slug)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
offset = min(offset, len(metas))
results := []map[string]any{}
for _, m := range metas[offset : offset+min(limit, len(metas)-offset)] {
results = append(results, releaseObject(m))
}
page := func(off int) any {
if off < 0 || off >= len(metas) && off != 0 {
return nil
}
v := url.Values{"limit": {strconv.Itoa(limit)}, "offset": {strconv.Itoa(off)}}
if slug != "" {
v.Set("module", slug)
}
return "/v3/releases?" + v.Encode()
}
var previous any
if offset > 0 {
previous = page(max(offset-limit, 0))
}
writeJSON(w, map[string]any{
"pagination": map[string]any{
"limit": limit,
"offset": offset,
"first": page(0),
"previous": previous,
"current": page(offset),
"next": page(offset + limit),
"total": len(metas),
},
"results": results,
})
}
func owner(slug string) map[string]any {
name, _, _ := strings.Cut(slug, "-")
return map[string]any{"uri": "/v3/users/" + name, "slug": name, "username": name}
}
func moduleRef(slug string) map[string]any {
_, name, _ := strings.Cut(slug, "-")
return map[string]any{"uri": "/v3/modules/" + slug, "slug": slug, "name": name, "deprecated_at": nil, "owner": owner(slug)}
}
const forgeTime = "2006-01-02 15:04:05 -0700"
func releaseObject(m provider.PuppetMetadata) map[string]any {
slug := m.Slug + "-" + m.Version
return map[string]any{
"uri": "/v3/releases/" + slug,
"slug": slug,
"module": moduleRef(m.Slug),
"version": m.Version,
"metadata": json.RawMessage(m.MetadataJSON),
"tags": []string{},
"supported": false,
"pdk": false,
"file_uri": "/" + m.FilePath,
"file_size": m.Size,
"file_md5": m.MD5,
"file_sha256": strings.TrimPrefix(m.ContentHash, "sha256:"),
"downloads": 0,
"readme": nil,
"changelog": nil,
"license": nil,
"created_at": m.CreatedAt.Format(forgeTime),
"updated_at": m.CreatedAt.Format(forgeTime),
"deleted_at": nil,
}
}
// moduleObject expects metas sorted newest version first.
func moduleObject(metas []provider.PuppetMetadata) map[string]any {
obj := moduleRef(metas[0].Slug)
releases := make([]map[string]any, 0, len(metas))
for _, m := range metas {
slug := m.Slug + "-" + m.Version
releases = append(releases, map[string]any{
"uri": "/v3/releases/" + slug, "slug": slug, "version": m.Version,
"file_uri": "/" + m.FilePath, "file_size": m.Size, "supported": false,
"created_at": m.CreatedAt.Format(forgeTime), "deleted_at": nil,
})
}
obj["current_release"] = releaseObject(metas[0])
obj["releases"] = releases
obj["downloads"] = 0
obj["created_at"] = metas[len(metas)-1].CreatedAt.Format(forgeTime)
obj["updated_at"] = metas[0].CreatedAt.Format(forgeTime)
return obj
}
func writeJSON(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
enc := json.NewEncoder(w)
enc.SetEscapeHTML(false)
_ = enc.Encode(v)
}
func notFound(w http.ResponseWriter) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusNotFound)
_, _ = io.WriteString(w, `{"message":"404 Not Found","errors":["The requested resource could not be found"]}`)
}
// sortReleases orders by module slug, then newest semver first.
func sortReleases(metas []provider.PuppetMetadata) {
slices.SortStableFunc(metas, func(a, b provider.PuppetMetadata) int {
if c := strings.Compare(a.Slug, b.Slug); c != 0 {
return c
}
return compareSemver(b.Version, a.Version)
})
}
func compareSemver(a, b string) int {
a, _, _ = strings.Cut(a, "+")
b, _, _ = strings.Cut(b, "+")
aCore, aPre, aHasPre := strings.Cut(a, "-")
bCore, bPre, bHasPre := strings.Cut(b, "-")
if c := compareIdents(strings.Split(aCore, "."), strings.Split(bCore, ".")); c != 0 {
return c
}
switch {
case aHasPre && !bHasPre:
return -1
case !aHasPre && bHasPre:
return 1
}
return compareIdents(strings.Split(aPre, "."), strings.Split(bPre, "."))
}
func compareIdents(a, b []string) int {
for i := 0; i < len(a) && i < len(b); i++ {
an, aErr := strconv.Atoi(a[i])
bn, bErr := strconv.Atoi(b[i])
var c int
switch {
case aErr == nil && bErr == nil:
c = an - bn
case aErr == nil:
c = -1
case bErr == nil:
c = 1
default:
c = strings.Compare(a[i], b[i])
}
if c != 0 {
return c
}
}
return len(a) - len(b)
}
+301
View File
@@ -0,0 +1,301 @@
package puppet
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"crypto/md5"
"encoding/hex"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func moduleTarball(t *testing.T, dir, metadata string) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for name, body := range map[string]string{dir + "/metadata.json": metadata, dir + "/manifests/init.pp": "class hello {}\n"} {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body))}); err != nil {
t.Fatal(err)
}
_, _ = tw.Write([]byte(body))
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
const helloMetadata = `{"name":"unkin-hello","version":"0.1.0","dependencies":[{"name":"puppetlabs/stdlib","version_requirement":">= 4.0.0"}]}`
type fakeBlobs struct{ data []byte }
func (f fakeBlobs) Download(context.Context, string) (io.ReadCloser, int64, error) {
return io.NopCloser(bytes.NewReader(f.data)), int64(len(f.data)), nil
}
type fakeStore struct{ inserted *provider.PuppetMetadata }
func (f *fakeStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
func (f *fakeStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
func (f *fakeStore) InsertPuppetMetadata(_ context.Context, m *provider.PuppetMetadata) error {
f.inserted = m
return nil
}
type fakeReader struct{ metas []provider.PuppetMetadata }
func (f fakeReader) ListPuppetMetadataEntries(_ context.Context, _, slug string) ([]provider.PuppetMetadata, error) {
var out []provider.PuppetMetadata
for _, m := range f.metas {
if slug == "" || m.Slug == slug {
out = append(out, m)
}
}
return out, nil
}
func (fakeReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
return nil, nil
}
func (fakeReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
func TestValidateUpload(t *testing.T) {
p := &Provider{}
sp, ct, err := p.ValidateUpload("some/dir/Unkin-hello-0.1.0.tar.gz")
if err != nil || sp != "v3/files/unkin-hello-0.1.0.tar.gz" || ct != "application/gzip" {
t.Errorf("sp=%q ct=%q err=%v", sp, ct, err)
}
for _, bad := range []string{"hello-0.1.0.tar.gz", "unkin-hello-0.1.tar.gz", "unkin-hello-0.1.0.zip", "unkin-Hello_x-0.1.0.tgz"} {
if _, _, err := p.ValidateUpload(bad); err == nil {
t.Errorf("%q accepted", bad)
}
}
}
func TestAfterUploadRecordsMetadata(t *testing.T) {
raw := moduleTarball(t, "unkin-hello-0.1.0", helloMetadata)
store := &fakeStore{}
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
m := store.inserted
if m == nil {
t.Fatal("nothing inserted")
}
sum := md5.Sum(raw)
if m.Slug != "unkin-hello" || m.Version != "0.1.0" || m.MD5 != hex.EncodeToString(sum[:]) || m.Size != int64(len(raw)) || m.MetadataJSON != helloMetadata {
t.Errorf("got %+v", m)
}
}
func TestAfterUploadRejectsMismatchedFilename(t *testing.T) {
raw := moduleTarball(t, "unkin-hello-0.2.0", `{"name":"unkin-hello","version":"0.2.0"}`)
store := &fakeStore{}
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
if store.inserted != nil {
t.Error("mismatched version indexed")
}
}
func TestParseModuleRequiresTopLevelMetadata(t *testing.T) {
if _, err := parseModule(bytes.NewReader(moduleTarball(t, "a/b", helloMetadata))); err == nil {
t.Error("nested metadata.json accepted")
}
if _, err := parseModule(bytes.NewReader(moduleTarball(t, "x", `{"name":"unkin-hello"}`))); err == nil {
t.Error("missing version accepted")
}
}
func tarballOf(t *testing.T, entries ...*tar.Header) []byte {
t.Helper()
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
tw := tar.NewWriter(zw)
for _, h := range entries {
if err := tw.WriteHeader(h); err != nil {
t.Fatal(err)
}
if h.Typeflag == tar.TypeReg {
body := helloMetadata
if !strings.HasSuffix(h.Name, "metadata.json") {
body = strings.Repeat("x", int(h.Size))
}
_, _ = tw.Write([]byte(body))
}
}
_ = tw.Close()
_ = zw.Close()
return buf.Bytes()
}
func TestParseModuleSkipsNonRegularMetadata(t *testing.T) {
raw := tarballOf(t,
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json", Typeflag: tar.TypeSymlink, Linkname: "/etc/passwd"},
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json/", Typeflag: tar.TypeDir, Mode: 0o755},
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json", Typeflag: tar.TypeReg, Mode: 0o644, Size: int64(len(helloMetadata))},
)
meta, err := parseModule(bytes.NewReader(raw))
if err != nil || meta.Slug != "unkin-hello" || meta.Version != "0.1.0" {
t.Fatalf("got %v, %v", meta, err)
}
}
func TestParseModuleCapsDecompressedScan(t *testing.T) {
defer func(n int64) { maxModuleScan = n }(maxModuleScan)
maxModuleScan = 64 << 10
raw := tarballOf(t,
&tar.Header{Name: "unkin-hello-0.1.0/big", Typeflag: tar.TypeReg, Mode: 0o644, Size: 1 << 20},
&tar.Header{Name: "unkin-hello-0.1.0/metadata.json", Typeflag: tar.TypeReg, Mode: 0o644, Size: int64(len(helloMetadata))},
)
if _, err := parseModule(bytes.NewReader(raw)); err == nil || !strings.Contains(err.Error(), "scan limit") {
t.Fatalf("expected scan limit error, got %v", err)
}
maxModuleScan = 2 << 20
if _, err := parseModule(bytes.NewReader(raw)); err != nil {
t.Fatalf("under cap: %v", err)
}
}
func TestAfterUploadCapsTarballSize(t *testing.T) {
defer func(n int64) { maxModuleTarball = n }(maxModuleTarball)
raw := moduleTarball(t, "unkin-hello-0.1.0", helloMetadata)
maxModuleTarball = int64(len(raw)) - 1
store := &fakeStore{}
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
if store.inserted != nil {
t.Error("oversized tarball indexed")
}
maxModuleTarball = int64(len(raw))
(&Provider{}).AfterUpload(context.Background(), "forge", "v3/files/unkin-hello-0.1.0.tar.gz", "sha256:abc", fakeBlobs{raw}, store)
if store.inserted == nil {
t.Error("tarball at cap not indexed")
}
}
func TestReleaseListClampsPagination(t *testing.T) {
files := releases("0.1.0", "0.2.0", "0.3.0")
for _, q := range []string{"offset=9223372036854775807", "offset=9223372036854775807&limit=9223372036854775807", "offset=-5&limit=-5", "limit=9223372036854775807"} {
code, body := get(t, files, "v3/releases?module=unkin-hello&"+q)
if code != http.StatusOK {
t.Fatalf("%s: code %d", q, code)
}
pg := body["pagination"].(map[string]any)
if l := pg["limit"].(float64); l < 1 || l > 100 {
t.Errorf("%s: limit %v", q, l)
}
if o := pg["offset"].(float64); o < 0 || o > 3 {
t.Errorf("%s: offset %v", q, o)
}
}
_, body := get(t, files, "v3/releases?module=unkin-hello&offset=9223372036854775807")
if len(body["results"].([]any)) != 0 || body["pagination"].(map[string]any)["next"] != nil {
t.Errorf("past-end page: %v", body)
}
}
func get(t *testing.T, files provider.FileStore, target string) (int, map[string]any) {
t.Helper()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/x/"+target, nil)
path, _, _ := strings.Cut(target, "?")
if !(&Provider{}).ServeLocalIndex(rec, req, files, "forge", path) {
return 0, nil
}
var body map[string]any
_ = json.Unmarshal(rec.Body.Bytes(), &body)
return rec.Code, body
}
func releases(versions ...string) fakeReader {
var f fakeReader
for i, v := range versions {
f.metas = append(f.metas, provider.PuppetMetadata{
FilePath: "v3/files/unkin-hello-" + v + ".tar.gz", ContentHash: "sha256:feed", Slug: "unkin-hello",
Version: v, MetadataJSON: `{"name":"unkin-hello","version":"` + v + `"}`, MD5: "m" + v,
CreatedAt: time.Unix(int64(i), 0),
})
}
f.metas = append(f.metas, provider.PuppetMetadata{Slug: "other-mod", Version: "9.9.9", MetadataJSON: `{}`})
return f
}
func TestReleaseListPaginatesNewestFirst(t *testing.T) {
files := releases("0.1.0", "0.10.0", "0.2.0", "1.0.0-rc.1", "1.0.0")
code, body := get(t, files, "v3/releases?module=unkin/hello&limit=2")
if code != http.StatusOK {
t.Fatalf("code %d", code)
}
results := body["results"].([]any)
pg := body["pagination"].(map[string]any)
if len(results) != 2 || results[0].(map[string]any)["version"] != "1.0.0" || results[1].(map[string]any)["version"] != "1.0.0-rc.1" {
t.Fatalf("page 1: %v", results)
}
if pg["total"] != float64(5) || pg["next"] != "/v3/releases?limit=2&module=unkin-hello&offset=2" || pg["previous"] != nil {
t.Errorf("pagination: %v", pg)
}
_, body = get(t, files, "v3/releases?module=unkin-hello&limit=2&offset=4")
results = body["results"].([]any)
pg = body["pagination"].(map[string]any)
if len(results) != 1 || results[0].(map[string]any)["version"] != "0.1.0" || pg["next"] != nil || pg["previous"] != "/v3/releases?limit=2&module=unkin-hello&offset=2" {
t.Errorf("last page: %v %v", results, pg)
}
_, body = get(t, files, "v3/releases?module=nobody-none")
if len(body["results"].([]any)) != 0 {
t.Error("unknown module returned results")
}
}
func TestReleaseObject(t *testing.T) {
code, body := get(t, releases("0.1.0"), "v3/releases/unkin-hello-0.1.0")
if code != http.StatusOK {
t.Fatalf("code %d", code)
}
if body["file_uri"] != "/v3/files/unkin-hello-0.1.0.tar.gz" || body["file_md5"] != "m0.1.0" || body["file_sha256"] != "feed" {
t.Errorf("release: %v", body)
}
if body["metadata"].(map[string]any)["name"] != "unkin-hello" || body["module"].(map[string]any)["owner"].(map[string]any)["username"] != "unkin" {
t.Errorf("release refs: %v", body)
}
if code, _ := get(t, releases("0.1.0"), "v3/releases/unkin-hello-0.2.0"); code != http.StatusNotFound {
t.Errorf("missing version: %d", code)
}
}
func TestModuleObject(t *testing.T) {
code, body := get(t, releases("0.9.0", "0.10.0"), "v3/modules/unkin-hello")
if code != http.StatusOK {
t.Fatalf("code %d", code)
}
if body["slug"] != "unkin-hello" || body["name"] != "hello" || body["current_release"].(map[string]any)["version"] != "0.10.0" || len(body["releases"].([]any)) != 2 {
t.Errorf("module: %v", body)
}
if code, _ := get(t, releases(), "v3/modules/unkin-missing"); code != http.StatusNotFound {
t.Errorf("missing module: %d", code)
}
}
func TestFilesFallThrough(t *testing.T) {
if code, _ := get(t, releases(), "v3/files/unkin-hello-0.1.0.tar.gz"); code != 0 {
t.Error("file downloads must fall through to the stored file")
}
}
func TestCompareSemver(t *testing.T) {
ordered := []string{"0.1.0", "0.2.0", "0.10.0", "1.0.0-alpha", "1.0.0-alpha.1", "1.0.0-beta", "1.0.0-beta.2", "1.0.0-beta.11", "1.0.0", "1.0.1+build"}
for i := 1; i < len(ordered); i++ {
if compareSemver(ordered[i-1], ordered[i]) >= 0 || compareSemver(ordered[i], ordered[i-1]) <= 0 {
t.Errorf("%s should sort before %s", ordered[i-1], ordered[i])
}
}
if compareSemver("1.0.0+a", "1.0.0+b") != 0 {
t.Error("build metadata must not affect precedence")
}
}
+1
View File
@@ -29,6 +29,7 @@ import (
_ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
_ "git.unkin.net/unkin/artifactapi/internal/provider/helm"
_ "git.unkin.net/unkin/artifactapi/internal/provider/npm"
_ "git.unkin.net/unkin/artifactapi/internal/provider/nuget"
_ "git.unkin.net/unkin/artifactapi/internal/provider/puppet"
_ "git.unkin.net/unkin/artifactapi/internal/provider/pypi"
"git.unkin.net/unkin/artifactapi/internal/provider/rpm"
+15
View File
@@ -0,0 +1,15 @@
CREATE TABLE IF NOT EXISTS puppet_metadata (
id BIGSERIAL PRIMARY KEY,
repo_name TEXT NOT NULL,
file_path TEXT NOT NULL,
content_hash TEXT NOT NULL,
slug TEXT NOT NULL,
version TEXT NOT NULL,
metadata_json TEXT NOT NULL,
file_size BIGINT NOT NULL DEFAULT 0,
file_md5 TEXT NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(repo_name, file_path)
);
CREATE INDEX IF NOT EXISTS idx_puppet_metadata_repo_slug ON puppet_metadata(repo_name, slug);
+2
View File
@@ -17,6 +17,7 @@ const (
PackageTerraform PackageType = "terraform"
PackageGoProxy PackageType = "goproxy"
PackageCargo PackageType = "cargo"
PackageNuGet PackageType = "nuget"
PackageGitHubRPM PackageType = "github_rpm"
PackageGitHubDeb PackageType = "github_deb"
PackageGitHubAlpine PackageType = "github_alpine"
@@ -35,6 +36,7 @@ var validPackageTypes = map[PackageType]bool{
PackageTerraform: true,
PackageGoProxy: true,
PackageCargo: true,
PackageNuGet: true,
PackageGitHubRPM: true,
PackageGitHubDeb: true,
PackageGitHubAlpine: true,
+1
View File
@@ -19,6 +19,7 @@ func TestPackageTypeValid(t *testing.T) {
models.PackageTerraform,
models.PackageGoProxy,
models.PackageCargo,
models.PackageNuGet,
models.PackageGitHubRPM,
models.PackageGitHubDeb,
models.PackageGitHubAlpine,
+59 -10
View File
@@ -218,15 +218,38 @@ sudo apk add --allow-untrusted <package>`,
];
case 'goproxy':
return [
{
title: 'Point the Go module proxy here',
language: 'bash',
code: `export GOPROXY=${proxy}
return isLocal
? [
{
title: 'Fetch private modules from this proxy',
language: 'bash',
code: `export GOPROXY=${url}/api/v1/local/${name},https://proxy.golang.org,direct
export GONOSUMDB=example.unkin.net
go get example.unkin.net/hello@latest`,
note: 'Set GONOSUMDB (or GOPRIVATE) to your module prefix: private modules are not in sum.golang.org.',
},
{
title: 'Publish a module version',
language: 'bash',
code: `# build the zip with golang.org/x/mod/zip (CreateFromDir), or reuse
# the .zip/.mod/.info from $(go env GOMODCACHE)/cache/download after go mod download
MOD=example.unkin.net/hello VER=v0.1.0
for ext in zip mod info; do
curl -fsSL --upload-file $VER.$ext \\
${url}/api/v2/remotes/${name}/files/$MOD/@v/$VER.$ext
done`,
note: 'Paths use the case-encoded module path (uppercase as !lower). Zips must hold every file under module@version/ and are rejected otherwise; .info is optional.',
},
]
: [
{
title: 'Point the Go module proxy here',
language: 'bash',
code: `export GOPROXY=${proxy}
go mod download`,
note: 'Append ,direct to fall back to VCS for modules this proxy does not cover.',
},
];
note: 'Append ,direct to fall back to VCS for modules this proxy does not cover.',
},
];
case 'cargo':
return [
@@ -243,14 +266,40 @@ index = "sparse+${proxy}/"`,
},
];
case 'nuget':
return [
{
title: 'Use this feed instead of nuget.org',
language: 'xml',
code: `<!-- nuget.config -->
<configuration>
<packageSources>
<clear />
<add key="${name}" value="${proxy}/index.json" protocolVersion="3" />
</packageSources>
</configuration>`,
note: 'Serve artifactapi over HTTPS: NuGet rejects the repository signatures resource over plain HTTP.',
},
];
case 'puppet':
return [
{
title: 'Install a module from the Forge proxy',
title: isLocal ? 'Install a module from this forge' : 'Install a module from the Forge proxy',
language: 'bash',
code: `puppet module install <author>-<module> \\
--module_repository ${proxy}`,
--module_repository ${isLocal ? `${url}/api/v1/local/${name}` : proxy}`,
},
...(isLocal
? [
{
title: 'Publish a module (versions are immutable)',
language: 'bash',
code: `puppet module build && curl -fsSL --upload-file pkg/<author>-<module>-<version>.tar.gz \\
${url}/api/v2/remotes/${name}/files/<author>-<module>-<version>.tar.gz`,
},
]
: []),
];
case 'deb':
+1
View File
@@ -18,6 +18,7 @@ const typeColors: Record<string, 'blue' | 'green' | 'yellow' | 'red' | 'default'
terraform: 'blue',
goproxy: 'green',
cargo: 'yellow',
nuget: 'blue',
};
export function Locals() {
+1
View File
@@ -18,6 +18,7 @@ const typeColors: Record<string, 'blue' | 'green' | 'yellow' | 'red' | 'default'
terraform: 'blue',
goproxy: 'green',
cargo: 'yellow',
nuget: 'blue',
};
export function Remotes() {