Add alpine virtual repositories (#164)

Virtual repos cover rpm, deb, npm and others but not alpine, so a local alpine repo and github_alpine remotes cannot be served as one apk repository.

- merge members' `<dir>/APKINDEX.tar.gz` per directory; member order wins duplicate (P, V, A); a member 404 is skipped
- 302 `<dir>/<P>-<V>.apk` downloads to the member whose index listed the package
- serve the merged index unsigned, like local and github_alpine indexes
- cache merges per directory on the shared virtual cache; member errors fail with 502
- add docs/alpine/virtual.md, unit tests and an apk e2e with abuild-built packages

Requires unkin/artifactapi#163.

Reviewed-on: #164
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>

README.md

@@ -26,7 +26,7 @@ Classification precedence, TTLs and eviction for every remote: [docs/common/cach
 
 Each type's paths, Terraform examples and local/virtual behaviour are in `docs/<type>/`:
 
-- `alpine`: [remote](docs/alpine/remote.md), [remote-github](docs/alpine/remote-github.md)
+- `alpine`: [remote](docs/alpine/remote.md), [remote-github](docs/alpine/remote-github.md), [virtual](docs/alpine/virtual.md)
 - `ansible`: [remote](docs/ansible/remote.md), [local](docs/ansible/local.md), [virtual](docs/ansible/virtual.md)
 - `cargo`: [remote](docs/cargo/remote.md)
 - `deb`: [remote](docs/deb/remote.md), [remote-github](docs/deb/remote-github.md), [virtual](docs/deb/virtual.md)
@@ -41,6 +41,7 @@ Each type's paths, Terraform examples and local/virtual behaviour are in `docs/<
 - `pypi`: [remote](docs/pypi/remote.md)
 - `rpm`: [remote](docs/rpm/remote.md), [remote-github](docs/rpm/remote-github.md)
 - `terraform`: [remote](docs/terraform/remote.md), [local](docs/terraform/local.md)
+- `vcs`: [remote](docs/vcs/remote.md)
 
 ### GitHub authentication
 

docs/alpine/virtual.md

@@ -0,0 +1,30 @@
+# `alpine` virtual
+
+An alpine virtual serves one apk repository over local and remote alpine
+members at `/api/v1/virtual/<name>/`.
+
+- `<dir>/APKINDEX.tar.gz` merges each member's index for the same `<dir>`
+  (e.g. `x86_64/`); on a duplicate (P, V, A) the earlier member wins. A member
+  that 404s for that directory is skipped.
+- apk builds download paths itself (`<dir>/<P>-<V>.apk`), so the virtual 302s
+  each download to the member whose index listed it.
+- The merged index is unsigned, like local and `github_alpine` indexes: clients
+  need `--allow-untrusted` (packages keep their own signatures).
+- Any other member error fails the request with 502. Merges are cached for 60s
+  per directory, up to 32 directories per virtual.
+
+```hcl
+resource "artifactapi_virtual" "alpine" {
+  name         = "alpine"
+  package_type = "alpine"
+  members      = [artifactapi_local_alpine.internal.name, artifactapi_remote_github_alpine.tool.name]
+}
+```
+
+```sh
+echo https://artifactapi.example.com/api/v1/virtual/alpine >> /etc/apk/repositories
+apk update --allow-untrusted
+apk add --allow-untrusted mytool
+```
+
+Upload to the local member directly; the virtual is read-only.

docs/common/caching.md

@@ -98,4 +98,4 @@ Each type's page lists its auto-detected mutable and immutable paths:
 [docker](../docker/remote.md), [generic](../generic/remote.md), [gems](../gems/remote.md), [goproxy](../goproxy/remote.md),
 [helm](../helm/remote.md), [npm](../npm/remote.md), [nuget](../nuget/remote.md),
 [puppet](../puppet/remote.md), [pypi](../pypi/remote.md), [rpm](../rpm/remote.md),
-[rpm github](../rpm/remote-github.md), [terraform](../terraform/remote.md).
+[rpm github](../rpm/remote-github.md), [terraform](../terraform/remote.md), [vcs](../vcs/remote.md).

docs/vcs/remote.md

@@ -0,0 +1,29 @@
+# `vcs` remote
+
+Shared classification, TTL and eviction rules: [caching](../common/caching.md).
+
+Caches source archives and release assets from GitHub (`base_url = "https://github.com"`) or Gitea (any other `base_url`). GitHub Enterprise is not supported: a non-github.com host is treated as Gitea. Only the paths below are served; anything else is a 404.
+
+| Path | Behaviour |
+|---|---|
+| `<owner>/<repo>/archive/refs/tags/<tag>.(tar.gz\|zip)` | immutable |
+| `<owner>/<repo>/archive/<40-char sha>.(tar.gz\|zip)` | immutable |
+| `<owner>/<repo>/releases/download/<tag>/<asset>` | immutable |
+| `<owner>/<repo>/releases/latest/download/<asset>` | mutable |
+| `<owner>/<repo>/archive/refs/heads/<branch>.(tar.gz\|zip)` | `302` to the archive of the branch's current commit, resolved via the provider's branch API and reused for `mutable_ttl` |
+
+Branch archives redirect, so `curl` needs `-L`; pip and npm follow redirects. The archive is the host's commit archive (GitHub names its top-level directory `<repo>-<sha>`). `patterns` and `blocklist` match both the branch path and the commit path it redirects to, so allowlist repos by prefix (`^cli/cli/`); a branch whose commit path is denied is a `403`. A missing branch is a `404`, cached for `mutable_ttl` (at least 60s). An API rate limit (`429`, or `403` with rate-limit headers) is a `503` with `Retry-After`, unless `stale_on_error` serves the last known commit. Each remote caches up to 4096 branch lookups. GitHub API calls use the [server GitHub credential](../../README.md#github-authentication); Gitea API calls use the remote's `username`/`password`.
+
+```hcl
+resource "artifactapi_remote_vcs" "github" {
+  name        = "github-vcs"
+  base_url    = "https://github.com"
+  mutable_ttl = 600
+  patterns    = ["^cli/cli/", "^psf/requests/"]
+}
+```
+
+```bash
+curl -fsSLO https://artifactapi.example/api/v1/remote/github-vcs/cli/cli/archive/refs/tags/v2.60.0.tar.gz
+pip install https://artifactapi.example/api/v1/remote/github-vcs/psf/requests/archive/refs/tags/v2.32.3.tar.gz
+```

e2e-docker/alpine_virtual_test.go

@@ -0,0 +1,105 @@
+//go:build dockere2e
+
+package e2edocker
+
+import (
+	"net/http"
+	"os"
+	"os/exec"
+	"strings"
+	"testing"
+	"time"
+)
+
+// abuildScript builds a real, abuild-signed $N-$V-r0.apk and writes it to stdout.
+const abuildScript = `set -eu
+apk add -q abuild >&2
+adduser -D b && addgroup b abuild
+su b -c 'set -eu; abuild-keygen -an >&2; mkdir -p ~/p && cd ~/p
+printf "%s\n" "pkgname=$N" "pkgver=$V" pkgrel=0 "pkgdesc=\"e2e\"" "url=\"https://example.com\"" "arch=\"x86_64\"" "license=\"MIT\"" "options=\"!check\"" \
+  "package() { mkdir -p \"\$pkgdir/usr/share/$N\"; echo ok > \"\$pkgdir/usr/share/$N/ok\"; }" > APKBUILD
+abuild -d >&2 || true'
+cat /home/b/packages/*/x86_64/$N-$V-r0.apk`
+
+func buildApk(t *testing.T, name, version string) []byte {
+	t.Helper()
+	cmd := exec.Command("docker", "run", "--rm", "-e", "N="+name, "-e", "V="+version, "alpine:3.20", "sh", "-c", abuildScript)
+	var stderr strings.Builder
+	cmd.Stderr = &stderr
+	out, err := cmd.Output()
+	if err != nil || len(out) == 0 {
+		t.Fatalf("abuild %s-%s: %v\n%s", name, version, err, stderr.String())
+	}
+	return out
+}
+
+// TestVirtualAlpineMerge merges two local alpine repos carrying the same
+// abuild-built package: the first member wins the duplicate and its download,
+// and a real apk installs packages from both members through the virtual.
+func TestVirtualAlpineMerge(t *testing.T) {
+	network := os.Getenv("COMPOSE_NETWORK")
+	internal := os.Getenv("ARTIFACTAPI_INTERNAL")
+	if network == "" || internal == "" {
+		t.Skip("COMPOSE_NETWORK/ARTIFACTAPI_INTERNAL not set; skipping real apk")
+	}
+	if _, err := exec.LookPath("docker"); err != nil {
+		t.Skip("docker not available on the test host")
+	}
+	one, two := buildApk(t, "vapk-one", "1.0"), buildApk(t, "vapk-two", "2.0")
+
+	createRepo(t, `{"name":"vapk-a","package_type":"alpine","repo_type":"local"}`)
+	createRepo(t, `{"name":"vapk-b","package_type":"alpine","repo_type":"local"}`)
+	defer deleteRepo(t, "vapk-a")
+	defer deleteRepo(t, "vapk-b")
+
+	const apkType = "application/vnd.android.package-archive"
+	uploadFile(t, "vapk-a", "x86_64/vapk-one-1.0-r0.apk", one, apkType)
+	uploadFile(t, "vapk-b", "x86_64/vapk-one-1.0-r0.apk", one, apkType)
+	uploadFile(t, "vapk-b", "x86_64/vapk-two-2.0-r0.apk", two, apkType)
+
+	// Local indexes list uploads once their metadata is parsed asynchronously,
+	// and the virtual caches its merge, so merge only once both members list all.
+	for _, m := range []struct{ repo, pkg string }{{"vapk-a", "P:vapk-one\n"}, {"vapk-b", "P:vapk-two\n"}} {
+		var idx string
+		for deadline := time.Now().Add(15 * time.Second); !strings.Contains(idx, m.pkg) && time.Now().Before(deadline); time.Sleep(250 * time.Millisecond) {
+			_, body := doRequest(t, http.MethodGet, api("/api/v1/local/"+m.repo+"/x86_64/APKINDEX.tar.gz"), nil, "")
+			idx = apkIndexText(t, body)
+		}
+	}
+
+	createVirtual(t, `{"name":"vapk","package_type":"alpine","members":["vapk-a","vapk-b"]}`)
+	defer deleteVirtual(t, "vapk")
+
+	resp, body := doRequest(t, http.MethodGet, api("/api/v1/virtual/vapk/x86_64/APKINDEX.tar.gz"), nil, "")
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("virtual APKINDEX: status %d: %s", resp.StatusCode, body)
+	}
+	index := apkIndexText(t, body)
+	if strings.Count(index, "P:vapk-one\n") != 1 || !strings.Contains(index, "P:vapk-two\n") {
+		t.Fatalf("merged APKINDEX should hold vapk-one once and vapk-two:\n%s", index)
+	}
+
+	noRedirect := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
+	for pkg, member := range map[string]string{"vapk-one-1.0-r0.apk": "vapk-a", "vapk-two-2.0-r0.apk": "vapk-b"} {
+		resp, err := noRedirect.Get(api("/api/v1/virtual/vapk/x86_64/" + pkg))
+		if err != nil {
+			t.Fatal(err)
+		}
+		resp.Body.Close()
+		if loc := resp.Header.Get("Location"); resp.StatusCode != http.StatusFound || !strings.HasSuffix(loc, "/api/v1/local/"+member+"/x86_64/"+pkg) {
+			t.Fatalf("%s: want 302 to %s, got %d %q", pkg, member, resp.StatusCode, loc)
+		}
+	}
+
+	script := "set -eu; " +
+		"echo \"$URL\" > /etc/apk/repositories; " +
+		"apk update --allow-untrusted; " +
+		"apk add --allow-untrusted vapk-one vapk-two; " +
+		"test -f /usr/share/vapk-one/ok && test -f /usr/share/vapk-two/ok"
+	out, err := exec.Command("docker", "run", "--rm", "--network", network,
+		"-e", "URL="+strings.TrimRight(internal, "/")+"/api/v1/virtual/vapk",
+		"alpine:3.20", "sh", "-c", script).CombinedOutput()
+	if err != nil {
+		t.Fatalf("real apk against alpine virtual failed: %v\n%s", err, out)
+	}
+}

e2e-docker/caching_vcs_test.go

@@ -0,0 +1,7 @@
+//go:build dockere2e
+
+package e2edocker
+
+func init() {
+	cachingCases = append(cachingCases, cachingCase{"vcs", "octo/repo/archive/refs/tags/v1.0.0.tar.gz", "octo/repo/archive/refs/tags/v1.0.0.tar.gz"})
+}

e2e-docker/local_test.go

@@ -150,31 +150,15 @@ func TestLocalAlpineIndex(t *testing.T) {
 	apk := testsupport.MinimalApk("e2e-testpkg", "1.0-r0", "x86_64")
 	uploadFile(t, "local-alpine", "x86_64/e2e-testpkg-1.0-r0.apk", apk, "application/vnd.android.package-archive")
 
-	// The index is generated asynchronously after upload; poll for it.
-	resp, body := getEventually(t, api("/api/v1/local/local-alpine/x86_64/APKINDEX.tar.gz"), 15*time.Second)
-	if resp.StatusCode != http.StatusOK {
-		t.Fatalf("APKINDEX: status %d: %s", resp.StatusCode, body)
-	}
-
-	zr, err := gzip.NewReader(bytes.NewReader(body))
-	if err != nil {
-		t.Fatalf("APKINDEX not gzip: %v", err)
-	}
-	tarBytes, _ := io.ReadAll(zr)
-	tr := tar.NewReader(bytes.NewReader(tarBytes))
+	// The index is served (empty) at once but lists the package only once the
+	// upload's metadata is parsed asynchronously; poll until it does.
 	var index string
-	for {
-		hdr, err := tr.Next()
-		if err == io.EOF {
-			break
-		}
-		if err != nil {
-			t.Fatalf("APKINDEX not tar: %v", err)
-		}
-		if hdr.Name == "APKINDEX" {
-			b, _ := io.ReadAll(tr)
-			index = string(b)
+	for deadline := time.Now().Add(15 * time.Second); !strings.Contains(index, "P:e2e-testpkg") && time.Now().Before(deadline); time.Sleep(250 * time.Millisecond) {
+		resp, body := doRequest(t, http.MethodGet, api("/api/v1/local/local-alpine/x86_64/APKINDEX.tar.gz"), nil, "")
+		if resp.StatusCode != http.StatusOK {
+			t.Fatalf("APKINDEX: status %d: %s", resp.StatusCode, body)
 		}
+		index = apkIndexText(t, body)
 	}
 	for _, want := range []string{"P:e2e-testpkg", "V:1.0-r0", "A:x86_64", "C:Q1", "S:", "I:"} {
 		if !strings.Contains(index, want) {
@@ -183,7 +167,7 @@ func TestLocalAlpineIndex(t *testing.T) {
 	}
 
 	// The .apk downloads back byte-identical from its arch path.
-	resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-alpine/x86_64/e2e-testpkg-1.0-r0.apk"), nil, "")
+	resp, body := doRequest(t, http.MethodGet, api("/api/v1/local/local-alpine/x86_64/e2e-testpkg-1.0-r0.apk"), nil, "")
 	if resp.StatusCode != http.StatusOK {
 		t.Fatalf("download apk: status %d: %s", resp.StatusCode, body)
 	}
@@ -191,3 +175,25 @@ func TestLocalAlpineIndex(t *testing.T) {
 		t.Fatalf("apk content mismatch")
 	}
 }
+
+func apkIndexText(t *testing.T, body []byte) string {
+	t.Helper()
+	zr, err := gzip.NewReader(bytes.NewReader(body))
+	if err != nil {
+		t.Fatalf("APKINDEX not gzip: %v", err)
+	}
+	tr := tar.NewReader(zr)
+	for {
+		hdr, err := tr.Next()
+		if err == io.EOF {
+			return ""
+		}
+		if err != nil {
+			t.Fatalf("APKINDEX not tar: %v", err)
+		}
+		if hdr.Name == "APKINDEX" {
+			b, _ := io.ReadAll(tr)
+			return string(b)
+		}
+	}
+}

internal/database/database_test.go

@@ -307,6 +307,34 @@ func TestLocalFiles(t *testing.T) {
 	}
 }
 
+// A repo recreated under a deleted repo's name must not inherit its files or
+// metadata, or re-uploading the same path 409s.
+func TestDeleteRemoteDropsLocalFiles(t *testing.T) {
+	requireDB(t)
+	seedRemote(t, "r-recreate")
+	hash := seedBlob(t, "recreatehash")
+	if err := testDB.CreateLocalFile(ctx(), "r-recreate", "x86_64/a-1.0-r0.apk", hash); err != nil {
+		t.Fatal(err)
+	}
+	if err := testDB.InsertAlpineMetadata(ctx(), &provider.AlpineMetadata{RepoName: "r-recreate", FilePath: "x86_64/a-1.0-r0.apk", ContentHash: hash, Name: "a", Version: "1.0-r0", Arch: "x86_64"}); err != nil {
+		t.Fatal(err)
+	}
+	if err := testDB.DeleteRemote(ctx(), "r-recreate"); err != nil {
+		t.Fatal(err)
+	}
+	seedRemote(t, "r-recreate")
+	defer testDB.DeleteRemote(ctx(), "r-recreate")
+	if f, err := testDB.GetLocalFile(ctx(), "r-recreate", "x86_64/a-1.0-r0.apk"); err != nil || f != nil {
+		t.Fatalf("recreated repo inherited local file: %v %v", f, err)
+	}
+	if rows, err := testDB.ListAlpineMetadataEntries(ctx(), "r-recreate"); err != nil || len(rows) != 0 {
+		t.Fatalf("recreated repo inherited alpine metadata: %v %v", rows, err)
+	}
+	if err := testDB.CreateLocalFile(ctx(), "r-recreate", "x86_64/a-1.0-r0.apk", hash); err != nil {
+		t.Fatalf("re-upload after recreate: %v", err)
+	}
+}
+
 func TestVirtualsCRUD(t *testing.T) {
 	requireDB(t)
 	if err := testDB.CreateVirtual(ctx(), &models.Virtual{

internal/database/remotes.go

@@ -3,6 +3,8 @@ package database
 import (
 	"context"
 
+	"github.com/jackc/pgx/v5"
+
 	"git.unkin.net/unkin/artifactapi/pkg/models"
 )
 
@@ -116,7 +118,16 @@ func (db *DB) UpdateRemote(ctx context.Context, r *models.Remote) error {
 	return err
 }
 
+// DeleteRemote drops a remote with its local files and package metadata, so a
+// repo recreated under the same name starts empty.
 func (db *DB) DeleteRemote(ctx context.Context, name string) error {
-	_, err := db.Pool.Exec(ctx, `DELETE FROM remotes WHERE name = $1`, name)
-	return err
+	return pgx.BeginFunc(ctx, db.Pool, func(tx pgx.Tx) error {
+		for _, table := range []string{"local_files", "rpm_metadata", "deb_metadata", "alpine_metadata", "puppet_metadata", "helm_metadata"} {
+			if _, err := tx.Exec(ctx, `DELETE FROM `+table+` WHERE repo_name = $1`, name); err != nil {
+				return err
+			}
+		}
+		_, err := tx.Exec(ctx, `DELETE FROM remotes WHERE name = $1`, name)
+		return err
+	})
 }

internal/provider/all/vcs.go

@@ -0,0 +1,3 @@
+package all
+
+import _ "git.unkin.net/unkin/artifactapi/internal/provider/vcs"

internal/provider/generic/generic.go

@@ -71,7 +71,7 @@ func (p *Provider) AuthHeaders(ctx context.Context, remote models.Remote) (http.
 		h.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(remote.Username+":"+remote.Password)))
 		return h, nil
 	}
-	if isGitHubHost(remote.BaseURL) {
+	if IsGitHubHost(remote.BaseURL) {
 		if c := githubauth.Server(); c != nil {
 			tok, err := c.Token(ctx)
 			if err != nil {
@@ -85,11 +85,11 @@ func (p *Provider) AuthHeaders(ctx context.Context, remote models.Remote) (http.
 	return h, nil
 }
 
-// isGitHubHost reports whether rawURL targets a GitHub API/download host that
+// IsGitHubHost reports whether rawURL targets a GitHub API/download host that
 // accepts the server credential. objects.githubusercontent.com is deliberately
 // excluded: release-asset downloads 302-redirect there with a pre-signed URL
 // that must not carry an Authorization header.
-func isGitHubHost(rawURL string) bool {
+func IsGitHubHost(rawURL string) bool {
 	u, err := url.Parse(rawURL)
 	if err != nil {
 		return false

internal/provider/vcs/vcs.go

@@ -0,0 +1,309 @@
+// Package vcs caches source archives and release assets from GitHub or Gitea,
+// keyed by ref: tags, commits and release assets are immutable, and branch
+// archives redirect to the archive of the commit the branch currently points at.
+package vcs
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"regexp"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"golang.org/x/sync/singleflight"
+
+	"git.unkin.net/unkin/artifactapi/internal/provider"
+	"git.unkin.net/unkin/artifactapi/internal/provider/generic"
+	"git.unkin.net/unkin/artifactapi/internal/proxy"
+	"git.unkin.net/unkin/artifactapi/pkg/models"
+)
+
+func init() {
+	provider.Register(New())
+}
+
+type kind int
+
+const (
+	tagArchive kind = iota + 1
+	branchArchive
+	commitArchive
+	releaseAsset
+	latestReleaseAsset
+)
+
+// target is a parsed vcs request path.
+type target struct {
+	Kind        kind
+	Owner, Repo string
+	Name        string // tag, branch, commit or asset
+	Ext         string
+}
+
+var (
+	segmentRe = regexp.MustCompile(`^[A-Za-z0-9._+-]+$`)
+	commitRe  = regexp.MustCompile(`^[0-9a-f]{40}$`)
+)
+
+var errNotFound = errors.New("ref not found")
+
+const (
+	maxRefs     = 4096
+	negativeTTL = time.Minute
+)
+
+// Provider embeds generic for upstream URLs, content types and GitHub/Basic auth.
+type Provider struct {
+	generic.Provider
+
+	client    *http.Client
+	githubAPI string
+
+	group singleflight.Group
+	mu    sync.Mutex
+	refs  map[string]map[string]resolved
+}
+
+// resolved is a branch lookup; an empty commit records a missing branch.
+type resolved struct {
+	commit string
+	at     time.Time
+}
+
+func New() *Provider {
+	return &Provider{
+		client:    &http.Client{Timeout: 30 * time.Second},
+		githubAPI: "https://api.github.com",
+		refs:      map[string]map[string]resolved{},
+	}
+}
+
+func (p *Provider) Type() models.PackageType { return models.PackageVCS }
+
+func (p *Provider) Classify(path string) provider.Mutability {
+	if ref, ok := parsePath(path); ok && (ref.Kind == branchArchive || ref.Kind == latestReleaseAsset) {
+		return provider.Mutable
+	}
+	return provider.Immutable
+}
+
+// parsePath accepts only the archive and release-download shapes, rejecting
+// any segment that could escape owner/repo or a ref.
+func parsePath(path string) (target, bool) {
+	parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3)
+	if len(parts) != 3 || !validSegment(parts[0]) || !validSegment(parts[1]) {
+		return target{}, false
+	}
+	ref := target{Owner: parts[0], Repo: parts[1]}
+	rest := parts[2]
+	switch {
+	case strings.HasPrefix(rest, "archive/"):
+		name, ext, ok := archiveName(strings.TrimPrefix(rest, "archive/"))
+		if !ok {
+			return target{}, false
+		}
+		ref.Ext = ext
+		switch {
+		case strings.HasPrefix(name, "refs/tags/"):
+			ref.Kind, ref.Name = tagArchive, strings.TrimPrefix(name, "refs/tags/")
+		case strings.HasPrefix(name, "refs/heads/"):
+			ref.Kind, ref.Name = branchArchive, strings.TrimPrefix(name, "refs/heads/")
+		case commitRe.MatchString(name):
+			return target{Kind: commitArchive, Owner: ref.Owner, Repo: ref.Repo, Name: name, Ext: ext}, true
+		default:
+			return target{}, false
+		}
+		return ref, validRefName(ref.Name)
+	case strings.HasPrefix(rest, "releases/latest/download/"):
+		ref.Kind, ref.Name = latestReleaseAsset, strings.TrimPrefix(rest, "releases/latest/download/")
+		return ref, validSegment(ref.Name)
+	case strings.HasPrefix(rest, "releases/download/"):
+		tagAsset := strings.TrimPrefix(rest, "releases/download/")
+		i := strings.LastIndex(tagAsset, "/")
+		if i < 0 {
+			return target{}, false
+		}
+		ref.Kind, ref.Name = releaseAsset, tagAsset[i+1:]
+		return ref, validRefName(tagAsset[:i]) && validSegment(ref.Name)
+	}
+	return target{}, false
+}
+
+func archiveName(s string) (string, string, bool) {
+	for _, ext := range []string{".tar.gz", ".zip"} {
+		if name, ok := strings.CutSuffix(s, ext); ok && name != "" {
+			return name, ext, true
+		}
+	}
+	return "", "", false
+}
+
+func validSegment(s string) bool {
+	return segmentRe.MatchString(s) && s != "." && s != ".." && !strings.Contains(s, "..")
+}
+
+func validRefName(s string) bool {
+	for _, seg := range strings.Split(s, "/") {
+		if !validSegment(seg) || strings.HasPrefix(seg, ".") {
+			return false
+		}
+	}
+	return true
+}
+
+// ServeRemote rejects unsupported paths and redirects branch archives to the
+// immutable archive of the branch's current commit; everything else is proxied.
+func (p *Provider) ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, _ provider.RemoteMetadataStore) bool {
+	ref, ok := parsePath(path)
+	if !ok {
+		http.Error(w, "unsupported vcs path", http.StatusNotFound)
+		return true
+	}
+	if ref.Kind != branchArchive {
+		return false
+	}
+	classifier := proxy.NewClassifier(p)
+	if classifier.Classify(remote, path) == proxy.ClassDenied {
+		http.Error(w, "access denied", http.StatusForbidden)
+		return true
+	}
+	commit, err := p.resolve(r.Context(), remote, ref)
+	var se *provider.UpstreamStatusError
+	switch {
+	case errors.Is(err, errNotFound):
+		http.Error(w, "branch not found", http.StatusNotFound)
+		return true
+	case errors.As(err, &se) && (se.Status == http.StatusTooManyRequests || se.Status == http.StatusForbidden && !se.RetryAt.IsZero()):
+		secs := 60
+		if !se.RetryAt.IsZero() {
+			secs = max(1, int(time.Until(se.RetryAt).Seconds()+1))
+		}
+		w.Header().Set("Retry-After", strconv.Itoa(secs))
+		http.Error(w, "upstream rate limited", http.StatusServiceUnavailable)
+		return true
+	case err != nil:
+		http.Error(w, "bad gateway", http.StatusBadGateway)
+		return true
+	}
+	target := ref.Owner + "/" + ref.Repo + "/archive/" + commit + ref.Ext
+	if classifier.Classify(remote, target) == proxy.ClassDenied {
+		http.Error(w, "access denied", http.StatusForbidden)
+		return true
+	}
+	http.Redirect(w, r, strings.TrimRight(proxyBaseURL, "/")+provider.RemoteMountPath+remote.Name+"/"+target, http.StatusFound)
+	return true
+}
+
+// resolve maps a branch to its commit, reusing the answer for mutable_ttl (a
+// missing branch for at least negativeTTL) and, with stale_on_error, past an
+// upstream failure. Concurrent lookups of one branch share a single API call.
+func (p *Provider) resolve(ctx context.Context, remote models.Remote, ref target) (string, error) {
+	key := ref.Owner + "/" + ref.Repo + "\x00" + ref.Name
+	ttl := time.Duration(remote.MutableTTL) * time.Second
+	p.mu.Lock()
+	prev, ok := p.refs[remote.Name][key]
+	p.mu.Unlock()
+	if ok && prev.commit != "" && time.Since(prev.at) < ttl {
+		return prev.commit, nil
+	}
+	if ok && prev.commit == "" && time.Since(prev.at) < max(ttl, negativeTTL) {
+		return "", errNotFound
+	}
+	v, err, _ := p.group.Do(remote.Name+"\x00"+key, func() (any, error) {
+		commit, err := p.lookup(context.WithoutCancel(ctx), remote, ref)
+		if err == nil || errors.Is(err, errNotFound) {
+			p.store(remote.Name, key, commit)
+		}
+		return commit, err
+	})
+	if err != nil {
+		if ok && prev.commit != "" && remote.StaleOnError && !errors.Is(err, errNotFound) {
+			return prev.commit, nil
+		}
+		return "", err
+	}
+	return v.(string), nil
+}
+
+// store records a lookup, evicting the remote's oldest entry once it holds maxRefs.
+func (p *Provider) store(remote, key, commit string) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	m := p.refs[remote]
+	if m == nil {
+		m = map[string]resolved{}
+		p.refs[remote] = m
+	}
+	if _, ok := m[key]; !ok && len(m) >= maxRefs {
+		var oldest string
+		for k, e := range m {
+			if oldest == "" || e.at.Before(m[oldest].at) {
+				oldest = k
+			}
+		}
+		delete(m, oldest)
+	}
+	m[key] = resolved{commit: commit, at: time.Now()}
+}
+
+func (p *Provider) lookup(ctx context.Context, remote models.Remote, ref target) (string, error) {
+	segs := strings.Split(ref.Name, "/")
+	for i, s := range segs {
+		segs[i] = url.PathEscape(s)
+	}
+	branch := strings.Join(segs, "/")
+	github := generic.IsGitHubHost(remote.BaseURL)
+	api := strings.TrimRight(remote.BaseURL, "/") + "/api/v1/repos/" + ref.Owner + "/" + ref.Repo + "/branches/" + branch
+	if github {
+		api = p.githubAPI + "/repos/" + ref.Owner + "/" + ref.Repo + "/git/ref/heads/" + branch
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, api, nil)
+	if err != nil {
+		return "", err
+	}
+	h, err := p.AuthHeaders(ctx, remote)
+	if err != nil {
+		return "", err
+	}
+	req.Header = h
+	if github {
+		req.Header.Set("Accept", "application/vnd.github+json")
+	}
+	resp, err := p.client.Do(req)
+	if err != nil {
+		return "", err
+	}
+	defer func() { _ = resp.Body.Close() }()
+	if resp.StatusCode == http.StatusNotFound || resp.StatusCode == http.StatusUnprocessableEntity {
+		return "", errNotFound
+	}
+	if resp.StatusCode != http.StatusOK {
+		return "", provider.NewUpstreamStatusError(api, resp)
+	}
+	var b struct {
+		Commit struct {
+			ID string `json:"id"`
+		} `json:"commit"`
+		Object struct {
+			SHA string `json:"sha"`
+		} `json:"object"`
+	}
+	if err := json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(&b); err != nil {
+		return "", err
+	}
+	commit := b.Commit.ID
+	if github {
+		commit = b.Object.SHA
+	}
+	if !commitRe.MatchString(commit) {
+		return "", fmt.Errorf("branch lookup %s: unexpected commit %q", api, commit)
+	}
+	return commit, nil
+}

internal/provider/vcs/vcs_test.go

@@ -0,0 +1,378 @@
+package vcs
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+
+	"git.unkin.net/unkin/artifactapi/internal/githubauth"
+	"git.unkin.net/unkin/artifactapi/internal/provider"
+	"git.unkin.net/unkin/artifactapi/pkg/models"
+)
+
+const (
+	shaA = "4f2cfe92e3499f8581bf39db77e0a1431fa7b55c"
+	shaB = "7fa5434dbc083c853eef435d1915020345219d2f"
+)
+
+func TestParsePath(t *testing.T) {
+	tests := []struct {
+		path string
+		ok   bool
+		kind kind
+		name string
+	}{
+		{"cli/cli/archive/refs/tags/v2.60.0.tar.gz", true, tagArchive, "v2.60.0"},
+		{"/cli/cli/archive/refs/tags/v2.60.0.zip", true, tagArchive, "v2.60.0"},
+		{"o/r/archive/refs/tags/release/1.0.tar.gz", true, tagArchive, "release/1.0"},
+		{"o/r/archive/refs/heads/feature/x.tar.gz", true, branchArchive, "feature/x"},
+		{"o/r/archive/" + shaA + ".tar.gz", true, commitArchive, shaA},
+		{"o/r/releases/download/v1.0/tool_linux_amd64.tar.gz", true, releaseAsset, "tool_linux_amd64.tar.gz"},
+		{"o/r/releases/latest/download/tool.zip", true, latestReleaseAsset, "tool.zip"},
+		{"o/r/archive/main.tar.gz", false, 0, ""},
+		{"o/r/archive/" + shaA[:7] + ".tar.gz", false, 0, ""},
+		{"o/r/archive/refs/tags/v1.tar.bz2", false, 0, ""},
+		{"o/r/archive/refs/tags/.tar.gz", false, 0, ""},
+		{"../r/archive/refs/tags/v1.tar.gz", false, 0, ""},
+		{"o/../archive/refs/tags/v1.tar.gz", false, 0, ""},
+		{"o/r/archive/refs/tags/../../../etc/passwd.tar.gz", false, 0, ""},
+		{"o/r/archive/refs/heads/a//b.tar.gz", false, 0, ""},
+		{"o/r/archive/refs/heads/.hidden.tar.gz", false, 0, ""},
+		{"o/r/archive/refs/heads/a..b.tar.gz", false, 0, ""},
+		{"o/r/archive/refs/tags/v1%2f.tar.gz", false, 0, ""},
+		{"o/r/releases/download/v1/../x", false, 0, ""},
+		{"o/r/releases/download/asset", false, 0, ""},
+		{"o/r/raw/main/README.md", false, 0, ""},
+		{"o/r", false, 0, ""},
+	}
+	for _, tt := range tests {
+		ref, ok := parsePath(tt.path)
+		if ok != tt.ok || (ok && (ref.Kind != tt.kind || ref.Name != tt.name)) {
+			t.Errorf("parsePath(%q) = %+v, %v; want kind %v name %q ok %v", tt.path, ref, ok, tt.kind, tt.name, tt.ok)
+		}
+	}
+}
+
+func TestClassify(t *testing.T) {
+	p := New()
+	for path, want := range map[string]provider.Mutability{
+		"o/r/archive/refs/tags/v1.tar.gz":        provider.Immutable,
+		"o/r/archive/" + shaA + ".zip":           provider.Immutable,
+		"o/r/releases/download/v1/a.bin":         provider.Immutable,
+		"o/r/archive/refs/heads/main.tar.gz":     provider.Mutable,
+		"o/r/releases/latest/download/webui.zip": provider.Mutable,
+	} {
+		if got := p.Classify(path); got != want {
+			t.Errorf("Classify(%q) = %v, want %v", path, got, want)
+		}
+	}
+}
+
+type upstream struct {
+	srv    *httptest.Server
+	hits   atomic.Int32
+	commit atomic.Value
+	status atomic.Int32
+	header atomic.Value
+	accept atomic.Value
+	auth   atomic.Value
+	path   atomic.Value
+	gate   chan struct{}
+}
+
+func newUpstream(t *testing.T, gitea bool) *upstream {
+	u := &upstream{}
+	u.commit.Store(shaA)
+	u.status.Store(http.StatusOK)
+	u.header.Store(http.Header{})
+	u.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		u.hits.Add(1)
+		if u.gate != nil {
+			<-u.gate
+		}
+		u.accept.Store(r.Header.Get("Accept"))
+		u.auth.Store(r.Header.Get("Authorization"))
+		u.path.Store(r.URL.EscapedPath())
+		for k, v := range u.header.Load().(http.Header) {
+			w.Header()[k] = v
+		}
+		if s := int(u.status.Load()); s != http.StatusOK {
+			w.WriteHeader(s)
+			return
+		}
+		if gitea {
+			_, _ = w.Write([]byte(`{"name":"main","commit":{"id":"` + u.commit.Load().(string) + `"}}`))
+			return
+		}
+		_, _ = w.Write([]byte(`{"ref":"refs/heads/main","object":{"sha":"` + u.commit.Load().(string) + `","type":"commit"}}`))
+	}))
+	t.Cleanup(u.srv.Close)
+	return u
+}
+
+func serve(p *Provider, remote models.Remote, path string) (*httptest.ResponseRecorder, bool) {
+	w := httptest.NewRecorder()
+	r := httptest.NewRequest(http.MethodGet, "/api/v1/remote/"+remote.Name+"/"+path, nil)
+	return w, p.ServeRemote(w, r, remote, path, "http://proxy", nil)
+}
+
+// age backdates every cached lookup of a remote by d.
+func age(p *Provider, remote string, d time.Duration) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	for k, e := range p.refs[remote] {
+		e.at = e.at.Add(-d)
+		p.refs[remote][k] = e
+	}
+}
+
+type staticCred string
+
+func (s staticCred) Token(context.Context) (string, error) { return string(s), nil }
+
+func TestServeRemoteGiteaBranchRedirectsToCommit(t *testing.T) {
+	u := newUpstream(t, true)
+	p := New()
+	remote := models.Remote{Name: "gitea", BaseURL: u.srv.URL, MutableTTL: 3600, Username: "bot", Password: "pw"}
+
+	w, handled := serve(p, remote, "unkin/app/archive/refs/heads/feature/x.zip")
+	if !handled || w.Code != http.StatusFound {
+		t.Fatalf("handled=%v code=%d", handled, w.Code)
+	}
+	if got, want := w.Header().Get("Location"), "http://proxy/api/v1/remote/gitea/unkin/app/archive/"+shaA+".zip"; got != want {
+		t.Errorf("Location = %q, want %q", got, want)
+	}
+	if got := u.path.Load(); got != "/api/v1/repos/unkin/app/branches/feature/x" {
+		t.Errorf("api path = %q", got)
+	}
+	if got := u.auth.Load(); got != "Basic Ym90OnB3" {
+		t.Errorf("Authorization = %q, want remote basic auth", got)
+	}
+}
+
+func TestServeRemoteGitHubBranchUsesAPI(t *testing.T) {
+	githubauth.SetServer(staticCred("ghs_server"))
+	t.Cleanup(func() { githubauth.SetServer(nil) })
+	u := newUpstream(t, false)
+	p := New()
+	p.githubAPI = u.srv.URL
+
+	for _, base := range []string{"https://github.com", "https://www.github.com/", "https://api.github.com"} {
+		remote := models.Remote{Name: base, BaseURL: base, MutableTTL: 3600}
+		w, _ := serve(p, remote, "cli/cli/archive/refs/heads/feature/x.tar.gz")
+		if !strings.HasSuffix(w.Header().Get("Location"), "/cli/cli/archive/"+shaA+".tar.gz") {
+			t.Fatalf("%s: Location = %q", base, w.Header().Get("Location"))
+		}
+		if u.path.Load() != "/repos/cli/cli/git/ref/heads/feature/x" || u.accept.Load() != "application/vnd.github+json" {
+			t.Errorf("%s: api path %q accept %q", base, u.path.Load(), u.accept.Load())
+		}
+		if got := u.auth.Load(); got != "Bearer ghs_server" {
+			t.Errorf("%s: Authorization = %q, want server credential", base, got)
+		}
+	}
+}
+
+func TestServeRemoteBranchResolutionHonoursMutableTTL(t *testing.T) {
+	u := newUpstream(t, true)
+	p := New()
+	remote := models.Remote{Name: "g", BaseURL: u.srv.URL, MutableTTL: 600}
+	path := "o/r/archive/refs/heads/main.tar.gz"
+
+	serve(p, remote, path)
+	u.commit.Store(shaB)
+	age(p, remote.Name, 599*time.Second)
+	w, _ := serve(p, remote, path)
+	if u.hits.Load() != 1 || !strings.Contains(w.Header().Get("Location"), shaA) {
+		t.Fatalf("within ttl: hits=%d location=%q", u.hits.Load(), w.Header().Get("Location"))
+	}
+
+	age(p, remote.Name, 2*time.Second)
+	w, _ = serve(p, remote, path)
+	if u.hits.Load() != 2 || !strings.Contains(w.Header().Get("Location"), shaB) {
+		t.Fatalf("past ttl: hits=%d location=%q", u.hits.Load(), w.Header().Get("Location"))
+	}
+}
+
+func TestServeRemoteBranchUpstreamFailure(t *testing.T) {
+	path := "o/r/archive/refs/heads/main.tar.gz"
+	for _, tc := range []struct {
+		name  string
+		stale bool
+		want  int
+	}{
+		{"stale", true, http.StatusFound},
+		{"strict", false, http.StatusBadGateway},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			u := newUpstream(t, true)
+			p := New()
+			remote := models.Remote{Name: tc.name, BaseURL: u.srv.URL, StaleOnError: tc.stale}
+			serve(p, remote, path)
+			u.status.Store(http.StatusInternalServerError)
+			if w, _ := serve(p, remote, path); w.Code != tc.want {
+				t.Fatalf("code = %d, want %d", w.Code, tc.want)
+			}
+		})
+	}
+}
+
+func TestServeRemoteBranchRateLimited(t *testing.T) {
+	reset := strconv.FormatInt(time.Now().Add(120*time.Second).Unix(), 10)
+	for _, tc := range []struct {
+		name   string
+		status int
+		header http.Header
+		want   int
+		retry  [2]int
+	}{
+		{"primary 403", http.StatusForbidden, http.Header{"X-Ratelimit-Remaining": {"0"}, "X-Ratelimit-Reset": {reset}}, http.StatusServiceUnavailable, [2]int{100, 122}},
+		{"secondary 403", http.StatusForbidden, http.Header{"Retry-After": {"30"}}, http.StatusServiceUnavailable, [2]int{30, 31}},
+		{"429", http.StatusTooManyRequests, http.Header{"Retry-After": {"5"}}, http.StatusServiceUnavailable, [2]int{5, 6}},
+		{"429 no hint", http.StatusTooManyRequests, http.Header{}, http.StatusServiceUnavailable, [2]int{60, 60}},
+		{"plain 403", http.StatusForbidden, http.Header{}, http.StatusBadGateway, [2]int{}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			u := newUpstream(t, false)
+			u.status.Store(int32(tc.status))
+			u.header.Store(tc.header)
+			p := New()
+			p.githubAPI = u.srv.URL
+			w, _ := serve(p, models.Remote{Name: "gh", BaseURL: "https://github.com"}, "o/r/archive/refs/heads/main.tar.gz")
+			if w.Code != tc.want {
+				t.Fatalf("code = %d, want %d", w.Code, tc.want)
+			}
+			ra := w.Header().Get("Retry-After")
+			if tc.want != http.StatusServiceUnavailable {
+				if ra != "" {
+					t.Fatalf("Retry-After = %q on %d", ra, w.Code)
+				}
+				return
+			}
+			if n, err := strconv.Atoi(ra); err != nil || n < tc.retry[0] || n > tc.retry[1] {
+				t.Fatalf("Retry-After = %q, want %d..%d", ra, tc.retry[0], tc.retry[1])
+			}
+		})
+	}
+}
+
+func TestServeRemoteBranchNotFoundIsCached(t *testing.T) {
+	u := newUpstream(t, false)
+	u.status.Store(http.StatusNotFound)
+	p := New()
+	p.githubAPI = u.srv.URL
+	remote := models.Remote{Name: "gh", BaseURL: "https://github.com"}
+	path := "o/r/archive/refs/heads/nope.tar.gz"
+
+	for range 3 {
+		if w, _ := serve(p, remote, path); w.Code != http.StatusNotFound {
+			t.Fatalf("code = %d, want 404", w.Code)
+		}
+	}
+	if u.hits.Load() != 1 {
+		t.Fatalf("hits = %d, want 1 within negative ttl", u.hits.Load())
+	}
+	u.status.Store(http.StatusOK)
+	age(p, remote.Name, negativeTTL)
+	if w, _ := serve(p, remote, path); w.Code != http.StatusFound || u.hits.Load() != 2 {
+		t.Fatalf("after negative ttl: code=%d hits=%d", w.Code, u.hits.Load())
+	}
+}
+
+func TestServeRemoteConcurrentLookupsShareOneCall(t *testing.T) {
+	u := newUpstream(t, true)
+	u.gate = make(chan struct{})
+	p := New()
+	remote := models.Remote{Name: "g", BaseURL: u.srv.URL, MutableTTL: 600}
+
+	var wg sync.WaitGroup
+	codes := make([]int, 20)
+	for i := range codes {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			w, _ := serve(p, remote, "o/r/archive/refs/heads/main.tar.gz")
+			codes[i] = w.Code
+		}()
+	}
+	for u.hits.Load() == 0 {
+		time.Sleep(time.Millisecond)
+	}
+	time.Sleep(50 * time.Millisecond)
+	close(u.gate)
+	wg.Wait()
+	if u.hits.Load() != 1 {
+		t.Fatalf("hits = %d, want 1", u.hits.Load())
+	}
+	for i, c := range codes {
+		if c != http.StatusFound {
+			t.Fatalf("request %d: code %d", i, c)
+		}
+	}
+}
+
+func TestStoreBoundsEntriesPerRemote(t *testing.T) {
+	p := New()
+	for i := range maxRefs + 10 {
+		p.store("a", fmt.Sprint(i), "")
+	}
+	p.store("b", "x", shaA)
+	if n := len(p.refs["a"]); n != maxRefs {
+		t.Fatalf("remote a holds %d entries, want %d", n, maxRefs)
+	}
+	if len(p.refs["b"]) != 1 {
+		t.Fatalf("remote b must keep its own budget")
+	}
+	if _, ok := p.refs["a"][fmt.Sprint(maxRefs+9)]; !ok {
+		t.Fatalf("newest entry evicted")
+	}
+}
+
+func TestServeRemoteRejectsBadCommit(t *testing.T) {
+	u := newUpstream(t, true)
+	u.commit.Store("../../etc")
+	w, _ := serve(New(), models.Remote{Name: "g", BaseURL: u.srv.URL}, "o/r/archive/refs/heads/main.tar.gz")
+	if w.Code != http.StatusBadGateway {
+		t.Fatalf("code = %d, want 502", w.Code)
+	}
+}
+
+func TestServeRemoteAllowlist(t *testing.T) {
+	u := newUpstream(t, true)
+	remote := models.Remote{Name: "g", BaseURL: u.srv.URL, Patterns: []string{"^allowed/repo/"}}
+	if w, _ := serve(New(), remote, "other/repo/archive/refs/heads/main.tar.gz"); w.Code != http.StatusForbidden {
+		t.Fatalf("code = %d, want 403", w.Code)
+	}
+	if u.hits.Load() != 0 {
+		t.Fatalf("denied branch must not hit the API")
+	}
+}
+
+func TestServeRemoteBlockedRedirectTarget(t *testing.T) {
+	u := newUpstream(t, true)
+	remote := models.Remote{Name: "g", BaseURL: u.srv.URL, Blocklist: []string{"/archive/" + shaA}}
+	w, _ := serve(New(), remote, "o/r/archive/refs/heads/main.tar.gz")
+	if w.Code != http.StatusForbidden || w.Header().Get("Location") != "" {
+		t.Fatalf("code = %d location %q, want 403 without redirect", w.Code, w.Header().Get("Location"))
+	}
+}
+
+func TestServeRemotePassesThroughAndRejects(t *testing.T) {
+	p := New()
+	remote := models.Remote{Name: "g", BaseURL: "https://github.com"}
+	if _, handled := serve(p, remote, "o/r/archive/refs/tags/v1.tar.gz"); handled {
+		t.Error("tag archives must go through the proxy engine")
+	}
+	if _, handled := serve(p, remote, "o/r/releases/download/v1/a.bin"); handled {
+		t.Error("release assets must go through the proxy engine")
+	}
+	if w, handled := serve(p, remote, "o/r/raw/main/x"); !handled || w.Code != http.StatusNotFound {
+		t.Errorf("unsupported path: handled=%v code=%d", handled, w.Code)
+	}
+}

internal/virtual/alpine_merger.go

@@ -0,0 +1,90 @@
+package virtual
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"errors"
+	"fmt"
+	"io"
+	"strings"
+	"time"
+)
+
+// AlpineMember is one member's decompressed APKINDEX for a directory; Index is
+// nil when the member does not serve that directory.
+type AlpineMember struct {
+	RemoteName string
+	Index      []byte
+}
+
+// AlpineRepo is a merged directory: its APKINDEX.tar.gz plus, per
+// "<name>-<version>.apk" apk will request, the member that owns it.
+type AlpineRepo struct {
+	Index  []byte
+	Owners map[string][]byte
+}
+
+func (r *AlpineRepo) index() []byte            { return r.Index }
+func (r *AlpineRepo) files() map[string][]byte { return r.Owners }
+
+// MergeAlpine merges members' APKINDEX records. Members are in priority order:
+// the first to carry a (P, V, A) wins. The index is unsigned, like the local
+// and github_alpine indexes it merges.
+func MergeAlpine(members []AlpineMember) *AlpineRepo {
+	repo := &AlpineRepo{Owners: map[string][]byte{}}
+	seen := map[string]bool{}
+	var b bytes.Buffer
+	for _, m := range members {
+		for _, rec := range splitStanzas(m.Index) {
+			p, v, a := stanzaField(rec, "P"), stanzaField(rec, "V"), stanzaField(rec, "A")
+			key := p + "\x00" + v + "\x00" + a
+			if p == "" || v == "" || seen[key] {
+				continue
+			}
+			seen[key] = true
+			if _, ok := repo.Owners[p+"-"+v+".apk"]; !ok {
+				repo.Owners[p+"-"+v+".apk"] = []byte(m.RemoteName)
+			}
+			b.WriteString(rec)
+			b.WriteString("\n\n")
+		}
+	}
+	var t bytes.Buffer
+	tw := tar.NewWriter(&t)
+	_ = tw.WriteHeader(&tar.Header{Name: "APKINDEX", Mode: 0o644, Size: int64(b.Len()), Typeflag: tar.TypeReg, ModTime: time.Unix(0, 0)})
+	_, _ = tw.Write(b.Bytes())
+	_ = tw.Close()
+	repo.Index = gzipDeterministic(t.Bytes())
+	return repo
+}
+
+// apkIndexText extracts the APKINDEX entry from an APKINDEX.tar.gz, skipping a
+// leading signature stream if present.
+func apkIndexText(body []byte) ([]byte, error) {
+	zr, err := gzip.NewReader(bytes.NewReader(body))
+	if err != nil {
+		return nil, err
+	}
+	tr := tar.NewReader(zr)
+	for {
+		hdr, err := tr.Next()
+		if errors.Is(err, io.EOF) {
+			return nil, fmt.Errorf("no APKINDEX entry")
+		}
+		if err != nil {
+			return nil, err
+		}
+		if hdr.Name == "APKINDEX" {
+			return io.ReadAll(tr)
+		}
+	}
+}
+
+// alpinePath splits a request into its directory and file, or ok is false for
+// anything but an APKINDEX.tar.gz or .apk.
+func alpinePath(p string) (dir, file string, ok bool) {
+	i := strings.LastIndex(p, "/")
+	dir, file = p[:i+1], p[i+1:]
+	return dir, file, dir != "" && (file == "APKINDEX.tar.gz" || strings.HasSuffix(file, ".apk"))
+}

internal/virtual/alpine_merger_test.go

@@ -0,0 +1,140 @@
+package virtual
+
+import (
+	"archive/tar"
+	"bytes"
+	"compress/gzip"
+	"context"
+	"errors"
+	"strings"
+	"testing"
+
+	"git.unkin.net/unkin/artifactapi/pkg/models"
+)
+
+func apkRec(p, v, a string) string {
+	return "C:Q1" + p + v + "\nP:" + p + "\nV:" + v + "\nA:" + a + "\nS:10\n"
+}
+
+func apkTarGz(t *testing.T, name string, body []byte, closeTar bool) []byte {
+	t.Helper()
+	var tb bytes.Buffer
+	tw := tar.NewWriter(&tb)
+	if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(body)), Typeflag: tar.TypeReg}); err != nil {
+		t.Fatal(err)
+	}
+	tw.Write(body)
+	if closeTar {
+		tw.Close()
+	} else {
+		tw.Flush()
+	}
+	var gb bytes.Buffer
+	gz := gzip.NewWriter(&gb)
+	gz.Write(tb.Bytes())
+	gz.Close()
+	return gb.Bytes()
+}
+
+func TestMergeAlpine(t *testing.T) {
+	repo := MergeAlpine([]AlpineMember{
+		{RemoteName: "a", Index: []byte(apkRec("foo", "1.0-r0", "x86_64") + "\n" + apkRec("bar", "1-r0", "noarch") + "\nC:Q1x\nV:9\n")},
+		{RemoteName: "b"},
+		{RemoteName: "c", Index: []byte(apkRec("foo", "1.0-r0", "x86_64") + "\n" + apkRec("foo", "2.0-r0", "x86_64") + "\n" + apkRec("bar", "1-r0", "x86_64"))},
+	})
+	idx, err := apkIndexText(repo.Index)
+	if err != nil {
+		t.Fatal(err)
+	}
+	recs := splitStanzas(idx)
+	if len(recs) != 4 || strings.Count(string(idx), "P:foo\nV:1.0-r0\n") != 1 {
+		t.Fatalf("want foo 1.0 once, foo 2.0 and bar for both arches, record without P dropped:\n%s", idx)
+	}
+	for file, want := range map[string]string{"foo-1.0-r0.apk": "a", "foo-2.0-r0.apk": "c", "bar-1-r0.apk": "a"} {
+		if got := string(repo.Owners[file]); got != want {
+			t.Errorf("%s owned by %q, want %q", file, got, want)
+		}
+	}
+	if again := MergeAlpine([]AlpineMember{{RemoteName: "a", Index: idx}}); !bytes.Equal(again.Index, MergeAlpine([]AlpineMember{{RemoteName: "a", Index: idx}}).Index) {
+		t.Fatal("merged index must be byte-stable across merges")
+	}
+}
+
+func TestApkIndexTextSkipsSignature(t *testing.T) {
+	signed := append(apkTarGz(t, ".SIGN.RSA.alpine.rsa.pub", []byte("sig"), false), apkTarGz(t, "APKINDEX", []byte(apkRec("foo", "1", "x86_64")), true)...)
+	idx, err := apkIndexText(signed)
+	if err != nil || !strings.Contains(string(idx), "P:foo\n") {
+		t.Fatalf("signed index: %v\n%s", err, idx)
+	}
+	if _, err := apkIndexText(apkTarGz(t, "DESCRIPTION", []byte("x"), true)); err == nil {
+		t.Fatal("index without APKINDEX entry should fail")
+	}
+}
+
+func alpineEngine(members map[string]string) *Engine {
+	e := fakeEngine(nil)
+	e.apkMember = func(_ context.Context, name, dir string) (*AlpineMember, error) {
+		idx, ok := members[name]
+		if !ok {
+			return nil, errors.New("upstream down")
+		}
+		if dir != "x86_64/" {
+			return &AlpineMember{RemoteName: name}, nil
+		}
+		return &AlpineMember{RemoteName: name, Index: []byte(idx)}, nil
+	}
+	return e
+}
+
+func alpineVirt(members ...string) models.Virtual {
+	return models.Virtual{Name: "v", PackageType: models.PackageAlpine, Members: members}
+}
+
+func TestFetchAlpineIndexAndRedirect(t *testing.T) {
+	e := alpineEngine(map[string]string{"loc": apkRec("foo", "1-r0", "x86_64"), "gh": apkRec("foo", "1-r0", "x86_64") + "\n" + apkRec("tool", "2-r0", "x86_64")})
+	virt := alpineVirt("loc", "gh")
+
+	body, ct, err := e.Fetch(context.Background(), virt, "./x86_64/APKINDEX.tar.gz", "")
+	if err != nil || ct != "application/gzip" {
+		t.Fatalf("fetch index: %v %q", err, ct)
+	}
+	if idx, _ := apkIndexText(body); strings.Count(string(idx), "P:") != 2 {
+		t.Fatalf("merged index:\n%s", idx)
+	}
+	for _, tc := range []struct {
+		path, want string
+		ok         bool
+	}{
+		{path: "x86_64/foo-1-r0.apk", ok: true, want: "http://h/api/v1/remote/loc/x86_64/foo-1-r0.apk"},
+		{path: "./x86_64/tool-2-r0.apk", ok: true, want: "http://h/api/v1/remote/gh/x86_64/tool-2-r0.apk"},
+		{path: "x86_64/../x86_64/tool-2-r0.apk", ok: true, want: "http://h/api/v1/remote/gh/x86_64/tool-2-r0.apk"},
+		{path: "x86_64/missing-1-r0.apk"},
+		{path: "aarch64/foo-1-r0.apk"},
+		{path: "x86_64/APKINDEX.tar.gz"},
+		{path: "foo-1-r0.apk"},
+	} {
+		got, ok, err := e.MemberRedirect(context.Background(), virt, tc.path, "", "http://h/")
+		if got != tc.want || ok != tc.ok || err != nil {
+			t.Errorf("%s: got (%q, %v, %v), want (%q, %v)", tc.path, got, ok, err, tc.want, tc.ok)
+		}
+	}
+	for _, p := range []string{"x86_64/missing-1-r0.apk", "x86_64/DESCRIPTION", "APKINDEX.tar.gz"} {
+		if _, _, err := e.Fetch(context.Background(), virt, p, ""); !errors.Is(err, ErrNotFound) {
+			t.Errorf("%s: want not found, got %v", p, err)
+		}
+	}
+}
+
+func TestFetchAlpineFailsClosed(t *testing.T) {
+	e := alpineEngine(map[string]string{"a": apkRec("foo", "1-r0", "x86_64")})
+	virt := alpineVirt("a", "down")
+	if _, _, err := e.Fetch(context.Background(), virt, "x86_64/APKINDEX.tar.gz", ""); err == nil || errors.Is(err, ErrNotFound) {
+		t.Fatalf("index with a member down: want upstream error, got %v", err)
+	}
+	if _, ok, err := e.MemberRedirect(context.Background(), virt, "x86_64/foo-1-r0.apk", "", "http://h/"); ok || err != nil {
+		t.Fatalf("redirect with a member down must defer to Fetch: ok %v err %v", ok, err)
+	}
+	if _, _, err := e.Fetch(context.Background(), virt, "x86_64/foo-1-r0.apk", ""); err == nil || errors.Is(err, ErrNotFound) {
+		t.Fatalf("download with a member down: want upstream error, got %v", err)
+	}
+}

internal/virtual/engine.go

@@ -28,6 +28,7 @@ type Engine struct {
 	getRemote   func(context.Context, string) (*models.Remote, error)
 	rpmMember   func(context.Context, string) (*RPMMember, error)
 	debMember   func(ctx context.Context, name, base string) (*DebMember, error)
+	apkMember   func(ctx context.Context, name, dir string) (*AlpineMember, error)
 	npmMember   func(ctx context.Context, name, pkg string) (*NPMMember, error)
 
 	// NPMLocal returns a local npm repo's packument with tarball paths relative
@@ -76,13 +77,15 @@ type repoGen struct {
 
 const rpmMergeTTL = 60 * time.Second
 
-// maxDebSuites bounds the client-chosen suites cached per deb virtual.
+// maxDebSuites bounds the client-chosen deb suites or alpine directories
+// cached per virtual.
 const maxDebSuites = 32
 
 func NewEngine(db *database.DB, proxyEngine *proxy.Engine) *Engine {
 	e := &Engine{db: db, proxyEngine: proxyEngine, getRemote: db.GetRemote, mergeTTL: rpmMergeTTL}
 	e.rpmMember = e.fetchRPMMember
 	e.debMember = e.fetchDebMember
+	e.apkMember = e.fetchAlpineMember
 	e.npmMember = e.fetchNPMMember
 	e.gemsMember = e.fetchGemsMember
 	e.ansibleDoc = e.ansibleMemberDoc
@@ -97,6 +100,8 @@ func (e *Engine) Fetch(ctx context.Context, virt models.Virtual, path string, pr
 		return e.fetchRPM(ctx, virt, path)
 	case models.PackageDeb:
 		return e.fetchDeb(ctx, virt, path)
+	case models.PackageAlpine:
+		return e.fetchAlpine(ctx, virt, path)
 	}
 
 	merger, err := GetMerger(virt.PackageType)
@@ -216,8 +221,8 @@ var (
 )
 
 // MemberRedirect maps an rpm, deb, npm or ansible virtual package path
-// (<member>/<href>, as written by MergeRPM/MergeDeb/MergeNPM/AnsibleVersion)
-// to the owning member's route. ok is false for any other path; ErrBadPath
+// (<member>/<href>, as written by MergeRPM/MergeDeb/MergeNPM/AnsibleVersion),
+// or an alpine <dir>/<name>-<version>.apk, to the owning member's route. ok is false for any other path; ErrBadPath
 // means the path is absolute or escapes the member.
 func (e *Engine) MemberRedirect(ctx context.Context, virt models.Virtual, path, rawQuery, proxyBaseURL string) (string, bool, error) {
 	switch virt.PackageType {
@@ -231,6 +236,8 @@ func (e *Engine) MemberRedirect(ctx context.Context, virt models.Virtual, path,
 		if strings.HasPrefix(path, "dists/") {
 			return "", false, nil
 		}
+	case models.PackageAlpine:
+		return e.alpineRedirect(ctx, virt, path, rawQuery, proxyBaseURL)
 	case models.PackageNPM, models.PackageAnsible:
 	default:
 		return "", false, nil
@@ -445,6 +452,78 @@ func (e *Engine) fetchDebMember(ctx context.Context, name, base string) (*DebMem
 	return m, nil
 }
 
+func (e *Engine) mergedAlpine(ctx context.Context, virt models.Virtual, dir string) (mergedRepo, error) {
+	repo, err := e.merged(ctx, "alpine\x00"+virt.Name+"\x00"+dir, func(ctx context.Context) (mergedRepo, error) {
+		members, err := fetchMembers(virt, func(name string) (*AlpineMember, error) { return e.apkMember(ctx, name, dir) })
+		if err != nil {
+			return nil, err
+		}
+		return MergeAlpine(members), nil
+	})
+	if err == nil {
+		e.evictLRU("alpine\x00"+virt.Name+"\x00", maxDebSuites)
+	}
+	return repo, err
+}
+
+func (e *Engine) fetchAlpine(ctx context.Context, virt models.Virtual, reqPath string) ([]byte, string, error) {
+	dir, file, ok := alpinePath(strings.TrimPrefix(path.Clean("/"+reqPath), "/"))
+	if !ok {
+		return nil, "", ErrNotFound
+	}
+	repo, err := e.mergedAlpine(ctx, virt, dir)
+	if err != nil {
+		return nil, "", err
+	}
+	if file != "APKINDEX.tar.gz" {
+		return nil, "", ErrNotFound
+	}
+	return repo.index(), "application/gzip", nil
+}
+
+// alpineRedirect finds the member whose index lists the requested .apk; apk
+// builds the download path itself, so the index cannot carry a member prefix.
+// A failed merge returns ok=false so Fetch re-merges and fails with 502.
+func (e *Engine) alpineRedirect(ctx context.Context, virt models.Virtual, reqPath, rawQuery, proxyBaseURL string) (string, bool, error) {
+	dir, file, ok := alpinePath(strings.TrimPrefix(path.Clean("/"+reqPath), "/"))
+	if !ok || file == "APKINDEX.tar.gz" {
+		return "", false, nil
+	}
+	owner, ok := e.cachedFile("alpine\x00"+virt.Name+"\x00"+dir, file)
+	if !ok {
+		repo, err := e.mergedAlpine(ctx, virt, dir)
+		if err != nil {
+			return "", false, nil
+		}
+		if owner, ok = repo.files()[file]; !ok {
+			return "", false, nil
+		}
+	}
+	return e.memberLocation(ctx, string(owner), dir+file, rawQuery, proxyBaseURL)
+}
+
+// fetchAlpineMember reads a member's APKINDEX under dir; a member without that
+// directory contributes nothing.
+func (e *Engine) fetchAlpineMember(ctx context.Context, name, dir string) (*AlpineMember, error) {
+	remote, err := e.getRemote(ctx, name)
+	if err != nil {
+		return nil, fmt.Errorf("remote %q: %w", name, err)
+	}
+	raw, err := e.fetchMemberPath(ctx, *remote, dir+"APKINDEX.tar.gz")
+	var pe *proxy.ProxyError
+	if errors.Is(err, ErrNotFound) || errors.As(err, &pe) && pe.Status == http.StatusNotFound {
+		return &AlpineMember{RemoteName: name}, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	idx, err := apkIndexText(raw)
+	if err != nil {
+		return nil, fmt.Errorf("%s/%sAPKINDEX.tar.gz: %w", name, dir, err)
+	}
+	return &AlpineMember{RemoteName: name, Index: idx}, nil
+}
+
 // evictLRU drops the least recently used merges under prefix beyond keep.
 func (e *Engine) evictLRU(prefix string, keep int) {
 	e.mu.Lock()

pkg/models/package_type_vcs.go

@@ -0,0 +1,5 @@
+package models
+
+const PackageVCS PackageType = "vcs"
+
+func init() { registerPackageType(PackageVCS) }

ui/src/packagetypes/defs/vcs.ts

@@ -0,0 +1,26 @@
+import type { PackageTypeUI } from '../types';
+
+const def: PackageTypeUI = {
+  color: 'default',
+  snippets: ({ proxy }) => {
+    return [
+      {
+        title: 'Download a tag archive',
+        language: 'bash',
+        code: `curl -fsSLO ${proxy}/<owner>/<repo>/archive/refs/tags/<tag>.tar.gz`,
+      },
+      {
+        title: 'Download a branch archive (redirects to its current commit)',
+        language: 'bash',
+        code: `curl -fsSL -o src.tar.gz ${proxy}/<owner>/<repo>/archive/refs/heads/<branch>.tar.gz`,
+      },
+      {
+        title: 'Download a release asset',
+        language: 'bash',
+        code: `curl -fsSLO ${proxy}/<owner>/<repo>/releases/download/<tag>/<asset>`,
+      },
+    ];
+  },
+};
+
+export default def;