Merge master
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful

This commit is contained in:
2026-10-25 15:25:14 +11:00
25 changed files with 837 additions and 37 deletions
+3 -1
View File
@@ -23,6 +23,7 @@ API: `http://localhost:8000` | Frontend: `http://localhost:5173`
Providers classify paths automatically. Users only configure what to proxy and TTLs.
Classification precedence, TTLs and eviction for every remote: [docs/common/caching.md](docs/common/caching.md).
Write authentication and per-client token usage: [docs/common/auth.md](docs/common/auth.md).
Each type's paths, Terraform examples and local/virtual behaviour are in `docs/<type>/`:
@@ -208,8 +209,9 @@ additive and idempotent for the same reason; a test enforces it.
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
| `ARTIFACTAPI_OIDC_ISSUER` | | OIDC issuer URL (e.g. Vault `https://vault.service.consul:8200/v1/identity/oidc`). When set, `POST /api/v2/remotes`, `POST /api/v2/virtuals` and `PUT`/`DELETE` on `/api/v2/{remotes,virtuals}/{name}` require `Authorization: Bearer <jwt>` verified against the issuer's JWKS. Uploads, object/file deletes, probe, reads and `/v2` stay open. Unset = no auth |
| `ARTIFACTAPI_OIDC_ISSUER` | | OIDC issuer URL (e.g. Vault `https://vault.example.com/v1/identity/oidc`). When set, `POST /api/v2/remotes`, `POST /api/v2/virtuals` and `PUT`/`DELETE` on `/api/v2/{remotes,virtuals}/{name}` require a token verified against the issuer's JWKS. Unset = no auth. See [docs/common/auth.md](docs/common/auth.md) |
| `ARTIFACTAPI_OIDC_AUDIENCE` | | Comma-separated allowed `aud` values (e.g. the Vault OIDC role's client_id). Required when `ARTIFACTAPI_OIDC_ISSUER` is set |
| `ARTIFACTAPI_AUTH_UPLOADS` | `false` | Also require a token for every local repo write: file uploads/deletes, object evictions, `/api/v1/local` publishes and `/v2` push. Requires `ARTIFACTAPI_OIDC_ISSUER` |
## Development
+1 -1
View File
@@ -2,7 +2,7 @@
A local `ansible` repo is a Galaxy v3 server: `ansible-galaxy collection publish`
and `ansible-galaxy collection install` work against `/api/v1/local/<repo>`.
`publish` needs an API key; any value works.
`publish` needs an API key; any value works unless [uploads are gated](../common/auth.md).
```bash
ansible-galaxy collection build
+53
View File
@@ -0,0 +1,53 @@
# Write authentication
artifactapi verifies writes against an OIDC issuer's JWKS, such as a Vault
identity OIDC role. Reads, probes and anonymous pulls never need a token.
| Variable | Gates |
|---|---|
| `ARTIFACTAPI_OIDC_ISSUER` + `ARTIFACTAPI_OIDC_AUDIENCE` | creating, updating and deleting remotes and virtuals |
| `ARTIFACTAPI_AUTH_UPLOADS=true` | also every local repo write: `PUT`/`DELETE /api/v2/remotes/<name>/files/*`, `DELETE /api/v2/{remotes,locals}/<name>/objects/*`, `PUT`/`POST`/`DELETE /api/v1/local/<repo>/*` (npm publish, gem push/yank, ansible-galaxy publish) and Docker Registry V2 push/delete under `/v2` |
Any token from the issuer whose `aud` is in `ARTIFACTAPI_OIDC_AUDIENCE` may
write to every repo.
## Minting a token
```sh
vault read -field=token identity/oidc/token/<role> > token
```
The role's `client_id` must be one of the configured audiences.
## Passing the token
The token is accepted as `Authorization: Bearer <token>`, as `Token <token>`,
as the password of HTTP Basic credentials with any username, or as a bare
header value.
```sh
# curl
curl -fsSL -H "Authorization: Bearer $(cat token)" --upload-file f.tgz \
https://artifactapi.example.com/api/v2/remotes/<repo>/files/f.tgz
curl -fsSL -u "ci:$(cat token)" --upload-file f.tgz \
https://artifactapi.example.com/api/v2/remotes/<repo>/files/f.tgz
# docker / podman / helm (OCI): any username, token as password
docker login artifactapi.example.com -u ci --password-stdin < token
helm registry login artifactapi.example.com -u ci --password-stdin < token
# npm (.npmrc)
//artifactapi.example.com/api/v1/local/<repo>/:_authToken=<token>
# gem
GEM_HOST_API_KEY="$(cat token)" gem push --host https://artifactapi.example.com/api/v1/local/<repo> mygem-1.0.0.gem
# ansible-galaxy
ansible-galaxy collection publish c.tar.gz \
--server https://artifactapi.example.com/api/v1/local/<repo>/ --api-key "$(cat token)"
```
With uploads gated, `/v2/` answers anonymous requests with a Bearer challenge
pointing at `/v2/token`. That endpoint returns the verified Basic password as
the registry token, or an anonymous token that only reads, so `docker pull`
keeps working without a login.
+19 -1
View File
@@ -16,7 +16,7 @@ Every proxied request on a remote is classified as **denied**, **immutable** or
| `check_mutable` | When a mutable path's `mutable_ttl` expires, sends a conditional `HEAD` (`If-None-Match`) with the stored upstream ETag. A `304` re-arms the TTL and serves the stored copy; anything else refetches in full. The ETag is kept for `max(10 × mutable_ttl, 7 days)`, re-armed on each `304`, so idle paths age out. It is only stored when `check_mutable` is set and `mutable_ttl` is non-zero, and never for immutable paths. |
| `stale_on_error` | On an upstream network error, serve the stored copy instead of failing. |
| `serve_stale` | rpm/deb/alpine virtual members only. When a virtual re-merges, use this member's previous index at once and refresh it in the background; the next merge picks up the refresh. The first merge still waits, and so does any merge once the previous index is over an hour old, so a member that keeps failing surfaces its error. Default `false`. |
| `quarantine_enabled` / `quarantine_days` | Stored on the remote but not enforced by the proxy yet. |
| `quarantine_enabled` / `quarantine_days` | Refuses immutable artifacts published less than `quarantine_days` (default `3`) ago. See [Quarantine](#quarantine). |
| `mirrorlist` / `mirror_strategy` | rpm/deb/alpine only. Upstream pool is `[base_url] + mirrorlist`; `round_robin` (default) rotates the starting mirror, `least_conn` starts at the one with the fewest in-flight requests. Network errors and 5xx fail over to the next mirror; other statuses return as-is. |
## Precedence
@@ -53,6 +53,24 @@ So `blocklist` beats the allowlist, and `immutable_patterns` beats
Concurrent misses on one path share a fetch lock. Waiters poll the store for
up to 5s instead of all going upstream.
## Quarantine
With `quarantine_enabled`, an uncached **immutable** path whose upstream copy
was published less than `quarantine_days` ago answers `403` and is not cached:
```json
{"error": "quarantined: published 2026-10-24T09:00:00Z, available from 2026-10-27T09:00:00Z",
"published": "2026-10-24T09:00:00Z", "available": "2026-10-27T09:00:00Z"}
```
- `quarantine_days` must be at least `1` when quarantine is enabled.
- The publish time is the upstream `Last-Modified` header (capped at now), else the first time artifactapi was asked for the path.
- If that first-seen lookup fails, the path answers `503` with `Retry-After` instead.
- Mutable paths (indexes, metadata) are never quarantined.
- Paths already cached keep serving, including ones cached before quarantine was enabled.
- A virtual redirects to the member that owns the path, so its `403` reaches the client. A goproxy virtual probing for an unlisted version skips a quarantining member and only redirects to it when no other member serves the version.
- Indexes still list fresh versions, so a client resolving "latest" may pick one and get the `403`.
## Eviction
```sh
+3
View File
@@ -10,6 +10,9 @@ docker push artifactapi.example.com/docker-internal/myapp:latest
docker pull artifactapi.example.com/docker-internal/myapp:latest
```
With [uploads gated](../common/auth.md), `docker login` with any username and
a token as the password before pushing; pulls stay anonymous.
The first path segment after `/v2/` is the artifactapi repo name; the remainder
is the image name. Blobs and manifests are stored through the shared
content-addressable store (deduplicated by digest, reaped by GC once
+1 -1
View File
@@ -2,7 +2,7 @@
A local `gems` repo is a RubyGems source: `gem push`, `gem yank`, `gem install`
and `bundle install` work against `/api/v1/local/<repo>`. `gem push` needs an
API key; any value works.
API key; any value works unless [uploads are gated](../common/auth.md).
```bash
GEM_HOST_API_KEY=unused gem push --host https://artifactapi.example.com/api/v1/local/<repo> mygem-1.0.0.gem
+2 -1
View File
@@ -2,7 +2,8 @@
A local `npm` repo is a real npm registry: `npm publish` (scoped and unscoped)
and `npm install` work against `/api/v1/local/<repo>/`. npm requires an
`_authToken` for the registry before it will publish; any value works.
`_authToken` for the registry before it will publish; any value works unless
[uploads are gated](../common/auth.md).
```ini
# .npmrc
+22 -2
View File
@@ -2,6 +2,7 @@ package v1
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
@@ -11,6 +12,7 @@ import (
"net/url"
"strconv"
"strings"
"time"
"github.com/go-chi/chi/v5"
@@ -121,7 +123,7 @@ func (h *ProxyHandler) serveFetch(w http.ResponseWriter, r *http.Request, remote
if err != nil {
var proxyErr *proxy.ProxyError
if errors.As(err, &proxyErr) {
http.Error(w, proxyErr.Message, proxyErr.Status)
writeProxyError(w, proxyErr)
return
}
slog.Error("proxy fetch failed", "remote", remote.Name, "path", path, "error", err)
@@ -149,6 +151,24 @@ func (h *ProxyHandler) serveFetch(w http.ResponseWriter, r *http.Request, remote
io.Copy(w, body)
}
// writeProxyError answers a quarantine refusal as JSON and anything else as text.
func writeProxyError(w http.ResponseWriter, pe *proxy.ProxyError) {
if pe.RetryAfter > 0 {
w.Header().Set("Retry-After", strconv.Itoa(pe.RetryAfter))
}
if pe.Published.IsZero() {
http.Error(w, pe.Message, pe.Status)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(pe.Status)
_ = json.NewEncoder(w).Encode(map[string]string{
"error": pe.Message,
"published": pe.Published.Format(time.RFC3339),
"available": pe.Available.Format(time.RFC3339),
})
}
func rewritesMutable(prov provider.Provider, path string) bool {
_, sr := prov.(provider.ServeRewriter)
_, rr := prov.(provider.ResponseRewriter)
@@ -176,7 +196,7 @@ func (h *ProxyHandler) handleProxyHead(w http.ResponseWriter, r *http.Request) {
if err != nil {
var proxyErr *proxy.ProxyError
if errors.As(err, &proxyErr) {
http.Error(w, proxyErr.Message, proxyErr.Status)
writeProxyError(w, proxyErr)
return
}
slog.Error("proxy head failed", "remote", remote.Name, "path", path, "error", err)
+16
View File
@@ -0,0 +1,16 @@
package v1
import (
"net/http/httptest"
"testing"
"git.unkin.net/unkin/artifactapi/internal/proxy"
)
func TestWriteProxyErrorRetryAfter(t *testing.T) {
w := httptest.NewRecorder()
writeProxyError(w, &proxy.ProxyError{Status: 503, Message: "quarantine check unavailable", RetryAfter: 30})
if w.Code != 503 || w.Header().Get("Retry-After") != "30" {
t.Errorf("code %d, Retry-After %q", w.Code, w.Header().Get("Retry-After"))
}
}
+10 -1
View File
@@ -72,7 +72,7 @@ func (h *RemotesHandler) get(w http.ResponseWriter, r *http.Request) {
const defaultMutableTTL = 3600
func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
remote := models.Remote{MutableTTL: defaultMutableTTL}
remote := models.Remote{MutableTTL: defaultMutableTTL, QuarantineDays: models.DefaultQuarantineDays}
if err := json.NewDecoder(r.Body).Decode(&remote); err != nil {
http.Error(w, "invalid json", http.StatusBadRequest)
return
@@ -104,6 +104,10 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := remote.ValidateQuarantine(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := h.db.CreateRemote(r.Context(), &remote); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
@@ -126,6 +130,7 @@ func (h *RemotesHandler) update(w http.ResponseWriter, r *http.Request) {
var remote models.Remote
if oldKnown {
remote.MutableTTL = existing.MutableTTL
remote.QuarantineDays = existing.QuarantineDays
}
if err := json.NewDecoder(r.Body).Decode(&remote); err != nil {
http.Error(w, "invalid json", http.StatusBadRequest)
@@ -144,6 +149,10 @@ func (h *RemotesHandler) update(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := remote.ValidateQuarantine(); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if err := h.db.UpdateRemote(r.Context(), &remote); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
+64
View File
@@ -55,3 +55,67 @@ func TestMutableTTLOmittedVsZero(t *testing.T) {
t.Errorf("explicit 0 stored as %d", got)
}
}
func TestQuarantineDaysDefault(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
h := NewRemotesHandler(db, nil, nil).Routes()
days := func() int {
t.Helper()
r, err := db.GetRemote(context.Background(), "q-days")
if err != nil {
t.Fatal(err)
}
return r.QuarantineDays
}
const body = `"package_type":"npm","base_url":"https://example.com","quarantine_enabled":true`
for i, s := range []struct {
method, path, json string
want int
}{
{"POST", "/", `{"name":"q-days",` + body + `}`, 3},
{"PUT", "/q-days", `{` + body + `,"quarantine_days":14}`, 14},
{"PUT", "/q-days", `{` + body + `}`, 14},
} {
if c := do(t, h, s.method, s.path, s.json); c/100 != 2 {
t.Fatalf("step %d = %d", i, c)
}
if got := days(); got != s.want {
t.Errorf("step %d: quarantine_days = %d, want %d", i, got, s.want)
}
}
}
func TestQuarantineDaysValidation(t *testing.T) {
if testDSN == "" {
t.Skip("Docker unavailable")
}
db, err := database.New(testDSN)
if err != nil {
t.Fatal(err)
}
defer db.Close()
h := NewRemotesHandler(db, nil, nil).Routes()
const body = `"package_type":"npm","base_url":"https://example.com"`
for i, s := range []struct {
method, path, json string
want int
}{
{"POST", "/", `{"name":"q-val",` + body + `,"quarantine_enabled":true,"quarantine_days":0}`, 400},
{"POST", "/", `{"name":"q-val",` + body + `,"quarantine_enabled":true,"quarantine_days":-1}`, 400},
{"POST", "/", `{"name":"q-val",` + body + `,"quarantine_days":0}`, 201},
{"PUT", "/q-val", `{` + body + `,"quarantine_enabled":true}`, 400},
{"PUT", "/q-val", `{` + body + `,"quarantine_enabled":true,"quarantine_days":1}`, 200},
{"PUT", "/q-val", `{` + body + `,"quarantine_enabled":true,"quarantine_days":0}`, 400},
} {
if c := do(t, h, s.method, s.path, s.json); c != s.want {
t.Errorf("step %d %s %s = %d, want %d", i, s.method, s.path, c, s.want)
}
}
}
+93 -17
View File
@@ -2,7 +2,9 @@ package auth
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"slices"
@@ -104,42 +106,116 @@ func (b *BearerOIDC) discoverOnce(done chan struct{}) (v *oidc.IDTokenVerifier,
// Require gates every request through next on a valid token. A nil receiver
// passes everything through.
func (b *BearerOIDC) Require(next http.Handler) http.Handler {
return b.require(next, "Bearer")
}
func (b *BearerOIDC) require(next http.Handler, challenge string) http.Handler {
if b == nil {
return next
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
scheme, raw, _ := strings.Cut(strings.TrimSpace(r.Header.Get("Authorization")), " ")
raw = strings.TrimSpace(raw)
if !strings.EqualFold(scheme, "Bearer") || raw == "" {
unauthorized(w, "missing bearer token")
raw := token(r)
if raw == "" {
unauthorized(w, challenge, "missing token")
return
}
v, err := b.getVerifier(r.Context())
if err != nil {
sub, err := b.verify(r.Context(), raw)
if errors.Is(err, errIssuerUnavailable) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusServiceUnavailable)
_, _ = w.Write([]byte(`{"error":"token issuer unavailable"}`))
return
}
tok, err := v.Verify(r.Context(), raw)
if err != nil {
slog.Info("bearer token rejected", "error", err)
unauthorized(w, "invalid token")
unauthorized(w, challenge, "invalid token")
return
}
if len(b.audiences) > 0 && !slices.ContainsFunc(tok.Audience, func(a string) bool { return slices.Contains(b.audiences, a) }) {
slog.Info("bearer token rejected", "error", "audience not allowed", "aud", tok.Audience)
unauthorized(w, "invalid token")
return
}
slog.Info("bearer token accepted", "sub", tok.Subject, "method", r.Method, "path", r.URL.Path)
slog.Info("bearer token accepted", "sub", sub, "method", r.Method, "path", r.URL.Path)
next.ServeHTTP(w, r)
})
}
func (b *BearerOIDC) verify(ctx context.Context, raw string) (string, error) {
v, err := b.getVerifier(ctx)
if err != nil {
return "", errIssuerUnavailable
}
tok, err := v.Verify(ctx, raw)
if err != nil {
return "", err
}
if len(b.audiences) > 0 && !slices.ContainsFunc(tok.Audience, func(a string) bool { return slices.Contains(b.audiences, a) }) {
return "", fmt.Errorf("audience %v not allowed", tok.Audience)
}
return tok.Subject, nil
}
// token reads the JWT from `Bearer`/`Token` credentials, the password of
// `Basic` credentials (any username), or a bare header value as `gem push`
// sends its API key.
func token(r *http.Request) string {
if _, pass, ok := r.BasicAuth(); ok {
return pass
}
scheme, raw, found := strings.Cut(strings.TrimSpace(r.Header.Get("Authorization")), " ")
if !found {
return scheme
}
if strings.EqualFold(scheme, "Bearer") || strings.EqualFold(scheme, "Token") {
return strings.TrimSpace(raw)
}
return ""
}
// RequireWrites gates only mutating methods.
func (b *BearerOIDC) RequireWrites(next http.Handler) http.Handler {
gated := b.Require(next)
return writesOnly(next, b.Require(next))
}
// RequireRegistryWrites gates Docker Registry V2 writes behind the token flow
// Docker clients need: an unauthenticated ping gets a Bearer challenge naming
// realm, which serves the Basic password back as the token once verified, or
// an anonymous token so pulls stay open.
func (b *BearerOIDC) RequireRegistryWrites(next http.Handler, realm func(*http.Request) string) http.Handler {
if b == nil {
return next
}
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
challenge := `Bearer realm="` + realm(r) + `",service="artifactapi"`
hasAuth := r.Header.Get("Authorization") != ""
switch {
case r.URL.Path == "/v2/token":
writeToken(w, b.registryToken(r))
case r.URL.Path == "/v2/" && !hasAuth:
unauthorized(w, challenge, "authentication required")
default:
writesOnly(next, b.require(next, challenge)).ServeHTTP(w, r)
}
})
}
// registryToken echoes a valid credential back as the registry token. Missing
// or invalid credentials (e.g. an expired `docker login`) get the anonymous
// token so pulls keep working; writes are still verified per request.
func (b *BearerOIDC) registryToken(r *http.Request) string {
raw := token(r)
if raw == "" {
return "anonymous"
}
if _, err := b.verify(r.Context(), raw); err != nil {
slog.Info("registry credential rejected, issuing anonymous token", "error", err)
return "anonymous"
}
return raw
}
func writeToken(w http.ResponseWriter, tok string) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]string{"token": tok, "access_token": tok})
}
func writesOnly(next, gated http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet, http.MethodHead, http.MethodOptions:
@@ -150,9 +226,9 @@ func (b *BearerOIDC) RequireWrites(next http.Handler) http.Handler {
})
}
func unauthorized(w http.ResponseWriter, msg string) {
func unauthorized(w http.ResponseWriter, challenge, msg string) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("WWW-Authenticate", "Bearer")
w.Header().Set("WWW-Authenticate", challenge)
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte(`{"error":"` + msg + `"}`))
}
+88
View File
@@ -8,6 +8,7 @@ import (
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"sync/atomic"
"testing"
@@ -253,3 +254,90 @@ func TestRequireWritesLeavesReadsOpen(t *testing.T) {
}
}
}
func basic(user, pass string) string {
return "Basic " + base64.StdEncoding.EncodeToString([]byte(user+":"+pass))
}
func TestTokenSchemes(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
h := NewBearerOIDC(iss.URL, []string{"artifactapi"}).RequireWrites(ok)
valid := iss.Token(nil, "artifactapi", "ci", time.Hour)
cases := []struct {
name string
authz string
want int
}{
{"bearer", "Bearer " + valid, http.StatusNoContent},
{"token scheme", "Token " + valid, http.StatusNoContent},
{"basic password", basic("anyone", valid), http.StatusNoContent},
{"basic empty user", basic("", valid), http.StatusNoContent},
{"bare jwt", valid, http.StatusNoContent},
{"basic wrong aud", basic("ci", iss.Token(nil, "nope", "ci", time.Hour)), http.StatusUnauthorized},
{"basic token as user", basic(valid, "x"), http.StatusUnauthorized},
{"bare garbage", "garbage", http.StatusUnauthorized},
{"unknown scheme", "Digest " + valid, http.StatusUnauthorized},
}
for _, c := range cases {
if got := serveAuth(h, http.MethodPut, c.authz); got != c.want {
t.Errorf("%s: status %d, want %d", c.name, got, c.want)
}
}
}
func TestRequireRegistryWrites(t *testing.T) {
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
h := NewBearerOIDC(iss.URL, []string{"artifactapi"}).RequireRegistryWrites(ok, func(*http.Request) string { return "https://reg.example/v2/token" })
valid := iss.Token(nil, "artifactapi", "ci", time.Hour)
do := func(method, path, authz string) *httptest.ResponseRecorder {
r := httptest.NewRequest(method, path, nil)
if authz != "" {
r.Header.Set("Authorization", authz)
}
w := httptest.NewRecorder()
h.ServeHTTP(w, r)
return w
}
challenge := `Bearer realm="https://reg.example/v2/token",service="artifactapi"`
w := do(http.MethodGet, "/v2/", "")
if w.Code != http.StatusUnauthorized || w.Header().Get("WWW-Authenticate") != challenge {
t.Errorf("anonymous ping: %d %q", w.Code, w.Header().Get("WWW-Authenticate"))
}
if w := do(http.MethodGet, "/v2/", "Bearer anonymous"); w.Code != http.StatusNoContent {
t.Errorf("ping with token: %d", w.Code)
}
if w := do(http.MethodGet, "/v2/token", ""); w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `"token":"anonymous"`) {
t.Errorf("anonymous token: %d %s", w.Code, w.Body)
}
if w := do(http.MethodGet, "/v2/token", basic("ci", valid)); w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `"token":"`+valid+`"`) {
t.Errorf("token exchange: %d %s", w.Code, w.Body)
}
expired := iss.Token(nil, "artifactapi", "ci", -time.Minute)
for _, authz := range []string{basic("ci", "bogus"), basic("ci", expired), "Bearer " + expired} {
if w := do(http.MethodGet, "/v2/token", authz); w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `"token":"anonymous"`) {
t.Errorf("rejected credential %q: %d %s", authz, w.Code, w.Body)
}
}
for _, m := range []string{http.MethodGet, http.MethodHead} {
if w := do(m, "/v2/repo/img/manifests/t", ""); w.Code != http.StatusNoContent {
t.Errorf("anonymous %s: %d", m, w.Code)
}
}
for _, m := range []string{http.MethodPost, http.MethodPatch, http.MethodPut, http.MethodDelete} {
w := do(m, "/v2/repo/img/blobs/uploads/", "Bearer anonymous")
if w.Code != http.StatusUnauthorized || w.Header().Get("WWW-Authenticate") != challenge {
t.Errorf("anonymous %s: %d %q", m, w.Code, w.Header().Get("WWW-Authenticate"))
}
for _, authz := range []string{"Bearer " + valid, basic("ci", valid)} {
if w := do(m, "/v2/repo/img/blobs/uploads/", authz); w.Code != http.StatusNoContent {
t.Errorf("authenticated %s: %d", m, w.Code)
}
}
}
if got := serve(NewBearerOIDC("", nil).RequireRegistryWrites(ok, nil), http.MethodPut, ""); got != http.StatusNoContent {
t.Errorf("disabled: status %d", got)
}
}
+14
View File
@@ -84,6 +84,20 @@ func (r *Redis) GetETag(ctx context.Context, remote, path string) (string, error
return val, err
}
// FirstSeen records now as the first sighting of path unless one is already
// stored, and returns the stored sighting.
func (r *Redis) FirstSeen(ctx context.Context, remote, path string, ttl time.Duration) (time.Time, error) {
key := fmt.Sprintf("firstseen:%s:%s", remote, path)
if err := r.client.SetNX(ctx, key, time.Now().Unix(), ttl).Err(); err != nil {
return time.Time{}, err
}
sec, err := r.client.Get(ctx, key).Int64()
if err != nil {
return time.Time{}, err
}
return time.Unix(sec, 0), nil
}
func (r *Redis) GetToken(ctx context.Context, key string) (string, error) {
val, err := r.client.Get(ctx, "token:"+key).Result()
if err == redis.Nil {
+10
View File
@@ -61,6 +61,8 @@ type Config struct {
// Bearer-JWT auth on configuration writes; empty issuer disables it.
OIDCIssuer string
OIDCAudiences []string
// AuthUploads extends that auth to local repo uploads, deletes and evictions.
AuthUploads bool
}
func (c *Config) DatabaseDSN() string {
@@ -105,6 +107,13 @@ func Load() (*Config, error) {
if oidcIssuer != "" && len(oidcAudiences) == 0 {
return nil, fmt.Errorf("ARTIFACTAPI_OIDC_AUDIENCE is required when ARTIFACTAPI_OIDC_ISSUER is set")
}
authUploads, err := strconv.ParseBool(getenv("ARTIFACTAPI_AUTH_UPLOADS", "false"))
if err != nil {
return nil, fmt.Errorf("invalid ARTIFACTAPI_AUTH_UPLOADS: %w", err)
}
if authUploads && oidcIssuer == "" {
return nil, fmt.Errorf("ARTIFACTAPI_OIDC_ISSUER is required when ARTIFACTAPI_AUTH_UPLOADS is set")
}
cfg := &Config{
ListenAddr: getenv("LISTEN_ADDR", ":8000"),
@@ -142,6 +151,7 @@ func Load() (*Config, error) {
OIDCIssuer: oidcIssuer,
OIDCAudiences: oidcAudiences,
AuthUploads: authUploads,
}
return cfg, nil
+20
View File
@@ -81,3 +81,23 @@ func TestLoadOIDC(t *testing.T) {
t.Error("expected error for issuer without audience")
}
}
func TestLoadAuthUploads(t *testing.T) {
cfg, err := Load()
if err != nil || cfg.AuthUploads {
t.Fatalf("default: %v, AuthUploads=%v", err, cfg.AuthUploads)
}
t.Setenv("ARTIFACTAPI_AUTH_UPLOADS", "true")
if _, err := Load(); err == nil {
t.Error("expected error for AUTH_UPLOADS without issuer")
}
t.Setenv("ARTIFACTAPI_OIDC_ISSUER", "https://vault.example/v1/identity/oidc")
t.Setenv("ARTIFACTAPI_OIDC_AUDIENCE", "artifactapi")
if cfg, err := Load(); err != nil || !cfg.AuthUploads {
t.Fatalf("enabled: %v, AuthUploads=%v", err, cfg.AuthUploads)
}
t.Setenv("ARTIFACTAPI_AUTH_UPLOADS", "maybe")
if _, err := Load(); err == nil {
t.Error("expected error for invalid ARTIFACTAPI_AUTH_UPLOADS")
}
}
+62 -4
View File
@@ -325,12 +325,12 @@ func (e *Engine) Head(ctx context.Context, remote models.Remote, path string, pr
return &HeadResult{ContentType: info.ContentType, Size: info.Size, Source: "cache"}, nil
}
return e.headUpstream(ctx, remote, path, prov)
return e.headUpstream(ctx, remote, path, prov, class)
}
// headUpstream issues an upstream HEAD, load-balancing across the remote's base
// URLs and failing over to the next mirror on a network error or 5xx.
func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*HeadResult, error) {
func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path string, prov provider.Provider, class Classification) (*HeadResult, error) {
order := e.baseURLAttemptOrder(remote)
if len(order) == 0 {
return nil, &ProxyError{Status: http.StatusBadGateway, Message: "no upstream base_url configured"}
@@ -338,7 +338,7 @@ func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path st
var lastErr error
for i, url := range order {
ctr := e.beginAttempt(remote, url)
result, err := e.headUpstreamOnce(ctx, withBaseURL(remote, url), path, prov)
result, err := e.headUpstreamOnce(ctx, withBaseURL(remote, url), path, prov, class)
endAttempt(ctr)
if err == nil {
return result, nil
@@ -353,7 +353,7 @@ func (e *Engine) headUpstream(ctx context.Context, remote models.Remote, path st
return nil, lastErr
}
func (e *Engine) headUpstreamOnce(ctx context.Context, remote models.Remote, path string, prov provider.Provider) (*HeadResult, error) {
func (e *Engine) headUpstreamOnce(ctx context.Context, remote models.Remote, path string, prov provider.Provider, class Classification) (*HeadResult, error) {
url := prov.UpstreamURL(remote, path)
authHeaders, err := prov.AuthHeaders(ctx, remote)
@@ -400,6 +400,11 @@ func (e *Engine) headUpstreamOnce(ctx context.Context, remote models.Remote, pat
if resp.StatusCode != http.StatusOK {
return nil, &ProxyError{Status: resp.StatusCode, Message: fmt.Sprintf("upstream returned %d", resp.StatusCode)}
}
if class == ClassImmutable {
if err := e.checkQuarantine(ctx, remote, path, resp.Header); err != nil {
return nil, err
}
}
contentType := prov.ContentType(path)
if ct := resp.Header.Get("Content-Type"); ct != "" {
@@ -527,6 +532,11 @@ func (e *Engine) fetchFromUpstreamOnce(ctx context.Context, remote models.Remote
}, nil
}
if err := e.checkQuarantine(ctx, remote, path, resp.Header); err != nil {
_ = resp.Body.Close()
return nil, err
}
// Immutable blobs are streamed through the content-addressable store
// (tempfile -> sha256 -> S3) so arbitrarily large artifacts never sit
// fully in memory. Immutable content is never rewritten in the proxy path.
@@ -663,6 +673,43 @@ func (e *Engine) checkUpstreamOnce(ctx context.Context, remote models.Remote, pa
return resp.StatusCode == http.StatusNotModified, nil
}
// checkQuarantine refuses an uncached immutable artifact published less than
// quarantine_days ago. The publish time is the upstream Last-Modified, capped
// at now, else the first time this path was seen.
func (e *Engine) checkQuarantine(ctx context.Context, remote models.Remote, path string, upstream http.Header) error {
if !remote.QuarantineEnabled {
return nil
}
if a, err := e.db.GetArtifact(ctx, remote.Name, path); err == nil && a != nil {
return nil
}
days := remote.QuarantineDays
if days < 1 {
days = models.DefaultQuarantineDays
}
window := time.Duration(days) * 24 * time.Hour
now := time.Now()
published, err := http.ParseTime(upstream.Get("Last-Modified"))
if err != nil {
if published, err = e.cache.FirstSeen(ctx, remote.Name, path, window+24*time.Hour); err != nil {
slog.Warn("first-seen lookup failed", "remote", remote.Name, "path", path, "error", err)
return &ProxyError{Status: http.StatusServiceUnavailable, Message: "quarantine check unavailable", RetryAfter: 30}
}
}
if published.After(now) {
published = now
}
if until := published.Add(window); now.Before(until) {
return &ProxyError{
Status: http.StatusForbidden,
Message: fmt.Sprintf("quarantined: published %s, available from %s", published.UTC().Format(time.RFC3339), until.UTC().Format(time.RFC3339)),
Published: published.UTC(),
Available: until.UTC(),
}
}
return nil
}
// etagTTL keeps a mutable path's ETag well past its freshness key so an expired
// path revalidates with a conditional HEAD, while idle paths still age out.
func etagTTL(ttl time.Duration) time.Duration {
@@ -821,10 +868,21 @@ func fetchBearerToken(ctx context.Context, wwwAuth string, remote models.Remote)
type ProxyError struct {
Status int
Message string
// Published and Available are set only on a quarantine refusal.
Published time.Time
Available time.Time
// RetryAfter, in seconds, is sent as Retry-After when set.
RetryAfter int
}
func (e *ProxyError) Error() string { return e.Message }
// Quarantined reports whether err is a quarantine refusal.
func Quarantined(err error) bool {
var pe *ProxyError
return errors.As(err, &pe) && !pe.Published.IsZero()
}
type UpstreamError struct {
Err error
}
+1 -1
View File
@@ -227,7 +227,7 @@ func TestHeadUpstreamUsesRemoteTimeout(t *testing.T) {
r := models.Remote{Name: "slow-head", PackageType: models.PackageGeneric, BaseURL: slow.URL, UpstreamResponseHeaderTimeout: 1}
start := time.Now()
_, err := (&Engine{}).headUpstreamOnce(context.Background(), r, "blob.bin", prov(t, models.PackageGeneric))
_, err := (&Engine{}).headUpstreamOnce(context.Background(), r, "blob.bin", prov(t, models.PackageGeneric), ClassImmutable)
if err == nil || time.Since(start) > 2*time.Second {
t.Fatalf("HEAD ignored the remote's response header timeout: err=%v after %s", err, time.Since(start))
}
+186
View File
@@ -0,0 +1,186 @@
package proxy
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/cache"
"git.unkin.net/unkin/artifactapi/pkg/models"
goredis "github.com/redis/go-redis/v9"
)
func must[T any](v T, err error) T {
if err != nil {
panic(err)
}
return v
}
func quarantineUpstream(t *testing.T) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/fresh.tgz":
w.Header().Set("Last-Modified", time.Now().Add(-time.Hour).UTC().Format(http.TimeFormat))
case "/old.tgz":
w.Header().Set("Last-Modified", time.Now().Add(-30*24*time.Hour).UTC().Format(http.TimeFormat))
case "/future.tgz":
w.Header().Set("Last-Modified", time.Now().Add(30*24*time.Hour).UTC().Format(http.TimeFormat))
case "/undated.tgz", "/pkg":
default:
http.NotFound(w, r)
return
}
_, _ = w.Write([]byte("body " + r.URL.Path))
}))
t.Cleanup(srv.Close)
return srv
}
func quarantineRemote(t *testing.T, name string, enabled bool) models.Remote {
r := models.Remote{Name: name, PackageType: models.PackageNPM, RepoType: models.RepoTypeRemote,
BaseURL: quarantineUpstream(t).URL, QuarantineEnabled: enabled, QuarantineDays: 3}
return seed(t, r)
}
func wantQuarantined(t *testing.T, err error) *ProxyError {
t.Helper()
var pe *ProxyError
if !errors.As(err, &pe) || pe.Status != http.StatusForbidden || !Quarantined(err) {
t.Fatalf("want quarantine 403, got %v", err)
}
return pe
}
func TestQuarantineBlocksFreshArtifact(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := quarantineRemote(t, "q-fresh", true)
pe := wantQuarantined(t, func() error { _, err := testEngine.Fetch(ctx, r, "fresh.tgz", prov(t, models.PackageNPM)); return err }())
if d := time.Until(pe.Available); d < 70*time.Hour || d > 72*time.Hour {
t.Errorf("available in %s, want ~71h", d)
}
if a, _ := testDB.GetArtifact(ctx, r.Name, "fresh.tgz"); a != nil {
t.Error("quarantined artifact was cached")
}
_, err := testEngine.Head(ctx, r, "fresh.tgz", prov(t, models.PackageNPM))
_ = wantQuarantined(t, err)
}
func TestQuarantineServesOldArtifact(t *testing.T) {
requireStack(t)
r := quarantineRemote(t, "q-old", true)
res, err := testEngine.Fetch(context.Background(), r, "old.tgz", prov(t, models.PackageNPM))
if err != nil {
t.Fatal(err)
}
if got := readAll(t, res); got != "body /old.tgz" {
t.Errorf("body = %q", got)
}
}
func TestQuarantineFallsBackToFirstSeen(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := quarantineRemote(t, "q-undated", true)
p := prov(t, models.PackageNPM)
_, err := testEngine.Fetch(ctx, r, "undated.tgz", p)
first := wantQuarantined(t, err).Published
if time.Since(first) > time.Minute {
t.Fatalf("first seen %s, want now", first)
}
_, err = testEngine.Fetch(ctx, r, "undated.tgz", p)
if again := wantQuarantined(t, err).Published; !again.Equal(first) {
t.Errorf("first seen moved: %s -> %s", first, again)
}
r.QuarantineDays = 1
c := goredis.NewClient(must(goredis.ParseURL(testRedis)))
defer func() { _ = c.Close() }()
if err := c.Set(ctx, "firstseen:q-undated:undated.tgz", time.Now().Add(-48*time.Hour).Unix(), time.Hour).Err(); err != nil {
t.Fatal(err)
}
if _, err := testEngine.Fetch(ctx, r, "undated.tgz", p); err != nil {
t.Errorf("seen 48h ago with 1 day quarantine: %v", err)
}
}
func TestQuarantineSkipsMutable(t *testing.T) {
requireStack(t)
r := quarantineRemote(t, "q-mutable", true)
if _, err := testEngine.Fetch(context.Background(), r, "pkg", prov(t, models.PackageNPM)); err != nil {
t.Errorf("mutable index blocked: %v", err)
}
}
func TestQuarantineDisabled(t *testing.T) {
requireStack(t)
r := quarantineRemote(t, "q-off", false)
if _, err := testEngine.Fetch(context.Background(), r, "fresh.tgz", prov(t, models.PackageNPM)); err != nil {
t.Errorf("disabled quarantine blocked: %v", err)
}
}
func TestQuarantineKeepsServingCached(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := quarantineRemote(t, "q-cached", false)
p := prov(t, models.PackageNPM)
if _, err := testEngine.Fetch(ctx, r, "fresh.tgz", p); err != nil {
t.Fatal(err)
}
r.QuarantineEnabled = true
expire(t, r.Name, "fresh.tgz")
if _, err := testEngine.Fetch(ctx, r, "fresh.tgz", p); err != nil {
t.Errorf("artifact cached before quarantine refused on refetch: %v", err)
}
}
func TestQuarantineCapsFutureLastModified(t *testing.T) {
requireStack(t)
r := quarantineRemote(t, "q-future", true)
_, err := testEngine.Fetch(context.Background(), r, "future.tgz", prov(t, models.PackageNPM))
pe := wantQuarantined(t, err)
if d := time.Until(pe.Available); d < 71*time.Hour || d > 72*time.Hour {
t.Errorf("available in %s, want ~72h", d)
}
}
func TestQuarantineRedisErrorUnavailable(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := quarantineRemote(t, "q-redis-down", true)
r.StaleOnError = true
down := must(cache.NewRedis(testRedis))
_ = down.Close()
eng := NewEngine(testDB, down, testEngine.store)
_, err := eng.Fetch(ctx, r, "undated.tgz", prov(t, models.PackageNPM))
var pe *ProxyError
if !errors.As(err, &pe) || pe.Status != http.StatusServiceUnavailable || pe.RetryAfter == 0 || Quarantined(err) {
t.Fatalf("want 503 with Retry-After, got %#v", err)
}
if a, _ := testDB.GetArtifact(ctx, r.Name, "undated.tgz"); a != nil {
t.Error("artifact cached while quarantine check unavailable")
}
}
func TestQuarantineStaleOnErrorServesNothingUncached(t *testing.T) {
requireStack(t)
ctx := context.Background()
r := quarantineRemote(t, "q-stale", true)
r.StaleOnError = true
p := prov(t, models.PackageNPM)
_, err := testEngine.Fetch(ctx, r, "fresh.tgz", p)
_ = wantQuarantined(t, err)
r.BaseURL = "http://127.0.0.1:1"
if _, err := testEngine.Fetch(ctx, r, "fresh.tgz", p); err == nil {
t.Error("stale_on_error served an uncached quarantined path")
}
if a, _ := testDB.GetArtifact(ctx, r.Name, "fresh.tgz"); a != nil {
t.Error("quarantined artifact was cached")
}
}
+96
View File
@@ -1,6 +1,7 @@
package server
import (
"encoding/base64"
"io"
"net/http"
"net/http/httptest"
@@ -78,3 +79,98 @@ func TestServerOIDCWriteAuth(t *testing.T) {
expect(204, "DELETE", "/api/v2/virtuals/srv-oidc-v", "", good)
expect(204, "DELETE", "/api/v2/remotes/srv-oidc", "", good)
}
func TestServerAuthUploads(t *testing.T) {
requireStack(t)
iss := testsupport.NewOIDCIssuer()
defer iss.Close()
cfg := *testSrv.cfg
cfg.OIDCIssuer = iss.URL
cfg.OIDCAudiences = []string{"artifactapi"}
cfg.AuthUploads = true
s := *testSrv
s.cfg = &cfg
ts := httptest.NewServer(s.routes())
defer ts.Close()
good := iss.Token(nil, "artifactapi", "ci", time.Hour)
do := func(method, path, body, authz string) int {
t.Helper()
rq, _ := http.NewRequest(method, ts.URL+path, strings.NewReader(body))
rq.Header.Set("Content-Type", "application/json")
if authz != "" {
rq.Header.Set("Authorization", authz)
}
resp, err := http.DefaultClient.Do(rq)
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
return resp.StatusCode
}
bearer := "Bearer " + good
for _, typ := range []string{"generic", "docker", "npm", "gems"} {
body := `{"name":"srv-up-` + typ + `","package_type":"` + typ + `","repo_type":"local"}`
if got := do("POST", "/api/v2/remotes", body, bearer); got != 201 {
t.Fatalf("create %s: status %d", typ, got)
}
}
writes := []struct{ method, path, body string }{
{"PUT", "/api/v2/remotes/srv-up-generic/files/a.bin", "payload"},
{"DELETE", "/api/v2/remotes/srv-up-generic/files/a.bin", ""},
{"DELETE", "/api/v2/remotes/srv-up-generic/objects/a.bin", ""},
{"DELETE", "/api/v2/locals/srv-up-generic/objects/a.bin", ""},
{"PUT", "/api/v1/local/srv-up-npm/pkg", "{}"},
{"POST", "/api/v1/local/srv-up-gems/api/v1/gems", "x"},
{"DELETE", "/api/v1/local/srv-up-gems/api/v1/gems/yank", ""},
{"POST", "/v2/srv-up-docker/img/blobs/uploads/", ""},
{"PATCH", "/v2/srv-up-docker/img/blobs/uploads/x", ""},
{"PUT", "/v2/srv-up-docker/img/manifests/t", "{}"},
{"DELETE", "/v2/srv-up-docker/img/manifests/t", ""},
}
for _, w := range writes {
if got := do(w.method, w.path, w.body, ""); got != 401 {
t.Errorf("anonymous %s %s: status %d, want 401", w.method, w.path, got)
}
}
if got := do("PUT", "/api/v2/remotes/srv-up-generic/files/a.bin", "payload", bearer); got != 201 {
t.Errorf("bearer upload: status %d", got)
}
if got := do("PUT", "/api/v2/remotes/srv-up-generic/files/b.bin", "payload", basic(good)); got != 201 {
t.Errorf("basic upload: status %d", got)
}
if got := do("POST", "/v2/srv-up-docker/img/blobs/uploads/", "", basic(good)); got != 202 {
t.Errorf("basic docker upload: status %d", got)
}
expectOpen := func(method, path string) {
t.Helper()
if got := do(method, path, "", ""); got >= 400 {
t.Errorf("anonymous %s %s: status %d", method, path, got)
}
}
expectOpen("GET", "/api/v2/remotes/srv-up-generic/files/a.bin")
expectOpen("GET", "/api/v2/remotes/srv-up-generic/objects")
expectOpen("GET", "/api/v1/local/srv-up-generic/a.bin")
expectOpen("GET", "/v2/token")
if got := do("GET", "/v2/", "", ""); got != 401 {
t.Errorf("anonymous ping: status %d, want 401 challenge", got)
}
if got := do("POST", "/api/v2/probe", `{}`, ""); got == 401 {
t.Error("probe gated")
}
if got := do("DELETE", "/api/v2/remotes/srv-up-generic/files/a.bin", "", bearer); got >= 400 {
t.Errorf("bearer delete: status %d", got)
}
for _, typ := range []string{"generic", "docker", "npm", "gems"} {
do("DELETE", "/api/v2/remotes/srv-up-"+typ, "", bearer)
}
}
func basic(tok string) string {
return "Basic " + base64.StdEncoding.EncodeToString([]byte("ci:"+tok))
}
+16 -6
View File
@@ -169,8 +169,17 @@ func (s *Server) routes() chi.Router {
s.virtEngine.NPMLocal = proxyHandler.LocalNPMPackument
s.virtEngine.GemsLocal = proxyHandler.LocalGemsIndex
s.virtEngine.AnsibleLocal = proxyHandler.LocalAnsible
r.Mount("/api/v1", proxyHandler.Routes())
r.Mount("/v2", proxyHandler.DockerV2Routes())
writeAuth := auth.NewBearerOIDC(s.cfg.OIDCIssuer, s.cfg.OIDCAudiences)
uploadAuth := passthrough
dockerV2 := http.Handler(proxyHandler.DockerV2Routes())
if s.cfg.AuthUploads {
uploadAuth = writeAuth.RequireWrites
dockerV2 = writeAuth.RequireRegistryWrites(dockerV2, func(r *http.Request) string {
return v1.Scheme(r) + "://" + r.Host + "/v2/token"
})
}
r.Mount("/api/v1", uploadAuth(proxyHandler.Routes()))
r.Mount("/v2", dockerV2)
remotesHandler := v2.NewRemotesHandler(s.db, s.cache, map[models.PackageType]v2.Primer{
models.PackageGitHubRPM: s.syncer,
@@ -182,7 +191,6 @@ func (s *Server) routes() chi.Router {
statsHandler := v2.NewStatsHandler(s.db)
eventsHandler := v2.NewEventsHandler()
probeHandler := v2.NewProbeHandler(s.engine, s.db)
writeAuth := auth.NewBearerOIDC(s.cfg.OIDCIssuer, s.cfg.OIDCAudiences)
r.Route("/api/v2", func(r chi.Router) {
r.Mount("/remotes", writeAuth.RequireWrites(remotesHandler.Routes()))
@@ -192,19 +200,19 @@ func (s *Server) routes() chi.Router {
r.Mount("/events", eventsHandler.Routes())
r.Mount("/probe", probeHandler.Routes())
r.Route("/remotes/{name}/objects", func(r chi.Router) {
r.With(uploadAuth).Route("/remotes/{name}/objects", func(r chi.Router) {
objHandler := v2.NewObjectsHandler(s.db)
r.Get("/", objHandler.Routes(s.engine).ServeHTTP)
r.Delete("/*", objHandler.Routes(s.engine).ServeHTTP)
})
r.Route("/locals/{name}/objects", func(r chi.Router) {
r.With(uploadAuth).Route("/locals/{name}/objects", func(r chi.Router) {
objHandler := v2.NewObjectsHandler(s.db)
r.Get("/", objHandler.LocalRoutes().ServeHTTP)
r.Delete("/*", objHandler.LocalRoutes().ServeHTTP)
})
r.Route("/remotes/{name}/files", func(r chi.Router) {
r.With(uploadAuth).Route("/remotes/{name}/files", func(r chi.Router) {
r.Put("/*", s.localHandler.Routes().ServeHTTP)
r.Get("/*", s.localHandler.Routes().ServeHTTP)
r.Delete("/*", s.localHandler.Routes().ServeHTTP)
@@ -214,6 +222,8 @@ func (s *Server) routes() chi.Router {
return r
}
func passthrough(h http.Handler) http.Handler { return h }
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusOK)
+12 -1
View File
@@ -194,10 +194,18 @@ func (e *Engine) goList(ctx context.Context, virt models.Virtual, escMod string)
return nil, ErrNotFound
}
// goProbe returns the first member, in member order, that serves path.
// goProbe returns the first member, in member order, that serves path, else
// the first member quarantining it so the client sees that refusal.
func (e *Engine) goProbe(ctx context.Context, virt models.Virtual, path string) (string, error) {
held := ""
for _, name := range virt.Members {
m, err := e.goMember(ctx, name, path)
if proxy.Quarantined(err) {
if held == "" {
held = name
}
continue
}
if err != nil {
return "", fmt.Errorf("virtual %q: member %q: %v", virt.Name, name, err)
}
@@ -205,6 +213,9 @@ func (e *Engine) goProbe(ctx context.Context, virt models.Virtual, path string)
return name, nil
}
}
if held != "" {
return held, nil
}
return "", ErrNotFound
}
+33
View File
@@ -3,11 +3,15 @@ package virtual
import (
"context"
"errors"
"fmt"
"net/http"
"strings"
"sync"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/database"
"git.unkin.net/unkin/artifactapi/internal/proxy"
"git.unkin.net/unkin/artifactapi/internal/testsupport"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -181,3 +185,32 @@ func TestGoProxyLocalMemberNotFound(t *testing.T) {
t.Fatalf("module only absent from a local member must be not-found, got %v", err)
}
}
func TestGoProbeFallsThroughQuarantine(t *testing.T) {
quarantined := &proxy.ProxyError{Status: http.StatusForbidden, Message: "quarantined", Published: time.Now()}
served := map[string]bool{}
e := &Engine{
mergeTTL: time.Minute,
getRemote: func(_ context.Context, name string) (*models.Remote, error) {
return &models.Remote{Name: name, RepoType: models.RepoTypeRemote}, nil
},
goMember: func(_ context.Context, name, path string) (*GoMember, error) {
if name == "internal" && strings.HasSuffix(path, ".info") {
return nil, fmt.Errorf("fetch: %w", quarantined)
}
if served[name] {
return &GoMember{RemoteName: name, Body: []byte("{}")}, nil
}
return &GoMember{RemoteName: name}, nil
},
}
zip := "example.com/mod/@v/v0.0.0-20260101-abcdef.zip"
served["goproxy"] = true
if _, loc, err := e.GoProxy(context.Background(), goVirt(), zip, ""); err != nil || loc != "/api/v1/remote/goproxy/"+zip {
t.Errorf("next member serving: %q %v", loc, err)
}
served["goproxy"] = false
if _, loc, err := e.GoProxy(context.Background(), goVirt(), zip, ""); err != nil || loc != "/api/v1/remote/internal/"+zip {
t.Errorf("only quarantined member: %q %v, want redirect to its refusal", loc, err)
}
}
+1
View File
@@ -0,0 +1 @@
UPDATE remotes SET quarantine_days = 3 WHERE quarantine_days IS NULL OR quarantine_days <= 0;
+11
View File
@@ -152,6 +152,17 @@ func (r *Remote) ValidateMirrorStrategy() error {
return nil
}
// DefaultQuarantineDays matches the remotes.quarantine_days column default.
const DefaultQuarantineDays = 3
// ValidateQuarantine rejects an enabled quarantine with no window.
func (r *Remote) ValidateQuarantine() error {
if r.QuarantineEnabled && r.QuarantineDays < 1 {
return fmt.Errorf("quarantine_days must be at least 1 when quarantine_enabled is set")
}
return nil
}
// ValidatePatterns ensures every configured regex compiles. Storing an
// invalid pattern would otherwise be silently dropped at match time, which
// for the blocklist is a fail-open: a mistyped deny rule becomes a no-op.