Compare commits
12 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 26c37024ae | |||
| 5fde0ee58e | |||
| 60f008debc | |||
| 109ba2ce27 | |||
| e24c35f534 | |||
| d154fbf3f3 | |||
| eee8ee1c31 | |||
| f6b0afc5d6 | |||
| 649f89f58b | |||
| a92ede23f6 | |||
| 936cf8846a | |||
| 3a3b7fe7b7 |
+1
-1
@@ -1,5 +1,5 @@
|
|||||||
bin/
|
bin/
|
||||||
terraform/
|
/terraform/
|
||||||
|
|
||||||
# e2e-docker fixtures are real package files (.rpm, .tgz, .whl, .zip, ...) that
|
# e2e-docker fixtures are real package files (.rpm, .tgz, .whl, .zip, ...) that
|
||||||
# are intentionally tracked, overriding any global ignore of those extensions.
|
# are intentionally tracked, overriding any global ignore of those extensions.
|
||||||
|
|||||||
@@ -32,9 +32,150 @@ API: `http://localhost:8000` | Frontend: `http://localhost:5173`
|
|||||||
| `puppet` | `v3/modules/*`, `v3/releases*` | `.tar.gz` |
|
| `puppet` | `v3/modules/*`, `v3/releases*` | `.tar.gz` |
|
||||||
| `terraform` | `*/versions` | `*/download/*/*` |
|
| `terraform` | `*/versions` | `*/download/*/*` |
|
||||||
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
|
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
|
||||||
|
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
|
||||||
|
|
||||||
Providers classify paths automatically. Users only configure what to proxy and TTLs.
|
Providers classify paths automatically. Users only configure what to proxy and TTLs.
|
||||||
|
|
||||||
|
### `github_rpm` — GitHub releases as a yum repo (metadata-only, no precache)
|
||||||
|
|
||||||
|
A `github_rpm` remote turns a GitHub repo's **releases** into a real `dnf`/`yum`
|
||||||
|
repository without ever caching the packages. It scans releases for `.rpm`
|
||||||
|
assets, derives each package's metadata (NEVRA, requires/provides/conflicts/
|
||||||
|
obsoletes, files, checksum) and **synthesizes `repodata/` on the fly**. Package
|
||||||
|
metadata comes from a **ranged GET of just the RPM header** (the header sits at
|
||||||
|
the front of the file, so the whole package is never downloaded); the sha256
|
||||||
|
checksum comes from the GitHub asset `digest` when present, else a one-time
|
||||||
|
lazy stream. Derived metadata is cached (keyed by asset) so repodata generation
|
||||||
|
is served from primed DB rows, never a cold on-demand derive.
|
||||||
|
|
||||||
|
Each package's `<location>` points back at the remote, which **302-redirects**
|
||||||
|
the download to the `releases_remote` — an existing generic `github.com` remote
|
||||||
|
that streams the actual bytes. `dnf` follows the redirect transparently.
|
||||||
|
|
||||||
|
#### Background syncer
|
||||||
|
|
||||||
|
A single process-wide **background syncer** keeps every `github_rpm` remote's
|
||||||
|
derived metadata current off the client request path:
|
||||||
|
|
||||||
|
- **Prime on create.** Creating a `github_rpm` remote enqueues a background prime
|
||||||
|
scan, so its metadata is derived right away without blocking the create call.
|
||||||
|
The first `dnf` request is served from cache. If a request arrives before the
|
||||||
|
prime lands, it returns a retryable `503` (with `Retry-After`) rather than
|
||||||
|
serving an empty repo or blocking on a multi-minute derive.
|
||||||
|
- **Periodic re-check, driven by `mutable_ttl`.** Each remote is re-checked for
|
||||||
|
new or changed releases no more often than its `mutable_ttl`. New/changed
|
||||||
|
assets are derived incrementally; assets already cached are never re-fetched,
|
||||||
|
and assets that disappear upstream are pruned.
|
||||||
|
- **ETag / 304 conditional requests.** The releases-list `ETag` is stored per
|
||||||
|
remote and sent as `If-None-Match`; a `304 Not Modified` means nothing changed
|
||||||
|
and the syncer derives nothing. GitHub does not count `304` conditional
|
||||||
|
responses against the rate limit, so an unchanged repo is nearly free — this is
|
||||||
|
the main lever keeping GitHub traffic low.
|
||||||
|
- **Global rate limit.** Every GitHub call (releases list + each ranged asset
|
||||||
|
header GET) passes through a single token-bucket limiter **shared across all
|
||||||
|
remotes**, so GitHub is never hammered. Configure a token (`password`) on the
|
||||||
|
remote for the higher authenticated rate limit (~5000/hr vs ~60/hr
|
||||||
|
unauthenticated).
|
||||||
|
- **Multi-replica coordination.** State is shared through the database. Before a
|
||||||
|
periodic scan a replica must atomically claim a per-remote lease
|
||||||
|
(`github_rpm_sync_state`: `last_synced_at`, `etag`, `sync_lease_owner`,
|
||||||
|
`sync_lease_expires`); only the winner scans. This bounds total GitHub load to
|
||||||
|
~once per `mutable_ttl` regardless of replica count, and the shared `etag`
|
||||||
|
lets any replica issue the conditional request.
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
# Backend that serves the actual .rpm bytes from github.com.
|
||||||
|
resource "artifactapi_remote_generic" "github" {
|
||||||
|
name = "github"
|
||||||
|
base_url = "https://github.com"
|
||||||
|
patterns = [
|
||||||
|
"acme/tools/releases/download/.*\\.rpm$", # allowlist the repo's release assets
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
resource "artifactapi_remote_github_rpm" "acme-tools" {
|
||||||
|
name = "acme-tools"
|
||||||
|
base_url = "https://api.github.com/repos/acme/tools" # the releases API root
|
||||||
|
releases_remote = "github" # backend for downloads
|
||||||
|
mutable_ttl = 3600 # release re-scan interval
|
||||||
|
|
||||||
|
# Optional: restrict which release assets become packages (regex on filename).
|
||||||
|
patterns = [".*\\.x86_64\\.rpm$", ".*\\.noarch\\.rpm$"]
|
||||||
|
|
||||||
|
# Optional: a token for private repos / higher API rate limits.
|
||||||
|
# password = "ghp_..."
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`dnf` config: `baseurl=https://artifactapi.example/api/v1/remote/acme-tools`.
|
||||||
|
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
|
||||||
|
from the synthesized metadata.
|
||||||
|
|
||||||
|
### GitHub authentication
|
||||||
|
|
||||||
|
Anonymous GitHub is capped at **60 requests/hour** and cannot read private
|
||||||
|
repositories. Configure a **server-level GitHub credential** to raise the ceiling
|
||||||
|
to roughly **5000 requests/hour** and to read private-repo release assets. The
|
||||||
|
credential is a process-wide machine identity applied by default to *every*
|
||||||
|
outbound GitHub request — the releases scan, the ranged asset-header fetches, and
|
||||||
|
the generic-github byte proxy that streams private release assets.
|
||||||
|
|
||||||
|
The credential is read from the environment (deliver it from a Vault or
|
||||||
|
Kubernetes secret). It is **never** stored per-remote in the database, **never**
|
||||||
|
returned by any API, and **never** logged. Configure **exactly one** mode.
|
||||||
|
|
||||||
|
**Precedence.** A remote's own `username`/`password` credential still wins for
|
||||||
|
that remote's requests; the server credential is the default for everything else.
|
||||||
|
With no credential configured at all, requests stay anonymous (current behavior).
|
||||||
|
Partial configuration (e.g. an App id with no private key) is a **startup error**
|
||||||
|
— artifactapi fails closed rather than silently falling back to anonymous.
|
||||||
|
|
||||||
|
Both modes share the syncer's single global rate limiter, so a token simply
|
||||||
|
raises the effective GitHub ceiling; the default limiter settings stay safe.
|
||||||
|
|
||||||
|
#### Mode 1 — Personal Access Token (minimum viable, recommended for free accounts)
|
||||||
|
|
||||||
|
Set `GITHUB_TOKEN`. It is sent as `Authorization: Bearer <token>`.
|
||||||
|
|
||||||
|
Recommended free-account setup — a **fine-grained PAT** scoped to just the target
|
||||||
|
repositories:
|
||||||
|
|
||||||
|
1. GitHub → *Settings → Developer settings → Personal access tokens →
|
||||||
|
Fine-grained tokens → Generate new token*.
|
||||||
|
2. Limit *Repository access* to the specific repo(s) serving releases.
|
||||||
|
3. Grant repository permissions **Contents: Read-only** and **Metadata:
|
||||||
|
Read-only** (Metadata is mandatory and auto-selected).
|
||||||
|
|
||||||
|
A classic PAT with the `repo` scope also works but is broader than necessary.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GITHUB_TOKEN=github_pat_xxxxxxxx
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Mode 2 — GitHub App installation token (proper machine identity)
|
||||||
|
|
||||||
|
A GitHub App is not tied to a personal account and can be created and installed on
|
||||||
|
free personal repos. artifactapi mints a short-lived RS256 **JWT** from the app
|
||||||
|
private key, exchanges it at `POST /app/installations/{id}/access_tokens` for a
|
||||||
|
~1-hour **installation access token**, caches that token, and refreshes it a few
|
||||||
|
minutes before expiry (thread-safe, single-flighted).
|
||||||
|
|
||||||
|
1. GitHub → *Settings → Developer settings → GitHub Apps → New GitHub App*.
|
||||||
|
2. Under *Permissions → Repository permissions* grant **Contents: Read-only**
|
||||||
|
(Metadata: Read-only is implied).
|
||||||
|
3. Generate a **private key** (downloads a PEM) and note the **App ID**.
|
||||||
|
4. *Install* the App on the account and select the target repositories, then read
|
||||||
|
the **Installation ID** from the installation URL
|
||||||
|
(`.../settings/installations/<installation-id>`).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GITHUB_APP_ID=123456
|
||||||
|
GITHUB_APP_INSTALLATION_ID=7654321
|
||||||
|
GITHUB_APP_PRIVATE_KEY_PATH=/etc/artifactapi/github-app.pem
|
||||||
|
# or inline PEM (e.g. mounted from a secret):
|
||||||
|
# GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
|
||||||
|
```
|
||||||
|
|
||||||
## Terraform
|
## Terraform
|
||||||
|
|
||||||
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
|
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
|
||||||
@@ -89,6 +230,54 @@ resource "artifactapi_virtual" "helm" {
|
|||||||
|
|
||||||
Provider: [terraform-provider-artifactapi](../terraform-provider-artifactapi)
|
Provider: [terraform-provider-artifactapi](../terraform-provider-artifactapi)
|
||||||
|
|
||||||
|
### Serving providers as a registry
|
||||||
|
|
||||||
|
A local `terraform` repo is a real provider registry: upload
|
||||||
|
`terraform-provider-{type}_{version}_{os}_{arch}.zip` files under
|
||||||
|
`{namespace}/{type}/`, and Terraform installs them from a bare source address —
|
||||||
|
no `.terraformrc` mirror config:
|
||||||
|
|
||||||
|
```hcl
|
||||||
|
terraform {
|
||||||
|
required_providers {
|
||||||
|
artifactapi = {
|
||||||
|
source = "artifactapi.k8s.syd1.au.unkin.net/<repo>/<type>"
|
||||||
|
version = "0.1.2"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The Terraform *namespace* segment is the artifactapi repo name; the provider is
|
||||||
|
matched by *type*. The registry serves service discovery
|
||||||
|
(`/.well-known/terraform.json`), the `providers.v1` version/download endpoints,
|
||||||
|
and a GPG-signed `SHA256SUMS` per the provider registry protocol.
|
||||||
|
|
||||||
|
Signing needs a GPG key. By default artifactapi generates one on first start and
|
||||||
|
stores it in the database (`signing_keys` table), so every replica shares it and
|
||||||
|
there's nothing to provision. To bring your own key instead, point
|
||||||
|
`TF_SIGNING_KEY_PATH` at an armored private key (optionally
|
||||||
|
`TF_SIGNING_KEY_PASSPHRASE`), which takes precedence over the generated one.
|
||||||
|
`TF_PROVIDER_PROTOCOLS` (default `5.0,6.0`) sets the advertised plugin protocols.
|
||||||
|
|
||||||
|
### Local docker registry
|
||||||
|
|
||||||
|
A local `docker` repo is a real container registry, not a mirror: it serves the
|
||||||
|
Docker Registry HTTP API V2 for both push and pull, so any client (`docker`,
|
||||||
|
`podman`, `skopeo`, `buildah`) can use it directly.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker tag myapp:latest artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
|
||||||
|
docker push artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
|
||||||
|
docker pull artifactapi.k8s.syd1.au.unkin.net/docker-internal/myapp:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
The first path segment after `/v2/` is the artifactapi repo name; the remainder
|
||||||
|
is the image name. Blobs and manifests are stored through the shared
|
||||||
|
content-addressable store (deduplicated by digest, reaped by GC once
|
||||||
|
unreferenced); tags are mutable references and re-pushing a tag moves it. Blob
|
||||||
|
uploads support both the monolithic and chunked (`POST`/`PATCH`/`PUT`) flows.
|
||||||
|
|
||||||
## Access Control
|
## Access Control
|
||||||
|
|
||||||
| Field | Default | Behaviour |
|
| Field | Default | Behaviour |
|
||||||
@@ -149,6 +338,15 @@ S3 client supports MinIO, Ceph RGW, and AWS S3 (via minio-go).
|
|||||||
| `MINIO_BUCKET` | `artifacts` | S3 bucket |
|
| `MINIO_BUCKET` | `artifacts` | S3 bucket |
|
||||||
| `MINIO_SECURE` | `false` | Use HTTPS for S3 |
|
| `MINIO_SECURE` | `false` | Use HTTPS for S3 |
|
||||||
| `MINIO_REGION` | | S3 region (AWS) |
|
| `MINIO_REGION` | | S3 region (AWS) |
|
||||||
|
| `GITHUB_SYNC_RATE` | `1` | `github_rpm` syncer global GitHub request rate (req/s), shared across all remotes. `1`/s = 3600/hr, under an authenticated token's ~5000/hr; unauthenticated (~60/hr) relies on ETag/304 |
|
||||||
|
| `GITHUB_SYNC_BURST` | `5` | Token-bucket burst for the shared limiter |
|
||||||
|
| `GITHUB_SYNC_WORKERS` | `3` | Concurrent `github_rpm` scan workers |
|
||||||
|
| `GITHUB_SYNC_POLL_INTERVAL` | `60` | Base scheduler tick in seconds; per-remote cadence is its `mutable_ttl`, enforced by the DB lease |
|
||||||
|
| `GITHUB_TOKEN` | | Server-level GitHub PAT (fine-grained or classic), sent as `Authorization: Bearer`. Applies to every GitHub request; per-remote creds override it. See [GitHub authentication](#github-authentication) |
|
||||||
|
| `GITHUB_APP_ID` | | GitHub App id (App auth mode; mutually exclusive with `GITHUB_TOKEN`) |
|
||||||
|
| `GITHUB_APP_INSTALLATION_ID` | | GitHub App installation id |
|
||||||
|
| `GITHUB_APP_PRIVATE_KEY` | | GitHub App private key, inline PEM |
|
||||||
|
| `GITHUB_APP_PRIVATE_KEY_PATH` | | GitHub App private key, file path (alternative to inline PEM) |
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,177 @@
|
|||||||
|
//go:build dockere2e
|
||||||
|
|
||||||
|
package e2edocker
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func digestOf(b []byte) string {
|
||||||
|
sum := sha256.Sum256(b)
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
// pushBlobMonolithic uploads a blob with POST (open session) then PUT?digest
|
||||||
|
// (whole body) — the monolithic-after-POST flow.
|
||||||
|
func pushBlobMonolithic(t *testing.T, repo, image string, blob []byte) {
|
||||||
|
t.Helper()
|
||||||
|
dgst := digestOf(blob)
|
||||||
|
|
||||||
|
resp, body := doRequest(t, http.MethodPost, api("/v2/"+repo+"/"+image+"/blobs/uploads/"), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusAccepted {
|
||||||
|
t.Fatalf("start upload: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
loc := resp.Header.Get("Location")
|
||||||
|
if loc == "" {
|
||||||
|
t.Fatalf("start upload: no Location header")
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, body = doRequest(t, http.MethodPut, baseURL()+loc+"?digest="+dgst, blob, "application/octet-stream")
|
||||||
|
if resp.StatusCode != http.StatusCreated {
|
||||||
|
t.Fatalf("finish upload: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
if got := resp.Header.Get("Docker-Content-Digest"); got != dgst {
|
||||||
|
t.Fatalf("finish upload: digest mismatch: got %q want %q", got, dgst)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// pushBlobChunked uploads a blob with POST then PATCH (body) then PUT?digest
|
||||||
|
// (empty) — the chunked flow a real docker daemon uses.
|
||||||
|
func pushBlobChunked(t *testing.T, repo, image string, blob []byte) {
|
||||||
|
t.Helper()
|
||||||
|
dgst := digestOf(blob)
|
||||||
|
|
||||||
|
resp, body := doRequest(t, http.MethodPost, api("/v2/"+repo+"/"+image+"/blobs/uploads/"), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusAccepted {
|
||||||
|
t.Fatalf("start upload: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
loc := resp.Header.Get("Location")
|
||||||
|
|
||||||
|
resp, body = doRequest(t, http.MethodPatch, baseURL()+loc, blob, "application/octet-stream")
|
||||||
|
if resp.StatusCode != http.StatusAccepted {
|
||||||
|
t.Fatalf("patch upload: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
if got := resp.Header.Get("Range"); got != fmt.Sprintf("0-%d", len(blob)-1) {
|
||||||
|
t.Fatalf("patch upload: unexpected Range %q", got)
|
||||||
|
}
|
||||||
|
loc = resp.Header.Get("Location")
|
||||||
|
|
||||||
|
resp, body = doRequest(t, http.MethodPut, baseURL()+loc+"?digest="+dgst, nil, "")
|
||||||
|
if resp.StatusCode != http.StatusCreated {
|
||||||
|
t.Fatalf("finish upload: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLocalDockerPushPull exercises a full container push and pull against a
|
||||||
|
// local docker repo using the Docker Registry HTTP API V2, the way a docker
|
||||||
|
// client would: upload the config and layer blobs, push the manifest under a
|
||||||
|
// tag, then pull the manifest and blobs back byte-identically.
|
||||||
|
func TestLocalDockerPushPull(t *testing.T) {
|
||||||
|
createRepo(t, `{"name":"docker-internal","package_type":"docker","repo_type":"local"}`)
|
||||||
|
defer deleteRepo(t, "docker-internal")
|
||||||
|
|
||||||
|
const image = "team/app"
|
||||||
|
const tag = "v1.0.0"
|
||||||
|
|
||||||
|
// /v2/ version check.
|
||||||
|
resp, _ := doRequest(t, http.MethodGet, api("/v2/"), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("/v2/ ping: status %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
config := []byte(`{"architecture":"amd64","os":"linux","config":{},"rootfs":{"type":"layers","diff_ids":["sha256:0000000000000000000000000000000000000000000000000000000000000000"]}}`)
|
||||||
|
layer := bytes.Repeat([]byte("artifactapi-layer-data-"), 4096) // ~90 KB opaque layer
|
||||||
|
|
||||||
|
configDigest := digestOf(config)
|
||||||
|
layerDigest := digestOf(layer)
|
||||||
|
|
||||||
|
// A brand-new blob should be absent (this is the client's mount check).
|
||||||
|
resp, _ = doRequest(t, http.MethodHead, api("/v2/"+"docker-internal/"+image+"/blobs/"+configDigest), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusNotFound {
|
||||||
|
t.Fatalf("pre-push blob HEAD: expected 404, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
pushBlobMonolithic(t, "docker-internal", image, config)
|
||||||
|
pushBlobChunked(t, "docker-internal", image, layer)
|
||||||
|
|
||||||
|
manifest := []byte(fmt.Sprintf(`{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":%d,"digest":%q},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":%d,"digest":%q}]}`,
|
||||||
|
len(config), configDigest, len(layer), layerDigest))
|
||||||
|
manifestDigest := digestOf(manifest)
|
||||||
|
manifestType := "application/vnd.docker.distribution.manifest.v2+json"
|
||||||
|
|
||||||
|
resp, body := doRequest(t, http.MethodPut, api("/v2/docker-internal/"+image+"/manifests/"+tag), manifest, manifestType)
|
||||||
|
if resp.StatusCode != http.StatusCreated {
|
||||||
|
t.Fatalf("push manifest: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
if got := resp.Header.Get("Docker-Content-Digest"); got != manifestDigest {
|
||||||
|
t.Fatalf("push manifest: digest %q want %q", got, manifestDigest)
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- pull back ---
|
||||||
|
|
||||||
|
// Manifest by tag.
|
||||||
|
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/manifests/"+tag), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("pull manifest by tag: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(body, manifest) {
|
||||||
|
t.Fatalf("pulled manifest bytes differ from pushed")
|
||||||
|
}
|
||||||
|
if ct := resp.Header.Get("Content-Type"); ct != manifestType {
|
||||||
|
t.Fatalf("pulled manifest content-type %q want %q", ct, manifestType)
|
||||||
|
}
|
||||||
|
if got := resp.Header.Get("Docker-Content-Digest"); got != manifestDigest {
|
||||||
|
t.Fatalf("pulled manifest digest %q want %q", got, manifestDigest)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Manifest by digest.
|
||||||
|
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/manifests/"+manifestDigest), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK || !bytes.Equal(body, manifest) {
|
||||||
|
t.Fatalf("pull manifest by digest: status %d, equal=%v", resp.StatusCode, bytes.Equal(body, manifest))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Blobs by digest.
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
digest string
|
||||||
|
want []byte
|
||||||
|
}{
|
||||||
|
{"config", configDigest, config},
|
||||||
|
{"layer", layerDigest, layer},
|
||||||
|
} {
|
||||||
|
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/blobs/"+tc.digest), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("pull %s blob: status %d", tc.name, resp.StatusCode)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(body, tc.want) {
|
||||||
|
t.Fatalf("pulled %s blob bytes differ", tc.name)
|
||||||
|
}
|
||||||
|
if got := resp.Header.Get("Docker-Content-Digest"); got != tc.digest {
|
||||||
|
t.Fatalf("pulled %s blob digest %q want %q", tc.name, got, tc.digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// tags/list reflects the pushed tag.
|
||||||
|
resp, body = doRequest(t, http.MethodGet, api("/v2/docker-internal/"+image+"/tags/list"), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("tags/list: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(body), `"`+tag+`"`) {
|
||||||
|
t.Fatalf("tags/list missing tag %q: %s", tag, body)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(body), `"docker-internal/`+image+`"`) {
|
||||||
|
t.Fatalf("tags/list wrong repository name: %s", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A now-present blob HEAD should succeed (client would skip re-upload).
|
||||||
|
resp, _ = doRequest(t, http.MethodHead, api("/v2/docker-internal/"+image+"/blobs/"+layerDigest), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("post-push blob HEAD: expected 200, got %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,6 +8,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/testsupport"
|
||||||
)
|
)
|
||||||
|
|
||||||
func uploadFile(t *testing.T, repo, filePath string, body []byte, contentType string) {
|
func uploadFile(t *testing.T, repo, filePath string, body []byte, contentType string) {
|
||||||
@@ -91,3 +93,46 @@ func TestLocalRPMRepodata(t *testing.T) {
|
|||||||
t.Fatalf("repomd.xml not a valid repodata document: %s", s)
|
t.Fatalf("repomd.xml not a valid repodata document: %s", s)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestLocalDebRepo uploads a .deb and validates that the flat apt index
|
||||||
|
// (Packages / Release) is generated automatically from the parsed control
|
||||||
|
// stanza (the deb-local analog of rpm repodata generation).
|
||||||
|
func TestLocalDebRepo(t *testing.T) {
|
||||||
|
createRepo(t, `{"name":"local-deb","package_type":"deb","repo_type":"local"}`)
|
||||||
|
defer deleteRepo(t, "local-deb")
|
||||||
|
|
||||||
|
deb := testsupport.MinimalDeb("e2e-testpkg", "1.0.0", "amd64")
|
||||||
|
uploadFile(t, "local-deb", "e2e-testpkg_1.0.0_amd64.deb", deb, "application/vnd.debian.binary-package")
|
||||||
|
|
||||||
|
// The index is generated asynchronously after upload; poll for it.
|
||||||
|
resp, body := getEventually(t, api("/api/v1/local/local-deb/Packages"), 15*time.Second)
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("Packages: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
pkgs := string(body)
|
||||||
|
for _, want := range []string{"Package: e2e-testpkg", "Version: 1.0.0", "Architecture: amd64", "Filename: pool/e2e-testpkg_1.0.0_amd64.deb", "SHA256:"} {
|
||||||
|
if !strings.Contains(pkgs, want) {
|
||||||
|
t.Fatalf("Packages missing %q:\n%s", want, pkgs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-deb/Release"), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("Release: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
rel := string(body)
|
||||||
|
for _, want := range []string{"Architectures: amd64", "SHA256:", "Packages"} {
|
||||||
|
if !strings.Contains(rel, want) {
|
||||||
|
t.Fatalf("Release missing %q:\n%s", want, rel)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The .deb downloads back byte-identical from its pool path.
|
||||||
|
resp, body = doRequest(t, http.MethodGet, api("/api/v1/local/local-deb/pool/e2e-testpkg_1.0.0_amd64.deb"), nil, "")
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
t.Fatalf("download deb: status %d: %s", resp.StatusCode, body)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(body, deb) {
|
||||||
|
t.Fatalf("deb content mismatch")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -7,12 +7,17 @@ require (
|
|||||||
github.com/charmbracelet/bubbletea v1.3.10
|
github.com/charmbracelet/bubbletea v1.3.10
|
||||||
github.com/charmbracelet/lipgloss v1.1.0
|
github.com/charmbracelet/lipgloss v1.1.0
|
||||||
github.com/go-chi/chi/v5 v5.3.0
|
github.com/go-chi/chi/v5 v5.3.0
|
||||||
|
github.com/google/uuid v1.6.0
|
||||||
github.com/jackc/pgx/v5 v5.10.0
|
github.com/jackc/pgx/v5 v5.10.0
|
||||||
|
github.com/klauspost/compress v1.19.2
|
||||||
github.com/minio/minio-go/v7 v7.2.0
|
github.com/minio/minio-go/v7 v7.2.0
|
||||||
github.com/redis/go-redis/v9 v9.20.0
|
github.com/redis/go-redis/v9 v9.20.0
|
||||||
github.com/testcontainers/testcontainers-go v0.42.0
|
github.com/testcontainers/testcontainers-go v0.42.0
|
||||||
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
|
github.com/testcontainers/testcontainers-go/modules/postgres v0.42.0
|
||||||
github.com/testcontainers/testcontainers-go/modules/redis v0.42.0
|
github.com/testcontainers/testcontainers-go/modules/redis v0.42.0
|
||||||
|
github.com/ulikunitz/xz v0.5.16
|
||||||
|
golang.org/x/crypto v0.51.0
|
||||||
|
golang.org/x/time v0.15.0
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -45,11 +50,9 @@ require (
|
|||||||
github.com/go-logr/logr v1.4.3 // indirect
|
github.com/go-logr/logr v1.4.3 // indirect
|
||||||
github.com/go-logr/stdr v1.2.2 // indirect
|
github.com/go-logr/stdr v1.2.2 // indirect
|
||||||
github.com/go-ole/go-ole v1.2.6 // indirect
|
github.com/go-ole/go-ole v1.2.6 // indirect
|
||||||
github.com/google/uuid v1.6.0 // indirect
|
|
||||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||||
github.com/klauspost/compress v1.18.6 // indirect
|
|
||||||
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
|
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
|
||||||
github.com/klauspost/crc32 v1.3.0 // indirect
|
github.com/klauspost/crc32 v1.3.0 // indirect
|
||||||
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
|
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
|
||||||
@@ -96,7 +99,6 @@ require (
|
|||||||
go.opentelemetry.io/otel/trace v1.41.0 // indirect
|
go.opentelemetry.io/otel/trace v1.41.0 // indirect
|
||||||
go.uber.org/atomic v1.11.0 // indirect
|
go.uber.org/atomic v1.11.0 // indirect
|
||||||
go.yaml.in/yaml/v3 v3.0.4 // indirect
|
go.yaml.in/yaml/v3 v3.0.4 // indirect
|
||||||
golang.org/x/crypto v0.51.0 // indirect
|
|
||||||
golang.org/x/net v0.53.0 // indirect
|
golang.org/x/net v0.53.0 // indirect
|
||||||
golang.org/x/sync v0.20.0 // indirect
|
golang.org/x/sync v0.20.0 // indirect
|
||||||
golang.org/x/sys v0.44.0 // indirect
|
golang.org/x/sys v0.44.0 // indirect
|
||||||
|
|||||||
@@ -85,8 +85,8 @@ github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
|
|||||||
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||||
github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao=
|
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
|
||||||
github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||||
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
|
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
|
||||||
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||||
@@ -189,6 +189,8 @@ github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYI
|
|||||||
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
|
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
|
||||||
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
|
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
|
||||||
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
|
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
|
||||||
|
github.com/ulikunitz/xz v0.5.16 h1:ld6NyySjx5lowVKwJvMRLnW5nxKX/xnpSiFYZ/Lxur0=
|
||||||
|
github.com/ulikunitz/xz v0.5.16/go.mod h1:H9Rt/W6/Qj27PGauhQc6nfCDy7vHpzsOThBSaYDoEhw=
|
||||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
|
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
|
||||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
|
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
|
||||||
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
|
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
|
||||||
@@ -234,6 +236,8 @@ golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4=
|
|||||||
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
|
golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk=
|
||||||
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
|
||||||
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
|
||||||
|
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
||||||
|
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
||||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||||
|
|||||||
@@ -0,0 +1,301 @@
|
|||||||
|
// Package terraform serves local terraform repos as a real Terraform provider
|
||||||
|
// registry: service discovery, version listing, and GPG-signed downloads, so
|
||||||
|
// `terraform init` installs from a bare source address with no client config.
|
||||||
|
package terraform
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"path"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/database"
|
||||||
|
tfprov "git.unkin.net/unkin/artifactapi/internal/provider/terraform"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/tfsign"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ProvidersV1Path is the base the service-discovery document advertises (Terraform
|
||||||
|
// appends "{namespace}/{type}/versions" etc). MountPath is the same prefix without
|
||||||
|
// the trailing slash, for chi.Mount.
|
||||||
|
const (
|
||||||
|
ProvidersV1Path = "/terraform/v1/providers/"
|
||||||
|
MountPath = "/terraform/v1/providers"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Handler struct {
|
||||||
|
db *database.DB
|
||||||
|
signer *tfsign.Signer
|
||||||
|
protocols []string
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewHandler(db *database.DB, signer *tfsign.Signer, protocols string) *Handler {
|
||||||
|
var protos []string
|
||||||
|
for _, p := range strings.Split(protocols, ",") {
|
||||||
|
if p = strings.TrimSpace(p); p != "" {
|
||||||
|
protos = append(protos, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(protos) == 0 {
|
||||||
|
protos = []string{"5.0", "6.0"}
|
||||||
|
}
|
||||||
|
return &Handler{db: db, signer: signer, protocols: protos}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Enabled reports whether a signing key is configured. Without one the registry
|
||||||
|
// cannot produce the signed SHA256SUMS the protocol requires, so it stays off.
|
||||||
|
func (h *Handler) Enabled() bool { return h.signer != nil }
|
||||||
|
|
||||||
|
func (h *Handler) Routes() chi.Router {
|
||||||
|
r := chi.NewRouter()
|
||||||
|
r.Get("/{namespace}/{type}/versions", h.versions)
|
||||||
|
r.Get("/{namespace}/{type}/{version}/download/{os}/{arch}", h.download)
|
||||||
|
r.Get("/{namespace}/{type}/{version}/sha256sums", h.sha256sums)
|
||||||
|
r.Get("/{namespace}/{type}/{version}/sha256sums.sig", h.sha256sumsSig)
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServiceDiscovery answers /.well-known/terraform.json, pointing Terraform at the
|
||||||
|
// providers.v1 protocol base.
|
||||||
|
func (h *Handler) ServiceDiscovery(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !h.Enabled() {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, map[string]string{"providers.v1": ProvidersV1Path})
|
||||||
|
}
|
||||||
|
|
||||||
|
// providerFile is one resolved platform artifact within a repo.
|
||||||
|
type providerFile struct {
|
||||||
|
version string
|
||||||
|
os string
|
||||||
|
arch string
|
||||||
|
filePath string // path within the repo, e.g. unkin/artifactapi/...zip
|
||||||
|
sha256 string // hex, no "sha256:" prefix
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolve finds every provider zip of the given type in the repo (namespace).
|
||||||
|
// The Terraform source namespace maps to the artifactapi repo name; the provider
|
||||||
|
// is matched by type across whatever in-repo folder it was uploaded under.
|
||||||
|
func (h *Handler) resolve(r *http.Request, namespace, typeName string) ([]providerFile, error) {
|
||||||
|
remote, err := h.db.GetRemote(r.Context(), namespace)
|
||||||
|
if err != nil || remote.PackageType != models.PackageTerraform {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
rows, err := h.db.ListLocalFiles(r.Context(), namespace, 10000, 0)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var out []providerFile
|
||||||
|
for _, row := range rows {
|
||||||
|
parsed := tfprov.ParseProviderZip(path.Base(row.FilePath))
|
||||||
|
if !parsed.Ok || parsed.Type != typeName {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, providerFile{
|
||||||
|
version: parsed.Version,
|
||||||
|
os: parsed.OS,
|
||||||
|
arch: parsed.Arch,
|
||||||
|
filePath: row.FilePath,
|
||||||
|
sha256: strings.TrimPrefix(row.ContentHash, "sha256:"),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *Handler) versions(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !h.Enabled() {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
namespace := chi.URLParam(r, "namespace")
|
||||||
|
typeName := chi.URLParam(r, "type")
|
||||||
|
|
||||||
|
files, err := h.resolve(r, namespace, typeName)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if len(files) == 0 {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Group platforms by version, de-duplicated and stably ordered.
|
||||||
|
type platform struct {
|
||||||
|
OS string `json:"os"`
|
||||||
|
Arch string `json:"arch"`
|
||||||
|
}
|
||||||
|
platforms := map[string]map[string]platform{}
|
||||||
|
for _, f := range files {
|
||||||
|
if platforms[f.version] == nil {
|
||||||
|
platforms[f.version] = map[string]platform{}
|
||||||
|
}
|
||||||
|
platforms[f.version][f.os+"_"+f.arch] = platform{OS: f.os, Arch: f.arch}
|
||||||
|
}
|
||||||
|
|
||||||
|
type versionEntry struct {
|
||||||
|
Version string `json:"version"`
|
||||||
|
Protocols []string `json:"protocols"`
|
||||||
|
Platforms []platform `json:"platforms"`
|
||||||
|
}
|
||||||
|
out := struct {
|
||||||
|
Versions []versionEntry `json:"versions"`
|
||||||
|
}{}
|
||||||
|
for version, plats := range platforms {
|
||||||
|
entry := versionEntry{Version: version, Protocols: h.protocols}
|
||||||
|
for _, p := range plats {
|
||||||
|
entry.Platforms = append(entry.Platforms, p)
|
||||||
|
}
|
||||||
|
sort.Slice(entry.Platforms, func(i, j int) bool {
|
||||||
|
return entry.Platforms[i].OS+entry.Platforms[i].Arch < entry.Platforms[j].OS+entry.Platforms[j].Arch
|
||||||
|
})
|
||||||
|
out.Versions = append(out.Versions, entry)
|
||||||
|
}
|
||||||
|
sort.Slice(out.Versions, func(i, j int) bool { return out.Versions[i].Version < out.Versions[j].Version })
|
||||||
|
|
||||||
|
writeJSON(w, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *Handler) download(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !h.Enabled() {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
namespace := chi.URLParam(r, "namespace")
|
||||||
|
typeName := chi.URLParam(r, "type")
|
||||||
|
version := chi.URLParam(r, "version")
|
||||||
|
osName := chi.URLParam(r, "os")
|
||||||
|
arch := chi.URLParam(r, "arch")
|
||||||
|
|
||||||
|
files, err := h.resolve(r, namespace, typeName)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
var match *providerFile
|
||||||
|
for i := range files {
|
||||||
|
if files[i].version == version && files[i].os == osName && files[i].arch == arch {
|
||||||
|
match = &files[i]
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if match == nil {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
base := baseURL(r)
|
||||||
|
verBase := fmt.Sprintf("%s%s/%s/%s", base+ProvidersV1Path, namespace, typeName, version)
|
||||||
|
|
||||||
|
type gpgKey struct {
|
||||||
|
KeyID string `json:"key_id"`
|
||||||
|
ASCIIArmor string `json:"ascii_armor"`
|
||||||
|
}
|
||||||
|
resp := struct {
|
||||||
|
Protocols []string `json:"protocols"`
|
||||||
|
OS string `json:"os"`
|
||||||
|
Arch string `json:"arch"`
|
||||||
|
Filename string `json:"filename"`
|
||||||
|
DownloadURL string `json:"download_url"`
|
||||||
|
SHASumsURL string `json:"shasums_url"`
|
||||||
|
SHASumsSignatureURL string `json:"shasums_signature_url"`
|
||||||
|
SHASum string `json:"shasum"`
|
||||||
|
SigningKeys struct {
|
||||||
|
GPGPublicKeys []gpgKey `json:"gpg_public_keys"`
|
||||||
|
} `json:"signing_keys"`
|
||||||
|
}{
|
||||||
|
Protocols: h.protocols,
|
||||||
|
OS: match.os,
|
||||||
|
Arch: match.arch,
|
||||||
|
Filename: path.Base(match.filePath),
|
||||||
|
DownloadURL: fmt.Sprintf("%s/api/v1/local/%s/%s", base, namespace, match.filePath),
|
||||||
|
SHASumsURL: verBase + "/sha256sums",
|
||||||
|
SHASumsSignatureURL: verBase + "/sha256sums.sig",
|
||||||
|
SHASum: match.sha256,
|
||||||
|
}
|
||||||
|
resp.SigningKeys.GPGPublicKeys = []gpgKey{{
|
||||||
|
KeyID: h.signer.KeyID(),
|
||||||
|
ASCIIArmor: h.signer.PublicKeyArmor(),
|
||||||
|
}}
|
||||||
|
|
||||||
|
writeJSON(w, resp)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *Handler) sha256sums(w http.ResponseWriter, r *http.Request) {
|
||||||
|
sums, ok := h.buildSums(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||||
|
w.Write(sums)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *Handler) sha256sumsSig(w http.ResponseWriter, r *http.Request) {
|
||||||
|
sums, ok := h.buildSums(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
sig, err := h.signer.Sign(sums)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/octet-stream")
|
||||||
|
w.Write(sig)
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildSums renders the SHA256SUMS body for one version: one "<hex> <filename>"
|
||||||
|
// line per platform zip, sorted by filename so the signed bytes are stable.
|
||||||
|
func (h *Handler) buildSums(w http.ResponseWriter, r *http.Request) ([]byte, bool) {
|
||||||
|
if !h.Enabled() {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
namespace := chi.URLParam(r, "namespace")
|
||||||
|
typeName := chi.URLParam(r, "type")
|
||||||
|
version := chi.URLParam(r, "version")
|
||||||
|
|
||||||
|
files, err := h.resolve(r, namespace, typeName)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
|
||||||
|
var lines []string
|
||||||
|
for _, f := range files {
|
||||||
|
if f.version != version {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
lines = append(lines, fmt.Sprintf("%s %s", f.sha256, path.Base(f.filePath)))
|
||||||
|
}
|
||||||
|
if len(lines) == 0 {
|
||||||
|
http.NotFound(w, r)
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
sort.Strings(lines)
|
||||||
|
return []byte(strings.Join(lines, "\n") + "\n"), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSON(w http.ResponseWriter, v any) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
json.NewEncoder(w).Encode(v)
|
||||||
|
}
|
||||||
|
|
||||||
|
func baseURL(r *http.Request) string {
|
||||||
|
scheme := "http"
|
||||||
|
if r.TLS != nil {
|
||||||
|
scheme = "https"
|
||||||
|
}
|
||||||
|
if fwd := r.Header.Get("X-Forwarded-Proto"); fwd != "" {
|
||||||
|
scheme = fwd
|
||||||
|
}
|
||||||
|
return scheme + "://" + r.Host
|
||||||
|
}
|
||||||
@@ -0,0 +1,186 @@
|
|||||||
|
package terraform
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
"golang.org/x/crypto/openpgp"
|
||||||
|
"golang.org/x/crypto/openpgp/armor"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/database"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/testsupport"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/tfsign"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
var testDSN string
|
||||||
|
|
||||||
|
func TestMain(m *testing.M) {
|
||||||
|
ctx := context.Background()
|
||||||
|
dsn, terminate, err := testsupport.StartPostgres(ctx)
|
||||||
|
if err != nil {
|
||||||
|
os.Exit(m.Run())
|
||||||
|
}
|
||||||
|
testDSN = dsn
|
||||||
|
code := m.Run()
|
||||||
|
terminate()
|
||||||
|
os.Exit(code)
|
||||||
|
}
|
||||||
|
|
||||||
|
// testSigner writes a throwaway armored key and loads it.
|
||||||
|
func testSigner(t *testing.T) *tfsign.Signer {
|
||||||
|
t.Helper()
|
||||||
|
e, err := openpgp.NewEntity("artifactapi test", "tf", "tf@example.com", nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var buf bytes.Buffer
|
||||||
|
w, _ := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
|
||||||
|
if err := e.SerializePrivate(w, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
w.Close()
|
||||||
|
p := filepath.Join(t.TempDir(), "private-key.asc")
|
||||||
|
if err := os.WriteFile(p, buf.Bytes(), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s, err := tfsign.Load(p, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProviderRegistryFlow(t *testing.T) {
|
||||||
|
if testDSN == "" {
|
||||||
|
t.Skip("Docker unavailable")
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
db, err := database.New(testDSN)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
|
||||||
|
const repo = "tf-reg" // Terraform namespace == repo name
|
||||||
|
const filePath = "unkin/artifactapi/terraform-provider-artifactapi_1.2.3_linux_amd64.zip"
|
||||||
|
const hash = "sha256:983cdb25cb7b976538e4334d26e52dee5f44749b9be1500c760cf5cf66be659b"
|
||||||
|
const wantSha = "983cdb25cb7b976538e4334d26e52dee5f44749b9be1500c760cf5cf66be659b"
|
||||||
|
|
||||||
|
if err := db.CreateRemote(ctx, &models.Remote{Name: repo, PackageType: models.PackageTerraform, RepoType: models.RepoTypeLocal}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.UpsertBlob(ctx, hash, "blobs/98/3c", 6381007, "application/zip"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.CreateLocalFile(ctx, repo, filePath, hash); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
signer := testSigner(t)
|
||||||
|
h := NewHandler(db, signer, "5.0,6.0")
|
||||||
|
router := chi.NewRouter()
|
||||||
|
router.Get("/.well-known/terraform.json", h.ServiceDiscovery)
|
||||||
|
router.Mount(MountPath, h.Routes())
|
||||||
|
|
||||||
|
get := func(p string) *httptest.ResponseRecorder {
|
||||||
|
req := httptest.NewRequest("GET", p, nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(w, req)
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
// Service discovery.
|
||||||
|
w := get("/.well-known/terraform.json")
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("discovery = %d", w.Code)
|
||||||
|
}
|
||||||
|
var disc map[string]string
|
||||||
|
json.Unmarshal(w.Body.Bytes(), &disc)
|
||||||
|
if disc["providers.v1"] != ProvidersV1Path {
|
||||||
|
t.Errorf("providers.v1 = %q", disc["providers.v1"])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Versions.
|
||||||
|
w = get("/terraform/v1/providers/tf-reg/artifactapi/versions")
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("versions = %d %s", w.Code, w.Body)
|
||||||
|
}
|
||||||
|
var vresp struct {
|
||||||
|
Versions []struct {
|
||||||
|
Version string `json:"version"`
|
||||||
|
Protocols []string `json:"protocols"`
|
||||||
|
Platforms []map[string]string `json:"platforms"`
|
||||||
|
} `json:"versions"`
|
||||||
|
}
|
||||||
|
json.Unmarshal(w.Body.Bytes(), &vresp)
|
||||||
|
if len(vresp.Versions) != 1 || vresp.Versions[0].Version != "1.2.3" {
|
||||||
|
t.Fatalf("unexpected versions: %+v", vresp)
|
||||||
|
}
|
||||||
|
if len(vresp.Versions[0].Platforms) != 1 || vresp.Versions[0].Platforms[0]["os"] != "linux" {
|
||||||
|
t.Fatalf("unexpected platforms: %+v", vresp.Versions[0].Platforms)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Download.
|
||||||
|
w = get("/terraform/v1/providers/tf-reg/artifactapi/1.2.3/download/linux/amd64")
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("download = %d %s", w.Code, w.Body)
|
||||||
|
}
|
||||||
|
var dl struct {
|
||||||
|
Filename string `json:"filename"`
|
||||||
|
DownloadURL string `json:"download_url"`
|
||||||
|
SHASumsURL string `json:"shasums_url"`
|
||||||
|
SHASumsSignatureURL string `json:"shasums_signature_url"`
|
||||||
|
SHASum string `json:"shasum"`
|
||||||
|
SigningKeys struct {
|
||||||
|
GPGPublicKeys []struct {
|
||||||
|
KeyID string `json:"key_id"`
|
||||||
|
ASCIIArmor string `json:"ascii_armor"`
|
||||||
|
} `json:"gpg_public_keys"`
|
||||||
|
} `json:"signing_keys"`
|
||||||
|
}
|
||||||
|
json.Unmarshal(w.Body.Bytes(), &dl)
|
||||||
|
if dl.SHASum != wantSha {
|
||||||
|
t.Errorf("shasum = %q", dl.SHASum)
|
||||||
|
}
|
||||||
|
wantURL := "http://example.com/api/v1/local/tf-reg/" + filePath
|
||||||
|
if dl.DownloadURL != wantURL {
|
||||||
|
t.Errorf("download_url = %q, want %q", dl.DownloadURL, wantURL)
|
||||||
|
}
|
||||||
|
if len(dl.SigningKeys.GPGPublicKeys) != 1 || dl.SigningKeys.GPGPublicKeys[0].KeyID != signer.KeyID() {
|
||||||
|
t.Errorf("signing key mismatch: %+v", dl.SigningKeys)
|
||||||
|
}
|
||||||
|
|
||||||
|
// SHA256SUMS + signature verify against the advertised key.
|
||||||
|
sums := get("/terraform/v1/providers/tf-reg/artifactapi/1.2.3/sha256sums")
|
||||||
|
wantLine := wantSha + " terraform-provider-artifactapi_1.2.3_linux_amd64.zip\n"
|
||||||
|
if sums.Body.String() != wantLine {
|
||||||
|
t.Errorf("sha256sums = %q, want %q", sums.Body.String(), wantLine)
|
||||||
|
}
|
||||||
|
sig := get("/terraform/v1/providers/tf-reg/artifactapi/1.2.3/sha256sums.sig")
|
||||||
|
keyring, err := openpgp.ReadArmoredKeyRing(bytes.NewReader([]byte(dl.SigningKeys.GPGPublicKeys[0].ASCIIArmor)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := openpgp.CheckDetachedSignature(keyring, bytes.NewReader(sums.Body.Bytes()), bytes.NewReader(sig.Body.Bytes())); err != nil {
|
||||||
|
t.Errorf("sha256sums.sig did not verify: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegistryDisabledWithoutSigner(t *testing.T) {
|
||||||
|
h := NewHandler(nil, nil, "")
|
||||||
|
router := chi.NewRouter()
|
||||||
|
router.Get("/.well-known/terraform.json", h.ServiceDiscovery)
|
||||||
|
req := httptest.NewRequest("GET", "/.well-known/terraform.json", nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
router.ServeHTTP(w, req)
|
||||||
|
if w.Code != 404 {
|
||||||
|
t.Errorf("disabled discovery = %d, want 404", w.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,486 @@
|
|||||||
|
package v1
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/database"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/storage"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// This file implements the write half of the Docker Registry HTTP API V2 for
|
||||||
|
// *local* docker repositories, so a `docker push` / `docker pull` against
|
||||||
|
// artifactapi treats a local docker repo as a genuine registry (matching the
|
||||||
|
// project's "local repos are the real thing" principle) rather than a mirror.
|
||||||
|
//
|
||||||
|
// Storage reuses the existing content-addressable primitives:
|
||||||
|
// - blob and manifest bytes are stored via the CAS (deduplicated by sha256)
|
||||||
|
// - a local_files row per (repo, "<image>/blobs/<digest>") and
|
||||||
|
// (repo, "<image>/manifests/<ref>") keeps the blob referenced so the GC
|
||||||
|
// does not reap it, and lets pulls resolve a reference back to a blob.
|
||||||
|
// Tags are mutable references (UpsertLocalFile); digests and blobs are
|
||||||
|
// immutable (CreateLocalFile, tolerating an already-exists on re-push).
|
||||||
|
|
||||||
|
const dockerAPIVersionHeader = "registry/2.0"
|
||||||
|
|
||||||
|
// Chunked blob uploads are staged in object storage under uploads/<uuid> rather
|
||||||
|
// than in process memory, so the POST / PATCH / PUT of a single push can each be
|
||||||
|
// served by a different replica (the API runs with minReplicas>1 and no session
|
||||||
|
// affinity). The upload UUID travels in the Location URL handed back to the
|
||||||
|
// client, so any replica reconstructs the staging key with no shared in-process
|
||||||
|
// state. Abandoned stages are dropped by the GC's uploads sweep.
|
||||||
|
func uploadKey(id string) string { return "uploads/" + id }
|
||||||
|
|
||||||
|
var errUploadUnknown = errors.New("unknown upload")
|
||||||
|
|
||||||
|
// appendUpload appends a chunk to the staged upload object and returns the new
|
||||||
|
// total size. The staged bytes live entirely in object storage (download,
|
||||||
|
// append to a per-request temp file, re-upload), which keeps the session state
|
||||||
|
// replica-independent. Docker sends the whole layer in one PATCH, so this is a
|
||||||
|
// single append in the common case.
|
||||||
|
func (h *ProxyHandler) appendUpload(ctx context.Context, id string, chunk io.Reader) (int64, error) {
|
||||||
|
key := uploadKey(id)
|
||||||
|
reader, info, err := h.store.Download(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return 0, errUploadUnknown
|
||||||
|
}
|
||||||
|
|
||||||
|
tmp, err := os.CreateTemp("", "docker-upload-*")
|
||||||
|
if err != nil {
|
||||||
|
reader.Close()
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
defer os.Remove(tmp.Name())
|
||||||
|
defer tmp.Close()
|
||||||
|
|
||||||
|
if _, err := io.Copy(tmp, reader); err != nil {
|
||||||
|
reader.Close()
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
reader.Close()
|
||||||
|
|
||||||
|
n, err := io.Copy(tmp, chunk)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
size := info.Size + n
|
||||||
|
if _, err := tmp.Seek(0, io.SeekStart); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if err := h.store.Upload(ctx, key, tmp, size, "application/octet-stream"); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
return size, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerReq is a parsed /v2/<remote>/<image>/... request. kind is one of
|
||||||
|
// "manifest", "blob", "upload", "tags".
|
||||||
|
type dockerReq struct {
|
||||||
|
image string
|
||||||
|
kind string
|
||||||
|
ref string // tag, digest, or upload uuid depending on kind
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseDockerPath splits the chi "*" remainder (everything after the repo name)
|
||||||
|
// into the image name and the registry operation. The image name may itself
|
||||||
|
// contain slashes, so operations are located by their well-known infixes.
|
||||||
|
func parseDockerPath(rest string) (dockerReq, bool) {
|
||||||
|
rest = strings.TrimPrefix(rest, "/")
|
||||||
|
switch {
|
||||||
|
case strings.HasSuffix(rest, "/tags/list"):
|
||||||
|
return dockerReq{image: strings.TrimSuffix(rest, "/tags/list"), kind: "tags"}, true
|
||||||
|
case rest == "tags/list":
|
||||||
|
return dockerReq{}, false // no image
|
||||||
|
}
|
||||||
|
if i := strings.Index(rest, "/blobs/uploads"); i >= 0 {
|
||||||
|
image := rest[:i]
|
||||||
|
ref := strings.TrimPrefix(rest[i+len("/blobs/uploads"):], "/")
|
||||||
|
return dockerReq{image: image, kind: "upload", ref: ref}, image != ""
|
||||||
|
}
|
||||||
|
if i := strings.LastIndex(rest, "/manifests/"); i >= 0 {
|
||||||
|
return dockerReq{image: rest[:i], kind: "manifest", ref: rest[i+len("/manifests/"):]}, true
|
||||||
|
}
|
||||||
|
if i := strings.LastIndex(rest, "/blobs/"); i >= 0 {
|
||||||
|
return dockerReq{image: rest[:i], kind: "blob", ref: rest[i+len("/blobs/"):]}, true
|
||||||
|
}
|
||||||
|
return dockerReq{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
func isDigest(ref string) bool { return strings.HasPrefix(ref, "sha256:") }
|
||||||
|
|
||||||
|
// localDockerRemote returns the repo if name is a local docker repository.
|
||||||
|
func (h *ProxyHandler) localDockerRemote(r *http.Request, name string) (*models.Remote, bool) {
|
||||||
|
remote, err := h.db.GetRemote(r.Context(), name)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
return remote, remote.RepoType == models.RepoTypeLocal && remote.PackageType == models.PackageDocker
|
||||||
|
}
|
||||||
|
|
||||||
|
func dockerError(w http.ResponseWriter, status int, code, msg string) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(status)
|
||||||
|
fmt.Fprintf(w, `{"errors":[{"code":%q,"message":%q}]}`, code, msg)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerGet dispatches a registry GET to the local handler for local docker
|
||||||
|
// repos and falls through to the upstream proxy for everything else.
|
||||||
|
func (h *ProxyHandler) dockerGet(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := chi.URLParam(r, "remoteName")
|
||||||
|
if remote, ok := h.localDockerRemote(r, name); ok {
|
||||||
|
h.dockerLocalGet(w, r, remote, false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.handleProxy(w, r)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ProxyHandler) dockerHead(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := chi.URLParam(r, "remoteName")
|
||||||
|
if remote, ok := h.localDockerRemote(r, name); ok {
|
||||||
|
h.dockerLocalGet(w, r, remote, true)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.handleProxyHead(w, r)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ProxyHandler) dockerPost(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := chi.URLParam(r, "remoteName")
|
||||||
|
remote, ok := h.localDockerRemote(r, name)
|
||||||
|
if !ok {
|
||||||
|
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "push is only supported for local docker repositories")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.dockerStartUpload(w, r, remote)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ProxyHandler) dockerPatch(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := chi.URLParam(r, "remoteName")
|
||||||
|
remote, ok := h.localDockerRemote(r, name)
|
||||||
|
if !ok {
|
||||||
|
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "push is only supported for local docker repositories")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
h.dockerPatchUpload(w, r, remote)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ProxyHandler) dockerPut(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := chi.URLParam(r, "remoteName")
|
||||||
|
remote, ok := h.localDockerRemote(r, name)
|
||||||
|
if !ok {
|
||||||
|
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "push is only supported for local docker repositories")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
req, ok := parseDockerPath(chi.URLParam(r, "*"))
|
||||||
|
if !ok {
|
||||||
|
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
switch req.kind {
|
||||||
|
case "upload":
|
||||||
|
h.dockerFinishUpload(w, r, remote, req)
|
||||||
|
case "manifest":
|
||||||
|
h.dockerPutManifest(w, r, remote, req)
|
||||||
|
default:
|
||||||
|
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "PUT not supported for this path")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ProxyHandler) dockerDelete(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := chi.URLParam(r, "remoteName")
|
||||||
|
remote, ok := h.localDockerRemote(r, name)
|
||||||
|
if !ok {
|
||||||
|
dockerError(w, http.StatusMethodNotAllowed, "UNSUPPORTED", "delete is only supported for local docker repositories")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
req, ok := parseDockerPath(chi.URLParam(r, "*"))
|
||||||
|
if !ok {
|
||||||
|
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Cancel an in-progress upload: drop its staging object.
|
||||||
|
if req.kind == "upload" && req.ref != "" {
|
||||||
|
_ = h.store.Delete(r.Context(), uploadKey(req.ref))
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if req.kind != "manifest" && req.kind != "blob" {
|
||||||
|
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
filePath := req.image + "/" + req.kind + "s/" + req.ref
|
||||||
|
if err := h.db.DeleteLocalFile(r.Context(), remote.Name, filePath); err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(http.StatusAccepted)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerLocalGet serves manifest / blob / tags-list reads for a local repo.
|
||||||
|
func (h *ProxyHandler) dockerLocalGet(w http.ResponseWriter, r *http.Request, remote *models.Remote, head bool) {
|
||||||
|
req, ok := parseDockerPath(chi.URLParam(r, "*"))
|
||||||
|
if !ok {
|
||||||
|
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
switch req.kind {
|
||||||
|
case "tags":
|
||||||
|
h.dockerTagsList(w, r, remote, req.image)
|
||||||
|
case "manifest":
|
||||||
|
h.dockerServeRef(w, r, remote, req.image+"/manifests/"+req.ref, head, true)
|
||||||
|
case "blob":
|
||||||
|
h.dockerServeRef(w, r, remote, req.image+"/blobs/"+req.ref, head, false)
|
||||||
|
default:
|
||||||
|
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerServeRef streams the blob backing a local_files path. isManifest
|
||||||
|
// controls only the default content type; the stored blob content type wins.
|
||||||
|
func (h *ProxyHandler) dockerServeRef(w http.ResponseWriter, r *http.Request, remote *models.Remote, filePath string, head, isManifest bool) {
|
||||||
|
file, err := h.db.GetLocalFile(r.Context(), remote.Name, filePath)
|
||||||
|
if err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if file == nil {
|
||||||
|
code := "BLOB_UNKNOWN"
|
||||||
|
if isManifest {
|
||||||
|
code = "MANIFEST_UNKNOWN"
|
||||||
|
}
|
||||||
|
dockerError(w, http.StatusNotFound, code, "not found")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
s3Key := storage.BlobKey(file.ContentHash[len("sha256:"):])
|
||||||
|
reader, info, err := h.store.Download(r.Context(), s3Key)
|
||||||
|
if err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer reader.Close()
|
||||||
|
|
||||||
|
contentType := info.ContentType
|
||||||
|
if contentType == "" {
|
||||||
|
if isManifest {
|
||||||
|
contentType = "application/vnd.docker.distribution.manifest.v2+json"
|
||||||
|
} else {
|
||||||
|
contentType = "application/octet-stream"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", contentType)
|
||||||
|
w.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size))
|
||||||
|
w.Header().Set("Docker-Content-Digest", file.ContentHash)
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.Header().Set("X-Artifact-Source", "local")
|
||||||
|
if head {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
io.Copy(w, reader)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ProxyHandler) dockerTagsList(w http.ResponseWriter, r *http.Request, remote *models.Remote, image string) {
|
||||||
|
prefix := image + "/manifests/"
|
||||||
|
files, err := h.db.ListLocalFilesByPrefix(r.Context(), remote.Name, prefix)
|
||||||
|
if err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
tags := []string{}
|
||||||
|
for _, f := range files {
|
||||||
|
ref := strings.TrimPrefix(f.FilePath, prefix)
|
||||||
|
if ref == "" || isDigest(ref) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
tags = append(tags, ref)
|
||||||
|
}
|
||||||
|
sort.Strings(tags)
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
fmt.Fprintf(w, `{"name":%q,"tags":`, remote.Name+"/"+image)
|
||||||
|
writeJSONStringList(w, tags)
|
||||||
|
fmt.Fprint(w, "}")
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSONStringList(w io.Writer, items []string) {
|
||||||
|
fmt.Fprint(w, "[")
|
||||||
|
for i, s := range items {
|
||||||
|
if i > 0 {
|
||||||
|
fmt.Fprint(w, ",")
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "%q", s)
|
||||||
|
}
|
||||||
|
fmt.Fprint(w, "]")
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerStartUpload begins a blob upload. It honours a monolithic
|
||||||
|
// POST?digest=... (blob in the POST body) and otherwise opens a chunked
|
||||||
|
// session, returning its Location for the client's PATCH/PUT.
|
||||||
|
func (h *ProxyHandler) dockerStartUpload(w http.ResponseWriter, r *http.Request, remote *models.Remote) {
|
||||||
|
req, ok := parseDockerPath(chi.URLParam(r, "*"))
|
||||||
|
if !ok || req.kind != "upload" {
|
||||||
|
dockerError(w, http.StatusNotFound, "NAME_UNKNOWN", "unrecognised registry path")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if digest := r.URL.Query().Get("digest"); digest != "" {
|
||||||
|
h.dockerCommitBlob(w, r, remote, req.image, digest, r.Body)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stage an empty object keyed by the upload UUID; PATCH/PUT append to it.
|
||||||
|
id := uuid.NewString()
|
||||||
|
if err := h.store.Upload(r.Context(), uploadKey(id), bytes.NewReader(nil), 0, "application/octet-stream"); err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
loc := fmt.Sprintf("/v2/%s/%s/blobs/uploads/%s", remote.Name, req.image, id)
|
||||||
|
w.Header().Set("Location", loc)
|
||||||
|
w.Header().Set("Docker-Upload-UUID", id)
|
||||||
|
w.Header().Set("Range", "0-0")
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(http.StatusAccepted)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ProxyHandler) dockerPatchUpload(w http.ResponseWriter, r *http.Request, remote *models.Remote) {
|
||||||
|
req, ok := parseDockerPath(chi.URLParam(r, "*"))
|
||||||
|
if !ok || req.kind != "upload" || req.ref == "" {
|
||||||
|
dockerError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "unknown upload")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
size, err := h.appendUpload(r.Context(), req.ref, r.Body)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, errUploadUnknown) {
|
||||||
|
dockerError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "unknown upload")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
loc := fmt.Sprintf("/v2/%s/%s/blobs/uploads/%s", remote.Name, req.image, req.ref)
|
||||||
|
w.Header().Set("Location", loc)
|
||||||
|
w.Header().Set("Docker-Upload-UUID", req.ref)
|
||||||
|
w.Header().Set("Range", fmt.Sprintf("0-%d", size-1))
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(http.StatusAccepted)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerFinishUpload completes a chunked upload: appends any final PUT body,
|
||||||
|
// stores the assembled blob, and verifies its digest.
|
||||||
|
func (h *ProxyHandler) dockerFinishUpload(w http.ResponseWriter, r *http.Request, remote *models.Remote, req dockerReq) {
|
||||||
|
digest := r.URL.Query().Get("digest")
|
||||||
|
if digest == "" {
|
||||||
|
dockerError(w, http.StatusBadRequest, "DIGEST_INVALID", "digest query parameter required")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if req.ref == "" {
|
||||||
|
// Monolithic PUT with no prior session: body is the whole blob.
|
||||||
|
h.dockerCommitBlob(w, r, remote, req.image, digest, r.Body)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
key := uploadKey(req.ref)
|
||||||
|
reader, _, err := h.store.Download(r.Context(), key)
|
||||||
|
if err != nil {
|
||||||
|
dockerError(w, http.StatusNotFound, "BLOB_UPLOAD_UNKNOWN", "unknown upload")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer reader.Close()
|
||||||
|
// Drop the staging object once we're done, regardless of outcome; a fresh
|
||||||
|
// context so cleanup still runs if the client disconnects.
|
||||||
|
defer h.store.Delete(context.Background(), key)
|
||||||
|
|
||||||
|
// Stream the staged bytes plus any trailing PUT body through the CAS in one
|
||||||
|
// pass — no extra round trip to re-assemble.
|
||||||
|
combined := io.MultiReader(reader, r.Body)
|
||||||
|
h.dockerCommitBlob(w, r, remote, req.image, digest, combined)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerCommitBlob stores blob bytes through the CAS, verifies the client's
|
||||||
|
// declared digest, and records the per-image local_files reference.
|
||||||
|
func (h *ProxyHandler) dockerCommitBlob(w http.ResponseWriter, r *http.Request, remote *models.Remote, image, digest string, body io.Reader) {
|
||||||
|
result, err := h.cas.Store(r.Context(), body, "application/octet-stream")
|
||||||
|
if err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", fmt.Sprintf("store failed: %v", err))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if result.ContentHash != digest {
|
||||||
|
dockerError(w, http.StatusBadRequest, "DIGEST_INVALID", fmt.Sprintf("digest mismatch: got %s, declared %s", result.ContentHash, digest))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.db.UpsertBlob(r.Context(), result.ContentHash, result.S3Key, result.SizeBytes, "application/octet-stream"); err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.db.CreateLocalFile(r.Context(), remote.Name, image+"/blobs/"+digest, result.ContentHash); err != nil && !errors.Is(err, database.ErrAlreadyExists) {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Location", fmt.Sprintf("/v2/%s/%s/blobs/%s", remote.Name, image, digest))
|
||||||
|
w.Header().Set("Docker-Content-Digest", digest)
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(http.StatusCreated)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dockerPutManifest stores a manifest and points its reference (tag or digest)
|
||||||
|
// at it. Tags are mutable so a re-push moves the tag; digests are immutable.
|
||||||
|
func (h *ProxyHandler) dockerPutManifest(w http.ResponseWriter, r *http.Request, remote *models.Remote, req dockerReq) {
|
||||||
|
body, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
contentType := r.Header.Get("Content-Type")
|
||||||
|
if contentType == "" {
|
||||||
|
contentType = "application/vnd.docker.distribution.manifest.v2+json"
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(body)
|
||||||
|
digest := "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
|
||||||
|
result, err := h.cas.Store(r.Context(), strings.NewReader(string(body)), contentType)
|
||||||
|
if err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", fmt.Sprintf("store failed: %v", err))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.db.UpsertBlob(r.Context(), result.ContentHash, result.S3Key, result.SizeBytes, contentType); err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Always addressable by digest (immutable).
|
||||||
|
if err := h.db.CreateLocalFile(r.Context(), remote.Name, req.image+"/manifests/"+digest, result.ContentHash); err != nil && !errors.Is(err, database.ErrAlreadyExists) {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// If pushed under a tag, (re)point the tag at this manifest.
|
||||||
|
if !isDigest(req.ref) {
|
||||||
|
if err := h.db.UpsertLocalFile(r.Context(), remote.Name, req.image+"/manifests/"+req.ref, result.ContentHash); err != nil {
|
||||||
|
dockerError(w, http.StatusInternalServerError, "UNKNOWN", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("local docker manifest pushed", "repo", remote.Name, "image", req.image, "ref", req.ref, "digest", digest)
|
||||||
|
w.Header().Set("Location", fmt.Sprintf("/v2/%s/%s/manifests/%s", remote.Name, req.image, req.ref))
|
||||||
|
w.Header().Set("Docker-Content-Digest", digest)
|
||||||
|
w.Header().Set("Docker-Distribution-Api-Version", dockerAPIVersionHeader)
|
||||||
|
w.WriteHeader(http.StatusCreated)
|
||||||
|
}
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
package v1
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestParseDockerPath(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
rest string
|
||||||
|
wantOK bool
|
||||||
|
wantImage string
|
||||||
|
wantKind string
|
||||||
|
wantRef string
|
||||||
|
}{
|
||||||
|
{"start upload trailing slash", "team/app/blobs/uploads/", true, "team/app", "upload", ""},
|
||||||
|
{"start upload no slash", "team/app/blobs/uploads", true, "team/app", "upload", ""},
|
||||||
|
{"patch upload with uuid", "team/app/blobs/uploads/abc-123", true, "team/app", "upload", "abc-123"},
|
||||||
|
{"single-segment image upload", "app/blobs/uploads/", true, "app", "upload", ""},
|
||||||
|
{"blob by digest", "team/app/blobs/sha256:deadbeef", true, "team/app", "blob", "sha256:deadbeef"},
|
||||||
|
{"manifest by tag", "team/app/manifests/v1.0.0", true, "team/app", "manifest", "v1.0.0"},
|
||||||
|
{"manifest by digest", "team/app/manifests/sha256:cafe", true, "team/app", "manifest", "sha256:cafe"},
|
||||||
|
{"tags list", "team/app/tags/list", true, "team/app", "tags", ""},
|
||||||
|
{"leading slash tolerated", "/team/app/manifests/latest", true, "team/app", "manifest", "latest"},
|
||||||
|
{"deep image name", "a/b/c/manifests/latest", true, "a/b/c", "manifest", "latest"},
|
||||||
|
{"unrecognised", "team/app/whatever", false, "", "", ""},
|
||||||
|
{"tags list without image", "tags/list", false, "", "", ""},
|
||||||
|
}
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
got, ok := parseDockerPath(tc.rest)
|
||||||
|
if ok != tc.wantOK {
|
||||||
|
t.Fatalf("ok = %v, want %v", ok, tc.wantOK)
|
||||||
|
}
|
||||||
|
if !tc.wantOK {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if got.image != tc.wantImage || got.kind != tc.wantKind || got.ref != tc.wantRef {
|
||||||
|
t.Fatalf("got %+v, want image=%q kind=%q ref=%q", got, tc.wantImage, tc.wantKind, tc.wantRef)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsDigest(t *testing.T) {
|
||||||
|
if !isDigest("sha256:abc") {
|
||||||
|
t.Fatal("sha256: prefix should be a digest")
|
||||||
|
}
|
||||||
|
if isDigest("v1.0.0") {
|
||||||
|
t.Fatal("a tag is not a digest")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -23,10 +23,18 @@ type ProxyHandler struct {
|
|||||||
db *database.DB
|
db *database.DB
|
||||||
store *storage.S3
|
store *storage.S3
|
||||||
local *v2.LocalHandler
|
local *v2.LocalHandler
|
||||||
|
cas *storage.CAS
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewProxyHandler(engine *proxy.Engine, virtualEngine *virtual.Engine, db *database.DB, store *storage.S3, local *v2.LocalHandler) *ProxyHandler {
|
func NewProxyHandler(engine *proxy.Engine, virtualEngine *virtual.Engine, db *database.DB, store *storage.S3, local *v2.LocalHandler) *ProxyHandler {
|
||||||
return &ProxyHandler{engine: engine, virtualEngine: virtualEngine, db: db, store: store, local: local}
|
return &ProxyHandler{
|
||||||
|
engine: engine,
|
||||||
|
virtualEngine: virtualEngine,
|
||||||
|
db: db,
|
||||||
|
store: store,
|
||||||
|
local: local,
|
||||||
|
cas: storage.NewCAS(store),
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (h *ProxyHandler) Routes() chi.Router {
|
func (h *ProxyHandler) Routes() chi.Router {
|
||||||
@@ -37,12 +45,20 @@ func (h *ProxyHandler) Routes() chi.Router {
|
|||||||
return r
|
return r
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DockerV2Routes mounts the Docker Registry HTTP API V2. Reads (GET/HEAD)
|
||||||
|
// dispatch to a local registry implementation for local docker repos and fall
|
||||||
|
// through to the upstream proxy otherwise; writes (POST/PATCH/PUT/DELETE) are
|
||||||
|
// only valid for local docker repos and drive push.
|
||||||
func (h *ProxyHandler) DockerV2Routes() chi.Router {
|
func (h *ProxyHandler) DockerV2Routes() chi.Router {
|
||||||
r := chi.NewRouter()
|
r := chi.NewRouter()
|
||||||
r.Get("/", h.handleDockerPing)
|
r.Get("/", h.handleDockerPing)
|
||||||
r.Head("/", h.handleDockerPing)
|
r.Head("/", h.handleDockerPing)
|
||||||
r.Get("/{remoteName}/*", h.handleProxy)
|
r.Get("/{remoteName}/*", h.dockerGet)
|
||||||
r.Head("/{remoteName}/*", h.handleProxyHead)
|
r.Head("/{remoteName}/*", h.dockerHead)
|
||||||
|
r.Post("/{remoteName}/*", h.dockerPost)
|
||||||
|
r.Patch("/{remoteName}/*", h.dockerPatch)
|
||||||
|
r.Put("/{remoteName}/*", h.dockerPut)
|
||||||
|
r.Delete("/{remoteName}/*", h.dockerDelete)
|
||||||
return r
|
return r
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -67,6 +83,15 @@ func (h *ProxyHandler) handleProxy(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Metadata-only remotes (e.g. github_rpm) synthesize their own responses and
|
||||||
|
// redirect package downloads to a backend remote instead of proxying bytes.
|
||||||
|
if rs, ok := prov.(provider.RemoteServer); ok {
|
||||||
|
proxyBaseURL := fmt.Sprintf("%s://%s", scheme(r), r.Host)
|
||||||
|
if rs.ServeRemote(w, r, *remote, path, proxyBaseURL, h.db) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
result, err := h.engine.Fetch(r.Context(), *remote, path, prov, r.Header)
|
result, err := h.engine.Fetch(r.Context(), *remote, path, prov, r.Header)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
var proxyErr *proxy.ProxyError
|
var proxyErr *proxy.ProxyError
|
||||||
|
|||||||
@@ -57,7 +57,7 @@ func do(t *testing.T, h http.Handler, method, path, body string) int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRemotesErrorPaths(t *testing.T) {
|
func TestRemotesErrorPaths(t *testing.T) {
|
||||||
h := NewRemotesHandler(closedDB(t)).Routes()
|
h := NewRemotesHandler(closedDB(t), nil).Routes()
|
||||||
if c := do(t, h, "GET", "/", ""); c != 500 {
|
if c := do(t, h, "GET", "/", ""); c != 500 {
|
||||||
t.Errorf("list with dead db = %d, want 500", c)
|
t.Errorf("list with dead db = %d, want 500", c)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,12 +11,21 @@ import (
|
|||||||
"git.unkin.net/unkin/artifactapi/pkg/models"
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
)
|
)
|
||||||
|
|
||||||
type RemotesHandler struct {
|
// Primer enqueues a background metadata prime for a newly created remote so the
|
||||||
db *database.DB
|
// create call never blocks on a derive. *rpm.Syncer and *deb.Syncer satisfy it.
|
||||||
|
type Primer interface {
|
||||||
|
EnqueuePrime(remote models.Remote)
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewRemotesHandler(db *database.DB) *RemotesHandler {
|
type RemotesHandler struct {
|
||||||
return &RemotesHandler{db: db}
|
db *database.DB
|
||||||
|
primers map[models.PackageType]Primer
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewRemotesHandler wires the handler to the per-type metadata primers. primers
|
||||||
|
// may be nil; a package type with no registered primer simply skips priming.
|
||||||
|
func NewRemotesHandler(db *database.DB, primers map[models.PackageType]Primer) *RemotesHandler {
|
||||||
|
return &RemotesHandler{db: db, primers: primers}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (h *RemotesHandler) Routes() chi.Router {
|
func (h *RemotesHandler) Routes() chi.Router {
|
||||||
@@ -77,6 +86,11 @@ func (h *RemotesHandler) create(w http.ResponseWriter, r *http.Request) {
|
|||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
// Prime a metadata-only remote (github_rpm/github_deb) in the background so
|
||||||
|
// its first index request is served from cache instead of a cold derive.
|
||||||
|
if primer := h.primers[remote.PackageType]; primer != nil {
|
||||||
|
primer.EnqueuePrime(remote)
|
||||||
|
}
|
||||||
writeJSON(w, http.StatusCreated, remote)
|
writeJSON(w, http.StatusCreated, remote)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -24,6 +24,38 @@ type Config struct {
|
|||||||
S3Bucket string
|
S3Bucket string
|
||||||
S3Secure bool
|
S3Secure bool
|
||||||
S3Region string
|
S3Region string
|
||||||
|
|
||||||
|
// Terraform provider registry signing. When TFSigningKeyPath points at a
|
||||||
|
// readable armored GPG private key, artifactapi serves local terraform
|
||||||
|
// repos as a real provider registry (service discovery + signed
|
||||||
|
// SHA256SUMS). Left empty, the registry endpoints stay disabled.
|
||||||
|
TFSigningKeyPath string
|
||||||
|
TFSigningKeyPassphrase string
|
||||||
|
TFProviderProtocols string
|
||||||
|
|
||||||
|
// github_rpm background syncer. The syncer keeps derived RPM metadata for
|
||||||
|
// every github_rpm remote fresh off the client request path, sharing a
|
||||||
|
// single global token-bucket limiter across all remotes so GitHub is never
|
||||||
|
// hammered. Defaults are conservative: 1 req/s (3600/hr) sits well under an
|
||||||
|
// authenticated token's 5000/hr. Unauthenticated remotes (60/hr) lean on
|
||||||
|
// ETag/304 — an unchanged repo costs nothing — so keep those repos small or
|
||||||
|
// configure a token.
|
||||||
|
GitHubSyncRatePerSec float64
|
||||||
|
GitHubSyncBurst int
|
||||||
|
GitHubSyncWorkers int
|
||||||
|
GitHubSyncPollInterval int
|
||||||
|
|
||||||
|
// Server-level GitHub machine credential, applied by default to every
|
||||||
|
// outbound GitHub request (releases scan, ranged asset fetches, and the
|
||||||
|
// generic-github byte proxy for private assets). Delivered via env/secret
|
||||||
|
// only — never stored per-remote, never returned by an API, never logged.
|
||||||
|
// Configure exactly one mode: a Personal Access Token, or a GitHub App
|
||||||
|
// (id + installation id + private key). Partial App config fails at startup.
|
||||||
|
GitHubToken string
|
||||||
|
GitHubAppID string
|
||||||
|
GitHubAppInstallationID string
|
||||||
|
GitHubAppPrivateKey string
|
||||||
|
GitHubAppPrivateKeyPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Config) DatabaseDSN() string {
|
func (c *Config) DatabaseDSN() string {
|
||||||
@@ -41,6 +73,23 @@ func Load() (*Config, error) {
|
|||||||
|
|
||||||
s3Secure, _ := strconv.ParseBool(getenv("MINIO_SECURE", "false"))
|
s3Secure, _ := strconv.ParseBool(getenv("MINIO_SECURE", "false"))
|
||||||
|
|
||||||
|
syncRate, err := strconv.ParseFloat(getenv("GITHUB_SYNC_RATE", "1"), 64)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid GITHUB_SYNC_RATE: %w", err)
|
||||||
|
}
|
||||||
|
syncBurst, err := strconv.Atoi(getenv("GITHUB_SYNC_BURST", "5"))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid GITHUB_SYNC_BURST: %w", err)
|
||||||
|
}
|
||||||
|
syncWorkers, err := strconv.Atoi(getenv("GITHUB_SYNC_WORKERS", "3"))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid GITHUB_SYNC_WORKERS: %w", err)
|
||||||
|
}
|
||||||
|
syncPoll, err := strconv.Atoi(getenv("GITHUB_SYNC_POLL_INTERVAL", "60"))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid GITHUB_SYNC_POLL_INTERVAL: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
cfg := &Config{
|
cfg := &Config{
|
||||||
ListenAddr: getenv("LISTEN_ADDR", ":8000"),
|
ListenAddr: getenv("LISTEN_ADDR", ":8000"),
|
||||||
|
|
||||||
@@ -59,6 +108,21 @@ func Load() (*Config, error) {
|
|||||||
S3Bucket: getenv("MINIO_BUCKET", "artifacts"),
|
S3Bucket: getenv("MINIO_BUCKET", "artifacts"),
|
||||||
S3Secure: s3Secure,
|
S3Secure: s3Secure,
|
||||||
S3Region: getenv("MINIO_REGION", ""),
|
S3Region: getenv("MINIO_REGION", ""),
|
||||||
|
|
||||||
|
TFSigningKeyPath: getenv("TF_SIGNING_KEY_PATH", ""),
|
||||||
|
TFSigningKeyPassphrase: getenv("TF_SIGNING_KEY_PASSPHRASE", ""),
|
||||||
|
TFProviderProtocols: getenv("TF_PROVIDER_PROTOCOLS", "5.0,6.0"),
|
||||||
|
|
||||||
|
GitHubSyncRatePerSec: syncRate,
|
||||||
|
GitHubSyncBurst: syncBurst,
|
||||||
|
GitHubSyncWorkers: syncWorkers,
|
||||||
|
GitHubSyncPollInterval: syncPoll,
|
||||||
|
|
||||||
|
GitHubToken: getenv("GITHUB_TOKEN", ""),
|
||||||
|
GitHubAppID: getenv("GITHUB_APP_ID", ""),
|
||||||
|
GitHubAppInstallationID: getenv("GITHUB_APP_INSTALLATION_ID", ""),
|
||||||
|
GitHubAppPrivateKey: getenv("GITHUB_APP_PRIVATE_KEY", ""),
|
||||||
|
GitHubAppPrivateKeyPath: getenv("GITHUB_APP_PRIVATE_KEY_PATH", ""),
|
||||||
}
|
}
|
||||||
|
|
||||||
return cfg, nil
|
return cfg, nil
|
||||||
|
|||||||
@@ -0,0 +1,70 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ListGitHubDebRemotes returns every github_deb remote so the syncer can sweep
|
||||||
|
// them on each poll tick.
|
||||||
|
func (db *DB) ListGitHubDebRemotes(ctx context.Context) ([]models.Remote, error) {
|
||||||
|
rows, err := db.Pool.Query(ctx, `SELECT `+remoteCols+` FROM remotes WHERE package_type = $1 ORDER BY name`, models.PackageGitHubDeb)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
var remotes []models.Remote
|
||||||
|
for rows.Next() {
|
||||||
|
var r models.Remote
|
||||||
|
if err := scanRemote(rows, &r); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
remotes = append(remotes, r)
|
||||||
|
}
|
||||||
|
return remotes, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ClaimGitHubDebSyncLease atomically claims the per-remote sync lease. It
|
||||||
|
// succeeds only when the remote is due (never synced, or synced longer than
|
||||||
|
// freshness ago) and no live lease is held by another replica. A zero freshness
|
||||||
|
// (prime scans) ignores the recency gate. The returned etag is the stored
|
||||||
|
// releases-list ETag, shared across replicas.
|
||||||
|
func (db *DB) ClaimGitHubDebSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
|
||||||
|
row := db.Pool.QueryRow(ctx, `
|
||||||
|
INSERT INTO github_deb_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
|
||||||
|
VALUES ($1, $2, now() + make_interval(secs => $4))
|
||||||
|
ON CONFLICT (remote_name) DO UPDATE
|
||||||
|
SET sync_lease_owner = $2,
|
||||||
|
sync_lease_expires = now() + make_interval(secs => $4)
|
||||||
|
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
|
||||||
|
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
|
||||||
|
RETURNING s.etag
|
||||||
|
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
|
||||||
|
|
||||||
|
var etag string
|
||||||
|
if err := row.Scan(&etag); err != nil {
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return false, "", nil
|
||||||
|
}
|
||||||
|
return false, "", err
|
||||||
|
}
|
||||||
|
return true, etag, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReleaseGitHubDebSyncLease records the completed scan and frees the lease. Only
|
||||||
|
// the owning replica may release; last_synced_at advances so the next poll waits
|
||||||
|
// a full freshness window, and etag is persisted for the next conditional request.
|
||||||
|
func (db *DB) ReleaseGitHubDebSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
|
||||||
|
_, err := db.Pool.Exec(ctx, `
|
||||||
|
UPDATE github_deb_sync_state
|
||||||
|
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
|
||||||
|
WHERE remote_name = $1 AND sync_lease_owner = $2
|
||||||
|
`, remoteName, owner, syncedAt, etag)
|
||||||
|
return err
|
||||||
|
}
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
func seedGitHubDebRemote(t *testing.T, name string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := testDB.CreateRemote(ctx(), &models.Remote{
|
||||||
|
Name: name, PackageType: models.PackageGitHubDeb, RepoType: models.RepoTypeRemote,
|
||||||
|
BaseURL: "https://api.github.com/repos/acme/tools", ReleasesRemote: "github", MutableTTL: 3600,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("seed github_deb remote: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGitHubDebSyncLease exercises the real SQL: exactly one replica may hold the
|
||||||
|
// lease, the recency window blocks a too-soon periodic re-claim, and a prime
|
||||||
|
// (freshness 0) bypasses recency but still respects a live lease.
|
||||||
|
func TestGitHubDebSyncLease(t *testing.T) {
|
||||||
|
requireDB(t)
|
||||||
|
name := "ghdeb-lease-" + time.Now().Format("150405.000000")
|
||||||
|
seedGitHubDebRemote(t, name)
|
||||||
|
|
||||||
|
const lease = 15 * time.Minute
|
||||||
|
freshness := time.Hour
|
||||||
|
|
||||||
|
claimed, etag, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-1", freshness, lease)
|
||||||
|
if err != nil || !claimed {
|
||||||
|
t.Fatalf("replica-1 first claim: claimed=%v err=%v", claimed, err)
|
||||||
|
}
|
||||||
|
if etag != "" {
|
||||||
|
t.Fatalf("initial etag should be empty, got %q", etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
claimed2, _, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-2", freshness, lease)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("replica-2 claim err: %v", err)
|
||||||
|
}
|
||||||
|
if claimed2 {
|
||||||
|
t.Fatal("replica-2 claimed while replica-1 holds the lease")
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := testDB.ReleaseGitHubDebSyncLease(ctx(), name, "replica-1", `"etag-1"`, time.Now()); err != nil {
|
||||||
|
t.Fatalf("release: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
claimed3, _, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-2", freshness, lease)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("replica-2 recency claim err: %v", err)
|
||||||
|
}
|
||||||
|
if claimed3 {
|
||||||
|
t.Fatal("periodic claim succeeded inside the freshness window")
|
||||||
|
}
|
||||||
|
|
||||||
|
claimed4, etag4, err := testDB.ClaimGitHubDebSyncLease(ctx(), name, "replica-2", 0, lease)
|
||||||
|
if err != nil || !claimed4 {
|
||||||
|
t.Fatalf("prime claim: claimed=%v err=%v", claimed4, err)
|
||||||
|
}
|
||||||
|
if etag4 != `"etag-1"` {
|
||||||
|
t.Fatalf("prime claim etag = %q, want persisted \"etag-1\"", etag4)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestListGitHubDebRemotes(t *testing.T) {
|
||||||
|
requireDB(t)
|
||||||
|
name := "ghdeb-list-" + time.Now().Format("150405.000000")
|
||||||
|
seedGitHubDebRemote(t, name)
|
||||||
|
seedRemote(t, "generic-"+time.Now().Format("150405.000000"))
|
||||||
|
|
||||||
|
remotes, err := testDB.ListGitHubDebRemotes(ctx())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("list: %v", err)
|
||||||
|
}
|
||||||
|
found := false
|
||||||
|
for _, r := range remotes {
|
||||||
|
if r.PackageType != models.PackageGitHubDeb {
|
||||||
|
t.Fatalf("non-github_deb remote returned: %s (%s)", r.Name, r.PackageType)
|
||||||
|
}
|
||||||
|
if r.Name == name {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("seeded remote %q not returned", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (db *DB) InsertDebMetadata(ctx context.Context, meta *provider.DebMetadata) error {
|
||||||
|
_, err := db.Pool.Exec(ctx, `
|
||||||
|
INSERT INTO deb_metadata (
|
||||||
|
repo_name, file_path, content_hash,
|
||||||
|
name, version, architecture, control,
|
||||||
|
size, md5, sha256
|
||||||
|
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
|
||||||
|
ON CONFLICT (repo_name, file_path) DO NOTHING
|
||||||
|
`,
|
||||||
|
meta.RepoName, meta.FilePath, meta.ContentHash,
|
||||||
|
meta.Name, meta.Version, meta.Architecture, meta.Control,
|
||||||
|
meta.Size, meta.MD5, meta.SHA256,
|
||||||
|
)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *DB) DeleteDebMetadata(ctx context.Context, repoName, filePath string) error {
|
||||||
|
_, err := db.Pool.Exec(ctx, `DELETE FROM deb_metadata WHERE repo_name = $1 AND file_path = $2`, repoName, filePath)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (db *DB) ListDebMetadataEntries(ctx context.Context, repoName string) ([]provider.DebMetadata, error) {
|
||||||
|
rows, err := db.Pool.Query(ctx, `
|
||||||
|
SELECT repo_name, file_path, content_hash,
|
||||||
|
name, version, architecture, control,
|
||||||
|
size, md5, sha256
|
||||||
|
FROM deb_metadata
|
||||||
|
WHERE repo_name = $1
|
||||||
|
ORDER BY name, version, architecture
|
||||||
|
`, repoName)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
var result []provider.DebMetadata
|
||||||
|
for rows.Next() {
|
||||||
|
var m provider.DebMetadata
|
||||||
|
if err := rows.Scan(
|
||||||
|
&m.RepoName, &m.FilePath, &m.ContentHash,
|
||||||
|
&m.Name, &m.Version, &m.Architecture, &m.Control,
|
||||||
|
&m.Size, &m.MD5, &m.SHA256,
|
||||||
|
); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
result = append(result, m)
|
||||||
|
}
|
||||||
|
return result, rows.Err()
|
||||||
|
}
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ListGitHubRPMRemotes returns every github_rpm remote so the syncer can sweep
|
||||||
|
// them on each poll tick.
|
||||||
|
func (db *DB) ListGitHubRPMRemotes(ctx context.Context) ([]models.Remote, error) {
|
||||||
|
rows, err := db.Pool.Query(ctx, `SELECT `+remoteCols+` FROM remotes WHERE package_type = $1 ORDER BY name`, models.PackageGitHubRPM)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
var remotes []models.Remote
|
||||||
|
for rows.Next() {
|
||||||
|
var r models.Remote
|
||||||
|
if err := scanRemote(rows, &r); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
remotes = append(remotes, r)
|
||||||
|
}
|
||||||
|
return remotes, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ClaimGitHubSyncLease atomically claims the per-remote sync lease. It succeeds
|
||||||
|
// (claimed=true) only when the remote is due — never synced, or synced longer
|
||||||
|
// than freshness ago — and no live lease is held by another replica. This bounds
|
||||||
|
// total GitHub load to roughly one scan per freshness window regardless of how
|
||||||
|
// many replicas poll. The returned etag is the stored releases-list ETag, shared
|
||||||
|
// across replicas so a conditional request can short-circuit an unchanged repo.
|
||||||
|
// A zero freshness (used for prime scans) ignores the recency gate and claims
|
||||||
|
// whenever no live lease is held.
|
||||||
|
func (db *DB) ClaimGitHubSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (bool, string, error) {
|
||||||
|
row := db.Pool.QueryRow(ctx, `
|
||||||
|
INSERT INTO github_rpm_sync_state AS s (remote_name, sync_lease_owner, sync_lease_expires)
|
||||||
|
VALUES ($1, $2, now() + make_interval(secs => $4))
|
||||||
|
ON CONFLICT (remote_name) DO UPDATE
|
||||||
|
SET sync_lease_owner = $2,
|
||||||
|
sync_lease_expires = now() + make_interval(secs => $4)
|
||||||
|
WHERE (s.last_synced_at IS NULL OR s.last_synced_at < now() - make_interval(secs => $3))
|
||||||
|
AND (s.sync_lease_expires IS NULL OR s.sync_lease_expires < now())
|
||||||
|
RETURNING s.etag
|
||||||
|
`, remoteName, owner, freshness.Seconds(), lease.Seconds())
|
||||||
|
|
||||||
|
var etag string
|
||||||
|
if err := row.Scan(&etag); err != nil {
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return false, "", nil
|
||||||
|
}
|
||||||
|
return false, "", err
|
||||||
|
}
|
||||||
|
return true, etag, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReleaseGitHubSyncLease records the completed scan and frees the lease. Only the
|
||||||
|
// owning replica may release; last_synced_at advances so the next poll waits a
|
||||||
|
// full freshness window, and etag is persisted for the next conditional request.
|
||||||
|
func (db *DB) ReleaseGitHubSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error {
|
||||||
|
_, err := db.Pool.Exec(ctx, `
|
||||||
|
UPDATE github_rpm_sync_state
|
||||||
|
SET last_synced_at = $3, etag = $4, sync_lease_owner = '', sync_lease_expires = NULL
|
||||||
|
WHERE remote_name = $1 AND sync_lease_owner = $2
|
||||||
|
`, remoteName, owner, syncedAt, etag)
|
||||||
|
return err
|
||||||
|
}
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
func seedGitHubRPMRemote(t *testing.T, name string) {
|
||||||
|
t.Helper()
|
||||||
|
if err := testDB.CreateRemote(ctx(), &models.Remote{
|
||||||
|
Name: name, PackageType: models.PackageGitHubRPM, RepoType: models.RepoTypeRemote,
|
||||||
|
BaseURL: "https://api.github.com/repos/acme/tools", ReleasesRemote: "github", MutableTTL: 3600,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("seed github_rpm remote: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGitHubSyncLease exercises the real SQL: exactly one replica may hold the
|
||||||
|
// lease, the recency window blocks a too-soon periodic re-claim, and a prime
|
||||||
|
// (freshness 0) bypasses recency but still respects a live lease.
|
||||||
|
func TestGitHubSyncLease(t *testing.T) {
|
||||||
|
requireDB(t)
|
||||||
|
name := "gh-lease-" + time.Now().Format("150405.000000")
|
||||||
|
seedGitHubRPMRemote(t, name)
|
||||||
|
|
||||||
|
const lease = 15 * time.Minute
|
||||||
|
freshness := time.Hour
|
||||||
|
|
||||||
|
// First claim on a never-synced remote wins; etag starts empty.
|
||||||
|
claimed, etag, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-1", freshness, lease)
|
||||||
|
if err != nil || !claimed {
|
||||||
|
t.Fatalf("replica-1 first claim: claimed=%v err=%v", claimed, err)
|
||||||
|
}
|
||||||
|
if etag != "" {
|
||||||
|
t.Fatalf("initial etag should be empty, got %q", etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A second replica cannot claim while the lease is held.
|
||||||
|
claimed2, _, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-2", freshness, lease)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("replica-2 claim err: %v", err)
|
||||||
|
}
|
||||||
|
if claimed2 {
|
||||||
|
t.Fatal("replica-2 claimed while replica-1 holds the lease")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Replica 1 finishes: record the sync and persist an etag.
|
||||||
|
if err := testDB.ReleaseGitHubSyncLease(ctx(), name, "replica-1", `"etag-1"`, time.Now()); err != nil {
|
||||||
|
t.Fatalf("release: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A periodic re-claim inside the freshness window is blocked by recency.
|
||||||
|
claimed3, _, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-2", freshness, lease)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("replica-2 recency claim err: %v", err)
|
||||||
|
}
|
||||||
|
if claimed3 {
|
||||||
|
t.Fatal("periodic claim succeeded inside the freshness window")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A prime (freshness 0) bypasses recency and reads the persisted etag.
|
||||||
|
claimed4, etag4, err := testDB.ClaimGitHubSyncLease(ctx(), name, "replica-2", 0, lease)
|
||||||
|
if err != nil || !claimed4 {
|
||||||
|
t.Fatalf("prime claim: claimed=%v err=%v", claimed4, err)
|
||||||
|
}
|
||||||
|
if etag4 != `"etag-1"` {
|
||||||
|
t.Fatalf("prime claim etag = %q, want persisted \"etag-1\"", etag4)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestListGitHubRPMRemotes(t *testing.T) {
|
||||||
|
requireDB(t)
|
||||||
|
name := "gh-list-" + time.Now().Format("150405.000000")
|
||||||
|
seedGitHubRPMRemote(t, name)
|
||||||
|
seedRemote(t, "generic-"+time.Now().Format("150405.000000"))
|
||||||
|
|
||||||
|
remotes, err := testDB.ListGitHubRPMRemotes(ctx())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("list: %v", err)
|
||||||
|
}
|
||||||
|
found := false
|
||||||
|
for _, r := range remotes {
|
||||||
|
if r.PackageType != models.PackageGitHubRPM {
|
||||||
|
t.Fatalf("non-github_rpm remote returned: %s (%s)", r.Name, r.PackageType)
|
||||||
|
}
|
||||||
|
if r.Name == name {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("seeded remote %q not returned", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -38,6 +38,20 @@ func (db *DB) CreateLocalFile(ctx context.Context, repoName, filePath, contentHa
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpsertLocalFile inserts a local file or repoints an existing path at a new
|
||||||
|
// blob. Unlike CreateLocalFile it never errors on a duplicate path — it is for
|
||||||
|
// mutable references such as Docker tags, where re-pushing a tag must move it to
|
||||||
|
// the newly-pushed manifest rather than being rejected as an overwrite.
|
||||||
|
func (db *DB) UpsertLocalFile(ctx context.Context, repoName, filePath, contentHash string) error {
|
||||||
|
_, err := db.Pool.Exec(ctx, `
|
||||||
|
INSERT INTO local_files (repo_name, file_path, content_hash)
|
||||||
|
VALUES ($1, $2, $3)
|
||||||
|
ON CONFLICT (repo_name, file_path)
|
||||||
|
DO UPDATE SET content_hash = EXCLUDED.content_hash, created_at = NOW()
|
||||||
|
`, repoName, filePath, contentHash)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
func (db *DB) GetLocalFile(ctx context.Context, repoName, filePath string) (*LocalFile, error) {
|
func (db *DB) GetLocalFile(ctx context.Context, repoName, filePath string) (*LocalFile, error) {
|
||||||
row := db.Pool.QueryRow(ctx, `
|
row := db.Pool.QueryRow(ctx, `
|
||||||
SELECT id, repo_name, file_path, content_hash, created_at
|
SELECT id, repo_name, file_path, content_hash, created_at
|
||||||
|
|||||||
@@ -151,6 +151,8 @@ func (db *DB) migrate() error {
|
|||||||
packager TEXT DEFAULT '',
|
packager TEXT DEFAULT '',
|
||||||
requires JSONB DEFAULT '[]',
|
requires JSONB DEFAULT '[]',
|
||||||
provides JSONB DEFAULT '[]',
|
provides JSONB DEFAULT '[]',
|
||||||
|
conflicts JSONB DEFAULT '[]',
|
||||||
|
obsoletes JSONB DEFAULT '[]',
|
||||||
files JSONB DEFAULT '[]',
|
files JSONB DEFAULT '[]',
|
||||||
changelogs JSONB DEFAULT '[]',
|
changelogs JSONB DEFAULT '[]',
|
||||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||||
@@ -158,6 +160,50 @@ func (db *DB) migrate() error {
|
|||||||
);
|
);
|
||||||
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_rpm_metadata_repo ON rpm_metadata(repo_name);
|
CREATE INDEX IF NOT EXISTS idx_rpm_metadata_repo ON rpm_metadata(repo_name);
|
||||||
|
|
||||||
|
ALTER TABLE rpm_metadata ADD COLUMN IF NOT EXISTS conflicts JSONB DEFAULT '[]';
|
||||||
|
ALTER TABLE rpm_metadata ADD COLUMN IF NOT EXISTS obsoletes JSONB DEFAULT '[]';
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS deb_metadata (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
repo_name TEXT NOT NULL,
|
||||||
|
file_path TEXT NOT NULL,
|
||||||
|
content_hash TEXT NOT NULL,
|
||||||
|
name TEXT NOT NULL,
|
||||||
|
version TEXT NOT NULL,
|
||||||
|
architecture TEXT NOT NULL,
|
||||||
|
control TEXT NOT NULL,
|
||||||
|
size BIGINT DEFAULT 0,
|
||||||
|
md5 TEXT DEFAULT '',
|
||||||
|
sha256 TEXT DEFAULT '',
|
||||||
|
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||||
|
UNIQUE(repo_name, file_path)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_deb_metadata_repo ON deb_metadata(repo_name);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS github_rpm_sync_state (
|
||||||
|
remote_name TEXT PRIMARY KEY,
|
||||||
|
etag TEXT DEFAULT '',
|
||||||
|
last_synced_at TIMESTAMPTZ,
|
||||||
|
sync_lease_owner TEXT DEFAULT '',
|
||||||
|
sync_lease_expires TIMESTAMPTZ
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS github_deb_sync_state (
|
||||||
|
remote_name TEXT PRIMARY KEY,
|
||||||
|
etag TEXT DEFAULT '',
|
||||||
|
last_synced_at TIMESTAMPTZ,
|
||||||
|
sync_lease_owner TEXT DEFAULT '',
|
||||||
|
sync_lease_expires TIMESTAMPTZ
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS signing_keys (
|
||||||
|
purpose TEXT PRIMARY KEY,
|
||||||
|
private_key_armor TEXT NOT NULL,
|
||||||
|
key_id TEXT NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||||
|
);
|
||||||
`)
|
`)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,6 +10,8 @@ import (
|
|||||||
func (db *DB) InsertRPMMetadata(ctx context.Context, meta *provider.RPMMetadata) error {
|
func (db *DB) InsertRPMMetadata(ctx context.Context, meta *provider.RPMMetadata) error {
|
||||||
requiresJSON, _ := json.Marshal(meta.Requires)
|
requiresJSON, _ := json.Marshal(meta.Requires)
|
||||||
providesJSON, _ := json.Marshal(meta.Provides)
|
providesJSON, _ := json.Marshal(meta.Provides)
|
||||||
|
conflictsJSON, _ := json.Marshal(meta.Conflicts)
|
||||||
|
obsoletesJSON, _ := json.Marshal(meta.Obsoletes)
|
||||||
filesJSON, _ := json.Marshal(meta.Files)
|
filesJSON, _ := json.Marshal(meta.Files)
|
||||||
changelogsJSON, _ := json.Marshal(meta.Changelogs)
|
changelogsJSON, _ := json.Marshal(meta.Changelogs)
|
||||||
|
|
||||||
@@ -19,15 +21,15 @@ func (db *DB) InsertRPMMetadata(ctx context.Context, meta *provider.RPMMetadata)
|
|||||||
name, epoch, version, release, arch,
|
name, epoch, version, release, arch,
|
||||||
summary, description, rpm_size, installed_size,
|
summary, description, rpm_size, installed_size,
|
||||||
license, vendor, build_group, build_host, source_rpm, url, packager,
|
license, vendor, build_group, build_host, source_rpm, url, packager,
|
||||||
requires, provides, files, changelogs
|
requires, provides, conflicts, obsoletes, files, changelogs
|
||||||
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23)
|
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25)
|
||||||
ON CONFLICT (repo_name, file_path) DO NOTHING
|
ON CONFLICT (repo_name, file_path) DO NOTHING
|
||||||
`,
|
`,
|
||||||
meta.RepoName, meta.FilePath, meta.ContentHash,
|
meta.RepoName, meta.FilePath, meta.ContentHash,
|
||||||
meta.Name, meta.Epoch, meta.Version, meta.Release, meta.Arch,
|
meta.Name, meta.Epoch, meta.Version, meta.Release, meta.Arch,
|
||||||
meta.Summary, meta.Description, meta.RPMSize, meta.InstalledSize,
|
meta.Summary, meta.Description, meta.RPMSize, meta.InstalledSize,
|
||||||
meta.License, meta.Vendor, meta.Group, meta.BuildHost, meta.SourceRPM, meta.URL, meta.Packager,
|
meta.License, meta.Vendor, meta.Group, meta.BuildHost, meta.SourceRPM, meta.URL, meta.Packager,
|
||||||
requiresJSON, providesJSON, filesJSON, changelogsJSON,
|
requiresJSON, providesJSON, conflictsJSON, obsoletesJSON, filesJSON, changelogsJSON,
|
||||||
)
|
)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -59,6 +61,8 @@ type RPMMetadataRow struct {
|
|||||||
Packager string
|
Packager string
|
||||||
Requires json.RawMessage
|
Requires json.RawMessage
|
||||||
Provides json.RawMessage
|
Provides json.RawMessage
|
||||||
|
Conflicts json.RawMessage
|
||||||
|
Obsoletes json.RawMessage
|
||||||
Files json.RawMessage
|
Files json.RawMessage
|
||||||
Changelogs json.RawMessage
|
Changelogs json.RawMessage
|
||||||
}
|
}
|
||||||
@@ -93,6 +97,8 @@ func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]pr
|
|||||||
}
|
}
|
||||||
json.Unmarshal(r.Requires, &meta.Requires)
|
json.Unmarshal(r.Requires, &meta.Requires)
|
||||||
json.Unmarshal(r.Provides, &meta.Provides)
|
json.Unmarshal(r.Provides, &meta.Provides)
|
||||||
|
json.Unmarshal(r.Conflicts, &meta.Conflicts)
|
||||||
|
json.Unmarshal(r.Obsoletes, &meta.Obsoletes)
|
||||||
json.Unmarshal(r.Files, &meta.Files)
|
json.Unmarshal(r.Files, &meta.Files)
|
||||||
json.Unmarshal(r.Changelogs, &meta.Changelogs)
|
json.Unmarshal(r.Changelogs, &meta.Changelogs)
|
||||||
result[i] = meta
|
result[i] = meta
|
||||||
@@ -106,7 +112,7 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
|
|||||||
name, epoch, version, release, arch,
|
name, epoch, version, release, arch,
|
||||||
summary, description, rpm_size, installed_size,
|
summary, description, rpm_size, installed_size,
|
||||||
license, vendor, build_group, build_host, source_rpm, url, packager,
|
license, vendor, build_group, build_host, source_rpm, url, packager,
|
||||||
requires, provides, files, changelogs
|
requires, provides, conflicts, obsoletes, files, changelogs
|
||||||
FROM rpm_metadata
|
FROM rpm_metadata
|
||||||
WHERE repo_name = $1
|
WHERE repo_name = $1
|
||||||
ORDER BY name, epoch, version, release, arch
|
ORDER BY name, epoch, version, release, arch
|
||||||
@@ -124,7 +130,7 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
|
|||||||
&r.Name, &r.Epoch, &r.Version, &r.Release, &r.Arch,
|
&r.Name, &r.Epoch, &r.Version, &r.Release, &r.Arch,
|
||||||
&r.Summary, &r.Description, &r.RPMSize, &r.InstalledSize,
|
&r.Summary, &r.Description, &r.RPMSize, &r.InstalledSize,
|
||||||
&r.License, &r.Vendor, &r.Group, &r.BuildHost, &r.SourceRPM, &r.URL, &r.Packager,
|
&r.License, &r.Vendor, &r.Group, &r.BuildHost, &r.SourceRPM, &r.URL, &r.Packager,
|
||||||
&r.Requires, &r.Provides, &r.Files, &r.Changelogs,
|
&r.Requires, &r.Provides, &r.Conflicts, &r.Obsoletes, &r.Files, &r.Changelogs,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// GetSigningKey returns the stored armored private key and key id for a purpose.
|
||||||
|
// found is false when no key has been generated yet.
|
||||||
|
func (db *DB) GetSigningKey(ctx context.Context, purpose string) (armor, keyID string, found bool, err error) {
|
||||||
|
row := db.Pool.QueryRow(ctx, `
|
||||||
|
SELECT private_key_armor, key_id FROM signing_keys WHERE purpose = $1
|
||||||
|
`, purpose)
|
||||||
|
if err := row.Scan(&armor, &keyID); err != nil {
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return "", "", false, nil
|
||||||
|
}
|
||||||
|
return "", "", false, err
|
||||||
|
}
|
||||||
|
return armor, keyID, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// InsertSigningKeyIfAbsent stores a freshly generated key, doing nothing if
|
||||||
|
// another replica already inserted one. Callers re-read with GetSigningKey to
|
||||||
|
// pick up whichever key won the race.
|
||||||
|
func (db *DB) InsertSigningKeyIfAbsent(ctx context.Context, purpose, armor, keyID string) error {
|
||||||
|
_, err := db.Pool.Exec(ctx, `
|
||||||
|
INSERT INTO signing_keys (purpose, private_key_armor, key_id)
|
||||||
|
VALUES ($1, $2, $3)
|
||||||
|
ON CONFLICT (purpose) DO NOTHING
|
||||||
|
`, purpose, armor, keyID)
|
||||||
|
return err
|
||||||
|
}
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
package database
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestSigningKeyRoundTripAndIdempotency(t *testing.T) {
|
||||||
|
requireDB(t)
|
||||||
|
|
||||||
|
const purpose = "terraform-provider-test"
|
||||||
|
|
||||||
|
// Absent to start.
|
||||||
|
if _, _, found, err := testDB.GetSigningKey(ctx(), purpose); err != nil || found {
|
||||||
|
t.Fatalf("expected no key, got found=%v err=%v", found, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := testDB.InsertSigningKeyIfAbsent(ctx(), purpose, "ARMOR-1", "KEYID1"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A second insert must not overwrite (models the replica race).
|
||||||
|
if err := testDB.InsertSigningKeyIfAbsent(ctx(), purpose, "ARMOR-2", "KEYID2"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
armor, keyID, found, err := testDB.GetSigningKey(ctx(), purpose)
|
||||||
|
if err != nil || !found {
|
||||||
|
t.Fatalf("expected key, found=%v err=%v", found, err)
|
||||||
|
}
|
||||||
|
if armor != "ARMOR-1" || keyID != "KEYID1" {
|
||||||
|
t.Errorf("key was overwritten: armor=%q key_id=%q", armor, keyID)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -14,6 +14,11 @@ import (
|
|||||||
// before the referencing artifact/local_files row exists.
|
// before the referencing artifact/local_files row exists.
|
||||||
const blobGracePeriod = 1 * time.Hour
|
const blobGracePeriod = 1 * time.Hour
|
||||||
|
|
||||||
|
// uploadGracePeriod is how long a docker blob-upload staging object
|
||||||
|
// (uploads/<uuid>) may sit idle before GC treats it as an abandoned push and
|
||||||
|
// reaps it. Generous so a slow but live push is never cut off mid-flight.
|
||||||
|
const uploadGracePeriod = 24 * time.Hour
|
||||||
|
|
||||||
type Collector struct {
|
type Collector struct {
|
||||||
db *database.DB
|
db *database.DB
|
||||||
store *storage.S3
|
store *storage.S3
|
||||||
@@ -43,6 +48,8 @@ func (c *Collector) Run(ctx context.Context) {
|
|||||||
func (c *Collector) sweep(ctx context.Context) {
|
func (c *Collector) sweep(ctx context.Context) {
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|
||||||
|
c.sweepUploads(ctx)
|
||||||
|
|
||||||
orphaned, err := c.db.FindOrphanedBlobs(ctx, blobGracePeriod)
|
orphaned, err := c.db.FindOrphanedBlobs(ctx, blobGracePeriod)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
slog.Error("gc: find orphaned blobs", "error", err)
|
slog.Error("gc: find orphaned blobs", "error", err)
|
||||||
@@ -70,3 +77,24 @@ func (c *Collector) sweep(ctx context.Context) {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sweepUploads reaps docker blob-upload staging objects abandoned longer than
|
||||||
|
// uploadGracePeriod (cancelled or interrupted pushes that never finalised).
|
||||||
|
func (c *Collector) sweepUploads(ctx context.Context) {
|
||||||
|
stale, err := c.store.ListStaleObjects(ctx, "uploads/", time.Now().Add(-uploadGracePeriod))
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("gc: list stale uploads", "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
reaped := 0
|
||||||
|
for _, key := range stale {
|
||||||
|
if err := c.store.Delete(ctx, key); err != nil {
|
||||||
|
slog.Warn("gc: delete stale upload", "key", key, "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
reaped++
|
||||||
|
}
|
||||||
|
if reaped > 0 {
|
||||||
|
slog.Info("gc: reaped stale docker uploads", "count", reaped)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,199 @@
|
|||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto"
|
||||||
|
"crypto/rand"
|
||||||
|
"crypto/rsa"
|
||||||
|
"crypto/sha256"
|
||||||
|
"crypto/x509"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"encoding/pem"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
defaultAPIBase = "https://api.github.com"
|
||||||
|
|
||||||
|
// jwtLifetime is how long the app JWT is valid. GitHub caps it at 10 minutes;
|
||||||
|
// 9 leaves headroom for clock skew.
|
||||||
|
jwtLifetime = 9 * time.Minute
|
||||||
|
// jwtBackdate backdates iat to tolerate the app server's clock running behind
|
||||||
|
// GitHub's, which otherwise rejects the JWT.
|
||||||
|
jwtBackdate = 60 * time.Second
|
||||||
|
// refreshSkew refreshes the installation token this long before it expires so
|
||||||
|
// a request never races an expiry.
|
||||||
|
refreshSkew = 5 * time.Minute
|
||||||
|
)
|
||||||
|
|
||||||
|
type httpDoer interface {
|
||||||
|
Do(*http.Request) (*http.Response, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// appCredential mints installation access tokens for a GitHub App. It signs a
|
||||||
|
// short-lived RS256 JWT with the app private key, exchanges it for a ~1h
|
||||||
|
// installation token, caches that token, and refreshes it shortly before expiry.
|
||||||
|
// Refreshes are single-flighted by holding the mutex across the exchange, so
|
||||||
|
// concurrent callers coalesce onto one HTTP request and reuse the cached token.
|
||||||
|
type appCredential struct {
|
||||||
|
appID string
|
||||||
|
installationID string
|
||||||
|
key *rsa.PrivateKey
|
||||||
|
apiBase string
|
||||||
|
client httpDoer
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
token string
|
||||||
|
expiry time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func newAppCredential(opts Options) (*appCredential, error) {
|
||||||
|
if opts.AppID == "" {
|
||||||
|
return nil, errors.New("github app: GITHUB_APP_ID is required")
|
||||||
|
}
|
||||||
|
if opts.InstallationID == "" {
|
||||||
|
return nil, errors.New("github app: GITHUB_APP_INSTALLATION_ID is required")
|
||||||
|
}
|
||||||
|
pemBytes, err := loadPrivateKeyPEM(opts)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
key, err := parseRSAPrivateKey(pemBytes)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
apiBase := opts.apiBaseURL
|
||||||
|
if apiBase == "" {
|
||||||
|
apiBase = defaultAPIBase
|
||||||
|
}
|
||||||
|
client := opts.httpClient
|
||||||
|
if client == nil {
|
||||||
|
client = &http.Client{Timeout: 30 * time.Second}
|
||||||
|
}
|
||||||
|
|
||||||
|
return &appCredential{
|
||||||
|
appID: opts.AppID,
|
||||||
|
installationID: opts.InstallationID,
|
||||||
|
key: key,
|
||||||
|
apiBase: strings.TrimRight(apiBase, "/"),
|
||||||
|
client: client,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Token returns a cached installation token, refreshing it under a single-flight
|
||||||
|
// lock when it is missing or within refreshSkew of expiry.
|
||||||
|
func (a *appCredential) Token(ctx context.Context) (string, error) {
|
||||||
|
a.mu.Lock()
|
||||||
|
defer a.mu.Unlock()
|
||||||
|
if a.token != "" && time.Now().Before(a.expiry.Add(-refreshSkew)) {
|
||||||
|
return a.token, nil
|
||||||
|
}
|
||||||
|
if err := a.refreshLocked(ctx); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return a.token, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *appCredential) refreshLocked(ctx context.Context) error {
|
||||||
|
jwt, err := mintJWT(a.appID, a.key, time.Now())
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("github app: mint jwt: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
u := fmt.Sprintf("%s/app/installations/%s/access_tokens", a.apiBase, a.installationID)
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, u, nil)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
req.Header.Set("Authorization", "Bearer "+jwt)
|
||||||
|
req.Header.Set("Accept", "application/vnd.github+json")
|
||||||
|
req.Header.Set("X-GitHub-Api-Version", "2022-11-28")
|
||||||
|
|
||||||
|
resp, err := a.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("github app: token exchange: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||||
|
if resp.StatusCode != http.StatusCreated && resp.StatusCode != http.StatusOK {
|
||||||
|
// Never echo the body verbatim — it can contain sensitive material.
|
||||||
|
return fmt.Errorf("github app: token exchange status %d", resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
var out struct {
|
||||||
|
Token string `json:"token"`
|
||||||
|
ExpiresAt time.Time `json:"expires_at"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(body, &out); err != nil {
|
||||||
|
return fmt.Errorf("github app: decode token response: %w", err)
|
||||||
|
}
|
||||||
|
if out.Token == "" {
|
||||||
|
return errors.New("github app: token exchange returned an empty token")
|
||||||
|
}
|
||||||
|
a.token = out.Token
|
||||||
|
a.expiry = out.ExpiresAt
|
||||||
|
if a.expiry.IsZero() {
|
||||||
|
// Defensive: assume the documented ~1h lifetime if GitHub omits it.
|
||||||
|
a.expiry = time.Now().Add(time.Hour)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// mintJWT builds and RS256-signs a GitHub App JWT (iss=app id, backdated iat,
|
||||||
|
// ≤10m exp) using stdlib crypto — no third-party JWT dependency.
|
||||||
|
func mintJWT(appID string, key *rsa.PrivateKey, now time.Time) (string, error) {
|
||||||
|
header := map[string]string{"alg": "RS256", "typ": "JWT"}
|
||||||
|
claims := map[string]any{
|
||||||
|
"iat": now.Add(-jwtBackdate).Unix(),
|
||||||
|
"exp": now.Add(jwtLifetime).Unix(),
|
||||||
|
"iss": appID,
|
||||||
|
}
|
||||||
|
hb, err := json.Marshal(header)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
cb, err := json.Marshal(claims)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
signingInput := b64url(hb) + "." + b64url(cb)
|
||||||
|
digest := sha256.Sum256([]byte(signingInput))
|
||||||
|
sig, err := rsa.SignPKCS1v15(rand.Reader, key, crypto.SHA256, digest[:])
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return signingInput + "." + b64url(sig), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func b64url(b []byte) string {
|
||||||
|
return base64.RawURLEncoding.EncodeToString(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseRSAPrivateKey accepts PKCS#1 ("RSA PRIVATE KEY") and PKCS#8 ("PRIVATE
|
||||||
|
// KEY") PEM, covering both GitHub App key export formats.
|
||||||
|
func parseRSAPrivateKey(pemBytes []byte) (*rsa.PrivateKey, error) {
|
||||||
|
block, _ := pem.Decode(pemBytes)
|
||||||
|
if block == nil {
|
||||||
|
return nil, errors.New("github app: private key is not valid PEM")
|
||||||
|
}
|
||||||
|
if key, err := x509.ParsePKCS1PrivateKey(block.Bytes); err == nil {
|
||||||
|
return key, nil
|
||||||
|
}
|
||||||
|
keyAny, err := x509.ParsePKCS8PrivateKey(block.Bytes)
|
||||||
|
if err != nil {
|
||||||
|
return nil, errors.New("github app: private key is not a supported RSA PKCS#1/PKCS#8 key")
|
||||||
|
}
|
||||||
|
rsaKey, ok := keyAny.(*rsa.PrivateKey)
|
||||||
|
if !ok {
|
||||||
|
return nil, errors.New("github app: private key is not an RSA key")
|
||||||
|
}
|
||||||
|
return rsaKey, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto"
|
||||||
|
"crypto/rand"
|
||||||
|
"crypto/rsa"
|
||||||
|
"crypto/sha256"
|
||||||
|
"crypto/x509"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"encoding/pem"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func testRSAKeyPEM(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
key, err := rsa.GenerateKey(rand.Reader, 2048)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("generate key: %v", err)
|
||||||
|
}
|
||||||
|
der := x509.MarshalPKCS1PrivateKey(key)
|
||||||
|
return string(pem.EncodeToMemory(&pem.Block{Type: "RSA PRIVATE KEY", Bytes: der}))
|
||||||
|
}
|
||||||
|
|
||||||
|
// appFixture serves the installation-token exchange endpoint, records requests,
|
||||||
|
// verifies the presented JWT against the app public key, and returns tokens with
|
||||||
|
// a controllable expiry.
|
||||||
|
type appFixture struct {
|
||||||
|
srv *httptest.Server
|
||||||
|
pub *rsa.PublicKey
|
||||||
|
mu sync.Mutex
|
||||||
|
exchanges int
|
||||||
|
lastJWT string
|
||||||
|
expiresAt func() time.Time
|
||||||
|
tokenSeq int
|
||||||
|
}
|
||||||
|
|
||||||
|
func newAppFixture(t *testing.T, pemKey string) *appFixture {
|
||||||
|
t.Helper()
|
||||||
|
block, _ := pem.Decode([]byte(pemKey))
|
||||||
|
key, err := x509.ParsePKCS1PrivateKey(block.Bytes)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse test key: %v", err)
|
||||||
|
}
|
||||||
|
f := &appFixture{
|
||||||
|
pub: &key.PublicKey,
|
||||||
|
expiresAt: func() time.Time { return time.Now().Add(time.Hour) },
|
||||||
|
}
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("/app/installations/456/access_tokens", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
auth := r.Header.Get("Authorization")
|
||||||
|
jwt := strings.TrimPrefix(auth, "Bearer ")
|
||||||
|
f.mu.Lock()
|
||||||
|
f.exchanges++
|
||||||
|
f.lastJWT = jwt
|
||||||
|
f.tokenSeq++
|
||||||
|
seq := f.tokenSeq
|
||||||
|
exp := f.expiresAt()
|
||||||
|
f.mu.Unlock()
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusCreated)
|
||||||
|
json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"token": fmt.Sprintf("ghs_installation_%d", seq),
|
||||||
|
"expires_at": exp.UTC().Format(time.RFC3339),
|
||||||
|
})
|
||||||
|
})
|
||||||
|
f.srv = httptest.NewServer(mux)
|
||||||
|
t.Cleanup(f.srv.Close)
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *appFixture) verifyJWT(t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
f.mu.Lock()
|
||||||
|
jwt := f.lastJWT
|
||||||
|
f.mu.Unlock()
|
||||||
|
parts := strings.Split(jwt, ".")
|
||||||
|
if len(parts) != 3 {
|
||||||
|
t.Fatalf("jwt not three-part: %q", jwt)
|
||||||
|
}
|
||||||
|
signingInput := parts[0] + "." + parts[1]
|
||||||
|
sig, err := base64.RawURLEncoding.DecodeString(parts[2])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decode sig: %v", err)
|
||||||
|
}
|
||||||
|
digest := sha256.Sum256([]byte(signingInput))
|
||||||
|
if err := rsa.VerifyPKCS1v15(f.pub, crypto.SHA256, digest[:], sig); err != nil {
|
||||||
|
t.Fatalf("jwt signature invalid: %v", err)
|
||||||
|
}
|
||||||
|
var claims struct {
|
||||||
|
Iss string `json:"iss"`
|
||||||
|
Iat int64 `json:"iat"`
|
||||||
|
Exp int64 `json:"exp"`
|
||||||
|
}
|
||||||
|
cb, _ := base64.RawURLEncoding.DecodeString(parts[1])
|
||||||
|
if err := json.Unmarshal(cb, &claims); err != nil {
|
||||||
|
t.Fatalf("decode claims: %v", err)
|
||||||
|
}
|
||||||
|
if claims.Iss != "123" {
|
||||||
|
t.Fatalf("iss = %q, want 123", claims.Iss)
|
||||||
|
}
|
||||||
|
if claims.Exp-claims.Iat > int64((10*time.Minute)/time.Second) {
|
||||||
|
t.Fatalf("jwt lifetime exceeds 10m: iat=%d exp=%d", claims.Iat, claims.Exp)
|
||||||
|
}
|
||||||
|
if claims.Iat > time.Now().Unix() {
|
||||||
|
t.Fatalf("iat not backdated: %d", claims.Iat)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newAppCred(t *testing.T, f *appFixture, pemKey string) *appCredential {
|
||||||
|
t.Helper()
|
||||||
|
c, err := newAppCredential(Options{
|
||||||
|
AppID: "123",
|
||||||
|
InstallationID: "456",
|
||||||
|
PrivateKeyPEM: pemKey,
|
||||||
|
apiBaseURL: f.srv.URL,
|
||||||
|
httpClient: f.srv.Client(),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("newAppCredential: %v", err)
|
||||||
|
}
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_MintsJWTAndExchangesForInstallationToken(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
tok, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token: %v", err)
|
||||||
|
}
|
||||||
|
if tok != "ghs_installation_1" {
|
||||||
|
t.Fatalf("token = %q, want ghs_installation_1", tok)
|
||||||
|
}
|
||||||
|
if f.exchanges != 1 {
|
||||||
|
t.Fatalf("exchanges = %d, want 1", f.exchanges)
|
||||||
|
}
|
||||||
|
f.verifyJWT(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_CachesInstallationToken(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
for i := 0; i < 5; i++ {
|
||||||
|
if _, err := c.Token(context.Background()); err != nil {
|
||||||
|
t.Fatalf("token: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if f.exchanges != 1 {
|
||||||
|
t.Fatalf("exchanges = %d, want 1 (token should be cached)", f.exchanges)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_RefreshesNearExpiry(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
// Token expires within refreshSkew, so every call must re-exchange.
|
||||||
|
f.expiresAt = func() time.Time { return time.Now().Add(2 * time.Minute) }
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
t1, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token 1: %v", err)
|
||||||
|
}
|
||||||
|
t2, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token 2: %v", err)
|
||||||
|
}
|
||||||
|
if f.exchanges != 2 {
|
||||||
|
t.Fatalf("exchanges = %d, want 2 (near-expiry token must refresh)", f.exchanges)
|
||||||
|
}
|
||||||
|
if t1 == t2 {
|
||||||
|
t.Fatalf("expected a fresh token after refresh, both = %q", t1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestApp_ConcurrentTokenSingleFlights(t *testing.T) {
|
||||||
|
pemKey := testRSAKeyPEM(t)
|
||||||
|
f := newAppFixture(t, pemKey)
|
||||||
|
c := newAppCred(t, f, pemKey)
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < 20; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
if _, err := c.Token(context.Background()); err != nil {
|
||||||
|
t.Errorf("token: %v", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if f.exchanges != 1 {
|
||||||
|
t.Fatalf("exchanges = %d, want 1 (concurrent calls must coalesce)", f.exchanges)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
// Package githubauth provides the process-wide GitHub machine credential used to
|
||||||
|
// authenticate every outbound GitHub request (releases scan, ranged asset header
|
||||||
|
// fetches, and the generic-github byte proxy for private assets). The credential
|
||||||
|
// is delivered via env/secret only — it is never stored per-remote in the DB,
|
||||||
|
// never returned by any API, and never logged.
|
||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Credential yields a bearer token for GitHub requests. Token may block to mint
|
||||||
|
// or refresh (the GitHub App path); an empty string means "no auth", which only
|
||||||
|
// happens when no credential is configured.
|
||||||
|
type Credential interface {
|
||||||
|
Token(ctx context.Context) (string, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Options is the raw, env-sourced auth configuration. Exactly one mode may be
|
||||||
|
// configured: a static token, or a GitHub App (id + installation id + private
|
||||||
|
// key). Partial App configuration is an error (fail closed); no fields at all is
|
||||||
|
// fine and yields a nil credential (anonymous, current behavior).
|
||||||
|
type Options struct {
|
||||||
|
// Token is a Personal Access Token (fine-grained or classic) sent verbatim
|
||||||
|
// as "Authorization: Bearer <token>".
|
||||||
|
Token string
|
||||||
|
|
||||||
|
// GitHub App fields. PrivateKeyPEM and PrivateKeyPath are alternatives; the
|
||||||
|
// inline PEM wins when both are set.
|
||||||
|
AppID string
|
||||||
|
InstallationID string
|
||||||
|
PrivateKeyPEM string
|
||||||
|
PrivateKeyPath string
|
||||||
|
|
||||||
|
// apiBaseURL overrides https://api.github.com for tests. Empty uses the real
|
||||||
|
// endpoint. httpClient likewise overrides the default client for tests.
|
||||||
|
apiBaseURL string
|
||||||
|
httpClient httpDoer
|
||||||
|
}
|
||||||
|
|
||||||
|
// New builds the process credential from options, validating that auth is either
|
||||||
|
// fully configured or fully absent. It returns (nil, nil) when nothing is set.
|
||||||
|
func New(opts Options) (Credential, error) {
|
||||||
|
hasToken := opts.Token != ""
|
||||||
|
hasAppField := opts.AppID != "" || opts.InstallationID != "" ||
|
||||||
|
opts.PrivateKeyPEM != "" || opts.PrivateKeyPath != ""
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case !hasToken && !hasAppField:
|
||||||
|
return nil, nil // no auth configured — anonymous is fine
|
||||||
|
case hasToken && hasAppField:
|
||||||
|
return nil, errors.New("github auth: both a token and GitHub App fields are set; configure exactly one")
|
||||||
|
case hasToken:
|
||||||
|
return staticToken{token: opts.Token}, nil
|
||||||
|
default:
|
||||||
|
return newAppCredential(opts)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// staticToken is a fixed PAT credential.
|
||||||
|
type staticToken struct{ token string }
|
||||||
|
|
||||||
|
func (s staticToken) Token(context.Context) (string, error) { return s.token, nil }
|
||||||
|
|
||||||
|
// server is the process-wide credential set once at startup. A nil value means
|
||||||
|
// no server credential (anonymous). Access is guarded so a late SetServer in a
|
||||||
|
// test is race-free.
|
||||||
|
var (
|
||||||
|
serverMu sync.RWMutex
|
||||||
|
server Credential
|
||||||
|
)
|
||||||
|
|
||||||
|
// SetServer installs the process credential. Call once during startup.
|
||||||
|
func SetServer(c Credential) {
|
||||||
|
serverMu.Lock()
|
||||||
|
server = c
|
||||||
|
serverMu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Server returns the process credential, or nil if none is configured.
|
||||||
|
func Server() Credential {
|
||||||
|
serverMu.RLock()
|
||||||
|
defer serverMu.RUnlock()
|
||||||
|
return server
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadPrivateKeyPEM resolves the App private key bytes from the inline PEM or a
|
||||||
|
// file path, without ever returning the key material in an error message.
|
||||||
|
func loadPrivateKeyPEM(opts Options) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(opts.PrivateKeyPEM) != "" {
|
||||||
|
return []byte(opts.PrivateKeyPEM), nil
|
||||||
|
}
|
||||||
|
if opts.PrivateKeyPath != "" {
|
||||||
|
b, err := os.ReadFile(opts.PrivateKeyPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("github app: read private key file: %w", err)
|
||||||
|
}
|
||||||
|
return b, nil
|
||||||
|
}
|
||||||
|
return nil, errors.New("github app: no private key configured")
|
||||||
|
}
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
package githubauth
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNew_NoConfigIsAnonymous(t *testing.T) {
|
||||||
|
c, err := New(Options{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if c != nil {
|
||||||
|
t.Fatalf("expected nil credential when nothing configured, got %T", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_TokenMode(t *testing.T) {
|
||||||
|
c, err := New(Options{Token: "ghp_example"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
tok, err := c.Token(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("token: %v", err)
|
||||||
|
}
|
||||||
|
if tok != "ghp_example" {
|
||||||
|
t.Fatalf("token = %q, want ghp_example", tok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_TokenAndAppConflict(t *testing.T) {
|
||||||
|
_, err := New(Options{Token: "ghp_example", AppID: "123"})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error when both token and app fields are set")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_PartialAppFailsClosed(t *testing.T) {
|
||||||
|
cases := map[string]Options{
|
||||||
|
"app id without key": {AppID: "123", InstallationID: "456"},
|
||||||
|
"key without app id": {InstallationID: "456", PrivateKeyPEM: testRSAKeyPEM(t)},
|
||||||
|
"app id without inst": {AppID: "123", PrivateKeyPEM: testRSAKeyPEM(t)},
|
||||||
|
}
|
||||||
|
for name, opts := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
if _, err := New(opts); err == nil {
|
||||||
|
t.Fatalf("expected fail-closed error for %q", name)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_AppModeParsesKey(t *testing.T) {
|
||||||
|
c, err := New(Options{
|
||||||
|
AppID: "123",
|
||||||
|
InstallationID: "456",
|
||||||
|
PrivateKeyPEM: testRSAKeyPEM(t),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unexpected error: %v", err)
|
||||||
|
}
|
||||||
|
if _, ok := c.(*appCredential); !ok {
|
||||||
|
t.Fatalf("expected *appCredential, got %T", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNew_AppModeRejectsBadKey(t *testing.T) {
|
||||||
|
_, err := New(Options{
|
||||||
|
AppID: "123",
|
||||||
|
InstallationID: "456",
|
||||||
|
PrivateKeyPEM: "-----BEGIN RSA PRIVATE KEY-----\nnope\n-----END RSA PRIVATE KEY-----",
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error for malformed private key")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,447 @@
|
|||||||
|
package deb
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"compress/gzip"
|
||||||
|
"context"
|
||||||
|
"crypto/md5"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"path"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/klauspost/compress/zstd"
|
||||||
|
"github.com/ulikunitz/xz"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/auth"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/storage"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
provider.Register(&Provider{})
|
||||||
|
}
|
||||||
|
|
||||||
|
// mutableRe marks the apt index surface (both the flat local repo and a proxied
|
||||||
|
// Debian/Ubuntu mirror's dists/ tree) so the caching engine revalidates it
|
||||||
|
// instead of freezing it like an immutable .deb.
|
||||||
|
var mutableRe = []*regexp.Regexp{
|
||||||
|
regexp.MustCompile(`(^|/)Packages(\.gz|\.xz|\.bz2)?$`),
|
||||||
|
regexp.MustCompile(`(^|/)Sources(\.gz|\.xz|\.bz2)?$`),
|
||||||
|
regexp.MustCompile(`(^|/)Release$`),
|
||||||
|
regexp.MustCompile(`(^|/)InRelease$`),
|
||||||
|
regexp.MustCompile(`(^|/)Release\.gpg$`),
|
||||||
|
regexp.MustCompile(`(^|/)Contents-`),
|
||||||
|
regexp.MustCompile(`^dists/`),
|
||||||
|
regexp.MustCompile(`/by-hash/`),
|
||||||
|
}
|
||||||
|
|
||||||
|
type Provider struct{}
|
||||||
|
|
||||||
|
func (p *Provider) Type() models.PackageType { return models.PackageDeb }
|
||||||
|
|
||||||
|
func (p *Provider) Classify(path string) provider.Mutability {
|
||||||
|
for _, re := range mutableRe {
|
||||||
|
if re.MatchString(path) {
|
||||||
|
return provider.Mutable
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return provider.Immutable
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) ContentType(path string) string {
|
||||||
|
switch {
|
||||||
|
case strings.HasSuffix(path, ".deb"):
|
||||||
|
return "application/vnd.debian.binary-package"
|
||||||
|
case strings.HasSuffix(path, ".gz"):
|
||||||
|
return "application/gzip"
|
||||||
|
case strings.HasSuffix(path, ".xz"):
|
||||||
|
return "application/x-xz"
|
||||||
|
case strings.HasSuffix(path, "Packages"), strings.HasSuffix(path, "Release"),
|
||||||
|
strings.HasSuffix(path, "InRelease"), strings.HasSuffix(path, "Sources"):
|
||||||
|
return "text/plain"
|
||||||
|
}
|
||||||
|
return "application/octet-stream"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) UpstreamURL(remote models.Remote, path string) string {
|
||||||
|
return strings.TrimRight(remote.BaseURL, "/") + "/" + strings.TrimLeft(path, "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
|
||||||
|
return auth.BasicHeaders(remote), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) ValidateUpload(filePath string) (storagePath, contentType string, err error) {
|
||||||
|
filename := filePath
|
||||||
|
if idx := strings.LastIndex(filePath, "/"); idx >= 0 {
|
||||||
|
filename = filePath[idx+1:]
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.HasSuffix(strings.ToLower(filename), ".deb") {
|
||||||
|
return "", "", fmt.Errorf("file must be a .deb package")
|
||||||
|
}
|
||||||
|
|
||||||
|
return "pool/" + filename, "application/vnd.debian.binary-package", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any {
|
||||||
|
filename := strings.TrimPrefix(storagePath, "pool/")
|
||||||
|
return map[string]any{
|
||||||
|
"filename": filename,
|
||||||
|
"content_hash": contentHash,
|
||||||
|
"size_bytes": sizeBytes,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs provider.BlobReader, db provider.MetadataStore) {
|
||||||
|
s3Key := storage.BlobKey(strings.TrimPrefix(contentHash, "sha256:"))
|
||||||
|
|
||||||
|
reader, blobSize, err := blobs.Download(ctx, s3Key)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("deb metadata: download failed", "repo", repoName, "path", storagePath, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer reader.Close()
|
||||||
|
|
||||||
|
raw, err := io.ReadAll(reader)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("deb metadata: read failed", "repo", repoName, "path", storagePath, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
control, err := extractControl(raw)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("deb metadata: parse failed", "repo", repoName, "path", storagePath, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fields := parseControlFields(control)
|
||||||
|
|
||||||
|
sum := md5.Sum(raw)
|
||||||
|
meta := &provider.DebMetadata{
|
||||||
|
RepoName: repoName,
|
||||||
|
FilePath: storagePath,
|
||||||
|
ContentHash: contentHash,
|
||||||
|
Name: fields["Package"],
|
||||||
|
Version: fields["Version"],
|
||||||
|
Architecture: fields["Architecture"],
|
||||||
|
Control: strings.TrimRight(control, "\n"),
|
||||||
|
Size: blobSize,
|
||||||
|
MD5: hex.EncodeToString(sum[:]),
|
||||||
|
SHA256: strings.TrimPrefix(contentHash, "sha256:"),
|
||||||
|
}
|
||||||
|
|
||||||
|
if meta.Name == "" {
|
||||||
|
slog.Error("deb metadata: control missing Package field", "repo", repoName, "path", storagePath)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := db.InsertDebMetadata(ctx, meta); err != nil {
|
||||||
|
slog.Error("deb metadata: insert failed", "repo", repoName, "path", storagePath, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("deb metadata: parsed", "repo", repoName, "name", meta.Name, "version", meta.Version, "arch", meta.Architecture)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) AfterDelete(ctx context.Context, repoName, storagePath string, db provider.MetadataDeleter) error {
|
||||||
|
if err := db.DeleteDebMetadata(ctx, repoName, storagePath); err != nil {
|
||||||
|
slog.Error("deb metadata: delete failed", "repo", repoName, "path", storagePath, "error", err)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
slog.Info("deb metadata: deleted", "repo", repoName, "path", storagePath)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// extractControl reads a .deb (an ar archive), locates the control.tar.* member,
|
||||||
|
// decompresses it, and returns the raw ./control paragraph. Pure Go: no dpkg.
|
||||||
|
func extractControl(deb []byte) (string, error) {
|
||||||
|
members, err := readAr(deb)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
var name string
|
||||||
|
var data []byte
|
||||||
|
for _, m := range members {
|
||||||
|
if strings.HasPrefix(m.name, "control.tar") {
|
||||||
|
name = m.name
|
||||||
|
data = m.data
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if data == nil {
|
||||||
|
return "", errors.New("no control.tar member in .deb")
|
||||||
|
}
|
||||||
|
|
||||||
|
tarBytes, err := decompress(name, data)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
return readControlParagraph(tarBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
// readControlParagraph scans a decompressed control.tar and returns the raw
|
||||||
|
// ./control paragraph. Shared by the local upload path (extractControl) and the
|
||||||
|
// github_deb ranged-prefix parser.
|
||||||
|
func readControlParagraph(controlTar []byte) (string, error) {
|
||||||
|
tr := tar.NewReader(bytes.NewReader(controlTar))
|
||||||
|
for {
|
||||||
|
hdr, err := tr.Next()
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("read control.tar: %w", err)
|
||||||
|
}
|
||||||
|
clean := strings.TrimPrefix(hdr.Name, "./")
|
||||||
|
if clean == "control" {
|
||||||
|
b, err := io.ReadAll(tr)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("read control file: %w", err)
|
||||||
|
}
|
||||||
|
return string(b), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", errors.New("no ./control in control.tar")
|
||||||
|
}
|
||||||
|
|
||||||
|
func decompress(name string, data []byte) ([]byte, error) {
|
||||||
|
switch {
|
||||||
|
case strings.HasSuffix(name, ".gz"):
|
||||||
|
zr, err := gzip.NewReader(bytes.NewReader(data))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer zr.Close()
|
||||||
|
return io.ReadAll(zr)
|
||||||
|
case strings.HasSuffix(name, ".xz"):
|
||||||
|
xr, err := xz.NewReader(bytes.NewReader(data))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return io.ReadAll(xr)
|
||||||
|
case strings.HasSuffix(name, ".zst"):
|
||||||
|
zr, err := zstd.NewReader(bytes.NewReader(data))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer zr.Close()
|
||||||
|
return io.ReadAll(zr)
|
||||||
|
case strings.HasSuffix(name, ".tar"):
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("unsupported control.tar compression: %s", name)
|
||||||
|
}
|
||||||
|
|
||||||
|
type arMember struct {
|
||||||
|
name string
|
||||||
|
data []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
// readAr parses the (trivial) Unix ar archive that wraps a .deb. Each member has
|
||||||
|
// a 60-byte header; the size field is decimal ASCII and data is padded to an
|
||||||
|
// even offset.
|
||||||
|
func readAr(data []byte) ([]arMember, error) {
|
||||||
|
const magic = "!<arch>\n"
|
||||||
|
if len(data) < len(magic) || string(data[:len(magic)]) != magic {
|
||||||
|
return nil, errors.New("not an ar archive")
|
||||||
|
}
|
||||||
|
off := len(magic)
|
||||||
|
|
||||||
|
var members []arMember
|
||||||
|
for off+60 <= len(data) {
|
||||||
|
hdr := data[off : off+60]
|
||||||
|
off += 60
|
||||||
|
|
||||||
|
name := strings.TrimRight(string(hdr[0:16]), " ")
|
||||||
|
name = strings.TrimSuffix(name, "/")
|
||||||
|
size, err := strconv.ParseInt(strings.TrimSpace(string(hdr[48:58])), 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("bad ar size for %q: %w", name, err)
|
||||||
|
}
|
||||||
|
if off+int(size) > len(data) {
|
||||||
|
return nil, fmt.Errorf("truncated ar member %q", name)
|
||||||
|
}
|
||||||
|
members = append(members, arMember{name: name, data: data[off : off+int(size)]})
|
||||||
|
off += int(size)
|
||||||
|
if size%2 == 1 {
|
||||||
|
off++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return members, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseControlFields reads the single-line fields of an RFC822-style control
|
||||||
|
// paragraph. Continuation lines (leading whitespace) belong to the previous
|
||||||
|
// field and are ignored here since only Package/Version/Architecture are read.
|
||||||
|
func parseControlFields(control string) map[string]string {
|
||||||
|
fields := map[string]string{}
|
||||||
|
sc := bufio.NewScanner(strings.NewReader(control))
|
||||||
|
sc.Buffer(make([]byte, 0, 64*1024), 1024*1024)
|
||||||
|
for sc.Scan() {
|
||||||
|
line := sc.Text()
|
||||||
|
if line == "" || line[0] == ' ' || line[0] == '\t' {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
idx := strings.IndexByte(line, ':')
|
||||||
|
if idx < 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := strings.TrimSpace(line[:idx])
|
||||||
|
if _, seen := fields[key]; seen {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fields[key] = strings.TrimSpace(line[idx+1:])
|
||||||
|
}
|
||||||
|
return fields
|
||||||
|
}
|
||||||
|
|
||||||
|
// normalizeIndexPath collapses apt's verbatim dist prefix from a flat-repo
|
||||||
|
// request. For `deb ... <repo>/ ./`, apt appends the "./" dist literally and asks
|
||||||
|
// for "./Packages" (and "./Release", "./InRelease"); dot-segments must be
|
||||||
|
// collapsed so the index matcher sees "Packages". A no-op for pool/*.deb paths.
|
||||||
|
func normalizeIndexPath(p string) string {
|
||||||
|
return strings.TrimPrefix(path.Clean("/"+p), "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) ServeLocalIndex(w http.ResponseWriter, r *http.Request, files provider.FileStore, repoName, reqPath string) bool {
|
||||||
|
path := normalizeIndexPath(reqPath)
|
||||||
|
switch path {
|
||||||
|
case "Packages", "Packages.gz", "Release":
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, ok := files.(provider.DebMetadataReader)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "deb metadata not available", http.StatusInternalServerError)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
metas, err := reader.ListDebMetadataEntries(r.Context(), repoName)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
slog.Warn("deb: metadata read canceled", "repo", repoName, "error", err)
|
||||||
|
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
switch path {
|
||||||
|
case "Packages":
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write(generatePackages(metas))
|
||||||
|
case "Packages.gz":
|
||||||
|
w.Header().Set("Content-Type", "application/gzip")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write(gzipBytes(generatePackages(metas)))
|
||||||
|
case "Release":
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write(generateRelease(metas))
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileStore, repoName, path string) ([]byte, error) {
|
||||||
|
return nil, fmt.Errorf("deb local index generation for virtual repos not supported")
|
||||||
|
}
|
||||||
|
|
||||||
|
// generatePackages emits the flat-repo Packages file: each package's raw control
|
||||||
|
// stanza followed by the apt-required Filename/Size/MD5sum/SHA256 fields, blank
|
||||||
|
// line separated.
|
||||||
|
func generatePackages(metas []provider.DebMetadata) []byte {
|
||||||
|
var b bytes.Buffer
|
||||||
|
for _, m := range metas {
|
||||||
|
b.WriteString(strings.TrimRight(m.Control, "\n"))
|
||||||
|
b.WriteString("\n")
|
||||||
|
fmt.Fprintf(&b, "Filename: %s\n", m.FilePath)
|
||||||
|
fmt.Fprintf(&b, "Size: %d\n", m.Size)
|
||||||
|
if m.MD5 != "" {
|
||||||
|
fmt.Fprintf(&b, "MD5sum: %s\n", m.MD5)
|
||||||
|
}
|
||||||
|
if m.SHA256 != "" {
|
||||||
|
fmt.Fprintf(&b, "SHA256: %s\n", m.SHA256)
|
||||||
|
}
|
||||||
|
b.WriteString("\n")
|
||||||
|
}
|
||||||
|
return b.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func generateRelease(metas []provider.DebMetadata) []byte {
|
||||||
|
packages := generatePackages(metas)
|
||||||
|
packagesGz := gzipBytes(packages)
|
||||||
|
|
||||||
|
arches := uniqueArches(metas)
|
||||||
|
|
||||||
|
var b bytes.Buffer
|
||||||
|
fmt.Fprintf(&b, "Date: %s\n", time.Now().UTC().Format(time.RFC1123Z))
|
||||||
|
fmt.Fprintf(&b, "Architectures: %s\n", strings.Join(arches, " "))
|
||||||
|
b.WriteString("Acquire-By-Hash: no\n")
|
||||||
|
|
||||||
|
b.WriteString("MD5Sum:\n")
|
||||||
|
writeReleaseEntry(&b, md5Hex(packages), len(packages), "Packages")
|
||||||
|
writeReleaseEntry(&b, md5Hex(packagesGz), len(packagesGz), "Packages.gz")
|
||||||
|
|
||||||
|
b.WriteString("SHA256:\n")
|
||||||
|
writeReleaseEntry(&b, sha256Hex(packages), len(packages), "Packages")
|
||||||
|
writeReleaseEntry(&b, sha256Hex(packagesGz), len(packagesGz), "Packages.gz")
|
||||||
|
|
||||||
|
return b.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeReleaseEntry(b *bytes.Buffer, hash string, size int, name string) {
|
||||||
|
fmt.Fprintf(b, " %s %d %s\n", hash, size, name)
|
||||||
|
}
|
||||||
|
|
||||||
|
func uniqueArches(metas []provider.DebMetadata) []string {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
var out []string
|
||||||
|
for _, m := range metas {
|
||||||
|
a := m.Architecture
|
||||||
|
if a == "" || seen[a] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[a] = true
|
||||||
|
out = append(out, a)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func gzipBytes(data []byte) []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
gz := gzip.NewWriter(&buf)
|
||||||
|
gz.Write(data)
|
||||||
|
gz.Close()
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func md5Hex(data []byte) string {
|
||||||
|
h := md5.Sum(data)
|
||||||
|
return hex.EncodeToString(h[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
func sha256Hex(data []byte) string {
|
||||||
|
h := sha256.Sum256(data)
|
||||||
|
return hex.EncodeToString(h[:])
|
||||||
|
}
|
||||||
@@ -0,0 +1,408 @@
|
|||||||
|
package deb
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"bytes"
|
||||||
|
"compress/gzip"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/klauspost/compress/zstd"
|
||||||
|
"github.com/ulikunitz/xz"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/testsupport"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeBlobReader struct{ data []byte }
|
||||||
|
|
||||||
|
func (f fakeBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
|
||||||
|
return io.NopCloser(bytes.NewReader(f.data)), int64(len(f.data)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type errBlobReader struct{}
|
||||||
|
|
||||||
|
func (errBlobReader) Download(_ context.Context, _ string) (io.ReadCloser, int64, error) {
|
||||||
|
return nil, 0, io.ErrUnexpectedEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
// fakeDebStore satisfies provider.MetadataStore (both insert methods) and
|
||||||
|
// records the deb row that AfterUpload writes.
|
||||||
|
type fakeDebStore struct{ inserted *provider.DebMetadata }
|
||||||
|
|
||||||
|
func (f *fakeDebStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
|
||||||
|
func (f *fakeDebStore) InsertDebMetadata(_ context.Context, m *provider.DebMetadata) error {
|
||||||
|
f.inserted = m
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeDebReader struct{ metas []provider.DebMetadata }
|
||||||
|
|
||||||
|
func (f fakeDebReader) ListDebMetadataEntries(context.Context, string) ([]provider.DebMetadata, error) {
|
||||||
|
return f.metas, nil
|
||||||
|
}
|
||||||
|
func (f fakeDebReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
func (f fakeDebReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
|
||||||
|
|
||||||
|
type errDebReader struct{}
|
||||||
|
|
||||||
|
func (errDebReader) ListDebMetadataEntries(context.Context, string) ([]provider.DebMetadata, error) {
|
||||||
|
return nil, io.ErrUnexpectedEOF
|
||||||
|
}
|
||||||
|
func (errDebReader) ListFilesByPrefix(context.Context, string, string) ([]provider.FileEntry, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
func (errDebReader) ListPackages(context.Context, string) ([]string, error) { return nil, nil }
|
||||||
|
|
||||||
|
func TestDebPureFuncs(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
if p.Type() != models.PackageDeb {
|
||||||
|
t.Errorf("type = %q", p.Type())
|
||||||
|
}
|
||||||
|
if out, _ := p.RewriteResponse(nil, models.Remote{}, "http://p"); out != nil {
|
||||||
|
t.Error("deb never rewrites")
|
||||||
|
}
|
||||||
|
if got := p.UpstreamURL(models.Remote{BaseURL: "https://mirror/"}, "/dists/bookworm/Release"); got != "https://mirror/dists/bookworm/Release" {
|
||||||
|
t.Errorf("upstream url %q", got)
|
||||||
|
}
|
||||||
|
h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
|
||||||
|
if h.Get("Authorization") == "" {
|
||||||
|
t.Error("auth header")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebClassify(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
tests := []struct {
|
||||||
|
path string
|
||||||
|
want provider.Mutability
|
||||||
|
}{
|
||||||
|
{"pool/foo_1.0_amd64.deb", provider.Immutable},
|
||||||
|
{"Packages", provider.Mutable},
|
||||||
|
{"Packages.gz", provider.Mutable},
|
||||||
|
{"Release", provider.Mutable},
|
||||||
|
{"InRelease", provider.Mutable},
|
||||||
|
{"Release.gpg", provider.Mutable},
|
||||||
|
{"dists/bookworm/main/binary-amd64/Packages", provider.Mutable},
|
||||||
|
{"dists/bookworm/Release", provider.Mutable},
|
||||||
|
{"dists/bookworm/main/by-hash/SHA256/abc", provider.Mutable},
|
||||||
|
{"dists/bookworm/main/Contents-amd64.gz", provider.Mutable},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
if got := p.Classify(tt.path); got != tt.want {
|
||||||
|
t.Errorf("Classify(%q) = %v, want %v", tt.path, got, tt.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebContentType(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
for path, want := range map[string]string{
|
||||||
|
"pool/foo_1.0_amd64.deb": "application/vnd.debian.binary-package",
|
||||||
|
"dists/bookworm/main/bin/Packages.gz": "application/gzip",
|
||||||
|
"dists/bookworm/main/bin/Packages.xz": "application/x-xz",
|
||||||
|
"Packages": "text/plain",
|
||||||
|
"Release": "text/plain",
|
||||||
|
"InRelease": "text/plain",
|
||||||
|
"pool/other": "application/octet-stream",
|
||||||
|
} {
|
||||||
|
if got := p.ContentType(path); got != want {
|
||||||
|
t.Errorf("ContentType(%q) = %q, want %q", path, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebValidateUpload(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
sp, ct, err := p.ValidateUpload("dir/foo_1.0_amd64.deb")
|
||||||
|
if err != nil || sp != "pool/foo_1.0_amd64.deb" || ct != "application/vnd.debian.binary-package" {
|
||||||
|
t.Errorf("sp=%q ct=%q err=%v", sp, ct, err)
|
||||||
|
}
|
||||||
|
if _, _, err := p.ValidateUpload("foo.rpm"); err == nil {
|
||||||
|
t.Error("expected error for non-deb")
|
||||||
|
}
|
||||||
|
resp := p.UploadResponse("pool/foo_1.0_amd64.deb", "sha256:abc", 42)
|
||||||
|
if resp["filename"] != "foo_1.0_amd64.deb" || resp["content_hash"] != "sha256:abc" || resp["size_bytes"] != int64(42) {
|
||||||
|
t.Errorf("upload response %v", resp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebAfterUpload(t *testing.T) {
|
||||||
|
data := testsupport.MinimalDeb("e2e-testpkg", "1.2.3", "amd64")
|
||||||
|
store := &fakeDebStore{}
|
||||||
|
(&Provider{}).AfterUpload(context.Background(), "myrepo", "pool/e2e-testpkg_1.2.3_amd64.deb",
|
||||||
|
"sha256:deadbeef", fakeBlobReader{data: data}, store)
|
||||||
|
|
||||||
|
m := store.inserted
|
||||||
|
if m == nil {
|
||||||
|
t.Fatal("no metadata inserted")
|
||||||
|
}
|
||||||
|
if m.Name != "e2e-testpkg" || m.Version != "1.2.3" || m.Architecture != "amd64" {
|
||||||
|
t.Errorf("unexpected metadata: %+v", m)
|
||||||
|
}
|
||||||
|
if m.Size != int64(len(data)) {
|
||||||
|
t.Errorf("Size = %d, want %d", m.Size, len(data))
|
||||||
|
}
|
||||||
|
if m.SHA256 != "deadbeef" {
|
||||||
|
t.Errorf("SHA256 = %q, want deadbeef", m.SHA256)
|
||||||
|
}
|
||||||
|
if m.MD5 == "" {
|
||||||
|
t.Error("MD5 not computed")
|
||||||
|
}
|
||||||
|
if !strings.Contains(m.Control, "Package: e2e-testpkg") {
|
||||||
|
t.Errorf("raw control not stored: %q", m.Control)
|
||||||
|
}
|
||||||
|
// The raw stanza is stored verbatim (no trailing newline) so Packages can
|
||||||
|
// reproduce it faithfully.
|
||||||
|
if strings.HasSuffix(m.Control, "\n") {
|
||||||
|
t.Error("control should be trimmed of trailing newline")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebAfterUploadErrors(t *testing.T) {
|
||||||
|
// Download failure: no insert, no panic.
|
||||||
|
store := &fakeDebStore{}
|
||||||
|
(&Provider{}).AfterUpload(context.Background(), "r", "p", "sha256:x", errBlobReader{}, store)
|
||||||
|
if store.inserted != nil {
|
||||||
|
t.Error("no metadata should be inserted on download error")
|
||||||
|
}
|
||||||
|
// Not a .deb (ar) archive.
|
||||||
|
store2 := &fakeDebStore{}
|
||||||
|
(&Provider{}).AfterUpload(context.Background(), "r", "p", "sha256:x", fakeBlobReader{data: []byte("not a deb")}, store2)
|
||||||
|
if store2.inserted != nil {
|
||||||
|
t.Error("no metadata should be inserted on parse error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebControlDecompression(t *testing.T) {
|
||||||
|
// The control tarball may be gzip, xz, or zstd (goreleaser/nfpm emit gzip or
|
||||||
|
// xz); each must round-trip to the same control stanza.
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
member string
|
||||||
|
comp func([]byte) []byte
|
||||||
|
}{
|
||||||
|
{"gzip", "control.tar.gz", gzipBytes},
|
||||||
|
{"xz", "control.tar.xz", xzBytes},
|
||||||
|
{"zstd", "control.tar.zst", zstdBytes},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
deb := buildDeb("pkg", "9.9", "arm64", tc.member, tc.comp)
|
||||||
|
control, err := extractControl(deb)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("extractControl: %v", err)
|
||||||
|
}
|
||||||
|
fields := parseControlFields(control)
|
||||||
|
if fields["Package"] != "pkg" || fields["Version"] != "9.9" || fields["Architecture"] != "arm64" {
|
||||||
|
t.Errorf("fields = %v", fields)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebParseControlContinuationLines(t *testing.T) {
|
||||||
|
control := "Package: p\nVersion: 1\n" +
|
||||||
|
"Description: short\n very long\n .\n more\n" +
|
||||||
|
"Architecture: all\n"
|
||||||
|
f := parseControlFields(control)
|
||||||
|
if f["Package"] != "p" || f["Version"] != "1" || f["Architecture"] != "all" {
|
||||||
|
t.Errorf("continuation lines corrupted parse: %v", f)
|
||||||
|
}
|
||||||
|
if f["Description"] != "short" {
|
||||||
|
t.Errorf("Description folded continuation into value: %q", f["Description"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebServeLocalIndex(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
reader := fakeDebReader{metas: []provider.DebMetadata{
|
||||||
|
{Name: "aaa", Version: "1.0", Architecture: "amd64", FilePath: "pool/aaa_1.0_amd64.deb",
|
||||||
|
Control: "Package: aaa\nVersion: 1.0\nArchitecture: amd64", Size: 100, MD5: "md5aaa", SHA256: "sha256aaa"},
|
||||||
|
{Name: "bbb", Version: "2.0", Architecture: "arm64", FilePath: "pool/bbb_2.0_arm64.deb",
|
||||||
|
Control: "Package: bbb\nVersion: 2.0\nArchitecture: arm64", Size: 200, MD5: "md5bbb", SHA256: "sha256bbb"},
|
||||||
|
}}
|
||||||
|
|
||||||
|
serve := func(path string) *httptest.ResponseRecorder {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
|
||||||
|
if !p.ServeLocalIndex(w, r, reader, "myrepo", path) {
|
||||||
|
t.Fatalf("ServeLocalIndex returned false for %q", path)
|
||||||
|
}
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
// Packages lists both packages with their apt fields.
|
||||||
|
w := serve("Packages")
|
||||||
|
body := w.Body.String()
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("Packages code %d", w.Code)
|
||||||
|
}
|
||||||
|
for _, want := range []string{
|
||||||
|
"Package: aaa", "Package: bbb",
|
||||||
|
"Filename: pool/aaa_1.0_amd64.deb", "Size: 100", "MD5sum: md5aaa", "SHA256: sha256aaa",
|
||||||
|
"Filename: pool/bbb_2.0_arm64.deb", "Size: 200",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(body, want) {
|
||||||
|
t.Errorf("Packages missing %q:\n%s", want, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Stanzas are blank-line separated.
|
||||||
|
if !strings.Contains(body, "SHA256: sha256aaa\n\n") {
|
||||||
|
t.Errorf("stanzas not blank-line separated:\n%s", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Packages.gz decompresses to exactly the plain Packages bytes.
|
||||||
|
w = serve("Packages.gz")
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("Packages.gz code %d", w.Code)
|
||||||
|
}
|
||||||
|
zr, err := gzip.NewReader(bytes.NewReader(w.Body.Bytes()))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Packages.gz not gzip: %v", err)
|
||||||
|
}
|
||||||
|
plain, _ := io.ReadAll(zr)
|
||||||
|
if !bytes.Equal(plain, []byte(body)) {
|
||||||
|
t.Error("Packages.gz does not decompress to Packages")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Release lists arches and both index files under MD5Sum/SHA256.
|
||||||
|
w = serve("Release")
|
||||||
|
rel := w.Body.String()
|
||||||
|
if w.Code != 200 {
|
||||||
|
t.Fatalf("Release code %d", w.Code)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"Date:", "Architectures: amd64 arm64", "Acquire-By-Hash: no", "MD5Sum:", "SHA256:", " Packages\n", " Packages.gz\n"} {
|
||||||
|
if !strings.Contains(rel, want) {
|
||||||
|
t.Errorf("Release missing %q:\n%s", want, rel)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Unsigned trust model: no InRelease / Release.gpg served here.
|
||||||
|
for _, path := range []string{"InRelease", "Release.gpg", "pool/aaa_1.0_amd64.deb"} {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
|
||||||
|
if p.ServeLocalIndex(w, r, reader, "myrepo", path) {
|
||||||
|
t.Errorf("ServeLocalIndex should return false for %q", path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Real apt appends the flat-repo dist "./" verbatim, so it requests "./Packages"
|
||||||
|
// / "./Release" (curl pre-normalizes /./ which masks this). The handler must
|
||||||
|
// collapse the dot-segment and return the same bytes as the un-prefixed request.
|
||||||
|
func TestDebServeLocalIndexAptDotSegment(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
reader := fakeDebReader{metas: []provider.DebMetadata{
|
||||||
|
{Name: "aaa", Version: "1.0", Architecture: "amd64", FilePath: "pool/aaa_1.0_amd64.deb",
|
||||||
|
Control: "Package: aaa\nVersion: 1.0\nArchitecture: amd64", Size: 100, MD5: "md5aaa", SHA256: "sha256aaa"},
|
||||||
|
}}
|
||||||
|
|
||||||
|
serve := func(path string) *httptest.ResponseRecorder {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
|
||||||
|
if !p.ServeLocalIndex(w, r, reader, "myrepo", path) {
|
||||||
|
t.Fatalf("ServeLocalIndex returned false for %q", path)
|
||||||
|
}
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
// Packages is deterministic: require exact byte identity.
|
||||||
|
if plain, dotted := serve("Packages"), serve("./Packages"); plain.Code != 200 || dotted.Code != 200 {
|
||||||
|
t.Fatalf("Packages: plain=%d dotted=%d, want 200/200", plain.Code, dotted.Code)
|
||||||
|
} else if !bytes.Equal(plain.Body.Bytes(), dotted.Body.Bytes()) {
|
||||||
|
t.Error("./Packages body differs from Packages body")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Release carries a Date: header stamped from time.Now(); compare the rest.
|
||||||
|
plain, dotted := serve("Release"), serve("./Release")
|
||||||
|
if plain.Code != 200 || dotted.Code != 200 {
|
||||||
|
t.Fatalf("Release: plain=%d dotted=%d, want 200/200", plain.Code, dotted.Code)
|
||||||
|
}
|
||||||
|
if stripDate(plain.Body.String()) != stripDate(dotted.Body.String()) {
|
||||||
|
t.Error("./Release body differs from Release body (ignoring Date)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func stripDate(s string) string {
|
||||||
|
var out []string
|
||||||
|
for _, line := range strings.Split(s, "\n") {
|
||||||
|
if strings.HasPrefix(line, "Date:") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, line)
|
||||||
|
}
|
||||||
|
return strings.Join(out, "\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebServeMetadataError(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
for _, path := range []string{"Packages", "Packages.gz", "Release"} {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
|
||||||
|
p.ServeLocalIndex(w, r, errDebReader{}, "repo", path)
|
||||||
|
if w.Code != 500 {
|
||||||
|
t.Errorf("%s with failing reader = %d, want 500", path, w.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDebGenerateLocalIndexUnsupported(t *testing.T) {
|
||||||
|
if _, err := (&Provider{}).GenerateLocalIndex(context.Background(), fakeDebReader{}, "r", "Packages"); err == nil {
|
||||||
|
t.Error("expected unsupported error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildDeb assembles an ar .deb whose control member uses the given name and
|
||||||
|
// compressor, so the decompression branches can be exercised directly.
|
||||||
|
func buildDeb(name, version, arch, member string, comp func([]byte) []byte) []byte {
|
||||||
|
control := "Package: " + name + "\nVersion: " + version + "\nArchitecture: " + arch + "\n"
|
||||||
|
controlTar := comp(tarSingle("./control", []byte(control)))
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
buf.WriteString("!<arch>\n")
|
||||||
|
arWrite(&buf, "debian-binary", []byte("2.0\n"))
|
||||||
|
arWrite(&buf, member, controlTar)
|
||||||
|
arWrite(&buf, "data.tar.gz", gzipBytes(tarSingle("./x", []byte("x"))))
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func tarSingle(name string, data []byte) []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
tw := tar.NewWriter(&buf)
|
||||||
|
tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(data)), Typeflag: tar.TypeReg})
|
||||||
|
tw.Write(data)
|
||||||
|
tw.Close()
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func arWrite(buf *bytes.Buffer, name string, data []byte) {
|
||||||
|
fmt.Fprintf(buf, "%-16s%-12s%-6s%-6s%-8s%-10d`\n", name, "0", "0", "0", "100644", len(data))
|
||||||
|
buf.Write(data)
|
||||||
|
if len(data)%2 == 1 {
|
||||||
|
buf.WriteByte('\n')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func xzBytes(data []byte) []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
w, _ := xz.NewWriter(&buf)
|
||||||
|
w.Write(data)
|
||||||
|
w.Close()
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func zstdBytes(data []byte) []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
w, _ := zstd.NewWriter(&buf)
|
||||||
|
w.Write(data)
|
||||||
|
w.Close()
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
@@ -0,0 +1,724 @@
|
|||||||
|
package deb
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/time/rate"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// gitHubProvider is the process-wide singleton for github_deb. The background
|
||||||
|
// Syncer binds its shared rate limiter and work queue onto this instance so the
|
||||||
|
// request path and the syncer drive the same derive machinery.
|
||||||
|
var gitHubProvider = newGitHubProvider()
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
provider.Register(gitHubProvider)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tuning knobs for the no-precache control fetch. A .deb is an ar archive whose
|
||||||
|
// control.tar member sits right after the tiny debian-binary member, so a small
|
||||||
|
// front prefix reliably covers it.
|
||||||
|
const (
|
||||||
|
defaultHeaderRangeInitial = 32 << 10 // 32 KiB — covers control.tar of almost every .deb
|
||||||
|
defaultHeaderRangeMax = 16 << 20 // 16 MiB — give up past this and skip the asset
|
||||||
|
defaultReleasePageCap = 10 // 100 releases/page * 10 pages
|
||||||
|
|
||||||
|
defaultScanTimeout = 10 * time.Minute
|
||||||
|
defaultServeTimeout = 30 * time.Second
|
||||||
|
defaultColdWait = 8 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// GitHubProvider is a metadata-only remote: it scans a GitHub repo's releases
|
||||||
|
// for .deb assets, derives per-asset control metadata via a ranged prefix fetch
|
||||||
|
// (never downloading whole packages), synthesizes a flat apt repository from that
|
||||||
|
// cached metadata, and redirects package downloads to a backend "releases_remote"
|
||||||
|
// (the generic github.com remote) that serves the actual bytes.
|
||||||
|
type GitHubProvider struct {
|
||||||
|
client *http.Client
|
||||||
|
|
||||||
|
headerInitial int64
|
||||||
|
headerMax int64
|
||||||
|
pageCap int
|
||||||
|
scanTimeout time.Duration
|
||||||
|
serveTimeout time.Duration
|
||||||
|
coldWait time.Duration
|
||||||
|
|
||||||
|
limiter *rate.Limiter
|
||||||
|
syncer *Syncer
|
||||||
|
|
||||||
|
serverCred githubauth.Credential
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
scanning map[string]bool
|
||||||
|
lastScan map[string]time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func newGitHubProvider() *GitHubProvider {
|
||||||
|
return &GitHubProvider{
|
||||||
|
client: &http.Client{},
|
||||||
|
headerInitial: defaultHeaderRangeInitial,
|
||||||
|
headerMax: defaultHeaderRangeMax,
|
||||||
|
pageCap: defaultReleasePageCap,
|
||||||
|
scanTimeout: defaultScanTimeout,
|
||||||
|
serveTimeout: defaultServeTimeout,
|
||||||
|
coldWait: defaultColdWait,
|
||||||
|
scanning: map[string]bool{},
|
||||||
|
lastScan: map[string]time.Time{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) limiterWait(ctx context.Context) error {
|
||||||
|
if p.limiter == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return p.limiter.Wait(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) Type() models.PackageType { return models.PackageGitHubDeb }
|
||||||
|
|
||||||
|
func (p *GitHubProvider) Classify(path string) provider.Mutability {
|
||||||
|
switch path {
|
||||||
|
case "Packages", "Packages.gz", "Release", "InRelease", "Release.gpg":
|
||||||
|
return provider.Mutable
|
||||||
|
}
|
||||||
|
return provider.Immutable
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) ContentType(path string) string {
|
||||||
|
switch {
|
||||||
|
case strings.HasSuffix(path, ".deb"):
|
||||||
|
return "application/vnd.debian.binary-package"
|
||||||
|
case strings.HasSuffix(path, ".gz"):
|
||||||
|
return "application/gzip"
|
||||||
|
case path == "Packages" || path == "Release" || path == "InRelease":
|
||||||
|
return "text/plain"
|
||||||
|
}
|
||||||
|
return "application/octet-stream"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) UpstreamURL(remote models.Remote, path string) string {
|
||||||
|
return strings.TrimRight(remote.BaseURL, "/") + "/" + strings.TrimLeft(path, "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
|
||||||
|
return p.githubHeaders(ctx, remote, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeRemote answers a request against a github_deb remote. It refreshes the
|
||||||
|
// derived metadata (bounded by mutable_ttl), serves a synthesized flat apt repo
|
||||||
|
// (Packages/Packages.gz/Release), 404s the signed index variants (the repo is
|
||||||
|
// consumed via [trusted=yes]), and 302-redirects .deb downloads to the backend
|
||||||
|
// releases_remote. Returns false only for paths it does not own.
|
||||||
|
func (p *GitHubProvider) ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, reqPath, proxyBaseURL string, store provider.RemoteMetadataStore) bool {
|
||||||
|
p.onRequest(remote, store)
|
||||||
|
|
||||||
|
// apt appends the flat-repo dist "./" verbatim, so it asks for "./Packages"
|
||||||
|
// etc.; collapse the dot-segment before matching the synthesized index.
|
||||||
|
path := normalizeIndexPath(reqPath)
|
||||||
|
|
||||||
|
switch path {
|
||||||
|
case "Packages", "Packages.gz", "Release":
|
||||||
|
p.serveIndex(w, r, remote, path, store)
|
||||||
|
return true
|
||||||
|
case "InRelease", "Release.gpg":
|
||||||
|
// Unsigned flat repo: apt consumes it with [trusted=yes]. Signal absence
|
||||||
|
// so apt falls back to the plain Release without waiting on a signature.
|
||||||
|
http.Error(w, "not found", http.StatusNotFound)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.HasSuffix(path, ".deb") {
|
||||||
|
if remote.ReleasesRemote == "" {
|
||||||
|
http.Error(w, "github_deb remote has no releases_remote configured for downloads", http.StatusInternalServerError)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
loc := strings.TrimRight(proxyBaseURL, "/") + "/api/v1/remote/" + remote.ReleasesRemote + "/" + strings.TrimLeft(path, "/")
|
||||||
|
http.Redirect(w, r, loc, http.StatusFound)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) serveIndex(w http.ResponseWriter, r *http.Request, remote models.Remote, path string, store provider.RemoteMetadataStore) {
|
||||||
|
// Serve on a context detached from the inbound request so a client disconnect
|
||||||
|
// never cancels the metadata DB read and surfaces as a 500.
|
||||||
|
sctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), p.serveTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
if p.syncer != nil && !p.ensurePrimed(sctx, remote, store) {
|
||||||
|
w.Header().Set("Retry-After", "5")
|
||||||
|
http.Error(w, "metadata is being prepared, retry shortly", http.StatusServiceUnavailable)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, ok := store.(provider.DebMetadataReader)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "deb metadata not available", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
metas, err := reader.ListDebMetadataEntries(sctx, remote.Name)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
switch path {
|
||||||
|
case "Packages":
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write(generatePackages(metas))
|
||||||
|
case "Packages.gz":
|
||||||
|
w.Header().Set("Content-Type", "application/gzip")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write(gzipBytes(generatePackages(metas)))
|
||||||
|
case "Release":
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
w.Write(generateRelease(metas))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// onRequest keeps a remote's derived metadata fresh off the request path.
|
||||||
|
func (p *GitHubProvider) onRequest(remote models.Remote, store provider.RemoteMetadataStore) {
|
||||||
|
if p.syncer != nil {
|
||||||
|
p.syncer.enqueue(remote, false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.refresh(remote, store)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ensurePrimed returns true once the remote has at least one cached row. On an
|
||||||
|
// empty cache it enqueues a prime and polls briefly for it to land.
|
||||||
|
func (p *GitHubProvider) ensurePrimed(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) bool {
|
||||||
|
if !p.cacheEmpty(ctx, store, remote.Name) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if p.syncer != nil {
|
||||||
|
p.syncer.enqueue(remote, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
deadline := time.Now().Add(p.coldWait)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return false
|
||||||
|
case <-time.After(400 * time.Millisecond):
|
||||||
|
}
|
||||||
|
if !p.cacheEmpty(ctx, store, remote.Name) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) cacheEmpty(ctx context.Context, store provider.RemoteMetadataStore, name string) bool {
|
||||||
|
reader, ok := store.(provider.DebMetadataReader)
|
||||||
|
if !ok {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
rows, err := reader.ListDebMetadataEntries(ctx, name)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return len(rows) == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// refresh brings the derived metadata up to date without coupling the scan to
|
||||||
|
// the inbound request (legacy inline path used without a syncer / in unit tests).
|
||||||
|
func (p *GitHubProvider) refresh(remote models.Remote, store provider.RemoteMetadataStore) {
|
||||||
|
ttl := time.Duration(remote.MutableTTL) * time.Second
|
||||||
|
if ttl <= 0 {
|
||||||
|
ttl = 5 * time.Minute
|
||||||
|
}
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
last, ok := p.lastScan[remote.Name]
|
||||||
|
fresh := ok && time.Since(last) < ttl
|
||||||
|
if fresh || p.scanning[remote.Name] {
|
||||||
|
p.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.scanning[remote.Name] = true
|
||||||
|
p.mu.Unlock()
|
||||||
|
|
||||||
|
if p.cacheEmpty(context.Background(), store, remote.Name) {
|
||||||
|
p.runScan(remote, store)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
go p.runScan(remote, store)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) runScan(remote models.Remote, store provider.RemoteMetadataStore) {
|
||||||
|
defer func() {
|
||||||
|
p.mu.Lock()
|
||||||
|
delete(p.scanning, remote.Name)
|
||||||
|
p.mu.Unlock()
|
||||||
|
}()
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), p.scanTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
if err := p.scan(ctx, remote, store); err != nil {
|
||||||
|
slog.Error("github_deb: release scan failed", "remote", remote.Name, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
p.lastScan[remote.Name] = time.Now()
|
||||||
|
p.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// scan runs a full unconditional derive. Retained for the legacy inline refresh
|
||||||
|
// path and existing tests; the syncer uses scanWithState.
|
||||||
|
func (p *GitHubProvider) scan(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) error {
|
||||||
|
_, _, err := p.scanWithState(ctx, remote, store, "")
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanWithState derives metadata incrementally. It sends the prior releases-list
|
||||||
|
// ETag as a conditional request: a 304 means nothing changed. On a 200 it diffs
|
||||||
|
// the release assets against the cache, derives only new/changed assets, prunes
|
||||||
|
// assets that disappeared, and returns the new ETag.
|
||||||
|
func (p *GitHubProvider) scanWithState(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore, etag string) (newEtag string, changed bool, err error) {
|
||||||
|
releases, newEtag, notModified, err := p.fetchReleases(ctx, remote, etag)
|
||||||
|
if err != nil {
|
||||||
|
return etag, false, err
|
||||||
|
}
|
||||||
|
if notModified {
|
||||||
|
return etag, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
reader, ok := store.(provider.DebMetadataReader)
|
||||||
|
if !ok {
|
||||||
|
return newEtag, false, errors.New("store does not support deb metadata reads")
|
||||||
|
}
|
||||||
|
existing, err := reader.ListDebMetadataEntries(ctx, remote.Name)
|
||||||
|
if err != nil {
|
||||||
|
return newEtag, false, err
|
||||||
|
}
|
||||||
|
existingByPath := make(map[string]provider.DebMetadata, len(existing))
|
||||||
|
for _, m := range existing {
|
||||||
|
existingByPath[m.FilePath] = m
|
||||||
|
}
|
||||||
|
|
||||||
|
allow, err := compilePatterns(remote.Patterns)
|
||||||
|
if err != nil {
|
||||||
|
return newEtag, false, err
|
||||||
|
}
|
||||||
|
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, rel := range releases {
|
||||||
|
if rel.Draft {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, asset := range rel.Assets {
|
||||||
|
if !strings.HasSuffix(strings.ToLower(asset.Name), ".deb") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !matchesAny(allow, asset.Name) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fp := assetPath(asset)
|
||||||
|
if fp == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[fp] = true
|
||||||
|
|
||||||
|
if cur, ok := existingByPath[fp]; ok {
|
||||||
|
if asset.Digest == "" || cur.ContentHash == asset.Digest {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
_ = store.DeleteDebMetadata(ctx, remote.Name, fp)
|
||||||
|
}
|
||||||
|
|
||||||
|
meta, err := p.deriveAsset(ctx, remote, asset, fp)
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("github_deb: derive asset failed", "remote", remote.Name, "asset", asset.Name, "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := store.InsertDebMetadata(ctx, meta); err != nil {
|
||||||
|
slog.Error("github_deb: insert metadata failed", "remote", remote.Name, "asset", asset.Name, "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
slog.Info("github_deb: derived asset", "remote", remote.Name, "name", meta.Name, "version", meta.Version, "arch", meta.Architecture)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for fp := range existingByPath {
|
||||||
|
if !seen[fp] {
|
||||||
|
_ = store.DeleteDebMetadata(ctx, remote.Name, fp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return newEtag, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type ghRelease struct {
|
||||||
|
TagName string `json:"tag_name"`
|
||||||
|
Draft bool `json:"draft"`
|
||||||
|
Assets []ghAsset `json:"assets"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ghAsset struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Size int64 `json:"size"`
|
||||||
|
BrowserDownloadURL string `json:"browser_download_url"`
|
||||||
|
Digest string `json:"digest"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetchReleases lists a repo's releases, sending the prior ETag as If-None-Match
|
||||||
|
// on page 1 so an unchanged repo short-circuits to notModified. Every call waits
|
||||||
|
// on the shared limiter first.
|
||||||
|
func (p *GitHubProvider) fetchReleases(ctx context.Context, remote models.Remote, etag string) (all []ghRelease, newEtag string, notModified bool, err error) {
|
||||||
|
base := strings.TrimRight(remote.BaseURL, "/") + "/releases"
|
||||||
|
for page := 1; page <= p.pageCap; page++ {
|
||||||
|
u := fmt.Sprintf("%s?per_page=100&page=%d", base, page)
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
hdr, err := p.githubHeaders(ctx, remote, true)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
|
if page == 1 && etag != "" {
|
||||||
|
req.Header.Set("If-None-Match", etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
if page == 1 && resp.StatusCode == http.StatusNotModified {
|
||||||
|
io.Copy(io.Discard, resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
return nil, etag, true, nil
|
||||||
|
}
|
||||||
|
body, err := io.ReadAll(resp.Body)
|
||||||
|
respEtag := resp.Header.Get("ETag")
|
||||||
|
resp.Body.Close()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return nil, "", false, fmt.Errorf("github releases API %s: status %d", u, resp.StatusCode)
|
||||||
|
}
|
||||||
|
if page == 1 {
|
||||||
|
newEtag = respEtag
|
||||||
|
}
|
||||||
|
var releases []ghRelease
|
||||||
|
if err := json.Unmarshal(body, &releases); err != nil {
|
||||||
|
return nil, "", false, fmt.Errorf("decode releases: %w", err)
|
||||||
|
}
|
||||||
|
if len(releases) == 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
all = append(all, releases...)
|
||||||
|
if len(releases) < 100 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return all, newEtag, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) deriveAsset(ctx context.Context, remote models.Remote, asset ghAsset, fp string) (*provider.DebMetadata, error) {
|
||||||
|
control, err := p.fetchControl(ctx, remote, asset.BrowserDownloadURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
fields := parseControlFields(control)
|
||||||
|
|
||||||
|
meta := &provider.DebMetadata{
|
||||||
|
RepoName: remote.Name,
|
||||||
|
FilePath: fp,
|
||||||
|
Name: fields["Package"],
|
||||||
|
Version: fields["Version"],
|
||||||
|
Architecture: fields["Architecture"],
|
||||||
|
Control: strings.TrimRight(control, "\n"),
|
||||||
|
Size: asset.Size,
|
||||||
|
}
|
||||||
|
if meta.Name == "" {
|
||||||
|
return nil, errors.New("control missing Package field")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The Packages SHA256 must be the sha256 of the whole .deb. Prefer GitHub's
|
||||||
|
// asset digest so we never download the body; only when it is absent (or not
|
||||||
|
// sha256) do we stream the asset once. MD5sum is left unset — apt verifies the
|
||||||
|
// download against SHA256 alone under [trusted=yes].
|
||||||
|
if h, ok := sha256FromDigest(asset.Digest); ok {
|
||||||
|
meta.ContentHash = "sha256:" + h
|
||||||
|
meta.SHA256 = h
|
||||||
|
} else {
|
||||||
|
h, err := p.computeSHA256(ctx, remote, asset.BrowserDownloadURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("compute sha256: %w", err)
|
||||||
|
}
|
||||||
|
meta.ContentHash = "sha256:" + h
|
||||||
|
meta.SHA256 = h
|
||||||
|
}
|
||||||
|
|
||||||
|
return meta, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetchControl pulls only the front of the .deb with a ranged GET and extracts
|
||||||
|
// the control paragraph from it. control.tar sits right after the tiny
|
||||||
|
// debian-binary member, so a small prefix suffices; a prefix that truncates the
|
||||||
|
// control member doubles the range and retries.
|
||||||
|
func (p *GitHubProvider) fetchControl(ctx context.Context, remote models.Remote, downloadURL string) (string, error) {
|
||||||
|
n := p.headerInitial
|
||||||
|
for {
|
||||||
|
body, full, err := p.rangeGet(ctx, remote, downloadURL, n)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
control, complete, perr := controlFromPrefix(body)
|
||||||
|
if perr != nil {
|
||||||
|
return "", fmt.Errorf("parse deb control: %w", perr)
|
||||||
|
}
|
||||||
|
if complete {
|
||||||
|
return control, nil
|
||||||
|
}
|
||||||
|
if full || n >= p.headerMax {
|
||||||
|
return "", fmt.Errorf("control.tar not found within %d bytes of %s", n, downloadURL)
|
||||||
|
}
|
||||||
|
n *= 2
|
||||||
|
if n > p.headerMax {
|
||||||
|
n = p.headerMax
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// controlFromPrefix parses the ar members present in a front prefix of a .deb.
|
||||||
|
// It returns the ./control paragraph once control.tar.* is fully covered
|
||||||
|
// (complete=true); a prefix too short to cover it returns complete=false so the
|
||||||
|
// caller can widen the range. Later members (data.tar.*) are ignored.
|
||||||
|
func controlFromPrefix(prefix []byte) (control string, complete bool, err error) {
|
||||||
|
const magic = "!<arch>\n"
|
||||||
|
if len(prefix) < len(magic) {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
if string(prefix[:len(magic)]) != magic {
|
||||||
|
return "", false, errors.New("not an ar archive")
|
||||||
|
}
|
||||||
|
off := len(magic)
|
||||||
|
for {
|
||||||
|
if off+60 > len(prefix) {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
hdr := prefix[off : off+60]
|
||||||
|
off += 60
|
||||||
|
name := strings.TrimSuffix(strings.TrimRight(string(hdr[0:16]), " "), "/")
|
||||||
|
size, err := strconv.ParseInt(strings.TrimSpace(string(hdr[48:58])), 10, 64)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, fmt.Errorf("bad ar size for %q: %w", name, err)
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(name, "control.tar") {
|
||||||
|
if off+int(size) > len(prefix) {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
tarBytes, err := decompress(name, prefix[off:off+int(size)])
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
c, err := readControlParagraph(tarBytes)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
return c, true, nil
|
||||||
|
}
|
||||||
|
if off+int(size) > len(prefix) {
|
||||||
|
return "", false, nil
|
||||||
|
}
|
||||||
|
off += int(size)
|
||||||
|
if size%2 == 1 {
|
||||||
|
off++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// rangeGet returns the first n bytes of downloadURL. full is true when the
|
||||||
|
// response body was shorter than n (i.e. we already have the whole object).
|
||||||
|
func (p *GitHubProvider) rangeGet(ctx context.Context, remote models.Remote, downloadURL string, n int64) ([]byte, bool, error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
hdr, err := p.githubHeaders(ctx, remote, false)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
|
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", n-1))
|
||||||
|
|
||||||
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
|
||||||
|
return nil, false, fmt.Errorf("range GET %s: status %d", downloadURL, resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
body, err := io.ReadAll(io.LimitReader(resp.Body, n))
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
full := int64(len(body)) < n
|
||||||
|
return body, full, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) computeSHA256(ctx context.Context, remote models.Remote, downloadURL string) (string, error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
hdr, err := p.githubHeaders(ctx, remote, false)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
|
|
||||||
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return "", fmt.Errorf("GET %s: status %d", downloadURL, resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
h := sha256.New()
|
||||||
|
if _, err := io.Copy(h, resp.Body); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return hex.EncodeToString(h.Sum(nil)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// assetPath is the package's location relative to github.com — the path the
|
||||||
|
// backend releases_remote (base https://github.com) proxies. It doubles as the
|
||||||
|
// deb_metadata key and the Filename field in the Packages index, so a .deb
|
||||||
|
// download resolves back to this remote and redirects to the backend.
|
||||||
|
func assetPath(asset ghAsset) string {
|
||||||
|
u, err := url.Parse(asset.BrowserDownloadURL)
|
||||||
|
if err != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.TrimPrefix(u.Path, "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func sha256FromDigest(digest string) (string, bool) {
|
||||||
|
if strings.HasPrefix(digest, "sha256:") {
|
||||||
|
return strings.TrimPrefix(digest, "sha256:"), true
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
// githubHeaders builds the outbound headers for a GitHub request, attaching a
|
||||||
|
// bearer credential when one is available. A per-remote credential wins; absent
|
||||||
|
// that, the process-wide server credential is used; absent both, the request is
|
||||||
|
// unauthenticated.
|
||||||
|
func (p *GitHubProvider) githubHeaders(ctx context.Context, remote models.Remote, api bool) (http.Header, error) {
|
||||||
|
h := http.Header{}
|
||||||
|
if api {
|
||||||
|
h.Set("Accept", "application/vnd.github+json")
|
||||||
|
h.Set("X-GitHub-Api-Version", "2022-11-28")
|
||||||
|
}
|
||||||
|
tok, err := p.githubToken(ctx, remote)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if tok != "" {
|
||||||
|
h.Set("Authorization", "Bearer "+tok)
|
||||||
|
}
|
||||||
|
return h, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// githubToken resolves the bearer token for a remote. Precedence: a per-remote
|
||||||
|
// credential (password, then username) overrides the server credential.
|
||||||
|
func (p *GitHubProvider) githubToken(ctx context.Context, remote models.Remote) (string, error) {
|
||||||
|
if remote.Password != "" {
|
||||||
|
return remote.Password, nil
|
||||||
|
}
|
||||||
|
if remote.Username != "" {
|
||||||
|
return remote.Username, nil
|
||||||
|
}
|
||||||
|
if c := p.serverCredential(); c != nil {
|
||||||
|
return c.Token(ctx)
|
||||||
|
}
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) serverCredential() githubauth.Credential {
|
||||||
|
if p.serverCred != nil {
|
||||||
|
return p.serverCred
|
||||||
|
}
|
||||||
|
return githubauth.Server()
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyHeaders(req *http.Request, h http.Header) {
|
||||||
|
for k, vals := range h {
|
||||||
|
for _, v := range vals {
|
||||||
|
req.Header.Add(k, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func compilePatterns(patterns []string) ([]*regexp.Regexp, error) {
|
||||||
|
var out []*regexp.Regexp
|
||||||
|
for _, p := range patterns {
|
||||||
|
re, err := regexp.Compile(p)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid pattern %q: %w", p, err)
|
||||||
|
}
|
||||||
|
out = append(out, re)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func matchesAny(res []*regexp.Regexp, s string) bool {
|
||||||
|
if len(res) == 0 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, re := range res {
|
||||||
|
if re.MatchString(s) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,462 @@
|
|||||||
|
package deb
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"compress/gzip"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/testsupport"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeStore is an in-memory provider.RemoteMetadataStore + DebMetadataReader
|
||||||
|
// keyed by file_path, mirroring the (repo_name, file_path) uniqueness of the
|
||||||
|
// real deb_metadata table.
|
||||||
|
type fakeStore struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
rows map[string]provider.DebMetadata
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeStore() *fakeStore { return &fakeStore{rows: map[string]provider.DebMetadata{}} }
|
||||||
|
|
||||||
|
func (f *fakeStore) InsertDebMetadata(_ context.Context, m *provider.DebMetadata) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
if _, ok := f.rows[m.FilePath]; ok {
|
||||||
|
return nil // ON CONFLICT DO NOTHING
|
||||||
|
}
|
||||||
|
f.rows[m.FilePath] = *m
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) DeleteDebMetadata(_ context.Context, _, filePath string) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
delete(f.rows, filePath)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) InsertRPMMetadata(context.Context, *provider.RPMMetadata) error { return nil }
|
||||||
|
func (f *fakeStore) DeleteRPMMetadata(context.Context, string, string) error { return nil }
|
||||||
|
func (f *fakeStore) ListRPMMetadataEntries(context.Context, string) ([]provider.RPMMetadata, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) ListDebMetadataEntries(ctx context.Context, _ string) ([]provider.DebMetadata, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
out := make([]provider.DebMetadata, 0, len(f.rows))
|
||||||
|
for _, m := range f.rows {
|
||||||
|
out = append(out, m)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// githubFixture serves the releases API and the .deb asset downloads (with Range
|
||||||
|
// support) for a set of packages. digest controls whether the asset carries a
|
||||||
|
// sha256 digest (no-download path) or not (compute path).
|
||||||
|
type githubFixture struct {
|
||||||
|
srv *httptest.Server
|
||||||
|
debBytes map[string][]byte
|
||||||
|
rangeHit map[string]int
|
||||||
|
fullHit map[string]int
|
||||||
|
etag string
|
||||||
|
releasesHit int
|
||||||
|
notModHit int
|
||||||
|
releaseAuth string
|
||||||
|
assetAuth string
|
||||||
|
mu sync.Mutex
|
||||||
|
}
|
||||||
|
|
||||||
|
func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
|
||||||
|
t.Helper()
|
||||||
|
f := &githubFixture{
|
||||||
|
debBytes: map[string][]byte{},
|
||||||
|
rangeHit: map[string]int{},
|
||||||
|
fullHit: map[string]int{},
|
||||||
|
}
|
||||||
|
f.debBytes["demo_1.2-3_amd64.deb"] = testsupport.MinimalDeb("demo", "1.2-3", "amd64")
|
||||||
|
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("/repos/acme/tools/releases", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
page := r.URL.Query().Get("page")
|
||||||
|
if page != "" && page != "1" {
|
||||||
|
w.Write([]byte("[]"))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f.mu.Lock()
|
||||||
|
f.releasesHit++
|
||||||
|
f.releaseAuth = r.Header.Get("Authorization")
|
||||||
|
etag := f.etag
|
||||||
|
if etag != "" && r.Header.Get("If-None-Match") == etag {
|
||||||
|
f.notModHit++
|
||||||
|
f.mu.Unlock()
|
||||||
|
w.WriteHeader(http.StatusNotModified)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f.mu.Unlock()
|
||||||
|
if etag != "" {
|
||||||
|
w.Header().Set("ETag", etag)
|
||||||
|
}
|
||||||
|
var assets []map[string]any
|
||||||
|
for name := range f.debBytes {
|
||||||
|
a := map[string]any{
|
||||||
|
"name": name,
|
||||||
|
"size": len(f.debBytes[name]),
|
||||||
|
"browser_download_url": f.srv.URL + "/acme/tools/releases/download/v1.2-3/" + name,
|
||||||
|
}
|
||||||
|
if withDigest {
|
||||||
|
sum := sha256.Sum256(f.debBytes[name])
|
||||||
|
a["digest"] = "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
assets = append(assets, a)
|
||||||
|
}
|
||||||
|
rel := []map[string]any{{"tag_name": "v1.2-3", "draft": false, "assets": assets}}
|
||||||
|
json.NewEncoder(w).Encode(rel)
|
||||||
|
})
|
||||||
|
mux.HandleFunc("/acme/tools/releases/download/", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
|
||||||
|
body, ok := f.debBytes[name]
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "not found", 404)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
rng := r.Header.Get("Range")
|
||||||
|
f.mu.Lock()
|
||||||
|
f.assetAuth = r.Header.Get("Authorization")
|
||||||
|
if rng != "" {
|
||||||
|
f.rangeHit[name]++
|
||||||
|
} else {
|
||||||
|
f.fullHit[name]++
|
||||||
|
}
|
||||||
|
f.mu.Unlock()
|
||||||
|
|
||||||
|
if rng == "" {
|
||||||
|
w.WriteHeader(200)
|
||||||
|
w.Write(body)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var end int
|
||||||
|
fmt.Sscanf(rng, "bytes=0-%d", &end)
|
||||||
|
if end >= len(body)-1 {
|
||||||
|
end = len(body) - 1
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Range", fmt.Sprintf("bytes 0-%d/%d", end, len(body)))
|
||||||
|
w.Header().Set("Content-Length", strconv.Itoa(end+1))
|
||||||
|
w.WriteHeader(http.StatusPartialContent)
|
||||||
|
w.Write(body[:end+1])
|
||||||
|
})
|
||||||
|
f.srv = httptest.NewServer(mux)
|
||||||
|
t.Cleanup(f.srv.Close)
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *githubFixture) remote() models.Remote {
|
||||||
|
return models.Remote{
|
||||||
|
Name: "acme-deb",
|
||||||
|
PackageType: models.PackageGitHubDeb,
|
||||||
|
BaseURL: f.srv.URL + "/repos/acme/tools",
|
||||||
|
ReleasesRemote: "github",
|
||||||
|
MutableTTL: 3600,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTestProvider() *GitHubProvider {
|
||||||
|
p := newGitHubProvider()
|
||||||
|
p.headerInitial = 32 // force the ranged-fetch retry loop against the tiny fixture
|
||||||
|
p.headerMax = 1 << 20
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
const demoPath = "acme/tools/releases/download/v1.2-3/demo_1.2-3_amd64.deb"
|
||||||
|
|
||||||
|
func TestGitHubScanDerivesControlFromPrefixAndDigest(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
metas, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb")
|
||||||
|
if len(metas) != 1 {
|
||||||
|
t.Fatalf("want 1 metadata row, got %d", len(metas))
|
||||||
|
}
|
||||||
|
m := metas[0]
|
||||||
|
if m.Name != "demo" || m.Version != "1.2-3" || m.Architecture != "amd64" {
|
||||||
|
t.Fatalf("bad control fields: %+v", m)
|
||||||
|
}
|
||||||
|
if m.FilePath != demoPath {
|
||||||
|
t.Fatalf("FilePath = %q, want %q", m.FilePath, demoPath)
|
||||||
|
}
|
||||||
|
if int(m.Size) != len(fx.debBytes["demo_1.2-3_amd64.deb"]) {
|
||||||
|
t.Fatalf("Size = %d, want %d", m.Size, len(fx.debBytes["demo_1.2-3_amd64.deb"]))
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(fx.debBytes["demo_1.2-3_amd64.deb"])
|
||||||
|
if m.SHA256 != hex.EncodeToString(sum[:]) {
|
||||||
|
t.Fatalf("SHA256 = %q, want digest", m.SHA256)
|
||||||
|
}
|
||||||
|
if m.ContentHash != "sha256:"+hex.EncodeToString(sum[:]) {
|
||||||
|
t.Fatalf("ContentHash = %q", m.ContentHash)
|
||||||
|
}
|
||||||
|
if m.MD5 != "" {
|
||||||
|
t.Fatalf("MD5 should be unset for metadata-only derive, got %q", m.MD5)
|
||||||
|
}
|
||||||
|
if fx.fullHit["demo_1.2-3_amd64.deb"] != 0 {
|
||||||
|
t.Fatalf("expected no full download when digest present, got %d", fx.fullHit["demo_1.2-3_amd64.deb"])
|
||||||
|
}
|
||||||
|
if fx.rangeHit["demo_1.2-3_amd64.deb"] == 0 {
|
||||||
|
t.Fatalf("expected ranged control fetch")
|
||||||
|
}
|
||||||
|
if !strings.Contains(m.Control, "Package: demo") {
|
||||||
|
t.Fatalf("raw control not captured: %q", m.Control)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubChecksumComputedWhenDigestAbsent(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, false)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
metas, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb")
|
||||||
|
if len(metas) != 1 {
|
||||||
|
t.Fatalf("want 1 row, got %d", len(metas))
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(fx.debBytes["demo_1.2-3_amd64.deb"])
|
||||||
|
if metas[0].SHA256 != hex.EncodeToString(sum[:]) {
|
||||||
|
t.Fatalf("computed checksum mismatch: %q", metas[0].SHA256)
|
||||||
|
}
|
||||||
|
if fx.fullHit["demo_1.2-3_amd64.deb"] == 0 {
|
||||||
|
t.Fatalf("expected a full download to compute sha256 when digest absent")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubServeRemoteIndexAndRedirect(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
const proxyBase = "https://artifactapi.example"
|
||||||
|
|
||||||
|
// Release is served and triggers the initial scan.
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Release", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "Release", proxyBase, store) {
|
||||||
|
t.Fatal("ServeRemote did not handle Release")
|
||||||
|
}
|
||||||
|
if rec.Code != 200 || !strings.Contains(rec.Body.String(), "Architectures:") {
|
||||||
|
t.Fatalf("Release bad: code=%d body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(rec.Body.String(), "amd64") {
|
||||||
|
t.Fatalf("Release missing arch: %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Packages carries the package with a Filename that is the github-relative
|
||||||
|
// download path (so it resolves back to this remote and redirects).
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
req = httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "Packages", proxyBase, store) {
|
||||||
|
t.Fatal("ServeRemote did not handle Packages")
|
||||||
|
}
|
||||||
|
pkgs := rec.Body.String()
|
||||||
|
if !strings.Contains(pkgs, "Package: demo") {
|
||||||
|
t.Fatalf("Packages missing package: %s", pkgs)
|
||||||
|
}
|
||||||
|
if !strings.Contains(pkgs, "Filename: "+demoPath) {
|
||||||
|
t.Fatalf("Packages missing/incorrect Filename: %s", pkgs)
|
||||||
|
}
|
||||||
|
if !strings.Contains(pkgs, "SHA256: ") {
|
||||||
|
t.Fatalf("Packages missing SHA256: %s", pkgs)
|
||||||
|
}
|
||||||
|
if strings.Contains(pkgs, "MD5sum:") {
|
||||||
|
t.Fatalf("Packages should omit empty MD5sum: %s", pkgs)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Packages.gz decompresses to the same content.
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
req = httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "Packages.gz", proxyBase, store) {
|
||||||
|
t.Fatal("ServeRemote did not handle Packages.gz")
|
||||||
|
}
|
||||||
|
gz, err := gzip.NewReader(rec.Body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("gzip: %v", err)
|
||||||
|
}
|
||||||
|
unz, _ := io.ReadAll(gz)
|
||||||
|
if !strings.Contains(string(unz), "Package: demo") {
|
||||||
|
t.Fatalf("Packages.gz missing package: %s", unz)
|
||||||
|
}
|
||||||
|
|
||||||
|
// InRelease/Release.gpg 404 (unsigned, consumed via [trusted=yes]).
|
||||||
|
for _, sp := range []string{"InRelease", "Release.gpg"} {
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
req = httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, sp, proxyBase, store) {
|
||||||
|
t.Fatalf("ServeRemote did not handle %s", sp)
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusNotFound {
|
||||||
|
t.Fatalf("%s want 404, got %d", sp, rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A .deb request redirects to the backend releases_remote.
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
req = httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/"+demoPath, nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, demoPath, proxyBase, store) {
|
||||||
|
t.Fatal("ServeRemote did not handle .deb")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusFound {
|
||||||
|
t.Fatalf("want 302, got %d", rec.Code)
|
||||||
|
}
|
||||||
|
wantLoc := proxyBase + "/api/v1/remote/github/" + demoPath
|
||||||
|
if got := rec.Header().Get("Location"); got != wantLoc {
|
||||||
|
t.Fatalf("Location = %q, want %q", got, wantLoc)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Real apt appends the flat-repo dist "./" verbatim, so the metadata-only remote
|
||||||
|
// receives "./Packages" / "./Release"; ServeRemote must collapse the dot-segment
|
||||||
|
// and synthesize the same index as the un-prefixed request.
|
||||||
|
func TestGitHubServeRemoteAptDotSegment(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
const proxyBase = "https://artifactapi.example"
|
||||||
|
|
||||||
|
serve := func(path string) *httptest.ResponseRecorder {
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/"+path, nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, path, proxyBase, store) {
|
||||||
|
t.Fatalf("ServeRemote did not handle %q", path)
|
||||||
|
}
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
// Packages is deterministic: byte-identical to the un-prefixed request.
|
||||||
|
plain, dotted := serve("Packages"), serve("./Packages")
|
||||||
|
if plain.Code != 200 || dotted.Code != 200 {
|
||||||
|
t.Fatalf("Packages: plain=%d dotted=%d, want 200/200", plain.Code, dotted.Code)
|
||||||
|
}
|
||||||
|
if !strings.Contains(dotted.Body.String(), "Package: demo") {
|
||||||
|
t.Fatalf("./Packages missing synthesized body: %s", dotted.Body.String())
|
||||||
|
}
|
||||||
|
if !bytes.Equal(plain.Body.Bytes(), dotted.Body.Bytes()) {
|
||||||
|
t.Error("./Packages body differs from Packages body")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Release carries a time.Now() Date: header; compare the rest.
|
||||||
|
rPlain, rDotted := serve("Release"), serve("./Release")
|
||||||
|
if rPlain.Code != 200 || rDotted.Code != 200 {
|
||||||
|
t.Fatalf("Release: plain=%d dotted=%d, want 200/200", rPlain.Code, rDotted.Code)
|
||||||
|
}
|
||||||
|
if stripDate(rPlain.Body.String()) != stripDate(rDotted.Body.String()) {
|
||||||
|
t.Error("./Release body differs from Release body (ignoring Date)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A canceled inbound request must still serve the warm cache (detached context),
|
||||||
|
// not turn the metadata read into a 500.
|
||||||
|
func TestGitHubServeRemoteCanceledRequestServesCache(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("warm scan: %v", err)
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
p.lastScan[remote.Name] = time.Now()
|
||||||
|
p.mu.Unlock()
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Packages", nil).WithContext(ctx)
|
||||||
|
|
||||||
|
if !p.ServeRemote(rec, req, remote, "Packages", "https://x", store) {
|
||||||
|
t.Fatal("ServeRemote did not handle Packages")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("canceled request must serve cache, not error; got code=%d body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(rec.Body.String(), "Package: demo") {
|
||||||
|
t.Fatalf("expected Packages served from cache, got %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubServeRemoteRedirectRequiresReleasesRemote(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
remote.ReleasesRemote = ""
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, demoPath, "https://x", store) {
|
||||||
|
t.Fatal("expected handled")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusInternalServerError {
|
||||||
|
t.Fatalf("want 500 when releases_remote unset, got %d", rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubScanPrunesRemovedAssets(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if rows, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb"); len(rows) != 1 {
|
||||||
|
t.Fatalf("want 1 row after first scan, got %d", len(rows))
|
||||||
|
}
|
||||||
|
|
||||||
|
delete(fx.debBytes, "demo_1.2-3_amd64.deb")
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("rescan: %v", err)
|
||||||
|
}
|
||||||
|
if rows, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb"); len(rows) != 0 {
|
||||||
|
t.Fatalf("want 0 rows after prune, got %d", len(rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubAssetPatternFilter(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.debBytes["other_9_arm64.deb"] = testsupport.MinimalDeb("other", "9", "arm64")
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
remote.Patterns = []string{`^demo_.*_amd64\.deb$`}
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
rows, _ := store.ListDebMetadataEntries(context.Background(), "acme-deb")
|
||||||
|
if len(rows) != 1 || rows[0].Name != "demo" {
|
||||||
|
t.Fatalf("pattern filter failed, rows=%+v", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,238 @@
|
|||||||
|
package deb
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/time/rate"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
syncLeaseDuration = 15 * time.Minute
|
||||||
|
defaultSyncFreshness = 5 * time.Minute
|
||||||
|
jobQueueDepth = 256
|
||||||
|
)
|
||||||
|
|
||||||
|
// SyncStore is the persistence surface the deb syncer needs: the metadata cache
|
||||||
|
// it primes plus the shared sync-state coordination (remote enumeration and the
|
||||||
|
// per-remote lease). *database.DB satisfies it.
|
||||||
|
type SyncStore interface {
|
||||||
|
provider.RemoteMetadataStore
|
||||||
|
ListGitHubDebRemotes(ctx context.Context) ([]models.Remote, error)
|
||||||
|
ClaimGitHubDebSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (claimed bool, etag string, err error)
|
||||||
|
ReleaseGitHubDebSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// SyncConfig tunes the shared syncer. Zero values fall back to safe defaults.
|
||||||
|
type SyncConfig struct {
|
||||||
|
RatePerSec float64
|
||||||
|
Burst int
|
||||||
|
Workers int
|
||||||
|
PollInterval time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
type syncJob struct {
|
||||||
|
remote models.Remote
|
||||||
|
prime bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// Syncer is the single per-process background worker that keeps every github_deb
|
||||||
|
// remote's derived metadata fresh. It owns a deduped work queue, a pool of
|
||||||
|
// workers, and a global token-bucket rate limiter shared across all remotes and
|
||||||
|
// bound onto the github_deb provider. Periodic checks are gated by a shared DB
|
||||||
|
// lease so, across replicas, only one performs each scan.
|
||||||
|
type Syncer struct {
|
||||||
|
store SyncStore
|
||||||
|
prov *GitHubProvider
|
||||||
|
limiter *rate.Limiter
|
||||||
|
cfg SyncConfig
|
||||||
|
owner string
|
||||||
|
|
||||||
|
jobs chan syncJob
|
||||||
|
mu sync.Mutex
|
||||||
|
active map[string]bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewSyncer builds the syncer bound to the process-wide github_deb provider
|
||||||
|
// singleton. Call Run to start it.
|
||||||
|
func NewSyncer(store SyncStore, cfg SyncConfig) *Syncer {
|
||||||
|
return newSyncer(store, gitHubProvider, cfg)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newSyncer(store SyncStore, prov *GitHubProvider, cfg SyncConfig) *Syncer {
|
||||||
|
if cfg.RatePerSec <= 0 {
|
||||||
|
cfg.RatePerSec = 1
|
||||||
|
}
|
||||||
|
if cfg.Burst <= 0 {
|
||||||
|
cfg.Burst = 5
|
||||||
|
}
|
||||||
|
if cfg.Workers <= 0 {
|
||||||
|
cfg.Workers = 3
|
||||||
|
}
|
||||||
|
if cfg.PollInterval <= 0 {
|
||||||
|
cfg.PollInterval = 60 * time.Second
|
||||||
|
}
|
||||||
|
|
||||||
|
lim := rate.NewLimiter(rate.Limit(cfg.RatePerSec), cfg.Burst)
|
||||||
|
s := &Syncer{
|
||||||
|
store: store,
|
||||||
|
prov: prov,
|
||||||
|
limiter: lim,
|
||||||
|
cfg: cfg,
|
||||||
|
owner: leaseOwner(),
|
||||||
|
jobs: make(chan syncJob, jobQueueDepth),
|
||||||
|
active: map[string]bool{},
|
||||||
|
}
|
||||||
|
prov.limiter = lim
|
||||||
|
prov.syncer = s
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run starts the worker pool and the periodic scheduler and blocks until ctx is
|
||||||
|
// canceled, at which point it drains in-flight scans and returns.
|
||||||
|
func (s *Syncer) Run(ctx context.Context) {
|
||||||
|
slog.Info("github_deb syncer started",
|
||||||
|
"rate_per_sec", s.cfg.RatePerSec, "burst", s.cfg.Burst,
|
||||||
|
"workers", s.cfg.Workers, "poll_interval", s.cfg.PollInterval, "owner", s.owner)
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < s.cfg.Workers; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
s.worker(ctx)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
ticker := time.NewTicker(s.cfg.PollInterval)
|
||||||
|
defer ticker.Stop()
|
||||||
|
|
||||||
|
s.schedule(ctx)
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
wg.Wait()
|
||||||
|
slog.Info("github_deb syncer stopped")
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
s.schedule(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// schedule enqueues a periodic check for every github_deb remote. The DB lease
|
||||||
|
// enforces the per-remote mutable_ttl cadence and cross-replica coordination.
|
||||||
|
func (s *Syncer) schedule(ctx context.Context) {
|
||||||
|
remotes, err := s.store.ListGitHubDebRemotes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("github_deb syncer: list remotes", "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, r := range remotes {
|
||||||
|
s.enqueue(r, false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// EnqueuePrime queues an immediate background prime for a freshly created remote.
|
||||||
|
func (s *Syncer) EnqueuePrime(remote models.Remote) {
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.enqueue(remote, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
// enqueue adds a job unless the remote is already queued or in-flight, coalescing
|
||||||
|
// duplicate requests down to one scan. It never blocks.
|
||||||
|
func (s *Syncer) enqueue(remote models.Remote, prime bool) {
|
||||||
|
s.mu.Lock()
|
||||||
|
if s.active[remote.Name] {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.active[remote.Name] = true
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case s.jobs <- syncJob{remote: remote, prime: prime}:
|
||||||
|
default:
|
||||||
|
s.mu.Lock()
|
||||||
|
delete(s.active, remote.Name)
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Syncer) worker(ctx context.Context) {
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case job := <-s.jobs:
|
||||||
|
s.process(ctx, job)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// process claims the shared lease and, if won, runs an incremental scan. Losing
|
||||||
|
// the claim (another replica scanning, or not yet due) is a no-op.
|
||||||
|
func (s *Syncer) process(ctx context.Context, job syncJob) {
|
||||||
|
defer func() {
|
||||||
|
s.mu.Lock()
|
||||||
|
delete(s.active, job.remote.Name)
|
||||||
|
s.mu.Unlock()
|
||||||
|
}()
|
||||||
|
|
||||||
|
freshness := time.Duration(job.remote.MutableTTL) * time.Second
|
||||||
|
if freshness <= 0 {
|
||||||
|
freshness = defaultSyncFreshness
|
||||||
|
}
|
||||||
|
if job.prime {
|
||||||
|
freshness = 0
|
||||||
|
}
|
||||||
|
|
||||||
|
claimed, etag, err := s.store.ClaimGitHubDebSyncLease(ctx, job.remote.Name, s.owner, freshness, syncLeaseDuration)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("github_deb syncer: claim lease", "remote", job.remote.Name, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !claimed {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
scanCtx, cancel := context.WithTimeout(ctx, s.prov.scanTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
newEtag, changed, scanErr := s.prov.scanWithState(scanCtx, job.remote, s.store, etag)
|
||||||
|
releaseEtag := etag
|
||||||
|
if scanErr == nil {
|
||||||
|
releaseEtag = newEtag
|
||||||
|
} else {
|
||||||
|
slog.Error("github_deb syncer: scan failed", "remote", job.remote.Name, "error", scanErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
relCtx, relCancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
|
||||||
|
defer relCancel()
|
||||||
|
if err := s.store.ReleaseGitHubDebSyncLease(relCtx, job.remote.Name, s.owner, releaseEtag, time.Now()); err != nil {
|
||||||
|
slog.Warn("github_deb syncer: release lease", "remote", job.remote.Name, "error", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if scanErr == nil && changed {
|
||||||
|
slog.Info("github_deb syncer: refreshed", "remote", job.remote.Name, "prime", job.prime)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// leaseOwner is a per-replica identity for the lease: hostname plus a random
|
||||||
|
// suffix so restarts and colocated replicas never collide.
|
||||||
|
func leaseOwner() string {
|
||||||
|
host, _ := os.Hostname()
|
||||||
|
var b [6]byte
|
||||||
|
_, _ = rand.Read(b[:])
|
||||||
|
return host + "-" + hex.EncodeToString(b[:])
|
||||||
|
}
|
||||||
@@ -0,0 +1,300 @@
|
|||||||
|
package deb
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/time/rate"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/testsupport"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeSyncStore is an in-memory SyncStore: the metadata cache (via the embedded
|
||||||
|
// fakeStore) plus the shared sync-state lease, whose claim mirrors the atomic
|
||||||
|
// semantics of the real SQL (recency gate AND no live lease).
|
||||||
|
type fakeSyncStore struct {
|
||||||
|
*fakeStore
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
remotes []models.Remote
|
||||||
|
leaseOwner map[string]string
|
||||||
|
leaseExp map[string]time.Time
|
||||||
|
lastSynced map[string]time.Time
|
||||||
|
etags map[string]string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeSyncStore() *fakeSyncStore {
|
||||||
|
return &fakeSyncStore{
|
||||||
|
fakeStore: newFakeStore(),
|
||||||
|
leaseOwner: map[string]string{},
|
||||||
|
leaseExp: map[string]time.Time{},
|
||||||
|
lastSynced: map[string]time.Time{},
|
||||||
|
etags: map[string]string{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSyncStore) ListGitHubDebRemotes(_ context.Context) ([]models.Remote, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
return append([]models.Remote(nil), f.remotes...), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSyncStore) ClaimGitHubDebSyncLease(_ context.Context, name, owner string, freshness, lease time.Duration) (bool, string, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
now := time.Now()
|
||||||
|
ls, hasLS := f.lastSynced[name]
|
||||||
|
exp, hasExp := f.leaseExp[name]
|
||||||
|
freshOK := !hasLS || now.Sub(ls) >= freshness
|
||||||
|
leaseOK := !hasExp || exp.Before(now)
|
||||||
|
if freshOK && leaseOK {
|
||||||
|
f.leaseOwner[name] = owner
|
||||||
|
f.leaseExp[name] = now.Add(lease)
|
||||||
|
return true, f.etags[name], nil
|
||||||
|
}
|
||||||
|
return false, "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSyncStore) ReleaseGitHubDebSyncLease(_ context.Context, name, owner, etag string, syncedAt time.Time) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
if f.leaseOwner[name] != owner {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
f.lastSynced[name] = syncedAt
|
||||||
|
f.etags[name] = etag
|
||||||
|
delete(f.leaseOwner, name)
|
||||||
|
delete(f.leaseExp, name)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func testSyncConfig() SyncConfig {
|
||||||
|
return SyncConfig{RatePerSec: 1000, Burst: 100, Workers: 1, PollInterval: time.Hour}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (a) A 304 conditional response must derive nothing: no asset fetches and
|
||||||
|
// changed=false, so an unchanged repo is nearly free.
|
||||||
|
func TestSyncerConditionalNotModifiedSkipsDerive(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
etag1, changed, err := p.scanWithState(context.Background(), remote, store, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first scan: %v", err)
|
||||||
|
}
|
||||||
|
if !changed || etag1 != `"v1"` {
|
||||||
|
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag1)
|
||||||
|
}
|
||||||
|
priorRange := fx.rangeHit["demo_1.2-3_amd64.deb"]
|
||||||
|
if priorRange == 0 {
|
||||||
|
t.Fatal("first scan should have fetched the asset control")
|
||||||
|
}
|
||||||
|
|
||||||
|
etag2, changed2, err := p.scanWithState(context.Background(), remote, store, etag1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second scan: %v", err)
|
||||||
|
}
|
||||||
|
if changed2 {
|
||||||
|
t.Fatal("304 scan must report changed=false")
|
||||||
|
}
|
||||||
|
if etag2 != etag1 {
|
||||||
|
t.Fatalf("etag changed across 304: %q -> %q", etag1, etag2)
|
||||||
|
}
|
||||||
|
if fx.notModHit != 1 {
|
||||||
|
t.Fatalf("want exactly one 304 releases response, got %d", fx.notModHit)
|
||||||
|
}
|
||||||
|
if got := fx.rangeHit["demo_1.2-3_amd64.deb"]; got != priorRange {
|
||||||
|
t.Fatalf("304 scan re-fetched asset control: %d -> %d", priorRange, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (b) On a real change, only the newly added asset is derived.
|
||||||
|
func TestSyncerIncrementalDerivesOnlyNewAsset(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
if _, _, err := p.scanWithState(context.Background(), remote, store, ""); err != nil {
|
||||||
|
t.Fatalf("first scan: %v", err)
|
||||||
|
}
|
||||||
|
demoRange := fx.rangeHit["demo_1.2-3_amd64.deb"]
|
||||||
|
|
||||||
|
fx.debBytes["other_9_arm64.deb"] = testsupport.MinimalDeb("other", "9", "arm64")
|
||||||
|
fx.etag = `"v2"`
|
||||||
|
|
||||||
|
if _, changed, err := p.scanWithState(context.Background(), remote, store, `"v1"`); err != nil || !changed {
|
||||||
|
t.Fatalf("second scan changed=%v err=%v", changed, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rows, _ := store.ListDebMetadataEntries(context.Background(), remote.Name)
|
||||||
|
if len(rows) != 2 {
|
||||||
|
t.Fatalf("want 2 cached rows after incremental derive, got %d", len(rows))
|
||||||
|
}
|
||||||
|
if got := fx.rangeHit["demo_1.2-3_amd64.deb"]; got != demoRange {
|
||||||
|
t.Fatalf("already-cached asset was re-fetched: %d -> %d", demoRange, got)
|
||||||
|
}
|
||||||
|
if fx.rangeHit["other_9_arm64.deb"] == 0 {
|
||||||
|
t.Fatal("newly added asset was not derived")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (c) The shared limiter caps the request rate.
|
||||||
|
func TestRateLimiterCapsRequestRate(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
p.limiter = rate.NewLimiter(rate.Every(120*time.Millisecond), 1)
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
if _, _, _, err := p.fetchReleases(context.Background(), remote, ""); err != nil {
|
||||||
|
t.Fatalf("fetchReleases %d: %v", i, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed < 200*time.Millisecond {
|
||||||
|
t.Fatalf("rate limiter did not throttle: 3 calls took %v, want >= 200ms", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (d) Concurrent enqueues for the same remote coalesce to a single queued job.
|
||||||
|
func TestSyncerEnqueueDedup(t *testing.T) {
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := models.Remote{Name: "acme-deb", PackageType: models.PackageGitHubDeb, MutableTTL: 3600}
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() { defer wg.Done(); s.enqueue(remote, false) }()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
if got := len(s.jobs); got != 1 {
|
||||||
|
t.Fatalf("want exactly 1 coalesced job, got %d", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (e) Prime-on-create enqueues a prime job.
|
||||||
|
func TestSyncerEnqueuePrime(t *testing.T) {
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := models.Remote{Name: "acme-deb", PackageType: models.PackageGitHubDeb, MutableTTL: 3600}
|
||||||
|
|
||||||
|
s.EnqueuePrime(remote)
|
||||||
|
select {
|
||||||
|
case job := <-s.jobs:
|
||||||
|
if !job.prime || job.remote.Name != "acme-deb" {
|
||||||
|
t.Fatalf("bad prime job: %+v", job)
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
t.Fatal("EnqueuePrime did not enqueue a job")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (f) A held lease prevents a second replica from scanning.
|
||||||
|
func TestSyncerLeasePreventsSecondReplica(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
claimed, _, err := store.ClaimGitHubDebSyncLease(context.Background(), remote.Name, "replica-1", time.Duration(remote.MutableTTL)*time.Second, syncLeaseDuration)
|
||||||
|
if err != nil || !claimed {
|
||||||
|
t.Fatalf("replica-1 claim: claimed=%v err=%v", claimed, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
s.process(context.Background(), syncJob{remote: remote})
|
||||||
|
|
||||||
|
if fx.releasesHit != 0 {
|
||||||
|
t.Fatalf("second replica scanned while lease held: %d releases calls", fx.releasesHit)
|
||||||
|
}
|
||||||
|
if rows, _ := store.ListDebMetadataEntries(context.Background(), remote.Name); len(rows) != 0 {
|
||||||
|
t.Fatalf("second replica derived metadata while lease held: %d rows", len(rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// With the syncer wired and the cache empty, an index request enqueues a prime
|
||||||
|
// and returns a retryable 503 when it has not landed within the cold wait.
|
||||||
|
func TestServeRemoteColdStartReturns503(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
p.coldWait = 300 * time.Millisecond
|
||||||
|
_ = newSyncer(store, p, testSyncConfig()) // binds p.syncer, but no workers running
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Packages", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "Packages", "https://x", store) {
|
||||||
|
t.Fatal("ServeRemote did not handle Packages")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusServiceUnavailable {
|
||||||
|
t.Fatalf("cold empty cache must return 503, got %d", rec.Code)
|
||||||
|
}
|
||||||
|
if rec.Header().Get("Retry-After") == "" {
|
||||||
|
t.Fatal("503 should carry Retry-After")
|
||||||
|
}
|
||||||
|
if got := len(p.syncer.jobs); got != 1 {
|
||||||
|
t.Fatalf("cold start did not enqueue a prime, jobs=%d", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// With the cache warm, the same request serves the index immediately (no 503).
|
||||||
|
func TestServeRemoteWarmCacheServesImmediately(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
_ = newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("warm scan: %v", err)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-deb/Packages", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "Packages", "https://x", store) {
|
||||||
|
t.Fatal("ServeRemote did not handle Packages")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("warm cache must serve 200, got %d body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A prime job (freshness 0) runs even right after a sync; a periodic job at the
|
||||||
|
// same moment is gated by the recency window.
|
||||||
|
func TestSyncerPrimeBypassesRecencyPeriodicDoesNot(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
var _ provider.RemoteMetadataStore = store
|
||||||
|
|
||||||
|
s.process(context.Background(), syncJob{remote: remote, prime: true})
|
||||||
|
if rows, _ := store.ListDebMetadataEntries(context.Background(), remote.Name); len(rows) != 1 {
|
||||||
|
t.Fatalf("prime did not derive: %d rows", len(rows))
|
||||||
|
}
|
||||||
|
releasesAfterPrime := fx.releasesHit
|
||||||
|
|
||||||
|
s.process(context.Background(), syncJob{remote: remote, prime: false})
|
||||||
|
if fx.releasesHit != releasesAfterPrime {
|
||||||
|
t.Fatalf("periodic scan ran inside recency window: %d -> %d releases calls", releasesAfterPrime, fx.releasesHit)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -4,9 +4,11 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/base64"
|
"encoding/base64"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/url"
|
||||||
"path"
|
"path"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/provider"
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
"git.unkin.net/unkin/artifactapi/pkg/models"
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
)
|
)
|
||||||
@@ -59,10 +61,42 @@ func (p *Provider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte,
|
|||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Provider) AuthHeaders(_ context.Context, remote models.Remote) (http.Header, error) {
|
// AuthHeaders authenticates outbound requests. A per-remote username/password
|
||||||
|
// (Basic auth) takes precedence. Otherwise, when the remote points at a GitHub
|
||||||
|
// host (e.g. a releases_remote proxying private release assets), the process-wide
|
||||||
|
// GitHub credential is attached as a bearer token so private downloads work.
|
||||||
|
func (p *Provider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
|
||||||
h := http.Header{}
|
h := http.Header{}
|
||||||
if remote.Username != "" {
|
if remote.Username != "" {
|
||||||
h.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(remote.Username+":"+remote.Password)))
|
h.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(remote.Username+":"+remote.Password)))
|
||||||
|
return h, nil
|
||||||
|
}
|
||||||
|
if isGitHubHost(remote.BaseURL) {
|
||||||
|
if c := githubauth.Server(); c != nil {
|
||||||
|
tok, err := c.Token(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if tok != "" {
|
||||||
|
h.Set("Authorization", "Bearer "+tok)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return h, nil
|
return h, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// isGitHubHost reports whether rawURL targets a GitHub API/download host that
|
||||||
|
// accepts the server credential. objects.githubusercontent.com is deliberately
|
||||||
|
// excluded: release-asset downloads 302-redirect there with a pre-signed URL
|
||||||
|
// that must not carry an Authorization header.
|
||||||
|
func isGitHubHost(rawURL string) bool {
|
||||||
|
u, err := url.Parse(rawURL)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
switch strings.ToLower(u.Hostname()) {
|
||||||
|
case "github.com", "www.github.com", "api.github.com", "codeload.github.com", "uploads.github.com":
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|||||||
@@ -4,11 +4,56 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/provider"
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
"git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
||||||
"git.unkin.net/unkin/artifactapi/pkg/models"
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
type staticCred string
|
||||||
|
|
||||||
|
func (s staticCred) Token(context.Context) (string, error) { return string(s), nil }
|
||||||
|
|
||||||
|
func TestProvider_AuthHeaders_GitHubServerCredential(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghs_server"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
p := &generic.Provider{}
|
||||||
|
h, err := p.AuthHeaders(context.Background(), models.Remote{BaseURL: "https://github.com"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("auth headers: %v", err)
|
||||||
|
}
|
||||||
|
if h.Get("Authorization") != "Bearer ghs_server" {
|
||||||
|
t.Fatalf("Authorization = %q, want Bearer ghs_server", h.Get("Authorization"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProvider_AuthHeaders_NonGitHubHostNoServerCredential(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghs_server"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
p := &generic.Provider{}
|
||||||
|
h, _ := p.AuthHeaders(context.Background(), models.Remote{BaseURL: "https://example.com/downloads"})
|
||||||
|
if h.Get("Authorization") != "" {
|
||||||
|
t.Fatalf("server credential must not be sent to non-github host, got %q", h.Get("Authorization"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProvider_AuthHeaders_PerRemoteOverridesServerCredential(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghs_server"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
p := &generic.Provider{}
|
||||||
|
h, _ := p.AuthHeaders(context.Background(), models.Remote{
|
||||||
|
BaseURL: "https://github.com",
|
||||||
|
Username: "user",
|
||||||
|
Password: "pass",
|
||||||
|
})
|
||||||
|
if got := h.Get("Authorization"); got != "Basic dXNlcjpwYXNz" {
|
||||||
|
t.Fatalf("per-remote Basic auth must win, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestProvider_Type(t *testing.T) {
|
func TestProvider_Type(t *testing.T) {
|
||||||
p := &generic.Provider{}
|
p := &generic.Provider{}
|
||||||
if p.Type() != models.PackageGeneric {
|
if p.Type() != models.PackageGeneric {
|
||||||
|
|||||||
@@ -61,16 +61,60 @@ type PostDeleteHook interface {
|
|||||||
|
|
||||||
type MetadataStore interface {
|
type MetadataStore interface {
|
||||||
InsertRPMMetadata(ctx context.Context, meta *RPMMetadata) error
|
InsertRPMMetadata(ctx context.Context, meta *RPMMetadata) error
|
||||||
|
InsertDebMetadata(ctx context.Context, meta *DebMetadata) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// RemoteServer lets a remote provider fully answer a request itself instead of
|
||||||
|
// going through the byte-proxy engine. It is the remote-side analog of
|
||||||
|
// LocalIndexer: a metadata-only remote (e.g. github_rpm) uses it to synthesize
|
||||||
|
// repodata from derived per-asset metadata and to redirect package downloads to
|
||||||
|
// a backend remote, without ever precaching the packages. Returning false lets
|
||||||
|
// the normal proxy path take over.
|
||||||
|
type RemoteServer interface {
|
||||||
|
ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store RemoteMetadataStore) bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// RemoteMetadataStore is the persistence surface a RemoteServer needs to cache
|
||||||
|
// and read the metadata it derives per upstream asset. *database.DB satisfies it.
|
||||||
|
type RemoteMetadataStore interface {
|
||||||
|
RPMMetadataReader
|
||||||
|
MetadataStore
|
||||||
|
MetadataDeleter
|
||||||
}
|
}
|
||||||
|
|
||||||
type MetadataDeleter interface {
|
type MetadataDeleter interface {
|
||||||
DeleteRPMMetadata(ctx context.Context, repoName, filePath string) error
|
DeleteRPMMetadata(ctx context.Context, repoName, filePath string) error
|
||||||
|
DeleteDebMetadata(ctx context.Context, repoName, filePath string) error
|
||||||
}
|
}
|
||||||
|
|
||||||
type RPMMetadataReader interface {
|
type RPMMetadataReader interface {
|
||||||
ListRPMMetadataEntries(ctx context.Context, repoName string) ([]RPMMetadata, error)
|
ListRPMMetadataEntries(ctx context.Context, repoName string) ([]RPMMetadata, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DebMetadataReader is the read surface the deb LocalIndexer needs to
|
||||||
|
// regenerate a flat apt repository (Packages/Release) from stored rows.
|
||||||
|
// *database.DB satisfies it; ServeLocalIndex type-asserts the FileStore to it,
|
||||||
|
// mirroring how the rpm provider reaches its RPMMetadataReader.
|
||||||
|
type DebMetadataReader interface {
|
||||||
|
ListDebMetadataEntries(ctx context.Context, repoName string) ([]DebMetadata, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DebMetadata is the derived per-package metadata for a Debian .deb, carrying
|
||||||
|
// the full raw control stanza so the Packages index can be regenerated
|
||||||
|
// faithfully alongside the computed size/md5/sha256 apt requires.
|
||||||
|
type DebMetadata struct {
|
||||||
|
RepoName string
|
||||||
|
FilePath string
|
||||||
|
ContentHash string
|
||||||
|
Name string
|
||||||
|
Version string
|
||||||
|
Architecture string
|
||||||
|
Control string
|
||||||
|
Size int64
|
||||||
|
MD5 string
|
||||||
|
SHA256 string
|
||||||
|
}
|
||||||
|
|
||||||
type RPMMetadata struct {
|
type RPMMetadata struct {
|
||||||
RepoName string
|
RepoName string
|
||||||
FilePath string
|
FilePath string
|
||||||
@@ -93,6 +137,8 @@ type RPMMetadata struct {
|
|||||||
Packager string
|
Packager string
|
||||||
Requires []RPMDep
|
Requires []RPMDep
|
||||||
Provides []RPMDep
|
Provides []RPMDep
|
||||||
|
Conflicts []RPMDep
|
||||||
|
Obsoletes []RPMDep
|
||||||
Files []RPMFile
|
Files []RPMFile
|
||||||
Changelogs []RPMChangelog
|
Changelogs []RPMChangelog
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,732 @@
|
|||||||
|
package rpm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
rpmlib "github.com/cavaliergopher/rpm"
|
||||||
|
"golang.org/x/time/rate"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// gitHubProvider is the process-wide singleton. The background Syncer binds its
|
||||||
|
// shared rate limiter and work queue onto this instance so the request path and
|
||||||
|
// the syncer drive the same derive machinery.
|
||||||
|
var gitHubProvider = newGitHubProvider()
|
||||||
|
|
||||||
|
func init() {
|
||||||
|
provider.Register(gitHubProvider)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tuning knobs for the no-precache header fetch. Fields (not consts) so tests
|
||||||
|
// can shrink them against small fixtures.
|
||||||
|
const (
|
||||||
|
defaultHeaderRangeInitial = 1 << 20 // 1 MiB — covers the header of almost every RPM
|
||||||
|
defaultHeaderRangeMax = 16 << 20 // 16 MiB — give up past this and skip the asset
|
||||||
|
defaultReleasePageCap = 10 // 100 releases/page * 10 pages
|
||||||
|
|
||||||
|
// defaultScanTimeout bounds a detached background scan (which may do one
|
||||||
|
// ranged fetch per asset across every release) so it can never run forever.
|
||||||
|
defaultScanTimeout = 10 * time.Minute
|
||||||
|
// defaultServeTimeout bounds a repodata DB read served on a detached context.
|
||||||
|
defaultServeTimeout = 30 * time.Second
|
||||||
|
|
||||||
|
// defaultColdWait bounds how long a repodata request blocks waiting for a
|
||||||
|
// just-enqueued prime to populate an empty cache before returning a
|
||||||
|
// retryable 503. Kept short so a client never hangs on a rate-limited derive
|
||||||
|
// of a large repo; small repos usually prime within this window.
|
||||||
|
defaultColdWait = 8 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// GitHubProvider is a metadata-only remote: it scans a GitHub repo's releases
|
||||||
|
// for .rpm assets, derives per-asset RPM metadata via a ranged header fetch
|
||||||
|
// (never downloading whole packages), synthesizes yum repodata from that cached
|
||||||
|
// metadata, and redirects package downloads to a backend "releases_remote"
|
||||||
|
// (the generic github.com remote) that serves the actual bytes.
|
||||||
|
type GitHubProvider struct {
|
||||||
|
client *http.Client
|
||||||
|
|
||||||
|
headerInitial int64
|
||||||
|
headerMax int64
|
||||||
|
pageCap int
|
||||||
|
scanTimeout time.Duration
|
||||||
|
serveTimeout time.Duration
|
||||||
|
coldWait time.Duration
|
||||||
|
|
||||||
|
// limiter, when set by the Syncer, gates every GitHub HTTP call (releases
|
||||||
|
// list + each ranged asset fetch) through a single process-wide token bucket.
|
||||||
|
// nil means unlimited (direct provider use / unit tests).
|
||||||
|
limiter *rate.Limiter
|
||||||
|
// syncer, when set, routes freshness refresh and cold-start priming through
|
||||||
|
// the shared background work queue instead of an inline per-replica scan.
|
||||||
|
syncer *Syncer
|
||||||
|
|
||||||
|
// serverCred overrides the process-wide GitHub credential for this provider
|
||||||
|
// instance. nil falls back to githubauth.Server(); set directly in tests.
|
||||||
|
serverCred githubauth.Credential
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
scanning map[string]bool
|
||||||
|
lastScan map[string]time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func newGitHubProvider() *GitHubProvider {
|
||||||
|
return &GitHubProvider{
|
||||||
|
client: &http.Client{},
|
||||||
|
headerInitial: defaultHeaderRangeInitial,
|
||||||
|
headerMax: defaultHeaderRangeMax,
|
||||||
|
pageCap: defaultReleasePageCap,
|
||||||
|
scanTimeout: defaultScanTimeout,
|
||||||
|
serveTimeout: defaultServeTimeout,
|
||||||
|
coldWait: defaultColdWait,
|
||||||
|
scanning: map[string]bool{},
|
||||||
|
lastScan: map[string]time.Time{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// limiterWait blocks until the shared rate limiter grants a token, or returns
|
||||||
|
// the context error if it is canceled first. A nil limiter is a no-op.
|
||||||
|
func (p *GitHubProvider) limiterWait(ctx context.Context) error {
|
||||||
|
if p.limiter == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return p.limiter.Wait(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) Type() models.PackageType { return models.PackageGitHubRPM }
|
||||||
|
|
||||||
|
// Classify/ContentType/UpstreamURL/RewriteResponse/AuthHeaders satisfy the
|
||||||
|
// Provider interface. The proxy engine never reaches them for this type because
|
||||||
|
// ServeRemote handles every request, but they must exist for registry lookup.
|
||||||
|
func (p *GitHubProvider) Classify(path string) provider.Mutability {
|
||||||
|
if strings.HasPrefix(path, "repodata/") {
|
||||||
|
return provider.Mutable
|
||||||
|
}
|
||||||
|
return provider.Immutable
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) ContentType(path string) string {
|
||||||
|
switch {
|
||||||
|
case strings.HasSuffix(path, ".rpm"):
|
||||||
|
return "application/x-rpm"
|
||||||
|
case strings.HasSuffix(path, ".xml.gz"):
|
||||||
|
return "application/gzip"
|
||||||
|
case strings.HasSuffix(path, ".xml"):
|
||||||
|
return "application/xml"
|
||||||
|
}
|
||||||
|
return "application/octet-stream"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) UpstreamURL(remote models.Remote, path string) string {
|
||||||
|
return strings.TrimRight(remote.BaseURL, "/") + "/" + strings.TrimLeft(path, "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) RewriteResponse(_ []byte, _ models.Remote, _ string) ([]byte, error) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error) {
|
||||||
|
return p.githubHeaders(ctx, remote, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ServeRemote answers a request against a github_rpm remote. It refreshes the
|
||||||
|
// derived metadata (bounded by mutable_ttl), serves synthesized repodata, and
|
||||||
|
// 302-redirects .rpm downloads to the backend releases_remote. Returns false
|
||||||
|
// only for paths it does not own, letting the normal proxy path take over.
|
||||||
|
func (p *GitHubProvider) ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store provider.RemoteMetadataStore) bool {
|
||||||
|
p.onRequest(remote, store)
|
||||||
|
|
||||||
|
if strings.HasPrefix(path, "repodata/") {
|
||||||
|
// Serve repodata on a context detached from the inbound request: a
|
||||||
|
// client disconnect (e.g. dnf makecache timing out) must never cancel
|
||||||
|
// the metadata DB read and surface as a 500.
|
||||||
|
sctx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), p.serveTimeout)
|
||||||
|
defer cancel()
|
||||||
|
sr := r.WithContext(sctx)
|
||||||
|
|
||||||
|
// Cold start: with the syncer wired, an empty cache means the prime has
|
||||||
|
// not landed yet. Enqueue it and wait briefly rather than serving empty
|
||||||
|
// repodata; if it still has not primed, return a retryable 503.
|
||||||
|
if p.syncer != nil && !p.ensurePrimed(sctx, remote, store) {
|
||||||
|
w.Header().Set("Retry-After", "5")
|
||||||
|
http.Error(w, "metadata is being prepared, retry shortly", http.StatusServiceUnavailable)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
tail := strings.TrimPrefix(path, "repodata/")
|
||||||
|
lp := &Provider{}
|
||||||
|
switch {
|
||||||
|
case tail == "repomd.xml":
|
||||||
|
lp.serveRepomd(w, sr, store, remote.Name)
|
||||||
|
case strings.HasSuffix(tail, "-primary.xml.gz"):
|
||||||
|
lp.servePrimary(w, sr, store, remote.Name)
|
||||||
|
case strings.HasSuffix(tail, "-filelists.xml.gz"):
|
||||||
|
lp.serveFilelists(w, sr, store, remote.Name)
|
||||||
|
case strings.HasSuffix(tail, "-other.xml.gz"):
|
||||||
|
lp.serveOther(w, sr, store, remote.Name)
|
||||||
|
default:
|
||||||
|
http.Error(w, "not found", http.StatusNotFound)
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.HasSuffix(path, ".rpm") {
|
||||||
|
if remote.ReleasesRemote == "" {
|
||||||
|
http.Error(w, "github_rpm remote has no releases_remote configured for downloads", http.StatusInternalServerError)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
loc := strings.TrimRight(proxyBaseURL, "/") + "/api/v1/remote/" + remote.ReleasesRemote + "/" + strings.TrimLeft(path, "/")
|
||||||
|
http.Redirect(w, r, loc, http.StatusFound)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// onRequest keeps a remote's derived metadata fresh off the request path. With
|
||||||
|
// the background syncer wired it enqueues a deduped, rate-limited, lease-gated
|
||||||
|
// refresh and returns immediately; the request always serves the current cache.
|
||||||
|
// Without a syncer (direct provider use / unit tests) it falls back to the
|
||||||
|
// legacy inline single-flight scan.
|
||||||
|
func (p *GitHubProvider) onRequest(remote models.Remote, store provider.RemoteMetadataStore) {
|
||||||
|
if p.syncer != nil {
|
||||||
|
p.syncer.enqueue(remote, false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.refresh(remote, store)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ensurePrimed returns true once the remote has at least one cached metadata
|
||||||
|
// row. On an empty cache it enqueues a prime and polls briefly for it to land,
|
||||||
|
// so the very first client after a remote is created gets real repodata instead
|
||||||
|
// of an empty index or a blocking multi-minute derive. Returns false if the
|
||||||
|
// cache is still empty after the bounded wait.
|
||||||
|
func (p *GitHubProvider) ensurePrimed(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) bool {
|
||||||
|
if !p.cacheEmpty(ctx, store, remote.Name) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if p.syncer != nil {
|
||||||
|
p.syncer.enqueue(remote, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
deadline := time.Now().Add(p.coldWait)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return false
|
||||||
|
case <-time.After(400 * time.Millisecond):
|
||||||
|
}
|
||||||
|
if !p.cacheEmpty(ctx, store, remote.Name) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) cacheEmpty(ctx context.Context, store provider.RemoteMetadataStore, name string) bool {
|
||||||
|
rows, err := store.ListRPMMetadataEntries(ctx, name)
|
||||||
|
if err != nil {
|
||||||
|
// Treat a failed read as "not empty" so a transient DB error becomes a
|
||||||
|
// normal serve attempt (which reports its own error) rather than a 503.
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return len(rows) == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// refresh brings the derived metadata up to date without coupling the scan to
|
||||||
|
// the inbound request. When the cache is stale it single-flights a scan: if the
|
||||||
|
// cache already holds rows the scan runs in the background and the caller serves
|
||||||
|
// the current cache immediately; only a completely empty cache blocks on a
|
||||||
|
// bounded first scan (so the first client sees packages rather than an empty or
|
||||||
|
// 500 repodata).
|
||||||
|
func (p *GitHubProvider) refresh(remote models.Remote, store provider.RemoteMetadataStore) {
|
||||||
|
ttl := time.Duration(remote.MutableTTL) * time.Second
|
||||||
|
if ttl <= 0 {
|
||||||
|
ttl = 5 * time.Minute
|
||||||
|
}
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
last, ok := p.lastScan[remote.Name]
|
||||||
|
fresh := ok && time.Since(last) < ttl
|
||||||
|
if fresh || p.scanning[remote.Name] {
|
||||||
|
p.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p.scanning[remote.Name] = true
|
||||||
|
p.mu.Unlock()
|
||||||
|
|
||||||
|
empty := true
|
||||||
|
if rows, err := store.ListRPMMetadataEntries(context.Background(), remote.Name); err == nil {
|
||||||
|
empty = len(rows) == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
if empty {
|
||||||
|
p.runScan(remote, store)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
go p.runScan(remote, store)
|
||||||
|
}
|
||||||
|
|
||||||
|
// runScan derives metadata on a detached, bounded context so a client cancel
|
||||||
|
// can neither abort the shared derive nor poison the metadata read. The caller
|
||||||
|
// must have already claimed the single-flight slot (scanning[name] = true).
|
||||||
|
func (p *GitHubProvider) runScan(remote models.Remote, store provider.RemoteMetadataStore) {
|
||||||
|
defer func() {
|
||||||
|
p.mu.Lock()
|
||||||
|
delete(p.scanning, remote.Name)
|
||||||
|
p.mu.Unlock()
|
||||||
|
}()
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), p.scanTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
if err := p.scan(ctx, remote, store); err != nil {
|
||||||
|
// Keep serving whatever metadata is already cached rather than 500ing.
|
||||||
|
slog.Error("github_rpm: release scan failed", "remote", remote.Name, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
p.lastScan[remote.Name] = time.Now()
|
||||||
|
p.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// scan runs a full unconditional derive. Retained for the legacy inline refresh
|
||||||
|
// path and existing tests; the syncer uses scanWithState to pass and receive the
|
||||||
|
// releases-list ETag.
|
||||||
|
func (p *GitHubProvider) scan(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore) error {
|
||||||
|
_, _, err := p.scanWithState(ctx, remote, store, "")
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanWithState derives metadata incrementally. It sends the prior releases-list
|
||||||
|
// ETag as a conditional request: a 304 means nothing changed, so it returns
|
||||||
|
// (etag, changed=false) without a single asset fetch. On a 200 it diffs the
|
||||||
|
// release assets against the cache, derives only new/changed assets, prunes
|
||||||
|
// assets that disappeared, and returns the new ETag.
|
||||||
|
func (p *GitHubProvider) scanWithState(ctx context.Context, remote models.Remote, store provider.RemoteMetadataStore, etag string) (newEtag string, changed bool, err error) {
|
||||||
|
releases, newEtag, notModified, err := p.fetchReleases(ctx, remote, etag)
|
||||||
|
if err != nil {
|
||||||
|
return etag, false, err
|
||||||
|
}
|
||||||
|
if notModified {
|
||||||
|
return etag, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
existing, err := store.ListRPMMetadataEntries(ctx, remote.Name)
|
||||||
|
if err != nil {
|
||||||
|
return newEtag, false, err
|
||||||
|
}
|
||||||
|
existingByPath := make(map[string]provider.RPMMetadata, len(existing))
|
||||||
|
for _, m := range existing {
|
||||||
|
existingByPath[m.FilePath] = m
|
||||||
|
}
|
||||||
|
|
||||||
|
allow, err := compilePatterns(remote.Patterns)
|
||||||
|
if err != nil {
|
||||||
|
return newEtag, false, err
|
||||||
|
}
|
||||||
|
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, rel := range releases {
|
||||||
|
if rel.Draft {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, asset := range rel.Assets {
|
||||||
|
if !strings.HasSuffix(strings.ToLower(asset.Name), ".rpm") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !matchesAny(allow, asset.Name) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fp := assetPath(asset)
|
||||||
|
if fp == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[fp] = true
|
||||||
|
|
||||||
|
if cur, ok := existingByPath[fp]; ok {
|
||||||
|
// Assets are effectively immutable; only re-derive when the
|
||||||
|
// upstream digest is known and no longer matches what we cached.
|
||||||
|
if asset.Digest == "" || cur.ContentHash == asset.Digest {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
_ = store.DeleteRPMMetadata(ctx, remote.Name, fp)
|
||||||
|
}
|
||||||
|
|
||||||
|
meta, err := p.deriveAsset(ctx, remote, asset, fp)
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("github_rpm: derive asset failed", "remote", remote.Name, "asset", asset.Name, "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := store.InsertRPMMetadata(ctx, meta); err != nil {
|
||||||
|
slog.Error("github_rpm: insert metadata failed", "remote", remote.Name, "asset", asset.Name, "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
slog.Info("github_rpm: derived asset", "remote", remote.Name, "name", meta.Name, "version", meta.Version, "arch", meta.Arch)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for fp := range existingByPath {
|
||||||
|
if !seen[fp] {
|
||||||
|
_ = store.DeleteRPMMetadata(ctx, remote.Name, fp)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return newEtag, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type ghRelease struct {
|
||||||
|
TagName string `json:"tag_name"`
|
||||||
|
Draft bool `json:"draft"`
|
||||||
|
Assets []ghAsset `json:"assets"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ghAsset struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Size int64 `json:"size"`
|
||||||
|
BrowserDownloadURL string `json:"browser_download_url"`
|
||||||
|
Digest string `json:"digest"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetchReleases lists a repo's releases. It sends the prior ETag as
|
||||||
|
// If-None-Match on page 1 (the newest releases, where a new one first appears):
|
||||||
|
// a 304 there means the repo is unchanged, so it returns notModified without
|
||||||
|
// paging further — GitHub does not count 304 conditional responses against the
|
||||||
|
// rate limit, making an unchanged repo nearly free. On a 200 it captures the
|
||||||
|
// page-1 ETag and pages through the rest normally. Every call waits on the
|
||||||
|
// shared limiter first.
|
||||||
|
func (p *GitHubProvider) fetchReleases(ctx context.Context, remote models.Remote, etag string) (all []ghRelease, newEtag string, notModified bool, err error) {
|
||||||
|
base := strings.TrimRight(remote.BaseURL, "/") + "/releases"
|
||||||
|
for page := 1; page <= p.pageCap; page++ {
|
||||||
|
u := fmt.Sprintf("%s?per_page=100&page=%d", base, page)
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
hdr, err := p.githubHeaders(ctx, remote, true)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
|
if page == 1 && etag != "" {
|
||||||
|
req.Header.Set("If-None-Match", etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
if page == 1 && resp.StatusCode == http.StatusNotModified {
|
||||||
|
io.Copy(io.Discard, resp.Body)
|
||||||
|
resp.Body.Close()
|
||||||
|
return nil, etag, true, nil
|
||||||
|
}
|
||||||
|
body, err := io.ReadAll(resp.Body)
|
||||||
|
respEtag := resp.Header.Get("ETag")
|
||||||
|
resp.Body.Close()
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", false, err
|
||||||
|
}
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return nil, "", false, fmt.Errorf("github releases API %s: status %d", u, resp.StatusCode)
|
||||||
|
}
|
||||||
|
if page == 1 {
|
||||||
|
newEtag = respEtag
|
||||||
|
}
|
||||||
|
var releases []ghRelease
|
||||||
|
if err := json.Unmarshal(body, &releases); err != nil {
|
||||||
|
return nil, "", false, fmt.Errorf("decode releases: %w", err)
|
||||||
|
}
|
||||||
|
if len(releases) == 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
all = append(all, releases...)
|
||||||
|
if len(releases) < 100 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return all, newEtag, false, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) deriveAsset(ctx context.Context, remote models.Remote, asset ghAsset, fp string) (*provider.RPMMetadata, error) {
|
||||||
|
pkg, err := p.fetchHeader(ctx, remote, asset.BrowserDownloadURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
meta := &provider.RPMMetadata{
|
||||||
|
RepoName: remote.Name,
|
||||||
|
FilePath: fp,
|
||||||
|
Name: pkg.Name(),
|
||||||
|
Epoch: pkg.Epoch(),
|
||||||
|
Version: pkg.Version(),
|
||||||
|
Release: pkg.Release(),
|
||||||
|
Arch: pkg.Architecture(),
|
||||||
|
Summary: pkg.Summary(),
|
||||||
|
Description: pkg.Description(),
|
||||||
|
RPMSize: asset.Size,
|
||||||
|
InstalledSize: int64(pkg.Size()),
|
||||||
|
License: pkg.License(),
|
||||||
|
Vendor: pkg.Vendor(),
|
||||||
|
Group: firstGroup(pkg.Groups()),
|
||||||
|
BuildHost: pkg.BuildHost(),
|
||||||
|
SourceRPM: pkg.SourceRPM(),
|
||||||
|
URL: pkg.URL(),
|
||||||
|
Packager: pkg.Packager(),
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, d := range pkg.Requires() {
|
||||||
|
meta.Requires = append(meta.Requires, rpmDepFromEntry(d))
|
||||||
|
}
|
||||||
|
for _, d := range pkg.Provides() {
|
||||||
|
meta.Provides = append(meta.Provides, rpmDepFromEntry(d))
|
||||||
|
}
|
||||||
|
for _, d := range pkg.Conflicts() {
|
||||||
|
meta.Conflicts = append(meta.Conflicts, rpmDepFromEntry(d))
|
||||||
|
}
|
||||||
|
for _, d := range pkg.Obsoletes() {
|
||||||
|
meta.Obsoletes = append(meta.Obsoletes, rpmDepFromEntry(d))
|
||||||
|
}
|
||||||
|
for _, f := range pkg.Files() {
|
||||||
|
rf := provider.RPMFile{Path: f.Name()}
|
||||||
|
if f.IsDir() {
|
||||||
|
rf.Type = "dir"
|
||||||
|
}
|
||||||
|
meta.Files = append(meta.Files, rf)
|
||||||
|
}
|
||||||
|
|
||||||
|
if meta.Requires == nil {
|
||||||
|
meta.Requires = []provider.RPMDep{}
|
||||||
|
}
|
||||||
|
if meta.Provides == nil {
|
||||||
|
meta.Provides = []provider.RPMDep{}
|
||||||
|
}
|
||||||
|
if meta.Conflicts == nil {
|
||||||
|
meta.Conflicts = []provider.RPMDep{}
|
||||||
|
}
|
||||||
|
if meta.Obsoletes == nil {
|
||||||
|
meta.Obsoletes = []provider.RPMDep{}
|
||||||
|
}
|
||||||
|
if meta.Files == nil {
|
||||||
|
meta.Files = []provider.RPMFile{}
|
||||||
|
}
|
||||||
|
meta.Changelogs = []provider.RPMChangelog{}
|
||||||
|
|
||||||
|
// The primary.xml pkgid checksum must be the sha256 of the whole package.
|
||||||
|
// Prefer GitHub's asset digest so we never download the body; only when it
|
||||||
|
// is absent (or not sha256) do we stream the asset once to compute it.
|
||||||
|
if h, ok := sha256FromDigest(asset.Digest); ok {
|
||||||
|
meta.ContentHash = "sha256:" + h
|
||||||
|
} else {
|
||||||
|
h, err := p.computeSHA256(ctx, remote, asset.BrowserDownloadURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("compute sha256: %w", err)
|
||||||
|
}
|
||||||
|
meta.ContentHash = "sha256:" + h
|
||||||
|
}
|
||||||
|
|
||||||
|
return meta, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// fetchHeader pulls only the front of the package with a ranged GET and parses
|
||||||
|
// the RPM header from it. The header sits before the payload, so a small prefix
|
||||||
|
// is enough; on a truncated-header parse error it doubles the range and retries.
|
||||||
|
func (p *GitHubProvider) fetchHeader(ctx context.Context, remote models.Remote, downloadURL string) (*rpmlib.Package, error) {
|
||||||
|
n := p.headerInitial
|
||||||
|
for {
|
||||||
|
body, full, err := p.rangeGet(ctx, remote, downloadURL, n)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
pkg, perr := rpmlib.Read(bytes.NewReader(body))
|
||||||
|
if perr == nil {
|
||||||
|
return pkg, nil
|
||||||
|
}
|
||||||
|
truncated := errors.Is(perr, io.ErrUnexpectedEOF) || errors.Is(perr, io.EOF)
|
||||||
|
if truncated && !full && n < p.headerMax {
|
||||||
|
n *= 2
|
||||||
|
if n > p.headerMax {
|
||||||
|
n = p.headerMax
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("parse rpm header: %w", perr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// rangeGet returns the first n bytes of downloadURL. full is true when the
|
||||||
|
// response body was shorter than n (i.e. we already have the whole object).
|
||||||
|
func (p *GitHubProvider) rangeGet(ctx context.Context, remote models.Remote, downloadURL string, n int64) ([]byte, bool, error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
hdr, err := p.githubHeaders(ctx, remote, false)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
|
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", n-1))
|
||||||
|
|
||||||
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
|
||||||
|
return nil, false, fmt.Errorf("range GET %s: status %d", downloadURL, resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
body, err := io.ReadAll(io.LimitReader(resp.Body, n))
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, err
|
||||||
|
}
|
||||||
|
full := int64(len(body)) < n
|
||||||
|
return body, full, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GitHubProvider) computeSHA256(ctx context.Context, remote models.Remote, downloadURL string) (string, error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, downloadURL, nil)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
hdr, err := p.githubHeaders(ctx, remote, false)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
copyHeaders(req, hdr)
|
||||||
|
|
||||||
|
if err := p.limiterWait(ctx); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
return "", fmt.Errorf("GET %s: status %d", downloadURL, resp.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
h := sha256.New()
|
||||||
|
if _, err := io.Copy(h, resp.Body); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return hex.EncodeToString(h.Sum(nil)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// assetPath is the package's location relative to github.com — the path the
|
||||||
|
// backend releases_remote (base https://github.com) proxies. It doubles as the
|
||||||
|
// rpm_metadata key and the <location href> in primary.xml.
|
||||||
|
func assetPath(asset ghAsset) string {
|
||||||
|
u, err := url.Parse(asset.BrowserDownloadURL)
|
||||||
|
if err != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.TrimPrefix(u.Path, "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
func sha256FromDigest(digest string) (string, bool) {
|
||||||
|
if strings.HasPrefix(digest, "sha256:") {
|
||||||
|
return strings.TrimPrefix(digest, "sha256:"), true
|
||||||
|
}
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
// githubHeaders builds the outbound headers for a GitHub request, attaching a
|
||||||
|
// bearer credential when one is available. A per-remote credential wins; absent
|
||||||
|
// that, the process-wide server credential is used; absent both, the request is
|
||||||
|
// unauthenticated (anonymous, subject to the 60/hr cap).
|
||||||
|
func (p *GitHubProvider) githubHeaders(ctx context.Context, remote models.Remote, api bool) (http.Header, error) {
|
||||||
|
h := http.Header{}
|
||||||
|
if api {
|
||||||
|
h.Set("Accept", "application/vnd.github+json")
|
||||||
|
h.Set("X-GitHub-Api-Version", "2022-11-28")
|
||||||
|
}
|
||||||
|
tok, err := p.githubToken(ctx, remote)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if tok != "" {
|
||||||
|
h.Set("Authorization", "Bearer "+tok)
|
||||||
|
}
|
||||||
|
return h, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// githubToken resolves the bearer token for a remote. Precedence: a per-remote
|
||||||
|
// credential (password, then username) overrides the server credential.
|
||||||
|
func (p *GitHubProvider) githubToken(ctx context.Context, remote models.Remote) (string, error) {
|
||||||
|
if remote.Password != "" {
|
||||||
|
return remote.Password, nil
|
||||||
|
}
|
||||||
|
if remote.Username != "" {
|
||||||
|
return remote.Username, nil
|
||||||
|
}
|
||||||
|
if c := p.serverCredential(); c != nil {
|
||||||
|
return c.Token(ctx)
|
||||||
|
}
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// serverCredential returns this provider's server credential, defaulting to the
|
||||||
|
// process-wide one installed at startup.
|
||||||
|
func (p *GitHubProvider) serverCredential() githubauth.Credential {
|
||||||
|
if p.serverCred != nil {
|
||||||
|
return p.serverCred
|
||||||
|
}
|
||||||
|
return githubauth.Server()
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyHeaders(req *http.Request, h http.Header) {
|
||||||
|
for k, vals := range h {
|
||||||
|
for _, v := range vals {
|
||||||
|
req.Header.Add(k, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func compilePatterns(patterns []string) ([]*regexp.Regexp, error) {
|
||||||
|
var out []*regexp.Regexp
|
||||||
|
for _, p := range patterns {
|
||||||
|
re, err := regexp.Compile(p)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid pattern %q: %w", p, err)
|
||||||
|
}
|
||||||
|
out = append(out, re)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func matchesAny(res []*regexp.Regexp, s string) bool {
|
||||||
|
if len(res) == 0 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, re := range res {
|
||||||
|
if re.MatchString(s) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
package rpm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// staticCred is a test Credential yielding a fixed token.
|
||||||
|
type staticCred string
|
||||||
|
|
||||||
|
func (s staticCred) Token(context.Context) (string, error) { return string(s), nil }
|
||||||
|
|
||||||
|
func TestGitHubServerCredentialAttachedToReleasesAndAssets(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
p.serverCred = staticCred("ghp_server_secret")
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if got := fx.releaseAuth; got != "Bearer ghp_server_secret" {
|
||||||
|
t.Fatalf("releases Authorization = %q, want Bearer ghp_server_secret", got)
|
||||||
|
}
|
||||||
|
if got := fx.assetAuth; got != "Bearer ghp_server_secret" {
|
||||||
|
t.Fatalf("asset Authorization = %q, want Bearer ghp_server_secret", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubPerRemoteCredentialOverridesServer(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
p.serverCred = staticCred("ghp_server_secret")
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
remote := fx.remote()
|
||||||
|
remote.Password = "ghp_remote_wins"
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if got := fx.releaseAuth; got != "Bearer ghp_remote_wins" {
|
||||||
|
t.Fatalf("releases Authorization = %q, want per-remote token to win", got)
|
||||||
|
}
|
||||||
|
if got := fx.assetAuth; got != "Bearer ghp_remote_wins" {
|
||||||
|
t.Fatalf("asset Authorization = %q, want per-remote token to win", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubNoCredentialSendsNoAuthHeader(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider() // serverCred nil, package Server() unset in unit tests
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if fx.releaseAuth != "" {
|
||||||
|
t.Fatalf("expected no Authorization header, got %q", fx.releaseAuth)
|
||||||
|
}
|
||||||
|
if fx.assetAuth != "" {
|
||||||
|
t.Fatalf("expected no asset Authorization header, got %q", fx.assetAuth)
|
||||||
|
}
|
||||||
|
// Requests still succeed anonymously.
|
||||||
|
if rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm"); len(rows) != 1 {
|
||||||
|
t.Fatalf("anonymous scan should still derive metadata, got %d rows", len(rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubETag304FlowWithAuth(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
p := newTestProvider()
|
||||||
|
p.serverCred = staticCred("ghp_server_secret")
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
etag, changed, err := p.scanWithState(context.Background(), fx.remote(), store, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first scan: %v", err)
|
||||||
|
}
|
||||||
|
if !changed || etag != `"v1"` {
|
||||||
|
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-scan with the captured ETag: a 304 means no change and no asset fetch.
|
||||||
|
etag2, changed2, err := p.scanWithState(context.Background(), fx.remote(), store, etag)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second scan: %v", err)
|
||||||
|
}
|
||||||
|
if changed2 {
|
||||||
|
t.Fatal("expected no change on 304")
|
||||||
|
}
|
||||||
|
if etag2 != `"v1"` {
|
||||||
|
t.Fatalf("etag = %q, want preserved \"v1\"", etag2)
|
||||||
|
}
|
||||||
|
if fx.notModHit != 1 {
|
||||||
|
t.Fatalf("expected exactly one 304 response, got %d", fx.notModHit)
|
||||||
|
}
|
||||||
|
// The conditional request still carried the credential.
|
||||||
|
if fx.releaseAuth != "Bearer ghp_server_secret" {
|
||||||
|
t.Fatalf("conditional request Authorization = %q, want the server credential", fx.releaseAuth)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGitHubCredentialAbsentFromRemoteJSON asserts the server credential never
|
||||||
|
// appears in a remote's serialized API representation, and per-remote secrets
|
||||||
|
// stay redacted by the models.Remote json:"-" tags.
|
||||||
|
func TestGitHubCredentialAbsentFromRemoteJSON(t *testing.T) {
|
||||||
|
githubauth.SetServer(staticCred("ghp_super_secret_server_token"))
|
||||||
|
t.Cleanup(func() { githubauth.SetServer(nil) })
|
||||||
|
|
||||||
|
remote := models.Remote{
|
||||||
|
Name: "acme-rpm",
|
||||||
|
PackageType: models.PackageGitHubRPM,
|
||||||
|
BaseURL: "https://api.github.com/repos/acme/tools",
|
||||||
|
Username: "per_remote_user",
|
||||||
|
Password: "per_remote_secret",
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(remote)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal remote: %v", err)
|
||||||
|
}
|
||||||
|
js := string(b)
|
||||||
|
for _, secret := range []string{"ghp_super_secret_server_token", "per_remote_secret", "per_remote_user"} {
|
||||||
|
if strings.Contains(js, secret) {
|
||||||
|
t.Fatalf("credential %q leaked into remote JSON: %s", secret, js)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,390 @@
|
|||||||
|
package rpm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"compress/gzip"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/testsupport"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeStore is an in-memory provider.RemoteMetadataStore keyed by file_path,
|
||||||
|
// mirroring the (repo_name, file_path) uniqueness of the real table.
|
||||||
|
type fakeStore struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
rows map[string]provider.RPMMetadata
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeStore() *fakeStore { return &fakeStore{rows: map[string]provider.RPMMetadata{}} }
|
||||||
|
|
||||||
|
func (f *fakeStore) InsertRPMMetadata(_ context.Context, m *provider.RPMMetadata) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
if _, ok := f.rows[m.FilePath]; ok {
|
||||||
|
return nil // ON CONFLICT DO NOTHING
|
||||||
|
}
|
||||||
|
f.rows[m.FilePath] = *m
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) DeleteRPMMetadata(_ context.Context, _, filePath string) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
delete(f.rows, filePath)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
|
||||||
|
func (f *fakeStore) DeleteDebMetadata(context.Context, string, string) error { return nil }
|
||||||
|
|
||||||
|
func (f *fakeStore) ListRPMMetadataEntries(ctx context.Context, _ string) ([]provider.RPMMetadata, error) {
|
||||||
|
// Mirror pgx: a canceled/expired context fails the read. This is what
|
||||||
|
// poisons the repodata response if the read runs on the inbound request.
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
out := make([]provider.RPMMetadata, 0, len(f.rows))
|
||||||
|
for _, m := range f.rows {
|
||||||
|
out = append(out, m)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// githubFixture serves the releases API and the .rpm asset downloads (with
|
||||||
|
// Range support) for a set of packages. digest controls whether the asset
|
||||||
|
// carries a sha256 digest (no-download path) or not (compute path).
|
||||||
|
type githubFixture struct {
|
||||||
|
srv *httptest.Server
|
||||||
|
rpmBytes map[string][]byte // asset filename -> bytes
|
||||||
|
rangeHit map[string]int // asset filename -> number of ranged GETs
|
||||||
|
fullHit map[string]int // asset filename -> number of full GETs
|
||||||
|
etag string // when set, served as ETag; matching If-None-Match yields 304
|
||||||
|
releasesHit int // total releases-list requests (200 + 304)
|
||||||
|
notModHit int // releases-list requests answered 304
|
||||||
|
releaseAuth string // Authorization header seen on the last releases request
|
||||||
|
assetAuth string // Authorization header seen on the last asset request
|
||||||
|
mu sync.Mutex
|
||||||
|
}
|
||||||
|
|
||||||
|
func newGitHubFixture(t *testing.T, withDigest bool) *githubFixture {
|
||||||
|
t.Helper()
|
||||||
|
f := &githubFixture{
|
||||||
|
rpmBytes: map[string][]byte{},
|
||||||
|
rangeHit: map[string]int{},
|
||||||
|
fullHit: map[string]int{},
|
||||||
|
}
|
||||||
|
f.rpmBytes["demo-1.2-3.x86_64.rpm"] = testsupport.MinimalRPM("demo", "1.2", "3", "x86_64")
|
||||||
|
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("/repos/acme/tools/releases", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
page := r.URL.Query().Get("page")
|
||||||
|
if page != "" && page != "1" {
|
||||||
|
w.Write([]byte("[]"))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f.mu.Lock()
|
||||||
|
f.releasesHit++
|
||||||
|
f.releaseAuth = r.Header.Get("Authorization")
|
||||||
|
etag := f.etag
|
||||||
|
if etag != "" && r.Header.Get("If-None-Match") == etag {
|
||||||
|
f.notModHit++
|
||||||
|
f.mu.Unlock()
|
||||||
|
w.WriteHeader(http.StatusNotModified)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f.mu.Unlock()
|
||||||
|
if etag != "" {
|
||||||
|
w.Header().Set("ETag", etag)
|
||||||
|
}
|
||||||
|
var assets []map[string]any
|
||||||
|
for name := range f.rpmBytes {
|
||||||
|
a := map[string]any{
|
||||||
|
"name": name,
|
||||||
|
"size": len(f.rpmBytes[name]),
|
||||||
|
"browser_download_url": f.srv.URL + "/acme/tools/releases/download/v1.2-3/" + name,
|
||||||
|
}
|
||||||
|
if withDigest {
|
||||||
|
sum := sha256.Sum256(f.rpmBytes[name])
|
||||||
|
a["digest"] = "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
assets = append(assets, a)
|
||||||
|
}
|
||||||
|
rel := []map[string]any{{"tag_name": "v1.2-3", "draft": false, "assets": assets}}
|
||||||
|
json.NewEncoder(w).Encode(rel)
|
||||||
|
})
|
||||||
|
mux.HandleFunc("/acme/tools/releases/download/", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
name := r.URL.Path[strings.LastIndex(r.URL.Path, "/")+1:]
|
||||||
|
body, ok := f.rpmBytes[name]
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "not found", 404)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
rng := r.Header.Get("Range")
|
||||||
|
f.mu.Lock()
|
||||||
|
f.assetAuth = r.Header.Get("Authorization")
|
||||||
|
if rng != "" {
|
||||||
|
f.rangeHit[name]++
|
||||||
|
} else {
|
||||||
|
f.fullHit[name]++
|
||||||
|
}
|
||||||
|
f.mu.Unlock()
|
||||||
|
|
||||||
|
if rng == "" {
|
||||||
|
w.WriteHeader(200)
|
||||||
|
w.Write(body)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Parse "bytes=0-N".
|
||||||
|
var end int
|
||||||
|
fmt.Sscanf(rng, "bytes=0-%d", &end)
|
||||||
|
if end >= len(body)-1 {
|
||||||
|
end = len(body) - 1
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Range", fmt.Sprintf("bytes 0-%d/%d", end, len(body)))
|
||||||
|
w.Header().Set("Content-Length", strconv.Itoa(end+1))
|
||||||
|
w.WriteHeader(http.StatusPartialContent)
|
||||||
|
w.Write(body[:end+1])
|
||||||
|
})
|
||||||
|
f.srv = httptest.NewServer(mux)
|
||||||
|
t.Cleanup(f.srv.Close)
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *githubFixture) remote() models.Remote {
|
||||||
|
return models.Remote{
|
||||||
|
Name: "acme-rpm",
|
||||||
|
PackageType: models.PackageGitHubRPM,
|
||||||
|
BaseURL: f.srv.URL + "/repos/acme/tools",
|
||||||
|
ReleasesRemote: "github",
|
||||||
|
MutableTTL: 3600,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTestProvider() *GitHubProvider {
|
||||||
|
p := newGitHubProvider()
|
||||||
|
p.headerInitial = 32 // force the ranged-fetch retry loop against the tiny fixture
|
||||||
|
p.headerMax = 1 << 20
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubScanDerivesMetadataFromHeaderAndDigest(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
metas, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm")
|
||||||
|
if len(metas) != 1 {
|
||||||
|
t.Fatalf("want 1 metadata row, got %d", len(metas))
|
||||||
|
}
|
||||||
|
m := metas[0]
|
||||||
|
if m.Name != "demo" || m.Version != "1.2" || m.Release != "3" || m.Arch != "x86_64" {
|
||||||
|
t.Fatalf("bad NEVRA: %+v", m)
|
||||||
|
}
|
||||||
|
// location href / redirect key must be the github-relative download path.
|
||||||
|
wantPath := "acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"
|
||||||
|
if m.FilePath != wantPath {
|
||||||
|
t.Fatalf("FilePath = %q, want %q", m.FilePath, wantPath)
|
||||||
|
}
|
||||||
|
if int(m.RPMSize) != len(fx.rpmBytes["demo-1.2-3.x86_64.rpm"]) {
|
||||||
|
t.Fatalf("RPMSize = %d, want %d", m.RPMSize, len(fx.rpmBytes["demo-1.2-3.x86_64.rpm"]))
|
||||||
|
}
|
||||||
|
// Digest present => checksum from digest, no full download.
|
||||||
|
sum := sha256.Sum256(fx.rpmBytes["demo-1.2-3.x86_64.rpm"])
|
||||||
|
if m.ContentHash != "sha256:"+hex.EncodeToString(sum[:]) {
|
||||||
|
t.Fatalf("ContentHash = %q, want digest", m.ContentHash)
|
||||||
|
}
|
||||||
|
if fx.fullHit["demo-1.2-3.x86_64.rpm"] != 0 {
|
||||||
|
t.Fatalf("expected no full download when digest present, got %d", fx.fullHit["demo-1.2-3.x86_64.rpm"])
|
||||||
|
}
|
||||||
|
if fx.rangeHit["demo-1.2-3.x86_64.rpm"] == 0 {
|
||||||
|
t.Fatalf("expected ranged header fetch")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubChecksumComputedWhenDigestAbsent(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, false)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
metas, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm")
|
||||||
|
if len(metas) != 1 {
|
||||||
|
t.Fatalf("want 1 row, got %d", len(metas))
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(fx.rpmBytes["demo-1.2-3.x86_64.rpm"])
|
||||||
|
if metas[0].ContentHash != "sha256:"+hex.EncodeToString(sum[:]) {
|
||||||
|
t.Fatalf("computed checksum mismatch: %q", metas[0].ContentHash)
|
||||||
|
}
|
||||||
|
if fx.fullHit["demo-1.2-3.x86_64.rpm"] == 0 {
|
||||||
|
t.Fatalf("expected a full download to compute sha256 when digest absent")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubServeRemoteRepodataAndRedirect(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
const proxyBase = "https://artifactapi.example"
|
||||||
|
|
||||||
|
// repomd.xml is served and triggers the initial scan.
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", proxyBase, store) {
|
||||||
|
t.Fatal("ServeRemote did not handle repomd.xml")
|
||||||
|
}
|
||||||
|
if rec.Code != 200 || !strings.Contains(rec.Body.String(), "<repomd") {
|
||||||
|
t.Fatalf("repomd bad: code=%d body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// primary.xml.gz must carry the package with a location href that is the
|
||||||
|
// github-relative download path (so it resolves back to this remote and
|
||||||
|
// redirects to the backend).
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
req = httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "repodata/abc-primary.xml.gz", proxyBase, store) {
|
||||||
|
t.Fatal("ServeRemote did not handle primary")
|
||||||
|
}
|
||||||
|
gz, err := gzip.NewReader(rec.Body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("gzip: %v", err)
|
||||||
|
}
|
||||||
|
xmlBytes, _ := io.ReadAll(gz)
|
||||||
|
primary := string(xmlBytes)
|
||||||
|
if !strings.Contains(primary, `<name>demo</name>`) {
|
||||||
|
t.Fatalf("primary missing package: %s", primary)
|
||||||
|
}
|
||||||
|
if !strings.Contains(primary, `<location href="acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"/>`) {
|
||||||
|
t.Fatalf("primary missing/incorrect location href: %s", primary)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A .rpm request redirects to the backend releases_remote.
|
||||||
|
rec = httptest.NewRecorder()
|
||||||
|
pkgPath := "acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"
|
||||||
|
req = httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/"+pkgPath, nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, pkgPath, proxyBase, store) {
|
||||||
|
t.Fatal("ServeRemote did not handle .rpm")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusFound {
|
||||||
|
t.Fatalf("want 302, got %d", rec.Code)
|
||||||
|
}
|
||||||
|
wantLoc := proxyBase + "/api/v1/remote/github/" + pkgPath
|
||||||
|
if got := rec.Header().Get("Location"); got != wantLoc {
|
||||||
|
t.Fatalf("Location = %q, want %q", got, wantLoc)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGitHubServeRemoteCanceledRequestServesCache reproduces the cold-makecache
|
||||||
|
// 500: when the inbound request context is already canceled (dnf timed out and
|
||||||
|
// disconnected), the repodata read must not be run on that context and turned
|
||||||
|
// into a 500. With the cache already warm, the handler serves it as 200.
|
||||||
|
// Before the fix the read used r.Context() and returned 500; after the fix it
|
||||||
|
// runs on a detached context and serves the cached repomd.
|
||||||
|
func TestGitHubServeRemoteCanceledRequestServesCache(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
// Warm the cache and mark the scan fresh so ServeRemote does not re-derive.
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("warm scan: %v", err)
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
p.lastScan[remote.Name] = time.Now()
|
||||||
|
p.mu.Unlock()
|
||||||
|
|
||||||
|
// Inbound request whose context is already canceled (client went away).
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil).WithContext(ctx)
|
||||||
|
|
||||||
|
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", "https://x", store) {
|
||||||
|
t.Fatal("ServeRemote did not handle repomd.xml")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("canceled request must serve cache, not error; got code=%d body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(rec.Body.String(), "<repomd") {
|
||||||
|
t.Fatalf("expected repomd served from cache, got %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubServeRemoteRedirectRequiresReleasesRemote(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
remote.ReleasesRemote = ""
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
pkgPath := "acme/tools/releases/download/v1.2-3/demo-1.2-3.x86_64.rpm"
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, pkgPath, "https://x", store) {
|
||||||
|
t.Fatal("expected handled")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusInternalServerError {
|
||||||
|
t.Fatalf("want 500 when releases_remote unset, got %d", rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubScanPrunesRemovedAssets(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
if rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm"); len(rows) != 1 {
|
||||||
|
t.Fatalf("want 1 row after first scan, got %d", len(rows))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remove the asset upstream; a rescan must prune the stale metadata row.
|
||||||
|
delete(fx.rpmBytes, "demo-1.2-3.x86_64.rpm")
|
||||||
|
if err := p.scan(context.Background(), fx.remote(), store); err != nil {
|
||||||
|
t.Fatalf("rescan: %v", err)
|
||||||
|
}
|
||||||
|
if rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm"); len(rows) != 0 {
|
||||||
|
t.Fatalf("want 0 rows after prune, got %d", len(rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGitHubAssetPatternFilter(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.rpmBytes["other-9-9.aarch64.rpm"] = testsupport.MinimalRPM("other", "9", "9", "aarch64")
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
remote.Patterns = []string{`^demo-.*\.x86_64\.rpm$`}
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("scan: %v", err)
|
||||||
|
}
|
||||||
|
rows, _ := store.ListRPMMetadataEntries(context.Background(), "acme-rpm")
|
||||||
|
if len(rows) != 1 || rows[0].Name != "demo" {
|
||||||
|
t.Fatalf("pattern filter failed, rows=%+v", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"encoding/hex"
|
"encoding/hex"
|
||||||
"encoding/xml"
|
"encoding/xml"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -133,6 +134,12 @@ func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, conte
|
|||||||
for _, prov := range pkg.Provides() {
|
for _, prov := range pkg.Provides() {
|
||||||
meta.Provides = append(meta.Provides, rpmDepFromEntry(prov))
|
meta.Provides = append(meta.Provides, rpmDepFromEntry(prov))
|
||||||
}
|
}
|
||||||
|
for _, con := range pkg.Conflicts() {
|
||||||
|
meta.Conflicts = append(meta.Conflicts, rpmDepFromEntry(con))
|
||||||
|
}
|
||||||
|
for _, obs := range pkg.Obsoletes() {
|
||||||
|
meta.Obsoletes = append(meta.Obsoletes, rpmDepFromEntry(obs))
|
||||||
|
}
|
||||||
|
|
||||||
if meta.Requires == nil {
|
if meta.Requires == nil {
|
||||||
meta.Requires = []provider.RPMDep{}
|
meta.Requires = []provider.RPMDep{}
|
||||||
@@ -140,6 +147,12 @@ func (p *Provider) AfterUpload(ctx context.Context, repoName, storagePath, conte
|
|||||||
if meta.Provides == nil {
|
if meta.Provides == nil {
|
||||||
meta.Provides = []provider.RPMDep{}
|
meta.Provides = []provider.RPMDep{}
|
||||||
}
|
}
|
||||||
|
if meta.Conflicts == nil {
|
||||||
|
meta.Conflicts = []provider.RPMDep{}
|
||||||
|
}
|
||||||
|
if meta.Obsoletes == nil {
|
||||||
|
meta.Obsoletes = []provider.RPMDep{}
|
||||||
|
}
|
||||||
meta.Files = []provider.RPMFile{}
|
meta.Files = []provider.RPMFile{}
|
||||||
meta.Changelogs = []provider.RPMChangelog{}
|
meta.Changelogs = []provider.RPMChangelog{}
|
||||||
|
|
||||||
@@ -229,10 +242,28 @@ func (p *Provider) GenerateLocalIndex(ctx context.Context, files provider.FileSt
|
|||||||
return nil, fmt.Errorf("rpm local index generation for virtual repos not supported")
|
return nil, fmt.Errorf("rpm local index generation for virtual repos not supported")
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
// readMetadataEntries loads the repo's derived metadata, translating the read
|
||||||
|
// error into an HTTP response. A canceled/deadline-exceeded context (typically a
|
||||||
|
// client that went away) becomes a retryable 503 rather than a hard 500, so a
|
||||||
|
// dnf disconnect never looks like a server fault. ok is false when a response
|
||||||
|
// has already been written.
|
||||||
|
func readMetadataEntries(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) ([]provider.RPMMetadata, bool) {
|
||||||
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
|
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||||
|
slog.Warn("rpm: metadata read canceled", "repo", repoName, "error", err)
|
||||||
|
http.Error(w, "metadata read canceled", http.StatusServiceUnavailable)
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
return metas, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
||||||
|
metas, ok := readMetadataEntries(w, r, reader, repoName)
|
||||||
|
if !ok {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -252,9 +283,8 @@ func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader pr
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (p *Provider) servePrimary(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
func (p *Provider) servePrimary(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
||||||
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
|
metas, ok := readMetadataEntries(w, r, reader, repoName)
|
||||||
if err != nil {
|
if !ok {
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -264,9 +294,8 @@ func (p *Provider) servePrimary(w http.ResponseWriter, r *http.Request, reader p
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (p *Provider) serveFilelists(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
func (p *Provider) serveFilelists(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
||||||
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
|
metas, ok := readMetadataEntries(w, r, reader, repoName)
|
||||||
if err != nil {
|
if !ok {
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -276,9 +305,8 @@ func (p *Provider) serveFilelists(w http.ResponseWriter, r *http.Request, reader
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (p *Provider) serveOther(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
func (p *Provider) serveOther(w http.ResponseWriter, r *http.Request, reader provider.RPMMetadataReader, repoName string) {
|
||||||
metas, err := reader.ListRPMMetadataEntries(r.Context(), repoName)
|
metas, ok := readMetadataEntries(w, r, reader, repoName)
|
||||||
if err != nil {
|
if !ok {
|
||||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -363,6 +391,20 @@ func generatePrimaryXMLGZ(metas []provider.RPMMetadata) []byte {
|
|||||||
}
|
}
|
||||||
xmlBuf.WriteString(" </rpm:requires>\n")
|
xmlBuf.WriteString(" </rpm:requires>\n")
|
||||||
}
|
}
|
||||||
|
if len(m.Conflicts) > 0 {
|
||||||
|
xmlBuf.WriteString(" <rpm:conflicts>\n")
|
||||||
|
for _, d := range m.Conflicts {
|
||||||
|
writeRPMEntry(&xmlBuf, d)
|
||||||
|
}
|
||||||
|
xmlBuf.WriteString(" </rpm:conflicts>\n")
|
||||||
|
}
|
||||||
|
if len(m.Obsoletes) > 0 {
|
||||||
|
xmlBuf.WriteString(" <rpm:obsoletes>\n")
|
||||||
|
for _, d := range m.Obsoletes {
|
||||||
|
writeRPMEntry(&xmlBuf, d)
|
||||||
|
}
|
||||||
|
xmlBuf.WriteString(" </rpm:obsoletes>\n")
|
||||||
|
}
|
||||||
|
|
||||||
fmt.Fprintf(&xmlBuf, " </format>\n")
|
fmt.Fprintf(&xmlBuf, " </format>\n")
|
||||||
fmt.Fprintf(&xmlBuf, "</package>\n")
|
fmt.Fprintf(&xmlBuf, "</package>\n")
|
||||||
|
|||||||
@@ -28,6 +28,8 @@ func (f *fakeMetaStore) InsertRPMMetadata(_ context.Context, m *provider.RPMMeta
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (f *fakeMetaStore) InsertDebMetadata(context.Context, *provider.DebMetadata) error { return nil }
|
||||||
|
|
||||||
type fakeRPMReader struct{ metas []provider.RPMMetadata }
|
type fakeRPMReader struct{ metas []provider.RPMMetadata }
|
||||||
|
|
||||||
func (f fakeRPMReader) ListRPMMetadataEntries(_ context.Context, _ string) ([]provider.RPMMetadata, error) {
|
func (f fakeRPMReader) ListRPMMetadataEntries(_ context.Context, _ string) ([]provider.RPMMetadata, error) {
|
||||||
|
|||||||
@@ -0,0 +1,256 @@
|
|||||||
|
package rpm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/time/rate"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// syncLeaseDuration is how long a claimed sync lease is held before it is
|
||||||
|
// considered abandoned. It comfortably exceeds a scan's own timeout so a live
|
||||||
|
// scan never loses its lease, while a crashed replica's lease still expires.
|
||||||
|
syncLeaseDuration = 15 * time.Minute
|
||||||
|
// defaultSyncFreshness is the periodic re-check interval used when a remote's
|
||||||
|
// mutable_ttl is unset.
|
||||||
|
defaultSyncFreshness = 5 * time.Minute
|
||||||
|
// jobQueueDepth bounds the pending work queue; enqueues past it are dropped
|
||||||
|
// (a later poll re-enqueues), never blocking the caller.
|
||||||
|
jobQueueDepth = 256
|
||||||
|
)
|
||||||
|
|
||||||
|
// SyncStore is the persistence surface the syncer needs: the metadata cache it
|
||||||
|
// primes plus the shared sync-state coordination (remote enumeration and the
|
||||||
|
// per-remote lease). *database.DB satisfies it.
|
||||||
|
type SyncStore interface {
|
||||||
|
provider.RemoteMetadataStore
|
||||||
|
ListGitHubRPMRemotes(ctx context.Context) ([]models.Remote, error)
|
||||||
|
ClaimGitHubSyncLease(ctx context.Context, remoteName, owner string, freshness, lease time.Duration) (claimed bool, etag string, err error)
|
||||||
|
ReleaseGitHubSyncLease(ctx context.Context, remoteName, owner, etag string, syncedAt time.Time) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// SyncConfig tunes the shared syncer. Zero values fall back to safe defaults.
|
||||||
|
type SyncConfig struct {
|
||||||
|
RatePerSec float64 // global GitHub request rate (req/s)
|
||||||
|
Burst int // token-bucket burst
|
||||||
|
Workers int // concurrent scan workers
|
||||||
|
PollInterval time.Duration // base scheduler tick; per-remote cadence is mutable_ttl
|
||||||
|
}
|
||||||
|
|
||||||
|
type syncJob struct {
|
||||||
|
remote models.Remote
|
||||||
|
prime bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// Syncer is the single per-process background worker that keeps every
|
||||||
|
// github_rpm remote's derived metadata fresh. It owns a deduped work queue, a
|
||||||
|
// pool of workers, and a global token-bucket rate limiter shared across all
|
||||||
|
// remotes and bound onto the github provider so every GitHub call it makes
|
||||||
|
// passes through the same bucket. Periodic checks are gated by a shared DB lease
|
||||||
|
// so, across replicas, only one performs each scan.
|
||||||
|
type Syncer struct {
|
||||||
|
store SyncStore
|
||||||
|
prov *GitHubProvider
|
||||||
|
limiter *rate.Limiter
|
||||||
|
cfg SyncConfig
|
||||||
|
owner string
|
||||||
|
|
||||||
|
jobs chan syncJob
|
||||||
|
mu sync.Mutex
|
||||||
|
active map[string]bool // remotes queued or in-flight, for dedup/coalescing
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewSyncer builds the syncer bound to the process-wide github provider
|
||||||
|
// singleton. Call Run to start it.
|
||||||
|
func NewSyncer(store SyncStore, cfg SyncConfig) *Syncer {
|
||||||
|
return newSyncer(store, gitHubProvider, cfg)
|
||||||
|
}
|
||||||
|
|
||||||
|
func newSyncer(store SyncStore, prov *GitHubProvider, cfg SyncConfig) *Syncer {
|
||||||
|
if cfg.RatePerSec <= 0 {
|
||||||
|
cfg.RatePerSec = 1
|
||||||
|
}
|
||||||
|
if cfg.Burst <= 0 {
|
||||||
|
cfg.Burst = 5
|
||||||
|
}
|
||||||
|
if cfg.Workers <= 0 {
|
||||||
|
cfg.Workers = 3
|
||||||
|
}
|
||||||
|
if cfg.PollInterval <= 0 {
|
||||||
|
cfg.PollInterval = 60 * time.Second
|
||||||
|
}
|
||||||
|
|
||||||
|
lim := rate.NewLimiter(rate.Limit(cfg.RatePerSec), cfg.Burst)
|
||||||
|
s := &Syncer{
|
||||||
|
store: store,
|
||||||
|
prov: prov,
|
||||||
|
limiter: lim,
|
||||||
|
cfg: cfg,
|
||||||
|
owner: leaseOwner(),
|
||||||
|
jobs: make(chan syncJob, jobQueueDepth),
|
||||||
|
active: map[string]bool{},
|
||||||
|
}
|
||||||
|
// Bind the shared limiter and back-reference so the request path routes
|
||||||
|
// through this syncer and every derive HTTP call is rate limited.
|
||||||
|
prov.limiter = lim
|
||||||
|
prov.syncer = s
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run starts the worker pool and the periodic scheduler and blocks until ctx is
|
||||||
|
// canceled, at which point it drains in-flight scans and returns.
|
||||||
|
func (s *Syncer) Run(ctx context.Context) {
|
||||||
|
slog.Info("github_rpm syncer started",
|
||||||
|
"rate_per_sec", s.cfg.RatePerSec, "burst", s.cfg.Burst,
|
||||||
|
"workers", s.cfg.Workers, "poll_interval", s.cfg.PollInterval, "owner", s.owner)
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < s.cfg.Workers; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
s.worker(ctx)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
ticker := time.NewTicker(s.cfg.PollInterval)
|
||||||
|
defer ticker.Stop()
|
||||||
|
|
||||||
|
s.schedule(ctx) // sweep at boot so existing remotes are checked immediately
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
wg.Wait()
|
||||||
|
slog.Info("github_rpm syncer stopped")
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
s.schedule(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// schedule enqueues a periodic check for every github_rpm remote. The DB lease
|
||||||
|
// (claimed in the worker) enforces the per-remote mutable_ttl cadence and cross
|
||||||
|
// replica coordination, so enqueuing every tick is cheap: a not-yet-due remote
|
||||||
|
// simply fails to claim and is skipped.
|
||||||
|
func (s *Syncer) schedule(ctx context.Context) {
|
||||||
|
remotes, err := s.store.ListGitHubRPMRemotes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("github_rpm syncer: list remotes", "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, r := range remotes {
|
||||||
|
s.enqueue(r, false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// EnqueuePrime queues an immediate background prime for a freshly created
|
||||||
|
// remote so its metadata is derived without blocking the create call.
|
||||||
|
func (s *Syncer) EnqueuePrime(remote models.Remote) {
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.enqueue(remote, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
// enqueue adds a job unless the remote is already queued or in-flight, coalescing
|
||||||
|
// duplicate requests down to one scan. It never blocks: a full queue drops the
|
||||||
|
// job (a later poll re-enqueues it) after clearing the dedup slot.
|
||||||
|
func (s *Syncer) enqueue(remote models.Remote, prime bool) {
|
||||||
|
s.mu.Lock()
|
||||||
|
if s.active[remote.Name] {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.active[remote.Name] = true
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case s.jobs <- syncJob{remote: remote, prime: prime}:
|
||||||
|
default:
|
||||||
|
s.mu.Lock()
|
||||||
|
delete(s.active, remote.Name)
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Syncer) worker(ctx context.Context) {
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case job := <-s.jobs:
|
||||||
|
s.process(ctx, job)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// process claims the shared lease and, if won, runs an incremental scan. The
|
||||||
|
// lease bounds total GitHub load to one scan per freshness window across all
|
||||||
|
// replicas; losing the claim (another replica scanning, or not yet due) is a
|
||||||
|
// no-op.
|
||||||
|
func (s *Syncer) process(ctx context.Context, job syncJob) {
|
||||||
|
defer func() {
|
||||||
|
s.mu.Lock()
|
||||||
|
delete(s.active, job.remote.Name)
|
||||||
|
s.mu.Unlock()
|
||||||
|
}()
|
||||||
|
|
||||||
|
freshness := time.Duration(job.remote.MutableTTL) * time.Second
|
||||||
|
if freshness <= 0 {
|
||||||
|
freshness = defaultSyncFreshness
|
||||||
|
}
|
||||||
|
if job.prime {
|
||||||
|
freshness = 0 // prime ignores the recency gate but still respects a live lease
|
||||||
|
}
|
||||||
|
|
||||||
|
claimed, etag, err := s.store.ClaimGitHubSyncLease(ctx, job.remote.Name, s.owner, freshness, syncLeaseDuration)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("github_rpm syncer: claim lease", "remote", job.remote.Name, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !claimed {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
scanCtx, cancel := context.WithTimeout(ctx, s.prov.scanTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
newEtag, changed, scanErr := s.prov.scanWithState(scanCtx, job.remote, s.store, etag)
|
||||||
|
releaseEtag := etag
|
||||||
|
if scanErr == nil {
|
||||||
|
releaseEtag = newEtag
|
||||||
|
} else {
|
||||||
|
slog.Error("github_rpm syncer: scan failed", "remote", job.remote.Name, "error", scanErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Release on a detached context so a clean shutdown mid-scan still frees the
|
||||||
|
// lease and advances last_synced_at (otherwise it simply expires).
|
||||||
|
relCtx, relCancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
|
||||||
|
defer relCancel()
|
||||||
|
if err := s.store.ReleaseGitHubSyncLease(relCtx, job.remote.Name, s.owner, releaseEtag, time.Now()); err != nil {
|
||||||
|
slog.Warn("github_rpm syncer: release lease", "remote", job.remote.Name, "error", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if scanErr == nil && changed {
|
||||||
|
slog.Info("github_rpm syncer: refreshed", "remote", job.remote.Name, "prime", job.prime)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// leaseOwner is a per-replica identity for the lease: hostname plus a random
|
||||||
|
// suffix so restarts and colocated replicas never collide.
|
||||||
|
func leaseOwner() string {
|
||||||
|
host, _ := os.Hostname()
|
||||||
|
var b [6]byte
|
||||||
|
_, _ = rand.Read(b[:])
|
||||||
|
return host + "-" + hex.EncodeToString(b[:])
|
||||||
|
}
|
||||||
@@ -0,0 +1,312 @@
|
|||||||
|
package rpm
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"golang.org/x/time/rate"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/testsupport"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeSyncStore is an in-memory SyncStore: the metadata cache (via the embedded
|
||||||
|
// fakeStore) plus the shared sync-state lease, whose claim mirrors the atomic
|
||||||
|
// semantics of the real SQL (recency gate AND no live lease).
|
||||||
|
type fakeSyncStore struct {
|
||||||
|
*fakeStore
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
remotes []models.Remote
|
||||||
|
leaseOwner map[string]string
|
||||||
|
leaseExp map[string]time.Time
|
||||||
|
lastSynced map[string]time.Time
|
||||||
|
etags map[string]string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeSyncStore() *fakeSyncStore {
|
||||||
|
return &fakeSyncStore{
|
||||||
|
fakeStore: newFakeStore(),
|
||||||
|
leaseOwner: map[string]string{},
|
||||||
|
leaseExp: map[string]time.Time{},
|
||||||
|
lastSynced: map[string]time.Time{},
|
||||||
|
etags: map[string]string{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSyncStore) ListGitHubRPMRemotes(_ context.Context) ([]models.Remote, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
return append([]models.Remote(nil), f.remotes...), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSyncStore) ClaimGitHubSyncLease(_ context.Context, name, owner string, freshness, lease time.Duration) (bool, string, error) {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
now := time.Now()
|
||||||
|
ls, hasLS := f.lastSynced[name]
|
||||||
|
exp, hasExp := f.leaseExp[name]
|
||||||
|
freshOK := !hasLS || now.Sub(ls) >= freshness
|
||||||
|
leaseOK := !hasExp || exp.Before(now)
|
||||||
|
if freshOK && leaseOK {
|
||||||
|
f.leaseOwner[name] = owner
|
||||||
|
f.leaseExp[name] = now.Add(lease)
|
||||||
|
return true, f.etags[name], nil
|
||||||
|
}
|
||||||
|
return false, "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeSyncStore) ReleaseGitHubSyncLease(_ context.Context, name, owner, etag string, syncedAt time.Time) error {
|
||||||
|
f.mu.Lock()
|
||||||
|
defer f.mu.Unlock()
|
||||||
|
if f.leaseOwner[name] != owner {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
f.lastSynced[name] = syncedAt
|
||||||
|
f.etags[name] = etag
|
||||||
|
delete(f.leaseOwner, name)
|
||||||
|
delete(f.leaseExp, name)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func testSyncConfig() SyncConfig {
|
||||||
|
return SyncConfig{RatePerSec: 1000, Burst: 100, Workers: 1, PollInterval: time.Hour}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (a) A 304 conditional response must derive nothing: no asset header GETs and
|
||||||
|
// changed=false, so an unchanged repo is nearly free.
|
||||||
|
func TestSyncerConditionalNotModifiedSkipsDerive(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
etag1, changed, err := p.scanWithState(context.Background(), remote, store, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first scan: %v", err)
|
||||||
|
}
|
||||||
|
if !changed || etag1 != `"v1"` {
|
||||||
|
t.Fatalf("first scan changed=%v etag=%q, want true and \"v1\"", changed, etag1)
|
||||||
|
}
|
||||||
|
priorRange := fx.rangeHit["demo-1.2-3.x86_64.rpm"]
|
||||||
|
if priorRange == 0 {
|
||||||
|
t.Fatal("first scan should have fetched the asset header")
|
||||||
|
}
|
||||||
|
|
||||||
|
etag2, changed2, err := p.scanWithState(context.Background(), remote, store, etag1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second scan: %v", err)
|
||||||
|
}
|
||||||
|
if changed2 {
|
||||||
|
t.Fatal("304 scan must report changed=false")
|
||||||
|
}
|
||||||
|
if etag2 != etag1 {
|
||||||
|
t.Fatalf("etag changed across 304: %q -> %q", etag1, etag2)
|
||||||
|
}
|
||||||
|
if fx.notModHit != 1 {
|
||||||
|
t.Fatalf("want exactly one 304 releases response, got %d", fx.notModHit)
|
||||||
|
}
|
||||||
|
if got := fx.rangeHit["demo-1.2-3.x86_64.rpm"]; got != priorRange {
|
||||||
|
t.Fatalf("304 scan re-fetched asset header: %d -> %d", priorRange, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (b) On a real change, only the newly added asset is derived; assets already
|
||||||
|
// cached are never re-fetched.
|
||||||
|
func TestSyncerIncrementalDerivesOnlyNewAsset(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
p := newTestProvider()
|
||||||
|
store := newFakeStore()
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
if _, _, err := p.scanWithState(context.Background(), remote, store, ""); err != nil {
|
||||||
|
t.Fatalf("first scan: %v", err)
|
||||||
|
}
|
||||||
|
demoRange := fx.rangeHit["demo-1.2-3.x86_64.rpm"]
|
||||||
|
|
||||||
|
// Add a new asset and bump the ETag so the conditional request returns 200.
|
||||||
|
fx.rpmBytes["other-9-9.aarch64.rpm"] = testsupport.MinimalRPM("other", "9", "9", "aarch64")
|
||||||
|
fx.etag = `"v2"`
|
||||||
|
|
||||||
|
if _, changed, err := p.scanWithState(context.Background(), remote, store, `"v1"`); err != nil || !changed {
|
||||||
|
t.Fatalf("second scan changed=%v err=%v", changed, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rows, _ := store.ListRPMMetadataEntries(context.Background(), remote.Name)
|
||||||
|
if len(rows) != 2 {
|
||||||
|
t.Fatalf("want 2 cached rows after incremental derive, got %d", len(rows))
|
||||||
|
}
|
||||||
|
if got := fx.rangeHit["demo-1.2-3.x86_64.rpm"]; got != demoRange {
|
||||||
|
t.Fatalf("already-cached asset was re-fetched: %d -> %d", demoRange, got)
|
||||||
|
}
|
||||||
|
if fx.rangeHit["other-9-9.aarch64.rpm"] == 0 {
|
||||||
|
t.Fatal("newly added asset was not derived")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (c) The shared limiter caps the request rate: three gated releases calls at
|
||||||
|
// one token per 120ms cannot complete faster than ~2 gaps.
|
||||||
|
func TestRateLimiterCapsRequestRate(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
p := newTestProvider()
|
||||||
|
p.limiter = rate.NewLimiter(rate.Every(120*time.Millisecond), 1)
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
if _, _, _, err := p.fetchReleases(context.Background(), remote, ""); err != nil {
|
||||||
|
t.Fatalf("fetchReleases %d: %v", i, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed < 200*time.Millisecond {
|
||||||
|
t.Fatalf("rate limiter did not throttle: 3 calls took %v, want >= 200ms", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (d) Concurrent enqueues for the same remote coalesce to a single queued job.
|
||||||
|
func TestSyncerEnqueueDedup(t *testing.T) {
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := models.Remote{Name: "acme-rpm", PackageType: models.PackageGitHubRPM, MutableTTL: 3600}
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() { defer wg.Done(); s.enqueue(remote, false) }()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
if got := len(s.jobs); got != 1 {
|
||||||
|
t.Fatalf("want exactly 1 coalesced job, got %d", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (e) Prime-on-create enqueues a prime job.
|
||||||
|
func TestSyncerEnqueuePrime(t *testing.T) {
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := models.Remote{Name: "acme-rpm", PackageType: models.PackageGitHubRPM, MutableTTL: 3600}
|
||||||
|
|
||||||
|
s.EnqueuePrime(remote)
|
||||||
|
select {
|
||||||
|
case job := <-s.jobs:
|
||||||
|
if !job.prime || job.remote.Name != "acme-rpm" {
|
||||||
|
t.Fatalf("bad prime job: %+v", job)
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
t.Fatal("EnqueuePrime did not enqueue a job")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (f) A held lease prevents a second replica from scanning: with the lease owned
|
||||||
|
// by another replica, process claims nothing and makes zero GitHub calls.
|
||||||
|
func TestSyncerLeasePreventsSecondReplica(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
// Replica 1 holds the lease.
|
||||||
|
claimed, _, err := store.ClaimGitHubSyncLease(context.Background(), remote.Name, "replica-1", time.Duration(remote.MutableTTL)*time.Second, syncLeaseDuration)
|
||||||
|
if err != nil || !claimed {
|
||||||
|
t.Fatalf("replica-1 claim: claimed=%v err=%v", claimed, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Replica 2 (this syncer) tries to process the same remote; it must skip.
|
||||||
|
s.process(context.Background(), syncJob{remote: remote})
|
||||||
|
|
||||||
|
if fx.releasesHit != 0 {
|
||||||
|
t.Fatalf("second replica scanned while lease held: %d releases calls", fx.releasesHit)
|
||||||
|
}
|
||||||
|
if rows, _ := store.ListRPMMetadataEntries(context.Background(), remote.Name); len(rows) != 0 {
|
||||||
|
t.Fatalf("second replica derived metadata while lease held: %d rows", len(rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// With the syncer wired and the cache empty, a repodata request enqueues a
|
||||||
|
// prime and, when it has not landed within the bounded cold wait, returns a
|
||||||
|
// retryable 503 rather than serving empty repodata (and without regressing the
|
||||||
|
// detached-context serve).
|
||||||
|
func TestServeRemoteColdStartReturns503(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
p.coldWait = 300 * time.Millisecond
|
||||||
|
_ = newSyncer(store, p, testSyncConfig()) // binds p.syncer, but no workers running
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", "https://x", store) {
|
||||||
|
t.Fatal("ServeRemote did not handle repomd.xml")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusServiceUnavailable {
|
||||||
|
t.Fatalf("cold empty cache must return 503, got %d", rec.Code)
|
||||||
|
}
|
||||||
|
if rec.Header().Get("Retry-After") == "" {
|
||||||
|
t.Fatal("503 should carry Retry-After")
|
||||||
|
}
|
||||||
|
// The prime was enqueued.
|
||||||
|
if got := len(p.syncer.jobs); got != 1 {
|
||||||
|
t.Fatalf("cold start did not enqueue a prime, jobs=%d", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// With the cache warm, the same request serves repodata immediately (no 503).
|
||||||
|
func TestServeRemoteWarmCacheServesImmediately(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
_ = newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
if err := p.scan(context.Background(), remote, store); err != nil {
|
||||||
|
t.Fatalf("warm scan: %v", err)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/remote/acme-rpm/repodata/repomd.xml", nil)
|
||||||
|
if !p.ServeRemote(rec, req, remote, "repodata/repomd.xml", "https://x", store) {
|
||||||
|
t.Fatal("ServeRemote did not handle repomd.xml")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("warm cache must serve 200, got %d body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A prime job (freshness 0) runs even right after a sync, deriving metadata,
|
||||||
|
// while a periodic job at the same moment is gated by the recency window.
|
||||||
|
func TestSyncerPrimeBypassesRecencyPeriodicDoesNot(t *testing.T) {
|
||||||
|
fx := newGitHubFixture(t, true)
|
||||||
|
fx.etag = `"v1"`
|
||||||
|
store := newFakeSyncStore()
|
||||||
|
p := newTestProvider()
|
||||||
|
s := newSyncer(store, p, testSyncConfig())
|
||||||
|
remote := fx.remote()
|
||||||
|
|
||||||
|
var _ provider.RemoteMetadataStore = store
|
||||||
|
|
||||||
|
// Prime derives despite no prior sync.
|
||||||
|
s.process(context.Background(), syncJob{remote: remote, prime: true})
|
||||||
|
if rows, _ := store.ListRPMMetadataEntries(context.Background(), remote.Name); len(rows) != 1 {
|
||||||
|
t.Fatalf("prime did not derive: %d rows", len(rows))
|
||||||
|
}
|
||||||
|
releasesAfterPrime := fx.releasesHit
|
||||||
|
|
||||||
|
// A periodic job immediately after is gated by mutable_ttl recency: no new
|
||||||
|
// releases call.
|
||||||
|
s.process(context.Background(), syncJob{remote: remote, prime: false})
|
||||||
|
if fx.releasesHit != releasesAfterPrime {
|
||||||
|
t.Fatalf("periodic scan ran inside recency window: %d -> %d releases calls", releasesAfterPrime, fx.releasesHit)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -26,6 +26,27 @@ var providerZipRe = regexp.MustCompile(
|
|||||||
|
|
||||||
var semverRe = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(?:-[a-zA-Z0-9.]+)?$`)
|
var semverRe = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+(?:-[a-zA-Z0-9.]+)?$`)
|
||||||
|
|
||||||
|
// ParsedProviderZip describes a terraform-provider-{type}_{version}_{os}_{arch}.zip
|
||||||
|
// filename. Ok is false when the name doesn't match that convention.
|
||||||
|
type ParsedProviderZip struct {
|
||||||
|
Type string
|
||||||
|
Version string
|
||||||
|
OS string
|
||||||
|
Arch string
|
||||||
|
Ok bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseProviderZip extracts the type, version and platform from a provider zip
|
||||||
|
// filename (the base name, not a full path). It's the canonical parser shared by
|
||||||
|
// the network-mirror index and the provider registry handler.
|
||||||
|
func ParseProviderZip(filename string) ParsedProviderZip {
|
||||||
|
m := providerZipRe.FindStringSubmatch(filename)
|
||||||
|
if m == nil {
|
||||||
|
return ParsedProviderZip{}
|
||||||
|
}
|
||||||
|
return ParsedProviderZip{Type: m[1], Version: m[2], OS: m[3], Arch: m[4], Ok: true}
|
||||||
|
}
|
||||||
|
|
||||||
type Provider struct{}
|
type Provider struct{}
|
||||||
|
|
||||||
func (p *Provider) Type() models.PackageType { return models.PackageTerraform }
|
func (p *Provider) Type() models.PackageType { return models.PackageTerraform }
|
||||||
|
|||||||
@@ -0,0 +1,171 @@
|
|||||||
|
package terraform
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeFileStore struct{ entries []provider.FileEntry }
|
||||||
|
|
||||||
|
func (f fakeFileStore) ListFilesByPrefix(_ context.Context, _, prefix string) ([]provider.FileEntry, error) {
|
||||||
|
var out []provider.FileEntry
|
||||||
|
for _, e := range f.entries {
|
||||||
|
if strings.HasPrefix(e.FilePath, prefix) {
|
||||||
|
out = append(out, e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
func (f fakeFileStore) ListPackages(_ context.Context, _ string) ([]string, error) { return nil, nil }
|
||||||
|
|
||||||
|
func TestTFPureFuncs(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
if p.Classify("hashicorp/aws/versions") != provider.Mutable {
|
||||||
|
t.Error("versions should be mutable")
|
||||||
|
}
|
||||||
|
if p.Classify("hashicorp/aws/terraform-provider-aws_1.0.0_linux_amd64.zip") != provider.Immutable {
|
||||||
|
t.Error("zip should be immutable")
|
||||||
|
}
|
||||||
|
if got := p.UpstreamURL(models.Remote{BaseURL: "https://registry.terraform.io"}, "hashicorp/aws/versions"); got != "https://registry.terraform.io/v1/providers/hashicorp/aws/versions" {
|
||||||
|
t.Errorf("upstream url %q", got)
|
||||||
|
}
|
||||||
|
h, _ := p.AuthHeaders(context.Background(), models.Remote{Username: "u", Password: "p"})
|
||||||
|
if h.Get("Authorization") == "" {
|
||||||
|
t.Error("auth header")
|
||||||
|
}
|
||||||
|
_ = p.ContentType("x.json")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTFValidateUpload(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
sp, ct, err := p.ValidateUpload("hashicorp/aws/terraform-provider-aws_1.2.3_linux_amd64.zip")
|
||||||
|
if err != nil || sp != "hashicorp/aws/terraform-provider-aws_1.2.3_linux_amd64.zip" || ct != "application/zip" {
|
||||||
|
t.Errorf("valid: sp=%q ct=%q err=%v", sp, ct, err)
|
||||||
|
}
|
||||||
|
if _, _, err := p.ValidateUpload("too/few"); err == nil {
|
||||||
|
t.Error("expected error for wrong path depth")
|
||||||
|
}
|
||||||
|
if _, _, err := p.ValidateUpload("ns/aws/not-a-provider.zip"); err == nil {
|
||||||
|
t.Error("expected error for bad filename")
|
||||||
|
}
|
||||||
|
if _, _, err := p.ValidateUpload("ns/gcp/terraform-provider-aws_1.0.0_linux_amd64.zip"); err == nil {
|
||||||
|
t.Error("expected error for type mismatch")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTFUploadResponse(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
resp := p.UploadResponse("hashicorp/aws/terraform-provider-aws_1.2.3_linux_amd64.zip", "sha256:abc", 100)
|
||||||
|
if resp["namespace"] != "hashicorp" || resp["type"] != "aws" || resp["version"] != "1.2.3" || resp["os"] != "linux" || resp["arch"] != "amd64" {
|
||||||
|
t.Errorf("structured response wrong: %v", resp)
|
||||||
|
}
|
||||||
|
fallback := p.UploadResponse("weird/path", "sha256:x", 1)
|
||||||
|
if fallback["path"] != "weird/path" {
|
||||||
|
t.Errorf("fallback response wrong: %v", fallback)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTFRewriteResponse(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
remote := models.Remote{Name: "tf", ReleasesRemote: "hashicorp-releases"}
|
||||||
|
|
||||||
|
if out, _ := p.RewriteResponse([]byte(`{"download_url":"x"}`), models.Remote{}, "http://proxy"); out != nil {
|
||||||
|
t.Error("no ReleasesRemote should be a no-op")
|
||||||
|
}
|
||||||
|
if out, _ := p.RewriteResponse([]byte("not json"), remote, "http://proxy"); out != nil {
|
||||||
|
t.Error("invalid json should be a no-op")
|
||||||
|
}
|
||||||
|
body := []byte(`{"download_url":"https://releases.hashicorp.com/terraform-provider-aws/1.0/aws.zip"}`)
|
||||||
|
out, err := p.RewriteResponse(body, remote, "http://proxy")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(out), "http://proxy/api/v1/remote/hashicorp-releases/") {
|
||||||
|
t.Errorf("download_url not rewritten: %s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTFServeLocalIndex(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
fs := fakeFileStore{entries: []provider.FileEntry{
|
||||||
|
{FilePath: "hashicorp/aws/terraform-provider-aws_1.0.0_linux_amd64.zip", ContentHash: "sha256:deadbeef"},
|
||||||
|
{FilePath: "hashicorp/aws/terraform-provider-aws_1.0.0_darwin_arm64.zip", ContentHash: "sha256:cafe"},
|
||||||
|
}}
|
||||||
|
|
||||||
|
serve := func(path string) *httptest.ResponseRecorder {
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
|
||||||
|
p.ServeLocalIndex(w, r, fs, "repo", path)
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
if w := serve("hashicorp/aws/index.json"); w.Code != 200 || !strings.Contains(w.Body.String(), "1.0.0") {
|
||||||
|
t.Errorf("index.json: code=%d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
if w := serve("hashicorp/aws/1.0.0.json"); w.Code != 200 || !strings.Contains(w.Body.String(), "linux_amd64") {
|
||||||
|
t.Errorf("version doc: code=%d body=%s", w.Code, w.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Not a terraform index path.
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r := httptest.NewRequest(http.MethodGet, "/x", nil)
|
||||||
|
if p.ServeLocalIndex(w, r, fs, "repo", "hashicorp/aws/other.txt") {
|
||||||
|
t.Error("non-index path should return false")
|
||||||
|
}
|
||||||
|
if p.ServeLocalIndex(httptest.NewRecorder(), r, fs, "repo", "too/short") {
|
||||||
|
t.Error("short path should return false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTFContentTypeAndEmptyIndex(t *testing.T) {
|
||||||
|
p := &Provider{}
|
||||||
|
for path, want := range map[string]string{
|
||||||
|
"x.zip": "application/zip",
|
||||||
|
"x.sig": "application/octet-stream",
|
||||||
|
"index.json": "application/json",
|
||||||
|
} {
|
||||||
|
if got := p.ContentType(path); got != want {
|
||||||
|
t.Errorf("ContentType(%q)=%q want %q", path, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// index / version doc with no matching files -> 404.
|
||||||
|
empty := fakeFileStore{}
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
r := httptest.NewRequest(http.MethodGet, "/hashicorp/aws/index.json", nil)
|
||||||
|
p.ServeLocalIndex(w, r, empty, "repo", "hashicorp/aws/index.json")
|
||||||
|
if w.Code != http.StatusNotFound {
|
||||||
|
t.Errorf("empty index should be 404, got %d", w.Code)
|
||||||
|
}
|
||||||
|
w = httptest.NewRecorder()
|
||||||
|
p.ServeLocalIndex(w, r, empty, "repo", "hashicorp/aws/1.0.0.json")
|
||||||
|
if w.Code != http.StatusNotFound {
|
||||||
|
t.Errorf("empty version doc should be 404, got %d", w.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRewriteDownloadURL(t *testing.T) {
|
||||||
|
// Empty proxy base -> unchanged.
|
||||||
|
if got := rewriteDownloadURL("https://x/a.zip", "rel", ""); got != "https://x/a.zip" {
|
||||||
|
t.Errorf("empty base: %q", got)
|
||||||
|
}
|
||||||
|
// Unparseable URL -> unchanged.
|
||||||
|
if got := rewriteDownloadURL("://bad", "rel", "http://p"); got != "://bad" {
|
||||||
|
t.Errorf("bad url: %q", got)
|
||||||
|
}
|
||||||
|
// Normal rewrite.
|
||||||
|
if got := rewriteDownloadURL("https://cdn/path/a.zip", "rel", "http://p"); got != "http://p/api/v1/remote/rel/path/a.zip" {
|
||||||
|
t.Errorf("rewrite: %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTFGenerateLocalIndexUnsupported(t *testing.T) {
|
||||||
|
if _, err := (&Provider{}).GenerateLocalIndex(context.Background(), fakeFileStore{}, "r", "x"); err == nil {
|
||||||
|
t.Error("expected unsupported error")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -12,13 +12,16 @@ import (
|
|||||||
"github.com/go-chi/chi/v5"
|
"github.com/go-chi/chi/v5"
|
||||||
"github.com/go-chi/chi/v5/middleware"
|
"github.com/go-chi/chi/v5/middleware"
|
||||||
|
|
||||||
|
tfregistry "git.unkin.net/unkin/artifactapi/internal/api/terraform"
|
||||||
v1 "git.unkin.net/unkin/artifactapi/internal/api/v1"
|
v1 "git.unkin.net/unkin/artifactapi/internal/api/v1"
|
||||||
v2 "git.unkin.net/unkin/artifactapi/internal/api/v2"
|
v2 "git.unkin.net/unkin/artifactapi/internal/api/v2"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/cache"
|
"git.unkin.net/unkin/artifactapi/internal/cache"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/config"
|
"git.unkin.net/unkin/artifactapi/internal/config"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/database"
|
"git.unkin.net/unkin/artifactapi/internal/database"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/gc"
|
"git.unkin.net/unkin/artifactapi/internal/gc"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/githubauth"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/alpine"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/alpine"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/provider/deb"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/docker"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/docker"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/generic"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/goproxy"
|
||||||
@@ -26,11 +29,13 @@ import (
|
|||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/npm"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/npm"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/puppet"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/puppet"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/pypi"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/pypi"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/rpm"
|
"git.unkin.net/unkin/artifactapi/internal/provider/rpm"
|
||||||
_ "git.unkin.net/unkin/artifactapi/internal/provider/terraform"
|
_ "git.unkin.net/unkin/artifactapi/internal/provider/terraform"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/proxy"
|
"git.unkin.net/unkin/artifactapi/internal/proxy"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/storage"
|
"git.unkin.net/unkin/artifactapi/internal/storage"
|
||||||
|
"git.unkin.net/unkin/artifactapi/internal/tfsign"
|
||||||
"git.unkin.net/unkin/artifactapi/internal/virtual"
|
"git.unkin.net/unkin/artifactapi/internal/virtual"
|
||||||
|
"git.unkin.net/unkin/artifactapi/pkg/models"
|
||||||
)
|
)
|
||||||
|
|
||||||
type Server struct {
|
type Server struct {
|
||||||
@@ -43,7 +48,10 @@ type Server struct {
|
|||||||
engine *proxy.Engine
|
engine *proxy.Engine
|
||||||
virtEngine *virtual.Engine
|
virtEngine *virtual.Engine
|
||||||
localHandler *v2.LocalHandler
|
localHandler *v2.LocalHandler
|
||||||
|
tfRegistry *tfregistry.Handler
|
||||||
gc *gc.Collector
|
gc *gc.Collector
|
||||||
|
syncer *rpm.Syncer
|
||||||
|
debSyncer *deb.Syncer
|
||||||
}
|
}
|
||||||
|
|
||||||
func New(cfg *config.Config, version string) (*Server, error) {
|
func New(cfg *config.Config, version string) (*Server, error) {
|
||||||
@@ -62,10 +70,60 @@ func New(cfg *config.Config, version string) (*Server, error) {
|
|||||||
return nil, fmt.Errorf("s3: %w", err)
|
return nil, fmt.Errorf("s3: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Install the process-wide GitHub credential before any provider makes an
|
||||||
|
// outbound call. A misconfiguration (e.g. App id without a private key) fails
|
||||||
|
// closed here rather than silently falling back to anonymous. No credential
|
||||||
|
// configured is fine — requests stay anonymous.
|
||||||
|
ghCred, err := githubauth.New(githubauth.Options{
|
||||||
|
Token: cfg.GitHubToken,
|
||||||
|
AppID: cfg.GitHubAppID,
|
||||||
|
InstallationID: cfg.GitHubAppInstallationID,
|
||||||
|
PrivateKeyPEM: cfg.GitHubAppPrivateKey,
|
||||||
|
PrivateKeyPath: cfg.GitHubAppPrivateKeyPath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("github auth: %w", err)
|
||||||
|
}
|
||||||
|
githubauth.SetServer(ghCred)
|
||||||
|
if ghCred != nil {
|
||||||
|
slog.Info("github machine credential configured")
|
||||||
|
}
|
||||||
|
|
||||||
engine := proxy.NewEngine(db, redis, s3)
|
engine := proxy.NewEngine(db, redis, s3)
|
||||||
localHandler := v2.NewLocalHandler(db, s3)
|
localHandler := v2.NewLocalHandler(db, s3)
|
||||||
virtEngine := virtual.NewEngine(db, engine)
|
virtEngine := virtual.NewEngine(db, engine)
|
||||||
collector := gc.New(db, s3, 1*time.Hour)
|
collector := gc.New(db, s3, 1*time.Hour)
|
||||||
|
syncer := rpm.NewSyncer(db, rpm.SyncConfig{
|
||||||
|
RatePerSec: cfg.GitHubSyncRatePerSec,
|
||||||
|
Burst: cfg.GitHubSyncBurst,
|
||||||
|
Workers: cfg.GitHubSyncWorkers,
|
||||||
|
PollInterval: time.Duration(cfg.GitHubSyncPollInterval) * time.Second,
|
||||||
|
})
|
||||||
|
debSyncer := deb.NewSyncer(db, deb.SyncConfig{
|
||||||
|
RatePerSec: cfg.GitHubSyncRatePerSec,
|
||||||
|
Burst: cfg.GitHubSyncBurst,
|
||||||
|
Workers: cfg.GitHubSyncWorkers,
|
||||||
|
PollInterval: time.Duration(cfg.GitHubSyncPollInterval) * time.Second,
|
||||||
|
})
|
||||||
|
|
||||||
|
// The terraform registry signs with a GPG key. A configured file wins (BYO
|
||||||
|
// key); otherwise artifactapi generates one on first start and persists it in
|
||||||
|
// the database so every replica shares it. A failure here must not take the
|
||||||
|
// server down — the registry just stays disabled.
|
||||||
|
var signer *tfsign.Signer
|
||||||
|
if cfg.TFSigningKeyPath != "" {
|
||||||
|
signer, err = tfsign.Load(cfg.TFSigningKeyPath, cfg.TFSigningKeyPassphrase)
|
||||||
|
} else {
|
||||||
|
signer, err = tfsign.LoadOrCreate(context.Background(), db, "terraform-provider")
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("terraform provider registry disabled", "error", err)
|
||||||
|
signer = nil
|
||||||
|
}
|
||||||
|
tfRegistry := tfregistry.NewHandler(db, signer, cfg.TFProviderProtocols)
|
||||||
|
if tfRegistry.Enabled() {
|
||||||
|
slog.Info("terraform provider registry enabled", "key_id", signer.KeyID())
|
||||||
|
}
|
||||||
|
|
||||||
s := &Server{
|
s := &Server{
|
||||||
cfg: cfg,
|
cfg: cfg,
|
||||||
@@ -76,7 +134,10 @@ func New(cfg *config.Config, version string) (*Server, error) {
|
|||||||
engine: engine,
|
engine: engine,
|
||||||
virtEngine: virtEngine,
|
virtEngine: virtEngine,
|
||||||
localHandler: localHandler,
|
localHandler: localHandler,
|
||||||
|
tfRegistry: tfRegistry,
|
||||||
gc: collector,
|
gc: collector,
|
||||||
|
syncer: syncer,
|
||||||
|
debSyncer: debSyncer,
|
||||||
}
|
}
|
||||||
|
|
||||||
s.router = s.routes()
|
s.router = s.routes()
|
||||||
@@ -95,12 +156,21 @@ func (s *Server) routes() chi.Router {
|
|||||||
|
|
||||||
r.Get("/health", s.handleHealth)
|
r.Get("/health", s.handleHealth)
|
||||||
r.Get("/", s.handleRoot)
|
r.Get("/", s.handleRoot)
|
||||||
|
r.Get("/version", s.handleVersion)
|
||||||
|
|
||||||
|
// Terraform provider registry: service discovery at the well-known path,
|
||||||
|
// providers.v1 protocol under /terraform/v1/providers.
|
||||||
|
r.Get("/.well-known/terraform.json", s.tfRegistry.ServiceDiscovery)
|
||||||
|
r.Mount(tfregistry.MountPath, s.tfRegistry.Routes())
|
||||||
|
|
||||||
proxyHandler := v1.NewProxyHandler(s.engine, s.virtEngine, s.db, s.store, s.localHandler)
|
proxyHandler := v1.NewProxyHandler(s.engine, s.virtEngine, s.db, s.store, s.localHandler)
|
||||||
r.Mount("/api/v1", proxyHandler.Routes())
|
r.Mount("/api/v1", proxyHandler.Routes())
|
||||||
r.Mount("/v2", proxyHandler.DockerV2Routes())
|
r.Mount("/v2", proxyHandler.DockerV2Routes())
|
||||||
|
|
||||||
remotesHandler := v2.NewRemotesHandler(s.db)
|
remotesHandler := v2.NewRemotesHandler(s.db, map[models.PackageType]v2.Primer{
|
||||||
|
models.PackageGitHubRPM: s.syncer,
|
||||||
|
models.PackageGitHubDeb: s.debSyncer,
|
||||||
|
})
|
||||||
virtualsHandler := v2.NewVirtualsHandler(s.db)
|
virtualsHandler := v2.NewVirtualsHandler(s.db)
|
||||||
healthHandler := v2.NewHealthHandler(s.db, s.cache, s.store)
|
healthHandler := v2.NewHealthHandler(s.db, s.cache, s.store)
|
||||||
statsHandler := v2.NewStatsHandler(s.db)
|
statsHandler := v2.NewStatsHandler(s.db)
|
||||||
@@ -143,7 +213,13 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
|
|||||||
fmt.Fprint(w, `{"status":"ok"}`)
|
fmt.Fprint(w, `{"status":"ok"}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handleRoot sends browsers landing on the bare domain to the web UI, which is
|
||||||
|
// served under /ui. The service identity that used to live here is at /version.
|
||||||
func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) {
|
||||||
|
http.Redirect(w, r, "/ui/", http.StatusFound)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) handleVersion(w http.ResponseWriter, r *http.Request) {
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
w.WriteHeader(http.StatusOK)
|
w.WriteHeader(http.StatusOK)
|
||||||
fmt.Fprintf(w, `{"name":"artifactapi","version":"%s"}`, s.version)
|
fmt.Fprintf(w, `{"name":"artifactapi","version":"%s"}`, s.version)
|
||||||
@@ -161,6 +237,8 @@ func (s *Server) newHTTPServer() *http.Server {
|
|||||||
|
|
||||||
func (s *Server) Run(ctx context.Context) error {
|
func (s *Server) Run(ctx context.Context) error {
|
||||||
go s.gc.Run(ctx)
|
go s.gc.Run(ctx)
|
||||||
|
go s.syncer.Run(ctx)
|
||||||
|
go s.debSyncer.Run(ctx)
|
||||||
|
|
||||||
httpServer := s.newHTTPServer()
|
httpServer := s.newHTTPServer()
|
||||||
|
|
||||||
@@ -181,6 +259,8 @@ func (s *Server) Run(ctx context.Context) error {
|
|||||||
|
|
||||||
func (s *Server) RunOnListener(ctx context.Context, ln net.Listener) error {
|
func (s *Server) RunOnListener(ctx context.Context, ln net.Listener) error {
|
||||||
go s.gc.Run(ctx)
|
go s.gc.Run(ctx)
|
||||||
|
go s.syncer.Run(ctx)
|
||||||
|
go s.debSyncer.Run(ctx)
|
||||||
|
|
||||||
httpServer := s.newHTTPServer()
|
httpServer := s.newHTTPServer()
|
||||||
|
|
||||||
|
|||||||
@@ -129,13 +129,32 @@ func req(t *testing.T, method, path string, body string) (*http.Response, []byte
|
|||||||
return resp, b
|
return resp, b
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// reqNoRedirect issues a request without following redirects so the response's
|
||||||
|
// status and Location header can be asserted directly.
|
||||||
|
func reqNoRedirect(t *testing.T, method, path string) *http.Response {
|
||||||
|
t.Helper()
|
||||||
|
rq, _ := http.NewRequest(method, testTS.URL+path, nil)
|
||||||
|
client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||||
|
return http.ErrUseLastResponse
|
||||||
|
}}
|
||||||
|
resp, err := client.Do(rq)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%s %s: %v", method, path, err)
|
||||||
|
}
|
||||||
|
resp.Body.Close()
|
||||||
|
return resp
|
||||||
|
}
|
||||||
|
|
||||||
func TestServerHealthAndRoot(t *testing.T) {
|
func TestServerHealthAndRoot(t *testing.T) {
|
||||||
requireStack(t)
|
requireStack(t)
|
||||||
if resp, _ := req(t, "GET", "/health", ""); resp.StatusCode != 200 {
|
if resp, _ := req(t, "GET", "/health", ""); resp.StatusCode != 200 {
|
||||||
t.Errorf("health: %d", resp.StatusCode)
|
t.Errorf("health: %d", resp.StatusCode)
|
||||||
}
|
}
|
||||||
if resp, b := req(t, "GET", "/", ""); resp.StatusCode != 200 || !strings.Contains(string(b), "test-version") {
|
if resp := reqNoRedirect(t, "GET", "/"); resp.StatusCode != http.StatusFound || resp.Header.Get("Location") != "/ui/" {
|
||||||
t.Errorf("root: %d %s", resp.StatusCode, b)
|
t.Errorf("root redirect: %d %q", resp.StatusCode, resp.Header.Get("Location"))
|
||||||
|
}
|
||||||
|
if resp, b := req(t, "GET", "/version", ""); resp.StatusCode != 200 || !strings.Contains(string(b), "test-version") {
|
||||||
|
t.Errorf("version: %d %s", resp.StatusCode, b)
|
||||||
}
|
}
|
||||||
if resp, _ := req(t, "GET", "/api/v2/health", ""); resp.StatusCode != 200 {
|
if resp, _ := req(t, "GET", "/api/v2/health", ""); resp.StatusCode != 200 {
|
||||||
t.Errorf("health v2: %d", resp.StatusCode)
|
t.Errorf("health v2: %d", resp.StatusCode)
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/minio/minio-go/v7"
|
"github.com/minio/minio-go/v7"
|
||||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||||
@@ -97,3 +98,18 @@ func (s *S3) Stat(ctx context.Context, key string) (*minio.ObjectInfo, error) {
|
|||||||
}
|
}
|
||||||
return &info, nil
|
return &info, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ListStaleObjects returns keys under prefix last modified before cutoff. Used
|
||||||
|
// by the GC to reap abandoned staging objects (e.g. cancelled docker pushes).
|
||||||
|
func (s *S3) ListStaleObjects(ctx context.Context, prefix string, cutoff time.Time) ([]string, error) {
|
||||||
|
var keys []string
|
||||||
|
for obj := range s.client.ListObjects(ctx, s.bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: true}) {
|
||||||
|
if obj.Err != nil {
|
||||||
|
return nil, obj.Err
|
||||||
|
}
|
||||||
|
if obj.LastModified.Before(cutoff) {
|
||||||
|
keys = append(keys, obj.Key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return keys, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
package testsupport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"bytes"
|
||||||
|
"compress/gzip"
|
||||||
|
"fmt"
|
||||||
|
)
|
||||||
|
|
||||||
|
// MinimalDeb builds a valid-enough Debian package in pure Go (no committed
|
||||||
|
// binary fixture, no dpkg-deb): an ar archive of debian-binary, a gzip
|
||||||
|
// control.tar.gz carrying ./control, and an (empty) gzip data.tar.gz. It is the
|
||||||
|
// deb analog of MinimalRPM and is parseable by the deb provider.
|
||||||
|
func MinimalDeb(name, version, arch string) []byte {
|
||||||
|
control := fmt.Sprintf(
|
||||||
|
"Package: %s\nVersion: %s\nArchitecture: %s\nMaintainer: e2e <e2e@example.com>\n"+
|
||||||
|
"Section: utils\nPriority: optional\nDescription: minimal test package\n",
|
||||||
|
name, version, arch)
|
||||||
|
|
||||||
|
controlTarGz := gzipBytes(tarSingle("./control", []byte(control)))
|
||||||
|
dataTarGz := gzipBytes(tarEmpty())
|
||||||
|
|
||||||
|
var buf bytes.Buffer
|
||||||
|
buf.WriteString("!<arch>\n")
|
||||||
|
arWrite(&buf, "debian-binary", []byte("2.0\n"))
|
||||||
|
arWrite(&buf, "control.tar.gz", controlTarGz)
|
||||||
|
arWrite(&buf, "data.tar.gz", dataTarGz)
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func arWrite(buf *bytes.Buffer, name string, data []byte) {
|
||||||
|
fmt.Fprintf(buf, "%-16s%-12s%-6s%-6s%-8s%-10d`\n", name, "0", "0", "0", "100644", len(data))
|
||||||
|
buf.Write(data)
|
||||||
|
if len(data)%2 == 1 {
|
||||||
|
buf.WriteByte('\n')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func tarSingle(name string, data []byte) []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
tw := tar.NewWriter(&buf)
|
||||||
|
tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: int64(len(data)), Typeflag: tar.TypeReg})
|
||||||
|
tw.Write(data)
|
||||||
|
tw.Close()
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func tarEmpty() []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
tw := tar.NewWriter(&buf)
|
||||||
|
tw.Close()
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
|
|
||||||
|
func gzipBytes(data []byte) []byte {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
gz := gzip.NewWriter(&buf)
|
||||||
|
gz.Write(data)
|
||||||
|
gz.Close()
|
||||||
|
return buf.Bytes()
|
||||||
|
}
|
||||||
@@ -0,0 +1,173 @@
|
|||||||
|
// Package tfsign loads a GPG signing key and produces the detached signatures
|
||||||
|
// the Terraform provider registry protocol requires over SHA256SUMS files.
|
||||||
|
package tfsign
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"golang.org/x/crypto/openpgp"
|
||||||
|
"golang.org/x/crypto/openpgp/armor"
|
||||||
|
)
|
||||||
|
|
||||||
|
// KeyStore persists a generated signing key. *database.DB satisfies it.
|
||||||
|
type KeyStore interface {
|
||||||
|
GetSigningKey(ctx context.Context, purpose string) (armor, keyID string, found bool, err error)
|
||||||
|
InsertSigningKeyIfAbsent(ctx context.Context, purpose, armor, keyID string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// LoadOrCreate returns a signer for purpose, generating and persisting a new key
|
||||||
|
// the first time it is needed. It is safe across replicas: a lost insert race
|
||||||
|
// just re-reads whichever key won.
|
||||||
|
func LoadOrCreate(ctx context.Context, store KeyStore, purpose string) (*Signer, error) {
|
||||||
|
armored, _, found, err := store.GetSigningKey(ctx, purpose)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
newArmor, keyID, err := Generate()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := store.InsertSigningKeyIfAbsent(ctx, purpose, newArmor, keyID); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if armored, _, _, err = store.GetSigningKey(ctx, purpose); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return LoadArmored(armored, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Signer holds a decrypted GPG entity and exposes what the registry download
|
||||||
|
// response needs: a detached signature, the armored public key, and the key ID.
|
||||||
|
type Signer struct {
|
||||||
|
entity *openpgp.Entity
|
||||||
|
publicASCII string
|
||||||
|
keyID string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load reads an armored private key from path, decrypting it with passphrase if
|
||||||
|
// the key is protected. A blank path returns (nil, nil): a nil *Signer means the
|
||||||
|
// caller should fall back to another source (e.g. a DB-stored key).
|
||||||
|
func Load(path, passphrase string) (*Signer, error) {
|
||||||
|
if path == "" {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("open signing key: %w", err)
|
||||||
|
}
|
||||||
|
return fromArmor(string(data), passphrase, path)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LoadArmored builds a signer from an in-memory armored private key, e.g. one
|
||||||
|
// read from the database. A blank key returns (nil, nil).
|
||||||
|
func LoadArmored(armored, passphrase string) (*Signer, error) {
|
||||||
|
if armored == "" {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return fromArmor(armored, passphrase, "stored key")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate creates a fresh signing keypair and returns the armored private key
|
||||||
|
// (to persist) and its uppercase key id.
|
||||||
|
func Generate() (armoredPrivateKey, keyID string, err error) {
|
||||||
|
entity, err := openpgp.NewEntity("artifactapi terraform registry", "provider signing", "artifactapi@localhost", nil)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
var buf bytes.Buffer
|
||||||
|
w, err := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
if err := entity.SerializePrivate(w, nil); err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
if err := w.Close(); err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
return buf.String(), strings.ToUpper(entity.PrimaryKey.KeyIdString()), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func fromArmor(armored, passphrase, src string) (*Signer, error) {
|
||||||
|
keyring, err := openpgp.ReadArmoredKeyRing(strings.NewReader(armored))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read signing key: %w", err)
|
||||||
|
}
|
||||||
|
if len(keyring) == 0 {
|
||||||
|
return nil, fmt.Errorf("signing key (%s) contains no entities", src)
|
||||||
|
}
|
||||||
|
entity := keyring[0]
|
||||||
|
|
||||||
|
if entity.PrivateKey == nil {
|
||||||
|
return nil, fmt.Errorf("signing key (%s) has no private key material", src)
|
||||||
|
}
|
||||||
|
if entity.PrivateKey.Encrypted {
|
||||||
|
if err := decrypt(entity, passphrase); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub, err := armorPublicKey(entity)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Signer{
|
||||||
|
entity: entity,
|
||||||
|
publicASCII: pub,
|
||||||
|
keyID: entity.PrimaryKey.KeyIdString(),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// decrypt unlocks the entity's private key and all subkeys with the passphrase.
|
||||||
|
func decrypt(entity *openpgp.Entity, passphrase string) error {
|
||||||
|
pw := []byte(passphrase)
|
||||||
|
if err := entity.PrivateKey.Decrypt(pw); err != nil {
|
||||||
|
return fmt.Errorf("decrypt signing key: %w", err)
|
||||||
|
}
|
||||||
|
for _, sub := range entity.Subkeys {
|
||||||
|
if sub.PrivateKey != nil && sub.PrivateKey.Encrypted {
|
||||||
|
_ = sub.PrivateKey.Decrypt(pw)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func armorPublicKey(entity *openpgp.Entity) (string, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
w, err := armor.Encode(&buf, openpgp.PublicKeyType, nil)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if err := entity.Serialize(w); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if err := w.Close(); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return buf.String(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sign returns a binary detached signature over message, matching the
|
||||||
|
// SHA256SUMS.sig format Terraform verifies.
|
||||||
|
func (s *Signer) Sign(message []byte) ([]byte, error) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := openpgp.DetachSign(&buf, s.entity, bytes.NewReader(message), nil); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return buf.Bytes(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// PublicKeyArmor returns the ASCII-armored public key for the registry's
|
||||||
|
// signing_keys response.
|
||||||
|
func (s *Signer) PublicKeyArmor() string { return s.publicASCII }
|
||||||
|
|
||||||
|
// KeyID returns the 16-hex-char uppercase key ID Terraform matches against the
|
||||||
|
// signature's issuer.
|
||||||
|
func (s *Signer) KeyID() string { return strings.ToUpper(s.keyID) }
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
package tfsign
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"golang.org/x/crypto/openpgp"
|
||||||
|
"golang.org/x/crypto/openpgp/armor"
|
||||||
|
)
|
||||||
|
|
||||||
|
// armoredPrivateKey generates a throwaway armored private key for tests.
|
||||||
|
func armoredPrivateKey(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
e, err := openpgp.NewEntity("artifactapi test", "tf registry", "tf@example.com", nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var buf bytes.Buffer
|
||||||
|
w, err := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := e.SerializePrivate(w, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
w.Close()
|
||||||
|
return buf.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeKey(t *testing.T, contents string) string {
|
||||||
|
t.Helper()
|
||||||
|
p := filepath.Join(t.TempDir(), "private-key.asc")
|
||||||
|
if err := os.WriteFile(p, []byte(contents), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadSignAndVerify(t *testing.T) {
|
||||||
|
path := writeKey(t, armoredPrivateKey(t))
|
||||||
|
s, err := Load(path, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if s == nil {
|
||||||
|
t.Fatal("expected a signer")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !regexp.MustCompile(`^[0-9A-F]{16}$`).MatchString(s.KeyID()) {
|
||||||
|
t.Errorf("key id %q is not 16 uppercase hex chars", s.KeyID())
|
||||||
|
}
|
||||||
|
|
||||||
|
msg := []byte("deadbeef terraform-provider-x_1.0.0_linux_amd64.zip\n")
|
||||||
|
sig, err := s.Sign(msg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The advertised public key must verify the signature over the same bytes.
|
||||||
|
keyring, err := openpgp.ReadArmoredKeyRing(bytes.NewReader([]byte(s.PublicKeyArmor())))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := openpgp.CheckDetachedSignature(keyring, bytes.NewReader(msg), bytes.NewReader(sig)); err != nil {
|
||||||
|
t.Errorf("signature did not verify: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGenerateAndLoadArmored(t *testing.T) {
|
||||||
|
priv, keyID, err := Generate()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !regexp.MustCompile(`^[0-9A-F]{16}$`).MatchString(keyID) {
|
||||||
|
t.Errorf("generated key id %q malformed", keyID)
|
||||||
|
}
|
||||||
|
|
||||||
|
s, err := LoadArmored(priv, "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if s.KeyID() != keyID {
|
||||||
|
t.Errorf("loaded key id %q != generated %q", s.KeyID(), keyID)
|
||||||
|
}
|
||||||
|
|
||||||
|
msg := []byte("abc terraform-provider-x_1.0.0_linux_amd64.zip\n")
|
||||||
|
sig, err := s.Sign(msg)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
keyring, _ := openpgp.ReadArmoredKeyRing(bytes.NewReader([]byte(s.PublicKeyArmor())))
|
||||||
|
if _, err := openpgp.CheckDetachedSignature(keyring, bytes.NewReader(msg), bytes.NewReader(sig)); err != nil {
|
||||||
|
t.Errorf("signature did not verify: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// memStore is an in-memory KeyStore that records how many keys it accepted.
|
||||||
|
type memStore struct {
|
||||||
|
armor, keyID string
|
||||||
|
found bool
|
||||||
|
inserts int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) GetSigningKey(_ context.Context, _ string) (string, string, bool, error) {
|
||||||
|
return m.armor, m.keyID, m.found, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memStore) InsertSigningKeyIfAbsent(_ context.Context, _, armor, keyID string) error {
|
||||||
|
if !m.found { // ON CONFLICT DO NOTHING
|
||||||
|
m.armor, m.keyID, m.found = armor, keyID, true
|
||||||
|
m.inserts++
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadOrCreateGeneratesOnceThenReuses(t *testing.T) {
|
||||||
|
store := &memStore{}
|
||||||
|
|
||||||
|
first, err := LoadOrCreate(context.Background(), store, "terraform-provider")
|
||||||
|
if err != nil || first == nil {
|
||||||
|
t.Fatalf("first LoadOrCreate: signer=%v err=%v", first, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
second, err := LoadOrCreate(context.Background(), store, "terraform-provider")
|
||||||
|
if err != nil || second == nil {
|
||||||
|
t.Fatalf("second LoadOrCreate: signer=%v err=%v", second, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if store.inserts != 1 {
|
||||||
|
t.Errorf("expected exactly one key generated, got %d", store.inserts)
|
||||||
|
}
|
||||||
|
if first.KeyID() != second.KeyID() {
|
||||||
|
t.Errorf("key id changed between loads: %q vs %q", first.KeyID(), second.KeyID())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadEmptyPathDisabled(t *testing.T) {
|
||||||
|
s, err := Load("", "")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if s != nil {
|
||||||
|
t.Error("empty path should yield a nil (disabled) signer")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadMissingFile(t *testing.T) {
|
||||||
|
if _, err := Load(filepath.Join(t.TempDir(), "nope.asc"), ""); err == nil {
|
||||||
|
t.Error("expected an error for a missing key file")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,10 +11,13 @@ const (
|
|||||||
PackagePyPI PackageType = "pypi"
|
PackagePyPI PackageType = "pypi"
|
||||||
PackageNPM PackageType = "npm"
|
PackageNPM PackageType = "npm"
|
||||||
PackageRPM PackageType = "rpm"
|
PackageRPM PackageType = "rpm"
|
||||||
|
PackageDeb PackageType = "deb"
|
||||||
PackageAlpine PackageType = "alpine"
|
PackageAlpine PackageType = "alpine"
|
||||||
PackagePuppet PackageType = "puppet"
|
PackagePuppet PackageType = "puppet"
|
||||||
PackageTerraform PackageType = "terraform"
|
PackageTerraform PackageType = "terraform"
|
||||||
PackageGoProxy PackageType = "goproxy"
|
PackageGoProxy PackageType = "goproxy"
|
||||||
|
PackageGitHubRPM PackageType = "github_rpm"
|
||||||
|
PackageGitHubDeb PackageType = "github_deb"
|
||||||
)
|
)
|
||||||
|
|
||||||
var validPackageTypes = map[PackageType]bool{
|
var validPackageTypes = map[PackageType]bool{
|
||||||
@@ -24,10 +27,13 @@ var validPackageTypes = map[PackageType]bool{
|
|||||||
PackagePyPI: true,
|
PackagePyPI: true,
|
||||||
PackageNPM: true,
|
PackageNPM: true,
|
||||||
PackageRPM: true,
|
PackageRPM: true,
|
||||||
|
PackageDeb: true,
|
||||||
PackageAlpine: true,
|
PackageAlpine: true,
|
||||||
PackagePuppet: true,
|
PackagePuppet: true,
|
||||||
PackageTerraform: true,
|
PackageTerraform: true,
|
||||||
PackageGoProxy: true,
|
PackageGoProxy: true,
|
||||||
|
PackageGitHubRPM: true,
|
||||||
|
PackageGitHubDeb: true,
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p PackageType) Valid() bool {
|
func (p PackageType) Valid() bool {
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ func TestPackageTypeValid(t *testing.T) {
|
|||||||
models.PackagePuppet,
|
models.PackagePuppet,
|
||||||
models.PackageTerraform,
|
models.PackageTerraform,
|
||||||
models.PackageGoProxy,
|
models.PackageGoProxy,
|
||||||
|
models.PackageGitHubRPM,
|
||||||
}
|
}
|
||||||
for _, pt := range valid {
|
for _, pt := range valid {
|
||||||
if !pt.Valid() {
|
if !pt.Valid() {
|
||||||
|
|||||||
@@ -0,0 +1,99 @@
|
|||||||
|
.usage-panel {
|
||||||
|
margin: 24px 0;
|
||||||
|
background: var(--bg-surface);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: var(--radius);
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-toggle {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
width: 100%;
|
||||||
|
padding: 14px 18px;
|
||||||
|
background: transparent;
|
||||||
|
border: none;
|
||||||
|
color: var(--text-bright);
|
||||||
|
font-size: 0.95em;
|
||||||
|
font-weight: 600;
|
||||||
|
cursor: pointer;
|
||||||
|
text-align: left;
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-toggle:hover {
|
||||||
|
background: var(--bg-elevated);
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-caret {
|
||||||
|
display: inline-block;
|
||||||
|
transition: transform 0.15s;
|
||||||
|
color: var(--text-muted);
|
||||||
|
font-size: 0.9em;
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-caret.open {
|
||||||
|
transform: rotate(90deg);
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-body {
|
||||||
|
padding: 4px 18px 18px;
|
||||||
|
border-top: 1px solid var(--border);
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 18px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-snippet-title {
|
||||||
|
font-size: 0.85em;
|
||||||
|
font-weight: 600;
|
||||||
|
color: var(--text-muted);
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.03em;
|
||||||
|
margin: 14px 0 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-codebox {
|
||||||
|
position: relative;
|
||||||
|
background: var(--bg);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: var(--radius);
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-codebox pre {
|
||||||
|
margin: 0;
|
||||||
|
padding: 14px 16px;
|
||||||
|
overflow-x: auto;
|
||||||
|
font-family: var(--font-mono);
|
||||||
|
font-size: 0.85em;
|
||||||
|
line-height: 1.5;
|
||||||
|
color: var(--text-bright);
|
||||||
|
white-space: pre;
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-copy-btn {
|
||||||
|
position: absolute;
|
||||||
|
top: 8px;
|
||||||
|
right: 8px;
|
||||||
|
padding: 3px 10px;
|
||||||
|
font-size: 0.75em;
|
||||||
|
font-family: var(--font-sans);
|
||||||
|
color: var(--text-muted);
|
||||||
|
background: var(--bg-elevated);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
border-radius: 4px;
|
||||||
|
cursor: pointer;
|
||||||
|
transition: all 0.15s;
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-copy-btn:hover {
|
||||||
|
color: var(--text-bright);
|
||||||
|
border-color: var(--accent);
|
||||||
|
}
|
||||||
|
|
||||||
|
.usage-note {
|
||||||
|
margin-top: 8px;
|
||||||
|
font-size: 0.82em;
|
||||||
|
color: var(--text-muted);
|
||||||
|
line-height: 1.45;
|
||||||
|
}
|
||||||
@@ -0,0 +1,331 @@
|
|||||||
|
import { useState } from 'react';
|
||||||
|
import './UsageInstructions.css';
|
||||||
|
|
||||||
|
// repoClass distinguishes the three ways a repository is consumed. remotes are
|
||||||
|
// caching proxies, locals are real registries you also publish to, virtuals are
|
||||||
|
// merged read-only indexes.
|
||||||
|
type RepoClass = 'remote' | 'local' | 'virtual';
|
||||||
|
|
||||||
|
interface Snippet {
|
||||||
|
title: string;
|
||||||
|
language: string;
|
||||||
|
code: string;
|
||||||
|
note?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// baseURL resolves the externally reachable origin of this artifactapi instance.
|
||||||
|
// The UI is served on the same origin as the API (client BASE is ''), so
|
||||||
|
// window.location.origin is the address a host would actually curl/pull against
|
||||||
|
// — no hardcoded hostname, works in prod and in `npm run dev` behind a proxy.
|
||||||
|
function baseURL(): string {
|
||||||
|
if (typeof window !== 'undefined' && window.location?.origin) {
|
||||||
|
return window.location.origin.replace(/\/$/, '');
|
||||||
|
}
|
||||||
|
return 'https://artifactapi.k8s.syd1.au.unkin.net';
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostOnly is the bare host[:port] with no scheme, for docker/terraform source
|
||||||
|
// addresses which are scheme-less.
|
||||||
|
function hostOnly(): string {
|
||||||
|
try {
|
||||||
|
return new URL(baseURL()).host;
|
||||||
|
} catch {
|
||||||
|
return 'artifactapi.k8s.syd1.au.unkin.net';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// remoteProxyBase is where a remote (or virtual) repo's proxied artifacts live.
|
||||||
|
function remoteProxyBase(cls: RepoClass, name: string): string {
|
||||||
|
const seg = cls === 'virtual' ? 'virtual' : 'remote';
|
||||||
|
return `${baseURL()}/api/v1/${seg}/${name}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildSnippets(packageType: string, repoClass: RepoClass, name: string): Snippet[] {
|
||||||
|
const url = baseURL();
|
||||||
|
const host = hostOnly();
|
||||||
|
const proxy = remoteProxyBase(repoClass, name);
|
||||||
|
const isLocal = repoClass === 'local';
|
||||||
|
|
||||||
|
switch (packageType) {
|
||||||
|
case 'rpm':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: isLocal
|
||||||
|
? 'Add the yum repo (real yum repo, repodata auto-regenerated)'
|
||||||
|
: 'Add the yum repo (caching proxy)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `sudo tee /etc/yum.repos.d/${name}.repo >/dev/null <<'EOF'
|
||||||
|
[${name}]
|
||||||
|
name=${name} (artifactapi)
|
||||||
|
baseurl=${isLocal ? `${url}/api/v2/remotes/${name}/files/` : `${proxy}/`}
|
||||||
|
enabled=1
|
||||||
|
gpgcheck=0
|
||||||
|
repo_gpgcheck=0
|
||||||
|
EOF
|
||||||
|
|
||||||
|
sudo dnf install <package>`,
|
||||||
|
note: isLocal
|
||||||
|
? 'gpgcheck=0: artifactapi serves the repo unsigned. If you sign your RPMs, import your key and set gpgcheck=1.'
|
||||||
|
: 'gpgcheck=0 trusts upstream over the proxy. To verify package signatures, import the upstream GPG key and set gpgcheck=1.',
|
||||||
|
},
|
||||||
|
...(isLocal
|
||||||
|
? [
|
||||||
|
{
|
||||||
|
title: 'Publish an RPM (repodata regenerates automatically)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `curl -fsSL --upload-file ./my-package-1.0-1.el9.x86_64.rpm \\
|
||||||
|
${url}/api/v2/remotes/${name}/files/my-package-1.0-1.el9.x86_64.rpm`,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
: []),
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'pypi':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Install a package (one-off)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `pip install --index-url ${proxy}/simple/ <package>`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
title: 'Configure pip persistently',
|
||||||
|
language: 'bash',
|
||||||
|
code: `mkdir -p ~/.config/pip
|
||||||
|
cat > ~/.config/pip/pip.conf <<'EOF'
|
||||||
|
[global]
|
||||||
|
index-url = ${proxy}/simple/
|
||||||
|
EOF
|
||||||
|
|
||||||
|
pip install <package>`,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'npm':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Point npm at this registry',
|
||||||
|
language: 'bash',
|
||||||
|
code: `npm config set registry ${proxy}/
|
||||||
|
npm install <package>`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
title: 'Per-project (.npmrc)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `echo 'registry=${proxy}/' >> .npmrc
|
||||||
|
npm install`,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'docker':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Pull an image',
|
||||||
|
language: 'bash',
|
||||||
|
code: `docker pull ${host}/${name}/<image>:<tag>`,
|
||||||
|
note: 'The first path segment after the host is the artifactapi repo name; the rest is the image name.',
|
||||||
|
},
|
||||||
|
...(isLocal
|
||||||
|
? [
|
||||||
|
{
|
||||||
|
title: 'Push an image (this is a real Registry V2)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `docker tag myapp:latest ${host}/${name}/myapp:latest
|
||||||
|
docker push ${host}/${name}/myapp:latest`,
|
||||||
|
note: 'Works with docker, podman, skopeo and buildah. If the registry requires auth, run `docker login ' + host + '` first.',
|
||||||
|
},
|
||||||
|
]
|
||||||
|
: []),
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'terraform':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Use as a provider source (bare address, no mirror config)',
|
||||||
|
language: 'hcl',
|
||||||
|
code: `terraform {
|
||||||
|
required_providers {
|
||||||
|
${name} = {
|
||||||
|
source = "${host}/${name}/<type>"
|
||||||
|
version = ">= 0.1.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}`,
|
||||||
|
note: 'The namespace segment is this repo name; <type> is the provider type. artifactapi signs SHA256SUMS server-side with its GPG key, so `terraform init` installs with no .terraformrc.',
|
||||||
|
},
|
||||||
|
...(isLocal
|
||||||
|
? [
|
||||||
|
{
|
||||||
|
title: 'Publish a provider build',
|
||||||
|
language: 'bash',
|
||||||
|
code: `curl -fsSL --upload-file terraform-provider-<type>_0.1.0_linux_amd64.zip \\
|
||||||
|
${url}/api/v2/remotes/${name}/files/${name}/<type>/terraform-provider-<type>_0.1.0_linux_amd64.zip`,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
: []),
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'helm':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Add the Helm repo',
|
||||||
|
language: 'bash',
|
||||||
|
code: `helm repo add ${name} ${proxy}/
|
||||||
|
helm repo update
|
||||||
|
helm install <release> ${name}/<chart>`,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'alpine':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Add the APK repository',
|
||||||
|
language: 'bash',
|
||||||
|
code: `echo '${proxy}/' | sudo tee -a /etc/apk/repositories
|
||||||
|
sudo apk update
|
||||||
|
sudo apk add <package>`,
|
||||||
|
note: 'If the index is unsigned over the proxy, add --allow-untrusted or install the signing key into /etc/apk/keys.',
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'goproxy':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Point the Go module proxy here',
|
||||||
|
language: 'bash',
|
||||||
|
code: `export GOPROXY=${proxy}
|
||||||
|
go mod download`,
|
||||||
|
note: 'Append ,direct to fall back to VCS for modules this proxy does not cover.',
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'puppet':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Install a module from the Forge proxy',
|
||||||
|
language: 'bash',
|
||||||
|
code: `puppet module install <author>-<module> \\
|
||||||
|
--module_repository ${proxy}`,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'deb':
|
||||||
|
return isLocal
|
||||||
|
? [
|
||||||
|
{
|
||||||
|
title: 'Add the apt repo (real apt repo, flat — Packages/Release auto-generated)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `echo 'deb [trusted=yes] ${url}/api/v1/local/${name}/ ./' | sudo tee /etc/apt/sources.list.d/${name}.list
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install <package>`,
|
||||||
|
note: '[trusted=yes]: artifactapi serves the flat repo unsigned (matches the rpm repo\'s gpgcheck=0). The `./` is the flat-repo suite — apt fetches Packages/Release from the repo root.',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
title: 'Publish a .deb (index regenerates automatically)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `curl -fsSL --upload-file ./my-package_1.0_amd64.deb \\
|
||||||
|
${url}/api/v2/remotes/${name}/files/my-package_1.0_amd64.deb`,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
: [
|
||||||
|
{
|
||||||
|
title: 'Add the apt repo (caching proxy)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `echo 'deb ${proxy} <suite> <component>' | sudo tee /etc/apt/sources.list.d/${name}.list
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install <package>`,
|
||||||
|
note: "Signatures are verified against the upstream mirror's real signed Release through the proxy (no [trusted=yes] needed). Example suite/component: bookworm main.",
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'github_deb':
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Add the apt repo (metadata-only, from GitHub releases)',
|
||||||
|
language: 'bash',
|
||||||
|
code: `echo 'deb [trusted=yes] ${proxy}/ ./' | sudo tee /etc/apt/sources.list.d/${name}.list
|
||||||
|
sudo apt-get update
|
||||||
|
sudo apt-get install <package>`,
|
||||||
|
note: "The apt index is synthesized from the configured GitHub repo's release .deb assets; package downloads are redirected to the backing releases remote. Served unsigned, so [trusted=yes].",
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
case 'generic':
|
||||||
|
default:
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
title: 'Download a file',
|
||||||
|
language: 'bash',
|
||||||
|
code: `curl -fsSLO ${proxy}/<path>`,
|
||||||
|
note:
|
||||||
|
packageType === 'generic'
|
||||||
|
? 'Generic repos are fetched as plain files at their upstream path.'
|
||||||
|
: `No tailored client instructions for "${packageType}" yet — fetch artifacts directly by path.`,
|
||||||
|
},
|
||||||
|
...(isLocal
|
||||||
|
? [
|
||||||
|
{
|
||||||
|
title: 'Publish a file',
|
||||||
|
language: 'bash',
|
||||||
|
code: `curl -fsSL --upload-file ./myfile \\
|
||||||
|
${url}/api/v2/remotes/${name}/files/<path>/myfile`,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
: []),
|
||||||
|
];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function CodeBox({ snippet }: { snippet: Snippet }) {
|
||||||
|
const [copied, setCopied] = useState(false);
|
||||||
|
|
||||||
|
async function copy() {
|
||||||
|
try {
|
||||||
|
await navigator.clipboard.writeText(snippet.code);
|
||||||
|
setCopied(true);
|
||||||
|
setTimeout(() => setCopied(false), 1500);
|
||||||
|
} catch {
|
||||||
|
// Clipboard API unavailable (e.g. non-secure context); silently ignore.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="usage-snippet">
|
||||||
|
<div className="usage-snippet-title">{snippet.title}</div>
|
||||||
|
<div className="usage-codebox">
|
||||||
|
<button className="usage-copy-btn" onClick={copy} type="button">
|
||||||
|
{copied ? 'copied' : 'copy'}
|
||||||
|
</button>
|
||||||
|
<pre className="mono">{snippet.code}</pre>
|
||||||
|
</div>
|
||||||
|
{snippet.note && <div className="usage-note">{snippet.note}</div>}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
interface UsageInstructionsProps {
|
||||||
|
packageType: string;
|
||||||
|
repoClass: RepoClass;
|
||||||
|
name: string;
|
||||||
|
defaultOpen?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function UsageInstructions({ packageType, repoClass, name, defaultOpen = false }: UsageInstructionsProps) {
|
||||||
|
const [open, setOpen] = useState(defaultOpen);
|
||||||
|
const snippets = buildSnippets(packageType, repoClass, name);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="usage-panel">
|
||||||
|
<button className="usage-toggle" onClick={() => setOpen(o => !o)} type="button" aria-expanded={open}>
|
||||||
|
<span className={`usage-caret ${open ? 'open' : ''}`}>▸</span>
|
||||||
|
How do I use this?
|
||||||
|
</button>
|
||||||
|
{open && (
|
||||||
|
<div className="usage-body">
|
||||||
|
{snippets.map((s, i) => (
|
||||||
|
<CodeBox key={i} snippet={s} />
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
// Per-repo-type "downloadable" capability map.
|
||||||
|
//
|
||||||
|
// When a repo's package_type has an entry here, file entries in the object
|
||||||
|
// browser render as direct-download links pointing at the URL the entry
|
||||||
|
// builds. Types absent from the map render as plain text. Enabling a new
|
||||||
|
// type is a one-line addition below.
|
||||||
|
//
|
||||||
|
// Download routes are same-origin, unauthenticated GETs (the API serves local
|
||||||
|
// repos as real registries with no token on reads), so a bare <a href download>
|
||||||
|
// works and carries no credentials.
|
||||||
|
|
||||||
|
// buildUrl receives the repo name and the artifact's full path (may contain
|
||||||
|
// slashes) and returns the direct-download URL for that file.
|
||||||
|
type DownloadUrlBuilder = (repo: string, path: string) => string;
|
||||||
|
|
||||||
|
export const downloadableTypes: Record<string, DownloadUrlBuilder> = {
|
||||||
|
// rpm locals are real yum repos; files are served at
|
||||||
|
// /api/v2/remotes/<repo>/files/<path>.
|
||||||
|
rpm: (repo, path) =>
|
||||||
|
`/api/v2/remotes/${encodeURIComponent(repo)}/files/${path
|
||||||
|
.split('/')
|
||||||
|
.map(encodeURIComponent)
|
||||||
|
.join('/')}`,
|
||||||
|
};
|
||||||
|
|
||||||
|
// downloadUrlFor returns the direct-download URL for a file when its repo type
|
||||||
|
// is downloadable, or null otherwise (render as plain text).
|
||||||
|
export function downloadUrlFor(
|
||||||
|
packageType: string | undefined,
|
||||||
|
repo: string,
|
||||||
|
path: string,
|
||||||
|
): string | null {
|
||||||
|
if (!packageType) return null;
|
||||||
|
const build = downloadableTypes[packageType];
|
||||||
|
return build ? build(repo, path) : null;
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ import { useParams, Link } from 'react-router-dom';
|
|||||||
import { api } from '../api/client';
|
import { api } from '../api/client';
|
||||||
import type { Remote } from '../api/types';
|
import type { Remote } from '../api/types';
|
||||||
import { Badge } from '../components/Badge';
|
import { Badge } from '../components/Badge';
|
||||||
|
import { UsageInstructions } from '../components/UsageInstructions';
|
||||||
import './RemoteDetail.css';
|
import './RemoteDetail.css';
|
||||||
|
|
||||||
export function LocalDetail() {
|
export function LocalDetail() {
|
||||||
@@ -36,6 +37,8 @@ export function LocalDetail() {
|
|||||||
<p className="detail-description">{remote.description}</p>
|
<p className="detail-description">{remote.description}</p>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
<UsageInstructions packageType={remote.package_type} repoClass="local" name={remote.name} />
|
||||||
|
|
||||||
<div className="detail-actions">
|
<div className="detail-actions">
|
||||||
<Link to={`/locals/${remote.name}/objects`} className="btn btn-primary">
|
<Link to={`/locals/${remote.name}/objects`} className="btn btn-primary">
|
||||||
Browse Files
|
Browse Files
|
||||||
|
|||||||
@@ -37,6 +37,15 @@
|
|||||||
word-break: break-all;
|
word-break: break-all;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.tree-file-link {
|
||||||
|
color: var(--accent, #4c9aff);
|
||||||
|
text-decoration: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.tree-file-link:hover {
|
||||||
|
text-decoration: underline;
|
||||||
|
}
|
||||||
|
|
||||||
.tree-dir {
|
.tree-dir {
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ import { useParams, useLocation, Link } from 'react-router-dom';
|
|||||||
import { api } from '../api/client';
|
import { api } from '../api/client';
|
||||||
import type { Artifact } from '../api/types';
|
import type { Artifact } from '../api/types';
|
||||||
import { formatBytes, timeAgo, truncateHash } from '../components/format';
|
import { formatBytes, timeAgo, truncateHash } from '../components/format';
|
||||||
|
import { downloadUrlFor } from '../components/downloads';
|
||||||
import './Objects.css';
|
import './Objects.css';
|
||||||
|
|
||||||
interface TreeNode {
|
interface TreeNode {
|
||||||
@@ -100,11 +101,16 @@ interface TreeRowProps {
|
|||||||
expanded: Set<string>;
|
expanded: Set<string>;
|
||||||
onToggle: (path: string) => void;
|
onToggle: (path: string) => void;
|
||||||
onEvict: (path: string) => void;
|
onEvict: (path: string) => void;
|
||||||
|
repo: string;
|
||||||
|
packageType?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
|
function TreeRow({ node, depth, expanded, onToggle, onEvict, repo, packageType }: TreeRowProps) {
|
||||||
const isDir = node.children.size > 0 && !node.artifact;
|
const isDir = node.children.size > 0 && !node.artifact;
|
||||||
const isExpanded = expanded.has(node.path);
|
const isExpanded = expanded.has(node.path);
|
||||||
|
const downloadUrl = node.artifact
|
||||||
|
? downloadUrlFor(packageType, repo, node.artifact.path)
|
||||||
|
: null;
|
||||||
|
|
||||||
const sortedChildren = useMemo(() => {
|
const sortedChildren = useMemo(() => {
|
||||||
if (!isDir) return [];
|
if (!isDir) return [];
|
||||||
@@ -124,9 +130,20 @@ function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
|
|||||||
{isDir && (
|
{isDir && (
|
||||||
<span className="tree-toggle">{isExpanded ? '▾' : '▸'}</span>
|
<span className="tree-toggle">{isExpanded ? '▾' : '▸'}</span>
|
||||||
)}
|
)}
|
||||||
<span className={isDir ? 'tree-dir-name' : 'mono tree-file-name'}>
|
{downloadUrl ? (
|
||||||
{node.name}{isDir ? '/' : ''}
|
<a
|
||||||
</span>
|
className="mono tree-file-name tree-file-link"
|
||||||
|
href={downloadUrl}
|
||||||
|
download
|
||||||
|
onClick={(e) => e.stopPropagation()}
|
||||||
|
>
|
||||||
|
{node.name}
|
||||||
|
</a>
|
||||||
|
) : (
|
||||||
|
<span className={isDir ? 'tree-dir-name' : 'mono tree-file-name'}>
|
||||||
|
{node.name}{isDir ? '/' : ''}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
</span>
|
</span>
|
||||||
</td>
|
</td>
|
||||||
<td className="num-cell">{formatBytes(node.totalSize)}</td>
|
<td className="num-cell">{formatBytes(node.totalSize)}</td>
|
||||||
@@ -163,6 +180,8 @@ function TreeRow({ node, depth, expanded, onToggle, onEvict }: TreeRowProps) {
|
|||||||
expanded={expanded}
|
expanded={expanded}
|
||||||
onToggle={onToggle}
|
onToggle={onToggle}
|
||||||
onEvict={onEvict}
|
onEvict={onEvict}
|
||||||
|
repo={repo}
|
||||||
|
packageType={packageType}
|
||||||
/>
|
/>
|
||||||
))}
|
))}
|
||||||
</>
|
</>
|
||||||
@@ -175,6 +194,7 @@ export function Objects() {
|
|||||||
const isLocal = location.pathname.startsWith('/locals/');
|
const isLocal = location.pathname.startsWith('/locals/');
|
||||||
const backLink = isLocal ? `/locals/${name}` : `/remotes/${name}`;
|
const backLink = isLocal ? `/locals/${name}` : `/remotes/${name}`;
|
||||||
const [artifacts, setArtifacts] = useState<Artifact[]>([]);
|
const [artifacts, setArtifacts] = useState<Artifact[]>([]);
|
||||||
|
const [packageType, setPackageType] = useState<string | undefined>(undefined);
|
||||||
const [loading, setLoading] = useState(true);
|
const [loading, setLoading] = useState(true);
|
||||||
const [filter, setFilter] = useState('');
|
const [filter, setFilter] = useState('');
|
||||||
const [expanded, setExpanded] = useState<Set<string>>(new Set());
|
const [expanded, setExpanded] = useState<Set<string>>(new Set());
|
||||||
@@ -190,6 +210,15 @@ export function Objects() {
|
|||||||
|
|
||||||
useEffect(() => { load(); }, [load]);
|
useEffect(() => { load(); }, [load]);
|
||||||
|
|
||||||
|
// The repo's package_type is the modularity hook: it decides whether file
|
||||||
|
// names render as direct-download links (see downloadableTypes).
|
||||||
|
useEffect(() => {
|
||||||
|
if (!name) return;
|
||||||
|
api.getRemote(name)
|
||||||
|
.then(r => setPackageType(r.package_type))
|
||||||
|
.catch(() => setPackageType(undefined));
|
||||||
|
}, [name]);
|
||||||
|
|
||||||
const handleEvict = async (path: string) => {
|
const handleEvict = async (path: string) => {
|
||||||
if (!name || !confirm(`Evict ${path}?`)) return;
|
if (!name || !confirm(`Evict ${path}?`)) return;
|
||||||
await (isLocal ? api.evictLocalObject(name, path) : api.evictObject(name, path));
|
await (isLocal ? api.evictLocalObject(name, path) : api.evictObject(name, path));
|
||||||
@@ -287,6 +316,8 @@ export function Objects() {
|
|||||||
expanded={expanded}
|
expanded={expanded}
|
||||||
onToggle={toggleExpand}
|
onToggle={toggleExpand}
|
||||||
onEvict={handleEvict}
|
onEvict={handleEvict}
|
||||||
|
repo={name!}
|
||||||
|
packageType={packageType}
|
||||||
/>
|
/>
|
||||||
))
|
))
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ import { useParams, Link } from 'react-router-dom';
|
|||||||
import { api } from '../api/client';
|
import { api } from '../api/client';
|
||||||
import type { Remote } from '../api/types';
|
import type { Remote } from '../api/types';
|
||||||
import { Badge } from '../components/Badge';
|
import { Badge } from '../components/Badge';
|
||||||
|
import { UsageInstructions } from '../components/UsageInstructions';
|
||||||
import './RemoteDetail.css';
|
import './RemoteDetail.css';
|
||||||
|
|
||||||
export function RemoteDetail() {
|
export function RemoteDetail() {
|
||||||
@@ -109,6 +110,8 @@ export function RemoteDetail() {
|
|||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<UsageInstructions packageType={remote.package_type} repoClass="remote" name={remote.name} />
|
||||||
|
|
||||||
<div className="detail-actions">
|
<div className="detail-actions">
|
||||||
<Link to={`/remotes/${remote.name}/objects`} className="btn btn-primary">
|
<Link to={`/remotes/${remote.name}/objects`} className="btn btn-primary">
|
||||||
Browse Objects
|
Browse Objects
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import { api } from '../api/client';
|
|||||||
import type { Remote, Virtual } from '../api/types';
|
import type { Remote, Virtual } from '../api/types';
|
||||||
import { Badge } from '../components/Badge';
|
import { Badge } from '../components/Badge';
|
||||||
import { DataTable } from '../components/DataTable';
|
import { DataTable } from '../components/DataTable';
|
||||||
|
import { UsageInstructions } from '../components/UsageInstructions';
|
||||||
import './Virtuals.css';
|
import './Virtuals.css';
|
||||||
|
|
||||||
export function Virtuals() {
|
export function Virtuals() {
|
||||||
@@ -98,6 +99,12 @@ export function Virtuals() {
|
|||||||
);
|
);
|
||||||
})}
|
})}
|
||||||
</ul>
|
</ul>
|
||||||
|
{(() => {
|
||||||
|
const v = virtuals.find(x => x.name === expanded);
|
||||||
|
return v ? (
|
||||||
|
<UsageInstructions packageType={v.package_type} repoClass="virtual" name={v.name} defaultOpen />
|
||||||
|
) : null;
|
||||||
|
})()}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
Reference in New Issue
Block a user