Compare commits

..

5 Commits

Author SHA1 Message Date
unkin-agent 73c0bfc670 deb/apk: make local repodata deterministic (#119)
ci/woodpecker/tag/docker Pipeline was successful
Part of #117. Local generated repodata must be byte-identical across the two no-affinity replicas and across every regeneration, so apt/apk never hit a checksum mismatch between an index's advertised hash and the bytes actually served. This does the deb+apk half (the rpm half landed in #118).

How:
- Derive the deb `Release` `Date:` from the newest persisted `created_at` (RFC1123Z, UTC) instead of `time.Now()`; carry `created_at` through the deb metadata SELECT and `DebMetadata`. An empty repo falls back to the Unix epoch. This also stops `Date:` running ahead of wall clock.
- Pin the apk `APKINDEX` tar header `ModTime` to the Unix epoch instead of the zero-value `time.Time`, so it is never wall-clock derived.
- Give both list queries a genuine total order by adding a `file_path` tiebreak (name/version/arch is not unique).
- Add guard tests: deb generators byte-identical across generations; the `Release` checksum/size matches the served `Packages`/`Packages.gz` bytes (the exact apt invariant); `Date:` pinned to `created_at`; apk index byte-identical and tar `ModTime` pinned to epoch.

Reviewed-on: #119
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 23:32:07 +10:00
unkin-agent a8aa0c231b rpm: make local repodata deterministic (fixes #117) (#118)
Local RPM repos regenerated `repomd.xml` on every request and advertised a `primary.xml.gz` sha256 that drifted every second, because `time.Now().Unix()` was embedded inside the gzipped `primary.xml` (and in `repomd` `<revision>`/`<timestamp>`). The advertised hash therefore never matched the content-addressed `<sha256>-primary.xml.gz` bytes a second later or on the other replica, so `dnf` failed with a checksum mismatch. Part of #117.

How:
- Derives `<time file=>` in `primary.xml` from the persisted `rpm_metadata.created_at` instead of the wall clock; unset timestamps collapse to a fixed `0`.
- Derives `repomd` `<revision>`/`<timestamp>` from the newest package upload time, so `repomd.xml` is byte-identical across replicas and requests.
- Adds `file_path` as a total-order tiebreak to the metadata `ORDER BY`.
- Pins the gzip header (`OS: 255`) so compressed bytes depend only on the payload.
- Adds regression tests: generators are byte-identical across two runs, and the sha256 in `repomd.xml` equals the sha256 of the bytes each `serve*` handler returns.

Reviewed-on: #118
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 23:14:51 +10:00
unkin-agent 0e26d99228 ui: add alpine local and github_alpine usage instructions (#116)
ci/woodpecker/tag/docker Pipeline was successful
## Why
The alpine local repo (a real apk repo with auto-generated per-arch APKINDEX) and the new github_alpine remote (metadata-only apk index synthesized from a GitHub repo's release .apk assets) shipped without any "How do I use this?" UI guidance, unlike deb/github_deb. This adds the matching client instructions so users can consume and publish to these repos.

## How
- Extend `case 'alpine':` in `UsageInstructions.tsx` to branch on `isLocal` (mirroring rpm/deb):
  - LOCAL: "Add the apk repo" snippet appends `${url}/api/v1/local/<name>` to `/etc/apk/repositories` with `apk update/add --allow-untrusted` (served unsigned, parity with rpm gpgcheck=0), plus a "Publish a .apk" snippet uploading to the canonical `<arch>/<name>-<version>.apk` path (APKINDEX carries no filename, so path must match or install 404s).
  - REMOTE: existing caching-proxy snippet kept unchanged.
- Add `case 'github_alpine':` (always remote-class): "Add the apk repo (metadata-only, from GitHub releases)" snippet with `--allow-untrusted`; note explains the per-arch APKINDEX is synthesized from release .apk assets and downloads redirect to the backing releases remote.
- No other cases touched. `npm run build` (tsc typecheck + vite) passes; pre-commit passes.

Reviewed-on: #116
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 20:53:23 +10:00
unkin-agent 5a06c16797 Add github_alpine metadata-only package type (#115)
## Why

`github_alpine` is the Alpine/apk analog of the existing `github_deb`/`github_rpm` metadata-only remotes. It lets a plain GitHub-releases repo of `.apk` files be consumed as a real apk repository without artifactapi ever precaching the packages: it scans the repo's releases, derives each package's `.PKGINFO` from a ranged prefix fetch, synthesizes a per-arch `APKINDEX.tar.gz` from the cached metadata, and redirects the actual `.apk` downloads to a backend `releases_remote`. It stacks on the apk-local branch, reusing that work's alpine APKINDEX generator, `.apk`/`.PKGINFO` parser, Q1 pull-checksum, and `AlpineMetadata` store.

## How

- **`pkg/models`**: add `PackageGitHubAlpine` to the enum + validators (and test).
- **`internal/provider/alpine/github.go`**: the `github_alpine` provider. `ServeRemote` serves per-arch `<arch>/APKINDEX.tar.gz` (reusing `generateAPKIndex` over arch-filtered `AlpineMetadata` rows, `normalizeIndexPath` for apk's `./` dot-segment), 302-redirects `*.apk` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`, and cold-starts with a 503 + `Retry-After`. `scanWithState` lists releases with ETag/If-None-Match and incrementally derives/prunes. `deriveAsset` does a **ranged GET of just the front of the `.apk`** — the control gzip stream carrying `.PKGINFO` sits near the front — doubling the range on truncation; it parses `.PKGINFO` and computes the `C:` Q1 checksum (`Q1`+base64(sha1(control stream))). `FilePath` = the github-relative asset path so the redirect resolves.
- **`internal/provider/alpine/syncer.go`**: a parallel background `Syncer` (own worker pool, shared rate limiter, deduped queue, DB lease), separate from the deb/rpm syncers.
- **`internal/database/alpine_github_sync.go`** + `github_alpine_sync_state` table: `ListGitHubAlpineRemotes` + Claim/Release per-remote sync lease, kept separate from the deb/rpm tables.
- **`internal/server/server.go`**: construct + `Run` the alpine syncer alongside deb/rpm and register it in the `PackageType→Primer` map (priming on create then flows through the existing generic `remotes.go` path).

The rpm/deb providers, syncers, and tables are untouched — this adds parallel alpine equivalents and reuses shared helpers already present in the alpine package.

## Tests

Mirror the deb github tests: scan derives `.PKGINFO`/Q1 from a ranged prefix of a `testsupport.MinimalApk` served over an httptest range server (no full download); diff/prune; pattern filter; per-arch `ServeRemote` routing (index served per-arch and grouped, dot-segment collapse, `.apk` → 302, cold-start 503, warm 200, canceled-request-serves-cache); DB lease prevents a second replica; prime bypasses the recency window. `go build`/`go vet`/`go mod tidy` clean, `make test` (-race) green, pre-commit green.

Reviewed-on: #115
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 20:48:42 +10:00
unkin-agent 7f77666709 Add Alpine/apk local repository support (#114)
## Why

The alpine provider only supported remote (proxy) repositories, so there was no way to publish first-party `.apk` packages the way `rpm-local` and `deb-local` already allow. This extends the existing alpine provider into a real apk repository: uploaded `.apk` files are parsed in pure Go and a per-arch `APKINDEX.tar.gz` is generated on demand, at parity with rpm repodata and deb Packages generation. (The metadata-only `github_alpine` type is a separate follow-up and is not part of this PR.)

## How

- Implements `LocalUploader` / `LocalIndexer` / `PostUploadHook` / `PostDeleteHook` on the existing `alpine` provider, leaving the remote proxy methods (`UpstreamURL`/`ContentType`/`AuthHeaders`/`RewriteResponse`/`Classify`) intact.
- Parses the `.apk` (up to three concatenated, independently gzipped tar streams) in pure Go: locates the control stream by its `.PKGINFO` member, reads the `key = value` fields, and computes the apk pull checksum `C:` = `Q1` + base64(sha1(**control gzip stream bytes**)) — the sha1 of the second gzip member, not of the whole file.
- Derives arch from `.PKGINFO` and records download size (`S:` blob size) and installed size (`I:` from `.PKGINFO size`).
- Generates an **unsigned** per-arch `APKINDEX.tar.gz` = gzip(tar(`APKINDEX`)) filtered by requested arch (clients use `--allow-untrusted`, matching rpm `gpgcheck=0` / deb `[trusted=yes]`), applying the same dot-segment normalization as deb so `./<arch>/APKINDEX.tar.gz` resolves. Non-index / `.apk` paths return `false` so the generic file streamer serves the stored blob.
- Adds `AlpineMetadata` plus **separate** `AlpineMetadataStore` / `AlpineMetadataReader` / `AlpineMetadataDeleter` interfaces (type-asserted from the generic hooks) so the shared rpm/deb metadata interfaces and their test doubles are untouched.
- Adds the `alpine_metadata` table (keyed by `repo_name` + `file_path`, per-arch index) and its `Insert`/`Delete`/`List` DB methods.
- Adds `testsupport.MinimalApk`, unit tests (`.PKGINFO` parse, Q1 checksum over the control stream, per-arch filtering, empty-field omission, `./` dot-segment handling, ValidateUpload accept/reject), and a `dockere2e` `TestLocalAlpineIndex`.

## Consumption

`/etc/apk/repositories` line = `<url>/api/v1/local/<name>` (apk appends `/<arch>/APKINDEX.tar.gz`); `apk update --allow-untrusted && apk add --allow-untrusted <pkg>`. Packages live at `/api/v1/local/<name>/<arch>/<file>.apk`.

## Verification

`go build ./...`, `go vet ./...` (incl. `-tags dockere2e`), `go mod tidy` (no change), `make test` (`-race`), and `pre-commit run --all-files` all pass.

Reviewed-on: #114
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-08-12 20:39:27 +10:00
11 changed files with 499 additions and 18 deletions
+1 -1
View File
@@ -41,7 +41,7 @@ func (db *DB) ListAlpineMetadataEntries(ctx context.Context, repoName string) ([
depends, provides, install_if
FROM alpine_metadata
WHERE repo_name = $1
ORDER BY name, version, arch
ORDER BY name, version, arch, file_path
`, repoName)
if err != nil {
return nil, err
+3 -3
View File
@@ -31,10 +31,10 @@ func (db *DB) ListDebMetadataEntries(ctx context.Context, repoName string) ([]pr
rows, err := db.Pool.Query(ctx, `
SELECT repo_name, file_path, content_hash,
name, version, architecture, control,
size, md5, sha256
size, md5, sha256, created_at
FROM deb_metadata
WHERE repo_name = $1
ORDER BY name, version, architecture
ORDER BY name, version, architecture, file_path
`, repoName)
if err != nil {
return nil, err
@@ -47,7 +47,7 @@ func (db *DB) ListDebMetadataEntries(ctx context.Context, repoName string) ([]pr
if err := rows.Scan(
&m.RepoName, &m.FilePath, &m.ContentHash,
&m.Name, &m.Version, &m.Architecture, &m.Control,
&m.Size, &m.MD5, &m.SHA256,
&m.Size, &m.MD5, &m.SHA256, &m.CreatedAt,
); err != nil {
return nil, err
}
+7 -2
View File
@@ -3,6 +3,7 @@ package database
import (
"context"
"encoding/json"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
@@ -65,6 +66,7 @@ type RPMMetadataRow struct {
Obsoletes json.RawMessage
Files json.RawMessage
Changelogs json.RawMessage
CreatedAt time.Time
}
func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]provider.RPMMetadata, error) {
@@ -94,6 +96,7 @@ func (db *DB) ListRPMMetadataEntries(ctx context.Context, repoName string) ([]pr
SourceRPM: r.SourceRPM,
URL: r.URL,
Packager: r.Packager,
CreatedAt: r.CreatedAt,
}
json.Unmarshal(r.Requires, &meta.Requires)
json.Unmarshal(r.Provides, &meta.Provides)
@@ -112,10 +115,11 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
name, epoch, version, release, arch,
summary, description, rpm_size, installed_size,
license, vendor, build_group, build_host, source_rpm, url, packager,
requires, provides, conflicts, obsoletes, files, changelogs
requires, provides, conflicts, obsoletes, files, changelogs,
created_at
FROM rpm_metadata
WHERE repo_name = $1
ORDER BY name, epoch, version, release, arch
ORDER BY name, epoch, version, release, arch, file_path
`, repoName)
if err != nil {
return nil, err
@@ -131,6 +135,7 @@ func (db *DB) ListRPMMetadata(ctx context.Context, repoName string) ([]RPMMetada
&r.Summary, &r.Description, &r.RPMSize, &r.InstalledSize,
&r.License, &r.Vendor, &r.Group, &r.BuildHost, &r.SourceRPM, &r.URL, &r.Packager,
&r.Requires, &r.Provides, &r.Conflicts, &r.Obsoletes, &r.Files, &r.Changelogs,
&r.CreatedAt,
); err != nil {
return nil, err
}
+5 -1
View File
@@ -15,6 +15,7 @@ import (
"path"
"strconv"
"strings"
"time"
"archive/tar"
@@ -376,7 +377,10 @@ func generateAPKIndex(metas []provider.AlpineMetadata) []byte {
var tarBuf bytes.Buffer
tw := tar.NewWriter(&tarBuf)
body := idx.Bytes()
tw.WriteHeader(&tar.Header{Name: "APKINDEX", Mode: 0o644, Size: int64(len(body)), Typeflag: tar.TypeReg})
// ModTime is pinned to the Unix epoch (never wall clock) so APKINDEX.tar.gz
// is byte-identical across replicas and regenerations (issue #117); apk
// clients ignore the tar mtime.
tw.WriteHeader(&tar.Header{Name: "APKINDEX", Mode: 0o644, Size: int64(len(body)), Typeflag: tar.TypeReg, ModTime: time.Unix(0, 0)})
tw.Write(body)
tw.Close()
@@ -0,0 +1,72 @@
package alpine
import (
"archive/tar"
"bytes"
"compress/gzip"
"io"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func apkFixture() []provider.AlpineMetadata {
return []provider.AlpineMetadata{
{
RepoName: "r", FilePath: "x86_64/aaa-1.0-r0.apk", Checksum: "Q1aaa",
Name: "aaa", Version: "1.0-r0", Arch: "x86_64", DownloadSize: 100, InstalledSize: 10,
Description: "pkg aaa", URL: "https://a", License: "MIT",
Depends: []string{"so:libc"}, Provides: []string{"cmd:aaa"}, BuildTime: 1710000000,
},
{
RepoName: "r", FilePath: "x86_64/bbb-2.0-r0.apk", Checksum: "Q1bbb",
Name: "bbb", Version: "2.0-r0", Arch: "x86_64", DownloadSize: 200, InstalledSize: 20,
},
}
}
// TestAPKIndexDeterministic asserts APKINDEX.tar.gz is byte-identical across two
// generations separated by wall-clock time, so the two no-affinity replicas and
// every regeneration serve the same bytes (issue #117).
func TestAPKIndexDeterministic(t *testing.T) {
metas := apkFixture()
first := generateAPKIndex(metas)
time.Sleep(10 * time.Millisecond)
second := generateAPKIndex(metas)
if !bytes.Equal(first, second) {
t.Error("APKINDEX.tar.gz differs across generations")
}
}
// TestAPKIndexTarModTimePinned guards the tar header: its ModTime must be the
// pinned Unix epoch, never wall clock. Fails if a future edit stamps time.Now().
func TestAPKIndexTarModTimePinned(t *testing.T) {
metas := apkFixture()
zr, err := gzip.NewReader(bytes.NewReader(generateAPKIndex(metas)))
if err != nil {
t.Fatalf("gzip: %v", err)
}
if !zr.ModTime.IsZero() && zr.ModTime.Unix() != 0 {
t.Errorf("gzip header ModTime = %v, want zero/epoch", zr.ModTime)
}
tarBytes, err := io.ReadAll(zr)
if err != nil {
t.Fatalf("gunzip: %v", err)
}
tr := tar.NewReader(bytes.NewReader(tarBytes))
hdr, err := tr.Next()
if err != nil {
t.Fatalf("tar: %v", err)
}
if hdr.Name != "APKINDEX" {
t.Fatalf("tar entry = %q, want APKINDEX", hdr.Name)
}
if hdr.ModTime.Unix() != 0 {
t.Errorf("APKINDEX tar ModTime = %v (unix %d), want epoch (0)", hdr.ModTime, hdr.ModTime.Unix())
}
}
+16 -1
View File
@@ -395,7 +395,7 @@ func generateRelease(metas []provider.DebMetadata) []byte {
arches := uniqueArches(metas)
var b bytes.Buffer
fmt.Fprintf(&b, "Date: %s\n", time.Now().UTC().Format(time.RFC1123Z))
fmt.Fprintf(&b, "Date: %s\n", releaseDate(metas).Format(time.RFC1123Z))
fmt.Fprintf(&b, "Architectures: %s\n", strings.Join(arches, " "))
b.WriteString("Acquire-By-Hash: no\n")
@@ -410,6 +410,21 @@ func generateRelease(metas []provider.DebMetadata) []byte {
return b.Bytes()
}
// releaseDate derives the Release Date: from the newest package's persisted
// created_at (in UTC) so the file is byte-identical across the no-affinity
// replicas and across regenerations (issue #117); an empty repo falls back to
// the Unix epoch. This never uses wall clock, which also keeps Date: from
// running ahead of any Valid-Until logic.
func releaseDate(metas []provider.DebMetadata) time.Time {
newest := time.Unix(0, 0)
for _, m := range metas {
if m.CreatedAt.After(newest) {
newest = m.CreatedAt
}
}
return newest.UTC()
}
func writeReleaseEntry(b *bytes.Buffer, hash string, size int, name string) {
fmt.Fprintf(b, " %s %d %s\n", hash, size, name)
}
@@ -0,0 +1,172 @@
package deb
import (
"bytes"
"strconv"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
// debFixture returns a fixed set of rows with persisted created_at values, in
// the total order ListDebMetadataEntries produces (name, version, arch,
// file_path), so the generators are exercised on a stable input.
func debFixture() []provider.DebMetadata {
t1 := time.Date(2026, 3, 1, 8, 30, 0, 0, time.UTC)
t2 := time.Date(2026, 4, 15, 12, 0, 0, 0, time.UTC) // newest
return []provider.DebMetadata{
{
RepoName: "r", FilePath: "pool/aaa_1.0_amd64.deb", ContentHash: "sha256:aa",
Name: "aaa", Version: "1.0", Architecture: "amd64",
Control: "Package: aaa\nVersion: 1.0\nArchitecture: amd64",
Size: 100, MD5: "d41d8cd98f00b204e9800998ecf8427e", SHA256: "aa", CreatedAt: t1,
},
{
RepoName: "r", FilePath: "pool/bbb_2.0_arm64.deb", ContentHash: "sha256:bb",
Name: "bbb", Version: "2.0", Architecture: "arm64",
Control: "Package: bbb\nVersion: 2.0\nArchitecture: arm64",
Size: 200, MD5: "0cc175b9c0f1b6a831c399e269772661", SHA256: "bb", CreatedAt: t2,
},
}
}
// TestDebGeneratorsDeterministic asserts the served bytes are a pure function of
// DB state: Packages, Packages.gz and Release are byte-identical across two
// generations separated by wall-clock time. Fails against the old
// time.Now()-stamped Release Date:.
func TestDebGeneratorsDeterministic(t *testing.T) {
metas := debFixture()
pkgs1 := generatePackages(metas)
rel1 := generateRelease(metas)
gz1 := gzipBytes(pkgs1)
time.Sleep(10 * time.Millisecond)
pkgs2 := generatePackages(metas)
rel2 := generateRelease(metas)
gz2 := gzipBytes(pkgs2)
if !bytes.Equal(pkgs1, pkgs2) {
t.Error("Packages differs across generations")
}
if !bytes.Equal(gz1, gz2) {
t.Error("Packages.gz differs across generations")
}
if !bytes.Equal(rel1, rel2) {
t.Errorf("Release differs across generations:\n--- first ---\n%s\n--- second ---\n%s", rel1, rel2)
}
}
// TestDebReleaseDateUsesPersistedCreatedAt pins the Release Date: to the newest
// persisted created_at (RFC1123Z, UTC), not wall clock. Fails against the old
// time.Now() code.
func TestDebReleaseDateUsesPersistedCreatedAt(t *testing.T) {
metas := debFixture()
want := time.Date(2026, 4, 15, 12, 0, 0, 0, time.UTC).Format(time.RFC1123Z)
rel := string(generateRelease(metas))
var got string
for _, line := range strings.Split(rel, "\n") {
if strings.HasPrefix(line, "Date:") {
got = strings.TrimSpace(strings.TrimPrefix(line, "Date:"))
break
}
}
if got != want {
t.Errorf("Release Date: = %q, want %q (newest created_at)", got, want)
}
}
// TestDebReleaseDateEmptyRepoIsEpoch guards the fallback: an empty repo yields a
// deterministic epoch Date: rather than wall clock.
func TestDebReleaseDateEmptyRepoIsEpoch(t *testing.T) {
want := time.Unix(0, 0).UTC().Format(time.RFC1123Z)
rel := string(generateRelease(nil))
if !strings.Contains(rel, "Date: "+want+"\n") {
t.Errorf("empty-repo Release missing epoch Date: %q\n%s", want, rel)
}
}
// TestDebReleaseChecksumsMatchServedBytes is the exact apt invariant: the
// sha256/size (and md5/size) advertised for Packages and Packages.gz in Release
// equal the sha256/size of the actual bytes ServeLocalIndex serves. apt rejects
// any mismatch.
func TestDebReleaseChecksumsMatchServedBytes(t *testing.T) {
metas := debFixture()
packages := generatePackages(metas)
packagesGz := gzipBytes(packages)
rel := string(generateRelease(metas))
wantSHA := map[string]struct {
hash string
size int
}{
"Packages": {sha256Hex(packages), len(packages)},
"Packages.gz": {sha256Hex(packagesGz), len(packagesGz)},
}
wantMD5 := map[string]struct {
hash string
size int
}{
"Packages": {md5Hex(packages), len(packages)},
"Packages.gz": {md5Hex(packagesGz), len(packagesGz)},
}
sha := parseReleaseSection(rel, "SHA256:")
md5s := parseReleaseSection(rel, "MD5Sum:")
for name, w := range wantSHA {
got, ok := sha[name]
if !ok {
t.Fatalf("Release SHA256 section missing %q", name)
}
if got.hash != w.hash || got.size != w.size {
t.Errorf("Release SHA256 %s = (%s, %d), served bytes are (%s, %d)", name, got.hash, got.size, w.hash, w.size)
}
}
for name, w := range wantMD5 {
got, ok := md5s[name]
if !ok {
t.Fatalf("Release MD5Sum section missing %q", name)
}
if got.hash != w.hash || got.size != w.size {
t.Errorf("Release MD5Sum %s = (%s, %d), served bytes are (%s, %d)", name, got.hash, got.size, w.hash, w.size)
}
}
}
type releaseEntry struct {
hash string
size int
}
// parseReleaseSection reads the indented " <hash> <size> <name>" lines that
// follow a "SHA256:" / "MD5Sum:" header until the next non-indented line.
func parseReleaseSection(release, header string) map[string]releaseEntry {
out := map[string]releaseEntry{}
lines := strings.Split(release, "\n")
in := false
for _, line := range lines {
if line == header {
in = true
continue
}
if !in {
continue
}
if !strings.HasPrefix(line, " ") {
break
}
fields := strings.Fields(line)
if len(fields) != 3 {
continue
}
size, _ := strconv.Atoi(fields[1])
out[fields[2]] = releaseEntry{hash: fields[0], size: size}
}
return out
}
+8
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"io"
"net/http"
"time"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
@@ -113,6 +114,10 @@ type DebMetadata struct {
Size int64
MD5 string
SHA256 string
// CreatedAt is the persisted insert time; the Release Date: is derived from
// the newest value so the index is byte-identical across replicas and
// regenerations (issue #117) rather than stamped from wall clock.
CreatedAt time.Time
}
// AlpineMetadataStore / AlpineMetadataDeleter / AlpineMetadataReader are the
@@ -185,6 +190,9 @@ type RPMMetadata struct {
Obsoletes []RPMDep
Files []RPMFile
Changelogs []RPMChangelog
// CreatedAt is the persisted upload timestamp; used as a stable, replica-independent
// value for the repodata <time>/<revision> fields so generated indexes are deterministic.
CreatedAt time.Time
}
type RPMDep struct {
+31 -4
View File
@@ -275,7 +275,7 @@ func (p *Provider) serveRepomd(w http.ResponseWriter, r *http.Request, reader pr
filelistsHash := sha256Hex(filelists)
otherHash := sha256Hex(other)
repomd := generateRepomd(primaryHash, len(primary), filelistsHash, len(filelists), otherHash, len(other))
repomd := generateRepomd(repomdRevision(metas), primaryHash, len(primary), filelistsHash, len(filelists), otherHash, len(other))
w.Header().Set("Content-Type", "application/xml")
w.WriteHeader(http.StatusOK)
@@ -315,8 +315,32 @@ func (p *Provider) serveOther(w http.ResponseWriter, r *http.Request, reader pro
w.Write(generateOtherXMLGZ(metas))
}
func generateRepomd(primaryHash string, primarySize int, filelistsHash string, filelistsSize int, otherHash string, otherSize int) []byte {
ts := fmt.Sprintf("%d", time.Now().Unix())
// stableUnix maps a persisted timestamp to a fixed integer for repodata's
// informational <time>/<timestamp> fields. Zero times (unset) collapse to 0 so
// output stays byte-identical across replicas and requests. dnf does not
// validate these values.
func stableUnix(t time.Time) int64 {
if t.IsZero() {
return 0
}
return t.Unix()
}
// repomdRevision derives repomd.xml's <revision>/<timestamp> from persisted
// state: the newest package upload time in the repo. It changes only when the
// repo's package set does, and is identical on every replica reading the same
// rows, so repomd.xml is byte-stable.
func repomdRevision(metas []provider.RPMMetadata) string {
var max int64
for _, m := range metas {
if u := stableUnix(m.CreatedAt); u > max {
max = u
}
}
return fmt.Sprintf("%d", max)
}
func generateRepomd(ts string, primaryHash string, primarySize int, filelistsHash string, filelistsSize int, otherHash string, otherSize int) []byte {
var b bytes.Buffer
b.WriteString(xml.Header)
b.WriteString(`<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">` + "\n")
@@ -359,7 +383,7 @@ func generatePrimaryXMLGZ(metas []provider.RPMMetadata) []byte {
if m.URL != "" {
fmt.Fprintf(&xmlBuf, " <url>%s</url>\n", xmlEscape(m.URL))
}
fmt.Fprintf(&xmlBuf, " <time file=\"%d\" build=\"0\"/>\n", time.Now().Unix())
fmt.Fprintf(&xmlBuf, " <time file=\"%d\" build=\"0\"/>\n", stableUnix(m.CreatedAt))
fmt.Fprintf(&xmlBuf, " <size package=\"%d\" installed=\"%d\" archive=\"0\"/>\n", m.RPMSize, m.InstalledSize)
fmt.Fprintf(&xmlBuf, " <location href=\"%s\"/>\n", xmlEscape(m.FilePath))
fmt.Fprintf(&xmlBuf, " <format>\n")
@@ -484,6 +508,9 @@ func xmlEscape(s string) string {
func gzipBytes(data []byte) []byte {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
// Pin every header field so the compressed bytes (and their sha256) depend
// only on the payload, never on wall-clock time or the Go version's gzip defaults.
gz.Header = gzip.Header{OS: 255}
gz.Write(data)
gz.Close()
return buf.Bytes()
@@ -0,0 +1,148 @@
package rpm
import (
"bytes"
"compress/gzip"
"encoding/xml"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"git.unkin.net/unkin/artifactapi/internal/provider"
)
func gunzip(t *testing.T, data []byte) string {
t.Helper()
zr, err := gzip.NewReader(bytes.NewReader(data))
if err != nil {
t.Fatalf("gzip reader: %v", err)
}
out, err := io.ReadAll(zr)
if err != nil {
t.Fatalf("gunzip: %v", err)
}
return string(out)
}
// sampleMetas returns a fixed two-package repo state whose upload timestamps are
// pinned, so any nondeterminism must come from the generators themselves.
func sampleMetas() []provider.RPMMetadata {
base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
return []provider.RPMMetadata{
{
Name: "alpha", Version: "1.0", Release: "1", Arch: "x86_64",
Summary: "a", Description: "d", ContentHash: "sha256:aaa",
FilePath: "Packages/alpha-1.0-1.x86_64.rpm", RPMSize: 10, InstalledSize: 20,
Provides: []provider.RPMDep{{Name: "alpha"}},
Requires: []provider.RPMDep{{Name: "libc", Flags: "GE", Version: "2.0"}},
CreatedAt: base,
},
{
Name: "beta", Version: "2.0", Release: "3", Arch: "noarch",
Summary: "b", Description: "d2", ContentHash: "sha256:bbb",
FilePath: "Packages/beta-2.0-3.noarch.rpm", RPMSize: 30, InstalledSize: 40,
CreatedAt: base.Add(time.Hour),
},
}
}
// TestRepodataGeneratorsDeterministic is the direct regression guard for #117:
// generating each metadata document twice from identical state must yield
// byte-identical output (hence an identical sha256). The old code embedded
// time.Now() inside primary.xml.gz, so its bytes/hash drifted every second.
func TestRepodataGeneratorsDeterministic(t *testing.T) {
metas := sampleMetas()
gens := map[string]func([]provider.RPMMetadata) []byte{
"primary": generatePrimaryXMLGZ,
"filelists": generateFilelistsXMLGZ,
"other": generateOtherXMLGZ,
}
for name, gen := range gens {
a := gen(metas)
b := gen(metas)
if sha256Hex(a) != sha256Hex(b) {
t.Errorf("%s: sha256 differs between two generations (nondeterministic): %s != %s",
name, sha256Hex(a), sha256Hex(b))
}
}
// repomd.xml itself must also be byte-stable across regenerations.
r1 := generateRepomd(repomdRevision(metas), sha256Hex(generatePrimaryXMLGZ(metas)), 1, "f", 2, "o", 3)
r2 := generateRepomd(repomdRevision(metas), sha256Hex(generatePrimaryXMLGZ(metas)), 1, "f", 2, "o", 3)
if string(r1) != string(r2) {
t.Error("repomd.xml differs between two generations")
}
}
// TestPrimaryTimeUsesPersistedCreatedAt proves the <time> element is a pure
// function of the persisted upload timestamp, not the wall clock.
func TestPrimaryTimeUsesPersistedCreatedAt(t *testing.T) {
metas := sampleMetas()
out := gunzip(t, generatePrimaryXMLGZ(metas))
if want := `<time file="1767323045" build="0"/>`; !strings.Contains(out, want) {
t.Errorf("primary.xml missing persisted <time> %q; got:\n%s", want, out)
}
// A zero (unset) CreatedAt collapses to a fixed 0, never a live clock value.
metas[0].CreatedAt = time.Time{}
out = gunzip(t, generatePrimaryXMLGZ(metas))
if !strings.Contains(out, `<time file="0" build="0"/>`) {
t.Errorf("zero CreatedAt should emit file=\"0\"; got:\n%s", out)
}
}
type repomdDoc struct {
Revision string `xml:"revision"`
Data []struct {
Type string `xml:"type,attr"`
Checksum struct {
Value string `xml:",chardata"`
} `xml:"checksum"`
Location struct {
Href string `xml:"href,attr"`
} `xml:"location"`
} `xml:"data"`
}
// TestRepomdHashMatchesServedBytes asserts the exact invariant #117 violated:
// the sha256 advertised in repomd.xml equals the sha256 of the bytes the
// content-addressed serve* handler returns for the same repo state.
func TestRepomdHashMatchesServedBytes(t *testing.T) {
p := &Provider{}
reader := fakeRPMReader{metas: sampleMetas()}
serve := func(path string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
r := httptest.NewRequest(http.MethodGet, "/"+path, nil)
if !p.ServeLocalIndex(w, r, reader, "repo", path) {
t.Fatalf("ServeLocalIndex false for %q", path)
}
if w.Code != http.StatusOK {
t.Fatalf("%s: code %d", path, w.Code)
}
return w
}
var doc repomdDoc
if err := xml.Unmarshal(serve("repodata/repomd.xml").Body.Bytes(), &doc); err != nil {
t.Fatalf("parse repomd: %v", err)
}
if len(doc.Data) != 3 {
t.Fatalf("expected 3 <data> entries, got %d", len(doc.Data))
}
for _, d := range doc.Data {
// The advertised location is content-addressed: repodata/<sha256>-<type>.xml.gz.
body := serve("repodata/" + d.Location.Href[len("repodata/"):]).Body.Bytes()
got := sha256Hex(body)
if got != d.Checksum.Value {
t.Errorf("%s: repomd advertises %s but served bytes hash to %s (dnf would reject)",
d.Type, d.Checksum.Value, got)
}
if d.Location.Href != "repodata/"+d.Checksum.Value+"-"+d.Type+".xml.gz" {
t.Errorf("%s: location %q not addressed by its checksum %s", d.Type, d.Location.Href, d.Checksum.Value)
}
}
}
+36 -6
View File
@@ -176,14 +176,44 @@ helm install <release> ${name}/<chart>`,
];
case 'alpine':
return [
{
title: 'Add the APK repository',
language: 'bash',
code: `echo '${proxy}/' | sudo tee -a /etc/apk/repositories
return isLocal
? [
{
title: 'Add the apk repo (real apk repo, APKINDEX auto-generated)',
language: 'bash',
code: `echo '${url}/api/v1/local/${name}' | sudo tee -a /etc/apk/repositories
sudo apk update --allow-untrusted
sudo apk add --allow-untrusted <package>`,
note: `Served unsigned (parity with the rpm repo's gpgcheck=0) — use --allow-untrusted, or install a signing key. apk fetches <arch>/APKINDEX.tar.gz under this base.`,
},
{
title: 'Publish a .apk (index regenerates automatically)',
language: 'bash',
code: `curl -fsSL --upload-file ./mypkg-1.0-r0.apk \\
${url}/api/v2/remotes/${name}/files/x86_64/mypkg-1.0-r0.apk`,
note: 'Upload each package at <arch>/<name>-<version>.apk — apk reconstructs that exact path from the index (APKINDEX carries no filename), so a mismatched path will 404 on install.',
},
]
: [
{
title: 'Add the APK repository',
language: 'bash',
code: `echo '${proxy}/' | sudo tee -a /etc/apk/repositories
sudo apk update
sudo apk add <package>`,
note: 'If the index is unsigned over the proxy, add --allow-untrusted or install the signing key into /etc/apk/keys.',
note: 'If the index is unsigned over the proxy, add --allow-untrusted or install the signing key into /etc/apk/keys.',
},
];
case 'github_alpine':
return [
{
title: 'Add the apk repo (metadata-only, from GitHub releases)',
language: 'bash',
code: `echo '${proxy}' | sudo tee -a /etc/apk/repositories
sudo apk update --allow-untrusted
sudo apk add --allow-untrusted <package>`,
note: "The per-arch APKINDEX is synthesized from the configured GitHub repo's release .apk assets; package downloads are redirected to the backing releases remote. Served unsigned, so --allow-untrusted.",
},
];