unkin/artifactapi
3
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

Add github_alpine metadata-only package type #115

Merged
benvin merged 2 commits from benvin/apk-github into master 2026-08-12 20:48:43 +10:00
Conversation 0 Commits 2 Files Changed 9
+1872 -29
unkin-agent commented 2026-08-12 01:15:00 +10:00
Member
Copy Link
Copy Source

Why

github_alpine is the Alpine/apk analog of the existing github_deb/github_rpm metadata-only remotes. It lets a plain GitHub-releases repo of .apk files be consumed as a real apk repository without artifactapi ever precaching the packages: it scans the repo's releases, derives each package's .PKGINFO from a ranged prefix fetch, synthesizes a per-arch APKINDEX.tar.gz from the cached metadata, and redirects the actual .apk downloads to a backend releases_remote. It stacks on the apk-local branch, reusing that work's alpine APKINDEX generator, .apk/.PKGINFO parser, Q1 pull-checksum, and AlpineMetadata store.

How

  • pkg/models: add PackageGitHubAlpine to the enum + validators (and test).
  • internal/provider/alpine/github.go: the github_alpine provider. ServeRemote serves per-arch <arch>/APKINDEX.tar.gz (reusing generateAPKIndex over arch-filtered AlpineMetadata rows, normalizeIndexPath for apk's ./ dot-segment), 302-redirects *.apk to {proxyBaseURL}/api/v1/remote/{releases_remote}/{path}, and cold-starts with a 503 + Retry-After. scanWithState lists releases with ETag/If-None-Match and incrementally derives/prunes. deriveAsset does a ranged GET of just the front of the .apk — the control gzip stream carrying .PKGINFO sits near the front — doubling the range on truncation; it parses .PKGINFO and computes the C: Q1 checksum (Q1+base64(sha1(control stream))). FilePath = the github-relative asset path so the redirect resolves.
  • internal/provider/alpine/syncer.go: a parallel background Syncer (own worker pool, shared rate limiter, deduped queue, DB lease), separate from the deb/rpm syncers.
  • internal/database/alpine_github_sync.go + github_alpine_sync_state table: ListGitHubAlpineRemotes + Claim/Release per-remote sync lease, kept separate from the deb/rpm tables.
  • internal/server/server.go: construct + Run the alpine syncer alongside deb/rpm and register it in the PackageType→Primer map (priming on create then flows through the existing generic remotes.go path).

The rpm/deb providers, syncers, and tables are untouched — this adds parallel alpine equivalents and reuses shared helpers already present in the alpine package.

Tests

Mirror the deb github tests: scan derives .PKGINFO/Q1 from a ranged prefix of a testsupport.MinimalApk served over an httptest range server (no full download); diff/prune; pattern filter; per-arch ServeRemote routing (index served per-arch and grouped, dot-segment collapse, .apk → 302, cold-start 503, warm 200, canceled-request-serves-cache); DB lease prevents a second replica; prime bypasses the recency window. go build/go vet/go mod tidy clean, make test (-race) green, pre-commit green.

## Why `github_alpine` is the Alpine/apk analog of the existing `github_deb`/`github_rpm` metadata-only remotes. It lets a plain GitHub-releases repo of `.apk` files be consumed as a real apk repository without artifactapi ever precaching the packages: it scans the repo's releases, derives each package's `.PKGINFO` from a ranged prefix fetch, synthesizes a per-arch `APKINDEX.tar.gz` from the cached metadata, and redirects the actual `.apk` downloads to a backend `releases_remote`. It stacks on the apk-local branch, reusing that work's alpine APKINDEX generator, `.apk`/`.PKGINFO` parser, Q1 pull-checksum, and `AlpineMetadata` store. ## How - **`pkg/models`**: add `PackageGitHubAlpine` to the enum + validators (and test). - **`internal/provider/alpine/github.go`**: the `github_alpine` provider. `ServeRemote` serves per-arch `<arch>/APKINDEX.tar.gz` (reusing `generateAPKIndex` over arch-filtered `AlpineMetadata` rows, `normalizeIndexPath` for apk's `./` dot-segment), 302-redirects `*.apk` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`, and cold-starts with a 503 + `Retry-After`. `scanWithState` lists releases with ETag/If-None-Match and incrementally derives/prunes. `deriveAsset` does a **ranged GET of just the front of the `.apk`** — the control gzip stream carrying `.PKGINFO` sits near the front — doubling the range on truncation; it parses `.PKGINFO` and computes the `C:` Q1 checksum (`Q1`+base64(sha1(control stream))). `FilePath` = the github-relative asset path so the redirect resolves. - **`internal/provider/alpine/syncer.go`**: a parallel background `Syncer` (own worker pool, shared rate limiter, deduped queue, DB lease), separate from the deb/rpm syncers. - **`internal/database/alpine_github_sync.go`** + `github_alpine_sync_state` table: `ListGitHubAlpineRemotes` + Claim/Release per-remote sync lease, kept separate from the deb/rpm tables. - **`internal/server/server.go`**: construct + `Run` the alpine syncer alongside deb/rpm and register it in the `PackageType→Primer` map (priming on create then flows through the existing generic `remotes.go` path). The rpm/deb providers, syncers, and tables are untouched — this adds parallel alpine equivalents and reuses shared helpers already present in the alpine package. ## Tests Mirror the deb github tests: scan derives `.PKGINFO`/Q1 from a ranged prefix of a `testsupport.MinimalApk` served over an httptest range server (no full download); diff/prune; pattern filter; per-arch `ServeRemote` routing (index served per-arch and grouped, dot-segment collapse, `.apk` → 302, cold-start 503, warm 200, canceled-request-serves-cache); DB lease prevents a second replica; prime bypasses the recency window. `go build`/`go vet`/`go mod tidy` clean, `make test` (-race) green, pre-commit green.
benvin changed target branch from benvin/apk-local to master 2026-08-12 20:39:47 +10:00
unkin-agent added 2 commits 2026-08-12 20:40:23 +10:00
Add github_alpine metadata-only package type aa96c8af70 
github_alpine is the Alpine/apk analog of github_deb/github_rpm: a
metadata-only remote that scans a GitHub repo's releases for .apk assets,
derives each package's .PKGINFO via a ranged prefix fetch (never
downloading whole packages), synthesizes a per-arch APKINDEX.tar.gz from
that cached metadata, and 302-redirects .apk downloads to a backend
releases_remote. It stacks on the apk-local work, reusing the alpine
provider's APKINDEX generator, .PKGINFO parser, Q1 checksum, and
AlpineMetadata store.

- pkg/models: add PackageGitHubAlpine to the enum + validators
- internal/provider/alpine/github.go: the github_alpine provider
  (ServeRemote per-arch index + .apk redirect, cold-start 503,
  scanWithState incremental derive, ranged .PKGINFO prefix fetch with
  range-doubling on truncation)
- internal/provider/alpine/syncer.go: parallel background Syncer
  (worker pool, shared limiter, deduped queue, DB lease)
- internal/database/alpine_github_sync.go + github_alpine_sync_state
  table: remote enumeration + per-remote sync lease
- internal/api/v2/remotes.go: primed on create via the shared Primer map
- internal/server/server.go: construct + Run the alpine syncer, register
  it in the Primer map
- tests mirror the deb github_test/syncer_test (scan/diff/prune, ranged
  .PKGINFO parse, per-arch ServeRemote routing, .apk 302, DB lease)
Fix github_alpine .apk redirect to resolve stored FilePath
ci/woodpecker/pr/build Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
d1c3c2fb6c 
apk reconstructs the download URL itself as <arch>/<name>-<version>.apk
because APKINDEX carries no filename field (unlike rpm's <location> or
deb's Filename:). ServeRemote forwarded that synthesized path verbatim
into the releases_remote redirect, pointing at a nonexistent,
allowlist-denied github.com path (404/403).

Look up the cached metadata row by arch plus the full reconstructed
filename (no hyphen-split, so -rN suffixes are preserved) and redirect
to the stored github-relative FilePath. Unknown packages now 404 instead
of redirecting to a bad path.
unkin-agent force-pushed benvin/apk-github from 6c6ad3066e to d1c3c2fb6c 2026-08-12 20:40:23 +10:00 Compare
benvin merged commit 5a06c16797 into master 2026-08-12 20:48:43 +10:00
benvin deleted branch benvin/apk-github 2026-08-12 20:48:43 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#115
Delete Branch "benvin/apk-github"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?