Add github_deb metadata-only package type #112

Merged

benvin merged 2 commits from benvin/deb-github-metadataonly into master 2026-08-11 23:28:12 +10:00

Conversation 0 Commits 2 Files Changed 11

+1935 -14

unkin-agent commented 2026-08-11 22:21:40 +10:00

Member

Copy Link

Copy Source

Why

This stacks the Debian/apt analog of github_rpm on top of the deb local+remote work (#111). It lets a GitHub repo's .deb release assets be consumed as a real apt repository without artifactapi ever precaching whole packages: it derives per-asset control metadata from a ranged prefix fetch, synthesizes a flat apt repo from the cache, and redirects the actual .deb downloads to a backend releases_remote (the generic github.com remote).

Base is benvin/deb-local-remote (stacked) to keep the diff atomic.

How

• Adds github_deb to the package-type enum and validity map.
• Adds the github_deb provider mirroring github_rpm: ServeRemote serves Packages/Packages.gz/Release, returns 404 for InRelease/Release.gpg (unsigned, consumed via [trusted=yes]), and 302-redirects *.deb to {proxyBaseURL}/api/v1/remote/{releases_remote}/{path}; cold-start prime with a retryable 503.
• deriveAsset ranged-GETs the front of the .deb (an ar archive), locates and fully reads control.tar.*, and parses the control paragraph — doubling the range if the control member is truncated. The Packages SHA256 comes from the GitHub asset digest when present, else a one-time full stream; MD5sum is left unset (apt verifies against SHA256 under [trusted=yes]).
• Adds a github_deb background Syncer (own worker pool, shared rate limiter, deduped queue) with per-remote DB-lease-gated scans so only one replica scans per window.
• Adds the github_deb_sync_state table plus ListGitHubDebRemotes / ClaimGitHubDebSyncLease / ReleaseGitHubDebSyncLease DB helpers, kept separate from the rpm ones.
• Primes github_deb remotes on create and runs the deb syncer alongside the rpm one; prime-on-create is routed by package type.
• Reuses the deb apt-index generators and control parser; the Packages generator now skips empty hash lines so a SHA256-only entry is valid.

Notes / deviations

• Filename convention: the Filename stored in the Packages index is the github-relative asset path (same as rpm's assetPath), not pool/<asset>. This is required for the .deb 302 to {releases_remote=github}/{path} to resolve against github.com; it still matches the *.deb redirect rule.
• GitHub client helpers (releases pagination, ranged GET, auth headers) are duplicated into the deb package rather than shared, because the rpm equivalents are unexported in package rpm and the task requires not modifying the rpm provider.
• go build, go vet, go mod tidy, and make test (-race, incl. the Postgres lease integration tests) all pass; pre-commit clean.

Do not merge — for review.

## Why This stacks the Debian/apt analog of `github_rpm` on top of the deb local+remote work (#111). It lets a GitHub repo's `.deb` release assets be consumed as a real apt repository without artifactapi ever precaching whole packages: it derives per-asset control metadata from a ranged prefix fetch, synthesizes a flat apt repo from the cache, and redirects the actual `.deb` downloads to a backend `releases_remote` (the generic github.com remote). Base is `benvin/deb-local-remote` (stacked) to keep the diff atomic. ## How - Adds `github_deb` to the package-type enum and validity map. - Adds the `github_deb` provider mirroring `github_rpm`: `ServeRemote` serves `Packages`/`Packages.gz`/`Release`, returns 404 for `InRelease`/`Release.gpg` (unsigned, consumed via `[trusted=yes]`), and 302-redirects `*.deb` to `{proxyBaseURL}/api/v1/remote/{releases_remote}/{path}`; cold-start prime with a retryable 503. - `deriveAsset` ranged-GETs the front of the `.deb` (an `ar` archive), locates and fully reads `control.tar.*`, and parses the control paragraph — doubling the range if the control member is truncated. The Packages `SHA256` comes from the GitHub asset `digest` when present, else a one-time full stream; `MD5sum` is left unset (apt verifies against SHA256 under `[trusted=yes]`). - Adds a `github_deb` background Syncer (own worker pool, shared rate limiter, deduped queue) with per-remote DB-lease-gated scans so only one replica scans per window. - Adds the `github_deb_sync_state` table plus `ListGitHubDebRemotes` / `ClaimGitHubDebSyncLease` / `ReleaseGitHubDebSyncLease` DB helpers, kept separate from the rpm ones. - Primes `github_deb` remotes on create and runs the deb syncer alongside the rpm one; prime-on-create is routed by package type. - Reuses the deb apt-index generators and control parser; the Packages generator now skips empty hash lines so a SHA256-only entry is valid. ## Notes / deviations - **Filename convention:** the `Filename` stored in the Packages index is the **github-relative** asset path (same as rpm's `assetPath`), not `pool/<asset>`. This is required for the `.deb` 302 to `{releases_remote=github}/{path}` to resolve against github.com; it still matches the `*.deb` redirect rule. - **GitHub client helpers** (releases pagination, ranged GET, auth headers) are duplicated into the deb package rather than shared, because the rpm equivalents are unexported in `package rpm` and the task requires not modifying the rpm provider. - `go build`, `go vet`, `go mod tidy`, and `make test` (`-race`, incl. the Postgres lease integration tests) all pass; pre-commit clean. Do not merge — for review.

benvin changed target branch from benvin/deb-local-remote to master 2026-08-11 23:21:25 +10:00

unkin-agent added 2 commits 2026-08-11 23:23:23 +10:00

feat: github_deb metadata-only package type ... b2a6be8eb5

Add the Debian/apt analog of github_rpm: a metadata-only remote that scans
a GitHub repo's releases for .deb assets, derives per-asset control metadata
via a ranged prefix fetch (never downloading whole packages), synthesizes a
flat apt repository, and redirects .deb downloads to a backend releases_remote.

- Add PackageGitHubDeb to the package-type enum + validity map.
- Add the github_deb provider (internal/provider/deb/github.go): ServeRemote
  serves Packages/Packages.gz/Release, 404s the signed index variants
  (consumed via [trusted=yes]), and 302-redirects *.deb to the releases_remote;
  deriveAsset ranged-GETs the ar prefix, locates control.tar.*, and parses the
  control paragraph, doubling the range on truncation; sha256 comes from the
  asset digest when present, else a one-time full stream.
- Add the github_deb background Syncer (internal/provider/deb/syncer.go): its
  own worker pool, shared rate limiter, deduped queue, and DB-lease-gated scans.
- Add github_deb_sync_state table plus ListGitHubDebRemotes/Claim/Release DB
  helpers (separate from the rpm ones).
- Prime github_deb remotes on create and run the deb syncer alongside the rpm
  one; route prime-on-create by package type.
- Reuse the deb apt-index generators and control parser; skip empty hash lines
  in the Packages index so a SHA256-only metadata entry is valid.

fix(deb): normalize apt flat-repo ./ dist prefix in github_deb serve ... fd4fb160df

The metadata-only github_deb flat repo has the same defect as the local repo:
real apt requests <repo>/./Packages (and ./Release), which ServeRemote matched
literally and 404d. Reuse deb.go normalizeIndexPath to collapse the dot-segment
before matching the synthesized index.

unkin-agent force-pushed benvin/deb-github-metadataonly from 992d4a68ff to fd4fb160df 2026-08-11 23:23:23 +10:00 Compare

benvin merged commit 5fde0ee58e into master 2026-08-11 23:28:12 +10:00

benvin deleted branch benvin/deb-github-metadataonly 2026-08-11 23:28:12 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#112