unkin/artifactapi
3
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

feat: github_rpm metadata-only remote (GitHub releases as a yum repo, no precache) #107

Merged
benvin merged 2 commits from benvin/github-rpm-remote into master 2026-08-10 20:54:44 +10:00
Conversation 0 Commits 2 Files Changed 10
+1069 -15
unkinben commented 2026-08-10 09:35:53 +10:00
Owner
Copy Link
Copy Source

Why

Publishing RPMs to GitHub releases is common, but consuming them with dnf requires repodata GitHub does not provide, and mirroring every package into a local repo wastes storage and staleness-tracking on artifacts that already have a durable home. This exposes GitHub releases as a first-class RPM source that synthesizes repodata on the fly and never precaches the packages.

What

Add a github_rpm remote package type backed by a metadata-only provider.

  • Introduce a RemoteServer interception hook (the remote-side analog of LocalIndexer): handleProxy lets a provider fully answer a request before the byte-proxy engine, passing the request-derived proxy base URL and the DB as a RemoteMetadataStore.
  • Scan a repo's releases via the GitHub API (base_url = the releases API root) for .rpm assets, filtered by the remote's patterns (regex on asset filename), and reuse the existing local-rpm repodata generators to emit repomd.xml/primary/filelists/other.
  • Derive per-asset metadata without precaching: fetch only the RPM header via a ranged GET (retrying with a larger range on a truncated-header parse) for NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from the GitHub asset digest when present, else compute it once by streaming.
  • Cache derived metadata in rpm_metadata keyed by asset path; re-scan no more often than mutable_ttl, pruning assets that disappear upstream.
  • Serve each package's <location> as the github-relative download path so the client comes back to this remote, which 302-redirects to the releases_remote (an existing generic github.com remote) that streams the actual bytes.

Reuse the existing releases_remote field as the redirect target — it already carries exactly this "downloads served by remote X" semantic end to end, so no new schema/model field is needed.

Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns, added idempotently) so both local and github_rpm repodata resolve upgrades and conflicts; the local upload path records them too.

No-precache mechanics

  • Dependency metadata: always from the ranged header fetch (header precedes payload; rpm.Read stops at the payload boundary), giving dnf full resolution. Default range 1 MiB, doubling to 16 MiB.
  • Checksum: prefer the GitHub asset digest (no download); fall back to a one-time streamed sha256 only when absent. Header-only "minimal mode" (no deps) is rejected as a default because dnf needs accurate provides/requires and a correct pkgid checksum to install.

Tests

Header-range parsing incl. the retry loop, digest-vs-computed checksum selection, repodata synthesis with the redirect-able <location href>, the 302 redirect path (and the guard when releases_remote is unset), asset pattern filtering, and stale-asset pruning. go build/vet/test green; pre-commit clean.

Follow-ups

  • github_apk / github_deb metadata-only remotes (same pattern; not in this PR).
  • Terraform provider support for artifactapi_remote_github_rpm ships as a separate PR against terraform-provider-artifactapi (depends on this API surface).
## Why Publishing RPMs to GitHub releases is common, but consuming them with `dnf` requires repodata GitHub does not provide, and mirroring every package into a local repo wastes storage and staleness-tracking on artifacts that already have a durable home. This exposes GitHub releases as a first-class RPM source that synthesizes repodata on the fly and **never precaches the packages**. ## What Add a `github_rpm` remote package type backed by a metadata-only provider. - Introduce a `RemoteServer` interception hook (the remote-side analog of `LocalIndexer`): `handleProxy` lets a provider fully answer a request before the byte-proxy engine, passing the request-derived proxy base URL and the DB as a `RemoteMetadataStore`. - Scan a repo's releases via the GitHub API (`base_url` = the releases API root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset filename), and reuse the existing local-rpm repodata generators to emit `repomd.xml`/`primary`/`filelists`/`other`. - Derive per-asset metadata without precaching: fetch only the RPM header via a ranged GET (retrying with a larger range on a truncated-header parse) for NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from the GitHub asset `digest` when present, else compute it once by streaming. - Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more often than `mutable_ttl`, pruning assets that disappear upstream. - Serve each package's `<location>` as the github-relative download path so the client comes back to this remote, which **302-redirects** to the `releases_remote` (an existing generic github.com remote) that streams the actual bytes. Reuse the existing `releases_remote` field as the redirect target — it already carries exactly this "downloads served by remote X" semantic end to end, so no new schema/model field is needed. Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns, added idempotently) so both local and `github_rpm` repodata resolve upgrades and conflicts; the local upload path records them too. ## No-precache mechanics - **Dependency metadata**: always from the ranged header fetch (header precedes payload; `rpm.Read` stops at the payload boundary), giving `dnf` full resolution. Default range 1 MiB, doubling to 16 MiB. - **Checksum**: prefer the GitHub asset `digest` (no download); fall back to a one-time streamed sha256 only when absent. Header-only "minimal mode" (no deps) is rejected as a default because `dnf` needs accurate provides/requires and a correct pkgid checksum to install. ## Tests Header-range parsing incl. the retry loop, digest-vs-computed checksum selection, repodata synthesis with the redirect-able `<location href>`, the 302 redirect path (and the guard when `releases_remote` is unset), asset pattern filtering, and stale-asset pruning. `go build`/`vet`/`test` green; pre-commit clean. ## Follow-ups - `github_apk` / `github_deb` metadata-only remotes (same pattern; not in this PR). - Terraform provider support for `artifactapi_remote_github_rpm` ships as a separate PR against `terraform-provider-artifactapi` (depends on this API surface).
unkinben added 1 commit 2026-08-10 09:35:54 +10:00
feat: add github_rpm metadata-only remote serving GitHub releases as a yum repo
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
b727b990a2 
Publishing RPMs to GitHub releases is common, but consuming them with dnf
requires repodata that GitHub does not provide, and mirroring every package
into a local repo wastes storage and staleness-tracking on artifacts that
already have a durable home. Expose GitHub releases as a first-class RPM source
that synthesizes repodata on the fly and never precaches the packages.

Add a `github_rpm` remote package type backed by a metadata-only provider:

- Introduce a `RemoteServer` interception hook (the remote-side analog of
  `LocalIndexer`): `handleProxy` lets a provider fully answer a request before
  the byte-proxy engine, passing the request-derived proxy base URL and the DB
  as a `RemoteMetadataStore`.
- Scan a repo's releases via the GitHub API (`base_url` = the releases API
  root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset
  filename) and reuse the existing local-rpm repodata generators to emit
  `repomd.xml`/`primary`/`filelists`/`other`.
- Derive per-asset metadata without precaching: fetch only the RPM header via a
  ranged GET (retrying with a larger range on a truncated-header parse) for
  NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from
  the GitHub asset `digest` when present, else compute it once by streaming.
- Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more
  often than `mutable_ttl`, pruning assets that disappear upstream.
- Serve each package's `<location>` as the github-relative download path so the
  client comes back to this remote, which 302-redirects to the `releases_remote`
  (an existing generic github.com remote) that streams the actual bytes.

Reuse the existing `releases_remote` field as the redirect target — it already
carries exactly this "downloads served by remote X" semantic end to end.

Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns,
added idempotently) so both local and github_rpm repodata resolve upgrades and
conflicts; the local upload path now records them too.

Tests cover header-range parsing with the retry loop, digest-vs-computed
checksum selection, repodata synthesis with the redirect-able location href,
the 302 redirect path, asset pattern filtering, and stale-asset pruning.
unkinben added 1 commit 2026-08-10 11:09:31 +10:00
fix: decouple github_rpm scan and repodata read from the client request
ci/woodpecker/pr/build Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
9ba96ace41 
A cold `dnf makecache` against a github_rpm remote with many release
assets 500s on the first request: ServeRemote derives metadata for every
asset synchronously on the inbound request context, so once dnf hits its
makecache timeout and disconnects the canceled request context both
aborts the in-flight derive and poisons the subsequent repodata DB read,
which surfaces as HTTP 500. It only "works" on a lucky client retry that
finds the partially-populated cache fresh.

- detach the release scan to a background, timeout-bounded context so a
  client cancel can neither abort the shared derive nor cancel the read
- single-flight the scan per remote so concurrent requests never launch
  duplicate derives
- serve the current cache immediately when it is non-empty and derive in
  the background; only a completely empty cache blocks on a bounded first
  scan
- serve repodata on a context detached from the request, and treat a
  canceled/deadline-exceeded metadata read as a retryable 503 instead of
  a hard 500
benvin merged commit d154fbf3f3 into master 2026-08-10 20:54:44 +10:00
benvin deleted branch benvin/github-rpm-remote 2026-08-10 20:54:44 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#107
Delete Branch "benvin/github-rpm-remote"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?