feat: github_rpm metadata-only remote (GitHub releases as a yum repo, no precache) (#107)
## Why Publishing RPMs to GitHub releases is common, but consuming them with `dnf` requires repodata GitHub does not provide, and mirroring every package into a local repo wastes storage and staleness-tracking on artifacts that already have a durable home. This exposes GitHub releases as a first-class RPM source that synthesizes repodata on the fly and **never precaches the packages**. ## What Add a `github_rpm` remote package type backed by a metadata-only provider. - Introduce a `RemoteServer` interception hook (the remote-side analog of `LocalIndexer`): `handleProxy` lets a provider fully answer a request before the byte-proxy engine, passing the request-derived proxy base URL and the DB as a `RemoteMetadataStore`. - Scan a repo's releases via the GitHub API (`base_url` = the releases API root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset filename), and reuse the existing local-rpm repodata generators to emit `repomd.xml`/`primary`/`filelists`/`other`. - Derive per-asset metadata without precaching: fetch only the RPM header via a ranged GET (retrying with a larger range on a truncated-header parse) for NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from the GitHub asset `digest` when present, else compute it once by streaming. - Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more often than `mutable_ttl`, pruning assets that disappear upstream. - Serve each package's `<location>` as the github-relative download path so the client comes back to this remote, which **302-redirects** to the `releases_remote` (an existing generic github.com remote) that streams the actual bytes. Reuse the existing `releases_remote` field as the redirect target — it already carries exactly this "downloads served by remote X" semantic end to end, so no new schema/model field is needed. Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns, added idempotently) so both local and `github_rpm` repodata resolve upgrades and conflicts; the local upload path records them too. ## No-precache mechanics - **Dependency metadata**: always from the ranged header fetch (header precedes payload; `rpm.Read` stops at the payload boundary), giving `dnf` full resolution. Default range 1 MiB, doubling to 16 MiB. - **Checksum**: prefer the GitHub asset `digest` (no download); fall back to a one-time streamed sha256 only when absent. Header-only "minimal mode" (no deps) is rejected as a default because `dnf` needs accurate provides/requires and a correct pkgid checksum to install. ## Tests Header-range parsing incl. the retry loop, digest-vs-computed checksum selection, repodata synthesis with the redirect-able `<location href>`, the 302 redirect path (and the guard when `releases_remote` is unset), asset pattern filtering, and stale-asset pruning. `go build`/`vet`/`test` green; pre-commit clean. ## Follow-ups - `github_apk` / `github_deb` metadata-only remotes (same pattern; not in this PR). - Terraform provider support for `artifactapi_remote_github_rpm` ships as a separate PR against `terraform-provider-artifactapi` (depends on this API surface). Reviewed-on: #107 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #107.
This commit is contained in:
@@ -32,9 +32,54 @@ API: `http://localhost:8000` | Frontend: `http://localhost:5173`
|
||||
| `puppet` | `v3/modules/*`, `v3/releases*` | `.tar.gz` |
|
||||
| `terraform` | `*/versions` | `*/download/*/*` |
|
||||
| `goproxy` | `@v/list`, `@latest` | `.info`, `.mod`, `.zip` |
|
||||
| `github_rpm` | `repodata/*` (synthesized) | `.rpm` (redirected) |
|
||||
|
||||
Providers classify paths automatically. Users only configure what to proxy and TTLs.
|
||||
|
||||
### `github_rpm` — GitHub releases as a yum repo (metadata-only, no precache)
|
||||
|
||||
A `github_rpm` remote turns a GitHub repo's **releases** into a real `dnf`/`yum`
|
||||
repository without ever caching the packages. It scans releases for `.rpm`
|
||||
assets, derives each package's metadata (NEVRA, requires/provides/conflicts/
|
||||
obsoletes, files, checksum) and **synthesizes `repodata/` on the fly**. Package
|
||||
metadata comes from a **ranged GET of just the RPM header** (the header sits at
|
||||
the front of the file, so the whole package is never downloaded); the sha256
|
||||
checksum comes from the GitHub asset `digest` when present, else a one-time
|
||||
lazy stream. Derived metadata is cached (keyed by asset) so repodata generation
|
||||
is cheap on repeat, and refreshed no more often than `mutable_ttl`.
|
||||
|
||||
Each package's `<location>` points back at the remote, which **302-redirects**
|
||||
the download to the `releases_remote` — an existing generic `github.com` remote
|
||||
that streams the actual bytes. `dnf` follows the redirect transparently.
|
||||
|
||||
```hcl
|
||||
# Backend that serves the actual .rpm bytes from github.com.
|
||||
resource "artifactapi_remote_generic" "github" {
|
||||
name = "github"
|
||||
base_url = "https://github.com"
|
||||
patterns = [
|
||||
"acme/tools/releases/download/.*\\.rpm$", # allowlist the repo's release assets
|
||||
]
|
||||
}
|
||||
|
||||
resource "artifactapi_remote_github_rpm" "acme-tools" {
|
||||
name = "acme-tools"
|
||||
base_url = "https://api.github.com/repos/acme/tools" # the releases API root
|
||||
releases_remote = "github" # backend for downloads
|
||||
mutable_ttl = 3600 # release re-scan interval
|
||||
|
||||
# Optional: restrict which release assets become packages (regex on filename).
|
||||
patterns = [".*\\.x86_64\\.rpm$", ".*\\.noarch\\.rpm$"]
|
||||
|
||||
# Optional: a token for private repos / higher API rate limits.
|
||||
# password = "ghp_..."
|
||||
}
|
||||
```
|
||||
|
||||
`dnf` config: `baseurl=https://artifactapi.example/api/v1/remote/acme-tools`.
|
||||
The repo is multi-arch (no `$basearch` needed) — `dnf` selects matching packages
|
||||
from the synthesized metadata.
|
||||
|
||||
## Terraform
|
||||
|
||||
Remotes and virtuals are managed by Terraform. Each package type has its own resource:
|
||||
|
||||
Reference in New Issue
Block a user