Files
artifactapi/internal/provider/provider.go
T
unkinben b727b990a2
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
feat: add github_rpm metadata-only remote serving GitHub releases as a yum repo
Publishing RPMs to GitHub releases is common, but consuming them with dnf
requires repodata that GitHub does not provide, and mirroring every package
into a local repo wastes storage and staleness-tracking on artifacts that
already have a durable home. Expose GitHub releases as a first-class RPM source
that synthesizes repodata on the fly and never precaches the packages.

Add a `github_rpm` remote package type backed by a metadata-only provider:

- Introduce a `RemoteServer` interception hook (the remote-side analog of
  `LocalIndexer`): `handleProxy` lets a provider fully answer a request before
  the byte-proxy engine, passing the request-derived proxy base URL and the DB
  as a `RemoteMetadataStore`.
- Scan a repo's releases via the GitHub API (`base_url` = the releases API
  root) for `.rpm` assets, filtered by the remote's `patterns` (regex on asset
  filename) and reuse the existing local-rpm repodata generators to emit
  `repomd.xml`/`primary`/`filelists`/`other`.
- Derive per-asset metadata without precaching: fetch only the RPM header via a
  ranged GET (retrying with a larger range on a truncated-header parse) for
  NEVRA, requires/provides/conflicts/obsoletes and files; take the sha256 from
  the GitHub asset `digest` when present, else compute it once by streaming.
- Cache derived metadata in `rpm_metadata` keyed by asset path; re-scan no more
  often than `mutable_ttl`, pruning assets that disappear upstream.
- Serve each package's `<location>` as the github-relative download path so the
  client comes back to this remote, which 302-redirects to the `releases_remote`
  (an existing generic github.com remote) that streams the actual bytes.

Reuse the existing `releases_remote` field as the redirect target — it already
carries exactly this "downloads served by remote X" semantic end to end.

Extend the shared RPM metadata model with conflicts/obsoletes (JSONB columns,
added idempotently) so both local and github_rpm repodata resolve upgrades and
conflicts; the local upload path now records them too.

Tests cover header-range parsing with the retry loop, digest-vs-computed
checksum selection, repodata synthesis with the redirect-able location href,
the 302 redirect path, asset pattern filtering, and stale-asset pruning.
2026-08-10 09:35:22 +10:00

165 lines
4.3 KiB
Go

package provider
import (
"context"
"fmt"
"io"
"net/http"
"git.unkin.net/unkin/artifactapi/pkg/models"
)
type Mutability int
const (
Immutable Mutability = iota
Mutable
)
type Provider interface {
Type() models.PackageType
Classify(path string) Mutability
ContentType(path string) string
UpstreamURL(remote models.Remote, path string) string
RewriteResponse(body []byte, remote models.Remote, proxyBaseURL string) ([]byte, error)
AuthHeaders(ctx context.Context, remote models.Remote) (http.Header, error)
}
type FileEntry struct {
FilePath string
ContentHash string
}
type FileStore interface {
ListFilesByPrefix(ctx context.Context, repoName, prefix string) ([]FileEntry, error)
ListPackages(ctx context.Context, repoName string) ([]string, error)
}
type LocalUploader interface {
ValidateUpload(filePath string) (storagePath, contentType string, err error)
UploadResponse(storagePath, contentHash string, sizeBytes int64) map[string]any
}
type LocalIndexer interface {
ServeLocalIndex(w http.ResponseWriter, r *http.Request, files FileStore, repoName, path string) bool
GenerateLocalIndex(ctx context.Context, files FileStore, repoName, path string) ([]byte, error)
}
type BlobReader interface {
Download(ctx context.Context, key string) (io.ReadCloser, int64, error)
}
type PostUploadHook interface {
AfterUpload(ctx context.Context, repoName, storagePath, contentHash string, blobs BlobReader, db MetadataStore)
}
// PostDeleteHook lets a provider clean up derived state (e.g. RPM metadata that
// feeds generated repodata) after a local file is removed.
type PostDeleteHook interface {
AfterDelete(ctx context.Context, repoName, storagePath string, db MetadataDeleter) error
}
type MetadataStore interface {
InsertRPMMetadata(ctx context.Context, meta *RPMMetadata) error
}
// RemoteServer lets a remote provider fully answer a request itself instead of
// going through the byte-proxy engine. It is the remote-side analog of
// LocalIndexer: a metadata-only remote (e.g. github_rpm) uses it to synthesize
// repodata from derived per-asset metadata and to redirect package downloads to
// a backend remote, without ever precaching the packages. Returning false lets
// the normal proxy path take over.
type RemoteServer interface {
ServeRemote(w http.ResponseWriter, r *http.Request, remote models.Remote, path, proxyBaseURL string, store RemoteMetadataStore) bool
}
// RemoteMetadataStore is the persistence surface a RemoteServer needs to cache
// and read the metadata it derives per upstream asset. *database.DB satisfies it.
type RemoteMetadataStore interface {
RPMMetadataReader
MetadataStore
MetadataDeleter
}
type MetadataDeleter interface {
DeleteRPMMetadata(ctx context.Context, repoName, filePath string) error
}
type RPMMetadataReader interface {
ListRPMMetadataEntries(ctx context.Context, repoName string) ([]RPMMetadata, error)
}
type RPMMetadata struct {
RepoName string
FilePath string
ContentHash string
Name string
Epoch int
Version string
Release string
Arch string
Summary string
Description string
RPMSize int64
InstalledSize int64
License string
Vendor string
Group string
BuildHost string
SourceRPM string
URL string
Packager string
Requires []RPMDep
Provides []RPMDep
Conflicts []RPMDep
Obsoletes []RPMDep
Files []RPMFile
Changelogs []RPMChangelog
}
type RPMDep struct {
Name string `json:"name"`
Flags string `json:"flags,omitempty"`
Epoch string `json:"epoch,omitempty"`
Version string `json:"version,omitempty"`
Release string `json:"release,omitempty"`
}
type RPMFile struct {
Path string `json:"path"`
Type string `json:"type,omitempty"`
}
type RPMChangelog struct {
Author string `json:"author"`
Date int64 `json:"date"`
Text string `json:"text"`
}
type IndexMerger interface {
MergeIndexes(members []MemberIndex, proxyBaseURL string) ([]byte, error)
}
type MemberIndex struct {
RemoteName string
Body []byte
}
var registry = map[models.PackageType]Provider{}
func Register(p Provider) {
registry[p.Type()] = p
}
func Get(t models.PackageType) (Provider, error) {
p, ok := registry[t]
if !ok {
return nil, fmt.Errorf("no provider registered for package type %q", t)
}
return p, nil
}
func All() map[models.PackageType]Provider {
return registry
}