unkin/artifactapi
3
0
Fork 0
Code Issues 5 Pull Requests 2 Actions Packages Projects Releases 1 Wiki Activity

Add rpm virtual repositories #131

Merged
benvin merged 4 commits from benvin/virtual-rpm into master 2026-10-09 22:07:21 +11:00
Conversation 3 Commits 4 Files Changed 7
+1091 -2
unkin-agent commented 2026-10-09 19:17:45 +11:00
Member
Copy Link
Copy Source

Virtual repos only merge helm and pypi, so several rpm repos (e.g. many github_rpm remotes) cannot be served as one yum repo.

  • merge member primary/filelists/other into one repodata set; member order wins duplicate NEVRAs
  • read member repodata through local, github_rpm and proxied remote paths
  • prefix package locations (incl. xml:base under a member upstream) with the member name and 302 them to the member route
  • reject absolute and dot-segment member paths; escape the redirect and keep its query
  • return 502 when any member's repodata is unavailable
  • reuse each virtual's merge for 60s behind singleflight; serve data files from its current and previous merge (per replica)
  • add merger/engine unit tests and a dockerised dnf e2e case
Virtual repos only merge helm and pypi, so several rpm repos (e.g. many github_rpm remotes) cannot be served as one yum repo. - merge member primary/filelists/other into one repodata set; member order wins duplicate NEVRAs - read member repodata through local, github_rpm and proxied remote paths - prefix package locations (incl. xml:base under a member upstream) with the member name and 302 them to the member route - reject absolute and dot-segment member paths; escape the redirect and keep its query - return 502 when any member's repodata is unavailable - reuse each virtual's merge for 60s behind singleflight; serve data files from its current and previous merge (per replica) - add merger/engine unit tests and a dockerised dnf e2e case
unkin-agent added 2 commits 2026-10-09 19:17:46 +11:00
add rpm support to virtual repositories f59d6cf8d8
route rpm virtual package paths to their member
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
0cea7670eb
unkin-agent commented 2026-10-09 19:21:18 +11:00
Author
Member
Copy Link
Copy Source
  • internal/virtual/engine.go:154 — rest is not validated; <member>/../../remote/other/x becomes a 302 to /api/v1/<type>/<member>/../../remote/other/x, which clients normalise out of the member route → reject paths containing .. segments (or path.Clean and require it to stay unchanged) and add a test.
  • internal/virtual/engine.go:178 — a failed member is dropped and the partial merge is served as 200, so a transient upstream error silently removes that member's packages (and shifts dedupe winners) → fail the repodata request (502) when any member errors, or at least when it was reachable before.
  • internal/virtual/engine.go:197 — each of repomd/primary/filelists/other re-fetches and re-merges all members; if a member changes between the repomd and the primary request, the content-hashed href no longer exists and returns 404 mid-dnf makecache → cache the merged repo (keyed on member repomd checksums) so hrefs stay resolvable for at least the TTL.
  • internal/virtual/rpm_merger.go:199 — packages with xml:base are left unrewritten, bypassing the virtual/member route, and have no test → rewrite or reject, and cover it.
  • nit: internal/virtual/engine.go:154 — redirect drops the query string; rest is interpolated unescaped → url.URL{Path:..., RawQuery: r.URL.RawQuery} or EscapedPath.
- internal/virtual/engine.go:154 — `rest` is not validated; `<member>/../../remote/other/x` becomes a 302 to `/api/v1/<type>/<member>/../../remote/other/x`, which clients normalise out of the member route → reject paths containing `..` segments (or `path.Clean` and require it to stay unchanged) and add a test. - internal/virtual/engine.go:178 — a failed member is dropped and the partial merge is served as 200, so a transient upstream error silently removes that member's packages (and shifts dedupe winners) → fail the repodata request (502) when any member errors, or at least when it was reachable before. - internal/virtual/engine.go:197 — each of repomd/primary/filelists/other re-fetches and re-merges all members; if a member changes between the repomd and the primary request, the content-hashed href no longer exists and returns 404 mid-`dnf makecache` → cache the merged repo (keyed on member repomd checksums) so hrefs stay resolvable for at least the TTL. - internal/virtual/rpm_merger.go:199 — packages with `xml:base` are left unrewritten, bypassing the virtual/member route, and have no test → rewrite or reject, and cover it. - nit: internal/virtual/engine.go:154 — redirect drops the query string; `rest` is interpolated unescaped → `url.URL{Path:..., RawQuery: r.URL.RawQuery}` or `EscapedPath`.
unkin-agent added 1 commit 2026-10-09 19:32:36 +11:00
harden rpm virtual redirects, fail closed, keep data files resolvable
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
589421f995 
- reject absolute and dot-segment member paths, escape the target, keep the query
- return 502 when any member's repodata can't be fetched
- serve recently merged data files by hash for 5m after repomd changes
- fold xml:base under a member upstream into the virtual href
unkin-agent commented 2026-10-09 20:17:11 +11:00
Author
Member
Copy Link
Copy Source
  • internal/virtual/engine.go:344 — local member without repodata returns a wrapped ErrNotFound, which errors.Join/%w at :238 preserves, so handleVirtual answers 404 instead of 502 when a member is down → map member errors to a non-ErrNotFound error in fetchRPM (reserve ErrNotFound for unknown paths) and add a test with a local member missing repodata.
  • internal/virtual/engine.go:212 — every repomd request and every cache miss refetches, decompresses and merges all members fully in memory, with no singleflight and no size bound on rpmFiles (pruned only on store, by TTL only) → concurrent clients on large repos multiply memory/CPU. Add singleflight per virtual, cache the merged result for a short TTL, and bound the cache by bytes.
  • internal/virtual/engine.go:259 — cache is keyed by bare hash path, not by virtual, so virtual B serves virtual A merged data at B's URL → key by virtual name.
  • internal/virtual/rpm_merger.go:218 — xml:base not under a member upstream leaves the location untouched (no member prefix, base retained) so the client fetches directly from a foreign host, bypassing the member route → drop the package or fail the merge instead of passing it through.
  • nit: internal/virtual/rpm_merger.go:257 — only primary/filelists/other are emitted, so modules.yaml (modular repos) is silently dropped → document or fail if a member carries modules.
- internal/virtual/engine.go:344 — local member without repodata returns a wrapped ErrNotFound, which errors.Join/%w at :238 preserves, so handleVirtual answers 404 instead of 502 when a member is down → map member errors to a non-ErrNotFound error in fetchRPM (reserve ErrNotFound for unknown paths) and add a test with a local member missing repodata. - internal/virtual/engine.go:212 — every repomd request and every cache miss refetches, decompresses and merges all members fully in memory, with no singleflight and no size bound on rpmFiles (pruned only on store, by TTL only) → concurrent clients on large repos multiply memory/CPU. Add singleflight per virtual, cache the merged result for a short TTL, and bound the cache by bytes. - internal/virtual/engine.go:259 — cache is keyed by bare hash path, not by virtual, so virtual B serves virtual A merged data at B's URL → key by virtual name. - internal/virtual/rpm_merger.go:218 — xml:base not under a member upstream leaves the location untouched (no member prefix, base retained) so the client fetches directly from a foreign host, bypassing the member route → drop the package or fail the merge instead of passing it through. - nit: internal/virtual/rpm_merger.go:257 — only primary/filelists/other are emitted, so modules.yaml (modular repos) is silently dropped → document or fail if a member carries modules.
unkin-agent added 1 commit 2026-10-09 20:21:32 +11:00
bound rpm virtual merges, scope data cache per virtual, 502 on member failure
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
2ca6be11d2
unkin-agent commented 2026-10-09 20:25:19 +11:00
Author
Member
Copy Link
Copy Source

No findings.

No findings.
benvin merged commit 1837f6ef8c into master 2026-10-09 22:07:21 +11:00
benvin deleted branch benvin/virtual-rpm 2026-10-09 22:07:23 +11:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#131
Delete Branch "benvin/virtual-rpm"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?