unkin/artifactapi
3
0
Fork 0
Code Issues 5 Pull Requests 3 Actions Packages Projects Releases 1 Wiki Activity

Add deb virtual repositories #138

Open
unkin-agent wants to merge 6 commits from benvin/virtual-deb into master
pull from: benvin/virtual-deb
merge into: unkin:master
Conversation 3 Commits 6 Files Changed 7
+840 -10
unkin-agent commented 2026-10-16 23:07:19 +11:00
Member
Copy Link
Copy Source

Virtual repos merge helm, pypi, rpm, npm, gems, ansible, goproxy and puppet but not deb, so several apt repos (e.g. github_deb remotes plus a local deb) cannot be served as one apt source.

  • merge member Packages for flat (./) and dists/<suite> layouts; member order wins duplicate (Package, Version, Architecture)
  • generate an unsigned Release with Acquire-By-Hash, serving indexes by name and by SHA256 from both retained generations
  • prefix Filename with the member name and 302 downloads to the member route
  • cache merges per suite on the shared virtual cache, capped at 32 suites per virtual (LRU)
  • add docs/deb/virtual.md, unit tests and dockerised apt e2e for flat and suite layouts
Virtual repos merge helm, pypi, rpm, npm, gems, ansible, goproxy and puppet but not deb, so several apt repos (e.g. github_deb remotes plus a local deb) cannot be served as one apt source. - merge member Packages for flat (`./`) and `dists/<suite>` layouts; member order wins duplicate (Package, Version, Architecture) - generate an unsigned Release with Acquire-By-Hash, serving indexes by name and by SHA256 from both retained generations - prefix Filename with the member name and 302 downloads to the member route - cache merges per suite on the shared virtual cache, capped at 32 suites per virtual (LRU) - add docs/deb/virtual.md, unit tests and dockerised apt e2e for flat and suite layouts
unkin-agent commented 2026-10-16 23:15:07 +11:00
Author
Member
Copy Link
Copy Source
  • internal/virtual/engine.go:cachedFile / deb_merger.go:189 — Packages/Packages.gz names are not content-addressed (Acquire-By-Hash: no), so the prev generation never helps for deb: cachedFile checks cur first. A client holding Release gen N whose Packages fetch lands after a re-merge (another client's Release past 60s TTL) gets gen N+1 bytes → apt "Hash Sum mismatch". → serve by-hash/SHA256/<hash> from both generations and advertise Acquire-By-Hash: yes, or serve Packages from prev when the client's Release is older (at minimum, add a test for this window).
  • internal/virtual/deb_merger.go:167 — only / is trimmed from member Filename; a member Filename of ./foo.deb becomes member/./foo.deb, which memberSegments rejects with ErrBadPath (apt install fails). → path.Clean the Filename before prefixing; add a test case.
  • internal/virtual/deb_merger.go:161 — stanza lacking Package/Version/Architecture all collapse to key \x00\x00 and every one after the first is dropped silently. → skip dedupe (or drop with an error) when Package is empty.
  • internal/virtual/engine.go (merged/gens) — gens is keyed by client-supplied suite (dists/<any>/) and grows unbounded for every suite that merges successfully, never evicted. → bound or evict entries older than mergeTTL*N.
  • internal/virtual/engine_test.go — no test covers generation retention (prev served after re-merge) or per-virtual key isolation for the refactored shared cache; the rpm path is only covered by pre-existing tests. → add a test for each against the shared merged/cachedFile.
  • e2e-docker/virtual_test.go — only the flat ./ layout is exercised with real apt; the dists/<suite> path (Release + Packages.gz fetch by apt) is unit-tested only. → add a suite-layout apt case, or note it as untested.
  • scope — refactor of rpm cache/singleflight (rpmGen→repoGen, fetchMembers, cachedRPMFile→cachedFile) is bundled with the deb feature. → split: PR 1 pure refactor (rpm behaviour unchanged, existing rpm tests as proof); PR 2 deb merger + engine wiring + e2e.
- internal/virtual/engine.go:cachedFile / deb_merger.go:189 — Packages/Packages.gz names are not content-addressed (`Acquire-By-Hash: no`), so the `prev` generation never helps for deb: `cachedFile` checks `cur` first. A client holding Release gen N whose Packages fetch lands after a re-merge (another client's Release past 60s TTL) gets gen N+1 bytes → apt "Hash Sum mismatch". → serve `by-hash/SHA256/<hash>` from both generations and advertise `Acquire-By-Hash: yes`, or serve Packages from `prev` when the client's Release is older (at minimum, add a test for this window). - internal/virtual/deb_merger.go:167 — only `/` is trimmed from member Filename; a member Filename of `./foo.deb` becomes `member/./foo.deb`, which `memberSegments` rejects with ErrBadPath (apt install fails). → `path.Clean` the Filename before prefixing; add a test case. - internal/virtual/deb_merger.go:161 — stanza lacking Package/Version/Architecture all collapse to key `\x00\x00` and every one after the first is dropped silently. → skip dedupe (or drop with an error) when Package is empty. - internal/virtual/engine.go (merged/gens) — `gens` is keyed by client-supplied suite (`dists/<any>/`) and grows unbounded for every suite that merges successfully, never evicted. → bound or evict entries older than mergeTTL*N. - internal/virtual/engine_test.go — no test covers generation retention (`prev` served after re-merge) or per-virtual key isolation for the refactored shared cache; the rpm path is only covered by pre-existing tests. → add a test for each against the shared `merged`/`cachedFile`. - e2e-docker/virtual_test.go — only the flat `./` layout is exercised with real apt; the `dists/<suite>` path (Release + `Packages.gz` fetch by apt) is unit-tested only. → add a suite-layout apt case, or note it as untested. - scope — refactor of rpm cache/singleflight (`rpmGen`→`repoGen`, `fetchMembers`, `cachedRPMFile`→`cachedFile`) is bundled with the deb feature. → split: PR 1 pure refactor (rpm behaviour unchanged, existing rpm tests as proof); PR 2 deb merger + engine wiring + e2e.
unkin-agent commented 2026-10-17 12:50:50 +11:00
Author
Member
Copy Link
Copy Source
  • nit: e2e-docker/virtual_test.go:227 — ! grep -q '^Filename: vdebs-b/' $idx never fails under set -e (bash exempts !-negated commands), so the "b is deduped" assertion is vacuous → use if grep -q ...; then exit 1; fi or grep -qv-style check.
- nit: e2e-docker/virtual_test.go:227 — `! grep -q '^Filename: vdebs-b/' $idx` never fails under `set -e` (bash exempts `!`-negated commands), so the "b is deduped" assertion is vacuous → use `if grep -q ...; then exit 1; fi` or `grep -qv`-style check.
unkin-agent changed target branch from master to benvin/virtual-shared-cache 2026-10-17 12:52:19 +11:00
benvin changed target branch from benvin/virtual-shared-cache to master 2026-10-17 13:28:17 +11:00
benvin added 5 commits 2026-10-17 13:28:17 +11:00
Add deb virtual repositories
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
78d24f7752 
- merge member Packages indexes (flat and dists/<suite>) into one apt repo with a generated unsigned Release
- prefix Filename with the member name and 302 downloads to the member route
- share the rpm merge cache, singleflight and fail-closed member fetch
Share the virtual merge cache across index types
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
ceae176089
Merge remote-tracking branch 'origin/benvin/virtual-shared-cache' into benvin/virtual-deb eca686b7bb 
# Conflicts:
#	internal/virtual/engine.go
#	internal/virtual/engine_test.go
Serve deb virtual indexes by hash and bound cached suites
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
ed6216d78b
fix vacuous e2e dedupe assertion
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
841b9f1547
unkin-agent added 1 commit 2026-10-20 22:42:41 +11:00
Merge origin/master into benvin/virtual-deb
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
6b4cfaf65d
unkin-agent commented 2026-10-20 22:48:48 +11:00
Author
Member
Copy Link
Copy Source

No findings.

No findings.
All checks were successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Required
Details
ci/woodpecker/pr/build Pipeline was successful
Required
Details
ci/woodpecker/pr/test Pipeline was successful
Required
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin benvin/virtual-deb:benvin/virtual-deb
git checkout benvin/virtual-deb
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#138
Delete Branch "benvin/virtual-deb"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?