Add deb virtual repositories #138

2026-10-16T23:07:19+11:00

unkin-agent commented

2026-10-16 23:07:19 +11:00

Virtual repos merge helm, pypi and rpm but not deb, so several apt repos (e.g. github_deb remotes plus a local deb) cannot be served as one apt source.

merge member Packages for flat (./) and dists/<suite> layouts; member order wins duplicate (Package, Version, Architecture), stanzas missing any are kept
generate an unsigned Release with Acquire-By-Hash, serving indexes by name and by SHA256 from both retained generations
prefix a cleaned Filename with the member name and 302 downloads to the member route
cache merges per suite on the shared virtual cache, capped at 32 suites per virtual (LRU)
add unit tests and dockerised apt e2e for flat and suite layouts

Requires unkin/artifactapi#147.

Virtual repos merge helm, pypi and rpm but not deb, so several apt repos (e.g. github_deb remotes plus a local deb) cannot be served as one apt source. - merge member Packages for flat (`./`) and `dists/<suite>` layouts; member order wins duplicate (Package, Version, Architecture), stanzas missing any are kept - generate an unsigned Release with Acquire-By-Hash, serving indexes by name and by SHA256 from both retained generations - prefix a cleaned Filename with the member name and 302 downloads to the member route - cache merges per suite on the shared virtual cache, capped at 32 suites per virtual (LRU) - add unit tests and dockerised apt e2e for flat and suite layouts Requires unkin/artifactapi#147.

unkin-agent commented

2026-10-16 23:15:07 +11:00

internal/virtual/engine.go:cachedFile / deb_merger.go:189 — Packages/Packages.gz names are not content-addressed (Acquire-By-Hash: no), so the prev generation never helps for deb: cachedFile checks cur first. A client holding Release gen N whose Packages fetch lands after a re-merge (another client's Release past 60s TTL) gets gen N+1 bytes → apt "Hash Sum mismatch". → serve by-hash/SHA256/<hash> from both generations and advertise Acquire-By-Hash: yes, or serve Packages from prev when the client's Release is older (at minimum, add a test for this window).
internal/virtual/deb_merger.go:167 — only / is trimmed from member Filename; a member Filename of ./foo.deb becomes member/./foo.deb, which memberSegments rejects with ErrBadPath (apt install fails). → path.Clean the Filename before prefixing; add a test case.
internal/virtual/deb_merger.go:161 — stanza lacking Package/Version/Architecture all collapse to key \x00\x00 and every one after the first is dropped silently. → skip dedupe (or drop with an error) when Package is empty.
internal/virtual/engine.go (merged/gens) — gens is keyed by client-supplied suite (dists/<any>/) and grows unbounded for every suite that merges successfully, never evicted. → bound or evict entries older than mergeTTL*N.
internal/virtual/engine_test.go — no test covers generation retention (prev served after re-merge) or per-virtual key isolation for the refactored shared cache; the rpm path is only covered by pre-existing tests. → add a test for each against the shared merged/cachedFile.
e2e-docker/virtual_test.go — only the flat ./ layout is exercised with real apt; the dists/<suite> path (Release + Packages.gz fetch by apt) is unit-tested only. → add a suite-layout apt case, or note it as untested.
scope — refactor of rpm cache/singleflight (rpmGen→repoGen, fetchMembers, cachedRPMFile→cachedFile) is bundled with the deb feature. → split: PR 1 pure refactor (rpm behaviour unchanged, existing rpm tests as proof); PR 2 deb merger + engine wiring + e2e.

- internal/virtual/engine.go:cachedFile / deb_merger.go:189 — Packages/Packages.gz names are not content-addressed (`Acquire-By-Hash: no`), so the `prev` generation never helps for deb: `cachedFile` checks `cur` first. A client holding Release gen N whose Packages fetch lands after a re-merge (another client's Release past 60s TTL) gets gen N+1 bytes → apt "Hash Sum mismatch". → serve `by-hash/SHA256/<hash>` from both generations and advertise `Acquire-By-Hash: yes`, or serve Packages from `prev` when the client's Release is older (at minimum, add a test for this window). - internal/virtual/deb_merger.go:167 — only `/` is trimmed from member Filename; a member Filename of `./foo.deb` becomes `member/./foo.deb`, which `memberSegments` rejects with ErrBadPath (apt install fails). → `path.Clean` the Filename before prefixing; add a test case. - internal/virtual/deb_merger.go:161 — stanza lacking Package/Version/Architecture all collapse to key `\x00\x00` and every one after the first is dropped silently. → skip dedupe (or drop with an error) when Package is empty. - internal/virtual/engine.go (merged/gens) — `gens` is keyed by client-supplied suite (`dists/<any>/`) and grows unbounded for every suite that merges successfully, never evicted. → bound or evict entries older than mergeTTL*N. - internal/virtual/engine_test.go — no test covers generation retention (`prev` served after re-merge) or per-virtual key isolation for the refactored shared cache; the rpm path is only covered by pre-existing tests. → add a test for each against the shared `merged`/`cachedFile`. - e2e-docker/virtual_test.go — only the flat `./` layout is exercised with real apt; the `dists/<suite>` path (Release + `Packages.gz` fetch by apt) is unit-tested only. → add a suite-layout apt case, or note it as untested. - scope — refactor of rpm cache/singleflight (`rpmGen`→`repoGen`, `fetchMembers`, `cachedRPMFile`→`cachedFile`) is bundled with the deb feature. → split: PR 1 pure refactor (rpm behaviour unchanged, existing rpm tests as proof); PR 2 deb merger + engine wiring + e2e.

unkin-agent commented

2026-10-17 12:50:50 +11:00

nit: e2e-docker/virtual_test.go:227 — ! grep -q '^Filename: vdebs-b/' $idx never fails under set -e (bash exempts !-negated commands), so the "b is deduped" assertion is vacuous → use if grep -q ...; then exit 1; fi or grep -qv-style check.

- nit: e2e-docker/virtual_test.go:227 — `! grep -q '^Filename: vdebs-b/' $idx` never fails under `set -e` (bash exempts `!`-negated commands), so the "b is deduped" assertion is vacuous → use `if grep -q ...; then exit 1; fi` or `grep -qv`-style check.

unkin-agent changed target branch from master to benvin/virtual-shared-cache

2026-10-17 12:52:19 +11:00

benvin changed target branch from benvin/virtual-shared-cache to master

2026-10-17 13:28:17 +11:00

benvin added 5 commits 2026-10-17 13:28:17 +11:00

Add deb virtual repositories

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

78d24f7752

- merge member Packages indexes (flat and dists/<suite>) into one apt repo with a generated unsigned Release
- prefix Filename with the member name and 302 downloads to the member route
- share the rpm merge cache, singleflight and fail-closed member fetch

Share the virtual merge cache across index types

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

ceae176089

Merge remote-tracking branch 'origin/benvin/virtual-shared-cache' into benvin/virtual-deb eca686b7bb

# Conflicts:
#	internal/virtual/engine.go
#	internal/virtual/engine_test.go

Serve deb virtual indexes by hash and bound cached suites

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

ed6216d78b

fix vacuous e2e dedupe assertion

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

841b9f1547

ci/woodpecker/pr/pre-commit Pipeline was successful

Required

Details

ci/woodpecker/pr/test Pipeline was successful

Required

Details

ci/woodpecker/pr/build Pipeline was successful

Required

Details

This pull request has changes conflicting with the target branch.

internal/virtual/engine.go
internal/virtual/engine_test.go

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin benvin/virtual-deb:benvin/virtual-deb

git checkout benvin/virtual-deb

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#138