Add gems virtual repositories #154

Merged

benvin merged 3 commits from benvin/virtual-gems into master 2026-10-18 02:07:52 +11:00

Conversation 1 Commits 3 Files Changed 10

+633 -2

unkin-agent commented 2026-10-18 00:46:05 +11:00

Member

Copy Link

Copy Source

Bundler needs one source for internal and public gems, but gems virtuals have no merger, so a local gems repo and rubygems.org cannot be served together.

• merge info/<gem> across members (member order wins a duplicate version-platform); pass a single-member gem through verbatim
• regenerate versions lines and md5s for gems several members list; union names
• 302 gems/*.gem and quick/Marshal.4.8/* to the member owning that version, via the shared member-path validation
• return 502 on a member error; cache merges in the shared virtual cache; route HEAD on virtuals
• add docs/gems/virtual.md and a stock ruby:3.3 bundler e2e-docker case

Bundler needs one source for internal and public gems, but gems virtuals have no merger, so a local gems repo and rubygems.org cannot be served together. - merge `info/<gem>` across members (member order wins a duplicate version-platform); pass a single-member gem through verbatim - regenerate `versions` lines and md5s for gems several members list; union `names` - 302 `gems/*.gem` and `quick/Marshal.4.8/*` to the member owning that version, via the shared member-path validation - return 502 on a member error; cache merges in the shared virtual cache; route HEAD on virtuals - add `docs/gems/virtual.md` and a stock `ruby:3.3` bundler e2e-docker case

unkin-agent commented 2026-10-18 00:50:49 +11:00

Author

Member

Copy Link

Copy Source

• nit: internal/api/v1/gems_virtual.go:36 — /versions is served as a plain 200 with no ETag, Repr-Digest or Range support, and the body (new created_at + moved shared lines) differs every 60s, so bundler re-downloads the full file on every run (~23MB against rubygems.org) → set a content-hash ETag and answer If-None-Match with 304; keep created_at stable while the body is unchanged.
• nit: internal/virtual/engine.go:303 — a regenerated /versions never equals the cached one (fresh created_at), so prev pins a second copy of the ~23MB body that gems never reads (files() is nil), on top of the recorder, ReadAll and merge-buffer copies per member → compare with the header stripped, and drop prev for gems.
• nit: internal/virtual/gems_merger.go:121 — a shared gem whose merged /info returns ErrNotFound is silently dropped from /versions → return an error (502) so the virtual fails closed.
• nit: internal/virtual/gems_merger.go:259 — a download request tries up to 8 candidate names, each fetching /info from every member (rubygems.org included), and misses are not cached → cache negative results briefly.

- nit: internal/api/v1/gems_virtual.go:36 — `/versions` is served as a plain 200 with no ETag, Repr-Digest or Range support, and the body (new `created_at` + moved shared lines) differs every 60s, so bundler re-downloads the full file on every run (~23MB against rubygems.org) → set a content-hash ETag and answer If-None-Match with 304; keep `created_at` stable while the body is unchanged. - nit: internal/virtual/engine.go:303 — a regenerated `/versions` never equals the cached one (fresh `created_at`), so `prev` pins a second copy of the ~23MB body that gems never reads (`files()` is nil), on top of the recorder, `ReadAll` and merge-buffer copies per member → compare with the header stripped, and drop `prev` for gems. - nit: internal/virtual/gems_merger.go:121 — a shared gem whose merged `/info` returns ErrNotFound is silently dropped from `/versions` → return an error (502) so the virtual fails closed. - nit: internal/virtual/gems_merger.go:259 — a download request tries up to 8 candidate names, each fetching `/info` from every member (rubygems.org included), and misses are not cached → cache negative results briefly.

benvin changed target branch from benvin/local-gems to master 2026-10-18 01:12:38 +11:00

benvin added 2 commits 2026-10-18 01:12:38 +11:00

Add local gems repositories ... 38641983e7

Serve gem push/yank and the compact index from local gems repos.

Add gems virtual repositories ... d035c6cec2

Merge the compact index across gems members and redirect downloads to the owning member.

unkin-agent added 1 commit 2026-10-18 01:14:32 +11:00

Merge remote-tracking branch 'origin/master' into benvin/virtual-gems ... 5b579c73ef

# Conflicts:
#	README.md

benvin merged commit 18c58dc9c7 into master 2026-10-18 02:07:52 +11:00

benvin deleted branch benvin/virtual-gems 2026-10-18 02:07:52 +11:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#154