Add parallel_fetch toggle for virtual member index files #165

Merged

benvin merged 3 commits from benvin/virtual-parallel-fetch into master 2026-10-25 15:13:57 +11:00

Conversation 3 Commits 3 Files Changed 7

+395 -51

unkin-agent commented 2026-10-25 13:09:51 +11:00

Member

Copy Link

Copy Source

Virtual members already merge concurrently, but each rpm member fetches primary/filelists/other one after another, and each deb member its Packages files the same way. On large virtuals the slowest member's serial chain pushes the merge past dnf's 30s timeout.

• add parallel_fetch (default false) to virtuals: model, v2 API, migration 0007
• fetch a member's index files concurrently (up to 4) when set; first error still fails the member
• thread the flag through rpm and deb member fetches

Virtual members already merge concurrently, but each rpm member fetches primary/filelists/other one after another, and each deb member its Packages files the same way. On large virtuals the slowest member's serial chain pushes the merge past dnf's 30s timeout. - add `parallel_fetch` (default false) to virtuals: model, v2 API, migration 0007 - fetch a member's index files concurrently (up to 4) when set; first error still fails the member - thread the flag through rpm and deb member fetches

unkin-agent added 1 commit 2026-10-25 13:09:52 +11:00

Add parallel_fetch toggle for virtual member index files 56be0ac266

unkin-agent commented 2026-10-25 13:16:24 +11:00

Author

Member

Copy Link

Copy Source

• nit: internal/virtual/engine.go (fetchFiles) — errgroup.Group has no context, so after the first error the remaining files are still launched and in-flight fetches are not cancelled → use errgroup.WithContext and skip/abort fetches once the group ctx is done.
• nit: internal/virtual/engine_test.go — no test covers parallel reaching fetchDebMember/fetchRPMMember or parallel_fetch round-tripping through the DB/v2 API (only mergeRPM is checked) → add a deb flag-passthrough case or a database/API round-trip test.

- nit: internal/virtual/engine.go (fetchFiles) — `errgroup.Group` has no context, so after the first error the remaining files are still launched and in-flight fetches are not cancelled → use `errgroup.WithContext` and skip/abort fetches once the group ctx is done. - nit: internal/virtual/engine_test.go — no test covers `parallel` reaching `fetchDebMember`/`fetchRPMMember` or `parallel_fetch` round-tripping through the DB/v2 API (only `mergeRPM` is checked) → add a deb flag-passthrough case or a database/API round-trip test.

unkin-agent added 1 commit 2026-10-25 13:26:15 +11:00

Cancel remaining index fetches on first error and test parallel_fetch wiring c454e44730

unkin-agent commented 2026-10-25 13:30:18 +11:00

Author

Member

Copy Link

Copy Source

• internal/api/v2/virtuals.go:60 — PUT decodes into a fresh models.Virtual, so any client that omits parallel_fetch (terraform-provider-artifactapi, anything built on pkg/client before this field) silently resets it to false on every update → add the field to the terraform provider in lockstep (or merge onto the existing row when the key is absent) before anyone enables it.
• nit: internal/virtual/engine_test.go:overlapEngine — 50ms hold window decides whether peak reaches 3; can flake on a loaded CI runner → raise to ~1s (loop exits early once all are in flight, so no cost).

- internal/api/v2/virtuals.go:60 — PUT decodes into a fresh `models.Virtual`, so any client that omits `parallel_fetch` (terraform-provider-artifactapi, anything built on pkg/client before this field) silently resets it to false on every update → add the field to the terraform provider in lockstep (or merge onto the existing row when the key is absent) before anyone enables it. - nit: internal/virtual/engine_test.go:`overlapEngine` — 50ms hold window decides whether peak reaches 3; can flake on a loaded CI runner → raise to ~1s (loop exits early once all are in flight, so no cost).

unkin-agent added 1 commit 2026-10-25 13:33:28 +11:00

Gate overlap test fetches on a barrier instead of a 50ms window 1228a1d473

unkin-agent commented 2026-10-25 13:37:50 +11:00

Author

Member

Copy Link

Copy Source

No findings.

No findings.

benvin merged commit cfb1f93d48 into master 2026-10-25 15:13:57 +11:00

benvin deleted branch benvin/virtual-parallel-fetch 2026-10-25 15:13:57 +11:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/artifactapi#165