unkin/artifactapi
3
0
Fork 0
Code Issues 5 Pull Requests 11 Actions Packages Projects Releases 1 Wiki Activity

Add ansible Galaxy remote type #137

Open
unkin-agent wants to merge 7 commits from benvin/remote-ansible into benvin/remote-query-forwarder
pull from: benvin/remote-ansible
merge into: unkin:benvin/remote-query-forwarder
Conversation 2 Commits 7 Files Changed 12
+309 -1
unkin-agent commented 2026-10-16 23:07:03 +11:00
Member
Copy Link
Copy Source

ansible-galaxy installs pull collections straight from galaxy.ansible.com because artifactapi has no Galaxy remote type.

  • add ansible package type proxying the Galaxy v3 API (discovery, collection versions, artifact downloads)
  • forward digits-only limit (capped at 100) and offset on metadata paths, canonicalised; tarball paths forward no query
  • rewrite only href, download_url and links.* to root-relative remote paths so tarballs flow through artifactapi
  • treat artifacts/*.tar.gz as immutable and JSON metadata as mutable
  • add a UI usage snippet and an ansible e2e caching case

requires unkin/artifactapi#146

ansible-galaxy installs pull collections straight from galaxy.ansible.com because artifactapi has no Galaxy remote type. - add `ansible` package type proxying the Galaxy v3 API (discovery, collection versions, artifact downloads) - forward digits-only `limit` (capped at 100) and `offset` on metadata paths, canonicalised; tarball paths forward no query - rewrite only `href`, `download_url` and `links.*` to root-relative remote paths so tarballs flow through artifactapi - treat `artifacts/*.tar.gz` as immutable and JSON metadata as mutable - add a UI usage snippet and an ansible e2e caching case requires unkin/artifactapi#146
unkin-agent commented 2026-10-16 23:21:40 +11:00
Author
Member
Copy Link
Copy Source
  • internal/api/v1/proxy.go:95 — raw client query is appended to the path, so it becomes the cache key (Redis TTL, artifacts row, S3 indexes/<remote>/<path>?<q>) and is sent upstream verbatim: any ?x=<random> mints unlimited cache entries/objects and upstream requests, and param order/encoding variants split keys → forward only an allowlist (limit, offset; add others only if needed), re-encode via url.Values.Encode() (sorted), cap value length/digits, drop everything else.
  • internal/provider/ansible/ansible.go:63 — isArtifact runs on path+query, so .../versions/?x=.tar.gz is classified Immutable (cached forever, never rewritten) and content-typed gzip → strip the query (path, _, _ = strings.Cut(path, "?")) before the suffix check; also require the artifacts path segment, not just .tar.gz.
  • internal/api/v1/proxy.go:95 — query is appended for every ansible path including tarballs, so ?n=1..N on one tarball creates N artifact rows/aliases for the same bytes → only forward the query on non-artifact (metadata) paths.
  • internal/provider/ansible/ansible.go:53 — text-level ReplaceAll of "/api/ rewrites any JSON string value starting with /api/ (descriptions, readme fields) and hardcodes the /api/v1/remote/ mount; remote.Name is spliced into the body unescaped → unmarshal and rewrite only href, download_url, and links.*, or at minimum JSON-escape the prefix.
  • internal/provider/ansible/ansible_test.go:81 — TestForwardsQuery only asserts the marker interface; nothing tests the proxy.go change (query appended, no query = unchanged path, non-forwarder providers unchanged) or the cache key → add a handler-level test with a forwarder and a non-forwarder, and a test that unknown params are dropped.
  • internal/provider/ansible/ansible_test.go:12 — Classify has no case with a query suffix (see .tar.gz finding) and rewrite has no case for a body with other /api/ strings or odd remote names → add them.
  • e2e-docker/caching_test.go:33 — only the tarball is exercised; the metadata path (paginated versions/?limit=&offset= + rewritten links) the PR exists for has no e2e → add a case following links.next through the proxy.
  • scope — QueryForwarder is a cross-cutting change to the shared proxy handler/cache-key semantics bundled with a new provider → split into PRs: (1) provider.QueryForwarder + proxy.go + allowlist + tests, (2) ansible provider/model/server registration, (3) UI + README + e2e (or fold into 2).
- internal/api/v1/proxy.go:95 — raw client query is appended to the path, so it becomes the cache key (Redis TTL, artifacts row, S3 `indexes/<remote>/<path>?<q>`) and is sent upstream verbatim: any `?x=<random>` mints unlimited cache entries/objects and upstream requests, and param order/encoding variants split keys → forward only an allowlist (`limit`, `offset`; add others only if needed), re-encode via `url.Values.Encode()` (sorted), cap value length/digits, drop everything else. - internal/provider/ansible/ansible.go:63 — `isArtifact` runs on path+query, so `.../versions/?x=.tar.gz` is classified Immutable (cached forever, never rewritten) and content-typed gzip → strip the query (`path, _, _ = strings.Cut(path, "?")`) before the suffix check; also require the artifacts path segment, not just `.tar.gz`. - internal/api/v1/proxy.go:95 — query is appended for every ansible path including tarballs, so `?n=1..N` on one tarball creates N artifact rows/aliases for the same bytes → only forward the query on non-artifact (metadata) paths. - internal/provider/ansible/ansible.go:53 — text-level `ReplaceAll` of `"/api/` rewrites any JSON string value starting with `/api/` (descriptions, readme fields) and hardcodes the `/api/v1/remote/` mount; remote.Name is spliced into the body unescaped → unmarshal and rewrite only `href`, `download_url`, and `links.*`, or at minimum JSON-escape the prefix. - internal/provider/ansible/ansible_test.go:81 — `TestForwardsQuery` only asserts the marker interface; nothing tests the proxy.go change (query appended, no query = unchanged path, non-forwarder providers unchanged) or the cache key → add a handler-level test with a forwarder and a non-forwarder, and a test that unknown params are dropped. - internal/provider/ansible/ansible_test.go:12 — Classify has no case with a query suffix (see `.tar.gz` finding) and rewrite has no case for a body with other `/api/` strings or odd remote names → add them. - e2e-docker/caching_test.go:33 — only the tarball is exercised; the metadata path (paginated `versions/?limit=&offset=` + rewritten links) the PR exists for has no e2e → add a case following `links.next` through the proxy. - scope — QueryForwarder is a cross-cutting change to the shared proxy handler/cache-key semantics bundled with a new provider → split into PRs: (1) `provider.QueryForwarder` + proxy.go + allowlist + tests, (2) ansible provider/model/server registration, (3) UI + README + e2e (or fold into 2).
unkin-agent commented 2026-10-17 12:48:52 +11:00
Author
Member
Copy Link
Copy Source

No findings.

No findings.
unkin-agent changed target branch from master to benvin/remote-query-forwarder 2026-10-17 12:52:24 +11:00
unkin-agent added 5 commits 2026-10-17 12:52:24 +11:00
Add ansible Galaxy remote type
ci/woodpecker/pr/build Pipeline failed
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
13d693b58a 
- add ansible package type proxying the Galaxy v3 API
- rewrite href/links/download_url back to the remote so tarballs flow through it
- forward the query string for providers that opt in, keeping Galaxy pagination working
- add a UI usage snippet and an ansible e2e caching case
Retrigger CI
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
0f826aa331
Merge benvin/remote-query-forwarder into benvin/remote-ansible 7bb11b4761
Forward ansible pagination params and rewrite only Galaxy link fields af6a088ef4
Document ansible pagination and artifact paths
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
9935691c6b
unkin-agent added 2 commits 2026-10-17 12:56:31 +11:00
Merge remote-tracking branch 'origin/benvin/remote-query-forwarder' into benvin/remote-ansible ef5a38475f
Validate and cap ansible pagination params
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
285e8674ca
Some checks are pending
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/test Pipeline was successful
Details
ci/woodpecker/pr/build Pipeline was successful
Details
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin benvin/remote-ansible:benvin/remote-ansible
git checkout benvin/remote-ansible
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/artifactapi#137
Delete Branch "benvin/remote-ansible"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?