unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Woodpecker k8s runner: trust the internal CA so buildx can push to artifactapi #325

New Issue
Open
opened 2026-08-02 16:24:31 +10:00 by unkinben · 2 comments
unkinben commented 2026-08-02 16:24:31 +10:00
Owner
Copy Link
Copy Source

Problem
The 12 open per-repo PRs (branch benvin/push-artifactapi) that repoint docker image pushes from the Gitea registry to artifactapi docker-internal will x509-fail on tag: plugin-docker-buildx builds via a nested moby/buildkit container that does not trust the internal unkin.net CA (PR CI is a false green — dry-run never contacts the registry). This is why nothing in the estate pushes to artifactapi via buildx.

Proposal

  • make the woodpecker k8s runner/buildkit trust the internal CA (mount the vault-ca-cert into the buildkit builder / buildkit_config CA), estate-wide — the durable fix; OR
  • switch the 12 push pipelines to build an OCI archive then publish with skopeo --dest-tls-verify=false (per-repo, bakes TLS-off into pipelines)
  • once resolved, merge the 12 benvin/push-artifactapi PRs (artifactapi's own image stays on Gitea — circular)

Acceptance

  • a tagged build in any migrated repo pushes to artifactapi docker-internal without x509; the 12 PRs mergeable; no false-green push path.
**Problem** The 12 open per-repo PRs (branch benvin/push-artifactapi) that repoint docker image pushes from the Gitea registry to artifactapi docker-internal will x509-fail on tag: plugin-docker-buildx builds via a nested moby/buildkit container that does not trust the internal unkin.net CA (PR CI is a false green — dry-run never contacts the registry). This is why nothing in the estate pushes to artifactapi via buildx. **Proposal** - make the woodpecker k8s runner/buildkit trust the internal CA (mount the vault-ca-cert into the buildkit builder / buildkit_config CA), estate-wide — the durable fix; OR - switch the 12 push pipelines to build an OCI archive then publish with skopeo --dest-tls-verify=false (per-repo, bakes TLS-off into pipelines) - once resolved, merge the 12 benvin/push-artifactapi PRs (artifactapi's own image stays on Gitea — circular) **Acceptance** - a tagged build in any migrated repo pushes to artifactapi docker-internal without x509; the 12 PRs mergeable; no false-green push path.
unkinben commented 2026-08-08 18:37:58 +10:00
Author
Owner
Copy Link
Copy Source

Assessment — not actionable in argocd-apps, recommend close

The x509 problem is still real: artifactapi.k8s.syd1.au.unkin.net still serves a vault-issuer (internal PKI) cert (gateway.yaml annotation + live kubectl get certificate artifactapi-tls → issuerRef vault-issuer), no repo pushes there yet (11 benvin/push-artifactapi branches still open; every buildx pipeline still targets git.unkin.net). But neither proposed fix is an argocd-apps change, and the one argocd-side enabler that is needed already exists:

  1. "Make the woodpecker k8s runner/buildkit trust the internal CA (estate-wide)" — not implementable here. Woodpecker's k8s backend has no global step-pod volume/secret/CA injection (volumes are per-step only; the chart's agent extraVolumeMounts attach to the long-lived agent container, which does not perform the push — buildkit runs in the ephemeral privileged plugin-docker-buildx pod). CA trust can therefore only be set per-repo in each pipeline, never via the runner. There is no WOODPECKER_BACKEND_K8S_* knob for it.

  2. Per-repo insecure buildx / skopeo — the only argocd-side enabler (the privileged woodpeckerci/plugin-docker-buildx:latest-insecure) is already in WOODPECKER_PLUGINS_PRIVILEGED (present since the initial migration, #13). argocd-apps already provides everything the per-repo path needs; the remaining work is entirely in the 11 per-repo PRs.

Durable alternative now on the table (post-#327/#331/#337) — but blocked on DNS, not argocd

Give artifactapi a publicly-trusted Let's Encrypt cert so buildkit trusts it natively — this deletes the internal CA from the picture for every client, not just woodpecker. Currently impossible for this host:

  • artifactapi.k8s.syd1.au.unkin.net is internal-only — no public A record; k8s.syd1.au.unkin.net has no public SOA/NS (split-horizon).
  • No public _acme-challenge delegation for it or a *.k8s.syd1.au.unkin.net wildcard — only _acme-challenge.unkin.net is CNAME'd into acme.unkin.net.
  • DNS-01 (cnameStrategy: Follow → rfc2136 to bind-external, authoritative for acme.unkin.net only) would fall back to an UPDATE on k8s.syd1.au.unkin.net → NOTAUTH, challenge fails.

To unblock: add _acme-challenge.artifactapi.k8s.syd1.au.unkin.net (or a *.k8s.syd1.au.unkin.net wildcard) CNAME _acme-challenge.acme.unkin.net in the public unkin.net view, then flip the artifactapi Gateway issuer vault-issuer → letsencrypt (one-line argocd change). That gateway flip is the only future argocd-apps deliverable, and it is gated on the DNS record.

Recommendation

Closing: there is no actionable argocd-apps change today (the insecure-plugin enabler is already present; runner-level CA trust is infeasible on the k8s backend). Track the push migration in the per-repo PRs; when the public _acme-challenge delegation lands, the Gateway → letsencrypt flip is a fresh, correctly-scoped argocd-apps change.

## Assessment — not actionable in argocd-apps, recommend close The x509 problem is still real: `artifactapi.k8s.syd1.au.unkin.net` still serves a **vault-issuer** (internal PKI) cert (`gateway.yaml` annotation + live `kubectl get certificate artifactapi-tls` → issuerRef vault-issuer), no repo pushes there yet (11 `benvin/push-artifactapi` branches still open; every buildx pipeline still targets `git.unkin.net`). But neither proposed fix is an argocd-apps change, and the one argocd-side enabler that is needed already exists: 1. **"Make the woodpecker k8s runner/buildkit trust the internal CA (estate-wide)" — not implementable here.** Woodpecker's k8s backend has no global step-pod volume/secret/CA injection (volumes are per-step only; the chart's agent `extraVolumeMounts` attach to the long-lived agent container, which does not perform the push — buildkit runs in the ephemeral privileged `plugin-docker-buildx` pod). CA trust can therefore only be set **per-repo** in each pipeline, never via the runner. There is no `WOODPECKER_BACKEND_K8S_*` knob for it. 2. **Per-repo insecure buildx / skopeo** — the only argocd-side enabler (the privileged `woodpeckerci/plugin-docker-buildx:latest-insecure`) is **already** in `WOODPECKER_PLUGINS_PRIVILEGED` (present since the initial migration, #13). argocd-apps already provides everything the per-repo path needs; the remaining work is entirely in the 11 per-repo PRs. ### Durable alternative now on the table (post-#327/#331/#337) — but blocked on DNS, not argocd Give artifactapi a publicly-trusted **Let's Encrypt** cert so buildkit trusts it natively — this deletes the internal CA from the picture for *every* client, not just woodpecker. Currently impossible for this host: - `artifactapi.k8s.syd1.au.unkin.net` is internal-only — no public A record; `k8s.syd1.au.unkin.net` has no public SOA/NS (split-horizon). - No public `_acme-challenge` delegation for it or a `*.k8s.syd1.au.unkin.net` wildcard — only `_acme-challenge.unkin.net` is CNAME'd into `acme.unkin.net`. - DNS-01 (`cnameStrategy: Follow` → rfc2136 to bind-external, authoritative for `acme.unkin.net` only) would fall back to an UPDATE on `k8s.syd1.au.unkin.net` → **NOTAUTH**, challenge fails. To unblock: add `_acme-challenge.artifactapi.k8s.syd1.au.unkin.net` (or a `*.k8s.syd1.au.unkin.net` wildcard) `CNAME _acme-challenge.acme.unkin.net` in the **public** unkin.net view, then flip the artifactapi `Gateway` issuer `vault-issuer` → `letsencrypt` (one-line argocd change). That gateway flip is the only future argocd-apps deliverable, and it is gated on the DNS record. ### Recommendation Closing: there is no actionable argocd-apps change today (the insecure-plugin enabler is already present; runner-level CA trust is infeasible on the k8s backend). Track the push migration in the per-repo PRs; when the public `_acme-challenge` delegation lands, the `Gateway` → `letsencrypt` flip is a fresh, correctly-scoped argocd-apps change.
unkinben reopened this issue 2026-08-08 18:38:52 +10:00
unkinben commented 2026-08-08 18:39:01 +10:00
Author
Owner
Copy Link
Copy Source

Reopening — this was closed without a resolving PR, but the assessment above confirms it's still required (the x509 blocker is real and ~11 per-repo push PRs remain blocked). Like #323, it's not fixable via an argocd-apps PR: estate-wide Woodpecker runner CA-trust isn't possible on the k8s backend, and the public-LE alternative is blocked on a missing public _acme-challenge delegation for the artifactapi host/wildcard. Keeping open and re-scoping to the per-repo push path + the DNS delegation, not closing until actually resolved.

Reopening — this was closed without a resolving PR, but the assessment above confirms it's still required (the x509 blocker is real and ~11 per-repo push PRs remain blocked). Like #323, it's not fixable via an argocd-apps PR: estate-wide Woodpecker runner CA-trust isn't possible on the k8s backend, and the public-LE alternative is blocked on a missing public _acme-challenge delegation for the artifactapi host/wildcard. Keeping open and re-scoping to the per-repo push path + the DNS delegation, not closing until actually resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#325
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?