Compare commits

..

196 Commits

Author SHA1 Message Date
Ben Vincent ef4f5f68a0 traefik-internal: add ldaps :636 entrypoint for authentik LDAPS
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
Why:
- traefik-internal has no 636 entrypoint, so the authentik-ldap Gateway's
  636/TLS passthrough listeners are invalid and the whole Gateway is
  rejected (Accepted: False), taking LDAPS down.
- The traefik-internal LoadBalancer only publishes 80/443, so nothing
  reaches the Authentik LDAP outpost on 636.

How:
- Add an `ldaps` entrypoint on :636 (raw TCP, no TLS termination) to the
  traefik-internal helm values so 636/TLS passthrough streams pass through
  to the LDAP outpost.
- Expose 636 on the traefik-internal LoadBalancer via the port's
  `expose.default`, keeping the pinned 198.18.200.4 / common group.
2026-08-08 23:16:50 +10:00
unkinben 4c2f275f04 puppet: reduce privilege in namespace workloads (#307) (#319)
Why: shrink the blast radius of the Puppet control-plane pods (CA/eyaml keys, compiled catalogs) per the security sweep in #307 — remove root where it is not required and strip cargo-culted capabilities.

How:
- puppetboard cert-generator init: root+APE:true -> uid 1000, drop:[all], APE:false; pod fsGroup 1000; removed trailing `chown -R 1000:1000` (PVC now group-owned).
- puppetdb create-log-dir init: root -> uid 999, drop:[all], APE:false; pod fsGroup 999; removed `chown 999:999`.
- All OpenVox capability add-lists: removed the duplicate CAP_-prefixed spellings (k8s normalises both to the same kernel cap) and dropped the unused AUDIT_WRITE.
- Added allowPrivilegeEscalation:false and seccompProfile RuntimeDefault across the workloads.

Stays root (evidence-backed, class-B fallback): the puppetserver master/compiler and puppetdb main containers, plus the perms-and-dirs and generate-types root containers. The OpenVox image entrypoint runs `chown -R puppet:puppet` over root-owned baked-in dirs and drops the JVM to the puppet user via `runuser` (needs CHOWN/SETUID/SETGID); a non-root start crashloops. Their cap sets are reduced to the minimum justified (CHOWN/DAC_OVERRIDE/FOWNER[/SETUID/SETGID]).

Validation: `kustomize build --enable-helm` clean; kubeconform 0 invalid / 0 errors; pre-commit (yamllint etc.) green. Confirmed against live pods: puppetserver/puppetdb JVMs already run as puppet/puppetdb via `runuser`; `pam_loginuid` is absent from the su/runuser PAM stacks and loginuid is unset, so dropping AUDIT_WRITE is safe.

Post-merge smoke test (puppet had an outage this session — watch closely): after argocd sync, confirm puppetserver master + a compiler reach `running` at /status/v1/simple, puppetdb reaches `running`, puppetboard serves 200, and the generate-types + g10k CronJobs complete — i.e. catalogs still compile and reports still ingest.

Closes #307

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #319
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 22:36:44 +10:00
unkinben 5ebf2cc581 Bump kea images to v0.1.2 (socket dir 0750 fix) (#343)
## Why
kea-dhcp4 and kea-ctrl-agent crash-loop because kea 2.6.5 refuses a unix-socket directory more relaxed than 0750, but the operator's shared emptyDir mounts `/var/run/kea` at 0777 (`'socket-name' is invalid: socket path:/var/run/kea ... more relaxed permissions than 750`). kea-operator v0.1.2 renders entrypoints that tighten it.

## How
- bump kea-operator, kea, and kea-api images to v0.1.2

---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #343
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 22:21:36 +10:00
unkinben da4a66046a Add Tier-2 per-app Vector transform pipelines (structured logs) (#320)
Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. **Stacks on #318 — merge after it.**

How:
- 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: **bind_query** (k8s bind-* + VM named), **rancher_audit** (cattle-system sidecar JSON), **cnpg_pg** (ONE transform for all 10 CNPG clusters via the `.postgres` container), **gitea** (router+access, k8s+VM), **puppet** (openvoxserver/openvoxdb logback + access), **litellm** (JSON request logs), **postfix** (per-line maillog).
- Carve `.postgres` out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact.
- Companion k8s flips in this PR: litellm `JSON_LOGS=True`; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on.
- 15 new `vector test` cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing `fields Map(String,String)` — no DDL change.

Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with `.file`/`.SYSLOG_IDENTIFIER` tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #320
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 22:12:46 +10:00
unkinben c722df415a puppetdb: isolate only the stockpile queue per pod (fix #340 crashloop) (#342) 2026-08-08 20:16:11 +10:00
unkinben 4c7c97ab80 puppetdb: unique per-pod command-queue directory (#340) 2026-08-08 20:04:17 +10:00
unkinben 5d1cc10588 Pin puppet master to a single Recreate replica (#341)
## Why

The puppet MASTER is the singleton CA/master. A second master, even transiently during a rolling update, races on CA/cert signing and shared state (the CA lives on a shared PVC mounted by every master pod). The master was previously driven by an HPA with `minReplicas: 2`, `maxReplicas: 5` and a `RollingUpdate` strategy, so 2-5 masters could coexist normally and a rollout would briefly run old+new masters against the same CA data — a latent CA-corruption/split-brain bug. Recreate guarantees the old pod terminates before the new one starts, so two masters never coexist.

## Changes

- Set `puppetserver-master` `spec.replicas: 1` and `spec.strategy.type: Recreate` (drops RollingUpdate).
- Remove the `puppetserver-masters-autoscaler` HPA and its kustomization entry, which forced 2-5 master replicas and would otherwise override `replicas: 1`.
- Refresh the `puppetserver-master-vpa` note to reflect the pinned-singleton, no-HPA state (VPA stays `updateMode: Off`, recommendation-only).

The compiler (`puppetserver-compiler`) remains the horizontally-scalable tier with its own HPA — untouched. puppetdb/puppetboard untouched.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #341
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 20:02:38 +10:00
unkinben ba7a1a9509 Enable Reloader secret watching, scope existing auto to configmap-only (#326) (#339)
## Why

The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret.

## Changes

- Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched.
- Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads.
- Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`.
- Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf.
- Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`).

## Safety review (secret-only / CA workloads)

`vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded.

- `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated.
- `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`).
- `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only.
- `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added.

CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`).

## Notes / coordination

- Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression).
- Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here.

## Validation

- `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains.
- `uvx pre-commit run --all-files` — all hooks pass.

Closes #326

---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #339
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 19:35:31 +10:00
unkinben ca5e29e685 Fix kea CrashLoopBackOff: drop DHCP ntp hostnames, bump to v0.1.1 (#338)
kea-0/kea-1 crash-looped after the dhcp-system deploy. Two root causes:

1. **kea-dhcp4** rejected the `ntp-servers` option (DHCP code 42) because that option carries IPv4 addresses only, but the KeaCluster supplied rotating `pool.ntp.org` hostnames (`DHCP4_CONFIG_LOAD_FAIL ... Failed to convert string to address '0.au.pool.ntp.org'`).
2. **kea-ctrl-agent/dhcp4** rejected the `/run/kea` unix socket path — kea 2.6.5 permits only `/var/run/kea` (exact-string check). Fixed in kea-operator v0.1.1 (`RunDir=/var/run/kea`).

- Remove `ntpServers` from the KeaCluster (not representable via DHCP option 42; add concrete NTP server IPs if ever needed).
- Bump kea-operator, kea, and kea-api images v0.1.0 -> v0.1.1 (socket-path fix).

kubeconform + pre-commit green.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #338
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 18:28:41 +10:00
unkinben b99682861b Point cert-manager DNS-01 at public recursive nameservers (#337)
unkin.net is split-horizon: the `_acme-challenge.unkin.net -> _acme-challenge.acme.unkin.net` delegation CNAME exists only in the public Google Cloud DNS view. cert-manager's CNAME following (`cnameStrategy: Follow`) resolves via in-cluster CoreDNS to the nodes' internal resolver, which serves an internal view of unkin.net lacking that CNAME; Follow therefore finds no delegation and still sends the rfc2136 UPDATE to zone unkin.net on bind-external (only authoritative for acme.unkin.net), returning NOTAUTH. Follow needs a public-view resolver for both the CNAME chase and the propagation self-check. TSIG is proven fine.

- Set `acmeDNS01.recursiveNameservers` to `8.8.8.8:53` and `1.1.1.1:53` with `acmeDNS01.recursiveNameserversOnly: true` in the cert-manager ControllerConfiguration so DNS-01 resolution and self-checks use the public DNS view.
- Keeps `cnameStrategy: Follow` on the ClusterIssuers (merged in #331); this PR gives that following a resolver that can see the delegation.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #337
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 18:14:27 +10:00
unkinben 2360534a38 artifactapi: serve plain HTTP without HTTPS redirect (#336)
## Why

Early-boot clients — anaconda/kickstart and yum in %post, PXE environments — need direct HTTP access to the artifactapi rpm repos. The current setup returns a 301 redirect from HTTP to HTTPS, which those minimal clients cannot follow (or downgrade insecurely), breaking rpm installs.

## Changes

- Attach the `api-route` HTTPRoute to the Gateway's `http` (port 80) listener alongside `https`, so `http://artifactapi.k8s.syd1.au.unkin.net/...` serves app content directly (200/40x from the app, no Location header).
- Remove the `http-redirect` HTTPRoute (RequestRedirect 301 `http`->`https`), which was the sole redirect mechanism — the traefik `web` entrypoint has no global `redirections`, so this is scoped strictly to artifactapi and does not affect other apps.
- Leave HTTPS unchanged.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #336
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-05 22:27:27 +10:00
unkinben 38743d58ab Rename terraform-ipam CI ServiceAccount -> terraform-infra (#335)
Follows the `terraform-ipam` -> `terraform-infra` repo rename. Renames the woodpecker ServiceAccount and its kustomization entry.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #335
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-05 19:40:34 +10:00
unkinben f31552e192 Deploy kea DHCP operator to dhcp-system (#333)
Replaces the isc-dhcpd PXE-boot VM with the kea-operator + an HA kea pair, GitOps-managed. Deploys on a new, unused anycast IP so nothing is cut over yet; the production cutover off the current dhcpd address (198.18.19.18) is a separate later task.

- Add `apps/base/dhcp-system`: namespace, kea-operator RBAC + Deployment (v0.1.0), VPA, and the 4 kea.unkin.net CRDs pulled from the operator repo at tag v0.1.0.
- Add CRs translating the legacy dhcpd config (source: puppet `roles/infra/dhcp/server.yaml`): KeaCluster `kea` (2 replicas, hot-standby HA, main.unkin.net, 1200/86400 leases, AU ntp pool); five KeaSubnets 198.18.13-17.0/24 with .200-.220 pools, gateways .254 except .17->.1, next-server 198.18.19.19; Legacy/UEFI-64 PXE client classes; KeaAPI.
- DHCP-advertised DNS points at the in-cluster bind-resolvers cluster (PureLB 198.18.200.7), not the legacy 198.18.19.15 forwarder.
- Pin the DHCP LoadBalancer Service to the free common-pool IP 198.18.200.10 via PureLB.
- KeaAPI bearer token is operator-generated (no plain Secret committed).
- Commit generated kea.unkin.net JSON schemas for kubeconform; register dhcp-system in the platform ApplicationSet + AppProject.

Client-class object names are lowercased (`legacy`/`uefi-64`) to satisfy RFC1123 since the operator renders the kea class name from metadata.name.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #333
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-03 00:18:05 +10:00
unkinben f296d0549a Add terraform-ipam CI ServiceAccount (#334)
ServiceAccount `terraform-ipam` in the `woodpecker` namespace for the terraform-ipam pipeline. The Vault k8s auth role `woodpecker_terraform_ipam` (terraform-vault PR) binds it. Mirrors the other terraform-* CI ServiceAccounts and is wired into the woodpecker kustomization.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #334
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-03 00:17:11 +10:00
unkinben f1c3b9617f Add agent-dns RBAC: static SA + ClusterRole + per-namespace RoleBindings (#332)
## Why
Vault's kubernetes secret engine will mint scoped tokens for a static \`agent-dns\` service account instead of generating cluster-wide RBAC, so agent DNS access is confined to exactly the bind namespaces. This is the GitOps half of the terraform-vault agent-dns role rework (PR unkin/terraform-vault#109). Ordering: this must sync before the Vault \`agent-dns\` creds are usable — Vault mints tokens for an SA that must already exist.

## How
- Add ServiceAccount \`agent-dns\` + ClusterRole \`agent-dns\` (definition only, no ClusterRoleBinding) in \`bind-system\`: full verbs on \`bind.unkin.net\` CRDs, get/list/watch pods/services/configmaps/events, get pods/log.
- Add RoleBinding \`agent-dns\` in each of \`bind-system\`, \`bind-internal\`, \`bind-external\`, \`externaldns\`, binding the SA to the ClusterRole in that namespace — confining all access (reads included) to those four namespaces.

Whitelist note: the platform AppProject already permits ClusterRole/ClusterRoleBinding and all four namespace destinations, so no project change is needed.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #332
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-02 21:49:37 +10:00
unkinben 4fc4aed358 Add cnameStrategy: Follow to ACME DNS-01 solvers (#331)
A live DNS-01 smoke test returned NOTAUTH because the solver walked _acme-challenge.unkin.net to zone unkin.net and sent the rfc2136 UPDATE there, but bind-external is only authoritative for acme.unkin.net; without cnameStrategy: Follow the solver does not chase the delegation CNAME.

- Set `cnameStrategy: Follow` on the `letsencrypt` and `letsencrypt-staging` ClusterIssuer DNS-01 solvers so cert-manager follows the `_acme-challenge.unkin.net -> _acme-challenge.acme.unkin.net` CNAME and updates the `acme.unkin.net` zone.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #331
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-02 19:01:32 +10:00
unkinben e5c84d0f74 Add kea-operator-ci ServiceAccount for Woodpecker CI (#330)
The new kea-operator repo's Woodpecker CI pipelines run under a dedicated Kubernetes ServiceAccount that must exist in the woodpecker namespace (cross-repo dependency; the .woodpecker/*.yaml steps set `serviceAccountName: kea-operator-ci`).

- Adds ServiceAccount `kea-operator-ci` in the `woodpecker` namespace
- Registers it in the woodpecker kustomization resources

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #330
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-02 19:01:02 +10:00
unkinben 239ea07d5c Import live vault-issuer ClusterIssuer into GitOps (#328)
## Why

The \`vault-issuer\` ClusterIssuer is applied out-of-band (Helm release \`cert-manager-clusterissuer\`) and is referenced by ~15 Gateways, but is not tracked in GitOps — so the live, load-bearing issuer is drift. This imports it so ArgoCD manages it. The committed spec matches the live object exactly (verified against \`kubectl get clusterissuer vault-issuer -o yaml\`), so adoption is a no-op.

## Changes

- Add \`apps/base/cert-manager/clusterissuer_vault-issuer.yaml\` capturing the live spec byte-faithfully: server \`https://vault.service.consul:8200\`, path \`pki_int/sign/servers_default\`, k8s auth mount \`/v1/auth/k8s/au/syd1\`, role \`cert_manager_issuer\`, serviceAccountRef \`cert-manager-vault-issuer\` (audience \`vault\`), caBundleSecretRef \`vault-ca-cert\`/\`ca.crt\`. Helm ownership labels/annotations kept so adoption produces zero diff.
- Register the manifest in the cert-manager base kustomization (inserted between \`clusterrolebinding.yaml\` and \`vmservicescrape.yaml\` to avoid the lines #327 touches).

## Depends on #327

ArgoCD can only adopt this resource once \`{group: cert-manager.io, kind: ClusterIssuer}\` is in the platform project \`clusterResourceWhitelist\`. That whitelist entry is added by #327, not here (to avoid a duplicate/conflicting change). **Merge #327 first.** There may be a small merge conflict with #327 in \`apps/base/cert-manager/kustomization.yaml\` (both append to the \`resources\` list); rebase on main after #327 merges.

## Note: SA name discrepancy (not fixed here — committing live spec unchanged)

The live issuer authenticates as SA **\`cert-manager-vault-issuer\`**, but the repo scaffolding \`serviceaccount.yaml\` creates SA **\`vault-issuer\`**. Both SAs exist live in \`cert-manager\` (192d and 136d). The issuer uses \`cert-manager-vault-issuer\`, so this PR commits that name (live truth). The repo-managed \`vault-issuer\` SA appears unused by this issuer — worth a follow-up to reconcile which SA is canonical, but out of scope for a zero-change import.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #328
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-02 18:00:00 +10:00
unkinben 131b4e1695 Add bind-external namespace for externally-reachable zones (#329)
We self-delegate `_acme-challenge.unkin.net` into an `acme.unkin.net` zone we serve ourselves, so cert-manager can solve Let's Encrypt DNS-01 over RFC2136/TSIG. That needs a publicly-reachable authoritative BIND, separate from the internal estate.

- Add app `bind-external` (base + au-syd1 overlay); register it in the platform ApplicationSet and AppProject destinations (bind-operator already watches all namespaces).
- Add BindCluster `bind-external`: authoritative-only, recursion off, no forwarding, transfers denied except the keyed catalog/zone AXFR; 2 replicas; primaryService is a dmz-pinned PureLB LoadBalancer at `198.18.199.53`.
- Add BindZone `acme.unkin.net` (primary, dynamicUpdate) and BindTSIGKey `certmanager` (hmac-sha256), whose Secret `certmanager-tsig` reflects into the `cert-manager` namespace for the rfc2136 solver.

Pairs with argocd-apps #327 (the ClusterIssuers) and a one-time Google Cloud DNS delegation + NAT of the public IP :53 to `198.18.199.53`.

---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #329
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-02 17:53:34 +10:00
unkinben c0c75d1bbb Add Let's Encrypt RFC2136/TSIG ClusterIssuers (#327)
Publicly-trusted wildcard certs via Let's Encrypt DNS-01, solved over RFC2136/TSIG against our own BIND. A one-time CNAME self-delegates `_acme-challenge.unkin.net` into the `acme.unkin.net` zone served by bind-external; cert-manager writes the challenge TXT there. No GCP/clouddns and no Vault secret involved. The existing `vault-issuer` (internal PKI) is untouched.

- Add ClusterIssuers `letsencrypt` (prod) and `letsencrypt-staging`, both using a dns01 rfc2136 solver: nameserver `198.18.199.53:53`, key `certmanager`, HMACSHA256, `tsigSecretSecretRef` -> reflected Secret `certmanager-tsig` key `secret`.
- Whitelist `cert-manager.io ClusterIssuer` in the platform AppProject.

Depends on #329 (bind-external: the acme.unkin.net zone, the certmanager TSIG key reflected into cert-manager, and the 198.18.199.53 nameserver) and on the one-time Google Cloud DNS delegation + NAT of the public IP :53 to 198.18.199.53. Earlier clouddns/Vault commits on this branch are reverted.

---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #327
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-02 17:47:39 +10:00
unkinben 8d70149467 Add Tier-1 per-app Vector transform pipelines (structured logs) (#318)
Why: the logging aggregator wrote every event through the generic catch-all shape. The Tier-1 survey picked six high-value log sources that warrant structured parsing into logs.raw columns/fields for real querying.

How:
- Two-stage routing in `aggregator.yaml`: `app_route` peels off the six Tier-1 streams by subject / VM source tag (mutually exclusive — no double-insert); everything else falls through `app_route._unmatched` to the unchanged generic k8s/vm catch-all.
- Six parse transforms emit the full `logs.raw` shape plus structured `.fields` (Map(String,String) — no DDL change): authentik (JSON), traefik (JSON access), vault audit (JSON), nginx access+error (regex), haproxy httplog (regex), glauth (JSON).
- Companion flip: traefik-system access logs to `format: json` (both overlays) so `traefik_parse` has structured input.
- 15 new `vector test` cases (routing + field extraction) in `aggregator-tests.yaml`; all green locally (vector 0.57).

Live now: authentik + traefik (k8s). Awaiting the puppet-side vector rollout (logs.vm.* with `.file`/`.SYSLOG_IDENTIFIER` tags per the documented convention): vault-file, nginx, haproxy, glauth — transforms are present and unit-tested so they light up automatically.

Note: geoip enrichment for nginx/traefik client IPs is a separate prerequisite — no enrichment table exists in the aggregator yet; these transforms extract `client_ip` ready for it.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #318
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-01 00:22:32 +10:00
unkinben 5f87d0c96d gitea: use git.k8s ROOT_URL for testing (#317)
Points the new gitea at its own k8s route so it can be exercised (login, browse, OIDC callback) before the git.unkin.net data cutover — ROOT_URL currently resolves to the live VM forge, which would break links on the k8s route. Flips back to git.unkin.net at cutover.

- set gitea DOMAIN/ROOT_URL/SSH_DOMAIN to git.k8s.syd1.au.unkin.net

Reviewed-on: #317
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-31 21:21:05 +10:00
unkinben d04940b1ea Pull in-estate service images from artifactapi docker-internal (#310)
Move the estate's own service/operator image pulls off the Gitea container registry (git.unkin.net/unkin) to the artifactapi local docker registry (docker-internal), ahead of the git.unkin.net forge migration which disables Gitea's container registry. The images were copied digest-for-digest into docker-internal and pulls verified before this repoint.

- repoint age-api, bind-operator, bind-tsig-api, cephrgw-operator, encapi, logarchiver, pdbmux image pulls to artifactapi.k8s.syd1.au.unkin.net/docker-internal

Deliberately not repointed here: artifactapi's own api/ui images (circular — it can't pull itself from itself), and the almalinux9-* base/CI images (huge, and their per-repo Woodpecker push targets move in a batched follow-up). Forge raw-CRD URLs and git clone sources are unaffected (those stay on the forge).

Reviewed-on: #310
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-31 20:07:16 +10:00
unkinben a04dcc2975 Add k8s Gitea deployment (migration target for git.unkin.net) (#309)
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md.

- add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute
- add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222
- register gitea in the platform ApplicationSet + AppProject
- add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback)

Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc.

Reviewed-on: #309
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-31 20:03:43 +10:00
unkinben 6a13ca758a cephrgw: recreate cnpg backup buckets on ec placement (step 3) (#316)
Final step of the ec migration: the old buckets were purged in #315, so the operator will now create fresh ones on the ec placement target. Restores the nine Bucket CRs with placementTarget: ec and retainOnDelete: true (purge disabled again for safety).

- re-add the nine cnpg backup Bucket resources on ec, retainOnDelete: true

Reviewed-on: #316
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-30 23:44:50 +10:00
unkinben 7daeb4af65 cephrgw: remove cnpg backup buckets (ec migration step 2) (#315)
Step 2 of the ec placement migration: with purge-on-delete now allowed (#313), removing the Bucket CRs makes the operator delete the underlying RGW buckets and their objects, freeing the names to be recreated on ec in step 3.

- remove the nine cnpg backup Bucket resources
- keep ObjectStoreUser and ScheduledBackup so the backup-s3 secrets and schedules survive

Reviewed-on: #315
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-30 23:35:55 +10:00
unkinben dfb495d771 Trust internal CA for Authentik SSO; canonical identity.unkin.net for NetBox (#314)
Authentik is canonical at https://identity.unkin.net, served by the internal
unkin.net CA. Grafana, LiteLLM and NetBox failed OIDC discovery because their
images don't trust that CA (x509: unknown authority); NetBox also still pointed
at the secondary admin host.

- grafana: mount the reflected vault-ca-cert; set generic_oauth `tls_client_ca`.
- litellm: `combine-certs` init builds public+internal CA bundle; `SSL_CERT_FILE`
  + `REQUESTS_CA_BUNDLE` point at it.
- netbox: flip OIDC issuer to identity.unkin.net; same combine bundle for
  python-social-auth (`requests`).
- docs: record the Rancher manual runtime step (issuer + CA in the auth config).

Validated: kustomize build + kubeconform + pre-commit.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #314
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-30 22:17:44 +10:00
unkinben 6e8a061b94 cephrgw: allow purge-on-delete for cnpg backup buckets (#313)
Step 1 of moving the CNPG backup buckets to ec placement: RGW can't move an existing bucket, so they must be dropped and recreated. This lets the operator actually delete the buckets (with their objects) when the CRs are removed in step 2.

- set retainOnDelete: false and purgeOnDelete: true on all nine cnpg backup Bucket CRs
- leave ObjectStoreUser/BucketAccess untouched so the backup-s3 secrets persist

Reviewed-on: #313
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-30 22:16:18 +10:00
unkinben 23c26e8cc2 cephrgw v0.4.0; move CNPG backup buckets to ec placement (#312)
cephrgw-operator v0.4.0 adds immutable placementTarget selection on Buckets; the nine CNPG backup buckets should live on the ec (4/1) placement instead of 3-replica. Existing buckets cannot change placement, so after this merges the buckets get deleted and recreated on ec and fresh base backups are triggered (day-old backups are accepted losses, per Ben).

- bump cephrgw-operator image and CRD ref to v0.4.0
- add placementTarget: ec to all nine cnpg backup Bucket CRs

Reviewed-on: #312
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-30 21:08:25 +10:00
unkinben 72c259a2a0 Fix nats-bootstrap: run from /tmp so the nats CLI works under readOnlyRootFS (#311)
## Why

Final-mile bringup: after #301/#306/#308 the auth chain was fixed and logs flowed, but the `nats-bootstrap` PostSync hook **failed** with:
```
nats: error: could not pick a Stream to operate on: ... could not load schema { ... }: stat .: permission denied
```
The nats CLI stats its **working directory** when loading response-validation schemas. Under the Job's `readOnlyRootFilesystem: true` + `runAsUser: 1000`, the nats-box image's default WORKDIR isn't accessible to uid 1000, so every `nats stream/consumer` call errored. (A throwaway pod using default securityContext worked, which is why manual stream creation succeeded.)

Consequence: the PostSync hook never completes → `logging-logging` stays **OutOfSync**. The `LOGS` stream + consumers persist in JetStream once created, so log flow is unaffected — but GitOps convergence is blocked and the hook would keep retrying.

## What

Set `workingDir: /tmp` on the bootstrap container (the writable emptyDir already mounted for `HOME`). The nats CLI can then stat/operate normally.

**Verified on the live cluster:** a nats-box pod with the Job's exact restrictive securityContext + `workingDir: /tmp` runs `nats stream info LOGS` cleanly (fails without it).

## Note (separate, pre-existing)

There is also a first-deploy ordering deadlock: the `nats-bootstrap` PostSync hook runs only after the Sync-phase resources are healthy, but the vector consumer Deployments can't become healthy until the hook creates the `LOGS` stream. On this deploy I broke the deadlock by creating the stream/consumers manually (idempotent with the Job); the stream now persists so it won't recur on normal re-syncs, but a fresh cluster / PVC loss would hit it again. A durable fix (sync-waves so bootstrap runs after NATS but before the consumers) is worth a follow-up — flagged, not included here to keep this fix minimal.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #311
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-30 00:11:16 +10:00
unkinben 9c10b9096a Replace vector-archiver with logarchiver (#308)
## Why

The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed.

## Changes

- Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring.
- Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch.
- Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`).
- Remove the vector-archiver Helm release, values and pipeline ConfigMap.

Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged).

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #308
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-29 21:07:20 +10:00
unkinben 96afbcf5e1 Fix NATS auth: wrap env-var passwords in << >> so the server expands them (#306)
## Why

After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient).

### Evidence
- Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped.
- The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected.
- **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**.

### Root cause
The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them.

## What

Wrap all three user passwords in `<< >>`:
```
password: << $NATS_ADMIN_PASSWORD >>      # (+ producer, consumer)
```
Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted).

This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password.

## Verified end-to-end
nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully.

## Expected recovery after merge + sync

1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords.
2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config.
3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes.
4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3.
5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises.

ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix.

## Validation
kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #306
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-29 19:28:27 +10:00
unkinben 7dddf8c5aa Fix logging deploy: enable Vector env interpolation; operator watches logging ns (#301)
## Why

PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here.

## Root cause 1 — Vector env-var interpolation is off by default in 0.57

`vector-aggregator` and `vector-archiver` crash-looped with `async_nats::connector: authorization violation`; `vector-agent` / `vector-vm-ingest` were "Running" but silently failing to publish.

Diagnosis (evidence):
- The NATS server config **does** expand `$NATS_*_PASSWORD` (a `nats` CLI login with the real secret value authenticated fine), and the ACL was **not** the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation).
- The failure reproduces locally: a hardcoded password connects; the **same value via `${NATS_CONSUMER_PASSWORD}` fails**. Configuring the server to expect the literal string `${NATS_CONSUMER_PASSWORD}` makes Vector connect — proving **Vector sends the literal, un-interpolated string**.
- `vector --help` shows `--dangerously-allow-env-var-interpolation` — in 0.57 `${VAR}` interpolation is **opt-in**. An unset-var test confirms interpolation is off (no "unknown env var" error).
- Verified fix: with `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` → **connects and authenticates**.

Every tier uses `${...}` for auth (`${NATS_*_PASSWORD}`, `${CLICKHOUSE_*}`), so the env var is added to **all four** vector deployments. (This slipped past CI because `vector test` never opens the NATS connection.)

## Root cause 2 — operator watches only its own namespace

`kubectl get chi -n logging` showed the `logs` CHI existed but with **empty status / no finalizer** — the operator never touched it, so the `logging-logging` Argo sync was stuck `Progressing` on *"waiting for healthy state of ClickHouseInstallation/logs"*, and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed).

Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states `watchNamespaces: []` (our value) makes the operator **watch only its own namespace** (`clickhouse-system`). The CHI is in `logging`.

Fix: `watchNamespaces: ["logging"]` → operator config `watch.namespaces.include: [logging]`.

## Changes

- `apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml`: add `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true`.
- `apps/overlays/au-syd1/clickhouse-system/values.yaml`: `watchNamespaces: ["logging"]`.

No NATS ACL change (the original narrow ACL is correct). No secret/base changes.

## Expected recovery after merge + sync

1. clickhouse-system syncs → operator config gains `logging` → operator restarts → reconciles the `logs` CHI → CHI pod comes up healthy.
2. `logging-logging` sync unblocks → PostSync hooks run → JetStream `LOGS` stream + `transform`/`archiver` consumers created; `logs.raw` table created.
3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3.
4. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` increases.

## Validation

kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders `watch.namespaces.include: [logging]`; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it).

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #301
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-29 00:11:33 +10:00
unkinben 57691ef1d5 certificates: restore the validly-signed intermediate in vault-ca-cert (#305)
CNPG WAL archiving to Ceph RGW fails with CERTIFICATE_VERIFY_FAILED on six clusters because the reflected vault-ca-cert bundle carries a corrupt intermediate: the genuinely-signed cert has a typo'd AIA URL (vault.servuce.consul), and the committed copy was text-edited at some point to fix the typo — flipping one byte of signed data and invalidating the signature (openssl verify: error 7 certificate signature failure). Only radosgw surfaces it because it serves a bare leaf, forcing clients to verify the stored intermediate against the root; services presenting their own intermediate never exercised the corrupt copy. terraform-k8s's copy is defunct per Ben — this file is the authoritative source.

- restore the original signed intermediate (one base64 character; sha256 E0:13:1B..., verified against the root, and the resulting bundle validates the live s3.ceph.unkin.net leaf)
- add an explicit allow-plain-secret marker mechanism to ci/validate-no-secrets.sh for public-data bootstrap secrets, and mark vault-ca-cert.yaml with it (a CA bundle is public and cannot be Vault-sourced since it establishes Vault trust)

After merge+sync the reflector propagates to all namespaces and barman's next retry (~1min) succeeds; base backups run on tonight's schedule. Follow-ups worth considering: re-issue the intermediate in Vault with a corrected AIA URL, and/or configure radosgw to serve its intermediate.

Reviewed-on: #305
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 23:14:14 +10:00
unkinben e0eeeb6b04 Use full postgres image on minimal CNPG clusters so WAL archiving works (#304)
## Why

The CNPG buckets are empty after PR #298. Live diagnosis found **two** independent
causes; this PR fixes one of them.

`litellm`, `puppet` and `paperclip` run the CNPG `17-minimal-trixie` image, which
omits the `barman-cloud` CLI. In-tree `barmanObjectStore` archiving shells out to
`barman-cloud-wal-archive`, so their archiver dies immediately:

```
ContinuousArchiving=False :: unexpected failure invoking barman-cloud-wal-archive:
exec: "barman-cloud-wal-archive": executable file not found in $PATH
```

(verified on the live `puppet-postgres` primary: `which barman-cloud-wal-archive` →
not found; on a `-system` cluster it resolves to `/usr/local/bin/barman-cloud-wal-archive`).

## How

Switch those three clusters from `17-minimal-trixie` to `17-system-trixie` — the
`-system` variant already used by the other six clusters, which bundles the
barman-cloud tools. Tag confirmed present upstream (ghcr manifest HTTP 200).

```
- imageName: ghcr.io/cloudnative-pg/postgresql:17-minimal-trixie
+ imageName: ghcr.io/cloudnative-pg/postgresql:17-system-trixie
```

CNPG applies this as a rolling image update (switchover, no data change).

## Not fixed here (separate, primary blocker)

The other six clusters (full image, barman present) fail with a **TLS trust**
error — the reflected `vault-ca-cert` bundle carries a **stale intermediate CA**,
so barman can't verify `s3.ceph.unkin.net`:

```
SSL: CERTIFICATE_VERIFY_FAILED ... certificate signature failure
```

That is a shared trust-anchor refresh (likely owned by terraform-k8s /
`config/certificates/secret.yaml`, `managed-by: terragrunt`), handled separately —
it also gates litellm/puppet once they have barman. See the investigation report.

## Validation

- `kustomize build --enable-helm` + `kubeconform` pass on `litellm`, `puppet`
  overlays and the `paperclip` base (paperclip has no overlay yet).
- `pre-commit run` passes on all changed files.

## Follow-ups

- Longer term, the Barman Cloud Plugin (sidecar) would let minimal images keep
  their size while still archiving — track with the plugin migration.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #304
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 23:11:30 +10:00
unkinben 7c1cbef722 Trust internal unkin.net CA for ArgoCD OIDC egress (#303)
## Why

ArgoCD SSO fails with `failed to query provider "https://identity.unkin.net/application/o/argocd/": tls: failed to verify certificate: x509: certificate signed by unknown authority`. argocd-server does OIDC discovery to identity.unkin.net over TLS served by the internal `unkin.net` CA. Unlike argocd-repo-server (which mounts `vault-ca-cert`), argocd-server has no internal CA in its trust store and no `rootCA` in `oidc.config`, so it never trusted the issuer.

## Change

- argocd-cm `oidc.config`: add `rootCA` (inline PEM) = the internal `unkin.net` root CA. argocd-server hot-reloads argocd-cm, so no rollout restart is required.

## Why the root, not the cluster vault-ca-cert bundle

The `unkin.net Intermediate Authority` was recently **re-keyed** (same serial, new key: bundle SHA1 `C4:48:78…` vs served `F1:DD:34…`). The cluster `vault-ca-cert` bundle still carries the **stale** intermediate and fails `openssl verify` against the currently-served identity cert. identity.unkin.net presents its current intermediate in the handshake, so anchoring on the long-lived, stable `unkin.net` root (valid to 2034, matches the host trust anchor) is both correct and rotation-proof. Verified: `openssl verify -CAfile <root> -untrusted <served-intermediate> <served-leaf>` = OK; the embedded PEM round-trips through the YAML patch and validates the served leaf.

## Verify after merge

argocd-server picks up argocd-cm live; retest SSO login. (Separately, the cluster `vault-ca-cert` reflected secret carries a stale intermediate and should be refreshed, but that is out of scope here.)

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #303
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 22:17:32 +10:00
unkinben d1a7646e09 Bump Woodpecker CI to v3.16.0 (#297)
## Why

Upgrade the Woodpecker CI instance from v3.14.1 to v3.16.0 to pick up upstream fixes and the k8s-backend privilege-escalation hardening (GHSA-qf34-295c-26v8). Server and agent must move together.

The chart is pulled at build time (`kustomize build --enable-helm`); the image tag defaults to the chart `appVersion`, so bumping the chart moves both images. Chart 3.6.5 -> appVersion 3.16.0 (verified against upstream `helm/charts/woodpecker/Chart.yaml`); subchart deps are unchanged from 3.6.3 (server 3.0.1, agent 2.0.1), so there is no values-schema migration.

## What

- Bump the woodpecker helmChart from `3.6.3` (appVersion 3.14.1) to `3.6.5` (appVersion 3.16.0) in the au-syd1 overlay. Server + agent both render as `v3.16.0`.
- Set `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true"` on the agent.

## CRITICAL: k8s backend serviceAccountName gating (required change)

v3.16.0 (PR #6792, GHSA-qf34-295c-26v8) gates step-level `serviceAccountName` behind a new agent flag `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP`, **default `false`**. When disabled, any `backend_options.kubernetes.serviceAccountName` set by a pipeline is **silently ignored** and the namespace `default` SA is used instead.

Every terraform pipeline in the estate sets `backend_options.kubernetes.serviceAccountName` (e.g. `terraform-git`, `terraform-vault`, `terraform-artifactapi`) and relies on that SA for Vault k8s auth / Consul state. Without this flag those jobs would run as `default` and lose their Vault identity. This PR sets the flag to `true` to preserve current behaviour. No other newly-gated backend_options keys (pod labels/annotations from step, native secrets) are used by the estate.

## Migration / rollback

- **DB migration:** Woodpecker auto-migrates the schema (xorm) on server start; migrations are forward-only and NOT reversible. The 3.14 -> 3.16 changelogs do not call out a data-destructive migration, but a **DB backup (CNPG cluster `woodpecker`) should be taken before merge**.
- **In-flight pipelines:** merging rolls the server StatefulSet and agents; any running pipelines are interrupted and will need re-running.
- **Rollback:** re-pin chart `3.6.5` -> `3.6.3` reverts the images to v3.14.1, but because migrations are one-way, a clean rollback requires **restoring the CNPG DB from the pre-merge backup**, not just pinning the old image.

## Validation

- `kustomize build --enable-helm apps/overlays/au-syd1/woodpecker` renders `woodpecker-server:v3.16.0` and `woodpecker-agent:v3.16.0`; agent carries the new env var.
- `kubeconform` (k8s 1.33.7): 24/24 resources valid.
- pre-commit (yamllint + checks): all pass.

## Follow-up (not in this PR)

Woodpecker images are pulled from `docker.io` / `ghcr.io` directly, not the artifactapi proxy. Proxying them via artifactapi is a possible follow-up but out of scope for this version bump.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #297
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 21:48:26 +10:00
unkinben f45cb6989f Add ClickHouse + Vector + NATS JetStream centralized logging (with S3 raw archive) (#296)
## Why

Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists.

## Topology

`edge (publishers) → JetStream → consumers → sinks`

- **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets.
- **Edge publishers (thin)** — no parsing, just a routing subject:
  - `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`.
  - `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.)
- **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now).
- **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`.
- **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job.

## Streams / consumers / auth

- Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment.
- **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret.

## S3 / retention

`ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here.

## Replay runbook

- **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time.
- **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window.

## Validation

- `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**.
- `pre-commit` (yamllint, check-json, no-plain-secrets) clean.
- **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs.
- **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets).

## Known upstream caveat

Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook.

## Prerequisites (manual, one-time)

```
# ClickHouse
PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1)
vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \
  username=vector password="$PW" password_sha256_hex="$HASH"
# NATS
vault kv put kv/kubernetes/namespace/logging/default/nats-auth \
  admin_password=$(openssl rand -base64 24) \
  producer_password=$(openssl rand -base64 24) \
  consumer_password=$(openssl rand -base64 24)
```
No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR.

## Open decisions (defaults chosen, flag to change)

- **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.**
- **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side).
- **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer.
- **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA.
- **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs).
- **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

---

## Update: images via artifactapi, DHI, stateless transform tier

**Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull.

### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`)

| Image | Upstream | artifactapi path | DHI? |
|---|---|---|---|
| clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job |
| altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI |
| altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI |
| bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI |
| nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats |
| natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI |
| natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI |
| timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened |
| timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step |

**DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace.

### Transform tier is now a stateless Deployment

Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**.

**Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.)

---

## Update: 7d retention, tunable limits ConfigMap, honest sizing

- **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict.

- **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3.

- **Honest 7d sizing (stated assumption — please sanity-check against real volume):**
  - Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica.
  - Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica.
  - **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.**
  - ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob.

- Replay window in the runbook is now **7d** (beyond that → the S3 archive).

---

## Update: retention cut to 3 days (both stores), PVCs shrunk

Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design.

| Store | Retention | Byte cap | PVC/node | Replicas | Total |
|---|---|---|---|---|---|
| NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) |
| ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) |

**NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC.
**ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table.

The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL.

⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit).

Reviewed-on: #296
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 19:54:27 +10:00
unkinben e6f2cbc363 authentik: raise CNPG postgres memory to stop replica OOMKills (#300)
Authentik's intermittent API 500s (which failed two terraform-authentik CI runs today) were traced to its CNPG postgres replicas being OOMKilled: 512Mi limits leave no headroom over shared_buffers 128MB + max_connections 200, both OOM events matched the 500 bursts to the second, and the session-pinned RO pooler turns each replica death into a batch of severed read connections. The primary is at 84% of its limit and is next.

- raise the authentik CNPG memory limit from 512Mi to 1Gi and request from 256Mi to 512Mi

Follow-up candidates (not in this PR): RO pooler poolMode session→transaction to shrink the blast radius of a replica loss; revisit max_connections/shared_buffers sizing.

Reviewed-on: #300
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 18:16:31 +10:00
unkinben 2ee4c9518c Deploy NetBox (CNPG Postgres, Valkey, Authentik SSO) (#299)
## Why
NetBox becomes the source of truth for host/interface/IPAM data as part of retiring Cobbler. The ENC role already moved to encapi; a kickstart-generation microservice that consumes NetBox comes later and is intentionally **out of scope** here.

## Change
Adds the `netbox` app (namespace `netbox`, platform project) using the netbox-community helm chart via the OCI helm-through-kustomize pattern, plus estate-native data stores:

- **NetBox** — chart `8.3.40` (appVersion `v4.6.5`), `oci://ghcr.io/netbox-community/netbox-chart`. 2 web replicas, 1 worker; bundled Postgres/Valkey subcharts disabled. Media on RWX CephFS so replicas share uploads. Chart `helm.sh/hook: test` Pod dropped via a kustomize delete patch (we deploy through ArgoCD, not `helm test`).
- **CNPG Postgres** — `netbox-postgres`, 2 instances, PG 18, `cephrbd-fast-delete`. Standard per-cluster S3 backup: `barmanObjectStore -> s3://cnpg-netbox`, cephrgw `ObjectStoreUser` + `Bucket`, nightly `ScheduledBackup` at **03:40** (`0 40 3 * * *` — next free slot after grafana's 03:20), 30d retention. A pgbouncer `Pooler` (session mode) fronts it; NetBox connects via `netbox-postgres-pooler-rw`.
- **Valkey** — standalone Deployment (`valkey/valkey:8-alpine`), AOF-persistent PVC on `cephrbd-fast-delete`. One instance: DB 0 = RQ task queue, DB 1 = cache. No auth (in-cluster, namespace-isolated). Chosen over the bundled Bitnami subchart to keep image control in-estate and avoid Bitnami's legacy-image churn; mirrors the litellm standalone-cache pattern.
- **Ingress** — `Gateway` + `HTTPRoute` at `netbox.k8s.syd1.au.unkin.net` (`traefik-internal`, `vault-issuer` cert into `netbox-tls`, external-dns to the internal VIP), HTTP->HTTPS 301.
- **Secrets** — all via VSO `VaultStaticSecret` (`postgres-credentials`, `netbox-secret-key`, `netbox-superuser`, `oauth-credentials`); no plain Secrets committed. The shared `default` k8s-auth role already binds `*` namespaces with a namespace-templated KV policy, so **no terraform-vault change is needed**.
- **Authentik OIDC SSO** — `remoteAuth` wires `REMOTE_AUTH_ENABLED` + the `OpenIdConnectAuth` backend via chart values; `SOCIAL_AUTH_OIDC_*` via `extraConfig` (the chart's config loader globs `/run/config/extra/*/*.yaml`). The client secret is injected as a YAML fragment mounted from the Vault-synced `oauth-credentials` secret. New users auto-provision on first login.
- Registers `netbox` in the platform ApplicationSet and AppProject (destination namespace + chart sourceRepo).

## Image table (source -> artifactapi -> DHI decision)
All images flow through the estate's containerd registry mirrors; the allowlist patterns gate them. DHI (Docker Hardened Images) require authenticated pulls from a Docker Hub `dhi/` org and are **not** reachable through the anonymous mirror, so upstream official is used throughout.

| Image | Upstream | Mirror / allowlist | Allowlisted? | DHI decision |
|---|---|---|---|---|
| NetBox app/worker/housekeeping | `ghcr.io/netbox-community/netbox:v4.6.5` | ghcr remote, `^netbox-community/` | **added in terraform-artifactapi #17** | No DHI published; upstream official |
| CNPG Postgres | `ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie` | ghcr, `^cloudnative-pg/` | already | estate-standard CNPG image |
| Valkey | `docker.io/valkey/valkey:8-alpine` | dockerhub, `^valkey/valkey` | already | DHI not anon-pullable; upstream official |
| NetBox init (perms) | `docker.io/busybox:1.38.0` | dockerhub, `^library/busybox` | already | upstream official |
| Worker wait-for-backend | `docker.io/rancher/kubectl:v1.36.2` | dockerhub, `^rancher/` | already | upstream official |

## Cross-repo PRs (merge order)
1. **terraform-artifactapi #17** (`^netbox-community/` ghcr allowlist) — **merge before** this PR so the NetBox image pulls on first sync.
2. **terraform-authentik #11** (OIDC provider/application) — independent; SSO works once applied + the secret is seeded.
3. This PR.

> Note: the CNPG S3-backup stanza pattern here is identical to PR #298 (already in main); it merges cleanly regardless of ordering.

## One-time Vault seeds (before/at first sync)
```
# App DB user (CNPG bootstrap + NetBox both consume this)
vault kv put kv/kubernetes/namespace/netbox/default/postgres-credentials \
  username=netbox password="$(openssl rand -base64 30)"

# Django SECRET_KEY
vault kv put kv/kubernetes/namespace/netbox/default/netbox-secret-key \
  secret_key="$(python3 -c 'import secrets;print(secrets.token_urlsafe(60))')"

# Bootstrap superuser
vault kv put kv/kubernetes/namespace/netbox/default/netbox-superuser \
  username=admin email=admin@unkin.net \
  password="$(openssl rand -base64 24)" api_token="$(openssl rand -hex 20)"

# OIDC client secret — ONE value stored two ways (raw for Authentik, YAML for NetBox)
CS="$(openssl rand -base64 30)"
vault kv put kv/kubernetes/namespace/netbox/default/oauth-credentials \
  client_secret="$CS" oidc.yaml="SOCIAL_AUTH_OIDC_SECRET: \"$CS\""
```
Seed `oauth-credentials` **before** applying terraform-authentik #11 (that apply reads `client_secret`).

## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/netbox` — clean.
- `kubeconform` (CI args, k8s 1.33.7): **27/27 valid, 0 invalid**.
- `pre-commit` (yamllint + no-plain-secrets) on all changed files — pass.

## Out of scope
The PXE/kickstart microservice that will consume NetBox — not scaffolded here.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #299
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 18:12:25 +10:00
unkinben 3c2bdf307a Add S3 backups to all CNPG Postgres clusters (#298)
## Why

None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A
lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable
data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip
and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for
every cluster, plus code-forward restore docs.

## What

- **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous
  WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2,
  30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected
  `vault-ca-cert` (`endpointCA`).
- **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator
  provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is
  hardcoded**) and a staggered nightly `ScheduledBackup`.
- **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can
  validate the new CRs.
- **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`).

## Design decisions (answers to the open questions)

**One bucket for all, or per-database?** → **Per-database (one bucket + owner user per
cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only
*within the same namespace*), and CNPG reads its S3 credential Secret from its *own*
namespace. A single shared bucket would require either cross-namespace bucket refs
(unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator
mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the
simplest correct topology and needs zero manual seeding. Each user owns exactly one
bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess`
grant needed.

**Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart
`cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree
`barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not
deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the
in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`).

## Schedule / retention (defaults — Ben to adjust)

| App | Cluster | Bucket | Nightly base backup |
| --- | --- | --- | --- |
| authentik | postgres | cnpg-authentik | 01:00 |
| litellm | litellm-postgres | cnpg-litellm | 01:20 |
| artifactapi | postgres | cnpg-artifactapi | 01:40 |
| woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 |
| puppet | puppet-postgres | cnpg-puppet | 02:20 |
| encapi | postgres | cnpg-encapi | 02:40 |
| paperclip | paperclip-postgres | cnpg-paperclip | 03:00 |
| grafana | postgres | cnpg-grafana | 03:20 |

Retention is **30d** across the board — flagged as a default to tune per cluster.
Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once.

## Validation

- `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph
  schemas) pass on all 8 affected overlays (paperclip validated at base — it has no
  overlay yet). ceph CRs resolve their schemas (`Skipped: 0`).
- `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.).
- Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated
  `cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) —
  pre-existing, reproduces on `origin/main`, unrelated to this change.

## Notes / caveats

- No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging
  PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json`
  files, which merge cleanly whichever lands first.
- Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The
  only prerequisite is the operator being healthy (it is, in `cephrgw-system`).

## Follow-ups

- Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs).
- Tune per-cluster retention / schedule if the defaults don't fit.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #298
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-27 23:57:08 +10:00
unkinben b725bf7dcf Add PTR to external-dns managed record types (#295)
Follow-up to #294: --create-ptr validates that PTR is in --managed-record-types, and setting that flag replaces the default list, so A/AAAA/CNAME are re-stated alongside PTR (verified against the v0.21.0 binary's --help: default A,AAAA,CNAME). Pod is crash-looping on config validation until this merges.

- Adds --managed-record-types=A,AAAA,CNAME,PTR to the external-dns args

MERGE ASAP.

Reviewed-on: #295
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-26 00:19:10 +10:00
unkinben faa256c764 Fix external-dns PTR flag (--create-ptr) (#294)
external-dns v0.21.0 rejects --rfc2136-create-ptr (my error in #292) and the pod is crash-looping on flag parsing. The correct flag in this version is the generic --create-ptr, which synthesizes PTRs for A records whose reverse zone is in the domain filter (200.18.198.in-addr.arpa already is).

- Replaces --rfc2136-create-ptr with --create-ptr

Record reconciliation is stalled until this merges (serving unaffected — bind answers normally). MERGE ASAP.

Reviewed-on: #294
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-26 00:15:33 +10:00
unkinben eff97b8d03 Enable PTR synthesis in external-dns (#292)
Post-cutover verification (argocd-apps#288) found the reverse zone 200.18.198.in-addr.arpa empty: external-dns only writes PTRs when --rfc2136-create-ptr is set, and nothing else feeds that zone. Reverse resolution for the k8s LB range was already absent for clients pre-migration (no forwarder existed), so this completes the reverse path rather than fixing a regression.

- Adds --rfc2136-create-ptr to the external-dns rfc2136 args

Verification after merge: dig -x 198.18.200.2 @198.18.200.7 returns puppetca.k8s.syd1.au.unkin.net (allow a reconcile cycle + negative-cache expiry).

Reviewed-on: #292
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-26 00:05:54 +10:00
unkinben aff06a633a Roll bind-operator to v0.2.6 (loop-free TSIG-keyed NOTIFY) (#293)
Deploy bind-operator v0.2.6 (bind-operator#15): NOTIFYs are now TSIG-signed via the catalog transfer key and secondaries accept by key, with no pod IPs in restart-scoped config — a regression test asserts the config-hash is invariant under pod IP churn, making the v0.2.5 roll-loop class impossible. Restores seconds-fast dynamic-zone propagation on bind-externaldns and bind-authoritative.

- Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.6 (confirmed in registry)
- Bumps the CRD install pin to the v0.2.6 tag

Expect exactly ONE settling roll of the bind statefulsets when the new config lands, then stability.

Reviewed-on: #293
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 23:53:55 +10:00
unkinben 0687c097d6 Revert bind-operator to v0.2.4 (v0.2.5 rolling-restart loop) (#291)
v0.2.5 renders the primary POD IP into the options-scope allow-notify. Options changes are restart-scoped (config-hash annotation), and every roll gives the primary a new pod IP, so the operator re-renders and rolls all bind clusters in an endless loop (externaldns, authoritative, resolvers all cycling ~45s pods right now).

- Reverts the operator image and CRD pin to v0.2.4

A v0.2.6 will re-do the NOTIFY fix loop-free (allow-notify via zone-scope/rndc-applied config or a TSIG-keyed notify instead of pod-IP-in-options). MERGE ASAP to stabilize DNS.

Reviewed-on: #291
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 23:12:38 +10:00
unkinben 7a889ca326 Flip resolver forwarding for k8s zones to the in-cluster bind (#288)
## Why

Step 3 of 3 in the external-dns → in-cluster bind migration: the client-visible cutover. The `openforwarder` resolvers currently forward `k8s.syd1.au.unkin.net` to the legacy VM anycast `198.18.19.20` (a temporary measure — commit 7ee5dfb) and have NO forwarder at all for the reverse zone `200.18.198.in-addr.arpa`. Once external-dns publishes to the in-cluster `bind-externaldns` (PR 2), resolvers must read from it.

## Changes

- Repoints the `fwd-k8s-syd1-au-unkin-net` forwarder from `198.18.19.20` (legacy VM) to `198.18.200.8` (in-cluster `bind-externaldns` VIP).
- Adds `fwd-200-18-198-in-addr-arpa` forwarding `200.18.198.in-addr.arpa` → `198.18.200.8`, closing the reverse-zone gap so PTR lookups for the k8s LB range keep resolving after cutover. Modeled exactly on the existing forward-zone entries.
- Refreshes the header comment to describe the in-cluster upstream.

`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean; both zones render with forwarder `198.18.200.8` and there is no residual `198.18.19.20`.

## Merge gate

- PR 2 (`benvin/externaldns-incluster`) merged, AND
- record parity confirmed between legacy and in-cluster for the forward zone. Spot-check (repeat for each name):
  ```
  for n in puppetca puppet puppetdb encapi pdbmux artifactapi consul; do
    echo "$n:"
    dig +short @198.18.19.20 A $n.k8s.syd1.au.unkin.net
    dig +short @198.18.200.8  A $n.k8s.syd1.au.unkin.net
  done
  # plus 2-3 PTRs in the reverse zone:
  dig +short @198.18.19.20 -x 198.18.200.8
  dig +short @198.18.200.8  -x 198.18.200.8
  ```
  A/PTR answers from `198.18.200.8` must match those from `198.18.19.20` before merging.

## Verification (after merge)

```
dig +short @198.18.200.7 A puppet.k8s.syd1.au.unkin.net   # resolvers VIP
dig +short @198.18.200.7 -x 198.18.200.8                  # reverse via resolvers
```
Resolution through the `bind-resolvers` VIP should now answer for both the forward and reverse k8s zones.

## Rollback

Revert this PR — the `fwd-k8s-syd1-au-unkin-net` forwarder returns to `198.18.19.20` and the reverse forward is removed. The legacy VM is untouched and remains authoritative until decommission.

Reviewed-on: #288
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 23:08:22 +10:00
unkinben 9452473bf6 Roll cephrgw-operator to v0.3.1 (CRD staleness warning) (#290)
## Why

cephrgw-operator **v0.3.1** logs a startup WARNING when its installed CRDs are missing or older than the operator (operator repo #6, merged) — added precisely because the CRD tag drifted behind the image before. The new startup check reads the `CustomResourceDefinition` objects, so it needs a small RBAC grant.

## Changes

- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.1`
- bump the CRD `install.yaml` tag → `v0.3.1` (keep CRDs in step with the image — the invariant the v0.3.1 warning enforces)
- add `apiextensions.k8s.io/customresourcedefinitions: [get, list]` to the operator ClusterRole so the startup check is not RBAC-denied

Validated with `kustomize build` on the au-syd1 overlay; the v0.3.1 CRD URL resolves. Supersedes nothing outstanding (the earlier CRD-tag PR #286 to v0.3.0 already merged).

https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #290
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 23:03:36 +10:00
unkinben 9abb82afb4 Roll bind-operator to v0.2.5 (intra-cluster NOTIFY fix) (#289)
Deploy bind-operator v0.2.5 (bind-operator#14): secondaries now carry an explicit allow-notify for the primary pod IP, so dynamic-zone updates propagate in seconds instead of the ~1h SOA refresh. Unblocks the external-dns migration parity gate (argocd-apps#288) and speeds up the dns-updater zones on bind-authoritative.

- Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.5 (confirmed in registry)
- Bumps the CRD install pin to the v0.2.5 tag

On sync the operator re-renders cluster ConfigMaps; the config-hash change rolls the bind secondaries, which then accept the primary's NOTIFYs.

Reviewed-on: #289
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 23:00:12 +10:00
unkinben 8dc65779a5 Repoint external-dns at the in-cluster bind-externaldns primary (#287)
## Why

Step 2 of 3 in the external-dns → in-cluster bind migration. external-dns currently pushes RFC2136 updates to the legacy VM `ausyd1nxvm2127.main.unkin.net`. This repoints it at the in-cluster `bind-externaldns` primary so the in-cluster zone becomes the live source of truth for `k8s.syd1.au.unkin.net` + `200.18.198.in-addr.arpa`.

## Changes

- Points `--rfc2136-host` at `bind-externaldns-primary.bind-internal.svc.cluster.local` (verified live: ClusterIP Service `bind-externaldns-primary` exists in `bind-internal`).
- Reads TSIG `secret` + `algorithm` from Secret `externaldns-key-tsig` (reflected by PR 1) instead of the Vault-backed `externaldns-tsig`.
- Keeps port, zones, keyname, and `txtOwnerId: k8s` unchanged.
- Leaves the old Vault manifests (`apps/base/externaldns/{vaultauth,vaultstaticsecret}.yaml`) in place as the rollback path; their removal is a later cleanup PR.

## Merge gate

- PR 1 (`benvin/externaldns-tsig-reflect`) merged, AND
- `kubectl -n externaldns get secret externaldns-key-tsig` returns keys `secret` + `algorithm`.

## TXT-registry note

`policy: sync` + `registry: txt`: on first reconcile against the (currently empty) in-cluster zone, external-dns re-creates all managed A/CNAME records and their ownership TXTs from scratch. This is expected and populates the zone.

## Verification (after merge)

```
kubectl -n externaldns logs deploy/externaldns --tail=100 | grep -Ei 'rfc2136|BADKEY|NOTAUTH|added|update'
dig +short @198.18.200.8 A puppet.k8s.syd1.au.unkin.net
```
Logs should show updates to `bind-externaldns-primary...` with NO `BADKEY`/`NOTAUTH`; sampled A records should start resolving against the in-cluster server (198.18.200.8) as the sync populates the zone.

## Rollback

Revert this PR (host + secret name back to `ausyd1nxvm2127.main.unkin.net` / `externaldns-tsig`). The legacy VM is untouched and still authoritative.

Reviewed-on: #287
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 22:41:20 +10:00
unkinben 8eeca2d1fb Reflect externaldns TSIG key into the externaldns namespace (#285)
## Why

Step 1 of 3 in the external-dns → in-cluster bind migration. For external-dns to send RFC2136 updates to the in-cluster `bind-externaldns` primary, it must present the exact TSIG key the primary's `allow-update` accepts. The bind-operator generates that key material into Secret `externaldns-key-tsig` in `bind-internal`; reflecting it into the `externaldns` namespace removes the manual eyaml→Vault key sync and guarantees key parity.

## Changes

- Adds `spec.secretTemplate.annotations` to BindTSIGKey `externaldns-key` with the emberstack reflector hints: `reflection-allowed`, `reflection-allowed-namespaces: externaldns`, `reflection-auto-enabled`, `reflection-auto-namespaces: externaldns`.
- Regenerates `schemas/bind.unkin.net/bindtsigkey_v1alpha1.json` from the live CRD (deployed bind-operator v0.2.4 already exposes `secretTemplate` — the WIP branch's ">= v0.3.0" claim is stale) to add the `secretTemplate` property. Schema output is byte-identical to running `ci/generate-schemas.sh`.

`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean and renders the annotations onto the BindTSIGKey.

## Verification (after merge)

```
kubectl -n externaldns get secret externaldns-key-tsig \
  -o jsonpath='{.data.secret} {.data.algorithm}{"\n"}'
```
Both `secret` and `algorithm` keys must be present (reflector mirrored the source Secret from bind-internal).

## Rollback

Revert this PR. The source Secret in bind-internal is unaffected; only the reflected mirror in `externaldns` is removed.

Merge order: this is PR 1/3. PR 2 (repoint external-dns) must not merge until the reflected secret is verified.

Reviewed-on: #285
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 22:37:39 +10:00
unkinben e4afdf59ba Bump cephrgw-operator CRDs to v0.3.0 (#286)
## Why

The `cephrgw-system` kustomization pinned the CRD source to **`raw/tag/v0.1.0/config/crd/install.yaml`**, so the in-cluster CRDs never gained the fields added since v0.1.0 — v0.2.0's fine-grained BucketAccess policy fields (`paths`/`actions`/`conditions`/`rawStatements`) and v0.3.0's adoption fields (`retainOnDelete`, `managePolicy`, `status.adopted`). The running operator is v0.3.0, so applying those specs fails with `strict decoding error: unknown field`. The image bumps (#273, #279) should have moved this tag too.

## Changes

- point the CRD `install.yaml` at `raw/tag/v0.3.0`

The tag must track the operator image tag on future bumps. Verified the v0.3.0 URL serves the new schema and `kustomize build` renders all 3 CRDs.

https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #286
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 22:34:58 +10:00
unkinben 1d87c42fc1 Add advise-mode VPA for every workload (#283)
## Why

Turns on right-sizing telemetry for the whole estate. Adds a `VerticalPodAutoscaler` with `updateMode: "Off"` (recommendation-only, advise mode) for every Deployment and StatefulSet in `apps/base`. Off mode never evicts or mutates pods, so this is purely observational: the VPA recommender (added in the vpa-system PR) publishes suggested requests/limits in each VPA's status, and nothing acts on them until someone deliberately flips a mode.

## Changes

- Add one `vpa.yaml` per app under `apps/base/<app>/` containing a `<workload>-vpa` VerticalPodAutoscaler for each workload, and register it in that app's `kustomization.yaml`.
- Coverage: 17 workloads across 11 apps.
  - age-api (age-api), artifactapi (api, redis, ui), authentik (redis), bind-system (bind-operator), cephrgw-system (cephrgw-operator), encapi (encapi), kanidm (kanidm StatefulSet), litellm (litellm, redis), paperclip (paperclip), pdbmux (pdbmux), puppet (puppetboard, puppetdb, puppetserver-compiler, puppetserver-master).

## Skipped (intentionally)

- **CNPG `Cluster` objects** (artifactapi, authentik, encapi, grafana, litellm, paperclip, puppet, woodpecker) — Postgres is managed by CloudNativePG, not a VPA target.
- **CronJobs** (puppet g10k/generate-types, reposync x4) — not VPA-able.

## HPA / VPA caveat

api, ui (artifactapi), litellm, and all four puppet deployments also carry an HPA. With `updateMode: "Off"` there is no conflict today (VPA only recommends). VPA objects targeting these carry an inline comment: do **not** flip to `Auto`/`Initial` while the HPA still autoscales on CPU/memory, or the two controllers will fight over the same resource. Move the HPA to a custom/non-resource metric first.

## Verification

- `kubectl kustomize` over every touched overlay: 9/11 overlays PASS rendering all their VPAs; 2 failures are pre-existing and unrelated to this change — `authentik` fails on a local helm-tooling flag error identically on origin/main, and `paperclip` has no `apps/overlays/au-syd1/paperclip` directory yet. Both apps' base kustomizations build clean and render their VPA.

## Merge gate

- **Requires the vpa-system CRDs PR (argocd-apps #281) to merge first.** These manifests use `autoscaling.k8s.io/v1 VerticalPodAutoscaler`; without the CRDs installed, ArgoCD sync fails on an unknown kind.

Reviewed-on: #283
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 18:30:23 +10:00
unkinben 4d877531b0 Add vpa-system: VPA CRDs + recommender (advise mode) (#281)
## Why

Rolls out the Vertical Pod Autoscaler control plane so the estate can gather right-sizing recommendations for every workload (advise mode, follow-up PR adds the per-workload VPA objects). Deploys the **recommender only**: advise mode never mutates pods, so the updater and admission-controller (and its mutating webhook) are intentionally omitted — fewer moving parts, no webhook in the admission path.

## Changes

- Add `apps/base/vpa-system/`: namespace, VPA CRDs (verticalpodautoscalers + verticalpodautoscalercheckpoints) pulled from the kubernetes/autoscaler repo at the pinned tag (same upstream-raw pattern node-feature-discovery uses), recommender-scoped RBAC (SA + metrics-reader/actor/status-actor/checkpoint-actor/target-reader), and the recommender Deployment.
- Add `apps/overlays/au-syd1/vpa-system/` referencing the base.
- Register `apps/overlays/*/vpa-system` in the platform ApplicationSet.

## Notes

- Pins upstream **vertical-pod-autoscaler-1.7.0** (latest stable, 2026-05-29) for both CRDs and the `registry.k8s.io/autoscaling/vpa-recommender:1.7.0` image.
- No platform AppProject change needed: `*-system` namespace destination plus CustomResourceDefinition/ClusterRole/ClusterRoleBinding are already whitelisted.
- The recommender image is pull-through-cached via terraform-artifactapi PR #13 (merge gate below).

## Merge gate

- Requires terraform-artifactapi PR #13 (adds the `autoscaling/vpa-` pattern to the k8s-registry docker remote) to land first so the recommender image is served through the cache.

Reviewed-on: #281
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 18:26:15 +10:00
unkinben 614c4b1aa6 Route consul API hostname to the HTTP API (8500) (#282)
Phase 2 of the consul migration: expose the HTTP API (not just the UI) at consul.k8s.syd1.au.unkin.net, now rebased onto main post-#280 (ACLs enabled).

- Adds a consul-http ClusterIP service targeting the server pods on 8500 (API + UI share the port, so the UI stays reachable at /ui/)
- Repoints the consul and consul-svc HTTPRoutes from consul-ui:80 to consul-http:8500
- Documents ACL-authenticated access in apps/base/consul/README.md: token from kv/kubernetes/namespace/consul/default/bootstrap-acl-token (VSO-synced), X-Consul-Token curl and consul CLI usage, UI token login, and the prefer-vault-minted-tokens note

Verification post-merge (ACLs are live, so authenticated): CONSUL_HTTP_TOKEN=$(vault kv get -field=token kv/kubernetes/namespace/consul/default/bootstrap-acl-token) && curl -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" https://consul.k8s.syd1.au.unkin.net/v1/status/leader

Rollback: revert the HTTPRoute backends to consul-ui:80.
Reviewed-on: #282
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 18:19:38 +10:00
unkinben 3920d76ba2 Add VPA CRDs and kubeconform schemas (#284)
Split out of #281 per review: the VerticalPodAutoscaler CRDs land first, together with their generated kubeconform schemas, so CI can validate the VPA objects that follow.

- Adds apps/base/vpa-system with the v1.7.0 VPA CRDs served via the artifactapi github remote (terraform-artifactapi#14, merged)
- Adds schemas/autoscaling.k8s.io/ (verticalpodautoscaler + checkpoint, v1 and v1beta2) generated with the same transform as ci/generate-schemas.sh (from the CRD manifest rather than the live cluster, since the CRDs are not installed yet)
- Wires the vpa-system overlay into the platform applicationset

Verified: kustomize renders both CRDs; a sample updateMode Off VPA passes kubeconform against the new schemas. Merge before #281 (recommender) and #283 (per-workload VPAs).
Reviewed-on: #284
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 18:16:15 +10:00
unkinben a633835550 Enable ACLs on k8s consul with Vault-sourced bootstrap token (#280)
## Why

Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with `default_policy: deny` and `down_policy: extend-cache`; the k8s cluster currently runs with **ACLs disabled**. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the **same** `initial_management` token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git.

## Changes

- Enable `global.acls.manageSystemACLs` so the chart manages system ACL tokens/policies for consul components.
- Point `global.acls.bootstrapToken` at a pre-existing Kubernetes secret `consul-bootstrap-acl-token` (key `token`); chart 1.9.7 supports this, and when the secret is populated the `server-acl-init` job **skips bootstrapping** and adopts that token as the management token (renders `-bootstrap-token-secret-name`/`-bootstrap-token-secret-key`, verified in the kustomize output).
- Add a `VaultAuth` (mount `k8s/au/syd1`, role `default`, SA `default`) and `VaultStaticSecret` in the `consul` namespace that sync `kv/kubernetes/namespace/consul/default/bootstrap-acl-token` into the `consul-bootstrap-acl-token` k8s secret via VSO (mirrors the encapi pattern).
- Merge the `acl` block (`enabled`, `default_policy: deny`, `down_policy: extend-cache`, `enable_token_persistence`) into the server `extraConfig` to match the VM posture.

## OPERATIONAL NOTE — required BEFORE merge

The user MUST place the VM cluster's `initial_management` token in Vault first, or ACL bootstrapping will generate a *different* token and break the mirror:

```
vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token>
```

VSO then syncs it into the `consul-bootstrap-acl-token` secret before the `server-acl-init` job runs. No terraform-vault change is needed: the wildcard `default` k8s-auth role (`bound_service_account_namespaces: ['*']`) plus the templated `kv/kubernetes/default` policy already grant the `consul` namespace `default` SA read on `kv/kubernetes/namespace/consul/default/*`.

## Risk / expected behavior

- Enabling ACLs **rolls the 5 servers** (StatefulSet update) and runs a `server-acl-init` job.
- With `default_policy: deny`, previously-anonymous operations are denied. The `vault` service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace `vault`) via their `service_registration "consul"` stanza (catalog entry has `ServiceMeta.external-source: vault`, port 8200). After the flip this registration will be **denied** unless Vault is given a Consul ACL token with `service:write` on `vault` (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration.
- Anonymous HTTP API reads (e.g. `/v1/status/leader`) will also be denied post-merge unless a token is supplied — see PR 2 verification note.

## Ordering

Independent of the API-route PR (`benvin/consul-api-route`). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done.

Reviewed-on: #280
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 17:39:18 +10:00
unkinben 69f304f60d Roll cephrgw-operator to v0.3.0 (safe adoption) (#279)
## Why

cephrgw-operator **v0.3.0** makes adopting pre-existing radosgw buckets/users safe (operator repo #5, merged): `retainOnDelete` on `ObjectStoreUser`/`BucketAccess`, non-destructive bucket-policy **merge** (+ `managePolicy`), non-destructive user attributes, and `status.adopted`.

## Changes

- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.0`

No credential or manifest change beyond the tag. Validated with `kustomize build` on the au-syd1 overlay.

https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #279
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 17:09:21 +10:00
unkinben cc81e90a75 Roll artifactapi to v3.7.7 (#278)
Deploy the two new UI features released in v3.7.7: per-repo usage-instruction panels (artifactapi#105) and direct-download links for local repo files (artifactapi#106).

- Bumps artifactapi api image to git.unkin.net/unkin/artifactapi:v3.7.7
- Bumps artifactapi ui image to git.unkin.net/unkin/artifactapi-ui:v3.7.7

Both v3.7.7 images are confirmed published to the registry; kustomize builds clean.

Reviewed-on: #278
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 16:48:12 +10:00
unkinben 05318553d6 Swap puppet compiler ENC to the encapic Go binary (#277)
## Why

The puppet-on-k8s compilers classify nodes with a uv/python ENC script (`encapi-enc`). Each fresh compiler pod resolves the script's python dependencies on first invocation, and that resolution fails on cold pods (observed exits 135/2), breaking puppet agent catalog compilation. `encapic` (git.unkin.net/unkin/encapic) is a stdlib-only Go replacement with no runtime dependency resolution — a behavioural drop-in whose output matches the python script byte-for-byte.

## Changes

- Points the compiler `external_nodes` at `/opt/bin/encapic`.
- Reworks the `setup-shared-bins` init container to `curl` the encapic `v0.1.0` `encapic_linux_amd64` release binary (sha256-verified against the published `.sha256`, installed mode 0755) into the shared bins dir, instead of copying the python script and installing uv.
- Removes the `puppet-encapi-enc` configmap generator, its volume and mount, and the `resources/encapi-enc` script. uv was consumed solely by that script (grep of `apps/base/puppet` confirms no other consumer), so its installation is removed too.

`kubectl kustomize apps/overlays/au-syd1/puppet` builds clean.

## Merge gate

Do not merge until the encapic `v0.1.0` release assets exist:
`https://git.unkin.net/unkin/encapic/releases/download/v0.1.0/encapic_linux_amd64` (+ `.sha256`). The init container pulls them at pod start.

## Rollback

Revert this PR to restore the `encapi-enc` configmap script + uv install and repoint `external_nodes`.

---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #277
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-25 12:40:42 +10:00
unkinben 3af12180fd Switch puppet compiler ENC from Cobbler to encapi (#272)
## Why

The k8s puppetserver compilers classify nodes via an exec ENC that today queries legacy Cobbler (`https://cobbler.main.unkin.net`) over TLS. `encapi` now runs in-cluster and exposes a cobbler-wire-compatible endpoint (`GET /cblr/svc/op/puppet/hostname/<certname>`), a drop-in for the Cobbler URL. This cuts the puppet-on-k8s ENC over from Cobbler to encapi — a prerequisite for migrating VM agents onto puppet-on-k8s.

## Changes

- Rename the ENC script `resources/cobbler-enc` -> `resources/encapi-enc`, and its configmap `puppet-cobbler-enc` -> `puppet-encapi-enc` (kustomization configMapGenerator + deployment volume, initContainer copy path, and volumeMount subPath).
- Point `external_nodes` in the compiler `puppet.conf` at `/opt/bin/encapi-enc`.
- Target the in-cluster encapi service `http://encapi.encapi.svc.cluster.local` (plain HTTP), overridable via the `ENCAPI_URL` env var.
- Drop the `/opt/vault-ca-cert.crt` verify for the ENC request (no TLS needed in-cluster).
- Leave the response normalization identical: classes coerced to a list, `enc_role`/`enc_env` params set, `environment` stripped when it equals `testing`.

Verified with `kubectl kustomize apps/overlays/au-syd1/puppet` (builds clean, exit 0); the generated `puppet-encapi-enc` configmap contains the new URL and env var.

## 🚨 Merge gate

**Do not merge until encapi is seeded** (terraform-incus `benvin/encapi-seed` PR applied). An empty encapi means every node resolves to a 404. On 404 the ENC script exits non-zero, so puppet fails the compile rather than classifying the node with zero classes — nodes will fail to run until they exist in encapi. Seed encapi first so real nodes classify correctly; only unknown nodes should 404.

Reviewed-on: #272
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-24 23:42:23 +10:00
unkinben 6d7cdd59e3 pdbmux: deploy the merging PuppetDB proxy in-cluster (#275)
## Why

During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move
between them as they migrate. `node-lookup` (and `pblastreport`) need a single,
consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy
that provides exactly that. Per the all-in-kubernetes estate direction it runs
as an in-cluster service, not a per-VM systemd unit.

pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) —
split out of the earlier node-lookup prototype — and is released as a container
image on its own `v*` tags.

## Changes

- Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway,
  httproute), modeled directly on the encapi app.
- Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080,
  `/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap.
- Backends: `old=http://puppetdbapi.service.consul:8080`,
  `new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified
  against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the
  in-cluster address is preferred over the external gateway). `new` is
  primary/prefer, merge = freshness.
- Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal`
  Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a
  port-80 Service — same shape as the puppetdb/encapi gateways — so
  VM/workstation `node-lookup` can reach it.
- Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform
  ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject
  (`pdbmux` namespace destination), exactly as encapi is wired.

No new woodpecker ServiceAccount is required: the pdbmux image push uses the
`docker-buildx` plugin against the Gitea registry with the `default` SA (same as
encapi), not artifactapi.

## Verification

- `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves
  to `git.unkin.net/unkin/pdbmux:v0.1.0`).
- ApplicationSet + AppProject YAML validated.

## Merge gates

1. The pdbmux repo initial-content PR
   (unkin/pdbmux#1) must merge first.
2. `v0.1.0` must then be tagged on the pdbmux repo so the image
   `git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s
   `.woodpecker/docker.yaml`.
3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update
   the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.)

---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #275
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-24 23:40:56 +10:00
unkinben 4580a5f6a4 Trust internal CA in cephrgw-operator (fix radosgw TLS) (#276)
## Why

cephrgw-operator v0.2.0 talks to radosgw over HTTPS (`radosgw.service.consul:443`, fronted by nginx presenting the internal `unkin.net` Vault-PKI cert). With no CA configured the operator fails:

```
Get "https://radosgw.service.consul:443/admin/user?...": tls: failed to verify certificate: x509: certificate signed by unknown authority
```

The `vault-ca-cert` Secret (the `unkin.net` intermediate+root) is already reflected into every namespace — including `cephrgw-system` — so the fix is deployment-only.

## Changes

- mount the `vault-ca-cert` Secret (key `ca.crt`) read-only at `/etc/vault-ca/ca.crt`, following the puppet/artifactapi pattern
- set `CEPH_RGW_CA_FILE=/etc/vault-ca/ca.crt` so the operator adds the CA to its TLS trust

No image change (still `v0.2.0`); `reloader` + a normal reconcile pick it up. Validated with `kustomize build` on the au-syd1 overlay.

https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #276
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-24 23:25:03 +10:00
unkinben 739de129e1 Add terraform-enc woodpecker ServiceAccount (#274)
The new **terragrunt-enc** repo (single source of truth for encapi ENC data) runs its Terraform apply/plan in Woodpecker and authenticates to Vault via kubernetes auth. The Vault k8s role `woodpecker_terraform_enc` (terraform-vault PR #98) binds to a ServiceAccount named `terraform-enc` in the `woodpecker` namespace, which must exist for that auth to work.

Changes:
- Add `apps/base/woodpecker/serviceaccount_terraform_enc.yaml` (SA `terraform-enc` in namespace `woodpecker`).
- Register it in the woodpecker kustomization.

Reviewed-on: #274
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-24 23:18:11 +10:00
unkinben 887a8d34cb Roll cephrgw-operator to v0.2.0 (radosgw-native) (#273)
## Why

cephrgw-operator **v0.2.0** rebuilds the Ceph integration to talk directly to radosgw via **go-ceph** (Admin Ops API) + **aws-sdk-go-v2** (S3), replacing the manager-dashboard client, and adds **fine-grained bucket-access policies** (paths / actions / conditions / rawStatements). The operator now authenticates with an **RGW admin user's access/secret key** instead of a dashboard login.

Operator repo PRs: unkin/cephrgw-operator #3 (rebuild) and #4 (fine-grained), both merged; tag `v0.2.0`.

## Changes

- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.2.0`
- update the `envFrom` / `VaultStaticSecret` comments to the `CEPH_RGW_*` credential keys the new image consumes

## Required manual step (runtime)

The VaultStaticSecret copies the KV secret's keys **verbatim**, so the seed must be re-put with the new keys before/with rollout — otherwise the operator fails auth:

```
vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
  CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
  CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
  CEPH_RGW_ACCESS_KEY=<key> CEPH_RGW_SECRET_KEY=<secret>
```

(The old `CEPH_DASHBOARD_*` keys are ignored by v0.2.0.) VSO refreshes within 5m and the `reloader` annotation restarts the operator.

https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #273
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-24 23:17:05 +10:00
unkinben 982bf1c837 Merge pull request 'Wire LiteLLM SSO to Authentik (generic OIDC)' (#268) from benvin/litellm-sso into main 2026-07-24 00:18:41 +10:00
unkinben 39b180aec9 bind-internal: allow k8s pod network to query the resolvers (#271)
Kubernetes nodes querying the bind-resolvers LoadBalancer VIP (198.18.200.7) get REFUSED (EDE 18 Prohibited).

The service is `externalTrafficPolicy: Local`, which preserves the client source IP for traffic entering the cluster from outside — but a node querying the VIP never leaves via OSPF. Its own kube-proxy DNATs the LB IP in the OUTPUT chain and masquerades the source to a cluster-internal address (the node's flannel.1, e.g. 10.42.x.x). That address is not in `acl-main.unkin.net`, so the openforwarder view's match-clients rejects the query.

External clients preserve their real source IP and match acl-main, which is why only in-cluster hosts were affected.

Add `10.42.0.0/16` to `acl-main.unkin.net` so node-originated (masqueraded) resolver queries are permitted. This mirrors the authoritative cluster, which already allows the pod network (`allow-query { ...; 10.42.0.0/16; }`).

Reviewed-on: #271
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-21 22:12:44 +10:00
unkinben f835a059f4 Merge pull request 'Bump bind-operator to v0.2.4' (#270) from benvin/bind-operator-v0.2.4 into main 2026-07-21 00:29:33 +10:00
unkinben 76db3bbfe0 Bump bind-operator to v0.2.4
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
Picks up immediate NOTIFY of secondaries on primary zone changes (also-notify
+ shorter seed SOA timers), so dynamic updates / CRD records replicate across
the authoritative replicas in seconds instead of waiting up to the SOA refresh.
Bumps both the operator image and the CRD install.yaml tag.
2026-07-21 00:27:34 +10:00
unkinben 7ee5dfbbcd TEMP: forward k8s.syd1.au.unkin.net to external external-dns service (#269)
The bind-resolvers `openforwarder` view forwarded `k8s.syd1.au.unkin.net` to the in-cluster bind-externaldns (198.18.200.8), which is not reliably serving those records yet, so lookups return NXDOMAIN.

Concrete impact: Gitea cannot resolve the k8s-hosted CI host, so its outbound webhook fails and tagged releases never trigger CI (e.g. cutting a new bind-operator release from a tag).

This points the `fwd-k8s-syd1-au-unkin-net` forwarder at the existing external external-dns bind service anycast **198.18.19.20** (puppet `roles::infra::dns::externaldns` — master `ausyd1nxvm2127` + slaves `2128`/`2129`, advertised via OSPF), which still holds the working `k8s.syd1.au.unkin.net` records. It is in the same 198.18.19.0/24 anycast family as the consul forwarder (198.18.19.14) the resolvers already use, so it is reachable from the pods.

Temporary measure. Revert to 198.18.200.8 once external-dns publishes to the in-cluster bind-externaldns service. Only the forward target changes; no CRD/schema changes.

Reviewed-on: #269
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-21 00:14:03 +10:00
unkinben 91080c1504 LiteLLM SSO: request litellm_role scope + map to role
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
Pairs with terraform-authentik#10: request the litellm_role scope (Authentik
emits the computed role claim) and read it via GENERIC_USER_ROLE_ATTRIBUTE so
akP-litellm-admin -> proxy_admin, akP-litellm-user -> internal_user.
2026-07-19 18:43:13 +10:00
unkinben 5380a9fbf4 Wire LiteLLM SSO to Authentik (generic OIDC)
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
App-side of the LiteLLM Authentik onboarding (terraform-authentik#8). Configures
LiteLLM's generic OIDC SSO against Authentik.

- VaultStaticSecret oauth-credentials: surfaces the OIDC client secret (same
  secret Authentik sets on the provider) as a k8s Secret.
- Deployment: GENERIC_CLIENT_SECRET from that Secret.
- litellm-env: GENERIC_CLIENT_ID, authorization/token/userinfo endpoints, scope,
  and PROXY_BASE_URL (required for SSO). reloader restarts on secret/config change.
2026-07-19 18:27:31 +10:00
unkinben 0c1156282f bind-internal: allow localhost to query the authoritative cluster (#267)
## Why

Every bind-operator dynamic update is refused (`update ... denied due to allow-query`) because the operator execs `nsupdate` against `127.0.0.1` inside the primary pod, and the BindCluster `allow-query` listed only the client subnets (`auth-acl-main`) and the pod net (`10.42.0.0/16`) — not loopback. This blocked ALL DNSRecords (identity, s3, dashboard, lb1) from ever applying.

## Change

- Add `localhost` to the BindCluster `allow-query` in `apps/base/bind-internal/authoritative/cluster.yaml`. The `client-update` TSIG key still gates the actual update.

---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #267
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-19 00:10:56 +10:00
unkinben f61d5ba16d bind-internal: add dashboard.ceph.unkin.net + lb1.unkin.net records (#266)
## Why

The cephrgw-operator (in-cluster) fails to reach the Ceph dashboard because CoreDNS/bind-internal has no record for `dashboard.ceph.unkin.net` (`no such host`). Publish it authoritatively so in-cluster clients can resolve it.

## Changes (apps/base/bind-internal/authoritative/records.yaml)

- `DNSRecord dashboard-ceph-cname`: CNAME `dashboard.ceph.unkin.net` -> `lb1.unkin.net.` (zone `ceph.unkin.net`, zoneRef `ceph-unkin-net`).
- `DNSRecord lb1-unkin-net`: A `lb1.unkin.net` -> `103.216.191.185` (zone `unkin.net`, zoneRef `unkin-net`).

Once applied, the operator's `sandbox-user`/`sandbox-bucket` (currently Error/Pending on DNS) will reconcile to Ready.

---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #266
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-18 23:50:59 +10:00
unkinben 5a22dd95ef cephrgw-system: source cephrgw-credentials from Vault via VSO (#262)
## Why

Follow-up to the cephrgw-operator deploy (#261): source the operator's Ceph dashboard credentials from Vault via VSO instead of a hand-created Secret.

## Changes

- Add `apps/base/cephrgw-system/vaultauth.yaml`: `VaultAuth` (mount `k8s/au/syd1`, role `cephrgw-operator`, SA `cephrgw-operator`, `vaultConnectionRef: vso-system/default`).
- Add `apps/base/cephrgw-system/vaultstaticsecret.yaml`: renders KV `service/cephrgw/dashboard-credentials` into the `cephrgw-credentials` Secret (keys copied verbatim → consumed by the Deployment via `envFrom`; the reloader annotation restarts the operator on rotation).
- Reference both from the base kustomization.

## Dependencies / ordering

- Requires the Vault role + policy from **terraform-vault #95** (merge/apply first), and the KV values to be seeded out-of-band:
  ```
  vault kv put kv/service/cephrgw/dashboard-credentials \
    CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
    CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
    CEPH_DASHBOARD_PASSWORD=... CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net
  ```
- Until VSO auth succeeds the `cephrgw-credentials` Secret won't exist and the operator pod stays in `CreateContainerConfigError` (expected).

---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #262
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-18 23:18:01 +10:00
unkinben 513c60492b bind-internal: add s3.ceph.unkin.net CNAME -> radosgw.service.consul (#265)
## Why

Publish the RGW S3 endpoint name (`s3.ceph.unkin.net`) that cephrgw-operator consumers use and that the radosgw hosts will carry as a cert SAN. For now it points at the Consul service; the real target will be changed later.

## Changes

- Add a `DNSRecord` in the `ceph.unkin.net` authoritative zone: `s3` CNAME `radosgw.service.consul.` (`apps/base/bind-internal/authoritative/records.yaml`, zoneRef `ceph-unkin-net`, TTL 600).

A companion puppet-prod change adds `s3.ceph.unkin.net` to the radosgw cert SANs and nginx server names.

---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #265
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-18 16:30:23 +10:00
unkinben 6f62b7873f Wire Grafana OAuth roles to Authentik ak_groups / akP-grafana-admin (#264)
## Why

Consume the two-tier Authentik RBAC from terraform-authentik#7. Grafana should grant Admin to the `akP-grafana-admin` permission group, which `akR-global-admin` members inherit.

## Change

- **grafana.yaml** (`auth.generic_oauth`): add `ak_groups` to `scopes`; `role_attribute_path` now keys off `ak_groups` and `akP-grafana-admin` (replaces the flat `grafana-admins`). Non-admins who can log in (gated to `akP-grafana-*` by the Authentik access policy) get Viewer; `role_attribute_strict: false` retained.

## Depends on
terraform-authentik#7 (creates `akP-grafana-admin`, the access binding, and the `ak_groups` mapping).

## Validation
`kustomize build` (base + overlay) renders; pre-commit clean.

Reviewed-on: #264
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-18 16:24:26 +10:00
unkinben 9bdb328d99 Wire ArgoCD RBAC to Authentik ak_groups / akP-argocd-admin (#263)
## Why

Consume the two-tier Authentik RBAC from terraform-authentik#7 (user → role → permissions). ArgoCD should grant admin to the `akP-argocd-admin` permission group, which `akR-global-admin` members inherit.

## Change

- **argocd-cm**: request the hierarchical `ak_groups` scope + id-token claim (carries permission groups inherited via role groups; distinct from the default `groups` claim to avoid collision).
- **argocd-rbac-cm**: `scopes: [ak_groups]`; `policy.csv`: `g, akP-argocd-admin, role:admin` (replaces the flat `argocd-admins`). Default stays `role:readonly`.

## Depends on
terraform-authentik#7 (creates `akP-argocd-admin`, the access binding, and the `ak_groups` mapping). Merge/apply that first; then add yourself to `akR-global-admin` in Authentik.

## Validation
`kustomize build` renders the patched configmaps; pre-commit clean. Note: argocd-server picks up argocd-cm/rbac-cm live.

Reviewed-on: #263
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-18 16:23:58 +10:00
unkinben 19f8055144 Deploy cephrgw-operator to cephrgw-system (#261)
## Why

The new `cephrgw-operator` provisions Ceph RGW (S3) buckets and access keys (RW/RO) from Kubernetes CRDs via the Ceph manager dashboard API. This deploys it as a platform app.

## Changes

- Add `apps/base/cephrgw-system`: namespace, ServiceAccount + ClusterRole/Binding (manage `ceph.unkin.net` CRDs, Secrets, leader-election leases), and the operator Deployment. CRDs are pulled from the operator repo at tag `v0.1.0`; the Deployment sources dashboard credentials from the `cephrgw-credentials` Secret via `envFrom` and carries the reloader annotation.
- Add `apps/overlays/au-syd1/cephrgw-system` referencing the base.
- Register `apps/overlays/*/cephrgw-system` in the platform ApplicationSet.

The platform AppProject already permits `*-system` namespaces and the Namespace/ClusterRole/CRD cluster resources, so no project change is needed.

## Ordering / dependencies

- Depends on the Gitea repo from terraform-git #34 and on the operator being pushed + tagged **v0.1.0** (image `git.unkin.net/unkin/cephrgw-operator:v0.1.0` and the raw CRD `install.yaml` at that tag). The `kubeconform` check will stay red until v0.1.0 exists, then go green.
- The `cephrgw-credentials` Secret must be created out-of-band in `cephrgw-system` (see the operator's `docs/ceph-setup.md`); it is intentionally **not** managed in GitOps.

---------

Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #261
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-18 14:32:11 +10:00
unkinben 706e30a202 bind-internal: add ceph.unkin.net authoritative zone (#260)
## Why
The ceph/halb host (`ausyd1nxvm2069`) publishes `dashboard.ceph.unkin.net` via nsupdate to a dedicated `zone ceph.unkin.net.`, which `bind-authoritative` was not authoritative for (NOTZONE). This adds the zone so that record has a home.

## Changes
- Add `ceph.unkin.net` BindZone (primary, dynamicUpdate, updateKeyRef client-update) to bind-authoritative, matching the unkin.net/main.unkin.net pattern.

## Note — not the root cause of the 6 missing hosts
Log evidence (VictoriaLogs, dns-update-apply on 2069/2070) shows the actual failure is a **host-side bug in the puppet `dns-update` script**: `fqdn()` appends the zone even to records whose name is already fully-qualified (e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`), producing a `..` empty label → `invalid owner name: empty label` → the whole main.unkin.net update `send` fails (reverse PTR, sent first, still lands). That script fix (puppet-prod) is the real blocker; this zone is still needed so the ceph record does not hit NOTZONE afterward.

Reviewed-on: #260
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-17 22:43:27 +10:00
unkinben 197991a6ca bind-internal: allow admin workstation to query the resolver (#259)
## Why
Follow-up to #258 (which added workstation+router to the *authoritative* ACL). The **resolver** (bind-resolvers, 198.18.200.7) has its own `acl-main.unkin.net` gating its `openforwarder` view; the workstation is not in it, so recursive queries return REFUSED. This lets the workstation use the resolver as its normal nameserver.

## Changes
- Add `10.10.12.200/32` (workstation, wireguard) to resolver `acl-main.unkin.net`
- Add `198.18.21.160/32` (router) explicitly for documentation; already covered by existing `198.18.21.160/27` and `198.18.21.0/24` (no functional change for the router)

Reviewed-on: #259
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-16 23:13:31 +10:00
unkinben 8f164873e0 bind-internal: allow admin workstation + router to query authoritative (#258)
## Why
Allow the operator's workstation and router to query the bind-authoritative servers directly. Their source addresses are outside the existing `auth-acl-main` client subnets, so named returns REFUSED to them today. The router sits on 198.18.21.0/24 which is not in the ACL at all.

## Changes
- Add `10.10.12.200/32` (workstation, over wireguard) to `auth-acl-main`
- Add `198.18.21.160/32` (router) to `auth-acl-main`

## Note
This grants query permission only. Reaching the LoadBalancer VIP (198.18.200.6) from off-datacenter paths is separately gated by `externalTrafficPolicy: Local`; the workstation-over-wireguard path still needs its L4 routing addressed to actually land on a node with a ready endpoint.

Reviewed-on: #258
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-16 22:56:22 +10:00
unkinben b1b2112c66 Add terraform-rancher woodpecker ServiceAccount (#257)
## Why

The new terraform-rancher CI pipeline runs as a pod in the woodpecker namespace; its ServiceAccount token is what Vault binds to the `woodpecker_terraform_rancher` k8s auth role (see terraform-vault#86) for rancher2 provider auth + Consul state.

## Change

- Add `ServiceAccount/terraform-rancher` (woodpecker ns) and wire it into the woodpecker kustomization, mirroring the other terraform-* runner SAs.

## Validation

`kustomize build apps/base/woodpecker` renders it; pre-commit clean.

Reviewed-on: #257
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-16 22:22:13 +10:00
unkinben 421df740ea Add identity.unkin.net internal DNS record (#256)
## Why

`identity.unkin.net` is configured as the Authentik OIDC issuer but has **no DNS record anywhere**, so in-cluster OIDC discovery fails (e.g. `argocd-server` → `lookup identity.unkin.net ... no such host`). Add an authoritative A record served by the internal bind system (bind-operator) so it resolves.

## Change

- New `DNSRecord/identity-dns-internal` → `198.18.200.4` (the traefik-internal gateway VIP, where the Authentik Gateway serves the `identity.unkin.net` hostname), in the `unkin-net` zone.
- Lives in the **`bind-internal` namespace** alongside the `BindZone`: the operator resolves `zoneRef`/`clusterRef`/`updateKeyRef` within the record's own namespace, so it can't live in the app (authentik) namespace.
- Wired into `apps/base/bind-internal/authoritative/kustomization.yaml`.
- `identity-dns-internal` name distinguishes this from the external DNS that Authentik will manage its own records from later.

## Validation

`kustomize build apps/base/bind-internal` + kubeconform (validates against the `dnsrecord_v1alpha1` schema): 57 valid, 0 invalid. pre-commit clean.

Reviewed-on: #256
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-15 20:53:39 +10:00
unkinben 8a9cdf35fd Bump bind-operator to v0.2.3 (deterministic render) (#255)
## Why

v0.2.2's config-hash rolling restart exposed a latent non-determinism: `client.List` returns cache-ordered results, so the resolver's forward zones reshuffled every reconcile, flipping the config hash and rolling `bind-resolvers-2` endlessly. v0.2.3 (bind-operator #11) sorts every rendered list so `named.conf` is byte-identical across reconciles and the hash is stable.

## Changes

- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.3`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.3`.

CRDs are unchanged from v0.2.2 (controller-only change), so the generated kubeconform schemas need no update.

## Validation

- `bind-system` renders with the v0.2.3 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.

## Deploy note

On deploy the deterministic operator stops churning the ConfigMap; the config hash stabilizes and the stuck resolver rolling update completes, leaving all three pods Ready on one revision.

Reviewed-on: #255
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 23:22:04 +10:00
unkinben 1f4c0e11bb Wire ArgoCD to Authentik OIDC (#253)
## Why

ArgoCD had no external ingress and only local admin auth. This exposes `argocd-server` behind the traefik-internal gateway and enables Authentik SSO, so operators log in with their Authentik identity and group membership. Pairs with unkin/terraform-authentik#3 (creates the OAuth2 provider).

## Changes

- **argocd-cm**: set `url` and `oidc.config` (Authentik issuer `identity.unkin.net/application/o/argocd/`, `argocd` client, openid/profile/email scopes). Client secret resolved from the `argocd-oidc` Secret via `$argocd-oidc:client_secret`.
- **argocd-rbac-cm**: match RBAC on the `groups` claim; default `role:readonly`; map the `argocd-admins` Authentik group to `role:admin`.
- **argocd-cmd-params-cm**: `server.insecure=true` so `argocd-server` serves HTTP behind the TLS-terminating gateway.
- Add **Gateway + HTTPRoutes** for `argocd.k8s.syd1.au.unkin.net` (mirrors the grafana pattern: traefik-internal, vault-issuer cert, external-dns).
- Add **VaultAuth + VaultStaticSecret** sourcing the OIDC client secret from `kv/kubernetes/namespace/argocd/default/oauth-credentials` into the `argocd-oidc` Secret (labelled `part-of=argocd` so ArgoCD will resolve the `$` reference).

## Notes / rollout

- Seed the client secret in Vault out of band (same path terraform-authentik reads).
- The argocd namespace `default` SA already has Vault read access via the `default` k8s role, so no terraform-vault change is needed.
- `argocd-server` needs a one-time rollout restart to pick up `server.insecure`.

Validated with `kustomize build --enable-helm clusters/au-syd1/bootstrap`, `make kubeconform`, and pre-commit (yamllint + no-plain-secrets).

Reviewed-on: #253
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 23:04:53 +10:00
unkinben 1f22ec2aa7 Bump bind-operator to v0.2.2 (config-hash rolling restart) (#254)
## Why

v0.2.2 (bind-operator #10) stamps a `bind.unkin.net/config-hash` on the pod template, so a ConfigMap or `keys.conf` change flips the hash and triggers an operator-driven rolling restart. This fixes the class of bug where config edits (ACLs, forwarders, `validate-except`, primary address, TSIG rotation) never reached running pods — they held a startup snapshot and needed manual pod deletes.

## Changes

- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.2`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.2`.

CRDs are unchanged from v0.2.1 (controller-only change), so the generated kubeconform schemas need no update.

## Validation

- `bind-system` renders with the v0.2.2 CRD bundle; pre-commit clean.

## Deploy note

When the v0.2.2 operator first reconciles it stamps the config-hash annotation, triggering **one rolling restart per bind StatefulSet** — expected, and it also pulls in any already-pending config. From then on, config changes roll pods automatically.

Reviewed-on: #254
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 22:52:19 +10:00
unkinben 69788a967c Fix validate-except syntax (resolver crash-loop) (#252)
## Why — urgent

PR #251 added `validate-except` to the resolver `BindCluster` but the list omitted the trailing semicolon after the final entry. `named` requires **every** entry in a list to be semicolon-terminated, including the last before the closing brace, so it fails config parse and the resolver pods crash-loop:

```
/run/named/named.conf:18: missing ';' before '}'
loading configuration: failure
exiting (due to fatal error)
```

The resolvers (`.7`) are down until this lands; the authoritative (`.6`/`.9`) and externaldns (`.8`) are unaffected.

## Fix

```diff
- validate-except { unkin.net; 18.198.in-addr.arpa; consul }
+ validate-except { unkin.net; 18.198.in-addr.arpa; consul; }
```

Renders to `validate-except { unkin.net; 18.198.in-addr.arpa; consul; };` — valid.

## Recovery

On merge + ArgoCD sync, the operator re-renders the ConfigMap with valid config and the crash-looping pods self-heal on their next restart (no manual `rollout restart` needed). Validated: `bind-internal` renders and pre-commit clean.

Reviewed-on: #252
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 22:26:23 +10:00
unkinben 5e7dc580b7 Exempt internal split-horizon zones from resolver DNSSEC validation (#251)
## Why

Resolving any `unkin.net` record through the resolver (`.7`) returns **SERVFAIL**, while the authoritative (`.6`) answers fine. Confirmed from the resolver's querylog:

```
view openforwarder:   validating unkin.net/SOA: got insecure response; parent indicates it should be secure
broken trust chain resolving 'ausyd1nxvm2120.main.unkin.net/A/IN': 198.18.200.6#53
query failed (broken trust chain)
```

The resolver runs `dnssec-validation auto`. The public `unkin.net` is DNSSEC-signed (the `.net` parent publishes a DS), but the in-cluster split-horizon authoritative serves `unkin.net` **unsigned**. The validator sees "parent says secure" + an insecure answer → treats it as spoofing → SERVFAIL. The authoritative works directly because it does no validation.

## Fix

Add `validate-except` (via `spec.extraOptions`) for the forwarded internal domains, so the resolver treats them as insecure and skips validation:

```
validate-except { unkin.net; 18.198.in-addr.arpa; consul }
```

- `unkin.net` covers all `*.unkin.net` (incl. `main.unkin.net`, `k8s.syd1.au.unkin.net`)
- `18.198.in-addr.arpa` covers every `NN.18.198.in-addr.arpa` reverse zone (subtree)
- `consul` covers the consul TLD

This also makes internal resolution independent of Internet egress (no DNSSEC chain-walk needed). External-name validation is unchanged. No operator change required.

## Validation

`bind-internal` renders and passes `kubeconform` (56/56); pre-commit clean.

## Activation

After merge + operator reconcile, the resolver ConfigMap re-renders; the running pods hold a startup snapshot, so they need a reload: `kubectl -n bind-internal rollout restart statefulset/bind-resolvers`.

Reviewed-on: #251
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 22:17:16 +10:00
unkinben 65f18a6380 Bump bind-operator to v0.2.1 (secondary replication fix) (#250)
## Why

v0.2.1 fixes authoritative **secondary replication**, which never actually worked — the master REFUSED the catalog AXFR. Root causes (bind-operator #9): secondaries presented no TSIG key, member zones had no `allow-transfer`, and secondaries pointed at the primary's ephemeral pod IP.

## Changes

- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.1`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.1`.

CRDs are unchanged from v0.2.0, so the generated kubeconform schemas need no update.

## Validation

- `bind-system` renders with the v0.2.1 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.

## Deploy note

Existing member zones pick up `allow-transfer` via `modzone`, and secondaries re-point at the stable primary Service ClusterIP with the transfer key, restoring replication without manual BIND surgery. A from-scratch namespace recreate also comes up clean (OrderedReady startup means secondaries snapshot a ClusterIP-correct config). Caveat for a full recreate: the operator regenerates the `client-update` TSIG key, so its new material must be re-synced into puppet eyaml before clients can nsupdate.

Reviewed-on: #250
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 21:43:03 +10:00
unkinben 38ef6c4a09 Bump bind-operator to v0.2.0 and deploy companion TSIG API (#249)
## Why

bind-operator v0.2.0 adds the `BindTSIGAPI` CRD and a companion API that `vault-plugin-secrets-bind-tsig` calls to create, rotate and delete TSIG keys (it does so by managing `BindTSIGKey` resources, which the operator reconciles into key material). This rolls the operator forward and deploys an API instance so Vault never talks to the Kubernetes API directly.

## Changes

- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.0`.
- Broaden the operator ClusterRole (`bind-system/rbac.yaml`) with `deployments`, `serviceaccounts` and `roles`/`rolebindings`, so the `BindTSIGAPI` reconciler can create the API Deployment and its namespaced Role/RoleBinding.
- Add a `BindTSIGAPI` (`bind-tsig-api`) in `bind-internal`; the operator reconciles it into a Deployment, Service, ConfigMap, master-token Secret and RBAC. Keys are created in `bind-internal`, alongside the authoritative cluster and its existing keys.
- Add the generated kubeconform schema for `BindTSIGAPI`.

## Notes

- The master access token Secret (`bind-tsig-api-token`) is generated by the operator when absent; the operator does not own it, so a `VaultStaticSecret` can later pre-seed/overwrite it to source the token from Vault.
- Validated: both overlays render (`kubectl kustomize`) and pass `kubeconform` (bind-internal 56/56 valid); pre-commit clean.

## Follow-up

- Point `vault-plugin-secrets-bind-tsig` config at `http://bind-tsig-api.bind-internal.svc:8443`.

Reviewed-on: #249
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 19:12:39 +10:00
unkinben 2409a4d3a0 reduce CPU requests across observability, authentik, bind, woodpecker (#248)
Reduce reserved-but-unused CPU requests across several workloads (limits unchanged).

| Workload | CPU request |
|---|---|
| vmstorage (vmcluster) | 1 → 250m |
| vmagent | 500m → 250m |
| authentik server | 250m → 50m |
| authentik worker | 250m → 100m |
| bind authoritative | 100m → 20m |
| bind externaldns | 100m → 20m |
| bind resolvers | 100m → 20m |
| woodpecker agent | 100m → 50m |

Reviewed-on: #248
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 18:41:08 +10:00
unkinben e0e0873ccb cnpg: reduce cluster CPU requests to 50m (#247)
Reduce CloudNativePG cluster CPU requests to `50m` across the platform to cut reserved-but-unused CPU.

Updated (memory and limits unchanged):
- artifactapi, authentik, encapi, paperclip (250m → 50m)
- grafana (100m → 50m)
- litellm (250m → 50m)

Excluded: woodpecker (already 50m), puppet (handled separately).
Reviewed-on: #247
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 18:39:13 +10:00
benvin 3320c823b3 chore: remove paperclip (#245)
no longer using paperclip, remove it from au-syd1 cluster

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #245
2026-07-12 18:05:54 +10:00
benvin 646d76da16 chore: reduce resource requirements of puppet (#246)
- lower reserved CPU for puppet related services

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #246
2026-07-12 18:04:50 +10:00
unkinben 05d2c83258 Authoritative zones: accept puppet client dynamic updates (#244)
Enables per-host RFC2136 updates from puppet (puppet-prod #475 profiles::dns::updater) to the bind-authoritative zones, via the .9 write endpoint.

## Changes
- add **client-update** BindTSIGKey (clusterRef bind-authoritative; operator generates the material into Secret client-update-tsig)
- set `dynamicUpdate: true` + `updateKeyRef: client-update` on all **18** authoritative zones → the operator renders `allow-update { key "client-update"; }`

## Key bridge (manual, per the TSIG plan)
The operator generates the client-update key value; it must reach puppet eyaml (`profiles::dns::updater::key_secret`) for clients to authenticate — until the planned Vault-sync/secret-reflection operator features exist. Get it with:
`kubectl -n bind-internal get secret client-update-tsig -o jsonpath='{.data.secret}' | base64 -d`

## Validated
kustomize build + kubeconform.

Reviewed-on: #244
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-11 00:57:52 +10:00
unkinben 38ab7199b3 grafana: add CloudNativePG dashboard (follow-up to #239) (#243)
Follow-up to #239, pairing with the CNPG VMPodScrape (#242). Imports the **CloudNativePG** dashboard (grafana.com 20417) as a `GrafanaDashboard` (gzipJson, datasources resolved to the in-cluster VictoriaMetrics uid). Now that #242 collects the postgres metrics, this dashboard renders real data for all CNPG clusters.

Reviewed-on: #243
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-07 00:10:36 +10:00
unkinben be6d07cf77 observability: scrape CloudNativePG postgres metrics (follow-up to #239) (#242)
Follow-up to #239. Every CNPG instance pod already exposes Prometheus metrics on `:9187` (`metrics` port), but nothing scraped them. Adds a single namespace-wide `VMPodScrape` (`namespaceSelector.any`, `selector cnpg.io/podRole=instance`) so the observability VMAgent collects postgres metrics for **all ~23 CNPG clusters** across the estate (authentik, grafana, woodpecker, artifactapi, puppet, litellm, …). No chart changes. Pairs well with the CNPG grafana.com dashboard (20417) as a further follow-up.

Reviewed-on: #242
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-07 00:08:32 +10:00
unkinben 8c9eb9befc cert-manager: scrape metrics (follow-up to #239) (#241)
Follow-up to #239. Adds a `VMServiceScrape` for cert-manager's existing webhook (`metrics` :9402) and cainjector (`http-metrics` :9402) services so the observability VMAgent collects them. No chart change needed. (The controller's own metrics need `prometheus.enabled` in the chart to expose a metrics service — separate follow-up.)

Reviewed-on: #241
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-07 00:08:06 +10:00
unkinben 55e41897ef vmagent: set consul datacenter (fix 0 discovered targets) (#240)
## Why
The Consul SD job discovered **0 targets**. vmagent logs showed it reaching the puppet Consul fine (TLS/connectivity OK) but getting `403` on `GET /v1/agent/self`: the anonymous token `lacks permission 'agent:read'`. VictoriaMetrics calls `/v1/agent/self` only to auto-detect the datacenter; catalog/health reads (what SD actually needs) work anonymously.

## Fix
Set `datacenter: au-syd1` on the consul_sd_config so VM skips the `agent/self` call. No consul token needed.

## Verify after sync
vmagent `/targets` → `consul` job shows the puppet targets (haproxy/ceph/gitea/node/…) up.

Reviewed-on: #240
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-07 00:05:30 +10:00
unkinben 9a56612ae6 observability: scrape existing app metrics (audit + ceph-csi batch) (#239)
## Why
Audit of existing helm deployments for (a) metrics collection and (b) bundled Grafana dashboards, per request. This PR enables the first, cleanest batch (ceph-csi) and documents the rest for follow-up review.

## Key constraint
The cluster has **no Prometheus-Operator CRDs** (`servicemonitors.monitoring.coreos.com` absent) — only the VM-native `VMServiceScrape`/`VMPodScrape`. So flipping charts' `serviceMonitor: true` toggles would fail. The correct mechanism here is native **VMServiceScrape** CRs, which the observability VMAgent picks up (`selectAllByDefault`, added in #234). Bundled chart dashboards ship as ConfigMaps that grafana-operator v5 ignores (it uses CRs), so "enabling" them means adding `GrafanaDashboard` CRs.

## This PR
- `VMServiceScrape` for **csi-cephfs** and **csi-cephrbd** (their `*-http-metrics` services already exist).

## Audit — metrics (apps already exposing a metrics service; enable via VMServiceScrape)
| App | Metrics service / port | Note |
|-----|------------------------|------|
| csi-cephfs / csi-cephrbd | `*-http-metrics` :8080 | **done here** |
| cert-manager | webhook :9402 `metrics`, cainjector :9402 `http-metrics` | multi-port; easy follow-up |
| argocd | `argocd-*-metrics` :8082/8083/8084/9001 | many components |
| media-apps | `prowlarr/radarr/sonarr-metrics` :9707 (exportarr) | follow-up |
| vm-system / grafana-operator | operator metrics | low value |

## Audit — charts with metrics *toggles* (need endpoint enabled + VMServiceScrape)
authentik, vault, purelb, externaldns, cnpg-system (`enablePodMonitor`), reloader, traefik. These expose `serviceMonitor`/`podMonitor` toggles that assume Prometheus CRDs — recommend enabling the metrics endpoint + a VMServiceScrape rather than the toggle.

## Audit — bundled Grafana dashboards
- **CNPG**: ships a dashboard (grafana.com id 20417) — add as `GrafanaDashboard` (grafanaCom).
- **traefik**: `dashboard: true` option + community dashboards — add as `GrafanaDashboard`.
- Others: none bundled.

## Review ask
Confirm the VMServiceScrape approach; I'll extend to the remaining apps (cert-manager/argocd/media-apps + CNPG/traefik dashboards) in follow-up batches.

Reviewed-on: #239
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-06 23:57:54 +10:00
unkinben 95c5d71222 grafana: deploy Grafana instance, datasource and dashboards (#238)
## Why
Deploys Grafana in-cluster (observability project) via the grafana-operator, mirroring the puppet Grafana but modernised — **CNPG** for state, **Authentik OIDC** for auth — and ports the live datasource + dashboards in as CRs.

Depends on: grafana-operator (#235, merged), grafana schemas (#236, merged), Authentik OIDC (terraform-authentik #2), Vault seeds (done), and `^grafana/` image proxy (terraform-artifactapi #5).

## Changes (`apps/base/grafana`)
- **CNPG** postgres Cluster + rw Pooler (db `grafana`); **VaultAuth** + **VaultStaticSecrets** pulling `postgres`/`oauth` credentials from `kv/kubernetes/namespace/grafana/default/*`.
- **Grafana CR**: postgres backend via the pooler; Authentik `generic_oauth` (client id/secret from the Vault-synced secret, openid/email/profile scopes, group→role mapping); `root_url` grafana.k8s.syd1.au.unkin.net.
- **1 GrafanaDatasource** — k8s VictoriaMetrics via the operator `vmselect-main` service; reuses the previous default datasource uid so the imported dashboards resolve unedited.
- **13 GrafanaDashboards** (gzipJson) exported from the current grafana.
- **Gateway API** (traefik-internal) + HTTPRoute for grafana.k8s.syd1.au.unkin.net.
- Registered in the observability ApplicationSet + project.

## Review notes
- OAuth `role_attribute_path` maps Authentik group `grafana-admins` → Admin, else Viewer — **confirm the group name**.
- `database.ssl_mode: require` against the CNPG pooler — adjust if the pooler isn't serving TLS.
- The `VictoriaLogs - cluster` dashboard has no in-cluster logs datasource yet (no VictoriaLogs in k8s) — included for completeness, will be empty until one exists.
- `make kubeconform` clean (24 resources, validated against the strict grafana schemas).

Reviewed-on: #238
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-06 23:54:51 +10:00
unkinben 391197ad1c Merge pull request 'schemas: add grafana.integreatly.org CRD schemas' (#236) from benvin/grafana-schemas into main 2026-07-05 22:22:19 +10:00
unkinben c17bbda89b schemas: add grafana.integreatly.org CRD schemas
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
Generated from the grafana-operator v5.24.0 CRDs (added in the previous
PR) so `make kubeconform` can validate the Grafana / GrafanaDashboard /
GrafanaDatasource / GrafanaFolder CRs introduced by the grafana instance
PR. Mirrors ci/generate-schemas.sh output for the grafana CRD group.
2026-07-05 22:19:40 +10:00
unkinben 0f6c9e5502 Merge pull request 'grafana-system: deploy grafana-operator' (#235) from benvin/grafana-operator into main 2026-07-05 22:18:57 +10:00
unkinben 5f8f2b1c70 observability: migrate VictoriaMetrics to operator CRDs + Consul SD (#234)
## Why
The k8s au-syd1 VictoriaMetrics stack ran as two helm charts and only scraped in-cluster targets. The victoria-metrics-operator already runs in vm-system, so this moves the stack onto operator-managed CRDs. That unlocks VMServiceScrape/VMPodScrape (auto-converted from Prometheus ServiceMonitors, used by a follow-up PR) and adds Consul service discovery so the cluster scrapes the **same puppet-prod targets** as the puppet vmagent. Also shrinks vmstorage 3 → 2 (Ceph-backed, replicationFactor 2).

## Changes
- Add **VMCluster `main`**: vmstorage 2 replicas (cephrbd-fast-delete 200Gi, 180d retention, replicationFactor 2), vminsert/vmselect 2 replicas + HPA (2–10, 60% cpu).
- Add **VMAgent `main`**: retains the kubernetes SD jobs (apiservers/nodes/cadvisor), `selectAllByDefault` for VMServiceScrape/VMPodScrape, and a **Consul SD job** against `consul.service.consul` (resolves to the puppet Consul from pods) replicating the puppet vmagent relabels — keep tag `metrics`, `__scheme__` from `metrics_scheme`, `job` from `metrics_job`. TLS is **verified against the reflected `vault-ca-cert`** (no insecure skip-verify).
- Expose vmselect/vminsert/vmagent via **Gateway API** (traefik-internal Gateway + HTTPRoute, http→https redirect), same hostnames.
- Remove the two helm charts, their values files, and vendored charts.

## Notes
- Data wipe on cutover is acceptable (confirmed) — old helm PVCs can be deleted.
- Verify at rollout: pods resolve `*.main.unkin.net` node FQDNs (needed for CA SAN match on scrape targets); `/targets` shows `job=consul`.

Reviewed-on: #234
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-05 22:17:32 +10:00
unkinben 3df8ac2779 grafana-system: deploy grafana-operator
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
Adds the grafana-operator (grafana.integreatly.org CRDs + controller) so
Grafana and its dashboards/datasources can be managed declaratively as
CRs in a follow-up PR. Sits in the platform project like the other
operators (vm-system, cnpg-system).

Changes:
- Add grafana-system namespace + grafana-operator helm chart v5.24.0
  (watches all namespaces).
- Render CRDs inline (crds.immutable: false) so ArgoCD installs/manages
  the 13 grafana.integreatly.org CRDs instead of the skipped helm crds/
  subchart.
- Register apps/overlays/*/grafana-system in the platform ApplicationSet.
2026-07-05 22:16:30 +10:00
benvin 53b55419a7 chore: add encapi to platform project (#233)
- ensure encapi is deployed to platform

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #233
2026-07-05 17:57:49 +10:00
benvin 56d60f969c bump artifactapi to 3.7.6 (#232)
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #232
2026-07-05 17:51:59 +10:00
unkinben 333e638e24 deploy encapi to au-syd1 (#230)
## Why
encapi is the new Postgres-backed Puppet ENC that replaces Cobbler (Go API + encapi-cli + terraform provider). It needs to run somewhere reachable by the puppet masters (`encapi-cli classify`) and every node's `enc_direct_facts` fact. Deploy it in k8s alongside artifactapi, exposed at `encapi.k8s.syd1.au.unkin.net`.

## Changes
- add `apps/base/encapi/`: namespace, deployment (`git.unkin.net/unkin/encapi`, port 8000, `/healthz` probes), service, gateway + httproute (`encapi.k8s.syd1.au.unkin.net`, traefik-internal), configmap (DB coordinates), CNPG cluster + pooler (database `encapi`), and VaultAuth + VaultStaticSecrets (`postgres-credentials`, `environment`)
- add `apps/overlays/au-syd1/encapi` overlay referencing the base
- register `apps/overlays/*/encapi` in the platform ApplicationSet so ArgoCD picks it up

## Notes
- Mirrors the artifactapi pattern (VaultAuth role `default`, namespace-scoped VSO paths `kv/kubernetes/namespace/encapi/default/*`).
- Before first sync, seed the Vault KV secrets: `environment` must carry `DBPASS` (matching the CNPG owner password) and `ENCAPI_WRITE_TOKEN`; `postgres-credentials` carries the CNPG owner username/password.
- `kustomize build apps/overlays/au-syd1/encapi` validates clean (11 resources).

---------

Co-authored-by: unkinben <neotheo@gmail.com>
Reviewed-on: #230
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-05 17:41:21 +10:00
unkinben 6ed436b973 chore: bump artifactapi to v3.7.5 (#231)
## Why

Roll out artifactapi `v3.7.5`, which ships the local docker registry (artifactapi#103): local `docker` repos now serve the Docker Registry HTTP API V2 for push and pull.

## Changes

- `apps/base/artifactapi/api-deployment.yaml`: `artifactapi` image `v3.7.4` → `v3.7.5`
- `apps/base/artifactapi/ui-deployment.yaml`: `artifactapi-ui` image `v3.7.4` → `v3.7.5`

## Heads-up (follow-up needed)

The API HPA runs `minReplicas: 2`. Local-docker **chunked** blob uploads keep the upload session in-memory per replica, so a real `docker push` (POST → PATCH → PUT across replicas, no session affinity) can intermittently 404 with `BLOB_UPLOAD_UNKNOWN`. Monolithic pushes are unaffected. Recommend a follow-up to make upload sessions replica-independent (S3-backed) or add session affinity for `/v2/*/blobs/uploads/` before relying on pushes in anger.

Reviewed-on: #231
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-05 17:14:08 +10:00
unkinben e030f07986 Add primary (write) Services to authoritative + externaldns (#229)
**Stacked on #228** (needs operator v0.1.5). Merge #228 first; the diff collapses to just this after.

## Why
Writes (RFC2136/nsupdate) must go to pod-0 — the round-robin read Service would land them on a secondary (rejected). Adds a dedicated write endpoint per cluster (operator v0.1.5 `primaryService`).

## Changes
- `bind-authoritative`: LoadBalancer write endpoint on **198.18.200.9** (`bind-authoritative-primary`)
- `bind-externaldns`: ClusterIP write endpoint (`bind-externaldns-primary`, for in-cluster writers)
- regenerate the bindcluster kubeconform schema (primaryService + externalTrafficPolicy)

## Deferred
external-dns is **not** repointed at `bind-externaldns-primary` yet: it authenticates with the existing TSIG key, which the operator-generated key won't match until the planned Vault-sync + secret-reflection features exist. Until then external-dns keeps writing to the puppet externaldns.

## Validated
kustomize build + kubeconform (3 BindClusters valid against the v0.1.5 schema).

---------

Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #229
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-05 16:37:49 +10:00
unkinben 15e70404ae Use externalTrafficPolicy: Local on the DNS services (#228)
- bump operator to v0.1.5 (CRD link + image)

Reviewed-on: #228
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-05 16:08:47 +10:00
unkinben 3ab8bcc34b Restrict authoritative queries to internal networks (#227)
Mirrors the puppet authoritative `master-zones` view (match-clients `acl-main.unkin.net`, recursion no) — restricting who can query bind-authoritative.

## Changes
- add `auth-acl-main` BindACL with the puppet authoritative acl-main.unkin.net networks (13-17,19,20,24-29)
- `allow-query { auth-acl-main; 10.42.0.0/16; }` on bind-authoritative via extraOptions

## Notes
- Implemented as a global `allow-query` rather than a BindView: dynamic *primary* zones inside a view would need per-view `allow-new-zones` (an operator gap). Functionally equivalent for the single master-zones view.
- `10.42.0.0/16` (pod network) is included so secondaries can SOA-refresh from the primary during catalog replication — without it, replication breaks.
- Works on the current operator (no HOLD).

## Caveat
The DNS Services use externalTrafficPolicy: Cluster, which SNATs external clients to node IPs (198.18.19.x, already in acl-main), so this ACL doesn't truly restrict *external* clients yet. True source-IP restriction needs externalTrafficPolicy: Local — happy to switch if wanted.

Reviewed-on: #227
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-04 22:15:43 +10:00
unkinben c8d61205ce Configure resolvers like puppet (openforwarder view + forward zones) (#226)
## Why
`dig google.com @198.18.200.7` was refused: the resolver never set allow-recursion, so BIND defaulted to localnets/localhost. This mirrors the puppet resolver (/etc/named/views.conf + acls.conf) exactly.

## Changes
- `openforwarder` BindView: `match-clients` = the 4 internal ACLs, recursion yes, allow-recursion/allow-query `any` (match-clients gates)
- 4 BindACLs from puppet acls.conf (acl-main.unkin.net/acl-dmz/acl-common/acl-nomad-jobs)
- 26 conditional forward zones in the view (unkin→198.18.19.15, consul→.14, k8s→.20, dmz/network/prod + 10.10.x reverse → 10.10.16.32/33)
- global forwarders 8.8.8.8/1.1.1.1
- operator image → v0.1.4

## Note
Forward-zone upstreams point at the **puppet anycast** servers (still authoritative during migration); flip to the in-cluster authoritative/externaldns LBs once zone data is migrated.

## Validated
kustomize build (59 docs), kubeconform clean.

Reviewed-on: #226
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-04 21:55:33 +10:00
unkinben ce8ebc71ce Consolidate BIND DNS into one bind-internal namespace (#225)
**HOLD until v0.1.3 is tagged/built** (operator #4 merged + tagged) — this PR bumps the operator to v0.1.3, whose CRD adds the `clusterRef` field these keys use.

## Why
Put all BIND DNS services in one `bind-internal` namespace and name the StatefulSets clearly.

## Changes
- 3 clusters consolidated into `bind-internal`, StatefulSets renamed **bind-authoritative** / **bind-resolvers** / **bind-externaldns**; LBs kept on 198.18.200.6/.7/.8; external-dns hostnames renamed to match
- `clusterRef` added to `transfer-key` (→ bind-authoritative) and `externaldns-key` (→ bind-externaldns) so keys are scoped per cluster
- removed the old `ns-auth`/`ns-resolver`/`ns-externaldns` apps; ApplicationSet + AppProject now list `bind-internal`
- bumped `bind-system` operator to **v0.1.3** (CRD link + image)
- operator stays in `bind-system`

## Deploy impact
ArgoCD prunes the old ns-* namespaces (StatefulSets/PVCs — data is only seed SOA+NS, no migrated records yet) and creates the renamed clusters in bind-internal.

## Validated
`kustomize build` → 28 docs (3 BindCluster, 20 BindZone, 2 catalog, 2 keys, ns); kubeconform clean.

Reviewed-on: #225
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-04 00:35:43 +10:00
unkinben 7c9a697452 Deploy binddns-externaldns (RFC2136 dynamic cluster) (#222)
Part of the bind rollout split. **Merge #219 (bind-operator) first** — stacked on it; diff reduces to the binddns-externaldns files once #219 merges.

## Why
The external-dns tier (replaces 3x Puppet external-dns servers): an authoritative cluster whose zones accept RFC2136 TSIG updates from external-dns.

## Changes
- `apps/base/binddns-externaldns`: authoritative `BindCluster` (3 replicas, LoadBalancer/PureLB), `BindTSIGKey` for RFC2136, namespace
- au-syd1 `binddns-externaldns` overlay

## Deploy impact
Creates the `binddns-externaldns` StatefulSet + LoadBalancer once merged.

Reviewed-on: #222
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 23:09:36 +10:00
unkinben 6affc5d8f4 Deploy binddns-resolver (recursive resolver cluster) (#221)
Part of the bind rollout split. **Merge #219 (bind-operator) first** — stacked on it; diff reduces to the binddns-resolver files once #219 merges.

## Why
The recursive-resolver tier (replaces 3x Puppet only-resolver servers): 3 identical recursive servers with upstream forwarders.

## Changes
- `apps/base/binddns-resolver`: resolver `BindCluster` (3 replicas, forwarders, LoadBalancer/PureLB), namespace
- au-syd1 `binddns-resolver` overlay

## Deploy impact
Creates the `binddns-resolver` StatefulSet + LoadBalancer once merged.

Reviewed-on: #221
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 23:08:06 +10:00
unkinben de123af1b1 Bump bind-operator image to v0.1.2 (#224)
**HOLD until v0.1.2 is tagged/built** (bind-operator #3 merged + tagged).

Picks up the zone-provisioning fix (seed glue A record + IP-based primaries + Pod watch) so the clusters stop failing to load their zones.

- `apps/base/bind-system/deployment.yaml`: image v0.1.1 -> v0.1.2

Reviewed-on: #224
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 23:03:37 +10:00
unkinben 649ed07ab0 Deploy binddns-auth (authoritative BIND cluster) (#220)
Part of the bind rollout split. **Merge #219 (bind-operator) first** — this PR is stacked on it, so its diff will reduce to just the binddns-auth files once #219 merges.

## Why
The authoritative masters tier (replaces 3x Puppet authoritative servers): pod-0 primary + 2 secondaries replicating via the catalog zone + AXFR/IXFR.

## Changes
- `apps/base/binddns-auth`: authoritative `BindCluster` (3 replicas, LoadBalancer/PureLB), `BindCatalogZone`, transfer `BindTSIGKey`, namespace
- au-syd1 `binddns-auth` overlay

## Deploy impact
Creates the `binddns-auth` StatefulSet + LoadBalancer once merged.

Reviewed-on: #220
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 21:23:25 +10:00
unkinben dbb5ad4f86 Rename bind DNS namespaces to ns-* (#223)
Renames the three BIND DNS app namespaces `binddns-{auth,resolver,externaldns}` -> `ns-{auth,resolver,externaldns}`.

## Why
Shorter, clearer namespace names for the DNS tiers.

## Changes
- `argocd/applicationsets/platform.yaml`: overlay path registrations renamed (the ApplicationSet derives each app's namespace from its overlay dir name)
- `argocd/projects/platform.yaml`: destination namespaces renamed

## Coupled with
The per-tier PRs (#220/#221/#222) rename the overlay dirs + namespaces + external-dns hostnames to match. No app deploys to a renamed namespace until both this and the tier PR are merged (harmless before then — the ApplicationSet only instantiates apps for existing dirs).

Reviewed-on: #223
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 21:16:24 +10:00
unkinben 4b8f9313c8 Deploy bind-operator (operator + CRDs) (#219)
First of a 4-PR split of the bind rollout (was #216). Deploys just the operator control plane so it can be verified before any DNS clusters exist.

## Why
Roll out incrementally: operator + CRDs first, then each BIND tier as its own PR.

## Changes
- `apps/base/bind-system`: operator Deployment (`git.unkin.net/unkin/bind-operator:v0.1.1`), RBAC, namespace; CRDs pulled from the operator repo by raw URL (`config/crd/install.yaml` @ v0.1.1)
- au-syd1 `bind-system` overlay
- register all four bind apps in `argocd/applicationsets/platform.yaml` (DNS overlays instantiate only when their dirs land in the follow-up PRs)
- add `binddns-*` namespaces to `argocd/projects/platform.yaml`
- add `schemas/bind.unkin.net/*.json` for kubeconform

## Deploy impact
Operator pod + CRDs only. No DNS services yet — the operator is idle until BindClusters exist.

## Follow-ups (merge after this)
binddns-auth, binddns-resolver, binddns-externaldns — one PR each.

Reviewed-on: #219
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 20:04:57 +10:00
unkinben bb330a0365 chore(artifactapi): deploy v3.7.4 (#218)
## Why

artifactapi `v3.7.4` images are built and pushed; au-syd1 is on `v3.7.3`. This rolls forward to ship the terraform provider registry.

## Changes

- `api-deployment`: `artifactapi` `v3.7.3` → `v3.7.4`
- `ui-deployment`: `artifactapi-ui` `v3.7.3` → `v3.7.4`

## What's new in v3.7.4

- Local terraform repos are now a real provider registry: `/.well-known/terraform.json` + `providers.v1` versions/download with GPG-signed SHA256SUMS (#102).
- The signing key self-provisions in the DB (`signing_keys` table) — no K8s secret to mount, so no deployment wiring needed.

Once synced, `terraform init` against `source = "artifactapi.k8s.syd1.au.unkin.net/<repo>/<type>"` works.

Reviewed-on: #218
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 19:40:38 +10:00
unkinben 15225433e9 chore(artifactapi): deploy v3.7.3 (#215)
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
## Why

artifactapi images \`v3.7.3\` are built and pushed to the registry, but au-syd1 is still running \`v3.6.5\`. This rolls the deployment forward to pick up the recent fixes.

## Changes

- \`api-deployment\`: \`artifactapi\` \`v3.6.5\` → \`v3.7.3\`
- \`ui-deployment\`: \`artifactapi-ui\` \`v3.6.5\` → \`v3.7.3\`

Included in v3.7.x since v3.6.5:
- Local-repo files now appear in the cached-objects UI (#99).
- Evicting a local RPM prunes its repodata metadata (#100).
- The bare domain redirects to the web UI at /ui (#101).

Reviewed-on: #215
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-03 15:14:28 +10:00
unkinben bbb9acba36 feat: add woodpecker service accounts for media terraform repos (#214)
Add Kubernetes ServiceAccounts in the woodpecker namespace for terraform-sonarr, terraform-radarr, and terraform-prowlarr CI pipelines.

Reviewed-on: #214
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-28 22:04:33 +10:00
benvin 48f32a044d fix: update TLSRoute to v1 (#213)
TLSRoutes are now in standard, no longer experimental

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #213
2026-06-28 17:50:27 +10:00
unkinben 7f1444fb38 Add Authentik identity provider deployment (#211)
## Summary
- Deploy Authentik (identity.unkin.net) via Helm chart 2026.5.3
- CNPG PostgreSQL cluster (3 instances) with separate rw/ro poolers (2 instances each)
- Redis with 5Gi persistent storage
- Gateway API for HTTPS (identity.unkin.net) and LDAPS (ldap.k8s.syd1.au.unkin.net, ldap.main.unkin.net)
- TLSRoute for LDAPS passthrough, HTTPRoute for external-dns record creation
- Vault secrets for postgres credentials, authentik secret key, and S3 storage credentials
- S3 storage via RadosGW (bucket: authentik)
- 3 server replicas, 2 worker replicas
- Woodpecker ServiceAccount for terraform-authentik CI
- Platform applicationset and project updated

## Dependencies
- terraform-git #15 (merged) — repo definition
- terraform-vault #78 (merged) — auth roles and Consul ACL

## Vault secrets needed before deploy
Write to `kv/kubernetes/namespace/authentik/default/`:
- `postgres-credentials`: username + password
- `authentik-credentials`: AUTHENTIK_SECRET_KEY
- `s3-credentials`: S3 access key + secret key

Reviewed-on: #211
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-28 17:42:49 +10:00
unkinben 784c3b5de1 Add JSON schema generation for kubeconform CRD validation (#212)
## Summary
- New `ci/generate-schemas.sh` script that generates JSON schemas from three sources:
  1. Live cluster CRDs via `kubectl get crds`
  2. Offline CRD manifests (ArgoCD v3.3.2, Gateway API v1.5.1)
  3. Kubernetes v1.33.7 swagger spec for native types
- Schemas follow Datree catalog convention (`<group>/<Kind>_<version>.json`)
- `validate-apps.sh` and `validate-clusters.sh` check local schemas first, falling back to remote
- Fixes TLSRoute (and other CRD) schema validation failures in kubeconform

## Sources
- ArgoCD: `artifactapi.../argoproj/argo-cd/refs/tags/v3.3.2/manifests/ha/install.yaml`
- Gateway API: `artifactapi.../kubernetes-sigs/gateway-api/releases/download/v1.5.1/standard-install.yaml`
- Kubernetes: `artifactapi.../kubernetes/kubernetes/refs/tags/v1.33.7/api/openapi-spec/swagger.json`

Reviewed-on: #212
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-28 17:26:08 +10:00
unkinben cfca1e5278 Add age-api deployment (#210)
## Summary
- Deploy age-api to the au-syd1 cluster
- Uses configMapGenerator for people config with jaidi, ben, and sudaporn
- Includes gateway, httproute, service, and deployment
- Image: git.unkin.net/unkin/age-api:v0.1.0

Reviewed-on: #210
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-28 12:19:38 +10:00
benvin 99a95f4e57 chore: source schema source for kubeconform (#209)
cache frequent lookups to prevent 400 errors from github. the schemas
are available via artifactapi.

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #209
2026-06-27 22:39:31 +10:00
unkinben feaec2c8a9 chore: bump artifactapi + ui to v3.6.5 (#208)
Adds bandwidth saved stat to dashboard.

Reviewed-on: #208
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 22:27:55 +10:00
unkinben d1cc467455 chore: bump artifactapi + ui to v3.6.4 (#207)
Fixes helm chart URL path duplication for same-host repos (stakater).

Reviewed-on: #207
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 08:06:26 +10:00
unkinben 0e9ac4d390 chore: bump artifactapi + ui to v3.6.3 (#206)
Includes Docker Accept header forwarding, Content-Type fix, nginx base path fix, and version endpoint fix.

Reviewed-on: #206
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 07:51:13 +10:00
unkinben 722ced3256 chore: bump artifactapi + ui to v3.6.2 (#205)
Includes Docker Bearer token auth (#60) and UI BASE_PATH build_args fix (#59).

Reviewed-on: #205
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 00:20:27 +10:00
unkinben 92e6f0f13b chore: bump artifactapi + ui to v3.6.1 (#204)
Rebuilds UI with BASE_PATH=/ui so assets serve under /ui/.

Reviewed-on: #204
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-27 00:03:58 +10:00
unkinben 825c46c91b chore: bump artifactapi + ui to v3.6.0 (#203)
Bumps API and UI images from v3.5.0 to v3.6.0.

Reviewed-on: #203
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-26 23:57:52 +10:00
unkinben 2c9c79d8f1 fix: update UI health check paths to /ui (#202)
The UI now serves under /ui (artifactapi#58). Health probes need /ui instead of /.

Reviewed-on: #202
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-26 23:57:27 +10:00
unkinben f695657d9d refactor: simplify artifactapi routes (#201)
Route /ui → UI service, everything else → API service.

Replaces the growing list of per-prefix rules (/api, /v2, /health) with a single catch-all to the API. No more needing to add a route rule every time the API adds a new top-level path.

Reviewed-on: #201
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-26 23:39:24 +10:00
unkinben 5dee768170 fix: route /v2 and /health to artifactapi API service (#200)
The v3 route migration (#198) split routes into /api → API and / → UI, but /v2/ (Docker Registry V2 API) and /health now hit the UI catch-all instead of the API backend.

This breaks `docker pull artifactapi.k8s.syd1.au.unkin.net/...` with context deadline exceeded.

Adds /v2 and /health prefix rules before the UI catch-all.

Reviewed-on: #200
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-26 23:31:47 +10:00
benvin f120f3b426 fix: rename environment2 to environment (#199)
update the environment secret reference to match what has been
 deployed. this prevents a containerconfigerror

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #199
2026-06-26 22:55:24 +10:00
benvin f6d60bd02d feat: artifactapi route change (#198)
complete cutover to artifactapi 3

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #198
2026-06-26 22:50:27 +10:00
benvin aac1b654bb feat: migrate to artifactapi 3+ (#197)
What changed:
- Adds new v3 API and UI deployments (separate api-deployment.yaml, ui-deployment.yaml) alongside the existing monolithic artifactapi-deployment.yaml
- Adds CNPG PostgreSQL cluster + pooler to replace the standalone postgres deployment
- Adds new api-env configmap, new Vault secrets (postgres-credentials, environment), and a second VaultAuth (default1)
- Adds new services targeting the split api and ui selectors
- Adds HPAs for both new deployments
- Updates kustomization to include all new resources

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #197
2026-06-26 22:18:07 +10:00
benvin 1c6e087116 chore: cleanup artifactory3 mess (#196)
attempted to let claude deploy a new version of artifactory with
terrible results. this change is to remove that mess so I can start
again.

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #196
2026-06-21 17:40:17 +10:00
benvin 9e6efb7c78 🤦 (#195)
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #195
2026-06-21 17:30:47 +10:00
benvin cae42b4896 feat: manage postgres-credentials for artifactapi3 (#194)
pull credentials for postgres/cnpg from vault

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #194
2026-06-21 17:26:26 +10:00
benvin 349dc5fd01 chore: remove middleware resource (#193)
there is no crd for this, preventing the deployment of artifactapi 3

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #193
2026-06-21 09:10:49 +10:00
benvin 8cbd645332 feat: deploy artifactapi3 (#192)
just-enough to test terraform deployment and begin migration. have
change to cnpg for the database and a new bucket for storage

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #192
2026-06-20 12:22:22 +10:00
benvin ad2cdd3b63 fix: update woodpecker kustomization (#191)
Reviewed-on: #191
2026-06-17 21:34:02 +10:00
benvin 17782d716c feat: enable terraform-artifactapi jobs (#190)
woodpecker jobs for terraform-artifactapi use the service account of the
same name to run jobs, so that it can access specific secrets

- add terraform-artifactapi serviceaccount

---------

Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #190
2026-06-17 21:23:49 +10:00
unkinben 188c39f85d feat: add terraform-git service account for woodpecker CI (#189)
## Summary
- Add ServiceAccount terraform-git in woodpecker namespace for terraform-git CI pipelines
- Add to kustomization.yaml

## Test plan
- [ ] Verify ArgoCD syncs the new service account
- [ ] Verify woodpecker CI can use the service account

Reviewed-on: #189
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-07 20:36:55 +10:00
unkinben 0b7819bda3 chore: bump almalinux9 image tags (#188)
Bump almalinux9 image tags to 20260606

Reviewed-on: #188
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-07 00:35:12 +10:00
benvin 3c6330ebfd benvin/gitea (#187)
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #187
2026-06-06 19:47:16 +10:00
unkinben a3a56d0c2b chore: add almalinux-vault repos (#186)
- 9.7 is end of life, ensure that we can still query packages

Reviewed-on: #186
2026-06-02 23:13:45 +10:00
unkinben 4b1fbe1fe1 feat(kanidm): scale down to single replica, remove replication (#185)
Drop from 3 replicas to 1. Remove init container, repl-certs secret,
replication port, podAntiAffinity, server-1/2 configs, and replication
stanza from server-0.toml. Mount configmap directly via subPath.

Reviewed-on: #185
2026-06-02 22:41:28 +10:00
unkinben 666f3d055c feat: add sessionaffinity to kanidm service (#184)
- required as traefik is in passthrough mode

Reviewed-on: #184
2026-06-02 21:17:51 +10:00
unkinben 3dc8801070 fix(kanidm): fix automatic_refresh TOML generation in init container (#182)
## Summary

- The `\n` escape in a shell variable wasn't interpreted as a newline when passed as a `printf %s` argument
- This caused `automatic_refresh = true` to be appended to the `partner_cert` string value on the same line, breaking TOML parsing on kanidm-2
- Fixed by using separate `printf` calls per peer type, with `\n` in the format string (not a variable) where it is correctly interpreted

## Test plan

- [ ] kanidm-2 init container generates valid TOML with `automatic_refresh = true` on its own line under the kanidm-0 peer section
- [ ] kanidm-1 and kanidm-2 start successfully and auto-refresh domain UUID from kanidm-0

Reviewed-on: #182
2026-05-31 00:25:21 +10:00
unkinben 60f1f3130b fix(kanidm): replicate 1/2 from 0 only with automatic_refresh (#181)
kanidm-0 is the authoritative supplier; kanidm-1 and kanidm-2 pull
from kanidm-0 only. automatic_refresh = true on the kanidm-0 peer
entry for kanidm-1/2 so fresh nodes auto-sync domain UUID on restart.

Reviewed-on: #181
2026-05-31 00:20:30 +10:00
unkinben b6f8cb0633 feat: autorestart statefulset (#180)
- ensure kanidm is restarted with vault secrets

Reviewed-on: #180
2026-05-30 23:40:07 +10:00
unkinben f11ec1056d fix(kanidm): remove invalid automatic_refresh from replication config (#179)
Reviewed-on: #179
2026-05-30 23:20:48 +10:00
unkinben ed7feaf19a Update apps/base/kanidm/vaultauth.yaml (#177)
Fix the VaultAuth object

Reviewed-on: #177
2026-05-30 23:11:38 +10:00
unkinben 4d594fbde7 feat(kanidm): vault-managed replication certs with auto-restart (#176)
- Store per-pod replication certs in Vault (kv/kubernetes/namespace/kanidm/default/repl-certs)
- VaultAuth + VaultStaticSecret sync certs to kanidm-repl-certs Secret
- busybox config-init init container injects peer certs from Secret into server.toml at startup
- Remove hardcoded partner_cert entries from per-pod server.toml templates
- Add automatic_refresh = true to all replication configs
- Add reloader.stakater.com/auto annotation to trigger rolling restart on ConfigMap/Secret changes
- Document domain UUID mismatch resolution and cert rotation in README

Reviewed-on: #176
2026-05-30 23:00:46 +10:00
unkinben 1b781e0885 feat(woodpecker): set workflow pod priority class to power (#175)
## Summary
Sets `WOODPECKER_BACKEND_K8S_PRIORITY_CLASS: power` on the Woodpecker agent so all CI pipeline pods are scheduled with the `power` PriorityClass (value 100, preemptionPolicy: Never).

This means pipeline pods can be evicted when the cluster is under pressure but won't preempt other workloads.

## Dependency
Requires the `power` PriorityClass to exist on the cluster — deploy PR #174 (priority-classes app) first.

## Test plan
- Trigger a pipeline run and confirm pods are created with `priorityClassName: power`
- `kubectl get pod -n woodpecker -o jsonpath='{.items[*].spec.priorityClassName}'`

Reviewed-on: #175
2026-05-26 23:58:57 +10:00
unkinben ede25a3858 feat(platform): add priority-classes app with low/power/medium/high classes (#174)
## Summary
- New `apps/base/priority-classes/` app with four `PriorityClass` objects managed via the `platform` ArgoCD project
- Adds `apps/overlays/*/priority-classes` to the platform ApplicationSet generator
- Adds `priority-classes` namespace to platform AppProject destinations (required even for cluster-scoped resources)

| Class | Value | PreemptionPolicy | Intent |
|---|---|---|---|
| `low` | 100 | Never | Background work; evictable, won't preempt others |
| `power` | 100 | Never | Compute-heavy but expendable (e.g. AI/ML workloads) |
| `medium` | 10000 | PreemptLowerPriority | Standard services |
| `high` | 100000 | PreemptLowerPriority | Critical services; preempts lower-priority pods |

`PriorityClass` is already in the platform project's `clusterResourceWhitelist` so no project policy changes were needed.

## Test plan
- ArgoCD syncs `platform-priority-classes` successfully
- `kubectl get priorityclasses low power medium high` shows all four classes

Reviewed-on: #174
2026-05-26 23:41:54 +10:00
unkinben f5f713fe86 feat(artifactapi): add open-webui/open-webui to ghcr immutable patterns (#173)
Part of #155 (prerequisite for open-webui deployment PR #172).

## Summary
- Adds `^open-webui/open-webui` to the `ghcr` remote's `immutable_patterns` in `remote-docker.yaml` so version-pinned open-webui image pulls are cached indefinitely through artifactapi

## Test plan
- artifactapi serves `ghcr.io/open-webui/open-webui:<version>` with `X-Artifact-Source: cache` on second fetch

Reviewed-on: #173
2026-05-26 23:28:27 +10:00
unkinben 3990fbfe06 feat(vault): switch to Kubernetes service registration (#171)
Replaces Consul service registration with the native Kubernetes provider so Vault labels its own pods with active/standby/perf-standby status without requiring a Consul dependency.

## Changes
- `values.yaml`: swap `service_registration "consul"` for `service_registration "kubernetes" {}`, add `VAULT_K8S_NAMESPACE` and `VAULT_K8S_POD_NAME` env vars via downward API
- `role_k8s-service-registration.yaml`: Role + RoleBinding granting the `vault` service account `get`/`update`/`patch` on pods
- `kustomization.yaml`: include new RBAC file

Reviewed-on: #171
2026-05-26 00:06:56 +10:00
unkinben d358098fff chore: update replication certs (#170)
- add replication certs for kanidm-0, kanidm-1 and kanidm-2

Reviewed-on: #170
2026-05-25 23:52:06 +10:00
unkinben 201e601737 feat: update kanidm replicaiton (#169)
- split to per-server configs
- remove init containers that attempted to automate the replication config
- add README.md

Reviewed-on: #169
2026-05-25 23:25:48 +10:00
unkinben d230d87ec9 feat(artifactapi): add conftest to GitHub generic remote cache (#168)
## Summary

- Adds `open-policy-agent/conftest/.*/conftest_.*_Linux_x86_64.tar.gz$` to the `github` remote immutable patterns in artifactapi

## Why

conftest v0.68.2 (https://github.com/open-policy-agent/conftest/releases/tag/v0.68.2) is now used for OPA policy checks in CI (see #167). Caching the release tarball in artifactapi reduces external dependency on GitHub during builds.

Reviewed-on: #168
2026-05-25 22:44:57 +10:00
unkinben 6497dab25e fix(puppet): remove explicit clusterIP: null from puppetdb Service (#166)
## Summary

- Removes `clusterIP: null` from the `puppetdb` Service spec

## Why

Setting `clusterIP: null` makes ArgoCD's desired state explicit about the field being null. Kubernetes assigns a real IP on creation and the field is immutable afterward. The null vs assigned-IP mismatch causes permanent OutOfSync on the puppetdb Service. Removing the field means ArgoCD no longer claims ownership of `clusterIP`, so the API server's value is authoritative.

Reviewed-on: #166
2026-05-25 22:44:24 +10:00
unkinben f403c6b05d fix(kanidm): add explicit group/kind/weight to TLSRoute refs (#165)
## Summary

- Adds `group: gateway.networking.k8s.io` and `kind: Gateway` to `parentRefs`
- Adds `group: ""`, `kind: Service`, and `weight: 1` to `backendRefs`

## Why

The Gateway API controller defaults these fields when creating/updating TLSRoute objects, so the live state always has them. ArgoCD diffs desired vs live by string comparison, causing the `kanidm` TLSRoute to show permanent OutOfSync. Same root cause as #162 (HTTPRoutes).

Reviewed-on: #165
2026-05-25 22:43:52 +10:00
unkinben ac8b8212bd fix(consul): normalize cpu limit to canonical string form (#164)
## Summary

- Changes `server.resources.limits.cpu` from `1000m` to `"1"` in consul Helm values

## Why

`1000m` (1000 milliCPU) is equivalent to `1` CPU, but Kubernetes normalizes the value to `"1"` when storing. ArgoCD diffs desired vs live by string comparison, so the mismatch causes a permanent OutOfSync on the `consul-server` StatefulSet. Same root cause as #163.

Reviewed-on: #164
2026-05-25 22:43:35 +10:00
unkinben dd282f59fb fix(litellm): normalize postgres cluster resource values (#163)
## Summary

- Changes `limits.memory` from `1024Mi` to `1Gi` (same value, canonical form)
- Changes `limits.cpu` from `1` (integer) to `"1"` (string, canonical form)

## Why

Kubernetes normalizes resource quantities on write — `1024Mi` becomes `1Gi` and integer `1` becomes string `"1"`. ArgoCD diffs by string comparison, so these equivalent values cause a permanent OutOfSync on the `litellm-postgres` Cluster.

Reviewed-on: #163
2026-05-24 23:30:10 +10:00
unkinben 1890dd4bda fix(gateways): add explicit group/kind/weight to all HTTPRoute refs (#162)
## Summary

- Adds `group: gateway.networking.k8s.io` and `kind: Gateway` to all `parentRefs` entries
- Adds `group: ""`, `kind: Service`, and `weight: 1` to all `backendRefs` entries
- Affects 9 HTTPRoute files across artifactapi, cattle-system, consul, kanidm, litellm, paperclip, puppet, and vault

## Why

ArgoCD diffs the desired manifest against the live Kubernetes object. The Gateway API controller defaults these fields when creating/updating objects, so the live state always has them — causing persistent OutOfSync for every HTTPRoute. Same root cause as #153 (certificateRefs).

## Test plan

- [ ] All affected ArgoCD applications show Synced after merge

Reviewed-on: #162
2026-05-24 20:32:37 +10:00
unkinben 6815b66010 fix(kanidm): use dockerhub image instead of ghcr.io (#161)
## Summary

- Changes both `config-init` init container and `kanidm` container images from `ghcr.io/kanidm/server:1.10.3` to `kanidm/server:1.10.3`

## Why

`kanidm/server` is published on Docker Hub, not ghcr.io. RKE2 rewrites dockerhub pulls through the artifactapi mirror automatically.

## Test plan

- [ ] Pods roll successfully after ArgoCD sync
- [ ] Verify kanidm cluster replication still healthy

Reviewed-on: #161
2026-05-24 20:27:21 +10:00
unkinben 7cbec33588 fix(artifactapi): move kanidm to dockerhub remote (#160)
## Summary

- Removes `^kanidm/` from the `ghcr` remote immutable_patterns
- Adds `^kanidm/` to the `dockerhub` remote immutable_patterns

## Why

`kanidm/server` is published on Docker Hub, not ghcr.io. Pulling via the `ghcr` cache was failing with 403 on anonymous token fetch → 502 Bad Gateway.

## Test plan

- [ ] `docker pull artifactapi.k8s.syd1.au.unkin.net/dockerhub/kanidm/server:1.10.3` succeeds after artifactapi redeploys

Reviewed-on: #160
2026-05-24 20:24:33 +10:00
unkinben 3756208ccd benvin/kanidm (#159)
Reviewed-on: #159
2026-05-24 19:55:22 +10:00
unkinben 6ce92e8ead benvin/artifactapi-mail-images (#158)
Reviewed-on: #158
2026-05-24 14:44:38 +10:00
unkinben af79d86db6 feat(artifactapi): cache stalwart webadmin zip (#157)
## Summary

- Adds \`stalwartlabs/webadmin/releases/latest/download/webadmin.zip\` to \`mutable_patterns\` in the \`github\` generic remote so the stalwart webadmin UI can be fetched through artifactapi rather than directly from GitHub.

## Notes

- Uses \`mutable_patterns\` (not \`immutable\`) because \`releases/latest\` resolves to whichever release is current and changes over time.
- Access URL: \`https://artifactapi.k8s.syd1.au.unkin.net/generic/github/stalwartlabs/webadmin/releases/latest/download/webadmin.zip\`

Reviewed-on: #157
2026-05-24 12:55:16 +10:00
unkinben 5f4c9225bb feat(artifactapi): add mail stack images to docker registry cache (#156)
- ghcr: stalwartlabs/stalwart (Stalwart mail server)
- dockerhub: rspamd/rspamd (spam filter), tozd/postfix (MTA gateway)

Reviewed-on: #156
2026-05-24 12:42:27 +10:00
unkinben cbc2c1cb9f fix(gateways): add explicit group: "" to all certificateRefs entries (#153)
The Gateway API admission server defaults certificateRefs[].group to ""
when it is omitted. ArgoCD diffed the desired state (no group field) against
the live state (group: "") and flagged every gateway as out of sync.

Fix: explicitly set group: "" in all certificateRefs entries so the
rendered manifest matches the API server's canonical form exactly.

Affected: artifactapi, cattle-system, consul, litellm, paperclip,
puppet (puppetboard + puppetdb), vault.

Reviewed-on: #153
2026-05-23 23:47:24 +10:00
unkinben c6f9893804 fix(argocd): add vault and consul to platform project destinations (#152)
Vault and consul namespaces were missing from the platform AppProject
allowed destinations, causing ArgoCD sync failures with:
  destination server 'https://kubernetes.default.svc' and namespace
  'vault' do not match any of the allowed destinations in project 'platform'

Reviewed-on: #152
2026-05-23 23:27:24 +10:00
unkinben e43fb742ad feat(artifactapi): add kanidm to ghcr docker immutable patterns (#151)
Prerequisite for kanidm deployment (PR benvin/kanidm).

Reviewed-on: #151
2026-05-23 23:09:38 +10:00
unkinben 11ac2ae91e feat(consul): deploy HashiCorp Consul 1.22.7 via Helm chart (5-replica cluster) (#149)
## Summary

- Deploys HashiCorp Consul 1.22.7 using Helm chart 1.9.7 with 5 server replicas
- Configuration modelled on production consul: \`datacenter=au-syd1\`, \`connect=true\`, \`raft_multiplier=10\`, HTTP on 8500, GRPC on 8502, HTTPS disabled
- 5-replica server cluster with \`bootstrapExpect=5\`
- 10Gi cephrbd-fast-delete PVC per server pod
- Gateway API: HTTPS gateway + HTTPRoute (443→consul-consul-ui:80→8500) at \`consul.k8s.syd1.au.unkin.net\`
- PodDisruptionBudget patched from \`policy/v1beta1\` to \`policy/v1\` (k8s 1.25+ compatibility)
- ArgoCD platform ApplicationSet updated to include consul overlay path
- Clients disabled (server-only deployment)
- ConnectInject disabled (can be enabled later for service mesh)

## Requires

- PR #147 (artifactapi: add hashicorp/consul to docker immutable patterns) to be merged first

## Test plan

- [ ] Sandbox tested in \`sandbox-consul\`: all 5 server pods 1/1 Running, cluster formed
- [ ] After merge: ArgoCD syncs consul namespace
- [ ] Verify \`consul.k8s.syd1.au.unkin.net\` is accessible via Gateway

Reviewed-on: #149
2026-05-23 22:40:49 +10:00
unkinben d2be521878 feat(vault): deploy HashiCorp Vault 2.0.1 via Helm chart (5-replica HA raft) (#148)
## Summary

- Deploys HashiCorp Vault 2.0.1 using Helm chart 0.32.0 in HA raft mode (5 replicas)
- Configuration modelled on production vault: \`disable_mlock=true\`, headless-DNS retry_join for all 5 pods
- IPC_LOCK capability added via \`server.statefulSet.securityContext.container\`
- 10Gi cephrbd-fast-delete PVC per pod via \`dataStorage\`
- Gateway API: HTTPS gateway + HTTPRoute (443→vault service port 8200) at \`vault.k8s.syd1.au.unkin.net\`
- ArgoCD platform ApplicationSet updated to include vault overlay path
- Injector disabled (no agent sidecar injection needed)

## Requires

- PR #147 (artifactapi: add hashicorp/vault to docker immutable patterns) to be merged first

## Test plan

- [ ] Sandbox tested in \`sandbox-vault\`: all 5 pods Running, raft cluster forming
- [ ] After merge: ArgoCD syncs vault namespace
- [ ] Operator runs \`vault operator init\` to initialize, then unseals all 5 nodes
- [ ] Verify \`vault.k8s.syd1.au.unkin.net\` is accessible via Gateway

Reviewed-on: #148
2026-05-23 22:39:41 +10:00
unkinben bcd4c1a722 feat(cert-manager): upgrade to v1.20.2 and enable Gateway API support (#150)
## Summary

- Upgrades cert-manager from v1.19.2 to v1.20.2
- Enables `enableGatewayAPI: true` via the `ControllerConfiguration` config block

## Why

cert-manager's Gateway API integration was not enabled. Without it, `cert-manager.io/*` annotations on Gateway resources are ignored and no certificates are issued. This is required for the consul and vault PRs (#148, #149) to have their TLS certs automatically provisioned from their Gateway annotations.

In v1.20.2, `ExperimentalGatewayAPISupport` is BETA and defaults to true — enabling `enableGatewayAPI` in the controller config activates the gateway-shim controller.

## Test plan

- [ ] cert-manager rolls out cleanly (v1.20.2 pods become Ready)
- [ ] After rollout, existing Gateway-annotated services (artifactapi, puppet, litellm) retain valid certs
- [ ] New Gateway resources with `cert-manager.io/cluster-issuer` annotations trigger Certificate creation

Reviewed-on: #150
2026-05-23 22:38:39 +10:00
unkinben 6d9530b1ee feat(artifactapi): add hashicorp/consul and hashicorp/vault to docker immutable patterns (#147)
## Summary

- Adds \`^hashicorp/consul\` and \`^hashicorp/vault\` to the dockerhub immutable_patterns in artifactapi's remote-docker.yaml
- Replaces the more specific \`^hashicorp/vault-secrets-operator\` pattern since \`^hashicorp/vault\` subsumes it
- Required for the benvin/vault and benvin/consul branches (vault:2.0.1 and consul:1.22.7)

## Test plan

- [ ] Verify artifactapi accepts requests for hashicorp/vault and hashicorp/consul images after merge

Reviewed-on: #147
2026-05-23 18:21:25 +10:00
unkinben dcea768c15 feat(woodpecker): upgrade to v3.14.1 (chart 3.6.3) (#146)
Reviewed-on: #146
2026-05-23 18:00:55 +10:00
unkinben e05f9bfd83 feat: increase litellm resources (#144)
finding litellm performance has dropped, crashed in multiple cases, and
then it had scaled to the maximum level using the majority of memory in
cluster.

- reduce the rate at which litellm autoscales
- increase the requests/limits to match usage

Reviewed-on: #144
2026-05-23 17:59:43 +10:00
unkinben 445d8b6e7e feat: add HTTP→HTTPS redirect to Gateway API services (#145)
Add port 80 HTTP listener and redirect HTTPRoute to artifactapi,
cattle-system (rancher), litellm, paperclip, and puppetboard — restoring
the redirect behaviour that existed on the previous nginx/traefik Ingress
resources.

Reviewed-on: #145
2026-05-23 17:34:07 +10:00
unkinben c2637da068 feat(artifactapi): migrate Ingress to Gateway API (#129)
## Summary

- Replace `Ingress` (nginx) with `Gateway` + `HTTPRoute` using `traefik-internal` GatewayClass
- TLS terminated at the Gateway listener; cert-manager provisions the certificate via `vault-issuer`
- external-dns annotations moved to the Gateway

## Notes

The original Ingress had nginx-specific annotations (`proxy-body-size: 10g`, `proxy-read-timeout: 600`) which are not portable to Gateway API. These can be re-introduced via a Traefik `Middleware` CRD if needed.

## Test plan

- [ ] ArgoCD syncs the app cleanly
- [ ] cert-manager issues the `artifactapi-tls` certificate
- [ ] external-dns creates the DNS record
- [ ] `https://artifactapi.k8s.syd1.au.unkin.net` is reachable

Reviewed-on: #129
2026-05-23 16:06:33 +10:00
unkinben 90ddd932fe feat(puppet): migrate puppetdb Ingress to Gateway API (#131)
## Summary

- Replace `Ingress` (nginx) with `Gateway` + `HTTPRoute` using `traefik-internal` GatewayClass
- TLS terminated at the Gateway listener; cert-manager provisions the certificate via `vault-issuer`
- external-dns annotations moved to the Gateway
- `ingress_puppetboard.yaml` is unchanged in this PR (separate PR)

## Test plan

- [ ] ArgoCD syncs the puppet app cleanly
- [ ] cert-manager issues the `puppetdb-tls` certificate
- [ ] external-dns creates the DNS record
- [ ] `https://puppetdb.k8s.syd1.au.unkin.net` is reachable

Reviewed-on: #131
2026-05-23 16:05:26 +10:00
unkinben 2c6d88aa6b feat(puppet): migrate puppetboard Ingress to Gateway API (#130)
## Summary

- Replace `Ingress` (nginx) with `Gateway` + `HTTPRoute` using `traefik-internal` GatewayClass
- TLS terminated at the Gateway listener; cert-manager provisions the certificate via `vault-issuer`
- external-dns annotations moved to the Gateway
- `ingress_puppetdb.yaml` is unchanged in this PR (separate PR)

## Test plan

- [ ] ArgoCD syncs the puppet app cleanly
- [ ] cert-manager issues the `puppetboard-tls` certificate
- [ ] external-dns creates the DNS record
- [ ] `https://puppetboard.k8s.syd1.au.unkin.net` is reachable

Reviewed-on: #130
2026-05-23 01:31:28 +10:00
unkinben 58368948d9 feat(paperclip): migrate Ingress to Gateway API (#133)
## Summary

- Replace `Ingress` (nginx) with `Gateway` + `HTTPRoute` using `traefik-internal` GatewayClass
- TLS terminated at the Gateway listener; cert-manager provisions the certificate via `vault-issuer`
- external-dns annotations moved to the Gateway

## Test plan

- [ ] ArgoCD syncs the paperclip app cleanly
- [ ] cert-manager issues the `paperclip-tls` certificate
- [ ] external-dns creates the DNS record
- [ ] `https://paperclip.k8s.syd1.au.unkin.net` is reachable

Reviewed-on: #133
2026-05-23 01:31:03 +10:00
unkinben 4f5c3f7ea0 feat(litellm): migrate Ingress to Gateway API (#134)
## Summary

- Replace `Ingress` (nginx) with `Gateway` + `HTTPRoute` using `traefik-internal` GatewayClass
- TLS terminated at the Gateway listener; cert-manager provisions the certificate via `vault-issuer`
- external-dns annotations moved to the Gateway

## Test plan

- [ ] ArgoCD syncs the litellm app cleanly
- [ ] cert-manager issues the `litellm-tls` certificate
- [ ] external-dns creates the DNS record
- [ ] `https://litellm.k8s.syd1.au.unkin.net` is reachable

Reviewed-on: #134
2026-05-23 01:29:54 +10:00
unkinben fd87cb96b5 feat(externaldns): upgrade to 1.21.1, fix sources for installed CRDs (#143)
## Changes

- Upgrade external-dns chart from 1.19.0 → 1.21.1 (app v0.19.0 → v0.21.0)
- Remove `gateway-tcproute` source — `TCPRoute` CRD is not installed, causing crash-loop
- Add `gateway-tlsroute` source — `TLSRoute` CRD is installed (Gateway API 1.5.1)

## Why

The pod was crash-looping every minute with `failed to list *v1alpha2.TCPRoute: the server could not find the requested resource` because the TCPRoute CRD doesn't exist in this cluster. TLSRoute was previously removed but its CRD does exist.

Reviewed-on: #143
2026-05-23 01:28:20 +10:00
unkinben d619f9195e benvin/externaldns_compatability (#142)
Reviewed-on: #142
2026-05-23 01:17:20 +10:00
668 changed files with 174943 additions and 1535 deletions
+273
View File
@@ -0,0 +1,273 @@
---
description: Pull master, read open issues, pick one, branch, implement, test, commit, PR, and comment.
---
# Solve a Gitea Issue
## Current repo state
```!
git status --short
echo "Current branch: $(git branch --show-current)"
echo "Remote: $(git remote get-url origin 2>/dev/null || echo 'none')"
```
## Open issues (with full body)
```!
echo "Fetching open issues..."
issue_ids=$(tea issues list --output simple 2>/dev/null | awk 'NF && $1 ~ /^[0-9]+$/ {print $1}')
if [ -z "$issue_ids" ]; then
echo "No open issues found (or tea is not logged in)."
else
for id in $issue_ids; do
echo ""
echo "══════════════════════════════════════"
tea issues view "$id" --fields index,title,body 2>/dev/null \
|| tea issue "$id" 2>/dev/null \
|| echo " (could not read issue #$id)"
echo "══════════════════════════════════════"
done
fi
```
---
## Your task
Follow these steps **in order**. Do not skip steps.
### 1 — Choose an issue
Present the issues above to the user as a numbered list (index, one-line title). Ask which one to work on. Wait for the answer before continuing.
### 2 — Sync master
```bash
git checkout master
git pull
```
Confirm you are on master and up to date.
### 3 — Create a branch
Name the branch `benvin/issue-<N>-<short-slug>` where `<short-slug>` is 24 kebab-case words from the issue title.
```bash
git checkout -b benvin/issue-<N>-<slug>
```
### 4 — Read the issue in full
Re-read the full issue body shown above. If any part is ambiguous, state your interpretation before coding.
**If you discover other problems while working:** do NOT solve them inline. Create a new Gitea issue with `tea issues create --title "..." --description "..."` and stay focused on the assigned issue.
### 5 — Implement the solution
Make the code changes needed to resolve the issue. Follow the conventions already in the repo:
- `main.py` route handlers each contain a single function call; logic lives in submodules.
- No comments unless the WHY is non-obvious.
- No new files unless the issue or architecture requires it.
- Security: no command injection, XSS, SQL injection, or secrets in code.
- **For performance improvements:** implement at the most generic call site possible so the fix applies to all current and future implementations, not just the one being tested.
### 6 — Update tests
Add or update tests that cover the new behaviour. Tests live in `tests/`. Check existing test structure before writing new ones — mirror the style and fixture patterns already in use.
### 7 — Update README
If the feature introduces new config keys, endpoints, or user-facing behaviour, document it in `README.md`. Keep additions concise — follow the existing section style.
### 8 — Run the full test suite
```bash
make test
```
All tests must pass. If any fail, fix them before proceeding. Do not skip or suppress failing tests.
### 9 — Live Docker test (new package type only)
**Skip this step if the issue does not add a new remote package type.**
If the issue adds a new package type (e.g. `deb`, `conda`, `cargo`, `rubygems`, or any type not already in `remotes.yaml`), do the following before committing.
#### 9a — Add a real test remote to remotes.yaml
Append a valid, publicly accessible remote of the new type to `remotes.yaml`. Use a real upstream URL and patterns that cover both an immutable file (versioned artifact) and a mutable file (index/metadata). Add a comment explaining which URLs to use for manual testing.
#### 9b — Start the stack
```bash
make docker-up
```
Wait until `curl -s http://localhost:8000/health` returns `{"status":"healthy"}`.
#### 9c — Test a mutable file (first fetch — cache miss)
Download the index or metadata file for the new remote. Confirm:
- HTTP 200
- `X-Artifact-Source: remote` header (or equivalent log line confirming a cache miss)
- Content looks correct (not empty, not an error page)
```bash
curl -sv "http://localhost:8000/api/v1/remote/<new-remote>/<mutable-path>" 2>&1 | grep -E "< HTTP|X-Artifact"
```
#### 9d — Test a mutable file (second fetch — cache hit)
Repeat the exact same request. Confirm:
- HTTP 200
- `X-Artifact-Source: cache`
```bash
curl -sv "http://localhost:8000/api/v1/remote/<new-remote>/<mutable-path>" 2>&1 | grep -E "< HTTP|X-Artifact"
```
#### 9e — Test an immutable file (first fetch — cache miss)
Download a versioned/immutable artifact. Confirm HTTP 200 and a cache-miss log line.
```bash
curl -sv "http://localhost:8000/api/v1/remote/<new-remote>/<immutable-path>" 2>&1 | grep -E "< HTTP|X-Artifact"
```
#### 9f — Test an immutable file (second fetch — cache hit)
Repeat. Confirm `X-Artifact-Source: cache`.
#### 9g — Check container logs
```bash
make docker-logs
```
Scan for:
- `Cache MISS` on first fetches, `Cache HIT` on second fetches
- `Cache ADD SUCCESS` with correct sizes
- No unhandled exceptions or ERROR lines
#### 9h — Exercise package-type tooling against the proxy
Use the native tooling for this package type to verify end-to-end behaviour. Examples:
| Package type | Command |
|---|---|
| `pypi` | `uv run --index-url http://localhost:8000/api/v1/remote/<remote>/simple <tool>` |
| `npm` | `npm install --registry http://localhost:8000/api/v1/remote/<remote>/ <pkg>` |
| `helm` | `helm repo add test http://localhost:8000/api/v1/remote/<remote> && helm search repo test && helm template test/<chart>` |
| `alpine` | `apk fetch --repository http://localhost:8000/api/v1/remote/<remote>/<branch>/<arch> <pkg>` |
| `rpm` | `dnf install --repofrompath ... <pkg>` or `repoquery` |
| `generic` | `curl` / `wget` as appropriate |
Confirm the tool resolves and downloads correctly through the proxy.
#### 9i — Tear down
```bash
make docker-down
```
Fix any failures found during 9b9h before moving on.
### 9.5 — Performance issues: measure before/after and gate the PR
**Skip this step if the issue is not a performance improvement.**
For performance issues, a PR is only warranted if there is a measurable gain. Use the Docker stack to compare before and after.
#### 9.5a — Baseline measurement (before)
Start the stack with the **unmodified** code (temporarily revert your change):
```bash
make docker-up
```
Warm or clear the cache as appropriate, then measure the relevant metric — e.g. concurrent request latency during a slow operation, response time for a specific endpoint, or throughput. Record the numbers.
#### 9.5b — Apply your change and rebuild
```bash
make docker-up # rebuilds the image
```
Repeat exactly the same measurement. Record the numbers.
#### 9.5c — Decide
If the improvement is not clearly measurable, **do not open a PR**. Instead:
1. Update the issue with your findings.
2. Note any conditions under which the improvement would be observable.
3. Skip steps 1114.
If the improvement is clear, proceed with the commit and PR. Include the before/after numbers in the PR description and the issue comment.
#### 9.5d — Tear down
```bash
make docker-down
```
### 10 — Build the wheel (smoke check)
```bash
uv build --wheel
```
Confirm the build succeeds.
### 11 — Stage and commit
Stage only the files you changed. Do not use `git add -A` or `git add .` — list files explicitly. Run:
```bash
git add <file1> <file2> ...
git commit
```
The commit message must:
- Start with a conventional-commit prefix (`feat:`, `fix:`, `refactor:`, `chore:`, etc.)
- Summarise the change in ≤ 72 characters on the first line
- Optionally include a short body explaining *why* (not *what*)
If the pre-commit hook auto-fixes files, re-stage the fixed files and commit again.
### 12 — Push the branch
```bash
git push origin <branch-name>
```
### 13 — Open a pull request
```bash
tea pulls create \
--base master \
--head <branch-name> \
--title "<same as commit subject>" \
--description "Closes #<N>\n\n## Summary\n<bullet points>\n\n## Test plan\n<what was verified>"
```
### 14 — Comment on the issue
```bash
tea comment <N> "<resolution comment>"
```
The comment must cover:
- **How it was resolved** — what changed and why
- **Issues encountered** — any non-obvious problems hit during implementation
- **Potential future improvements** — what could be done next
### 15 — Return to master
```bash
git checkout master
```
Report the PR URL and a one-sentence summary to the user.
+2
View File
@@ -7,6 +7,7 @@ repos:
- id: check-json
- id: check-added-large-files
args: ['--maxkb=500']
exclude: '^schemas/'
- id: check-merge-conflict
- id: check-shebang-scripts-are-executable
- id: check-symlinks
@@ -19,6 +20,7 @@ repos:
- id: end-of-file-fixer
- id: forbid-new-submodules
- id: pretty-format-json
args: ['--autofix']
- id: trailing-whitespace
# YAML linting
+1 -1
View File
@@ -3,7 +3,7 @@ when:
steps:
- name: kubeconform
image: git.unkin.net/unkin/almalinux9-kubetest:20260319
image: git.unkin.net/unkin/almalinux9-kubetest:20260606
commands:
- make kubeconform
backend_options:
+1 -1
View File
@@ -3,7 +3,7 @@ when:
steps:
- name: pre-commit
image: git.unkin.net/unkin/almalinux9-base:20260308
image: git.unkin.net/unkin/almalinux9-base:20260606
commands:
- uvx pre-commit run --all-files
backend_options:
+29
View File
@@ -0,0 +1,29 @@
when:
- event: pull_request
steps:
- name: vector-test
image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector:0.57.0-debian
commands:
# Dummy creds + writable dirs so the full topologies build; the unit tests
# only exercise the transforms (sources are not started).
- export CLICKHOUSE_USER=ci CLICKHOUSE_PASSWORD=ci
- export NATS_PRODUCER_PASSWORD=ci NATS_CONSUMER_PASSWORD=ci
- mkdir -p /vector-data-dir /etc/vault-ca
- cp /etc/ssl/certs/ca-certificates.crt /etc/vault-ca/ca.crt
# Transform tier + VM ingest: unit-tested transforms.
- vector test apps/base/logging/vector/aggregator.yaml apps/base/logging/vector/aggregator-tests.yaml
- vector test apps/base/logging/vector/vm-ingest.yaml apps/base/logging/vector/vm-ingest-tests.yaml
# Agent has no transforms to unit-test; validate it builds. (The archiver
# leg is now the logarchiver service, not a Vector pipeline.)
- vector validate --no-environment apps/base/logging/vector/agent.yaml
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 1Gi
cpu: 1
+261
View File
@@ -0,0 +1,261 @@
# AGENTS.md
## Project Overview
This is an **ArgoCD GitOps repository** that manages Kubernetes applications for the `au-syd1` cluster using a Kustomize + Helm pattern. Applications are deployed via ArgoCD ApplicationSets that watch directory patterns in this repo.
The migration pattern for this repo is: **Terragrunt/Terraform → ArgoCD** (see `migration.md` for full guide).
---
## Essential Commands
```bash
# Build and render manifests for a path (outputs to manifests/<path>/)
make build apps/overlays/au-syd1/<app-name>
make build clusters/au-syd1/bootstrap
# Validate all apps and clusters with kubeconform
make kubeconform
# Clean generated manifests
make clean
# Quick build + inspect without persisting output
kustomize build --enable-helm apps/overlays/au-syd1/<app-name>
# Check all resource kinds produced by an overlay
kustomize build --enable-helm apps/overlays/au-syd1/<app-name> | grep "^kind:" | sort | uniq -c
# Run pre-commit checks against all files
uvx pre-commit run --all-files
```
---
## Directory Structure
```
argocd-apps/
├── argocd/
│ ├── applicationsets/ # ArgoCD ApplicationSet definitions (platform.yaml, storage.yaml)
│ └── projects/ # ArgoCD AppProject definitions (platform.yaml, storage.yaml)
├── apps/
│ ├── base/ # Base Kustomize resources per app (no cluster-specific config)
│ │ └── <app-name>/
│ │ ├── kustomization.yaml
│ │ ├── namespace.yaml
│ │ ├── vaultauth.yaml # (if Vault-managed secrets)
│ │ └── vaultstaticsecret.yaml
│ └── overlays/
│ └── au-syd1/ # Cluster-specific overlays
│ └── <app-name>/
│ ├── kustomization.yaml # references base + helmCharts
│ └── values.yaml # Helm values for this cluster
├── clusters/
│ └── au-syd1/
│ ├── apps/ # Entry point: references apps/base (ArgoCD app-of-apps)
│ └── bootstrap/ # ArgoCD install + initial Application manifest
├── ci/
│ ├── validate-apps.sh # kubeconform over apps/overlays/*/kustomization.yaml
│ ├── validate-clusters.sh # kubeconform over clusters/*/kustomization.yaml
│ └── validate-no-secrets.sh # pre-commit hook: blocks plain Kubernetes Secrets
└── sources/ # Reference sources (Terraform configs, upstream charts, etc.)
└── terraform-k8s/ # Original Terraform configs — reference when migrating
```
---
## Adding a New Application
Follow these 10 steps (detailed in `migration.md`):
### 1. Create base resources
```
apps/base/<app-name>/
├── kustomization.yaml
├── namespace.yaml
├── vaultauth.yaml # if needed
└── vaultstaticsecret.yaml # if needed
```
### 2. Create cluster overlay
```
apps/overlays/au-syd1/<app-name>/
├── kustomization.yaml
└── values.yaml
```
**Overlay kustomization.yaml pattern:**
```yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../base/<app-name>
helmCharts:
- name: <chart-name>
repo: <helm-repo-url>
version: "<version>"
releaseName: <release-name>
namespace: <namespace>
valuesFile: values.yaml
```
### 3. Register in ApplicationSet
Add a directory entry to `argocd/applicationsets/platform.yaml` (or `storage.yaml` for `csi-*` apps):
```yaml
- path: apps/overlays/*/<app-name>
```
### 4. Update AppProject
In `argocd/projects/platform.yaml` (or `storage.yaml`):
- Add the Helm repo URL to `sourceRepos`
- Add the namespace to `destinations`
- Add any required cluster-scoped resource types to `clusterResourceWhitelist`
### 5. Validate
```bash
kustomize build --enable-helm apps/overlays/au-syd1/<app-name>
make kubeconform
```
---
## Secret Management
**Plain Kubernetes `Secret` objects are blocked** by the pre-commit hook. Use Vault Operator CRDs instead:
### VaultAuth template
```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: default
namespace: <namespace>
spec:
method: kubernetes
mount: k8s/au/syd1
vaultConnectionRef: vso-system/default
allowedNamespaces:
- <namespace>
kubernetes:
role: <role>
serviceAccount: <service-account>
audiences:
- vault
tokenExpirationSeconds: 600
```
### VaultStaticSecret template
```yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: <secret-name>
namespace: <namespace>
spec:
vaultAuthRef: default
mount: kv
type: kv-v2
path: kubernetes/namespace/<namespace>/default/<secret-name>
refreshAfter: 5m
destination:
name: <k8s-secret-name>
create: true
overwrite: true
hmacSecretData: true
```
---
## YAML Conventions
- **2-space indentation** (enforced by yamllint)
- All files must end with a newline (`end-of-file-fixer`)
- No trailing whitespace
- YAML linting uses relaxed rules with `line-length: disable` (long base64/URLs are fine)
- yamllint ignores `chart` directories (vendored Helm charts)
- `---` document separator at top of every YAML file
- Multiple documents in one file are allowed (e.g., `vaultstaticsecret.yaml` often contains multiple secrets)
---
## Kubernetes Labels Pattern
Use standard `app.kubernetes.io/*` labels consistently:
```yaml
labels:
app.kubernetes.io/component: <component>
app.kubernetes.io/instance: <release-name>
app.kubernetes.io/name: <app-name>
app.kubernetes.io/version: <version>
```
---
## Resource Naming Conventions
Files in `apps/base/<app-name>/` follow the pattern:
```
<kind>_<name>.yaml
```
Examples:
- `deployment_puppetserver-master.yaml`
- `cronjob_g10k-code.yaml`
- `configmap_puppetboard-config.yaml`
- `horizontalpodautoscaler_puppetserver-compilers-autoscaler.yaml`
- `service_puppet-headless.yaml`
---
## Helm Chart Vendoring
Some overlays vendor Helm charts locally under `apps/overlays/au-syd1/<app-name>/charts/<chart-name>/`. When a chart is vendored, the overlay's `kustomization.yaml` references the local path. When not vendored, it references the OCI or HTTP repo directly.
Current Kubernetes target version: **1.33.7** (used by kubeconform in CI).
---
## Project Boundaries
| Project | ApplicationSet | App pattern |
|------------|---------------------------|--------------------------|
| `platform` | `argocd/applicationsets/platform.yaml` | Named apps (cert-manager, puppet, woodpecker, etc.) |
| `storage` | `argocd/applicationsets/storage.yaml` | `csi-*` apps |
The `clusters/au-syd1/apps/` entry-point is deployed as a standalone ArgoCD `Application` (not an ApplicationSet) called `au-syd1-apps`.
---
## CI / Pre-commit Hooks
Runs on every PR via Woodpecker CI (`.woodpecker/`):
| Check | Tool | Trigger |
|---|---|---|
| YAML lint + general file checks | `pre-commit` (yamllint + pre-commit-hooks) | PR |
| No plain Secrets | `ci/validate-no-secrets.sh` | PR (staged files) |
| Kubernetes manifest validation | `kubeconform` via `make kubeconform` | PR |
kubeconform skips: `CustomResourceDefinition`, `GpuDevicePlugin` (for apps validation).
---
## Git Workflow
- Branch naming: `benvin/<app-name>` (user prefix)
- **Never `git add .`** — add only relevant files explicitly
- If pre-commit modifies files, `git add -u` then `git commit --amend --no-edit`
- Use `git push --force-with-lease` after amending
---
## Security Policies
- `reloader.stakater.com/auto: "true"` annotation triggers rolling restarts on ConfigMap/Secret changes
- Security contexts follow least-privilege: `drop: [all]` then add only required capabilities
- `fsGroup: 999` on pod security context for Puppet workloads
- `runAsUser: 0` is used only for init containers that need to set file permissions, then regular containers run as non-root
+5 -1
View File
@@ -1,4 +1,4 @@
.PHONY: build clean
.PHONY: build clean schemas
# Build a kustomization path to manifests directory
# Usage: make build clusters/au-syd1/bootstrap
@@ -6,6 +6,10 @@ build:
@mkdir -p manifests/$(filter-out $@,$(MAKECMDGOALS))
@kustomize build --enable-helm $(filter-out $@,$(MAKECMDGOALS)) --output manifests/$(filter-out $@,$(MAKECMDGOALS))
# Generate JSON schemas from CRDs and Kubernetes swagger spec (run manually, results committed)
schemas:
@ci/generate-schemas.sh schemas
# kubeconform
kubeconform:
@ci/validate-apps.sh && \
+45
View File
@@ -0,0 +1,45 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: age-api
namespace: age-api
spec:
replicas: 1
selector:
matchLabels:
app: age-api
template:
metadata:
annotations:
configmap.reloader.stakater.com/auto: "true"
labels:
app: age-api
spec:
containers:
- name: age-api
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/age-api:v0.1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
name: http
protocol: TCP
env:
- name: CONFIG_PATH
value: /etc/age-api/config.yaml
resources:
limits:
cpu: 100m
memory: 64Mi
requests:
cpu: 10m
memory: 32Mi
volumeMounts:
- mountPath: /etc/age-api/config.yaml
name: config
subPath: config.yaml
restartPolicy: Always
volumes:
- name: config
configMap:
name: age-api-config
+37
View File
@@ -0,0 +1,37 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
traefik.io/instance: internal
annotations:
cert-manager.io/cluster-issuer: vault-issuer
cert-manager.io/common-name: age-api.k8s.syd1.au.unkin.net
cert-manager.io/private-key-size: "4096"
external-dns.alpha.kubernetes.io/hostname: age-api.k8s.syd1.au.unkin.net
external-dns.alpha.kubernetes.io/target: 198.18.200.4
name: age-api
namespace: age-api
spec:
gatewayClassName: traefik-internal
listeners:
- allowedRoutes:
namespaces:
from: Same
hostname: age-api.k8s.syd1.au.unkin.net
name: http
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: Same
hostname: age-api.k8s.syd1.au.unkin.net
name: https
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ""
kind: Secret
name: age-api-tls
mode: Terminate
+49
View File
@@ -0,0 +1,49 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: age-api-http-redirect
namespace: age-api
spec:
hostnames:
- age-api.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: age-api
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: age-api
namespace: age-api
spec:
hostnames:
- age-api.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: age-api
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: age-api
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
+18
View File
@@ -0,0 +1,18 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- gateway.yaml
- httproute.yaml
- namespace.yaml
- service.yaml
- vpa.yaml
configMapGenerator:
- name: age-api-config
files:
- config.yaml=resources/config.yaml
options:
disableNameSuffixHash: true
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: age-api
+7
View File
@@ -0,0 +1,7 @@
people:
- name: jaidi
birthtime: 1773135720
- name: ben
birthtime: 559663200
- name: sudaporn
birthtime: 686757600
+17
View File
@@ -0,0 +1,17 @@
---
apiVersion: v1
kind: Service
metadata:
name: age-api
namespace: age-api
spec:
internalTrafficPolicy: Cluster
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: age-api
sessionAffinity: None
type: ClusterIP
+13
View File
@@ -0,0 +1,13 @@
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: age-api-vpa
namespace: age-api
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: age-api
updatePolicy:
updateMode: "Off"
+92
View File
@@ -0,0 +1,92 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: artifactapi
annotations:
configmap.reloader.stakater.com/auto: "true"
secret.reloader.stakater.com/reload: "vault-ca-cert"
spec:
selector:
matchLabels:
app: api
strategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: api
spec:
automountServiceAccountToken: true
initContainers:
- name: combine-certs
image: alpine:3
command:
- sh
- -c
- cat /etc/ssl/certs/ca-certificates.crt /custom-ca/ca.crt > /combined-certs/ca-certificates.crt
volumeMounts:
- name: vault-ca-cert
mountPath: /custom-ca
readOnly: true
- name: combined-certs
mountPath: /combined-certs
containers:
- name: api
image: git.unkin.net/unkin/artifactapi:v3.7.7
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8000
name: http
protocol: TCP
envFrom:
- configMapRef:
name: api-env
optional: false
- secretRef:
name: environment
optional: false
volumeMounts:
- name: combined-certs
mountPath: /etc/ssl/combined
readOnly: true
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: http
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: http
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: "1"
memory: 4Gi
requests:
cpu: 100m
memory: 256Mi
volumes:
- name: vault-ca-cert
secret:
secretName: vault-ca-cert
items:
- key: ca.crt
path: ca.crt
- name: combined-certs
emptyDir: {}
restartPolicy: Always
@@ -2,13 +2,13 @@
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: artifactapi-hpa
name: api-hpa
namespace: artifactapi
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: artifactapi-deployment
name: api
minReplicas: 2
maxReplicas: 10
metrics:
@@ -1,92 +0,0 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: artifactapi-deployment
namespace: artifactapi
annotations:
reloader.stakater.com/auto: "true"
spec:
selector:
matchLabels:
app: artifactapi
strategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
template:
spec:
automountServiceAccountToken: true
containers:
- name: artifactapi
image: git.unkin.net/unkin/artifactapi:v2.7.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8000
name: http
protocol: TCP
envFrom:
- configMapRef:
name: artifactapi-env
optional: false
- secretRef:
name: environment
optional: false
livenessProbe:
failureThreshold: 3
httpGet:
path: /health
port: http
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
failureThreshold: 3
httpGet:
path: /health
port: http
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: "1"
memory: 4Gi
requests:
cpu: 100m
memory: 256Mi
volumeMounts:
- mountPath: /etc/artifactapi/conf.d/config.yaml
name: remotes-config
subPath: config.yaml
- mountPath: /etc/artifactapi/conf.d/local-generic.yaml
name: remotes-config
subPath: local-generic.yaml
- mountPath: /etc/artifactapi/conf.d/remote-alpine.yaml
name: remotes-config
subPath: remote-alpine.yaml
- mountPath: /etc/artifactapi/conf.d/remote-docker.yaml
name: remotes-config
subPath: remote-docker.yaml
- mountPath: /etc/artifactapi/conf.d/remote-generic.yaml
name: remotes-config
subPath: remote-generic.yaml
- mountPath: /etc/artifactapi/conf.d/remote-helm.yaml
name: remotes-config
subPath: remote-helm.yaml
- mountPath: /etc/artifactapi/conf.d/remote-rpm.yaml
name: remotes-config
subPath: remote-rpm.yaml
- mountPath: /etc/artifactapi/conf.d/virtual-helm.yaml
name: remotes-config
subPath: virtual-helm.yaml
restartPolicy: Always
volumes:
- configMap:
name: remotes-config
optional: false
name: remotes-config
+56
View File
@@ -0,0 +1,56 @@
---
# Ceph RGW (S3) backup target for the artifactapi CNPG cluster, provisioned by the
# in-estate cephrgw-operator. One dedicated bucket + owner user per cluster:
# cephrgw CRs are namespace-scoped and CNPG reads its S3 credential Secret from
# its own namespace, so backups are per-database rather than one shared bucket.
apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
name: cnpg-artifactapi-backup
namespace: artifactapi
spec:
displayName: "CNPG backup owner (artifactapi)"
# RGW users are global; keep the uid namespace-qualified so it never collides.
uid: cnpg-artifactapi-backup
maxBuckets: 5
# Operator writes AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ RGW_UID,
# S3_ENDPOINT) into this Secret; the Cluster's barmanObjectStore consumes it.
secretName: cnpg-artifactapi-backup-s3
# Keep the RGW user (and thus the keys) if this CR is ever deleted, so an
# in-flight restore can still reach the archive.
retainOnDelete: true
---
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: cnpg-artifactapi
namespace: artifactapi
spec:
placementTarget: ec
bucketName: cnpg-artifactapi
# The owner user has full control of its own bucket (read + write), which is
# all the backup/restore identity needs — no extra BucketAccess grant.
ownerRef: cnpg-artifactapi-backup
versioning: false
tags:
app: artifactapi
purpose: cnpg-backup
# Never drop the backups if the CR is removed; retire buckets by hand.
retainOnDelete: true
---
# Nightly base backup. Continuous WAL archiving is always-on via the Cluster's
# spec.backup.barmanObjectStore; this schedules the periodic full backup that
# WAL is layered on top of. Schedules are staggered across clusters so the 8
# base backups do not hit RGW at once (CNPG cron is 6-field, seconds first).
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: cnpg-artifactapi-nightly
namespace: artifactapi
spec:
schedule: "0 40 1 * * *"
immediate: false
backupOwnerReference: self
method: barmanObjectStore
cluster:
name: postgres
+120
View File
@@ -0,0 +1,120 @@
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres
namespace: artifactapi
spec:
affinity:
podAntiAffinityType: preferred
backup:
# 30-day retention (DEFAULT — adjust per cluster if needed). Enforced by CNPG
# against the object store on each successful base backup.
retentionPolicy: 30d
barmanObjectStore:
# Dedicated per-cluster Ceph RGW bucket (cephrgw-operator provisions it).
destinationPath: s3://cnpg-artifactapi
endpointURL: https://s3.ceph.unkin.net
# radosgw serves a Vault-PKI cert; trust the internal CA (reflected into
# every namespace as the vault-ca-cert Secret).
endpointCA:
name: vault-ca-cert
key: ca.crt
# Keys minted by the ObjectStoreUser in cnpg_backup.yaml; never hardcoded.
s3Credentials:
accessKeyId:
name: cnpg-artifactapi-backup-s3
key: AWS_ACCESS_KEY_ID
secretAccessKey:
name: cnpg-artifactapi-backup-s3
key: AWS_SECRET_ACCESS_KEY
# Path prefix within the bucket; keep stable across restores (see docs).
serverName: artifactapi
data:
compression: bzip2
jobs: 2
wal:
compression: zstd
maxParallel: 2
bootstrap:
initdb:
database: artifacts
encoding: UTF8
localeCType: C
localeCollate: C
owner: artifacts
secret:
name: postgres-credentials
enablePDB: true
enableSuperuserAccess: false
failoverDelay: 0
imageName: ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie
instances: 3
logLevel: info
maxSyncReplicas: 0
minSyncReplicas: 0
monitoring:
customQueriesConfigMap:
- key: queries
name: cnpg-default-monitoring
disableDefaultQueries: false
enablePodMonitor: false
postgresql:
parameters:
archive_mode: "on"
archive_timeout: 5min
dynamic_shared_memory_type: posix
effective_cache_size: 256MB
full_page_writes: "on"
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
log_rotation_age: "0"
log_rotation_size: "0"
log_truncate_on_rotation: "false"
logging_collector: "on"
max_connections: "200"
max_parallel_workers: "16"
max_replication_slots: "16"
max_worker_processes: "16"
shared_buffers: 128MB
shared_memory_type: mmap
ssl_max_protocol_version: TLSv1.3
ssl_min_protocol_version: TLSv1.3
wal_keep_size: 256MB
wal_level: logical
wal_log_hints: "on"
wal_receiver_timeout: 5s
wal_sender_timeout: 5s
syncReplicaElectionConstraint:
enabled: false
primaryUpdateMethod: restart
primaryUpdateStrategy: unsupervised
probes:
liveness:
isolationCheck:
connectionTimeout: 1000
enabled: true
requestTimeout: 1000
replicationSlots:
highAvailability:
enabled: true
slotPrefix: _cnpg_
synchronizeReplicas:
enabled: true
updateInterval: 30
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 50m
memory: 256Mi
smartShutdownTimeout: 180
startDelay: 3600
stopDelay: 1800
storage:
resizeInUseVolumes: true
size: 20Gi
storageClass: cephrbd-fast-delete
switchoverDelay: 3600
+33
View File
@@ -0,0 +1,33 @@
---
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
name: postgres-pooler
namespace: artifactapi
spec:
cluster:
name: postgres
instances: 2
pgbouncer:
parameters:
default_pool_size: "100"
max_client_conn: "400"
paused: false
poolMode: session
template:
metadata:
labels:
app: pooler
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pooler
topologyKey: kubernetes.io/hostname
containers: []
type: rw
+5 -16
View File
@@ -2,26 +2,15 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: artifactapi-env
name: api-env
namespace: artifactapi
data:
CONFIG_PATH: /etc/artifactapi/conf.d/
DBHOST: postgres-service
DBHOST: postgres-pooler
DBNAME: artifacts
DBPORT: "5432"
DBUSER: artifacts
MINIO_BUCKET: artifactapi
MINIO_BUCKET: artifactapi-prod-k8s-syd1-au
MINIO_ENDPOINT: radosgw.service.consul
MINIO_SECURE: "true"
REDIS_URL: redis://redis-service:6379
REQUESTS_CA_BUNDLE: /etc/pki/tls/certs/ca-bundle.crt
SSL_CERT_FILE: /etc/pki/tls/certs/ca-bundle.crt
---
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-env
namespace: artifactapi
data:
POSTGRES_DB: artifacts
POSTGRES_USER: artifacts
REDIS_URL: redis://redis:6379
SSL_CERT_FILE: /etc/ssl/combined/ca-certificates.crt
+37
View File
@@ -0,0 +1,37 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
traefik.io/instance: internal
annotations:
cert-manager.io/cluster-issuer: vault-issuer
cert-manager.io/common-name: artifactapi.k8s.syd1.au.unkin.net
cert-manager.io/private-key-size: "4096"
external-dns.alpha.kubernetes.io/hostname: artifactapi.k8s.syd1.au.unkin.net
external-dns.alpha.kubernetes.io/target: 198.18.200.4
name: artifactapi
namespace: artifactapi
spec:
gatewayClassName: traefik-internal
listeners:
- allowedRoutes:
namespaces:
from: Same
hostname: artifactapi.k8s.syd1.au.unkin.net
name: http
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: Same
hostname: artifactapi.k8s.syd1.au.unkin.net
name: https
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ""
kind: Secret
name: artifactapi-tls
mode: Terminate
+41
View File
@@ -0,0 +1,41 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: api-route
namespace: artifactapi
spec:
hostnames:
- artifactapi.k8s.syd1.au.unkin.net
parentRefs:
# Early-boot clients (anaconda/kickstart, yum in %post, PXE) need plain HTTP
# for the rpm repos; serve the app directly on port 80 instead of redirecting.
- group: gateway.networking.k8s.io
kind: Gateway
name: artifactapi
sectionName: http
- group: gateway.networking.k8s.io
kind: Gateway
name: artifactapi
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: ui
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /ui
- backendRefs:
- group: ""
kind: Service
name: artifactapi
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
-32
View File
@@ -1,32 +0,0 @@
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: vault-issuer
cert-manager.io/common-name: artifactapi.k8s.syd1.au.unkin.net
cert-manager.io/private-key-size: "4096"
external-dns.alpha.kubernetes.io/hostname: artifactapi.k8s.syd1.au.unkin.net
external-dns.alpha.kubernetes.io/target: 198.18.200.0
nginx.ingress.kubernetes.io/proxy-body-size: 10g
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
name: artifactapi-ingress
namespace: artifactapi
spec:
ingressClassName: nginx
rules:
- host: artifactapi.k8s.syd1.au.unkin.net
http:
paths:
- backend:
service:
name: artifactapi-api
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- artifactapi.k8s.syd1.au.unkin.net
secretName: artifactapi-tls
+10 -19
View File
@@ -3,28 +3,19 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- artifactapi-deployment.yaml
- artifactapi-hpa.yaml
- api-deployment.yaml
- api-hpa.yaml
- configmap.yaml
- ingress.yaml
- cnpg_cluster.yaml
- cnpg_backup.yaml
- cnpg_pooler.yaml
- gateway.yaml
- httproute.yaml
- namespace.yaml
- postgres-deployment.yaml
- pvc.yaml
- redis-deployment.yaml
- services.yaml
- ui-deployment.yaml
- ui-hpa.yaml
- vaultauth.yaml
- vaultstaticsecret.yaml
configMapGenerator:
- name: remotes-config
files:
- resources/conf.d/config.yaml
- resources/conf.d/local-generic.yaml
- resources/conf.d/remote-generic.yaml
- resources/conf.d/remote-alpine.yaml
- resources/conf.d/remote-rpm.yaml
- resources/conf.d/remote-docker.yaml
- resources/conf.d/remote-helm.yaml
- resources/conf.d/virtual-helm.yaml
options:
disableNameSuffixHash: true
- vpa.yaml
@@ -1,76 +0,0 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-deployment
namespace: artifactapi
annotations:
reloader.stakater.com/auto: "true"
spec:
replicas: 1
selector:
matchLabels:
app: postgres
strategy:
type: Recreate
template:
spec:
automountServiceAccountToken: true
containers:
- name: postgres
image: postgres:15-alpine
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5432
name: postgres
protocol: TCP
envFrom:
- configMapRef:
name: postgres-env
optional: false
- secretRef:
name: postgres-password
optional: false
readinessProbe:
exec:
command:
- pg_isready
- -U
- artifacts
- -d
- artifacts
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
livenessProbe:
exec:
command:
- pg_isready
- -U
- artifacts
- -d
- artifacts
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 500m
memory: 1Gi
requests:
cpu: 50m
memory: 128Mi
volumeMounts:
- mountPath: /var/lib/postgresql/data
mountPropagation: None
name: pgdata
subPath: pgdata
restartPolicy: Always
volumes:
- name: pgdata
persistentVolumeClaim:
claimName: artifactapi-postgres-pgdata
-28
View File
@@ -1,28 +0,0 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: artifactapi-postgres-pgdata
namespace: artifactapi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: cephrbd-fast-delete
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: artifactapi-redis-data
namespace: artifactapi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: cephrbd-fast-delete
volumeMode: Filesystem
+11 -21
View File
@@ -2,23 +2,21 @@
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
name: redis-deployment
name: redis
namespace: artifactapi
spec:
replicas: 1
selector:
matchLabels:
app: redis
strategy:
type: Recreate
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:7-alpine
imagePullPolicy: IfNotPresent
command:
- redis-server
- --save
@@ -28,6 +26,13 @@ spec:
- containerPort: 6379
name: redis
protocol: TCP
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 50m
memory: 128Mi
livenessProbe:
exec:
command:
@@ -48,19 +53,4 @@ spec:
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 50m
memory: 128Mi
volumeMounts:
- mountPath: /data
mountPropagation: None
name: data
restartPolicy: Always
volumes:
- name: data
persistentVolumeClaim:
claimName: artifactapi-redis-data
@@ -1,3 +0,0 @@
# Global artifactapi configuration.
# S3, Redis, and database connection settings are injected via environment variables.
# Add any top-level overrides here if needed.
@@ -1,7 +0,0 @@
locals:
local-generic:
package: "generic"
description: "Local generic file repository"
cache:
immutable_ttl: 0
mutable_ttl: 0
@@ -1,10 +0,0 @@
remotes:
alpine:
base_url: "https://dl-cdn.alpinelinux.org"
package: "alpine"
description: "Alpine Linux APK package repository"
immutable_patterns:
- ".*/x86_64/.*\\.apk$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
@@ -1,96 +0,0 @@
remotes:
ghcr:
base_url: "https://ghcr.io"
package: "docker"
description: "GitHub Container Registry"
immutable_patterns:
- "^cloudnative-pg/cloudnative-pg"
- "^emberstack/helm-charts"
- "^openvoxproject/"
- "^stakater/reloader"
- "^voxpupuli/puppetboard"
- "^woodpecker-ci/helm"
cache:
immutable_ttl: 0
mutable_ttl: 300
dockerhub:
base_url: "https://registry-1.docker.io"
package: "docker"
description: "Docker Hub registry"
immutable_patterns:
- "^library/almalinux"
- "^library/busybox"
- "^library/debian"
- "^library/fedora"
- "^library/nginx"
- "^library/postgres"
- "^library/redis"
- "^beats/filebeat"
- "^bitnami/"
- "^curlimages/curl"
- "^emberstack/kubernetes-reflector"
- "^hashicorp/vault-secrets-operator"
- "^jfrog/"
- "^rancher/"
- "^traefik/"
- "^ubi9/ubi-minimal"
- "^victoriametrics/"
- "^woodpeckerci/"
cache:
immutable_ttl: 0
mutable_ttl: 300
quay:
base_url: "https://quay.io"
package: "docker"
description: "Quay.io container registry"
immutable_patterns:
- "^brancz/kube-rbac-proxy"
- "^cephcsi/cephcsi"
- "^jetstack/cert-manager-"
cache:
immutable_ttl: 0
mutable_ttl: 300
k8s-registry:
base_url: "https://registry.k8s.io"
package: "docker"
description: "Kubernetes container registry"
immutable_patterns:
- "^external-dns/external-dns"
- "^sig-storage/"
cache:
immutable_ttl: 0
mutable_ttl: 300
gitlab:
base_url: "https://registry.gitlab.com"
package: "docker"
description: "GitLab container registry"
immutable_patterns:
- "^purelb/purelb"
cache:
immutable_ttl: 0
mutable_ttl: 300
elastic:
base_url: "https://docker.elastic.co"
package: "docker"
description: "Elastic container registry"
immutable_patterns:
- "^eck/eck-operator"
cache:
immutable_ttl: 0
mutable_ttl: 300
gcr:
base_url: "https://gcr.io"
package: "docker"
description: "Google Container Registry"
immutable_patterns:
- "^k8s-staging-nfd/charts"
- "^k8s-staging-nfd/node-feature-discovery"
cache:
immutable_ttl: 0
mutable_ttl: 300
@@ -1,130 +0,0 @@
remotes:
github:
base_url: "https://github.com"
package: "generic"
description: "GitHub releases and files"
mutable_patterns:
- ".*/archive/refs/heads/.*.tar.gz$"
immutable_patterns:
- ".*/archive/refs/tags/.*.tar.gz$"
- "ahmetb/kubectx/.*/kubectx_.*_linux_x86_64.tar.gz$"
- "ahmetb/kubectx/.*/kubens_.*_linux_x86_64.tar.gz$"
- "apple/foundationdb/.*/libfdb_c.x86_64.so$"
- "astral-sh/ruff/.*/ruff-x86_64-unknown-linux-gnu.tar.gz$"
- "astral-sh/uv/.*/uv-x86_64-unknown-linux-gnu.tar.gz$"
- "camptocamp/prometheus-puppetdb-exporter/.*/prometheus-puppetdb-exporter-.*.linux-amd64.tar.gz$"
- "coder/code-server/.*/code-server-.*-amd64.rpm$"
- "containernetworking/plugins/.*/cni-plugins-linux-amd64-.*.tgz"
- "dandavison/delta/.*/delta-.*-x86_64-unknown-linux-musl.tar.gz$"
- "ducaale/xh/.*/xh-.*-x86_64-unknown-linux-musl.tar.gz$"
- "etcd-io/etcd/.*/etcd-.*-linux-amd64.tar.gz$"
- "getsops/sops/.*/sops-v.*\\.linux\\.amd64$"
- "grafana/jsonnet-language-server/.*/jsonnet-language-server_.*_linux_amd64$"
- "gruntwork-io/boilerplate/.*/boilerplate_linux_amd64$"
- "gruntwork-io/terragrunt/.*terragrunt_linux_amd64.*"
- "hadolint/hadolint/.*/hadolint-linux-x86_64$"
- "helmfile/helmfile/.*/helmfile_.*_linux_amd64.tar.gz$"
- "helmfile/vals/.*/vals_.*_linux_amd64.tar.gz$"
- "jesseduffield/lazydocker/.*/lazydocker_.*_Linux_x86_64.tar.gz$"
- "kubecolor/kubecolor/.*/kubecolor_.*_linux_amd64.tar.gz$"
- "kubernetes-sigs/gateway-api/.*/standard-install.yaml$"
- "kubernetes-sigs/kustomize/.*/kustomize_.*_linux_amd64.tar.gz$"
- "lxc/incus/.*.tar.gz$"
- "mikefarah/yq/.*/yq_linux_amd64$"
- "neovim/neovim-releases/.*/nvim-linux-x86_64.tar.gz$"
- "neovim/neovim/.*/nvim-linux-x86_64.tar.gz$"
- "nzbgetcom/nzbget/.*/nzbget-.*.x86_64.rpm$"
- "onedr0p/exportarr/.*/exportarr_.*_linux_amd64.tar.gz$"
- "openbao/openbao-plugins/.*/openbao-plugin-secrets-consul_linux_amd64_.*.tar.gz$"
- "openbao/openbao-plugins/.*/openbao-plugin-secrets-nomad_linux_amd64_.*.tar.gz$"
- "prometheus-community/bind_exporter/.*/bind_exporter-.*.linux-amd64.tar.gz$"
- "prometheus-community/pgbouncer_exporter/.*/pgbouncer_exporter-.*.linux-amd64.tar.gz$"
- "prometheus-community/postgres_exporter/.*/postgres_exporter-.*.linux-amd64.tar.gz$"
- "prometheus/node_exporter/.*/node_exporter-.*.linux-amd64.tar.gz$"
- "rancher/rke2/.*/rke2-images.linux-amd64.tar.zst$"
- "stalwartlabs/stalwart/.*/stalwart-cli-x86_64-unknown-linux-gnu.tar.gz$"
- "stalwartlabs/stalwart/.*/stalwart-foundationdb-x86_64-unknown-linux-gnu.tar.gz$"
- "stalwartlabs/stalwart/.*/stalwart-x86_64-unknown-linux-gnu.tar.gz$"
- "starship/starship/.*/starship-x86_64-unknown-linux-musl.tar.gz$"
- "stern/stern/.*/stern_.*_linux_amd64.tar.gz$"
- "terraform-linters/tflint/.*/tflint_linux_amd64.zip$"
- "tynany/frr_exporter/.*/frr_exporter-.*.linux-amd64.tar.gz$"
- "VictoriaMetrics/VictoriaLogs/.*/victoria-logs-linux-amd64-.*.tar.gz$"
- "VictoriaMetrics/VictoriaLogs/.*/vlutils-linux-amd64-.*.tar.gz$"
- "VictoriaMetrics/VictoriaMetrics/.*/victoria-logs-linux-amd64-.*.tar.gz$"
- "VictoriaMetrics/VictoriaMetrics/.*/victoria-metrics-linux-amd64-.*-cluster.tar.gz$"
- "VictoriaMetrics/VictoriaMetrics/.*/vlutils-linux-amd64-.*.tar.gz$"
- "VictoriaMetrics/VictoriaMetrics/.*/vmutils-linux-amd64-.*.tar.gz$"
- "xorpaul/g10k/.*/g10k-.*-linux-amd64.zip$"
- "yannh/kubeconform/.*/kubeconform-linux-amd64.tar.gz$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
github_user:
base_url: "https://raw.githubusercontent.com"
package: "generic"
description: "GitHub User Content"
immutable_patterns:
- "argoproj/argo-cd/.*.yaml$"
- "yannh/kubernetes-json-schema/master/.*.json$"
- "datreeio/CRDs-catalog/main/.*.json$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
gitea-dl:
base_url: "https://dl.gitea.com"
package: "generic"
description: "Gitea download site"
immutable_patterns:
- "act_runner/.*/act_runner-.*-linux-amd64$"
- "tea/.*/tea-.*-linux-amd64$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
hashicorp-releases:
base_url: "https://releases.hashicorp.com"
package: "generic"
description: "HashiCorp product releases"
immutable_patterns:
- "terraform/.*terraform_.*_linux_amd64\\.zip$"
- "terraform/.*terraform_.*_windows_amd64\\.zip$"
- "terraform/.*terraform_.*_darwin_amd64\\.zip$"
- "vault/.*vault_.*_linux_amd64\\.zip$"
- "vault/.*vault_.*_windows_amd64\\.zip$"
- "vault/.*vault_.*_darwin_amd64\\.zip$"
- "consul-cni/.*/consul-cni_.*_linux_amd64\\.zip$"
- "consul/.*/consul_.*_linux_amd64\\.zip$"
- "nomad-autoscaler/.*/nomad-autoscaler_.*_linux_amd64\\.zip$"
- "nomad/.*/nomad_.*_linux_amd64\\.zip$"
- "packer/.*/packer_.*_linux_amd64\\.zip$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
rarlab:
base_url: "https://www.rarlab.com"
package: "generic"
description: "RARLab"
immutable_patterns:
- "rar/rarlinux-x64-.*.tar.gz"
cache:
immutable_ttl: 0
mutable_ttl: 7200
claude-ai:
base_url: "https://downloads.claude.ai"
package: "generic"
description: "Anthropic Claude Code binary releases"
mutable_patterns:
- "claude-code-releases/.*/manifest.json$"
immutable_patterns:
- "claude-code-releases/.*/linux-x64/claude$"
- "claude-code-releases/.*/linux-arm64/claude$"
- "claude-code-releases/.*/linux-x64-musl/claude$"
- "claude-code-releases/.*/linux-arm64-musl/claude$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
@@ -1,143 +0,0 @@
remotes:
ceph-csi:
base_url: "https://ceph.github.io/csi-charts"
package: "helm"
description: "Ceph CSI driver Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
cnpg:
base_url: "https://cloudnative-pg.github.io/charts"
package: "helm"
description: "CloudNativePG operator Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
elastic-helm:
base_url: "https://helm.elastic.co"
package: "helm"
description: "Elastic stack Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
external-dns:
base_url: "https://kubernetes-sigs.github.io/external-dns/"
package: "helm"
description: "ExternalDNS Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
hashicorp-helm:
base_url: "https://helm.releases.hashicorp.com"
package: "helm"
description: "HashiCorp Helm charts (Vault Secrets Operator, etc.)"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
intel-helm:
base_url: "https://intel.github.io/helm-charts/"
package: "helm"
description: "Intel Helm charts (device plugins)"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
jetstack:
base_url: "https://charts.jetstack.io"
package: "helm"
description: "Jetstack Helm charts (cert-manager)"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
purelb:
base_url: "https://gitlab.com/api/v4/projects/20400619/packages/helm/stable"
package: "helm"
description: "PureLB load balancer Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
rancher-stable:
base_url: "https://releases.rancher.com/server-charts/stable"
package: "helm"
description: "Rancher stable Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
stakater:
base_url: "https://stakater.github.io/stakater-charts"
package: "helm"
description: "Stakater Helm charts (Reloader)"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
traefik:
base_url: "https://traefik.github.io/charts"
package: "helm"
description: "Traefik Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
victoriametrics:
base_url: "https://victoriametrics.github.io/helm-charts/"
package: "helm"
description: "VictoriaMetrics observability Helm charts"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
argo-helm:
base_url: "https://argoproj.github.io/argo-helm"
package: "helm"
description: "Argo Project Helm charts (ArgoCD, Image Updater, Rollouts, etc.)"
check_mutable_updates: true
immutable_patterns:
- "\\.tgz$"
cache:
immutable_ttl: 0
mutable_ttl: 3600
@@ -1,154 +0,0 @@
remotes:
almalinux:
base_url: "https://gsl-syd.mm.fcix.net/almalinux"
package: "rpm"
description: "AlmaLinux RPM package repository"
immutable_patterns:
- ".*/x86_64/.*\\.rpm$"
- ".*/noarch/.*\\.rpm$"
- ".*/repodata/.*\\.sqlite.*$"
- ".*/repodata/.*\\.xml.*$"
- ".*/repodata/.*\\.yaml.*$"
- ".*/install.img"
- ".*/squashfs.img"
- ".*/updates.img"
- ".*/RPM-GPG-KEY-.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
ceph-reef:
base_url: "https://download.ceph.com/rpm-reef/"
package: "rpm"
description: "Ceph Reef 18"
immutable_patterns:
- ".*/x86_64/.*\\.rpm$"
- ".*/noarch/.*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
ceph-squid:
base_url: "https://download.ceph.com/rpm-squid/"
package: "rpm"
description: "Ceph Squid 19"
immutable_patterns:
- ".*/x86_64/.*\\.rpm$"
- ".*/noarch/.*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
ceph-tentacle:
base_url: "https://download.ceph.com/rpm-tentacle/"
package: "rpm"
description: "Ceph Tentacle 20"
immutable_patterns:
- ".*/x86_64/.*\\.rpm$"
- ".*/noarch/.*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
epel:
base_url: "https://gsl-syd.mm.fcix.net/epel"
package: "rpm"
description: "EPEL (Extra Packages for Enterprise Linux)"
immutable_patterns:
- ".*/Everything/x86_64/.*\\.rpm$"
- ".*/noarch/.*\\.rpm$"
- ".*/repodata/.*\\.sqlite.*$"
- ".*/repodata/.*\\.xml.*$"
- ".*/repodata/.*\\.yaml.*$"
- "RPM-GPG-KEY-.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
fedora:
base_url: "https://gsl-syd.mm.fcix.net/fedora/linux"
package: "rpm"
description: "Fedora Linux RPM package repository"
immutable_patterns:
- "releases/.*/Everything/x86_64/.*\\.rpm$"
- "updates/.*/Everything/x86_64/.*\\.rpm$"
- "development/.*/Everything/x86_64/.*\\.rpm$"
- ".*/noarch/.*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
frr:
base_url: "https://rpm.frrouting.org/repo"
package: "rpm"
description: "FRR RPM package repository"
immutable_patterns:
- ".*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
mariadb:
base_url: "http://mariadb.mirror.digitalpacific.com.au/yum"
package: "rpm"
description: "MariaDB RPM package repository"
immutable_patterns:
- ".*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
- ".*/RPM-GPG-KEY-.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
openvox:
base_url: "https://yum.voxpupuli.org"
package: "rpm"
description: "OpenVox RPM package repository"
immutable_patterns:
- ".*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
- "GPG-KEY-.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
postgresql:
base_url: "https://download.postgresql.org/pub/repos/yum"
package: "rpm"
description: "PostgreSQL RPM package repository"
immutable_patterns:
- ".*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
- ".*/RPM-GPG-KEY-.*$"
- ".*/PGDG-RPM-GPG-KEY-.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
rke2:
base_url: "https://rpm.rancher.io"
package: "rpm"
description: "RKE2 RPM package repository"
immutable_patterns:
- ".*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
- "public.key$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
zfs:
base_url: "http://download.zfsonlinux.org"
package: "rpm"
description: "ZFS RPM package repository"
immutable_patterns:
- ".*\\.rpm$"
- ".*/repodata/.*\\.xml.*$"
cache:
immutable_ttl: 0
mutable_ttl: 7200
@@ -1,18 +0,0 @@
virtuals:
helm:
package: "helm"
description: "Virtual repository merging all helm remotes — member order is priority order for duplicate chart+version"
members:
- ceph-csi
- cnpg
- elastic-helm
- external-dns
- hashicorp-helm
- intel-helm
- jetstack
- purelb
- rancher-stable
- stakater
- traefik
- victoriametrics
- argo-helm
+8 -8
View File
@@ -2,7 +2,7 @@
apiVersion: v1
kind: Service
metadata:
name: artifactapi-api
name: artifactapi
namespace: artifactapi
spec:
internalTrafficPolicy: Cluster
@@ -12,31 +12,31 @@ spec:
protocol: TCP
targetPort: http
selector:
app: artifactapi
app: api
sessionAffinity: None
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: postgres-service
name: ui
namespace: artifactapi
spec:
internalTrafficPolicy: Cluster
ports:
- name: postgres
port: 5432
- name: http
port: 80
protocol: TCP
targetPort: postgres
targetPort: http
selector:
app: postgres
app: ui
sessionAffinity: None
type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
name: redis-service
name: redis
namespace: artifactapi
spec:
internalTrafficPolicy: Cluster
+58
View File
@@ -0,0 +1,58 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ui
namespace: artifactapi
annotations:
configmap.reloader.stakater.com/auto: "true"
spec:
selector:
matchLabels:
app: ui
strategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: ui
spec:
automountServiceAccountToken: true
containers:
- name: ui
image: git.unkin.net/unkin/artifactapi-ui:v3.7.7
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: http
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
path: /ui
port: http
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
failureThreshold: 3
httpGet:
path: /ui
port: http
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 50m
memory: 128Mi
restartPolicy: Always
+41
View File
@@ -0,0 +1,41 @@
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: ui-hpa
namespace: artifactapi
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ui
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60
behavior:
scaleUp:
stabilizationWindowSeconds: 0
selectPolicy: Max
policies:
- type: Percent
value: 100
periodSeconds: 30
- type: Pods
value: 4
periodSeconds: 30
scaleDown:
stabilizationWindowSeconds: 300
selectPolicy: Min
policies:
- type: Percent
value: 10
periodSeconds: 60
- type: Pods
value: 2
periodSeconds: 60
+1 -1
View File
@@ -10,7 +10,7 @@ spec:
kubernetes:
audiences:
- vault
role: artifactapi
role: default
serviceAccount: default
tokenExpirationSeconds: 600
method: kubernetes
+18 -18
View File
@@ -1,6 +1,23 @@
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: postgres-credentials
namespace: artifactapi
spec:
destination:
create: true
name: postgres-credentials
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/artifactapi/default/postgres-credentials
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: environment
namespace: artifactapi
@@ -8,27 +25,10 @@ spec:
destination:
create: true
name: environment
overwrite: false
hmacSecretData: true
mount: kv
path: service/artifactapi/environment
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: postgres-password
namespace: artifactapi
spec:
destination:
create: true
name: postgres-password
overwrite: true
hmacSecretData: true
mount: kv
path: service/artifactapi/postgres-password
path: kubernetes/namespace/artifactapi/default/environment
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
+45
View File
@@ -0,0 +1,45 @@
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: api-vpa
namespace: artifactapi
# NOTE: this workload also has an HPA. updateMode Off is recommendation-only
# and does not act, so there is no HPA/VPA conflict today. Do not flip to Auto/
# Initial without first moving the HPA off CPU/memory (VPA owns those under Auto).
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: api
updatePolicy:
updateMode: "Off"
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: redis-vpa
namespace: artifactapi
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: redis
updatePolicy:
updateMode: "Off"
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: ui-vpa
namespace: artifactapi
# NOTE: this workload also has an HPA. updateMode Off is recommendation-only
# and does not act, so there is no HPA/VPA conflict today. Do not flip to Auto/
# Initial without first moving the HPA off CPU/memory (VPA owns those under Auto).
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: ui
updatePolicy:
updateMode: "Off"
+56
View File
@@ -0,0 +1,56 @@
---
# Ceph RGW (S3) backup target for the authentik CNPG cluster, provisioned by the
# in-estate cephrgw-operator. One dedicated bucket + owner user per cluster:
# cephrgw CRs are namespace-scoped and CNPG reads its S3 credential Secret from
# its own namespace, so backups are per-database rather than one shared bucket.
apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
name: cnpg-authentik-backup
namespace: authentik
spec:
displayName: "CNPG backup owner (authentik)"
# RGW users are global; keep the uid namespace-qualified so it never collides.
uid: cnpg-authentik-backup
maxBuckets: 5
# Operator writes AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ RGW_UID,
# S3_ENDPOINT) into this Secret; the Cluster's barmanObjectStore consumes it.
secretName: cnpg-authentik-backup-s3
# Keep the RGW user (and thus the keys) if this CR is ever deleted, so an
# in-flight restore can still reach the archive.
retainOnDelete: true
---
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: cnpg-authentik
namespace: authentik
spec:
placementTarget: ec
bucketName: cnpg-authentik
# The owner user has full control of its own bucket (read + write), which is
# all the backup/restore identity needs — no extra BucketAccess grant.
ownerRef: cnpg-authentik-backup
versioning: false
tags:
app: authentik
purpose: cnpg-backup
# Never drop the backups if the CR is removed; retire buckets by hand.
retainOnDelete: true
---
# Nightly base backup. Continuous WAL archiving is always-on via the Cluster's
# spec.backup.barmanObjectStore; this schedules the periodic full backup that
# WAL is layered on top of. Schedules are staggered across clusters so the 8
# base backups do not hit RGW at once (CNPG cron is 6-field, seconds first).
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: cnpg-authentik-nightly
namespace: authentik
spec:
schedule: "0 0 1 * * *"
immediate: false
backupOwnerReference: self
method: barmanObjectStore
cluster:
name: postgres
+122
View File
@@ -0,0 +1,122 @@
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres
namespace: authentik
spec:
affinity:
podAntiAffinityType: preferred
backup:
# 30-day retention (DEFAULT — adjust per cluster if needed). Enforced by CNPG
# against the object store on each successful base backup.
retentionPolicy: 30d
barmanObjectStore:
# Dedicated per-cluster Ceph RGW bucket (cephrgw-operator provisions it).
destinationPath: s3://cnpg-authentik
endpointURL: https://s3.ceph.unkin.net
# radosgw serves a Vault-PKI cert; trust the internal CA (reflected into
# every namespace as the vault-ca-cert Secret).
endpointCA:
name: vault-ca-cert
key: ca.crt
# Keys minted by the ObjectStoreUser in cnpg_backup.yaml; never hardcoded.
s3Credentials:
accessKeyId:
name: cnpg-authentik-backup-s3
key: AWS_ACCESS_KEY_ID
secretAccessKey:
name: cnpg-authentik-backup-s3
key: AWS_SECRET_ACCESS_KEY
# Path prefix within the bucket; keep stable across restores (see docs).
serverName: authentik
data:
compression: bzip2
jobs: 2
wal:
compression: zstd
maxParallel: 2
bootstrap:
initdb:
database: authentik
encoding: UTF8
localeCType: C
localeCollate: C
owner: authentik
secret:
name: postgres-credentials
enablePDB: true
enableSuperuserAccess: false
failoverDelay: 0
imageName: ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie
instances: 3
logLevel: info
maxSyncReplicas: 0
minSyncReplicas: 0
monitoring:
customQueriesConfigMap:
- key: queries
name: cnpg-default-monitoring
disableDefaultQueries: false
enablePodMonitor: false
postgresql:
parameters:
archive_mode: "on"
archive_timeout: 5min
dynamic_shared_memory_type: posix
effective_cache_size: 256MB
full_page_writes: "on"
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
log_rotation_age: "0"
log_rotation_size: "0"
log_truncate_on_rotation: "false"
logging_collector: "on"
max_connections: "200"
max_parallel_workers: "16"
max_replication_slots: "16"
max_worker_processes: "16"
shared_buffers: 128MB
shared_memory_type: mmap
ssl_max_protocol_version: TLSv1.3
ssl_min_protocol_version: TLSv1.3
wal_keep_size: 256MB
wal_level: logical
wal_log_hints: "on"
wal_receiver_timeout: 5s
wal_sender_timeout: 5s
syncReplicaElectionConstraint:
enabled: false
primaryUpdateMethod: restart
primaryUpdateStrategy: unsupervised
probes:
liveness:
isolationCheck:
connectionTimeout: 1000
enabled: true
requestTimeout: 1000
replicationSlots:
highAvailability:
enabled: true
slotPrefix: _cnpg_
synchronizeReplicas:
enabled: true
updateInterval: 30
resources:
limits:
cpu: 500m
# 512Mi OOMKilled replicas under load (shared_buffers 128MB +
# max_connections 200 leave no headroom) — see incident 2026-07-28.
memory: 1Gi
requests:
cpu: 50m
memory: 512Mi
smartShutdownTimeout: 180
startDelay: 3600
stopDelay: 1800
storage:
resizeInUseVolumes: true
size: 20Gi
storageClass: cephrbd-fast-delete
switchoverDelay: 3600
+66
View File
@@ -0,0 +1,66 @@
---
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
name: postgres-pooler-rw
namespace: authentik
spec:
cluster:
name: postgres
instances: 2
pgbouncer:
parameters:
default_pool_size: "100"
max_client_conn: "400"
paused: false
poolMode: session
template:
metadata:
labels:
app: pooler-rw
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pooler-rw
topologyKey: kubernetes.io/hostname
containers: []
type: rw
---
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
name: postgres-pooler-ro
namespace: authentik
spec:
cluster:
name: postgres
instances: 2
pgbouncer:
parameters:
default_pool_size: "100"
max_client_conn: "400"
paused: false
poolMode: session
template:
metadata:
labels:
app: pooler-ro
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pooler-ro
topologyKey: kubernetes.io/hostname
containers: []
type: ro
+57
View File
@@ -0,0 +1,57 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
traefik.io/instance: internal
annotations:
cert-manager.io/cluster-issuer: vault-issuer
cert-manager.io/common-name: identity.unkin.net
cert-manager.io/private-key-size: "4096"
external-dns.alpha.kubernetes.io/hostname: identity.unkin.net,identity.k8s.syd1.au.unkin.net
external-dns.alpha.kubernetes.io/target: 198.18.200.4
name: authentik
namespace: authentik
spec:
gatewayClassName: traefik-internal
listeners:
- allowedRoutes:
namespaces:
from: Same
hostname: identity.unkin.net
name: http
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: Same
hostname: identity.unkin.net
name: https
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ""
kind: Secret
name: authentik-tls
mode: Terminate
- allowedRoutes:
namespaces:
from: Same
hostname: identity.k8s.syd1.au.unkin.net
name: http-internal
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: Same
hostname: identity.k8s.syd1.au.unkin.net
name: https-internal
port: 443
protocol: HTTPS
tls:
certificateRefs:
- group: ""
kind: Secret
name: authentik-tls
mode: Terminate
+59
View File
@@ -0,0 +1,59 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: authentik-http-redirect
namespace: authentik
spec:
hostnames:
- identity.unkin.net
- identity.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik
sectionName: http
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik
sectionName: http-internal
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: authentik
namespace: authentik
spec:
hostnames:
- identity.unkin.net
- identity.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik
sectionName: https
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik
sectionName: https-internal
rules:
- backendRefs:
- group: ""
kind: Service
name: authentik-server
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
+21
View File
@@ -0,0 +1,21 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cnpg_cluster.yaml
- cnpg_backup.yaml
- cnpg_pooler.yaml
- gateway.yaml
- httproute.yaml
- ldap-gateway.yaml
- ldap-httproute.yaml
- ldap-service.yaml
- ldap-tlsroute.yaml
- namespace.yaml
- redis-deployment.yaml
- redis-pvc.yaml
- redis-service.yaml
- vaultauth.yaml
- vaultstaticsecret.yaml
- vpa.yaml
+47
View File
@@ -0,0 +1,47 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
labels:
traefik.io/instance: internal
annotations:
cert-manager.io/cluster-issuer: vault-issuer
cert-manager.io/common-name: ldap.k8s.syd1.au.unkin.net
cert-manager.io/private-key-size: "4096"
name: authentik-ldap
namespace: authentik
spec:
gatewayClassName: traefik-internal
listeners:
- allowedRoutes:
namespaces:
from: Same
hostname: ldap.k8s.syd1.au.unkin.net
name: ldaps-internal
port: 636
protocol: TLS
tls:
mode: Passthrough
- allowedRoutes:
namespaces:
from: Same
hostname: ldap.main.unkin.net
name: ldaps-main
port: 636
protocol: TLS
tls:
mode: Passthrough
- allowedRoutes:
namespaces:
from: Same
hostname: ldap.k8s.syd1.au.unkin.net
name: http-dns
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: Same
hostname: ldap.main.unkin.net
name: http-dns-main
port: 80
protocol: HTTP
+32
View File
@@ -0,0 +1,32 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: authentik-ldap-dns
namespace: authentik
annotations:
external-dns.alpha.kubernetes.io/hostname: ldap.k8s.syd1.au.unkin.net,ldap.main.unkin.net
external-dns.alpha.kubernetes.io/target: 198.18.200.4
spec:
hostnames:
- ldap.k8s.syd1.au.unkin.net
- ldap.main.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik-ldap
sectionName: http-dns
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik-ldap
sectionName: http-dns-main
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
+18
View File
@@ -0,0 +1,18 @@
---
apiVersion: v1
kind: Service
metadata:
name: authentik-ldap
namespace: authentik
spec:
internalTrafficPolicy: Cluster
ports:
- name: ldaps
port: 6636
protocol: TCP
targetPort: 6636
selector:
app.kubernetes.io/name: authentik
app.kubernetes.io/component: ldap
sessionAffinity: None
type: ClusterIP
+26
View File
@@ -0,0 +1,26 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: TLSRoute
metadata:
name: authentik-ldaps
namespace: authentik
spec:
hostnames:
- ldap.k8s.syd1.au.unkin.net
- ldap.main.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik-ldap
sectionName: ldaps-internal
- group: gateway.networking.k8s.io
kind: Gateway
name: authentik-ldap
sectionName: ldaps-main
rules:
- backendRefs:
- group: ""
kind: Service
name: authentik-ldap
port: 6636
weight: 1
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: authentik
+58
View File
@@ -0,0 +1,58 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: authentik
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:7-alpine
imagePullPolicy: IfNotPresent
args:
- --save
- "20"
- "1"
ports:
- containerPort: 6379
name: redis
protocol: TCP
livenessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 5
periodSeconds: 10
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 50m
memory: 128Mi
volumeMounts:
- mountPath: /data
name: redis-data
volumes:
- name: redis-data
persistentVolumeClaim:
claimName: redis-data
+13
View File
@@ -0,0 +1,13 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: redis-data
namespace: authentik
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: cephrbd-fast-delete
+17
View File
@@ -0,0 +1,17 @@
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: authentik
spec:
internalTrafficPolicy: Cluster
ports:
- name: redis
port: 6379
protocol: TCP
targetPort: redis
selector:
app: redis
sessionAffinity: None
type: ClusterIP
+18
View File
@@ -0,0 +1,18 @@
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: default
namespace: authentik
spec:
allowedNamespaces:
- authentik
kubernetes:
audiences:
- vault
role: default
serviceAccount: default
tokenExpirationSeconds: 600
method: kubernetes
mount: k8s/au/syd1
vaultConnectionRef: vso-system/default
@@ -0,0 +1,51 @@
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: postgres-credentials
namespace: authentik
spec:
destination:
create: true
name: postgres-credentials
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/authentik/default/postgres-credentials
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: authentik-credentials
namespace: authentik
spec:
destination:
create: true
name: authentik-credentials
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/authentik/default/authentik-credentials
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: s3-credentials
namespace: authentik
spec:
destination:
create: true
name: s3-credentials
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/authentik/default/s3-credentials
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
+13
View File
@@ -0,0 +1,13 @@
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: redis-vpa
namespace: authentik
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: redis
updatePolicy:
updateMode: "Off"
@@ -0,0 +1,16 @@
---
# Confines the agent-dns service account (in bind-system) to the agent-dns
# ClusterRole within this namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: agent-dns
namespace: bind-external
subjects:
- kind: ServiceAccount
name: agent-dns
namespace: bind-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: agent-dns
+52
View File
@@ -0,0 +1,52 @@
---
# Externally-reachable authoritative BIND for zones we delegate to ourselves.
# First tenant: acme.unkin.net, the DNS-01 challenge zone Let's Encrypt validates
# via a one-time _acme-challenge.unkin.net CNAME. Authoritative-only, recursion
# off, no forwarding, no open transfers -- the primaryService is the single
# dmz-pinned LoadBalancer that public NAT targets and that cert-manager writes to.
apiVersion: bind.unkin.net/v1alpha1
kind: BindCluster
metadata:
name: bind-external
namespace: bind-external
spec:
mode: authoritative
recursion: false
replicas: 2
storageClassName: cephrbd-fast-delete
storageSize: 1Gi
# Public server: answer queries from anywhere (Let's Encrypt validates over the
# internet), deny recursion and open zone transfers. localhost + pod net are
# implied by "any" and cover in-pod nsupdate and secondary SOA refresh; per-zone
# allow-transfer (catalog + acme zone) still permits key-authenticated AXFR.
extraOptions:
- "allow-query { any; }"
- "allow-transfer { none; }"
service:
type: ClusterIP
primaryService:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
purelb.io/service-group: dmz
purelb.io/addresses: 198.18.199.53
external-dns.alpha.kubernetes.io/hostname: bind-external-primary.k8s.syd1.au.unkin.net
resources:
requests:
cpu: 20m
memory: 128Mi
limits:
cpu: "1"
memory: 512Mi
---
# Catalog zone so the acme zone replicates onto the secondary (AXFR/IXFR keyed
# with the certmanager TSIG key, reused here as the transfer key).
apiVersion: bind.unkin.net/v1alpha1
kind: BindCatalogZone
metadata:
name: bind-external-catalog
namespace: bind-external
spec:
clusterRef: bind-external
zoneName: catalog.external
transferKeyRef: certmanager
@@ -0,0 +1,10 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- cluster.yaml
- tsigkey.yaml
- zones.yaml
- agent-dns-rolebinding.yaml
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: bind-external
+20
View File
@@ -0,0 +1,20 @@
---
# TSIG key cert-manager uses to send RFC2136 dynamic updates (the DNS-01 TXT
# records) to the primary, and that the secondary reuses for AXFR. The operator
# generates the material into Secret certmanager-tsig in this namespace;
# secretTemplate stamps emberstack reflector hints so the Secret is mirrored into
# the cert-manager namespace, where the rfc2136 solver reads its "secret" key.
apiVersion: bind.unkin.net/v1alpha1
kind: BindTSIGKey
metadata:
name: certmanager
namespace: bind-external
spec:
clusterRef: bind-external
algorithm: hmac-sha256
secretTemplate:
annotations:
reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "cert-manager"
reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "cert-manager"
+19
View File
@@ -0,0 +1,19 @@
---
# Self-delegated ACME challenge zone. Google Cloud DNS holds a one-time
# _acme-challenge.unkin.net CNAME -> _acme-challenge.acme.unkin.net and an
# acme.unkin.net NS delegation pointing here; cert-manager writes the challenge
# TXT records via RFC2136 authenticated with the certmanager key.
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: acme-unkin-net
namespace: bind-external
spec:
clusterRef: bind-external
zoneName: acme.unkin.net
type: primary
defaultTTL: 60
dynamicUpdate: true
updateKeyRef: certmanager
allowTransfer:
- key certmanager
@@ -0,0 +1,16 @@
---
# Confines the agent-dns service account (in bind-system) to the agent-dns
# ClusterRole within this namespace.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: agent-dns
namespace: bind-internal
subjects:
- kind: ServiceAccount
name: agent-dns
namespace: bind-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: agent-dns
@@ -0,0 +1,29 @@
---
# Internal client networks allowed to query the authoritative servers,
# mirrored from the puppet authoritative /etc/named/acls.conf
# (acl-main.unkin.net). Named auth-acl-main because the resolver has its own,
# differently-scoped acl-main.unkin.net in the same namespace.
apiVersion: bind.unkin.net/v1alpha1
kind: BindACL
metadata:
name: auth-acl-main
namespace: bind-internal
spec:
clusterRef: bind-authoritative
entries:
- 198.18.13.0/24
- 198.18.14.0/24
- 198.18.15.0/24
- 198.18.16.0/24
- 198.18.17.0/24
- 198.18.19.0/24
- 198.18.20.0/24
- 198.18.24.0/24
- 198.18.25.0/24
- 198.18.26.0/24
- 198.18.27.0/24
- 198.18.28.0/24
- 198.18.29.0/24
# Admin/management access (individual hosts, not whole subnets)
- 10.10.12.200/32 # benvin workstation (wireguard)
- 198.18.21.160/32 # benvin router
@@ -0,0 +1,54 @@
---
# Authoritative masters (replaces the 3x Puppet authoritative servers).
# pod-0 is the primary; pods 1-2 replicate via the catalog zone + AXFR/IXFR.
apiVersion: bind.unkin.net/v1alpha1
kind: BindCluster
metadata:
name: bind-authoritative
namespace: bind-internal
spec:
mode: authoritative
replicas: 3
storageClassName: cephrbd-fast-delete
storageSize: 2Gi
# Restrict queries to internal networks (puppet acl-main.unkin.net).
# 10.42.0.0/16 (pod net) is required so secondaries can SOA-refresh
# from the primary during catalog replication. localhost is required so the
# operator's in-pod `nsupdate` (sent to 127.0.0.1) passes query-authorization;
# without it every dynamic update is "denied due to allow-query".
extraOptions:
- "allow-query { localhost; auth-acl-main; 10.42.0.0/16; }"
# Enable query logging for the Tier-2 vector bind_query pipeline (see the
# resolvers cluster for the routing rationale).
- "querylog yes"
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
purelb.io/service-group: common
purelb.io/addresses: 198.18.200.6
external-dns.alpha.kubernetes.io/hostname: bind-authoritative.k8s.syd1.au.unkin.net
primaryService:
type: LoadBalancer
annotations:
purelb.io/service-group: common
purelb.io/addresses: 198.18.200.9
external-dns.alpha.kubernetes.io/hostname: bind-authoritative-primary.k8s.syd1.au.unkin.net
resources:
requests:
cpu: 20m
memory: 128Mi
limits:
cpu: "1"
memory: 512Mi
---
# Catalog zone so new BindZones auto-provision onto the secondaries.
apiVersion: bind.unkin.net/v1alpha1
kind: BindCatalogZone
metadata:
name: bind-authoritative-catalog
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: catalog.internal
transferKeyRef: transfer-key
@@ -0,0 +1,10 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster.yaml
- tsigkey.yaml
- zones.yaml
- records.yaml
- acls.yaml
@@ -0,0 +1,90 @@
# Individually-managed authoritative records for the unkin.net zone.
# DNSRecords must live in the same namespace as their BindZone (the operator
# resolves zoneRef/clusterRef/updateKeyRef within the record's namespace), so
# these sit alongside the zone in bind-internal, not in the app namespace.
---
apiVersion: bind.unkin.net/v1alpha1
kind: DNSRecord
metadata:
# "internal" in the name distinguishes this from the external DNS that
# Authentik will manage its own records from later.
name: identity-dns-internal
namespace: bind-internal
spec:
zoneRef: unkin-net
name: identity
type: A
ttl: 600
values:
# traefik-internal gateway VIP; the authentik Gateway serves the
# identity.unkin.net hostname there.
- 198.18.200.4
---
# PRODUCTION CUTOVER RECORD — intentionally commented out.
# git.unkin.net currently resolves to the LIVE VM forge (HAProxy VRRP VIP
# 198.18.19.17), which holds every repo the estate depends on. Uncommenting this
# repoints the whole org's git.unkin.net at the new k8s Gitea gateway VIP, so it
# is the FINAL step of the forge migration — gated on the data migration (gitea
# dump/restore + SECRET_KEY copy) in argocd-apps docs/gitea-migration.md.
# NOTE: the live git.unkin.net answer is served by the puppet DNS master today
# (profiles::dns::master, records from PuppetDB); this k8s apex zone holds only
# SOA+NS + a few DNSRecords so far. Confirm the k8s bind cluster is the live
# authority for unkin.net (or update the puppet record instead) before relying
# on this CR at cutover.
# ---
# apiVersion: bind.unkin.net/v1alpha1
# kind: DNSRecord
# metadata:
# name: git-dns-internal
# namespace: bind-internal
# spec:
# zoneRef: unkin-net
# name: git
# type: A
# ttl: 600
# values:
# # traefik-internal gateway VIP; the gitea Gateway serves git.unkin.net there.
# - 198.18.200.4
---
apiVersion: bind.unkin.net/v1alpha1
kind: DNSRecord
metadata:
name: s3-ceph-cname
namespace: bind-internal
spec:
zoneRef: ceph-unkin-net
name: s3
type: CNAME
ttl: 600
values:
# radosgw S3 endpoint. Points at the Consul service for now; the real
# target will be changed later.
- radosgw.service.consul.
---
apiVersion: bind.unkin.net/v1alpha1
kind: DNSRecord
metadata:
name: dashboard-ceph-cname
namespace: bind-internal
spec:
zoneRef: ceph-unkin-net
name: dashboard
type: CNAME
ttl: 600
values:
# Ceph mgr dashboard, reached via lb1. Lets in-cluster clients (the
# cephrgw-operator) resolve dashboard.ceph.unkin.net.
- lb1.unkin.net.
---
apiVersion: bind.unkin.net/v1alpha1
kind: DNSRecord
metadata:
name: lb1-unkin-net
namespace: bind-internal
spec:
zoneRef: unkin-net
name: lb1
type: A
ttl: 600
values:
- 103.216.191.185
@@ -0,0 +1,24 @@
---
# Zone-transfer / catalog key. The operator generates the material into a
# Secret (transfer-key-tsig); nothing sensitive is committed to git.
apiVersion: bind.unkin.net/v1alpha1
kind: BindTSIGKey
metadata:
name: transfer-key
namespace: bind-internal
spec:
clusterRef: bind-authoritative
algorithm: hmac-sha256
---
# Client-update key: puppet clients (profiles::dns::updater) nsupdate their own
# records to the authoritative zones with this key. Operator generates the
# material into Secret client-update-tsig; the same value must reach puppet
# eyaml (or the planned Vault-sync bridge) for clients to authenticate.
apiVersion: bind.unkin.net/v1alpha1
kind: BindTSIGKey
metadata:
name: client-update
namespace: bind-internal
spec:
clusterRef: bind-authoritative
algorithm: hmac-sha256
@@ -0,0 +1,256 @@
# Authoritative zones migrated from puppet-prod
# (profiles::dns::master::zones in hieradata/roles/infra/dns/master.yaml).
# type primary, static (puppet dynamic:false); TTL 600 as in the puppet zone header.
# Record data is populated by PuppetDB exported resources upstream, so it is
# NOT in this repo — migrate it into these zones (AXFR from the current masters,
# or DNSRecord CRs) as a follow-up. The zones start with SOA+NS only.
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: unkin-net
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: unkin.net
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: main-unkin-net
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: main.unkin.net
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
# ceph.unkin.net: the ceph host (ausyd1nxvm2069/halb) publishes
# dashboard.ceph.unkin.net via nsupdate; puppet targets a dedicated
# `zone ceph.unkin.net.`, so it must exist here or the update gets NOTZONE.
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: ceph-unkin-net
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: ceph.unkin.net
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 13-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 13.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 14-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 14.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 15-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 15.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 16-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 16.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 17-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 17.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 19-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 19.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 20-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 20.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 21-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 21.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 22-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 22.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 23-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 23.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 24-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 24.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 25-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 25.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 26-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 26.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 27-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 27.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 28-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 28.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 29-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-authoritative
zoneName: 29.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: client-update
@@ -0,0 +1,42 @@
---
# external-dns tier (replaces the 3x Puppet external-dns servers). An ordinary
# authoritative cluster; external-dns writes to its zones via RFC2136 because
# those BindZones set dynamicUpdate (allow-update { key externaldns-key; }).
apiVersion: bind.unkin.net/v1alpha1
kind: BindCluster
metadata:
name: bind-externaldns
namespace: bind-internal
spec:
mode: authoritative
replicas: 3
storageClassName: cephrbd-fast-delete
storageSize: 1Gi
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
purelb.io/service-group: common
purelb.io/addresses: 198.18.200.8
external-dns.alpha.kubernetes.io/hostname: bind-externaldns.k8s.syd1.au.unkin.net
primaryService:
type: ClusterIP
resources:
requests:
cpu: 20m
memory: 128Mi
limits:
cpu: "1"
memory: 512Mi
---
# Catalog zone so the dynamic zones replicate onto the cluster's secondaries
# (external-dns writes to the primary; secondaries IXFR the result).
apiVersion: bind.unkin.net/v1alpha1
kind: BindCatalogZone
metadata:
name: bind-externaldns-catalog
namespace: bind-internal
spec:
clusterRef: bind-externaldns
zoneName: catalog.externaldns.internal
transferKeyRef: externaldns-key
@@ -0,0 +1,8 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster.yaml
- tsigkey.yaml
- zones.yaml
@@ -0,0 +1,21 @@
---
# Key that external-dns (and DNSRecord objects) use to send RFC2136 dynamic
# updates to the primary. The operator generates the material into a Secret
# (externaldns-key-tsig) in this namespace. secretTemplate stamps emberstack
# reflector hints onto that Secret so it is mirrored into the externaldns
# namespace, where the external-dns controller reads it -- guaranteeing
# external-dns presents exactly the key the primary's allow-update accepts.
apiVersion: bind.unkin.net/v1alpha1
kind: BindTSIGKey
metadata:
name: externaldns-key
namespace: bind-internal
spec:
clusterRef: bind-externaldns
algorithm: hmac-sha256
secretTemplate:
annotations:
reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "externaldns"
reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "externaldns"
@@ -0,0 +1,34 @@
# k8s external-dns zones migrated from puppet-prod
# (externaldns::k8s_zones in hieradata/roles/infra/dns/externaldns.yaml).
# Primary + dynamicUpdate: the Kubernetes external-dns controller writes
# records here via RFC2136 authenticated with externaldns-key.
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: k8s-syd1-au-unkin-net
namespace: bind-internal
spec:
clusterRef: bind-externaldns
zoneName: k8s.syd1.au.unkin.net
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: externaldns-key
allowTransfer:
- key externaldns-key
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: 200-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-externaldns
zoneName: 200.18.198.in-addr.arpa
type: primary
defaultTTL: 600
dynamicUpdate: true
updateKeyRef: externaldns-key
allowTransfer:
- key externaldns-key
@@ -0,0 +1,11 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- authoritative
- resolvers
- externaldns
- tsig-api
- agent-dns-rolebinding.yaml
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: bind-internal
@@ -0,0 +1,69 @@
# Internal client ACLs, mirrored from puppet /etc/named/acls.conf.
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindACL
metadata:
name: acl-main.unkin.net
namespace: bind-internal
spec:
clusterRef: bind-resolvers
entries:
- 10.42.0.0/16 # k8s pod network (kube-proxy masquerades node-originated LB queries)
- 198.18.1.10/32
- 198.18.2.160/27
- 198.18.21.160/27
- 198.18.2.192/27
- 198.18.21.192/27
# Admin/management access
- 10.10.12.200/32 # benvin workstation (wireguard)
- 198.18.21.160/32 # benvin router (also within 198.18.21.160/27 above)
- 198.18.13.0/24
- 198.18.14.0/24
- 198.18.15.0/24
- 198.18.16.0/24
- 198.18.17.0/24
- 198.18.18.0/24
- 198.18.19.0/24
- 198.18.20.0/24
- 198.18.21.0/24
- 198.18.22.0/24
- 198.18.23.0/24
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindACL
metadata:
name: acl-dmz
namespace: bind-internal
spec:
clusterRef: bind-resolvers
entries:
- 198.18.24.0/24
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindACL
metadata:
name: acl-common
namespace: bind-internal
spec:
clusterRef: bind-resolvers
entries:
- 198.18.25.0/24
- 198.18.26.0/24
- 198.18.27.0/24
- 198.18.28.0/24
- 198.18.29.0/24
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindACL
metadata:
name: acl-nomad-jobs
namespace: bind-internal
spec:
clusterRef: bind-resolvers
entries:
- 198.18.64.0/24
- 198.18.65.0/24
- 198.18.66.0/24
- 198.18.67.0/24
- 198.18.68.0/24
- 198.18.69.0/24
@@ -0,0 +1,44 @@
---
# Recursive resolvers (replaces the 3x Puppet only-resolver servers).
# Three identical recursive servers; no zone replication.
apiVersion: bind.unkin.net/v1alpha1
kind: BindCluster
metadata:
name: bind-resolvers
namespace: bind-internal
spec:
mode: resolver
replicas: 3
storageClassName: cephrbd-fast-delete
storageSize: 1Gi
service:
type: LoadBalancer
externalTrafficPolicy: Local
annotations:
purelb.io/service-group: common
purelb.io/addresses: 198.18.200.7
external-dns.alpha.kubernetes.io/hostname: bind-resolvers.k8s.syd1.au.unkin.net
forwarders:
- 8.8.8.8
- 1.1.1.1
# The internal split-horizon zones are served UNSIGNED by the in-cluster
# authoritative, but their public parents publish DS records (e.g. unkin.net
# is DNSSEC-signed on the Internet). With dnssec-validation on, the validator
# sees "parent indicates secure" but gets an insecure answer and returns
# SERVFAIL (broken trust chain). Treat the forwarded internal domains as
# insecure so they are not validated. unkin.net covers all *.unkin.net
# (incl. k8s.syd1.au.unkin.net); 18.198.in-addr.arpa covers every reverse zone.
extraOptions:
- "validate-except { unkin.net; 18.198.in-addr.arpa; consul; }"
# Enable query logging so the Tier-2 vector bind_query pipeline can parse
# client/qname/qtype. Routes to the `queries` category which, with no explicit
# logging{} clause, follows the default category to the named foreground
# stderr channel -> pod stdout -> vector (subject logs.k8s.bind-internal.*).
- "querylog yes"
resources:
requests:
cpu: 20m
memory: 128Mi
limits:
cpu: "1"
memory: 512Mi
@@ -0,0 +1,303 @@
# Conditional forward zones, from the puppet openforwarder view.
# Upstreams: unkin authoritative 198.18.200.6, consul 198.18.19.14,
# k8s 198.18.200.8 (in-cluster bind-externaldns VIP).
# k8s -> in-cluster bind-externaldns 198.18.200.8 for both the forward zone
# k8s.syd1.au.unkin.net and the reverse zone 200.18.198.in-addr.arpa, which
# external-dns now publishes to (see the external-dns migration PRs).
# (Zones that forwarded to 10.10.16.x were dropped; consul left as-is.)
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-unkin-net
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: unkin.net
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-main-unkin-net
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: main.unkin.net
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-consul
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: consul
type: forward
catalog: false
forwarders:
- 198.18.19.14
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-k8s-syd1-au-unkin-net
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: k8s.syd1.au.unkin.net
type: forward
catalog: false
forwarders:
- 198.18.200.8
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-200-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 200.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
# Reverse zone for the k8s LB range, published by external-dns to the
# in-cluster bind-externaldns alongside k8s.syd1.au.unkin.net.
- 198.18.200.8
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-13-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 13.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-14-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 14.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-15-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 15.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-16-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 16.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-17-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 17.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-19-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 19.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-20-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 20.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-21-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 21.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-22-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 22.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-23-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 23.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-24-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 24.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-25-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 25.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-26-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 26.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-27-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 27.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-28-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 28.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
---
apiVersion: bind.unkin.net/v1alpha1
kind: BindZone
metadata:
name: fwd-29-18-198-in-addr-arpa
namespace: bind-internal
spec:
clusterRef: bind-resolvers
viewRef: openforwarder
zoneName: 29.18.198.in-addr.arpa
type: forward
catalog: false
forwarders:
- 198.18.200.6
@@ -0,0 +1,9 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster.yaml
- acls.yaml
- view.yaml
- forward-zones.yaml
@@ -0,0 +1,23 @@
---
# openforwarder view, mirrored from puppet /etc/named/views.conf.
# match-clients gates access to internal networks; recursion/query are 'any'
# within the view since match-clients already restricts who reaches it.
apiVersion: bind.unkin.net/v1alpha1
kind: BindView
metadata:
name: openforwarder
namespace: bind-internal
spec:
clusterRef: bind-resolvers
order: 100
matchClients:
- acl-main.unkin.net
- acl-nomad-jobs
- acl-common
- acl-dmz
recursion: true
allowQuery:
- any
extraOptions:
- "allow-recursion { any; }"
- "allow-query-cache { any; }"
@@ -3,4 +3,4 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../base/paperclip
- tsig-api.yaml
@@ -0,0 +1,27 @@
---
# Companion TSIG API. The operator reconciles this into a Deployment, Service,
# ConfigMap, master-token Secret and namespaced RBAC. vault-plugin-secrets-bind-tsig
# calls it to create/rotate/delete TSIG keys, which it does by managing
# BindTSIGKey resources in this namespace (the operator reconciles the material).
#
# The master access token Secret (bind-tsig-api-token) is generated by the
# operator if absent; a VaultStaticSecret may later pre-seed/overwrite it so the
# token is sourced from Vault rather than generated in-cluster.
apiVersion: bind.unkin.net/v1alpha1
kind: BindTSIGAPI
metadata:
name: bind-tsig-api
namespace: bind-internal
spec:
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/bind-tsig-api:v0.2.3
replicas: 1
port: 8443
# targetNamespace defaults to this resource's namespace (bind-internal), where
# the authoritative cluster and its keys live.
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
cpu: 250m
memory: 128Mi
+38
View File
@@ -0,0 +1,38 @@
---
# Static service account that Vault's kubernetes secret engine mints scoped
# tokens for (agent-dns role). RBAC is confined to the bind namespaces via the
# per-namespace RoleBindings below, not a ClusterRoleBinding.
apiVersion: v1
kind: ServiceAccount
metadata:
name: agent-dns
namespace: bind-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: agent-dns
rules:
- apiGroups: ["bind.unkin.net"]
resources: ["*"]
verbs: ["*"]
- apiGroups: [""]
resources: ["pods", "services", "configmaps", "events"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: agent-dns
namespace: bind-system
subjects:
- kind: ServiceAccount
name: agent-dns
namespace: bind-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: agent-dns
+57
View File
@@ -0,0 +1,57 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bind-operator
namespace: bind-system
labels:
app.kubernetes.io/name: bind-operator
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: bind-operator
template:
metadata:
labels:
app.kubernetes.io/name: bind-operator
spec:
serviceAccountName: bind-operator
securityContext:
runAsNonRoot: true
containers:
- name: operator
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/bind-operator:v0.2.6
args:
- --metrics-bind-address=:8080
- --health-probe-bind-address=:8081
- --leader-elect
ports:
- containerPort: 8080
name: metrics
- containerPort: 8081
name: health
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 256Mi
+13
View File
@@ -0,0 +1,13 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
# CRDs are pulled from the bind-operator repo at the matching tag rather than
# vendored here, so they never drift from the operator.
- https://git.unkin.net/unkin/bind-operator/raw/tag/v0.2.6/config/crd/install.yaml
- rbac.yaml
- agent-dns-rbac.yaml
- deployment.yaml
- vpa.yaml
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: bind-system
+54
View File
@@ -0,0 +1,54 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: bind-operator
namespace: bind-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: bind-operator
rules:
- apiGroups: ["bind.unkin.net"]
resources: ["*"]
verbs: ["*"]
- apiGroups: [""]
resources: ["services", "configmaps", "secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create", "get"]
- apiGroups: ["apps"]
resources: ["statefulsets", "deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# The BindTSIGAPI reconciler deploys the companion API: a ServiceAccount plus
# a namespaced Role/RoleBinding granting it access to BindTSIGKey + Secrets.
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles", "rolebindings"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: bind-operator
subjects:
- kind: ServiceAccount
name: bind-operator
namespace: bind-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: bind-operator
+13
View File
@@ -0,0 +1,13 @@
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: bind-operator-vpa
namespace: bind-system
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: bind-operator
updatePolicy:
updateMode: "Off"
+9 -1
View File
@@ -15,6 +15,13 @@ metadata:
spec:
gatewayClassName: traefik-internal
listeners:
- allowedRoutes:
namespaces:
from: Same
hostname: rancher.k8s.syd1.au.unkin.net
name: http
port: 80
protocol: HTTP
- allowedRoutes:
namespaces:
from: Same
@@ -24,6 +31,7 @@ spec:
protocol: HTTPS
tls:
certificateRefs:
- kind: Secret
- group: ""
kind: Secret
name: rancher-tls
mode: Terminate
+31 -2
View File
@@ -1,6 +1,30 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: rancher-http-redirect
namespace: cattle-system
spec:
hostnames:
- rancher.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: rancher
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: rancher
namespace: cattle-system
@@ -8,12 +32,17 @@ spec:
hostnames:
- rancher.k8s.syd1.au.unkin.net
parentRefs:
- name: rancher
- group: gateway.networking.k8s.io
kind: Gateway
name: rancher
sectionName: https
rules:
- backendRefs:
- name: rancher
- group: ""
kind: Service
name: rancher
port: 80
weight: 1
matches:
- path:
type: PathPrefix
+84
View File
@@ -0,0 +1,84 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cephrgw-operator
namespace: cephrgw-system
labels:
app.kubernetes.io/name: cephrgw-operator
annotations:
# Restart on internal CA rotation only; cephrgw-credentials is Vault-rotated
# (VSO) and deliberately excluded so routine key rotation causes no restart.
configmap.reloader.stakater.com/auto: "true"
secret.reloader.stakater.com/reload: "vault-ca-cert"
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: cephrgw-operator
template:
metadata:
labels:
app.kubernetes.io/name: cephrgw-operator
spec:
serviceAccountName: cephrgw-operator
securityContext:
runAsNonRoot: true
containers:
- name: operator
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/cephrgw-operator:v0.4.0
args:
- --metrics-bind-address=:8080
- --health-probe-bind-address=:8081
- --leader-elect
envFrom:
# Provides CEPH_RGW_ACCESS_KEY/SECRET_KEY and the endpoints
# (CEPH_RGW_ENDPOINT / CEPH_RGW_ADMIN_ENDPOINT), plus optional
# CEPH_RGW_REGION / CEPH_RGW_CA / CEPH_RGW_INSECURE. Rendered from
# Vault per docs/ceph-setup.md; not managed in GitOps.
- secretRef:
name: cephrgw-credentials
env:
# Trust the internal unkin.net (Vault PKI) CA so the operator can
# verify radosgw's TLS cert. vault-ca-cert is reflected into every
# namespace from the certificates namespace.
- name: CEPH_RGW_CA_FILE
value: /etc/vault-ca/ca.crt
volumeMounts:
- name: vault-ca-cert
mountPath: /etc/vault-ca/ca.crt
subPath: ca.crt
readOnly: true
ports:
- containerPort: 8080
name: metrics
- containerPort: 8081
name: health
readinessProbe:
httpGet:
path: /readyz
port: 8081
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 8081
initialDelaySeconds: 15
periodSeconds: 20
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 256Mi
volumes:
- name: vault-ca-cert
secret:
secretName: vault-ca-cert
@@ -0,0 +1,14 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
# CRDs are pulled from the cephrgw-operator repo at the matching tag rather
# than vendored here, so they never drift from the operator.
- https://git.unkin.net/unkin/cephrgw-operator/raw/tag/v0.4.0/config/crd/install.yaml
- rbac.yaml
- deployment.yaml
- vaultauth.yaml
- vaultstaticsecret.yaml
- vpa.yaml
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: cephrgw-system
+42
View File
@@ -0,0 +1,42 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephrgw-operator
namespace: cephrgw-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cephrgw-operator
rules:
- apiGroups: ["ceph.unkin.net"]
resources: ["*"]
verbs: ["*"]
# The operator delivers RGW access/secret keys into Secrets.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# v0.3.1 startup check reads its own CRDs to warn if they are stale/missing.
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cephrgw-operator
subjects:
- kind: ServiceAccount
name: cephrgw-operator
namespace: cephrgw-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cephrgw-operator
+21
View File
@@ -0,0 +1,21 @@
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: default
namespace: cephrgw-system
spec:
method: kubernetes
mount: k8s/au/syd1
vaultConnectionRef: vso-system/default
allowedNamespaces:
- cephrgw-system
kubernetes:
# Shared "default" role: binds the namespace's default ServiceAccount and
# grants the templated kv/kubernetes/namespace/<ns>/<sa>/* read policy, so
# no per-app terraform-vault change is needed.
role: default
serviceAccount: default
audiences:
- vault
tokenExpirationSeconds: 600
@@ -0,0 +1,30 @@
---
# Renders the radosgw credentials from Vault into the cephrgw-credentials
# Secret the operator Deployment consumes via envFrom. The KV secret's keys
# (CEPH_RGW_ACCESS_KEY/SECRET_KEY, CEPH_RGW_ENDPOINT, optional
# CEPH_RGW_ADMIN_ENDPOINT/REGION/CA) are copied verbatim, so they land as the
# matching env vars.
#
# The path sits under the templated default policy
# (kv/data/kubernetes/namespace/<ns>/<sa>/*), so it needs no dedicated Vault
# role or policy. Seed the values with:
# vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
# CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
# CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
# CEPH_RGW_ACCESS_KEY=... CEPH_RGW_SECRET_KEY=...
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: cephrgw-credentials
namespace: cephrgw-system
spec:
vaultAuthRef: default
mount: kv
type: kv-v2
path: kubernetes/namespace/cephrgw-system/default/cephrgw-credentials
refreshAfter: 5m
hmacSecretData: true
destination:
name: cephrgw-credentials
create: true
overwrite: true

Some files were not shown because too many files have changed in this diff Show More