Turn the single-replica jellyfin-ha app into a proper high-availability
deployment so the fork's Redis-coordinated distributed transcoding and
PostgreSQL main database can actually be exercised.
- Replace the Deployment with a 2-replica StatefulSet for stable pod
identity; set JELLYFIN_INSTANCE_ID from metadata.name (the fork's Redis
transcode-lease owner id), add soft podAntiAffinity and a PDB
minAvailable 1.
- Move the main Jellyfin DB to PostgreSQL via a CloudNativePG trio
(3-instance Cluster, PgBouncer Pooler, Ceph RGW barman backups) mirroring
the litellm pattern; an init container writes database.xml selecting the
fork's Jellyfin-PostgreSQL provider and the DSN is composed from the
CNPG-generated app secret pointed at the pooler.
- Share /config on an RWX cephfs PVC across replicas; keep /cache per-pod
via a volumeClaimTemplate.
- Fix the transcode mount to the fork's real path /config/transcodes on the
RWX PVC (raid5) so a surviving pod can resume the segments of the pod it
takes over.
- Add Intel iGPU hardware transcoding via the gpu.intel.com/i915 device
plugin resource plus render/video supplemental groups.
- Switch the Service to sessionAffinity ClientIP to reduce transcode churn.
- Disable UDP auto-discovery. Library scans still run on every replica; a
single-scanner leader election is a planned follow-up.
Deploys the jellyfin-ha fork (git.unkin.net/unkin/jellyfin-ha) to au-syd1
via ArgoCD, under a dedicated media AppProject/ApplicationSet rather than
extending platform.
- New media AppProject + media-apps ApplicationSet (watches
apps/overlays/*/jellyfin); registered in the argocd kustomizations
- apps/base/jellyfin: namespace, deployment (single replica to start),
service, in-namespace Redis (transcode session store), gateway + httproute
at jellyfin.k8s.syd1.au.unkin.net
- Storage: RWO config (cephrbd), RWX transcode scratch and RWX media
library (cephfs) per the HA fork's pod-takeover requirement
- au-syd1 overlay
## Why
artifactapi v3.8.0 ships the github_rpm remote (GitHub releases exposed as a dnf/yum repo via synthesized repodata, a background metadata syncer with multi-replica leasing, and a server-level GitHub machine credential for authenticated requests). This rolls the running deployment onto that release.
## How
- Bump the api image `git.unkin.net/unkin/artifactapi` from v3.7.7 to v3.8.0.
- Bump the ui image `git.unkin.net/unkin/artifactapi-ui` from v3.7.7 to v3.8.0.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #351
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
ArgoCD fails to sync the puppet app with:
Deployment.apps "puppetserver-master" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is 'Recreate'
The manifest is already correct: #341 changed the master to `spec.strategy.type: Recreate` with no rollingUpdate block. The failure is a live-object artifact. When the master ran RollingUpdate the API server defaulted `spec.strategy.rollingUpdate` (maxSurge/maxUnavailable) onto the object. That defaulted field is owned by no applier, so neither a client-side merge nor server-side apply drops it when the desired manifest omits it. The live object therefore keeps `rollingUpdate` while gaining `type: Recreate`, which the API server rejects — blocking every sync.
k8s forbids any `rollingUpdate` field when `strategy.type` is `Recreate`; the two are mutually exclusive, so the sync cannot converge until the stale field is removed from the live object.
## Changes
- Annotate the `puppetserver-master` Deployment with `argocd.argoproj.io/sync-options: Replace=true`. Replace performs a full PUT that overwrites the whole object, dropping the stale `rollingUpdate` field and letting the Recreate strategy apply cleanly. The annotation is scoped to this one resource, so puppetdb/puppetboard/compiler keep the app-wide ServerSideApply behaviour.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/puppet` renders `puppetserver-master` with `strategy: { type: Recreate }` (no rollingUpdate) and the new sync-options annotation.
- `make kubeconform`: puppet overlay 34/34 valid; only the known cattle-system rancher kubeVersion incompatibility fails.
- `pre-commit`: all hooks pass.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #349
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Kubernetes defaults `apiVersion: v1` and `kind: PersistentVolumeClaim` onto every StatefulSet `volumeClaimTemplates` entry, but neither the raw manifests nor the rendered helm charts in this repo emit those fields. Live StatefulSets therefore carry TypeMeta that git lacks, and ArgoCD reports a diff that removes it.
`volumeClaimTemplates` are immutable on an existing StatefulSet, so ArgoCD can never apply the removal. The affected StatefulSets stay perpetually OutOfSync and the un-appliable diff can contribute to sync failures.
Rendering the overlays confirms this is a chronic render-vs-live mismatch, not a regression: the source manifests/charts have never emitted the TypeMeta. Affected StatefulSets (render lacks TypeMeta, live defaults it in):
- `consul-server` (consul, helm render)
- `nats` (logging, helm render)
- `kanidm` (kanidm, raw manifest)
`vault` and the `woodpecker` StatefulSets already emit TypeMeta from their charts and are unaffected.
## How
- Add a fleet-wide `resource.customizations.ignoreDifferences.apps_StatefulSet` to the `argocd-cm` patch, using `jqPathExpressions` to ignore the defaulted `apiVersion` and `kind` under every `volumeClaimTemplates` entry.
A single global customization is chosen over per-manifest edits because the affected StatefulSets span both raw manifests (kanidm) and helm renders (consul, nats) whose output cannot be edited; it is inert for StatefulSets that already emit TypeMeta (vault, woodpecker) and future-proof for new ones. The live StatefulSets are left untouched — their `volumeClaimTemplates` are immutable, and recreation would orphan PVCs.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #350
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
- The cert-manager v1.20.2 controller crashloops: strict decoding of its ControllerConfiguration rejects the unknown field `acmeDNS01` (`failed to load config file ... strict decoding error: unknown field "acmeDNS01"`), so `/var/cert-manager/config/config.yaml` fails to load and the controller never starts. The rollout is stuck with only the old pod running.
- PR #337 placed the DNS-01 recursive-nameserver settings under `acmeDNS01`, but the field in the `controller.config.cert-manager.io/v1alpha1` schema is `acmeDNS01Config` (`ACMEDNS01Config`, with `recursiveNameservers` / `recursiveNameserversOnly`). The recursive-ns settings belong in the config file, not `extraArgs`; the CLI flags feed the same struct but the chart already renders a `--config` ControllerConfiguration, so the correct fix is the correct field name.
## How
- Rename the `config:` block `acmeDNS01` to `acmeDNS01Config`, keeping `recursiveNameservers` (`8.8.8.8:53`, `1.1.1.1:53`) and `recursiveNameserversOnly: true` so DNS-01 resolution and self-checks still use the public DNS view for the split-horizon delegation.
Rendered `kustomize build --enable-helm` confirms the ConfigMap `config.yaml` now carries a valid `acmeDNS01Config` block and no longer contains the invalid `acmeDNS01`; the cert-manager overlay is kubeconform-clean (55 valid, 0 invalid).
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #347
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
NetBox 4.6.5 refuses to save v2 API tokens without `API_TOKEN_PEPPERS` ("Unable to save v2 tokens: API_TOKEN_PEPPERS is not defined"), which blocks creating the superuser token the NetBox Vault engine needs (it defaults to v2 tokens). The chart only auto-generates a pepper when it creates the config secret itself; it does not do that while `existingSecret` (`netbox-secret-key`) is set, so the config secret carries no `api_token_peppers` key.
## Changes
- Document `api_token_peppers` on the `netbox-secret-key` VaultStaticSecret: a JSON pepper map `{"1": "<random>"}` seeded once into Vault alongside `secret_key`. VSO syncs every key at the path into the config secret, which the chart already mounts as an optional file into `API_TOKEN_PEPPERS`.
- Add a reloader annotation via `commonAnnotations` so the `netbox` and `netbox-worker` Deployments roll when `netbox-secret-key` changes, picking up the seeded pepper (and any rotated `secret_key`) without a manual restart.
## Follow-up (out of band)
seed the pepper once (rotating it invalidates existing v2 tokens):
```
PEP=$(openssl rand -base64 48 | tr -d '\n')
vault kv patch kv/kubernetes/namespace/netbox/default/netbox-secret-key \
api_token_peppers="{\"1\": \"$PEP\"}"
```
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #346
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
kea-dhcp4 crash-loops on a cold container start: the HA hook resolves the StatefulSet peer URL hostnames once at config load, but the peer DNS records aren't resolvable in the first instant of a fresh container, and kea exits hard instead of retrying. Verified in-cluster that the rendered config validates once DNS is warm, so it's a startup race. kea-operator v0.1.3 gates dhcp4 startup on a bounded `kea-dhcp4 -t` retry (~120s, then proceeds/fails loud).
## How
- bump kea-operator, kea, and kea-api images to v0.1.3
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #344
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: shrink the blast radius of the Puppet control-plane pods (CA/eyaml keys, compiled catalogs) per the security sweep in #307 — remove root where it is not required and strip cargo-culted capabilities.
How:
- puppetboard cert-generator init: root+APE:true -> uid 1000, drop:[all], APE:false; pod fsGroup 1000; removed trailing `chown -R 1000:1000` (PVC now group-owned).
- puppetdb create-log-dir init: root -> uid 999, drop:[all], APE:false; pod fsGroup 999; removed `chown 999:999`.
- All OpenVox capability add-lists: removed the duplicate CAP_-prefixed spellings (k8s normalises both to the same kernel cap) and dropped the unused AUDIT_WRITE.
- Added allowPrivilegeEscalation:false and seccompProfile RuntimeDefault across the workloads.
Stays root (evidence-backed, class-B fallback): the puppetserver master/compiler and puppetdb main containers, plus the perms-and-dirs and generate-types root containers. The OpenVox image entrypoint runs `chown -R puppet:puppet` over root-owned baked-in dirs and drops the JVM to the puppet user via `runuser` (needs CHOWN/SETUID/SETGID); a non-root start crashloops. Their cap sets are reduced to the minimum justified (CHOWN/DAC_OVERRIDE/FOWNER[/SETUID/SETGID]).
Validation: `kustomize build --enable-helm` clean; kubeconform 0 invalid / 0 errors; pre-commit (yamllint etc.) green. Confirmed against live pods: puppetserver/puppetdb JVMs already run as puppet/puppetdb via `runuser`; `pam_loginuid` is absent from the su/runuser PAM stacks and loginuid is unset, so dropping AUDIT_WRITE is safe.
Post-merge smoke test (puppet had an outage this session — watch closely): after argocd sync, confirm puppetserver master + a compiler reach `running` at /status/v1/simple, puppetdb reaches `running`, puppetboard serves 200, and the generate-types + g10k CronJobs complete — i.e. catalogs still compile and reports still ingest.
Closes#307https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #319
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
kea-dhcp4 and kea-ctrl-agent crash-loop because kea 2.6.5 refuses a unix-socket directory more relaxed than 0750, but the operator's shared emptyDir mounts `/var/run/kea` at 0777 (`'socket-name' is invalid: socket path:/var/run/kea ... more relaxed permissions than 750`). kea-operator v0.1.2 renders entrypoints that tighten it.
## How
- bump kea-operator, kea, and kea-api images to v0.1.2
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #343
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. **Stacks on #318 — merge after it.**
How:
- 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: **bind_query** (k8s bind-* + VM named), **rancher_audit** (cattle-system sidecar JSON), **cnpg_pg** (ONE transform for all 10 CNPG clusters via the `.postgres` container), **gitea** (router+access, k8s+VM), **puppet** (openvoxserver/openvoxdb logback + access), **litellm** (JSON request logs), **postfix** (per-line maillog).
- Carve `.postgres` out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact.
- Companion k8s flips in this PR: litellm `JSON_LOGS=True`; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on.
- 15 new `vector test` cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing `fields Map(String,String)` — no DDL change.
Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with `.file`/`.SYSLOG_IDENTIFIER` tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #320
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The puppet MASTER is the singleton CA/master. A second master, even transiently during a rolling update, races on CA/cert signing and shared state (the CA lives on a shared PVC mounted by every master pod). The master was previously driven by an HPA with `minReplicas: 2`, `maxReplicas: 5` and a `RollingUpdate` strategy, so 2-5 masters could coexist normally and a rollout would briefly run old+new masters against the same CA data — a latent CA-corruption/split-brain bug. Recreate guarantees the old pod terminates before the new one starts, so two masters never coexist.
## Changes
- Set `puppetserver-master` `spec.replicas: 1` and `spec.strategy.type: Recreate` (drops RollingUpdate).
- Remove the `puppetserver-masters-autoscaler` HPA and its kustomization entry, which forced 2-5 master replicas and would otherwise override `replicas: 1`.
- Refresh the `puppetserver-master-vpa` note to reflect the pinned-singleton, no-HPA state (VPA stays `updateMode: Off`, recommendation-only).
The compiler (`puppetserver-compiler`) remains the horizontally-scalable tier with its own HPA — untouched. puppetdb/puppetboard untouched.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #341
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret.
## Changes
- Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched.
- Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads.
- Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`.
- Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf.
- Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`).
## Safety review (secret-only / CA workloads)
`vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded.
- `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated.
- `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`).
- `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only.
- `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added.
CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`).
## Notes / coordination
- Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression).
- Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here.
## Validation
- `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains.
- `uvx pre-commit run --all-files` — all hooks pass.
Closes#326
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #339
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
kea-0/kea-1 crash-looped after the dhcp-system deploy. Two root causes:
1. **kea-dhcp4** rejected the `ntp-servers` option (DHCP code 42) because that option carries IPv4 addresses only, but the KeaCluster supplied rotating `pool.ntp.org` hostnames (`DHCP4_CONFIG_LOAD_FAIL ... Failed to convert string to address '0.au.pool.ntp.org'`).
2. **kea-ctrl-agent/dhcp4** rejected the `/run/kea` unix socket path — kea 2.6.5 permits only `/var/run/kea` (exact-string check). Fixed in kea-operator v0.1.1 (`RunDir=/var/run/kea`).
- Remove `ntpServers` from the KeaCluster (not representable via DHCP option 42; add concrete NTP server IPs if ever needed).
- Bump kea-operator, kea, and kea-api images v0.1.0 -> v0.1.1 (socket-path fix).
kubeconform + pre-commit green.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #338
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin.net is split-horizon: the `_acme-challenge.unkin.net -> _acme-challenge.acme.unkin.net` delegation CNAME exists only in the public Google Cloud DNS view. cert-manager's CNAME following (`cnameStrategy: Follow`) resolves via in-cluster CoreDNS to the nodes' internal resolver, which serves an internal view of unkin.net lacking that CNAME; Follow therefore finds no delegation and still sends the rfc2136 UPDATE to zone unkin.net on bind-external (only authoritative for acme.unkin.net), returning NOTAUTH. Follow needs a public-view resolver for both the CNAME chase and the propagation self-check. TSIG is proven fine.
- Set `acmeDNS01.recursiveNameservers` to `8.8.8.8:53` and `1.1.1.1:53` with `acmeDNS01.recursiveNameserversOnly: true` in the cert-manager ControllerConfiguration so DNS-01 resolution and self-checks use the public DNS view.
- Keeps `cnameStrategy: Follow` on the ClusterIssuers (merged in #331); this PR gives that following a resolver that can see the delegation.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #337
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Early-boot clients — anaconda/kickstart and yum in %post, PXE environments — need direct HTTP access to the artifactapi rpm repos. The current setup returns a 301 redirect from HTTP to HTTPS, which those minimal clients cannot follow (or downgrade insecurely), breaking rpm installs.
## Changes
- Attach the `api-route` HTTPRoute to the Gateway's `http` (port 80) listener alongside `https`, so `http://artifactapi.k8s.syd1.au.unkin.net/...` serves app content directly (200/40x from the app, no Location header).
- Remove the `http-redirect` HTTPRoute (RequestRedirect 301 `http`->`https`), which was the sole redirect mechanism — the traefik `web` entrypoint has no global `redirections`, so this is scoped strictly to artifactapi and does not affect other apps.
- Leave HTTPS unchanged.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #336
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the isc-dhcpd PXE-boot VM with the kea-operator + an HA kea pair, GitOps-managed. Deploys on a new, unused anycast IP so nothing is cut over yet; the production cutover off the current dhcpd address (198.18.19.18) is a separate later task.
- Add `apps/base/dhcp-system`: namespace, kea-operator RBAC + Deployment (v0.1.0), VPA, and the 4 kea.unkin.net CRDs pulled from the operator repo at tag v0.1.0.
- Add CRs translating the legacy dhcpd config (source: puppet `roles/infra/dhcp/server.yaml`): KeaCluster `kea` (2 replicas, hot-standby HA, main.unkin.net, 1200/86400 leases, AU ntp pool); five KeaSubnets 198.18.13-17.0/24 with .200-.220 pools, gateways .254 except .17->.1, next-server 198.18.19.19; Legacy/UEFI-64 PXE client classes; KeaAPI.
- DHCP-advertised DNS points at the in-cluster bind-resolvers cluster (PureLB 198.18.200.7), not the legacy 198.18.19.15 forwarder.
- Pin the DHCP LoadBalancer Service to the free common-pool IP 198.18.200.10 via PureLB.
- KeaAPI bearer token is operator-generated (no plain Secret committed).
- Commit generated kea.unkin.net JSON schemas for kubeconform; register dhcp-system in the platform ApplicationSet + AppProject.
Client-class object names are lowercased (`legacy`/`uefi-64`) to satisfy RFC1123 since the operator renders the kea class name from metadata.name.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #333
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
ServiceAccount `terraform-ipam` in the `woodpecker` namespace for the terraform-ipam pipeline. The Vault k8s auth role `woodpecker_terraform_ipam` (terraform-vault PR) binds it. Mirrors the other terraform-* CI ServiceAccounts and is wired into the woodpecker kustomization.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #334
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Vault's kubernetes secret engine will mint scoped tokens for a static \`agent-dns\` service account instead of generating cluster-wide RBAC, so agent DNS access is confined to exactly the bind namespaces. This is the GitOps half of the terraform-vault agent-dns role rework (PR unkin/terraform-vault#109). Ordering: this must sync before the Vault \`agent-dns\` creds are usable — Vault mints tokens for an SA that must already exist.
## How
- Add ServiceAccount \`agent-dns\` + ClusterRole \`agent-dns\` (definition only, no ClusterRoleBinding) in \`bind-system\`: full verbs on \`bind.unkin.net\` CRDs, get/list/watch pods/services/configmaps/events, get pods/log.
- Add RoleBinding \`agent-dns\` in each of \`bind-system\`, \`bind-internal\`, \`bind-external\`, \`externaldns\`, binding the SA to the ClusterRole in that namespace — confining all access (reads included) to those four namespaces.
Whitelist note: the platform AppProject already permits ClusterRole/ClusterRoleBinding and all four namespace destinations, so no project change is needed.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #332
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
A live DNS-01 smoke test returned NOTAUTH because the solver walked _acme-challenge.unkin.net to zone unkin.net and sent the rfc2136 UPDATE there, but bind-external is only authoritative for acme.unkin.net; without cnameStrategy: Follow the solver does not chase the delegation CNAME.
- Set `cnameStrategy: Follow` on the `letsencrypt` and `letsencrypt-staging` ClusterIssuer DNS-01 solvers so cert-manager follows the `_acme-challenge.unkin.net -> _acme-challenge.acme.unkin.net` CNAME and updates the `acme.unkin.net` zone.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #331
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The new kea-operator repo's Woodpecker CI pipelines run under a dedicated Kubernetes ServiceAccount that must exist in the woodpecker namespace (cross-repo dependency; the .woodpecker/*.yaml steps set `serviceAccountName: kea-operator-ci`).
- Adds ServiceAccount `kea-operator-ci` in the `woodpecker` namespace
- Registers it in the woodpecker kustomization resources
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #330
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The \`vault-issuer\` ClusterIssuer is applied out-of-band (Helm release \`cert-manager-clusterissuer\`) and is referenced by ~15 Gateways, but is not tracked in GitOps — so the live, load-bearing issuer is drift. This imports it so ArgoCD manages it. The committed spec matches the live object exactly (verified against \`kubectl get clusterissuer vault-issuer -o yaml\`), so adoption is a no-op.
## Changes
- Add \`apps/base/cert-manager/clusterissuer_vault-issuer.yaml\` capturing the live spec byte-faithfully: server \`https://vault.service.consul:8200\`, path \`pki_int/sign/servers_default\`, k8s auth mount \`/v1/auth/k8s/au/syd1\`, role \`cert_manager_issuer\`, serviceAccountRef \`cert-manager-vault-issuer\` (audience \`vault\`), caBundleSecretRef \`vault-ca-cert\`/\`ca.crt\`. Helm ownership labels/annotations kept so adoption produces zero diff.
- Register the manifest in the cert-manager base kustomization (inserted between \`clusterrolebinding.yaml\` and \`vmservicescrape.yaml\` to avoid the lines #327 touches).
## Depends on #327
ArgoCD can only adopt this resource once \`{group: cert-manager.io, kind: ClusterIssuer}\` is in the platform project \`clusterResourceWhitelist\`. That whitelist entry is added by #327, not here (to avoid a duplicate/conflicting change). **Merge #327 first.** There may be a small merge conflict with #327 in \`apps/base/cert-manager/kustomization.yaml\` (both append to the \`resources\` list); rebase on main after #327 merges.
## Note: SA name discrepancy (not fixed here — committing live spec unchanged)
The live issuer authenticates as SA **\`cert-manager-vault-issuer\`**, but the repo scaffolding \`serviceaccount.yaml\` creates SA **\`vault-issuer\`**. Both SAs exist live in \`cert-manager\` (192d and 136d). The issuer uses \`cert-manager-vault-issuer\`, so this PR commits that name (live truth). The repo-managed \`vault-issuer\` SA appears unused by this issuer — worth a follow-up to reconcile which SA is canonical, but out of scope for a zero-change import.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #328
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
We self-delegate `_acme-challenge.unkin.net` into an `acme.unkin.net` zone we serve ourselves, so cert-manager can solve Let's Encrypt DNS-01 over RFC2136/TSIG. That needs a publicly-reachable authoritative BIND, separate from the internal estate.
- Add app `bind-external` (base + au-syd1 overlay); register it in the platform ApplicationSet and AppProject destinations (bind-operator already watches all namespaces).
- Add BindCluster `bind-external`: authoritative-only, recursion off, no forwarding, transfers denied except the keyed catalog/zone AXFR; 2 replicas; primaryService is a dmz-pinned PureLB LoadBalancer at `198.18.199.53`.
- Add BindZone `acme.unkin.net` (primary, dynamicUpdate) and BindTSIGKey `certmanager` (hmac-sha256), whose Secret `certmanager-tsig` reflects into the `cert-manager` namespace for the rfc2136 solver.
Pairs with argocd-apps #327 (the ClusterIssuers) and a one-time Google Cloud DNS delegation + NAT of the public IP :53 to `198.18.199.53`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #329
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Publicly-trusted wildcard certs via Let's Encrypt DNS-01, solved over RFC2136/TSIG against our own BIND. A one-time CNAME self-delegates `_acme-challenge.unkin.net` into the `acme.unkin.net` zone served by bind-external; cert-manager writes the challenge TXT there. No GCP/clouddns and no Vault secret involved. The existing `vault-issuer` (internal PKI) is untouched.
- Add ClusterIssuers `letsencrypt` (prod) and `letsencrypt-staging`, both using a dns01 rfc2136 solver: nameserver `198.18.199.53:53`, key `certmanager`, HMACSHA256, `tsigSecretSecretRef` -> reflected Secret `certmanager-tsig` key `secret`.
- Whitelist `cert-manager.io ClusterIssuer` in the platform AppProject.
Depends on #329 (bind-external: the acme.unkin.net zone, the certmanager TSIG key reflected into cert-manager, and the 198.18.199.53 nameserver) and on the one-time Google Cloud DNS delegation + NAT of the public IP :53 to 198.18.199.53. Earlier clouddns/Vault commits on this branch are reverted.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #327
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: the logging aggregator wrote every event through the generic catch-all shape. The Tier-1 survey picked six high-value log sources that warrant structured parsing into logs.raw columns/fields for real querying.
How:
- Two-stage routing in `aggregator.yaml`: `app_route` peels off the six Tier-1 streams by subject / VM source tag (mutually exclusive — no double-insert); everything else falls through `app_route._unmatched` to the unchanged generic k8s/vm catch-all.
- Six parse transforms emit the full `logs.raw` shape plus structured `.fields` (Map(String,String) — no DDL change): authentik (JSON), traefik (JSON access), vault audit (JSON), nginx access+error (regex), haproxy httplog (regex), glauth (JSON).
- Companion flip: traefik-system access logs to `format: json` (both overlays) so `traefik_parse` has structured input.
- 15 new `vector test` cases (routing + field extraction) in `aggregator-tests.yaml`; all green locally (vector 0.57).
Live now: authentik + traefik (k8s). Awaiting the puppet-side vector rollout (logs.vm.* with `.file`/`.SYSLOG_IDENTIFIER` tags per the documented convention): vault-file, nginx, haproxy, glauth — transforms are present and unit-tested so they light up automatically.
Note: geoip enrichment for nginx/traefik client IPs is a separate prerequisite — no enrichment table exists in the aggregator yet; these transforms extract `client_ip` ready for it.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #318
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Points the new gitea at its own k8s route so it can be exercised (login, browse, OIDC callback) before the git.unkin.net data cutover — ROOT_URL currently resolves to the live VM forge, which would break links on the k8s route. Flips back to git.unkin.net at cutover.
- set gitea DOMAIN/ROOT_URL/SSH_DOMAIN to git.k8s.syd1.au.unkin.net
Reviewed-on: #317
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Move the estate's own service/operator image pulls off the Gitea container registry (git.unkin.net/unkin) to the artifactapi local docker registry (docker-internal), ahead of the git.unkin.net forge migration which disables Gitea's container registry. The images were copied digest-for-digest into docker-internal and pulls verified before this repoint.
- repoint age-api, bind-operator, bind-tsig-api, cephrgw-operator, encapi, logarchiver, pdbmux image pulls to artifactapi.k8s.syd1.au.unkin.net/docker-internal
Deliberately not repointed here: artifactapi's own api/ui images (circular — it can't pull itself from itself), and the almalinux9-* base/CI images (huge, and their per-repo Woodpecker push targets move in a batched follow-up). Forge raw-CRD URLs and git clone sources are unaffected (those stay on the forge).
Reviewed-on: #310
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md.
- add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute
- add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222
- register gitea in the platform ApplicationSet + AppProject
- add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback)
Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc.
Reviewed-on: #309
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Final step of the ec migration: the old buckets were purged in #315, so the operator will now create fresh ones on the ec placement target. Restores the nine Bucket CRs with placementTarget: ec and retainOnDelete: true (purge disabled again for safety).
- re-add the nine cnpg backup Bucket resources on ec, retainOnDelete: true
Reviewed-on: #316
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Step 2 of the ec placement migration: with purge-on-delete now allowed (#313), removing the Bucket CRs makes the operator delete the underlying RGW buckets and their objects, freeing the names to be recreated on ec in step 3.
- remove the nine cnpg backup Bucket resources
- keep ObjectStoreUser and ScheduledBackup so the backup-s3 secrets and schedules survive
Reviewed-on: #315
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Authentik is canonical at https://identity.unkin.net, served by the internal
unkin.net CA. Grafana, LiteLLM and NetBox failed OIDC discovery because their
images don't trust that CA (x509: unknown authority); NetBox also still pointed
at the secondary admin host.
- grafana: mount the reflected vault-ca-cert; set generic_oauth `tls_client_ca`.
- litellm: `combine-certs` init builds public+internal CA bundle; `SSL_CERT_FILE`
+ `REQUESTS_CA_BUNDLE` point at it.
- netbox: flip OIDC issuer to identity.unkin.net; same combine bundle for
python-social-auth (`requests`).
- docs: record the Rancher manual runtime step (issuer + CA in the auth config).
Validated: kustomize build + kubeconform + pre-commit.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #314
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Step 1 of moving the CNPG backup buckets to ec placement: RGW can't move an existing bucket, so they must be dropped and recreated. This lets the operator actually delete the buckets (with their objects) when the CRs are removed in step 2.
- set retainOnDelete: false and purgeOnDelete: true on all nine cnpg backup Bucket CRs
- leave ObjectStoreUser/BucketAccess untouched so the backup-s3 secrets persist
Reviewed-on: #313
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
cephrgw-operator v0.4.0 adds immutable placementTarget selection on Buckets; the nine CNPG backup buckets should live on the ec (4/1) placement instead of 3-replica. Existing buckets cannot change placement, so after this merges the buckets get deleted and recreated on ec and fresh base backups are triggered (day-old backups are accepted losses, per Ben).
- bump cephrgw-operator image and CRD ref to v0.4.0
- add placementTarget: ec to all nine cnpg backup Bucket CRs
Reviewed-on: #312
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Final-mile bringup: after #301/#306/#308 the auth chain was fixed and logs flowed, but the `nats-bootstrap` PostSync hook **failed** with:
```
nats: error: could not pick a Stream to operate on: ... could not load schema { ... }: stat .: permission denied
```
The nats CLI stats its **working directory** when loading response-validation schemas. Under the Job's `readOnlyRootFilesystem: true` + `runAsUser: 1000`, the nats-box image's default WORKDIR isn't accessible to uid 1000, so every `nats stream/consumer` call errored. (A throwaway pod using default securityContext worked, which is why manual stream creation succeeded.)
Consequence: the PostSync hook never completes → `logging-logging` stays **OutOfSync**. The `LOGS` stream + consumers persist in JetStream once created, so log flow is unaffected — but GitOps convergence is blocked and the hook would keep retrying.
## What
Set `workingDir: /tmp` on the bootstrap container (the writable emptyDir already mounted for `HOME`). The nats CLI can then stat/operate normally.
**Verified on the live cluster:** a nats-box pod with the Job's exact restrictive securityContext + `workingDir: /tmp` runs `nats stream info LOGS` cleanly (fails without it).
## Note (separate, pre-existing)
There is also a first-deploy ordering deadlock: the `nats-bootstrap` PostSync hook runs only after the Sync-phase resources are healthy, but the vector consumer Deployments can't become healthy until the hook creates the `LOGS` stream. On this deploy I broke the deadlock by creating the stream/consumers manually (idempotent with the Job); the stream now persists so it won't recur on normal re-syncs, but a fresh cluster / PVC loss would hit it again. A durable fix (sync-waves so bootstrap runs after NATS but before the consumers) is worth a follow-up — flagged, not included here to keep this fix minimal.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #311
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed.
## Changes
- Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring.
- Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch.
- Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`).
- Remove the vector-archiver Helm release, values and pipeline ConfigMap.
Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged).
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #308
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient).
### Evidence
- Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped.
- The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected.
- **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**.
### Root cause
The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them.
## What
Wrap all three user passwords in `<< >>`:
```
password: << $NATS_ADMIN_PASSWORD >> # (+ producer, consumer)
```
Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted).
This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password.
## Verified end-to-end
nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully.
## Expected recovery after merge + sync
1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords.
2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config.
3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes.
4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3.
5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises.
ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix.
## Validation
kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #306
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here.
## Root cause 1 — Vector env-var interpolation is off by default in 0.57
`vector-aggregator` and `vector-archiver` crash-looped with `async_nats::connector: authorization violation`; `vector-agent` / `vector-vm-ingest` were "Running" but silently failing to publish.
Diagnosis (evidence):
- The NATS server config **does** expand `$NATS_*_PASSWORD` (a `nats` CLI login with the real secret value authenticated fine), and the ACL was **not** the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation).
- The failure reproduces locally: a hardcoded password connects; the **same value via `${NATS_CONSUMER_PASSWORD}` fails**. Configuring the server to expect the literal string `${NATS_CONSUMER_PASSWORD}` makes Vector connect — proving **Vector sends the literal, un-interpolated string**.
- `vector --help` shows `--dangerously-allow-env-var-interpolation` — in 0.57 `${VAR}` interpolation is **opt-in**. An unset-var test confirms interpolation is off (no "unknown env var" error).
- Verified fix: with `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` → **connects and authenticates**.
Every tier uses `${...}` for auth (`${NATS_*_PASSWORD}`, `${CLICKHOUSE_*}`), so the env var is added to **all four** vector deployments. (This slipped past CI because `vector test` never opens the NATS connection.)
## Root cause 2 — operator watches only its own namespace
`kubectl get chi -n logging` showed the `logs` CHI existed but with **empty status / no finalizer** — the operator never touched it, so the `logging-logging` Argo sync was stuck `Progressing` on *"waiting for healthy state of ClickHouseInstallation/logs"*, and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed).
Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states `watchNamespaces: []` (our value) makes the operator **watch only its own namespace** (`clickhouse-system`). The CHI is in `logging`.
Fix: `watchNamespaces: ["logging"]` → operator config `watch.namespaces.include: [logging]`.
## Changes
- `apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml`: add `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true`.
- `apps/overlays/au-syd1/clickhouse-system/values.yaml`: `watchNamespaces: ["logging"]`.
No NATS ACL change (the original narrow ACL is correct). No secret/base changes.
## Expected recovery after merge + sync
1. clickhouse-system syncs → operator config gains `logging` → operator restarts → reconciles the `logs` CHI → CHI pod comes up healthy.
2. `logging-logging` sync unblocks → PostSync hooks run → JetStream `LOGS` stream + `transform`/`archiver` consumers created; `logs.raw` table created.
3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3.
4. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` increases.
## Validation
kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders `watch.namespaces.include: [logging]`; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it).
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #301
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
CNPG WAL archiving to Ceph RGW fails with CERTIFICATE_VERIFY_FAILED on six clusters because the reflected vault-ca-cert bundle carries a corrupt intermediate: the genuinely-signed cert has a typo'd AIA URL (vault.servuce.consul), and the committed copy was text-edited at some point to fix the typo — flipping one byte of signed data and invalidating the signature (openssl verify: error 7 certificate signature failure). Only radosgw surfaces it because it serves a bare leaf, forcing clients to verify the stored intermediate against the root; services presenting their own intermediate never exercised the corrupt copy. terraform-k8s's copy is defunct per Ben — this file is the authoritative source.
- restore the original signed intermediate (one base64 character; sha256 E0:13:1B..., verified against the root, and the resulting bundle validates the live s3.ceph.unkin.net leaf)
- add an explicit allow-plain-secret marker mechanism to ci/validate-no-secrets.sh for public-data bootstrap secrets, and mark vault-ca-cert.yaml with it (a CA bundle is public and cannot be Vault-sourced since it establishes Vault trust)
After merge+sync the reflector propagates to all namespaces and barman's next retry (~1min) succeeds; base backups run on tonight's schedule. Follow-ups worth considering: re-issue the intermediate in Vault with a corrected AIA URL, and/or configure radosgw to serve its intermediate.
Reviewed-on: #305
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The CNPG buckets are empty after PR #298. Live diagnosis found **two** independent
causes; this PR fixes one of them.
`litellm`, `puppet` and `paperclip` run the CNPG `17-minimal-trixie` image, which
omits the `barman-cloud` CLI. In-tree `barmanObjectStore` archiving shells out to
`barman-cloud-wal-archive`, so their archiver dies immediately:
```
ContinuousArchiving=False :: unexpected failure invoking barman-cloud-wal-archive:
exec: "barman-cloud-wal-archive": executable file not found in $PATH
```
(verified on the live `puppet-postgres` primary: `which barman-cloud-wal-archive` →
not found; on a `-system` cluster it resolves to `/usr/local/bin/barman-cloud-wal-archive`).
## How
Switch those three clusters from `17-minimal-trixie` to `17-system-trixie` — the
`-system` variant already used by the other six clusters, which bundles the
barman-cloud tools. Tag confirmed present upstream (ghcr manifest HTTP 200).
```
- imageName: ghcr.io/cloudnative-pg/postgresql:17-minimal-trixie
+ imageName: ghcr.io/cloudnative-pg/postgresql:17-system-trixie
```
CNPG applies this as a rolling image update (switchover, no data change).
## Not fixed here (separate, primary blocker)
The other six clusters (full image, barman present) fail with a **TLS trust**
error — the reflected `vault-ca-cert` bundle carries a **stale intermediate CA**,
so barman can't verify `s3.ceph.unkin.net`:
```
SSL: CERTIFICATE_VERIFY_FAILED ... certificate signature failure
```
That is a shared trust-anchor refresh (likely owned by terraform-k8s /
`config/certificates/secret.yaml`, `managed-by: terragrunt`), handled separately —
it also gates litellm/puppet once they have barman. See the investigation report.
## Validation
- `kustomize build --enable-helm` + `kubeconform` pass on `litellm`, `puppet`
overlays and the `paperclip` base (paperclip has no overlay yet).
- `pre-commit run` passes on all changed files.
## Follow-ups
- Longer term, the Barman Cloud Plugin (sidecar) would let minimal images keep
their size while still archiving — track with the plugin migration.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #304
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
ArgoCD SSO fails with `failed to query provider "https://identity.unkin.net/application/o/argocd/": tls: failed to verify certificate: x509: certificate signed by unknown authority`. argocd-server does OIDC discovery to identity.unkin.net over TLS served by the internal `unkin.net` CA. Unlike argocd-repo-server (which mounts `vault-ca-cert`), argocd-server has no internal CA in its trust store and no `rootCA` in `oidc.config`, so it never trusted the issuer.
## Change
- argocd-cm `oidc.config`: add `rootCA` (inline PEM) = the internal `unkin.net` root CA. argocd-server hot-reloads argocd-cm, so no rollout restart is required.
## Why the root, not the cluster vault-ca-cert bundle
The `unkin.net Intermediate Authority` was recently **re-keyed** (same serial, new key: bundle SHA1 `C4:48:78…` vs served `F1:DD:34…`). The cluster `vault-ca-cert` bundle still carries the **stale** intermediate and fails `openssl verify` against the currently-served identity cert. identity.unkin.net presents its current intermediate in the handshake, so anchoring on the long-lived, stable `unkin.net` root (valid to 2034, matches the host trust anchor) is both correct and rotation-proof. Verified: `openssl verify -CAfile <root> -untrusted <served-intermediate> <served-leaf>` = OK; the embedded PEM round-trips through the YAML patch and validates the served leaf.
## Verify after merge
argocd-server picks up argocd-cm live; retest SSO login. (Separately, the cluster `vault-ca-cert` reflected secret carries a stale intermediate and should be refreshed, but that is out of scope here.)
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #303
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Upgrade the Woodpecker CI instance from v3.14.1 to v3.16.0 to pick up upstream fixes and the k8s-backend privilege-escalation hardening (GHSA-qf34-295c-26v8). Server and agent must move together.
The chart is pulled at build time (`kustomize build --enable-helm`); the image tag defaults to the chart `appVersion`, so bumping the chart moves both images. Chart 3.6.5 -> appVersion 3.16.0 (verified against upstream `helm/charts/woodpecker/Chart.yaml`); subchart deps are unchanged from 3.6.3 (server 3.0.1, agent 2.0.1), so there is no values-schema migration.
## What
- Bump the woodpecker helmChart from `3.6.3` (appVersion 3.14.1) to `3.6.5` (appVersion 3.16.0) in the au-syd1 overlay. Server + agent both render as `v3.16.0`.
- Set `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true"` on the agent.
## CRITICAL: k8s backend serviceAccountName gating (required change)
v3.16.0 (PR #6792, GHSA-qf34-295c-26v8) gates step-level `serviceAccountName` behind a new agent flag `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP`, **default `false`**. When disabled, any `backend_options.kubernetes.serviceAccountName` set by a pipeline is **silently ignored** and the namespace `default` SA is used instead.
Every terraform pipeline in the estate sets `backend_options.kubernetes.serviceAccountName` (e.g. `terraform-git`, `terraform-vault`, `terraform-artifactapi`) and relies on that SA for Vault k8s auth / Consul state. Without this flag those jobs would run as `default` and lose their Vault identity. This PR sets the flag to `true` to preserve current behaviour. No other newly-gated backend_options keys (pod labels/annotations from step, native secrets) are used by the estate.
## Migration / rollback
- **DB migration:** Woodpecker auto-migrates the schema (xorm) on server start; migrations are forward-only and NOT reversible. The 3.14 -> 3.16 changelogs do not call out a data-destructive migration, but a **DB backup (CNPG cluster `woodpecker`) should be taken before merge**.
- **In-flight pipelines:** merging rolls the server StatefulSet and agents; any running pipelines are interrupted and will need re-running.
- **Rollback:** re-pin chart `3.6.5` -> `3.6.3` reverts the images to v3.14.1, but because migrations are one-way, a clean rollback requires **restoring the CNPG DB from the pre-merge backup**, not just pinning the old image.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/woodpecker` renders `woodpecker-server:v3.16.0` and `woodpecker-agent:v3.16.0`; agent carries the new env var.
- `kubeconform` (k8s 1.33.7): 24/24 resources valid.
- pre-commit (yamllint + checks): all pass.
## Follow-up (not in this PR)
Woodpecker images are pulled from `docker.io` / `ghcr.io` directly, not the artifactapi proxy. Proxying them via artifactapi is a possible follow-up but out of scope for this version bump.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #297
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists.
## Topology
`edge (publishers) → JetStream → consumers → sinks`
- **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets.
- **Edge publishers (thin)** — no parsing, just a routing subject:
- `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`.
- `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.)
- **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now).
- **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`.
- **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job.
## Streams / consumers / auth
- Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment.
- **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret.
## S3 / retention
`ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here.
## Replay runbook
- **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time.
- **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window.
## Validation
- `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**.
- `pre-commit` (yamllint, check-json, no-plain-secrets) clean.
- **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs.
- **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets).
## Known upstream caveat
Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook.
## Prerequisites (manual, one-time)
```
# ClickHouse
PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1)
vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \
username=vector password="$PW" password_sha256_hex="$HASH"
# NATS
vault kv put kv/kubernetes/namespace/logging/default/nats-auth \
admin_password=$(openssl rand -base64 24) \
producer_password=$(openssl rand -base64 24) \
consumer_password=$(openssl rand -base64 24)
```
No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR.
## Open decisions (defaults chosen, flag to change)
- **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.**
- **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side).
- **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer.
- **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA.
- **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs).
- **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---
## Update: images via artifactapi, DHI, stateless transform tier
**Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull.
### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`)
| Image | Upstream | artifactapi path | DHI? |
|---|---|---|---|
| clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job |
| altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI |
| altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI |
| bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI |
| nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats |
| natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI |
| natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI |
| timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened |
| timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step |
**DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace.
### Transform tier is now a stateless Deployment
Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**.
**Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.)
---
## Update: 7d retention, tunable limits ConfigMap, honest sizing
- **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict.
- **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3.
- **Honest 7d sizing (stated assumption — please sanity-check against real volume):**
- Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica.
- Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica.
- **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.**
- ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob.
- Replay window in the runbook is now **7d** (beyond that → the S3 archive).
---
## Update: retention cut to 3 days (both stores), PVCs shrunk
Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design.
| Store | Retention | Byte cap | PVC/node | Replicas | Total |
|---|---|---|---|---|---|
| NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) |
| ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) |
**NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC.
**ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table.
The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL.
⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit).
Reviewed-on: #296
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Authentik's intermittent API 500s (which failed two terraform-authentik CI runs today) were traced to its CNPG postgres replicas being OOMKilled: 512Mi limits leave no headroom over shared_buffers 128MB + max_connections 200, both OOM events matched the 500 bursts to the second, and the session-pinned RO pooler turns each replica death into a batch of severed read connections. The primary is at 84% of its limit and is next.
- raise the authentik CNPG memory limit from 512Mi to 1Gi and request from 256Mi to 512Mi
Follow-up candidates (not in this PR): RO pooler poolMode session→transaction to shrink the blast radius of a replica loss; revisit max_connections/shared_buffers sizing.
Reviewed-on: #300
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
NetBox becomes the source of truth for host/interface/IPAM data as part of retiring Cobbler. The ENC role already moved to encapi; a kickstart-generation microservice that consumes NetBox comes later and is intentionally **out of scope** here.
## Change
Adds the `netbox` app (namespace `netbox`, platform project) using the netbox-community helm chart via the OCI helm-through-kustomize pattern, plus estate-native data stores:
- **NetBox** — chart `8.3.40` (appVersion `v4.6.5`), `oci://ghcr.io/netbox-community/netbox-chart`. 2 web replicas, 1 worker; bundled Postgres/Valkey subcharts disabled. Media on RWX CephFS so replicas share uploads. Chart `helm.sh/hook: test` Pod dropped via a kustomize delete patch (we deploy through ArgoCD, not `helm test`).
- **CNPG Postgres** — `netbox-postgres`, 2 instances, PG 18, `cephrbd-fast-delete`. Standard per-cluster S3 backup: `barmanObjectStore -> s3://cnpg-netbox`, cephrgw `ObjectStoreUser` + `Bucket`, nightly `ScheduledBackup` at **03:40** (`0 40 3 * * *` — next free slot after grafana's 03:20), 30d retention. A pgbouncer `Pooler` (session mode) fronts it; NetBox connects via `netbox-postgres-pooler-rw`.
- **Valkey** — standalone Deployment (`valkey/valkey:8-alpine`), AOF-persistent PVC on `cephrbd-fast-delete`. One instance: DB 0 = RQ task queue, DB 1 = cache. No auth (in-cluster, namespace-isolated). Chosen over the bundled Bitnami subchart to keep image control in-estate and avoid Bitnami's legacy-image churn; mirrors the litellm standalone-cache pattern.
- **Ingress** — `Gateway` + `HTTPRoute` at `netbox.k8s.syd1.au.unkin.net` (`traefik-internal`, `vault-issuer` cert into `netbox-tls`, external-dns to the internal VIP), HTTP->HTTPS 301.
- **Secrets** — all via VSO `VaultStaticSecret` (`postgres-credentials`, `netbox-secret-key`, `netbox-superuser`, `oauth-credentials`); no plain Secrets committed. The shared `default` k8s-auth role already binds `*` namespaces with a namespace-templated KV policy, so **no terraform-vault change is needed**.
- **Authentik OIDC SSO** — `remoteAuth` wires `REMOTE_AUTH_ENABLED` + the `OpenIdConnectAuth` backend via chart values; `SOCIAL_AUTH_OIDC_*` via `extraConfig` (the chart's config loader globs `/run/config/extra/*/*.yaml`). The client secret is injected as a YAML fragment mounted from the Vault-synced `oauth-credentials` secret. New users auto-provision on first login.
- Registers `netbox` in the platform ApplicationSet and AppProject (destination namespace + chart sourceRepo).
## Image table (source -> artifactapi -> DHI decision)
All images flow through the estate's containerd registry mirrors; the allowlist patterns gate them. DHI (Docker Hardened Images) require authenticated pulls from a Docker Hub `dhi/` org and are **not** reachable through the anonymous mirror, so upstream official is used throughout.
| Image | Upstream | Mirror / allowlist | Allowlisted? | DHI decision |
|---|---|---|---|---|
| NetBox app/worker/housekeeping | `ghcr.io/netbox-community/netbox:v4.6.5` | ghcr remote, `^netbox-community/` | **added in terraform-artifactapi #17** | No DHI published; upstream official |
| CNPG Postgres | `ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie` | ghcr, `^cloudnative-pg/` | already | estate-standard CNPG image |
| Valkey | `docker.io/valkey/valkey:8-alpine` | dockerhub, `^valkey/valkey` | already | DHI not anon-pullable; upstream official |
| NetBox init (perms) | `docker.io/busybox:1.38.0` | dockerhub, `^library/busybox` | already | upstream official |
| Worker wait-for-backend | `docker.io/rancher/kubectl:v1.36.2` | dockerhub, `^rancher/` | already | upstream official |
## Cross-repo PRs (merge order)
1. **terraform-artifactapi #17** (`^netbox-community/` ghcr allowlist) — **merge before** this PR so the NetBox image pulls on first sync.
2. **terraform-authentik #11** (OIDC provider/application) — independent; SSO works once applied + the secret is seeded.
3. This PR.
> Note: the CNPG S3-backup stanza pattern here is identical to PR #298 (already in main); it merges cleanly regardless of ordering.
## One-time Vault seeds (before/at first sync)
```
# App DB user (CNPG bootstrap + NetBox both consume this)
vault kv put kv/kubernetes/namespace/netbox/default/postgres-credentials \
username=netbox password="$(openssl rand -base64 30)"
# Django SECRET_KEY
vault kv put kv/kubernetes/namespace/netbox/default/netbox-secret-key \
secret_key="$(python3 -c 'import secrets;print(secrets.token_urlsafe(60))')"
# Bootstrap superuser
vault kv put kv/kubernetes/namespace/netbox/default/netbox-superuser \
username=admin email=admin@unkin.net \
password="$(openssl rand -base64 24)" api_token="$(openssl rand -hex 20)"
# OIDC client secret — ONE value stored two ways (raw for Authentik, YAML for NetBox)
CS="$(openssl rand -base64 30)"
vault kv put kv/kubernetes/namespace/netbox/default/oauth-credentials \
client_secret="$CS" oidc.yaml="SOCIAL_AUTH_OIDC_SECRET: \"$CS\""
```
Seed `oauth-credentials` **before** applying terraform-authentik #11 (that apply reads `client_secret`).
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/netbox` — clean.
- `kubeconform` (CI args, k8s 1.33.7): **27/27 valid, 0 invalid**.
- `pre-commit` (yamllint + no-plain-secrets) on all changed files — pass.
## Out of scope
The PXE/kickstart microservice that will consume NetBox — not scaffolded here.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #299
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A
lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable
data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip
and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for
every cluster, plus code-forward restore docs.
## What
- **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous
WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2,
30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected
`vault-ca-cert` (`endpointCA`).
- **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator
provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is
hardcoded**) and a staggered nightly `ScheduledBackup`.
- **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can
validate the new CRs.
- **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`).
## Design decisions (answers to the open questions)
**One bucket for all, or per-database?** → **Per-database (one bucket + owner user per
cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only
*within the same namespace*), and CNPG reads its S3 credential Secret from its *own*
namespace. A single shared bucket would require either cross-namespace bucket refs
(unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator
mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the
simplest correct topology and needs zero manual seeding. Each user owns exactly one
bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess`
grant needed.
**Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart
`cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree
`barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not
deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the
in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`).
## Schedule / retention (defaults — Ben to adjust)
| App | Cluster | Bucket | Nightly base backup |
| --- | --- | --- | --- |
| authentik | postgres | cnpg-authentik | 01:00 |
| litellm | litellm-postgres | cnpg-litellm | 01:20 |
| artifactapi | postgres | cnpg-artifactapi | 01:40 |
| woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 |
| puppet | puppet-postgres | cnpg-puppet | 02:20 |
| encapi | postgres | cnpg-encapi | 02:40 |
| paperclip | paperclip-postgres | cnpg-paperclip | 03:00 |
| grafana | postgres | cnpg-grafana | 03:20 |
Retention is **30d** across the board — flagged as a default to tune per cluster.
Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once.
## Validation
- `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph
schemas) pass on all 8 affected overlays (paperclip validated at base — it has no
overlay yet). ceph CRs resolve their schemas (`Skipped: 0`).
- `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.).
- Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated
`cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) —
pre-existing, reproduces on `origin/main`, unrelated to this change.
## Notes / caveats
- No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging
PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json`
files, which merge cleanly whichever lands first.
- Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The
only prerequisite is the operator being healthy (it is, in `cephrgw-system`).
## Follow-ups
- Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs).
- Tune per-cluster retention / schedule if the defaults don't fit.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #298
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #294: --create-ptr validates that PTR is in --managed-record-types, and setting that flag replaces the default list, so A/AAAA/CNAME are re-stated alongside PTR (verified against the v0.21.0 binary's --help: default A,AAAA,CNAME). Pod is crash-looping on config validation until this merges.
- Adds --managed-record-types=A,AAAA,CNAME,PTR to the external-dns args
MERGE ASAP.
Reviewed-on: #295
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
external-dns v0.21.0 rejects --rfc2136-create-ptr (my error in #292) and the pod is crash-looping on flag parsing. The correct flag in this version is the generic --create-ptr, which synthesizes PTRs for A records whose reverse zone is in the domain filter (200.18.198.in-addr.arpa already is).
- Replaces --rfc2136-create-ptr with --create-ptr
Record reconciliation is stalled until this merges (serving unaffected — bind answers normally). MERGE ASAP.
Reviewed-on: #294
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Post-cutover verification (argocd-apps#288) found the reverse zone 200.18.198.in-addr.arpa empty: external-dns only writes PTRs when --rfc2136-create-ptr is set, and nothing else feeds that zone. Reverse resolution for the k8s LB range was already absent for clients pre-migration (no forwarder existed), so this completes the reverse path rather than fixing a regression.
- Adds --rfc2136-create-ptr to the external-dns rfc2136 args
Verification after merge: dig -x 198.18.200.2 @198.18.200.7 returns puppetca.k8s.syd1.au.unkin.net (allow a reconcile cycle + negative-cache expiry).
Reviewed-on: #292
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy bind-operator v0.2.6 (bind-operator#15): NOTIFYs are now TSIG-signed via the catalog transfer key and secondaries accept by key, with no pod IPs in restart-scoped config — a regression test asserts the config-hash is invariant under pod IP churn, making the v0.2.5 roll-loop class impossible. Restores seconds-fast dynamic-zone propagation on bind-externaldns and bind-authoritative.
- Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.6 (confirmed in registry)
- Bumps the CRD install pin to the v0.2.6 tag
Expect exactly ONE settling roll of the bind statefulsets when the new config lands, then stability.
Reviewed-on: #293
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
v0.2.5 renders the primary POD IP into the options-scope allow-notify. Options changes are restart-scoped (config-hash annotation), and every roll gives the primary a new pod IP, so the operator re-renders and rolls all bind clusters in an endless loop (externaldns, authoritative, resolvers all cycling ~45s pods right now).
- Reverts the operator image and CRD pin to v0.2.4
A v0.2.6 will re-do the NOTIFY fix loop-free (allow-notify via zone-scope/rndc-applied config or a TSIG-keyed notify instead of pod-IP-in-options). MERGE ASAP to stabilize DNS.
Reviewed-on: #291
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 3 of 3 in the external-dns → in-cluster bind migration: the client-visible cutover. The `openforwarder` resolvers currently forward `k8s.syd1.au.unkin.net` to the legacy VM anycast `198.18.19.20` (a temporary measure — commit 7ee5dfb) and have NO forwarder at all for the reverse zone `200.18.198.in-addr.arpa`. Once external-dns publishes to the in-cluster `bind-externaldns` (PR 2), resolvers must read from it.
## Changes
- Repoints the `fwd-k8s-syd1-au-unkin-net` forwarder from `198.18.19.20` (legacy VM) to `198.18.200.8` (in-cluster `bind-externaldns` VIP).
- Adds `fwd-200-18-198-in-addr-arpa` forwarding `200.18.198.in-addr.arpa` → `198.18.200.8`, closing the reverse-zone gap so PTR lookups for the k8s LB range keep resolving after cutover. Modeled exactly on the existing forward-zone entries.
- Refreshes the header comment to describe the in-cluster upstream.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean; both zones render with forwarder `198.18.200.8` and there is no residual `198.18.19.20`.
## Merge gate
- PR 2 (`benvin/externaldns-incluster`) merged, AND
- record parity confirmed between legacy and in-cluster for the forward zone. Spot-check (repeat for each name):
```
for n in puppetca puppet puppetdb encapi pdbmux artifactapi consul; do
echo "$n:"
dig +short @198.18.19.20 A $n.k8s.syd1.au.unkin.net
dig +short @198.18.200.8 A $n.k8s.syd1.au.unkin.net
done
# plus 2-3 PTRs in the reverse zone:
dig +short @198.18.19.20 -x 198.18.200.8
dig +short @198.18.200.8 -x 198.18.200.8
```
A/PTR answers from `198.18.200.8` must match those from `198.18.19.20` before merging.
## Verification (after merge)
```
dig +short @198.18.200.7 A puppet.k8s.syd1.au.unkin.net # resolvers VIP
dig +short @198.18.200.7 -x 198.18.200.8 # reverse via resolvers
```
Resolution through the `bind-resolvers` VIP should now answer for both the forward and reverse k8s zones.
## Rollback
Revert this PR — the `fwd-k8s-syd1-au-unkin-net` forwarder returns to `198.18.19.20` and the reverse forward is removed. The legacy VM is untouched and remains authoritative until decommission.
Reviewed-on: #288
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator **v0.3.1** logs a startup WARNING when its installed CRDs are missing or older than the operator (operator repo #6, merged) — added precisely because the CRD tag drifted behind the image before. The new startup check reads the `CustomResourceDefinition` objects, so it needs a small RBAC grant.
## Changes
- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.1`
- bump the CRD `install.yaml` tag → `v0.3.1` (keep CRDs in step with the image — the invariant the v0.3.1 warning enforces)
- add `apiextensions.k8s.io/customresourcedefinitions: [get, list]` to the operator ClusterRole so the startup check is not RBAC-denied
Validated with `kustomize build` on the au-syd1 overlay; the v0.3.1 CRD URL resolves. Supersedes nothing outstanding (the earlier CRD-tag PR #286 to v0.3.0 already merged).
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #290
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy bind-operator v0.2.5 (bind-operator#14): secondaries now carry an explicit allow-notify for the primary pod IP, so dynamic-zone updates propagate in seconds instead of the ~1h SOA refresh. Unblocks the external-dns migration parity gate (argocd-apps#288) and speeds up the dns-updater zones on bind-authoritative.
- Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.5 (confirmed in registry)
- Bumps the CRD install pin to the v0.2.5 tag
On sync the operator re-renders cluster ConfigMaps; the config-hash change rolls the bind secondaries, which then accept the primary's NOTIFYs.
Reviewed-on: #289
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 2 of 3 in the external-dns → in-cluster bind migration. external-dns currently pushes RFC2136 updates to the legacy VM `ausyd1nxvm2127.main.unkin.net`. This repoints it at the in-cluster `bind-externaldns` primary so the in-cluster zone becomes the live source of truth for `k8s.syd1.au.unkin.net` + `200.18.198.in-addr.arpa`.
## Changes
- Points `--rfc2136-host` at `bind-externaldns-primary.bind-internal.svc.cluster.local` (verified live: ClusterIP Service `bind-externaldns-primary` exists in `bind-internal`).
- Reads TSIG `secret` + `algorithm` from Secret `externaldns-key-tsig` (reflected by PR 1) instead of the Vault-backed `externaldns-tsig`.
- Keeps port, zones, keyname, and `txtOwnerId: k8s` unchanged.
- Leaves the old Vault manifests (`apps/base/externaldns/{vaultauth,vaultstaticsecret}.yaml`) in place as the rollback path; their removal is a later cleanup PR.
## Merge gate
- PR 1 (`benvin/externaldns-tsig-reflect`) merged, AND
- `kubectl -n externaldns get secret externaldns-key-tsig` returns keys `secret` + `algorithm`.
## TXT-registry note
`policy: sync` + `registry: txt`: on first reconcile against the (currently empty) in-cluster zone, external-dns re-creates all managed A/CNAME records and their ownership TXTs from scratch. This is expected and populates the zone.
## Verification (after merge)
```
kubectl -n externaldns logs deploy/externaldns --tail=100 | grep -Ei 'rfc2136|BADKEY|NOTAUTH|added|update'
dig +short @198.18.200.8 A puppet.k8s.syd1.au.unkin.net
```
Logs should show updates to `bind-externaldns-primary...` with NO `BADKEY`/`NOTAUTH`; sampled A records should start resolving against the in-cluster server (198.18.200.8) as the sync populates the zone.
## Rollback
Revert this PR (host + secret name back to `ausyd1nxvm2127.main.unkin.net` / `externaldns-tsig`). The legacy VM is untouched and still authoritative.
Reviewed-on: #287
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 1 of 3 in the external-dns → in-cluster bind migration. For external-dns to send RFC2136 updates to the in-cluster `bind-externaldns` primary, it must present the exact TSIG key the primary's `allow-update` accepts. The bind-operator generates that key material into Secret `externaldns-key-tsig` in `bind-internal`; reflecting it into the `externaldns` namespace removes the manual eyaml→Vault key sync and guarantees key parity.
## Changes
- Adds `spec.secretTemplate.annotations` to BindTSIGKey `externaldns-key` with the emberstack reflector hints: `reflection-allowed`, `reflection-allowed-namespaces: externaldns`, `reflection-auto-enabled`, `reflection-auto-namespaces: externaldns`.
- Regenerates `schemas/bind.unkin.net/bindtsigkey_v1alpha1.json` from the live CRD (deployed bind-operator v0.2.4 already exposes `secretTemplate` — the WIP branch's ">= v0.3.0" claim is stale) to add the `secretTemplate` property. Schema output is byte-identical to running `ci/generate-schemas.sh`.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean and renders the annotations onto the BindTSIGKey.
## Verification (after merge)
```
kubectl -n externaldns get secret externaldns-key-tsig \
-o jsonpath='{.data.secret} {.data.algorithm}{"\n"}'
```
Both `secret` and `algorithm` keys must be present (reflector mirrored the source Secret from bind-internal).
## Rollback
Revert this PR. The source Secret in bind-internal is unaffected; only the reflected mirror in `externaldns` is removed.
Merge order: this is PR 1/3. PR 2 (repoint external-dns) must not merge until the reflected secret is verified.
Reviewed-on: #285
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The `cephrgw-system` kustomization pinned the CRD source to **`raw/tag/v0.1.0/config/crd/install.yaml`**, so the in-cluster CRDs never gained the fields added since v0.1.0 — v0.2.0's fine-grained BucketAccess policy fields (`paths`/`actions`/`conditions`/`rawStatements`) and v0.3.0's adoption fields (`retainOnDelete`, `managePolicy`, `status.adopted`). The running operator is v0.3.0, so applying those specs fails with `strict decoding error: unknown field`. The image bumps (#273, #279) should have moved this tag too.
## Changes
- point the CRD `install.yaml` at `raw/tag/v0.3.0`
The tag must track the operator image tag on future bumps. Verified the v0.3.0 URL serves the new schema and `kustomize build` renders all 3 CRDs.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #286
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Turns on right-sizing telemetry for the whole estate. Adds a `VerticalPodAutoscaler` with `updateMode: "Off"` (recommendation-only, advise mode) for every Deployment and StatefulSet in `apps/base`. Off mode never evicts or mutates pods, so this is purely observational: the VPA recommender (added in the vpa-system PR) publishes suggested requests/limits in each VPA's status, and nothing acts on them until someone deliberately flips a mode.
## Changes
- Add one `vpa.yaml` per app under `apps/base/<app>/` containing a `<workload>-vpa` VerticalPodAutoscaler for each workload, and register it in that app's `kustomization.yaml`.
- Coverage: 17 workloads across 11 apps.
- age-api (age-api), artifactapi (api, redis, ui), authentik (redis), bind-system (bind-operator), cephrgw-system (cephrgw-operator), encapi (encapi), kanidm (kanidm StatefulSet), litellm (litellm, redis), paperclip (paperclip), pdbmux (pdbmux), puppet (puppetboard, puppetdb, puppetserver-compiler, puppetserver-master).
## Skipped (intentionally)
- **CNPG `Cluster` objects** (artifactapi, authentik, encapi, grafana, litellm, paperclip, puppet, woodpecker) — Postgres is managed by CloudNativePG, not a VPA target.
- **CronJobs** (puppet g10k/generate-types, reposync x4) — not VPA-able.
## HPA / VPA caveat
api, ui (artifactapi), litellm, and all four puppet deployments also carry an HPA. With `updateMode: "Off"` there is no conflict today (VPA only recommends). VPA objects targeting these carry an inline comment: do **not** flip to `Auto`/`Initial` while the HPA still autoscales on CPU/memory, or the two controllers will fight over the same resource. Move the HPA to a custom/non-resource metric first.
## Verification
- `kubectl kustomize` over every touched overlay: 9/11 overlays PASS rendering all their VPAs; 2 failures are pre-existing and unrelated to this change — `authentik` fails on a local helm-tooling flag error identically on origin/main, and `paperclip` has no `apps/overlays/au-syd1/paperclip` directory yet. Both apps' base kustomizations build clean and render their VPA.
## Merge gate
- **Requires the vpa-system CRDs PR (argocd-apps #281) to merge first.** These manifests use `autoscaling.k8s.io/v1 VerticalPodAutoscaler`; without the CRDs installed, ArgoCD sync fails on an unknown kind.
Reviewed-on: #283
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Rolls out the Vertical Pod Autoscaler control plane so the estate can gather right-sizing recommendations for every workload (advise mode, follow-up PR adds the per-workload VPA objects). Deploys the **recommender only**: advise mode never mutates pods, so the updater and admission-controller (and its mutating webhook) are intentionally omitted — fewer moving parts, no webhook in the admission path.
## Changes
- Add `apps/base/vpa-system/`: namespace, VPA CRDs (verticalpodautoscalers + verticalpodautoscalercheckpoints) pulled from the kubernetes/autoscaler repo at the pinned tag (same upstream-raw pattern node-feature-discovery uses), recommender-scoped RBAC (SA + metrics-reader/actor/status-actor/checkpoint-actor/target-reader), and the recommender Deployment.
- Add `apps/overlays/au-syd1/vpa-system/` referencing the base.
- Register `apps/overlays/*/vpa-system` in the platform ApplicationSet.
## Notes
- Pins upstream **vertical-pod-autoscaler-1.7.0** (latest stable, 2026-05-29) for both CRDs and the `registry.k8s.io/autoscaling/vpa-recommender:1.7.0` image.
- No platform AppProject change needed: `*-system` namespace destination plus CustomResourceDefinition/ClusterRole/ClusterRoleBinding are already whitelisted.
- The recommender image is pull-through-cached via terraform-artifactapi PR #13 (merge gate below).
## Merge gate
- Requires terraform-artifactapi PR #13 (adds the `autoscaling/vpa-` pattern to the k8s-registry docker remote) to land first so the recommender image is served through the cache.
Reviewed-on: #281
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Phase 2 of the consul migration: expose the HTTP API (not just the UI) at consul.k8s.syd1.au.unkin.net, now rebased onto main post-#280 (ACLs enabled).
- Adds a consul-http ClusterIP service targeting the server pods on 8500 (API + UI share the port, so the UI stays reachable at /ui/)
- Repoints the consul and consul-svc HTTPRoutes from consul-ui:80 to consul-http:8500
- Documents ACL-authenticated access in apps/base/consul/README.md: token from kv/kubernetes/namespace/consul/default/bootstrap-acl-token (VSO-synced), X-Consul-Token curl and consul CLI usage, UI token login, and the prefer-vault-minted-tokens note
Verification post-merge (ACLs are live, so authenticated): CONSUL_HTTP_TOKEN=$(vault kv get -field=token kv/kubernetes/namespace/consul/default/bootstrap-acl-token) && curl -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" https://consul.k8s.syd1.au.unkin.net/v1/status/leader
Rollback: revert the HTTPRoute backends to consul-ui:80.
Reviewed-on: #282
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Split out of #281 per review: the VerticalPodAutoscaler CRDs land first, together with their generated kubeconform schemas, so CI can validate the VPA objects that follow.
- Adds apps/base/vpa-system with the v1.7.0 VPA CRDs served via the artifactapi github remote (terraform-artifactapi#14, merged)
- Adds schemas/autoscaling.k8s.io/ (verticalpodautoscaler + checkpoint, v1 and v1beta2) generated with the same transform as ci/generate-schemas.sh (from the CRD manifest rather than the live cluster, since the CRDs are not installed yet)
- Wires the vpa-system overlay into the platform applicationset
Verified: kustomize renders both CRDs; a sample updateMode Off VPA passes kubeconform against the new schemas. Merge before #281 (recommender) and #283 (per-workload VPAs).
Reviewed-on: #284
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with `default_policy: deny` and `down_policy: extend-cache`; the k8s cluster currently runs with **ACLs disabled**. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the **same** `initial_management` token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git.
## Changes
- Enable `global.acls.manageSystemACLs` so the chart manages system ACL tokens/policies for consul components.
- Point `global.acls.bootstrapToken` at a pre-existing Kubernetes secret `consul-bootstrap-acl-token` (key `token`); chart 1.9.7 supports this, and when the secret is populated the `server-acl-init` job **skips bootstrapping** and adopts that token as the management token (renders `-bootstrap-token-secret-name`/`-bootstrap-token-secret-key`, verified in the kustomize output).
- Add a `VaultAuth` (mount `k8s/au/syd1`, role `default`, SA `default`) and `VaultStaticSecret` in the `consul` namespace that sync `kv/kubernetes/namespace/consul/default/bootstrap-acl-token` into the `consul-bootstrap-acl-token` k8s secret via VSO (mirrors the encapi pattern).
- Merge the `acl` block (`enabled`, `default_policy: deny`, `down_policy: extend-cache`, `enable_token_persistence`) into the server `extraConfig` to match the VM posture.
## OPERATIONAL NOTE — required BEFORE merge
The user MUST place the VM cluster's `initial_management` token in Vault first, or ACL bootstrapping will generate a *different* token and break the mirror:
```
vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token>
```
VSO then syncs it into the `consul-bootstrap-acl-token` secret before the `server-acl-init` job runs. No terraform-vault change is needed: the wildcard `default` k8s-auth role (`bound_service_account_namespaces: ['*']`) plus the templated `kv/kubernetes/default` policy already grant the `consul` namespace `default` SA read on `kv/kubernetes/namespace/consul/default/*`.
## Risk / expected behavior
- Enabling ACLs **rolls the 5 servers** (StatefulSet update) and runs a `server-acl-init` job.
- With `default_policy: deny`, previously-anonymous operations are denied. The `vault` service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace `vault`) via their `service_registration "consul"` stanza (catalog entry has `ServiceMeta.external-source: vault`, port 8200). After the flip this registration will be **denied** unless Vault is given a Consul ACL token with `service:write` on `vault` (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration.
- Anonymous HTTP API reads (e.g. `/v1/status/leader`) will also be denied post-merge unless a token is supplied — see PR 2 verification note.
## Ordering
Independent of the API-route PR (`benvin/consul-api-route`). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done.
Reviewed-on: #280
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator **v0.3.0** makes adopting pre-existing radosgw buckets/users safe (operator repo #5, merged): `retainOnDelete` on `ObjectStoreUser`/`BucketAccess`, non-destructive bucket-policy **merge** (+ `managePolicy`), non-destructive user attributes, and `status.adopted`.
## Changes
- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.0`
No credential or manifest change beyond the tag. Validated with `kustomize build` on the au-syd1 overlay.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #279
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the two new UI features released in v3.7.7: per-repo usage-instruction panels (artifactapi#105) and direct-download links for local repo files (artifactapi#106).
- Bumps artifactapi api image to git.unkin.net/unkin/artifactapi:v3.7.7
- Bumps artifactapi ui image to git.unkin.net/unkin/artifactapi-ui:v3.7.7
Both v3.7.7 images are confirmed published to the registry; kustomize builds clean.
Reviewed-on: #278
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The puppet-on-k8s compilers classify nodes with a uv/python ENC script (`encapi-enc`). Each fresh compiler pod resolves the script's python dependencies on first invocation, and that resolution fails on cold pods (observed exits 135/2), breaking puppet agent catalog compilation. `encapic` (git.unkin.net/unkin/encapic) is a stdlib-only Go replacement with no runtime dependency resolution — a behavioural drop-in whose output matches the python script byte-for-byte.
## Changes
- Points the compiler `external_nodes` at `/opt/bin/encapic`.
- Reworks the `setup-shared-bins` init container to `curl` the encapic `v0.1.0` `encapic_linux_amd64` release binary (sha256-verified against the published `.sha256`, installed mode 0755) into the shared bins dir, instead of copying the python script and installing uv.
- Removes the `puppet-encapi-enc` configmap generator, its volume and mount, and the `resources/encapi-enc` script. uv was consumed solely by that script (grep of `apps/base/puppet` confirms no other consumer), so its installation is removed too.
`kubectl kustomize apps/overlays/au-syd1/puppet` builds clean.
## Merge gate
Do not merge until the encapic `v0.1.0` release assets exist:
`https://git.unkin.net/unkin/encapic/releases/download/v0.1.0/encapic_linux_amd64` (+ `.sha256`). The init container pulls them at pod start.
## Rollback
Revert this PR to restore the `encapi-enc` configmap script + uv install and repoint `external_nodes`.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #277
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The k8s puppetserver compilers classify nodes via an exec ENC that today queries legacy Cobbler (`https://cobbler.main.unkin.net`) over TLS. `encapi` now runs in-cluster and exposes a cobbler-wire-compatible endpoint (`GET /cblr/svc/op/puppet/hostname/<certname>`), a drop-in for the Cobbler URL. This cuts the puppet-on-k8s ENC over from Cobbler to encapi — a prerequisite for migrating VM agents onto puppet-on-k8s.
## Changes
- Rename the ENC script `resources/cobbler-enc` -> `resources/encapi-enc`, and its configmap `puppet-cobbler-enc` -> `puppet-encapi-enc` (kustomization configMapGenerator + deployment volume, initContainer copy path, and volumeMount subPath).
- Point `external_nodes` in the compiler `puppet.conf` at `/opt/bin/encapi-enc`.
- Target the in-cluster encapi service `http://encapi.encapi.svc.cluster.local` (plain HTTP), overridable via the `ENCAPI_URL` env var.
- Drop the `/opt/vault-ca-cert.crt` verify for the ENC request (no TLS needed in-cluster).
- Leave the response normalization identical: classes coerced to a list, `enc_role`/`enc_env` params set, `environment` stripped when it equals `testing`.
Verified with `kubectl kustomize apps/overlays/au-syd1/puppet` (builds clean, exit 0); the generated `puppet-encapi-enc` configmap contains the new URL and env var.
## 🚨 Merge gate
**Do not merge until encapi is seeded** (terraform-incus `benvin/encapi-seed` PR applied). An empty encapi means every node resolves to a 404. On 404 the ENC script exits non-zero, so puppet fails the compile rather than classifying the node with zero classes — nodes will fail to run until they exist in encapi. Seed encapi first so real nodes classify correctly; only unknown nodes should 404.
Reviewed-on: #272
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move
between them as they migrate. `node-lookup` (and `pblastreport`) need a single,
consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy
that provides exactly that. Per the all-in-kubernetes estate direction it runs
as an in-cluster service, not a per-VM systemd unit.
pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) —
split out of the earlier node-lookup prototype — and is released as a container
image on its own `v*` tags.
## Changes
- Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway,
httproute), modeled directly on the encapi app.
- Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080,
`/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap.
- Backends: `old=http://puppetdbapi.service.consul:8080`,
`new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified
against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the
in-cluster address is preferred over the external gateway). `new` is
primary/prefer, merge = freshness.
- Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal`
Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a
port-80 Service — same shape as the puppetdb/encapi gateways — so
VM/workstation `node-lookup` can reach it.
- Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform
ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject
(`pdbmux` namespace destination), exactly as encapi is wired.
No new woodpecker ServiceAccount is required: the pdbmux image push uses the
`docker-buildx` plugin against the Gitea registry with the `default` SA (same as
encapi), not artifactapi.
## Verification
- `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves
to `git.unkin.net/unkin/pdbmux:v0.1.0`).
- ApplicationSet + AppProject YAML validated.
## Merge gates
1. The pdbmux repo initial-content PR
(unkin/pdbmux#1) must merge first.
2. `v0.1.0` must then be tagged on the pdbmux repo so the image
`git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s
`.woodpecker/docker.yaml`.
3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update
the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.)
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #275
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator v0.2.0 talks to radosgw over HTTPS (`radosgw.service.consul:443`, fronted by nginx presenting the internal `unkin.net` Vault-PKI cert). With no CA configured the operator fails:
```
Get "https://radosgw.service.consul:443/admin/user?...": tls: failed to verify certificate: x509: certificate signed by unknown authority
```
The `vault-ca-cert` Secret (the `unkin.net` intermediate+root) is already reflected into every namespace — including `cephrgw-system` — so the fix is deployment-only.
## Changes
- mount the `vault-ca-cert` Secret (key `ca.crt`) read-only at `/etc/vault-ca/ca.crt`, following the puppet/artifactapi pattern
- set `CEPH_RGW_CA_FILE=/etc/vault-ca/ca.crt` so the operator adds the CA to its TLS trust
No image change (still `v0.2.0`); `reloader` + a normal reconcile pick it up. Validated with `kustomize build` on the au-syd1 overlay.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #276
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The new **terragrunt-enc** repo (single source of truth for encapi ENC data) runs its Terraform apply/plan in Woodpecker and authenticates to Vault via kubernetes auth. The Vault k8s role `woodpecker_terraform_enc` (terraform-vault PR #98) binds to a ServiceAccount named `terraform-enc` in the `woodpecker` namespace, which must exist for that auth to work.
Changes:
- Add `apps/base/woodpecker/serviceaccount_terraform_enc.yaml` (SA `terraform-enc` in namespace `woodpecker`).
- Register it in the woodpecker kustomization.
Reviewed-on: #274
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator **v0.2.0** rebuilds the Ceph integration to talk directly to radosgw via **go-ceph** (Admin Ops API) + **aws-sdk-go-v2** (S3), replacing the manager-dashboard client, and adds **fine-grained bucket-access policies** (paths / actions / conditions / rawStatements). The operator now authenticates with an **RGW admin user's access/secret key** instead of a dashboard login.
Operator repo PRs: unkin/cephrgw-operator #3 (rebuild) and #4 (fine-grained), both merged; tag `v0.2.0`.
## Changes
- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.2.0`
- update the `envFrom` / `VaultStaticSecret` comments to the `CEPH_RGW_*` credential keys the new image consumes
## Required manual step (runtime)
The VaultStaticSecret copies the KV secret's keys **verbatim**, so the seed must be re-put with the new keys before/with rollout — otherwise the operator fails auth:
```
vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
CEPH_RGW_ACCESS_KEY=<key> CEPH_RGW_SECRET_KEY=<secret>
```
(The old `CEPH_DASHBOARD_*` keys are ignored by v0.2.0.) VSO refreshes within 5m and the `reloader` annotation restarts the operator.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #273
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Kubernetes nodes querying the bind-resolvers LoadBalancer VIP (198.18.200.7) get REFUSED (EDE 18 Prohibited).
The service is `externalTrafficPolicy: Local`, which preserves the client source IP for traffic entering the cluster from outside — but a node querying the VIP never leaves via OSPF. Its own kube-proxy DNATs the LB IP in the OUTPUT chain and masquerades the source to a cluster-internal address (the node's flannel.1, e.g. 10.42.x.x). That address is not in `acl-main.unkin.net`, so the openforwarder view's match-clients rejects the query.
External clients preserve their real source IP and match acl-main, which is why only in-cluster hosts were affected.
Add `10.42.0.0/16` to `acl-main.unkin.net` so node-originated (masqueraded) resolver queries are permitted. This mirrors the authoritative cluster, which already allows the pod network (`allow-query { ...; 10.42.0.0/16; }`).
Reviewed-on: #271
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Picks up immediate NOTIFY of secondaries on primary zone changes (also-notify
+ shorter seed SOA timers), so dynamic updates / CRD records replicate across
the authoritative replicas in seconds instead of waiting up to the SOA refresh.
Bumps both the operator image and the CRD install.yaml tag.
The bind-resolvers `openforwarder` view forwarded `k8s.syd1.au.unkin.net` to the in-cluster bind-externaldns (198.18.200.8), which is not reliably serving those records yet, so lookups return NXDOMAIN.
Concrete impact: Gitea cannot resolve the k8s-hosted CI host, so its outbound webhook fails and tagged releases never trigger CI (e.g. cutting a new bind-operator release from a tag).
This points the `fwd-k8s-syd1-au-unkin-net` forwarder at the existing external external-dns bind service anycast **198.18.19.20** (puppet `roles::infra::dns::externaldns` — master `ausyd1nxvm2127` + slaves `2128`/`2129`, advertised via OSPF), which still holds the working `k8s.syd1.au.unkin.net` records. It is in the same 198.18.19.0/24 anycast family as the consul forwarder (198.18.19.14) the resolvers already use, so it is reachable from the pods.
Temporary measure. Revert to 198.18.200.8 once external-dns publishes to the in-cluster bind-externaldns service. Only the forward target changes; no CRD/schema changes.
Reviewed-on: #269
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Pairs with terraform-authentik#10: request the litellm_role scope (Authentik
emits the computed role claim) and read it via GENERIC_USER_ROLE_ATTRIBUTE so
akP-litellm-admin -> proxy_admin, akP-litellm-user -> internal_user.
App-side of the LiteLLM Authentik onboarding (terraform-authentik#8). Configures
LiteLLM's generic OIDC SSO against Authentik.
- VaultStaticSecret oauth-credentials: surfaces the OIDC client secret (same
secret Authentik sets on the provider) as a k8s Secret.
- Deployment: GENERIC_CLIENT_SECRET from that Secret.
- litellm-env: GENERIC_CLIENT_ID, authorization/token/userinfo endpoints, scope,
and PROXY_BASE_URL (required for SSO). reloader restarts on secret/config change.
## Why
Every bind-operator dynamic update is refused (`update ... denied due to allow-query`) because the operator execs `nsupdate` against `127.0.0.1` inside the primary pod, and the BindCluster `allow-query` listed only the client subnets (`auth-acl-main`) and the pod net (`10.42.0.0/16`) — not loopback. This blocked ALL DNSRecords (identity, s3, dashboard, lb1) from ever applying.
## Change
- Add `localhost` to the BindCluster `allow-query` in `apps/base/bind-internal/authoritative/cluster.yaml`. The `client-update` TSIG key still gates the actual update.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #267
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The cephrgw-operator (in-cluster) fails to reach the Ceph dashboard because CoreDNS/bind-internal has no record for `dashboard.ceph.unkin.net` (`no such host`). Publish it authoritatively so in-cluster clients can resolve it.
## Changes (apps/base/bind-internal/authoritative/records.yaml)
- `DNSRecord dashboard-ceph-cname`: CNAME `dashboard.ceph.unkin.net` -> `lb1.unkin.net.` (zone `ceph.unkin.net`, zoneRef `ceph-unkin-net`).
- `DNSRecord lb1-unkin-net`: A `lb1.unkin.net` -> `103.216.191.185` (zone `unkin.net`, zoneRef `unkin-net`).
Once applied, the operator's `sandbox-user`/`sandbox-bucket` (currently Error/Pending on DNS) will reconcile to Ready.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #266
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to the cephrgw-operator deploy (#261): source the operator's Ceph dashboard credentials from Vault via VSO instead of a hand-created Secret.
## Changes
- Add `apps/base/cephrgw-system/vaultauth.yaml`: `VaultAuth` (mount `k8s/au/syd1`, role `cephrgw-operator`, SA `cephrgw-operator`, `vaultConnectionRef: vso-system/default`).
- Add `apps/base/cephrgw-system/vaultstaticsecret.yaml`: renders KV `service/cephrgw/dashboard-credentials` into the `cephrgw-credentials` Secret (keys copied verbatim → consumed by the Deployment via `envFrom`; the reloader annotation restarts the operator on rotation).
- Reference both from the base kustomization.
## Dependencies / ordering
- Requires the Vault role + policy from **terraform-vault #95** (merge/apply first), and the KV values to be seeded out-of-band:
```
vault kv put kv/service/cephrgw/dashboard-credentials \
CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
CEPH_DASHBOARD_PASSWORD=... CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net
```
- Until VSO auth succeeds the `cephrgw-credentials` Secret won't exist and the operator pod stays in `CreateContainerConfigError` (expected).
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #262
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Publish the RGW S3 endpoint name (`s3.ceph.unkin.net`) that cephrgw-operator consumers use and that the radosgw hosts will carry as a cert SAN. For now it points at the Consul service; the real target will be changed later.
## Changes
- Add a `DNSRecord` in the `ceph.unkin.net` authoritative zone: `s3` CNAME `radosgw.service.consul.` (`apps/base/bind-internal/authoritative/records.yaml`, zoneRef `ceph-unkin-net`, TTL 600).
A companion puppet-prod change adds `s3.ceph.unkin.net` to the radosgw cert SANs and nginx server names.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #265
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Consume the two-tier Authentik RBAC from terraform-authentik#7. Grafana should grant Admin to the `akP-grafana-admin` permission group, which `akR-global-admin` members inherit.
## Change
- **grafana.yaml** (`auth.generic_oauth`): add `ak_groups` to `scopes`; `role_attribute_path` now keys off `ak_groups` and `akP-grafana-admin` (replaces the flat `grafana-admins`). Non-admins who can log in (gated to `akP-grafana-*` by the Authentik access policy) get Viewer; `role_attribute_strict: false` retained.
## Depends on
terraform-authentik#7 (creates `akP-grafana-admin`, the access binding, and the `ak_groups` mapping).
## Validation
`kustomize build` (base + overlay) renders; pre-commit clean.
Reviewed-on: #264
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Consume the two-tier Authentik RBAC from terraform-authentik#7 (user → role → permissions). ArgoCD should grant admin to the `akP-argocd-admin` permission group, which `akR-global-admin` members inherit.
## Change
- **argocd-cm**: request the hierarchical `ak_groups` scope + id-token claim (carries permission groups inherited via role groups; distinct from the default `groups` claim to avoid collision).
- **argocd-rbac-cm**: `scopes: [ak_groups]`; `policy.csv`: `g, akP-argocd-admin, role:admin` (replaces the flat `argocd-admins`). Default stays `role:readonly`.
## Depends on
terraform-authentik#7 (creates `akP-argocd-admin`, the access binding, and the `ak_groups` mapping). Merge/apply that first; then add yourself to `akR-global-admin` in Authentik.
## Validation
`kustomize build` renders the patched configmaps; pre-commit clean. Note: argocd-server picks up argocd-cm/rbac-cm live.
Reviewed-on: #263
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new `cephrgw-operator` provisions Ceph RGW (S3) buckets and access keys (RW/RO) from Kubernetes CRDs via the Ceph manager dashboard API. This deploys it as a platform app.
## Changes
- Add `apps/base/cephrgw-system`: namespace, ServiceAccount + ClusterRole/Binding (manage `ceph.unkin.net` CRDs, Secrets, leader-election leases), and the operator Deployment. CRDs are pulled from the operator repo at tag `v0.1.0`; the Deployment sources dashboard credentials from the `cephrgw-credentials` Secret via `envFrom` and carries the reloader annotation.
- Add `apps/overlays/au-syd1/cephrgw-system` referencing the base.
- Register `apps/overlays/*/cephrgw-system` in the platform ApplicationSet.
The platform AppProject already permits `*-system` namespaces and the Namespace/ClusterRole/CRD cluster resources, so no project change is needed.
## Ordering / dependencies
- Depends on the Gitea repo from terraform-git #34 and on the operator being pushed + tagged **v0.1.0** (image `git.unkin.net/unkin/cephrgw-operator:v0.1.0` and the raw CRD `install.yaml` at that tag). The `kubeconform` check will stay red until v0.1.0 exists, then go green.
- The `cephrgw-credentials` Secret must be created out-of-band in `cephrgw-system` (see the operator's `docs/ceph-setup.md`); it is intentionally **not** managed in GitOps.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #261
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The ceph/halb host (`ausyd1nxvm2069`) publishes `dashboard.ceph.unkin.net` via nsupdate to a dedicated `zone ceph.unkin.net.`, which `bind-authoritative` was not authoritative for (NOTZONE). This adds the zone so that record has a home.
## Changes
- Add `ceph.unkin.net` BindZone (primary, dynamicUpdate, updateKeyRef client-update) to bind-authoritative, matching the unkin.net/main.unkin.net pattern.
## Note — not the root cause of the 6 missing hosts
Log evidence (VictoriaLogs, dns-update-apply on 2069/2070) shows the actual failure is a **host-side bug in the puppet `dns-update` script**: `fqdn()` appends the zone even to records whose name is already fully-qualified (e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`), producing a `..` empty label → `invalid owner name: empty label` → the whole main.unkin.net update `send` fails (reverse PTR, sent first, still lands). That script fix (puppet-prod) is the real blocker; this zone is still needed so the ceph record does not hit NOTZONE afterward.
Reviewed-on: #260
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to #258 (which added workstation+router to the *authoritative* ACL). The **resolver** (bind-resolvers, 198.18.200.7) has its own `acl-main.unkin.net` gating its `openforwarder` view; the workstation is not in it, so recursive queries return REFUSED. This lets the workstation use the resolver as its normal nameserver.
## Changes
- Add `10.10.12.200/32` (workstation, wireguard) to resolver `acl-main.unkin.net`
- Add `198.18.21.160/32` (router) explicitly for documentation; already covered by existing `198.18.21.160/27` and `198.18.21.0/24` (no functional change for the router)
Reviewed-on: #259
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Allow the operator's workstation and router to query the bind-authoritative servers directly. Their source addresses are outside the existing `auth-acl-main` client subnets, so named returns REFUSED to them today. The router sits on 198.18.21.0/24 which is not in the ACL at all.
## Changes
- Add `10.10.12.200/32` (workstation, over wireguard) to `auth-acl-main`
- Add `198.18.21.160/32` (router) to `auth-acl-main`
## Note
This grants query permission only. Reaching the LoadBalancer VIP (198.18.200.6) from off-datacenter paths is separately gated by `externalTrafficPolicy: Local`; the workstation-over-wireguard path still needs its L4 routing addressed to actually land on a node with a ready endpoint.
Reviewed-on: #258
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new terraform-rancher CI pipeline runs as a pod in the woodpecker namespace; its ServiceAccount token is what Vault binds to the `woodpecker_terraform_rancher` k8s auth role (see terraform-vault#86) for rancher2 provider auth + Consul state.
## Change
- Add `ServiceAccount/terraform-rancher` (woodpecker ns) and wire it into the woodpecker kustomization, mirroring the other terraform-* runner SAs.
## Validation
`kustomize build apps/base/woodpecker` renders it; pre-commit clean.
Reviewed-on: #257
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`identity.unkin.net` is configured as the Authentik OIDC issuer but has **no DNS record anywhere**, so in-cluster OIDC discovery fails (e.g. `argocd-server` → `lookup identity.unkin.net ... no such host`). Add an authoritative A record served by the internal bind system (bind-operator) so it resolves.
## Change
- New `DNSRecord/identity-dns-internal` → `198.18.200.4` (the traefik-internal gateway VIP, where the Authentik Gateway serves the `identity.unkin.net` hostname), in the `unkin-net` zone.
- Lives in the **`bind-internal` namespace** alongside the `BindZone`: the operator resolves `zoneRef`/`clusterRef`/`updateKeyRef` within the record's own namespace, so it can't live in the app (authentik) namespace.
- Wired into `apps/base/bind-internal/authoritative/kustomization.yaml`.
- `identity-dns-internal` name distinguishes this from the external DNS that Authentik will manage its own records from later.
## Validation
`kustomize build apps/base/bind-internal` + kubeconform (validates against the `dnsrecord_v1alpha1` schema): 57 valid, 0 invalid. pre-commit clean.
Reviewed-on: #256
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2's config-hash rolling restart exposed a latent non-determinism: `client.List` returns cache-ordered results, so the resolver's forward zones reshuffled every reconcile, flipping the config hash and rolling `bind-resolvers-2` endlessly. v0.2.3 (bind-operator #11) sorts every rendered list so `named.conf` is byte-identical across reconciles and the hash is stable.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.3`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.3`.
CRDs are unchanged from v0.2.2 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.3 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.
## Deploy note
On deploy the deterministic operator stops churning the ConfigMap; the config hash stabilizes and the stuck resolver rolling update completes, leaving all three pods Ready on one revision.
Reviewed-on: #255
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
ArgoCD had no external ingress and only local admin auth. This exposes `argocd-server` behind the traefik-internal gateway and enables Authentik SSO, so operators log in with their Authentik identity and group membership. Pairs with unkin/terraform-authentik#3 (creates the OAuth2 provider).
## Changes
- **argocd-cm**: set `url` and `oidc.config` (Authentik issuer `identity.unkin.net/application/o/argocd/`, `argocd` client, openid/profile/email scopes). Client secret resolved from the `argocd-oidc` Secret via `$argocd-oidc:client_secret`.
- **argocd-rbac-cm**: match RBAC on the `groups` claim; default `role:readonly`; map the `argocd-admins` Authentik group to `role:admin`.
- **argocd-cmd-params-cm**: `server.insecure=true` so `argocd-server` serves HTTP behind the TLS-terminating gateway.
- Add **Gateway + HTTPRoutes** for `argocd.k8s.syd1.au.unkin.net` (mirrors the grafana pattern: traefik-internal, vault-issuer cert, external-dns).
- Add **VaultAuth + VaultStaticSecret** sourcing the OIDC client secret from `kv/kubernetes/namespace/argocd/default/oauth-credentials` into the `argocd-oidc` Secret (labelled `part-of=argocd` so ArgoCD will resolve the `$` reference).
## Notes / rollout
- Seed the client secret in Vault out of band (same path terraform-authentik reads).
- The argocd namespace `default` SA already has Vault read access via the `default` k8s role, so no terraform-vault change is needed.
- `argocd-server` needs a one-time rollout restart to pick up `server.insecure`.
Validated with `kustomize build --enable-helm clusters/au-syd1/bootstrap`, `make kubeconform`, and pre-commit (yamllint + no-plain-secrets).
Reviewed-on: #253
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2 (bind-operator #10) stamps a `bind.unkin.net/config-hash` on the pod template, so a ConfigMap or `keys.conf` change flips the hash and triggers an operator-driven rolling restart. This fixes the class of bug where config edits (ACLs, forwarders, `validate-except`, primary address, TSIG rotation) never reached running pods — they held a startup snapshot and needed manual pod deletes.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.2`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.2`.
CRDs are unchanged from v0.2.1 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.2 CRD bundle; pre-commit clean.
## Deploy note
When the v0.2.2 operator first reconciles it stamps the config-hash annotation, triggering **one rolling restart per bind StatefulSet** — expected, and it also pulls in any already-pending config. From then on, config changes roll pods automatically.
Reviewed-on: #254
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why — urgent
PR #251 added `validate-except` to the resolver `BindCluster` but the list omitted the trailing semicolon after the final entry. `named` requires **every** entry in a list to be semicolon-terminated, including the last before the closing brace, so it fails config parse and the resolver pods crash-loop:
```
/run/named/named.conf:18: missing ';' before '}'
loading configuration: failure
exiting (due to fatal error)
```
The resolvers (`.7`) are down until this lands; the authoritative (`.6`/`.9`) and externaldns (`.8`) are unaffected.
## Fix
```diff
- validate-except { unkin.net; 18.198.in-addr.arpa; consul }
+ validate-except { unkin.net; 18.198.in-addr.arpa; consul; }
```
Renders to `validate-except { unkin.net; 18.198.in-addr.arpa; consul; };` — valid.
## Recovery
On merge + ArgoCD sync, the operator re-renders the ConfigMap with valid config and the crash-looping pods self-heal on their next restart (no manual `rollout restart` needed). Validated: `bind-internal` renders and pre-commit clean.
Reviewed-on: #252
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Resolving any `unkin.net` record through the resolver (`.7`) returns **SERVFAIL**, while the authoritative (`.6`) answers fine. Confirmed from the resolver's querylog:
```
view openforwarder: validating unkin.net/SOA: got insecure response; parent indicates it should be secure
broken trust chain resolving 'ausyd1nxvm2120.main.unkin.net/A/IN': 198.18.200.6#53
query failed (broken trust chain)
```
The resolver runs `dnssec-validation auto`. The public `unkin.net` is DNSSEC-signed (the `.net` parent publishes a DS), but the in-cluster split-horizon authoritative serves `unkin.net` **unsigned**. The validator sees "parent says secure" + an insecure answer → treats it as spoofing → SERVFAIL. The authoritative works directly because it does no validation.
## Fix
Add `validate-except` (via `spec.extraOptions`) for the forwarded internal domains, so the resolver treats them as insecure and skips validation:
```
validate-except { unkin.net; 18.198.in-addr.arpa; consul }
```
- `unkin.net` covers all `*.unkin.net` (incl. `main.unkin.net`, `k8s.syd1.au.unkin.net`)
- `18.198.in-addr.arpa` covers every `NN.18.198.in-addr.arpa` reverse zone (subtree)
- `consul` covers the consul TLD
This also makes internal resolution independent of Internet egress (no DNSSEC chain-walk needed). External-name validation is unchanged. No operator change required.
## Validation
`bind-internal` renders and passes `kubeconform` (56/56); pre-commit clean.
## Activation
After merge + operator reconcile, the resolver ConfigMap re-renders; the running pods hold a startup snapshot, so they need a reload: `kubectl -n bind-internal rollout restart statefulset/bind-resolvers`.
Reviewed-on: #251
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.1 fixes authoritative **secondary replication**, which never actually worked — the master REFUSED the catalog AXFR. Root causes (bind-operator #9): secondaries presented no TSIG key, member zones had no `allow-transfer`, and secondaries pointed at the primary's ephemeral pod IP.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.1`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.1`.
CRDs are unchanged from v0.2.0, so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.1 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.
## Deploy note
Existing member zones pick up `allow-transfer` via `modzone`, and secondaries re-point at the stable primary Service ClusterIP with the transfer key, restoring replication without manual BIND surgery. A from-scratch namespace recreate also comes up clean (OrderedReady startup means secondaries snapshot a ClusterIP-correct config). Caveat for a full recreate: the operator regenerates the `client-update` TSIG key, so its new material must be re-synced into puppet eyaml before clients can nsupdate.
Reviewed-on: #250
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
bind-operator v0.2.0 adds the `BindTSIGAPI` CRD and a companion API that `vault-plugin-secrets-bind-tsig` calls to create, rotate and delete TSIG keys (it does so by managing `BindTSIGKey` resources, which the operator reconciles into key material). This rolls the operator forward and deploys an API instance so Vault never talks to the Kubernetes API directly.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.0`.
- Broaden the operator ClusterRole (`bind-system/rbac.yaml`) with `deployments`, `serviceaccounts` and `roles`/`rolebindings`, so the `BindTSIGAPI` reconciler can create the API Deployment and its namespaced Role/RoleBinding.
- Add a `BindTSIGAPI` (`bind-tsig-api`) in `bind-internal`; the operator reconciles it into a Deployment, Service, ConfigMap, master-token Secret and RBAC. Keys are created in `bind-internal`, alongside the authoritative cluster and its existing keys.
- Add the generated kubeconform schema for `BindTSIGAPI`.
## Notes
- The master access token Secret (`bind-tsig-api-token`) is generated by the operator when absent; the operator does not own it, so a `VaultStaticSecret` can later pre-seed/overwrite it to source the token from Vault.
- Validated: both overlays render (`kubectl kustomize`) and pass `kubeconform` (bind-internal 56/56 valid); pre-commit clean.
## Follow-up
- Point `vault-plugin-secrets-bind-tsig` config at `http://bind-tsig-api.bind-internal.svc:8443`.
Reviewed-on: #249
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Enables per-host RFC2136 updates from puppet (puppet-prod #475 profiles::dns::updater) to the bind-authoritative zones, via the .9 write endpoint.
## Changes
- add **client-update** BindTSIGKey (clusterRef bind-authoritative; operator generates the material into Secret client-update-tsig)
- set `dynamicUpdate: true` + `updateKeyRef: client-update` on all **18** authoritative zones → the operator renders `allow-update { key "client-update"; }`
## Key bridge (manual, per the TSIG plan)
The operator generates the client-update key value; it must reach puppet eyaml (`profiles::dns::updater::key_secret`) for clients to authenticate — until the planned Vault-sync/secret-reflection operator features exist. Get it with:
`kubectl -n bind-internal get secret client-update-tsig -o jsonpath='{.data.secret}' | base64 -d`
## Validated
kustomize build + kubeconform.
Reviewed-on: #244
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #239, pairing with the CNPG VMPodScrape (#242). Imports the **CloudNativePG** dashboard (grafana.com 20417) as a `GrafanaDashboard` (gzipJson, datasources resolved to the in-cluster VictoriaMetrics uid). Now that #242 collects the postgres metrics, this dashboard renders real data for all CNPG clusters.
Reviewed-on: #243
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #239. Every CNPG instance pod already exposes Prometheus metrics on `:9187` (`metrics` port), but nothing scraped them. Adds a single namespace-wide `VMPodScrape` (`namespaceSelector.any`, `selector cnpg.io/podRole=instance`) so the observability VMAgent collects postgres metrics for **all ~23 CNPG clusters** across the estate (authentik, grafana, woodpecker, artifactapi, puppet, litellm, …). No chart changes. Pairs well with the CNPG grafana.com dashboard (20417) as a further follow-up.
Reviewed-on: #242
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #239. Adds a `VMServiceScrape` for cert-manager's existing webhook (`metrics` :9402) and cainjector (`http-metrics` :9402) services so the observability VMAgent collects them. No chart change needed. (The controller's own metrics need `prometheus.enabled` in the chart to expose a metrics service — separate follow-up.)
Reviewed-on: #241
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Consul SD job discovered **0 targets**. vmagent logs showed it reaching the puppet Consul fine (TLS/connectivity OK) but getting `403` on `GET /v1/agent/self`: the anonymous token `lacks permission 'agent:read'`. VictoriaMetrics calls `/v1/agent/self` only to auto-detect the datacenter; catalog/health reads (what SD actually needs) work anonymously.
## Fix
Set `datacenter: au-syd1` on the consul_sd_config so VM skips the `agent/self` call. No consul token needed.
## Verify after sync
vmagent `/targets` → `consul` job shows the puppet targets (haproxy/ceph/gitea/node/…) up.
Reviewed-on: #240
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Deploys Grafana in-cluster (observability project) via the grafana-operator, mirroring the puppet Grafana but modernised — **CNPG** for state, **Authentik OIDC** for auth — and ports the live datasource + dashboards in as CRs.
Depends on: grafana-operator (#235, merged), grafana schemas (#236, merged), Authentik OIDC (terraform-authentik #2), Vault seeds (done), and `^grafana/` image proxy (terraform-artifactapi #5).
## Changes (`apps/base/grafana`)
- **CNPG** postgres Cluster + rw Pooler (db `grafana`); **VaultAuth** + **VaultStaticSecrets** pulling `postgres`/`oauth` credentials from `kv/kubernetes/namespace/grafana/default/*`.
- **Grafana CR**: postgres backend via the pooler; Authentik `generic_oauth` (client id/secret from the Vault-synced secret, openid/email/profile scopes, group→role mapping); `root_url` grafana.k8s.syd1.au.unkin.net.
- **1 GrafanaDatasource** — k8s VictoriaMetrics via the operator `vmselect-main` service; reuses the previous default datasource uid so the imported dashboards resolve unedited.
- **13 GrafanaDashboards** (gzipJson) exported from the current grafana.
- **Gateway API** (traefik-internal) + HTTPRoute for grafana.k8s.syd1.au.unkin.net.
- Registered in the observability ApplicationSet + project.
## Review notes
- OAuth `role_attribute_path` maps Authentik group `grafana-admins` → Admin, else Viewer — **confirm the group name**.
- `database.ssl_mode: require` against the CNPG pooler — adjust if the pooler isn't serving TLS.
- The `VictoriaLogs - cluster` dashboard has no in-cluster logs datasource yet (no VictoriaLogs in k8s) — included for completeness, will be empty until one exists.
- `make kubeconform` clean (24 resources, validated against the strict grafana schemas).
Reviewed-on: #238
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-06 23:54:51 +10:00
259 changed files with 12412 additions and 260 deletions
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Ben Vincent
unkinben
benvin