Why: shrink the blast radius of the Puppet control-plane pods (CA/eyaml keys, compiled catalogs) per the security sweep in #307 — remove root where it is not required and strip cargo-culted capabilities.
How:
- puppetboard cert-generator init: root+APE:true -> uid 1000, drop:[all], APE:false; pod fsGroup 1000; removed trailing `chown -R 1000:1000` (PVC now group-owned).
- puppetdb create-log-dir init: root -> uid 999, drop:[all], APE:false; pod fsGroup 999; removed `chown 999:999`.
- All OpenVox capability add-lists: removed the duplicate CAP_-prefixed spellings (k8s normalises both to the same kernel cap) and dropped the unused AUDIT_WRITE.
- Added allowPrivilegeEscalation:false and seccompProfile RuntimeDefault across the workloads.
Stays root (evidence-backed, class-B fallback): the puppetserver master/compiler and puppetdb main containers, plus the perms-and-dirs and generate-types root containers. The OpenVox image entrypoint runs `chown -R puppet:puppet` over root-owned baked-in dirs and drops the JVM to the puppet user via `runuser` (needs CHOWN/SETUID/SETGID); a non-root start crashloops. Their cap sets are reduced to the minimum justified (CHOWN/DAC_OVERRIDE/FOWNER[/SETUID/SETGID]).
Validation: `kustomize build --enable-helm` clean; kubeconform 0 invalid / 0 errors; pre-commit (yamllint etc.) green. Confirmed against live pods: puppetserver/puppetdb JVMs already run as puppet/puppetdb via `runuser`; `pam_loginuid` is absent from the su/runuser PAM stacks and loginuid is unset, so dropping AUDIT_WRITE is safe.
Post-merge smoke test (puppet had an outage this session — watch closely): after argocd sync, confirm puppetserver master + a compiler reach `running` at /status/v1/simple, puppetdb reaches `running`, puppetboard serves 200, and the generate-types + g10k CronJobs complete — i.e. catalogs still compile and reports still ingest.
Closes#307https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #319
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret.
## Changes
- Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched.
- Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads.
- Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`.
- Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf.
- Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`).
## Safety review (secret-only / CA workloads)
`vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded.
- `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated.
- `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`).
- `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only.
- `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added.
CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`).
## Notes / coordination
- Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression).
- Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here.
## Validation
- `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains.
- `uvx pre-commit run --all-files` — all hooks pass.
Closes#326
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #339
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- Update PuppetDB connections from HTTP (8080) to HTTPS (8081)
- Add automatic certificate generation for Puppetboard using Puppet CA
- Implement initContainers for proper certificate provisioning before app start
- Add dedicated PVC for Puppetboard certificates with RWX access
- Configure SSL verification and client authentication for secure PuppetDB access
Reviewed-on: #50
puppetdb_port has tcp:// in it, even though we pass the correct variable
in from a configmap.
```
ben@metabox ~/s/p/argocd-apps> kubectl --context admin run debug-pod --image=busybox --rm -it --restart=Never -n puppet -- env | grep -i puppetdb_port
PUPPETDB_PORT_8081_TCP_PORT=8081
PUPPETDB_PORT_8081_TCP_PROTO=tcp
PUPPETDB_PORT=tcp://10.43.101.142:8080
PUPPETDB_PORT_8080_TCP=tcp://10.43.101.142:8080
PUPPETDB_PORT_8080_TCP_ADDR=10.43.101.142
PUPPETDB_PORT_8081_TCP=tcp://10.43.101.142:8081
PUPPETDB_PORT_8080_TCP_PROTO=tcp
PUPPETDB_PORT_8081_TCP_ADDR=10.43.101.142
PUPPETDB_PORT_8080_TCP_PORT=8080
```
Reviewed-on: #32
complete the implementation of puppet in kubernetes, taking many
features from the openvox helm chart and improving on them. changes from
helm are:
- using vault for storing secrets
- using g10k instead of r10k
- using a single shared g10k cronjob for all masters/compilers
- using a single shared /etc/puppetlabs/code directory (shared, cephfs)
changes:
- deploy puppet master and compiler servers with statefulset/deployment
- deploy puppetdb with postgresql backend, taking advantage of cnpg cluster and pooler
- deploy puppetboard
- all supporting configmaps, services, ingresses, and hpas
- added vaultstaticsecret for eyaml private keys
- configured secure mounting of eyaml keys at /var/lib/puppet/keys/
- updated base kustomization to include all 23 new puppet resource files
Reviewed-on: #29
unkinben
benvin