## Why
ArgoCD fails to sync the puppet app with:
Deployment.apps "puppetserver-master" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is 'Recreate'
The manifest is already correct: #341 changed the master to `spec.strategy.type: Recreate` with no rollingUpdate block. The failure is a live-object artifact. When the master ran RollingUpdate the API server defaulted `spec.strategy.rollingUpdate` (maxSurge/maxUnavailable) onto the object. That defaulted field is owned by no applier, so neither a client-side merge nor server-side apply drops it when the desired manifest omits it. The live object therefore keeps `rollingUpdate` while gaining `type: Recreate`, which the API server rejects — blocking every sync.
k8s forbids any `rollingUpdate` field when `strategy.type` is `Recreate`; the two are mutually exclusive, so the sync cannot converge until the stale field is removed from the live object.
## Changes
- Annotate the `puppetserver-master` Deployment with `argocd.argoproj.io/sync-options: Replace=true`. Replace performs a full PUT that overwrites the whole object, dropping the stale `rollingUpdate` field and letting the Recreate strategy apply cleanly. The annotation is scoped to this one resource, so puppetdb/puppetboard/compiler keep the app-wide ServerSideApply behaviour.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/puppet` renders `puppetserver-master` with `strategy: { type: Recreate }` (no rollingUpdate) and the new sync-options annotation.
- `make kubeconform`: puppet overlay 34/34 valid; only the known cattle-system rancher kubeVersion incompatibility fails.
- `pre-commit`: all hooks pass.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #349
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: shrink the blast radius of the Puppet control-plane pods (CA/eyaml keys, compiled catalogs) per the security sweep in #307 — remove root where it is not required and strip cargo-culted capabilities.
How:
- puppetboard cert-generator init: root+APE:true -> uid 1000, drop:[all], APE:false; pod fsGroup 1000; removed trailing `chown -R 1000:1000` (PVC now group-owned).
- puppetdb create-log-dir init: root -> uid 999, drop:[all], APE:false; pod fsGroup 999; removed `chown 999:999`.
- All OpenVox capability add-lists: removed the duplicate CAP_-prefixed spellings (k8s normalises both to the same kernel cap) and dropped the unused AUDIT_WRITE.
- Added allowPrivilegeEscalation:false and seccompProfile RuntimeDefault across the workloads.
Stays root (evidence-backed, class-B fallback): the puppetserver master/compiler and puppetdb main containers, plus the perms-and-dirs and generate-types root containers. The OpenVox image entrypoint runs `chown -R puppet:puppet` over root-owned baked-in dirs and drops the JVM to the puppet user via `runuser` (needs CHOWN/SETUID/SETGID); a non-root start crashloops. Their cap sets are reduced to the minimum justified (CHOWN/DAC_OVERRIDE/FOWNER[/SETUID/SETGID]).
Validation: `kustomize build --enable-helm` clean; kubeconform 0 invalid / 0 errors; pre-commit (yamllint etc.) green. Confirmed against live pods: puppetserver/puppetdb JVMs already run as puppet/puppetdb via `runuser`; `pam_loginuid` is absent from the su/runuser PAM stacks and loginuid is unset, so dropping AUDIT_WRITE is safe.
Post-merge smoke test (puppet had an outage this session — watch closely): after argocd sync, confirm puppetserver master + a compiler reach `running` at /status/v1/simple, puppetdb reaches `running`, puppetboard serves 200, and the generate-types + g10k CronJobs complete — i.e. catalogs still compile and reports still ingest.
Closes#307https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #319
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The puppet MASTER is the singleton CA/master. A second master, even transiently during a rolling update, races on CA/cert signing and shared state (the CA lives on a shared PVC mounted by every master pod). The master was previously driven by an HPA with `minReplicas: 2`, `maxReplicas: 5` and a `RollingUpdate` strategy, so 2-5 masters could coexist normally and a rollout would briefly run old+new masters against the same CA data — a latent CA-corruption/split-brain bug. Recreate guarantees the old pod terminates before the new one starts, so two masters never coexist.
## Changes
- Set `puppetserver-master` `spec.replicas: 1` and `spec.strategy.type: Recreate` (drops RollingUpdate).
- Remove the `puppetserver-masters-autoscaler` HPA and its kustomization entry, which forced 2-5 master replicas and would otherwise override `replicas: 1`.
- Refresh the `puppetserver-master-vpa` note to reflect the pinned-singleton, no-HPA state (VPA stays `updateMode: Off`, recommendation-only).
The compiler (`puppetserver-compiler`) remains the horizontally-scalable tier with its own HPA — untouched. puppetdb/puppetboard untouched.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #341
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret.
## Changes
- Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched.
- Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads.
- Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`.
- Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf.
- Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`).
## Safety review (secret-only / CA workloads)
`vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded.
- `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated.
- `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`).
- `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only.
- `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added.
CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`).
## Notes / coordination
- Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression).
- Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here.
## Validation
- `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains.
- `uvx pre-commit run --all-files` — all hooks pass.
Closes#326
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #339
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- Mount vault-ca-cert secret at /opt/vault-ca-cert.crt in both deployments
- Update cobbler-enc script to use correct CA certificate path
- Resolves OSError about missing TLS CA certificate bundle
Reviewed-on: #62
- Add puppet-shared-bins PVC (10GB) for shared binaries
- Mount /opt/bin in both compiler and master deployments
- Add init container to install uv binary and cobbler script to shared volume
- Update cobbler-enc to use absolute path and uv cache directory
- Configure puppet.conf to reference cobbler-enc from /opt/bin
Reviewed-on: #61
- change puppet/puppetca -> LoadBalancer
- dedicate ip's for puppet and puppetca loadbalancers
- name the puppetserver port
- remove puppet/puppetca ingress
Reviewed-on: #35
complete the implementation of puppet in kubernetes, taking many
features from the openvox helm chart and improving on them. changes from
helm are:
- using vault for storing secrets
- using g10k instead of r10k
- using a single shared g10k cronjob for all masters/compilers
- using a single shared /etc/puppetlabs/code directory (shared, cephfs)
changes:
- deploy puppet master and compiler servers with statefulset/deployment
- deploy puppetdb with postgresql backend, taking advantage of cnpg cluster and pooler
- deploy puppetboard
- all supporting configmaps, services, ingresses, and hpas
- added vaultstaticsecret for eyaml private keys
- configured secure mounting of eyaml keys at /var/lib/puppet/keys/
- updated base kustomization to include all 23 new puppet resource files
Reviewed-on: #29
unkinben
benvin