Ben Vincent
8ce1b5f6bd
The internal intermediate CA re-key broke CA consumers because Reloader ignored Secrets and could not restart on the vault-ca-cert Secret. Enabling generic secret watching naively would restart every workload on each Vault/VSO secret rotation, so this scopes existing auto annotations to ConfigMaps and makes secret reload opt-in per Secret. - set reloader ignoreSecrets: false so Secrets are watched - convert every generic reloader.stakater.com/auto to the configmap-only configmap.reloader.stakater.com/auto form (22 annotations, 19 files) - add explicit secret.reloader.stakater.com/reload: "vault-ca-cert" to the CA consumers (artifactapi api, cephrgw-operator, puppetserver master+compiler, litellm, logarchiver) so CA rotation restarts them - add secret.reloader.stakater.com/reload: "kanidm-tls" so the cert-manager leaf renewal rolls kanidm - add docs/ca-rotation.md runbook and index it Closes #326 Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
kanidm
Three-replica kanidm identity server with Vault-managed replication certificates.
Architecture
- Per-pod
server-N.tomlinresources/— each has its own replication origin hardcoded config-initbusybox init container copies the right config and injects peer certs from the vault-syncedkanidm-repl-certsSecret at pod startupreloader.stakater.com/auto: "true"triggers a rolling restart when the ConfigMap or Secret changes- Vault path:
kv/kubernetes/namespace/kanidm/default/repl-certs- Keys:
kanidm-0,kanidm-1,kanidm-2— each holds that pod's replication certificate
- Keys:
Initial setup
After the first pod starts, generate the admin credentials:
kubectl exec -n kanidm kanidm-0 -- /sbin/kanidmd recover-account -c /config/server.toml admin
kubectl exec -n kanidm kanidm-0 -- /sbin/kanidmd recover-account -c /config/server.toml idm_admin
Replication certificate rotation
When certs need to be renewed, update vault and reloader will roll the StatefulSet:
# Get new cert from a pod
kubectl exec -it -n kanidm kanidm-N -- /sbin/kanidmd renew-replication-certificate -c /config/server.toml
# Write updated cert to vault (reloader triggers restart automatically)
vault kv patch kv/kubernetes/namespace/kanidm/default/repl-certs "kanidm-N=<cert>"
Resolving domain UUID mismatch
If pods initialized independently (each with a different domain UUID), replication will fail with
Consumer Domain UUID does not match. Fix by resetting kanidm-1 and kanidm-2 to sync from
kanidm-0 (the authoritative node):
# Scale down to avoid split-brain during reset
kubectl scale statefulset -n kanidm kanidm --replicas=1
# Delete the stale PVCs for the replica pods
kubectl delete pvc -n kanidm data-kanidm-1 data-kanidm-2
# Scale back up — replicas start with empty DBs and automatic_refresh=true
# will trigger a full sync from kanidm-0 once TLS peer certs are verified
kubectl scale statefulset -n kanidm kanidm --replicas=3