unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Enable Reloader secret watching, scope existing auto to configmap-only (#326) #339

Merged
benvin merged 2 commits from benvin/issue-326-reloader-secrets into main 2026-08-08 19:35:33 +10:00
Conversation 0 Commits 2 Files Changed 22
+105 -25
unkinben commented 2026-08-08 18:41:08 +10:00
Owner
Copy Link
Copy Source

Why

The re-keyed internal unkin.net intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with ignoreSecrets: true and could not restart on the vault-ca-cert Secret. Enabling secret watching naively is unsafe: many workloads carry the generic reloader.stakater.com/auto, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing auto to ConfigMaps, making secret-reload opt-in per Secret.

Changes

  • Set reloader.ignoreSecrets: false (au-syd1 reloader-system values) so Secrets are watched.
  • Convert every generic reloader.stakater.com/auto: "true" to the ConfigMap-only configmap.reloader.stakater.com/auto: "true" — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads.
  • Add explicit secret.reloader.stakater.com/reload: "vault-ca-cert" to the CA consumers that mount the CA and carry a Reloader annotation: artifactapi/api, cephrgw-operator, puppetserver-master, puppetserver-compiler, litellm, logarchiver.
  • Add secret.reloader.stakater.com/reload: "kanidm-tls" so kanidm rolls when cert-manager renews its leaf.
  • Add docs/ca-rotation.md runbook (indexed in docs/README.md).

Safety review (secret-only / CA workloads)

vault-ca-cert is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). kanidm-tls is a cert-manager leaf. Everything else mounted (environment, *-credentials, eyaml-keys, puppetboard-secrets, s3-credentials, nats-auth, clickhouse-credentials, woodpecker-*) is VSO/CNPG Vault-rotated and deliberately excluded.

  • cephrgw-operator — mounts only Secrets (cephrgw-credentials VSO + vault-ca-cert), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; cephrgw-credentials is VSO so that is now excluded, and reload is scoped to vault-ca-cert only. Comment updated.
  • nats (logging) — old comment "Roll the StatefulSet when nats-auth changes"; nats-auth is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO nats-auth/clickhouse-credentials).
  • artifactapi/ui — mounts neither a ConfigMap nor a Secret; its auto was already a no-op. Left as ConfigMap-only.
  • puppetdb / puppetboard — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added.

CA consumers that mount vault-ca-cert but have no Reloader annotation (CRD-managed or startup-cached) are documented in docs/ca-rotation.md for manual restart rather than annotated here: grafana, observability/vmagent, paperclip, argocd-repo-server, plus CNPG clusters (kubectl cnpg restart).

Notes / coordination

  • Annotations left in their existing location (some sit on the pod template, e.g. litellm, puppetdb; Reloader reads controller-level metadata — placement unchanged from before, no regression).
  • Touches apps/overlays/au-syd1/logging/values-vector-*.yaml, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here.

Validation

  • make kubeconform — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains.
  • uvx pre-commit run --all-files — all hooks pass.

Closes #326

## Why The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret. ## Changes - Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched. - Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads. - Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`. - Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf. - Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`). ## Safety review (secret-only / CA workloads) `vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded. - `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated. - `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`). - `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only. - `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added. CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`). ## Notes / coordination - Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression). - Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here. ## Validation - `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains. - `uvx pre-commit run --all-files` — all hooks pass. Closes #326
unkinben added 1 commit 2026-08-08 18:41:08 +10:00
Enable Reloader secret watching, scope existing auto to configmap-only (#326)
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
ci/woodpecker/pr/vector-test Pipeline failed
Details
ci/woodpecker/pr/pre-commit Pipeline failed
Details
8ce1b5f6bd 
The internal intermediate CA re-key broke CA consumers because Reloader
ignored Secrets and could not restart on the vault-ca-cert Secret. Enabling
generic secret watching naively would restart every workload on each
Vault/VSO secret rotation, so this scopes existing auto annotations to
ConfigMaps and makes secret reload opt-in per Secret.

- set reloader ignoreSecrets: false so Secrets are watched
- convert every generic reloader.stakater.com/auto to the configmap-only
  configmap.reloader.stakater.com/auto form (22 annotations, 19 files)
- add explicit secret.reloader.stakater.com/reload: "vault-ca-cert" to the CA
  consumers (artifactapi api, cephrgw-operator, puppetserver master+compiler,
  litellm, logarchiver) so CA rotation restarts them
- add secret.reloader.stakater.com/reload: "kanidm-tls" so the cert-manager
  leaf renewal rolls kanidm
- add docs/ca-rotation.md runbook and index it

Closes #326

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
unkinben added 1 commit 2026-08-08 18:53:27 +10:00
ci: retrigger pipeline (clone pod scheduling flake on #456)
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
68fdaa6864 
The pre-commit and vector-test workflows failed at the clone step with
"backoff: maximum elapsed time exceeded (pods ... not found)" — a transient
k8s pod-scheduling flake, not a content failure (kubeconform passed).

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
benvin merged commit ba7a1a9509 into main 2026-08-08 19:35:33 +10:00
benvin deleted branch benvin/issue-326-reloader-secrets 2026-08-08 19:35:33 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#339
Delete Branch "benvin/issue-326-reloader-secrets"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?