Roll pods on config change via a pod-template config hash #10

2026-07-12T22:40:03+10:00

unkinben commented

2026-07-12 22:40:03 +10:00

Why

Config changes never reached running pods, which caused the resolver validate-except fix (and earlier the primary-ClusterIP change) to require manual pod deletes. Two compounding causes:

Pods copy config from the projected volume into an emptyDir once at startup (the entrypoint's cp … /run/named/named.conf). The live ConfigMap updates at /etc/bind-operator, but the file named reads is never re-synced, and rndc reconfig re-reads that stale startup copy — so reloadReadyPods is a no-op for config changes.
A manual kubectl rollout restart is reverted: reconcileStatefulSet overwrites the whole pod template each reconcile (existing.Spec.Template = sts.Spec.Template), stripping the restartedAt annotation before the roll completes. (This is exactly why "only one resolver restarted".)

So the only thing that applies new config is a restart, and nothing triggered one reliably.

Fix

Stamp a hash of the projected config (the rendered ConfigMap + the keys.conf Secret) onto the pod template as bind.unkin.net/config-hash. When the config changes, the hash flips → the template changes → the StatefulSet performs a normal rolling restart, so every pod re-copies fresh config on startup. Because the operator owns the template, the restart is operator-driven and never reverted; a stable hash means no spurious restarts.

Covers all named.conf changes (ACLs, views, forwarders, validate-except, primary address) and TSIG key rotation (keys.conf).

Notes

The hash is read after the ConfigMap/keys Secret are reconciled (they're written earlier in the same Reconcile), so it reflects the desired config.
First rollout of this operator version stamps the annotation once, triggering a single rolling restart per cluster — which also picks up any already-pending config.

Tests

TestConfigHashStableAndSensitive: hash is stable when nothing changes and flips on both a ConfigMap change and a keys.conf change. go build/vet/test clean.

## Why Config changes never reached running pods, which caused the resolver `validate-except` fix (and earlier the primary-ClusterIP change) to require manual pod deletes. Two compounding causes: 1. Pods copy config from the projected volume into an emptyDir **once at startup** (the entrypoint's `cp … /run/named/named.conf`). The live ConfigMap updates at `/etc/bind-operator`, but the file `named` reads is never re-synced, and `rndc reconfig` re-reads that **stale startup copy** — so `reloadReadyPods` is a no-op for config changes. 2. A manual `kubectl rollout restart` is **reverted**: `reconcileStatefulSet` overwrites the whole pod template each reconcile (`existing.Spec.Template = sts.Spec.Template`), stripping the `restartedAt` annotation before the roll completes. (This is exactly why "only one resolver restarted".) So the only thing that applies new config is a restart, and nothing triggered one reliably. ## Fix Stamp a hash of the projected config (the rendered ConfigMap + the `keys.conf` Secret) onto the **pod template** as `bind.unkin.net/config-hash`. When the config changes, the hash flips → the template changes → the StatefulSet performs a normal **rolling restart**, so every pod re-copies fresh config on startup. Because the operator owns the template, the restart is operator-driven and never reverted; a stable hash means no spurious restarts. Covers all `named.conf` changes (ACLs, views, forwarders, `validate-except`, primary address) and TSIG key rotation (`keys.conf`). ## Notes - The hash is read after the ConfigMap/keys Secret are reconciled (they're written earlier in the same Reconcile), so it reflects the desired config. - First rollout of this operator version stamps the annotation once, triggering a single rolling restart per cluster — which also picks up any already-pending config. ## Tests - `TestConfigHashStableAndSensitive`: hash is stable when nothing changes and flips on both a ConfigMap change and a `keys.conf` change. `go build/vet/test` clean.

unkinben added 1 commit 2026-07-12 22:40:03 +10:00

Roll pods on config change via a pod-template config hash

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

2deea3e023

Pods copy config from the projected volume into an emptyDir once at
startup, so a ConfigMap or keys.conf change never reaches a running pod:
rndc reconfig re-reads the stale startup copy, and a manual
`kubectl rollout restart` is reverted because the operator overwrites the
pod template every reconcile. The only thing that applies new config is a
restart, and nothing triggered one.

Stamp a hash of the projected config (rendered ConfigMap + keys.conf
Secret) onto the pod template as bind.unkin.net/config-hash. When config
changes the hash flips, the template changes, and the StatefulSet does a
normal rolling restart so every pod re-copies fresh config. The operator
owns the template, so the restart is operator-driven and not reverted; a
stable hash means no spurious restarts.

Covers named.conf changes (ACLs, views, forwarders, validate-except,
primary address) and TSIG key rotation.

benvin merged commit 8fac152537 into main

2026-07-12 22:46:52 +10:00

benvin deleted branch benvin/config-hash-rollout

2026-07-12 22:46:52 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/bind-operator#10