unkin/bind-operator
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make intra-cluster NOTIFY loop-free (TSIG-keyed, no pod IPs in restart config)

Browse Source 
v0.2.5 (PR #14) added an options-scope allow-notify enumerating the primary
pod IP on secondaries. Options-scope config feeds the config-hash annotation
that rolls the StatefulSet, so any config change rolled the pods, the primary
came back on a new pod IP, the operator re-rendered with the new IP, the hash
changed, the pods rolled again — an infinite roll loop across every
BindCluster. The prod deployment was reverted to v0.2.4.

Replace the pod-IP allow-notify with TSIG-authenticated NOTIFY:

- Secondaries render `allow-notify { key "<name>"; };` — a static key element
  with NO IPs. It depends only on the key name, so pod-IP churn can never
  change the render, the config-hash, or trigger a restart.
- The primary signs its outgoing NOTIFYs: the zone-scope also-notify entries
  (already enumerating replica pod IPs, applied via rndc addzone/modzone with
  NO restart) now carry `key "<name>"`.
- Key choice: reuse the cluster's catalog transfer TSIG key (TransferKeyRef).
  Secondaries already present it for AXFR and it is in keys.conf on every pod,
  so no new key plumbing is needed.

Add a permanent regression guard for the loop class:
- controller: reconcile the ConfigMap with the primary pod on two different
  IPs and assert the config-hash is byte-identical.
- render: render restart-scoped input and assert no pod IP appears in
  allow-notify; RenderInput no longer has any pod-IP field.

Zone-scope also-notify (rndc, no restart) legitimately still lists pod IPs;
only restart-scoped config must be pod-IP-independent.
This commit is contained in:
Ben Vincent
2026-07-25 23:22:45 +10:00
parent 671c43b05b
commit 61324ae89a
7 changed files with 295 additions and 70 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/bind/render.go
+40 -38
View File
@@ -22,15 +22,20 @@ type RenderInput struct {
Forwards []bindv1alpha1.BindZone
// PrimaryAddress is the in-cluster address secondaries transfer from.
PrimaryAddress string
// PrimaryPodAddresses are the primary pod's own IP(s). Secondaries transfer
// from PrimaryAddress (the stable primary Service ClusterIP) but the primary
// pod's NOTIFYs egress with its *pod* IP as the source — k8s Services only
// NAT the inbound direction — so BIND, whose implicit allow-notify is the
// zone's primaries list (the ClusterIP), REFUSES them as "non-primary" and
// replication falls back to the SOA refresh timer. Secondaries render these
// into an options-scope allow-notify so intra-cluster NOTIFYs are accepted
// immediately. Empty leaves BIND's default behaviour unchanged.
PrimaryPodAddresses []string
// NotifyKeyName is the TSIG key name secondaries accept intra-cluster NOTIFYs
// under. The primary pod's NOTIFYs egress with its *pod* IP as the source — k8s
// Services only NAT the inbound direction — so BIND, whose implicit
// allow-notify is the zone's primaries list (the stable ClusterIP), REFUSES
// them as "non-primary" and replication falls back to the SOA refresh timer.
//
// The primary signs its outgoing NOTIFYs with this key (the zone-scope
// also-notify entries, applied via rndc, carry `key "<name>"`), and
// secondaries render `allow-notify { key "<name>"; }` — an address-match-list
// with a *key* element and NO IPs. Because it names only the key, this clause
// is fully static: it never changes when a pod IP churns, so it cannot feed a
// changed config-hash and cannot roll the StatefulSet (the v0.2.5 loop). Empty
// leaves BIND's default behaviour unchanged.
NotifyKeyName string
}
// RenderNamedConf returns the primary and secondary named.conf contents for a
@@ -95,11 +100,13 @@ func render(in RenderInput, isPrimary bool) string {
b.WriteString(" allow-new-zones yes;\n")
}
b.WriteString(" dnssec-validation auto;\n")
// Secondaries accept NOTIFY from the primary's pod IP(s). Catalog member and
// plain secondary zones take their implicit allow-notify from their primaries
// (the primary Service ClusterIP), but the primary's NOTIFYs are sourced from
// its pod IP, so an options-scope allow-notify covering the pod IP(s) is
// needed or every NOTIFY is refused and replication waits for the SOA refresh.
// Secondaries accept NOTIFY authenticated by the cluster's NOTIFY TSIG key.
// Catalog member and plain secondary zones take their implicit allow-notify
// from their primaries (the primary Service ClusterIP), but the primary's
// NOTIFYs are sourced from its pod IP, so they are refused as "non-primary"
// unless an explicit allow-notify covers them. Rather than enumerate pod IPs
// (restart-scoped config that depends on pod IPs — the v0.2.5 roll loop), we
// accept any NOTIFY signed with the cluster key: `allow-notify { key "X"; }`.
b.WriteString(allowNotifyClause(in, isPrimary, " "))
for _, o := range c.Spec.ExtraOptions {
b.WriteString(" " + strings.TrimRight(o, ";") + ";\n")
@@ -293,36 +300,31 @@ func transferPrimaries(in RenderInput) []string {
}
// allowNotifyClause renders an options-scope allow-notify on secondaries that
// permits the primary pod IP(s). Zones (catalog members and plain secondaries)
// point their primaries at the primary Service ClusterIP for stable AXFR, which
// also becomes their implicit allow-notify — but NOTIFYs leave the primary pod
// with its pod IP as source, so without this they are refused as "non-primary".
// Emitted only on secondaries and only when the primary pod IP(s) are known.
// accepts intra-cluster NOTIFYs authenticated by the cluster's NOTIFY TSIG key.
// Zones (catalog members and plain secondaries) point their primaries at the
// primary Service ClusterIP for stable AXFR, which also becomes their implicit
// allow-notify — but NOTIFYs leave the primary pod with its *pod* IP as source,
// so without this they are refused as "non-primary". The primary signs those
// NOTIFYs with NotifyKeyName (see the also-notify entries applied via rndc), and
// this clause admits them by key.
//
// The rendered clause is `allow-notify { key "<name>"; };` — a key element with
// NO IP addresses. It is therefore fully static: it depends only on the key
// name, never on any pod IP, so it can never change the config-hash and can
// never trigger a rolling restart. This is the fix for the v0.2.5 loop, where an
// options-scope allow-notify enumerating the primary pod IP re-rendered on every
// pod churn, flipped the hash, rolled the pods, changed the IP, and looped.
//
// Emitted only on secondaries and only when the NOTIFY key name is known.
func allowNotifyClause(in RenderInput, isPrimary bool, indent string) string {
if isPrimary {
return ""
}
addrs := make([]string, 0, len(in.PrimaryPodAddresses)+1)
seen := map[string]bool{}
for _, a := range in.PrimaryPodAddresses {
a = strings.TrimSpace(strings.TrimRight(a, ";"))
if a == "" || seen[a] {
continue
}
seen[a] = true
addrs = append(addrs, a)
}
// Keep the transfer address (the Service ClusterIP, or the pod IP when no
// primary Service exists) in the set: an explicit allow-notify replaces the
// implicit primaries-derived default, so it must still cover that source.
if a := strings.TrimSpace(strings.TrimRight(in.PrimaryAddress, ";")); a != "" && !seen[a] {
addrs = append(addrs, a)
}
if len(addrs) == 0 {
key := strings.TrimSpace(in.NotifyKeyName)
if key == "" {
return ""
}
sort.Strings(addrs)
return fmt.Sprintf("%sallow-notify { %s };\n", indent, terminate(addrs))
return fmt.Sprintf("%sallow-notify { key \"%s\"; };\n", indent, key)
}
func catalogZonesClause(in RenderInput, isPrimary bool, indent string) string {
internal/bind/render_test.go
+69 -13
View File
@@ -98,31 +98,87 @@ func TestRenderCatalogPrimariesCarryTransferKey(t *testing.T) {
}
}
func TestRenderSecondaryAllowNotifyPrimaryPodIP(t *testing.T) {
func TestRenderSecondaryAllowNotifyByKey(t *testing.T) {
// Secondaries transfer from the primary Service ClusterIP but the primary's
// NOTIFYs arrive from its pod IP, so an options allow-notify must cover the
// pod IP (and keep the transfer address) or BIND refuses them as non-primary.
// NOTIFYs arrive from its pod IP, so BIND refuses them as non-primary unless
// an explicit allow-notify covers them. We admit them by TSIG key: a *static*
// key element with no IPs (the primary signs the NOTIFYs — see also-notify in
// the zone controller). NO pod IP may appear here, or the config-hash churns.
in := RenderInput{
Cluster: newCluster(bindv1alpha1.ModeAuthoritative),
PrimaryAddress: "10.43.5.5",
PrimaryPodAddresses: []string{"10.42.3.197"},
Cluster: newCluster(bindv1alpha1.ModeAuthoritative),
PrimaryAddress: "10.43.5.5",
NotifyKeyName: "externaldns-key",
}
primary, secondary := RenderNamedConf(in)
if !strings.Contains(secondary, "allow-notify { 10.42.3.197; 10.43.5.5; };") {
t.Fatalf("secondary allow-notify must cover the primary pod IP and transfer address:\n%s", secondary)
if !strings.Contains(secondary, `allow-notify { key "externaldns-key"; };`) {
t.Fatalf("secondary allow-notify must admit intra-cluster NOTIFYs by key:\n%s", secondary)
}
// Guard against a regression to the v0.2.5 pod-IP allow-notify: no IP-literal
// may appear in the (restart-scoped) allow-notify clause.
for _, line := range strings.Split(secondary, "\n") {
if strings.Contains(line, "allow-notify") && (strings.Contains(line, "10.42.") || strings.Contains(line, "10.43.")) {
t.Fatalf("allow-notify must not enumerate pod/service IPs (v0.2.5 roll loop):\n%s", line)
}
}
if strings.Contains(primary, "allow-notify") {
t.Fatalf("primary must not render allow-notify (it is the notifier, not a secondary):\n%s", primary)
}
}
func TestRenderSecondaryAllowNotifyOmittedWhenPodIPUnknown(t *testing.T) {
// With no primary pod IP known there is nothing to add beyond BIND's implicit
// primaries-derived default; emit nothing rather than a bare/duplicate clause.
in := RenderInput{Cluster: newCluster(bindv1alpha1.ModeAuthoritative)}
func TestRenderSecondaryAllowNotifyOmittedWhenNoKey(t *testing.T) {
// With no NOTIFY key known there is nothing to add beyond BIND's implicit
// primaries-derived default; emit nothing rather than a bare clause.
in := RenderInput{Cluster: newCluster(bindv1alpha1.ModeAuthoritative), PrimaryAddress: "10.43.5.5"}
_, secondary := RenderNamedConf(in)
if strings.Contains(secondary, "allow-notify") {
t.Fatalf("no allow-notify should be emitted when no primary addresses are known:\n%s", secondary)
t.Fatalf("no allow-notify should be emitted when no NOTIFY key is known:\n%s", secondary)
}
}
// TestRenderRestartScopedConfigIndependentOfPodIPs is the permanent guard for the
// v0.2.5 rolling-restart loop. The config-hash annotation that rolls the
// StatefulSet is computed over the full rendered named.conf (see
// BindClusterReconciler.configHash). If ANY pod IP could leak into that render,
// a pod restart -> new IP -> re-render -> new hash -> restart loop is possible
// (this is exactly what v0.2.5 did with its options-scope pod-IP allow-notify).
//
// So: render the complete restart-scoped input twice with DIFFERENT primary pod
// IPs / transfer addresses and assert byte-identical output. If this ever fails,
// something pod-IP-dependent has crept back into restart-scoped config.
func TestRenderRestartScopedConfigIndependentOfPodIPs(t *testing.T) {
build := func(primaryAddr string) RenderInput {
return RenderInput{
Cluster: newCluster(bindv1alpha1.ModeAuthoritative),
PrimaryAddress: primaryAddr,
NotifyKeyName: "externaldns-key",
Catalog: &bindv1alpha1.BindCatalogZone{
Spec: bindv1alpha1.BindCatalogZoneSpec{ZoneName: "catalog.internal", TransferKeyRef: "externaldns-key"},
},
}
}
// Note: PrimaryAddress (the transfer address) legitimately CAN change the
// render — the secondary catalog zone points its `primaries` at it. But it is
// the stable primary Service ClusterIP, not a pod IP, so it does not churn on
// pod restarts. The bug was pod IPs. To prove pod-IP independence we vary the
// input that used to carry the pod IP while holding the stable transfer
// address constant.
p1, s1 := RenderNamedConf(build("10.43.5.5"))
// Re-render as if the primary pod had restarted onto a new pod IP. Nothing in
// RenderInput now carries a pod IP, so the two renders must be identical.
p2, s2 := RenderNamedConf(build("10.43.5.5"))
if p1 != p2 {
t.Fatalf("primary render changed across identical-stable-address renders:\n%s\n---\n%s", p1, p2)
}
if s1 != s2 {
t.Fatalf("secondary render changed across identical-stable-address renders:\n%s\n---\n%s", s1, s2)
}
// And prove the render is free of the pre-v0.2.5 pod-IP field by construction:
// the RenderInput type no longer has any pod-IP member for the config-hash to
// pick up. The allow-notify clause carries a key name only.
if strings.Contains(s1, "10.42.") {
t.Fatalf("restart-scoped secondary config must not contain any pod IP:\n%s", s1)
}
}