Reconcile zone apex NS off the pod IP placeholder #22

Merged
benvin merged 5 commits from benvin/zone-nameservers into main 2026-09-26 23:49:01 +10:00
Member

Every primary zone was seeded with an apex NS ns1.<zone> glued to the primary pod IP, and nothing replaced it: BIND ignores an RRset-wide delete at a zone apex, so neither a spec.records apex NS nor a later reconcile could displace it. acme.unkin.net has served a stale rke2 pod IP to public resolvers for 54 days.

  • add BindZone.spec.nameservers, the names published at the apex
  • converge the live apex NS RRset every reconcile, adding and retracting per record
  • fold a spec.records apex NS into that path, so it replaces instead of appending
  • read the live RRset with a TSIG-signed query, so a view-scoped zone is visible
  • publish without retracting when the RRset cannot be read
  • apply records before the apex NS, so an in-zone nameserver has its address first
  • default to the primary stable in-cluster name, so a seed needs no pod-IP glue

Closes #21

Every primary zone was seeded with an apex `NS ns1.<zone>` glued to the primary pod IP, and nothing replaced it: BIND ignores an RRset-wide delete at a zone apex, so neither a `spec.records` apex NS nor a later reconcile could displace it. acme.unkin.net has served a stale rke2 pod IP to public resolvers for 54 days. - add `BindZone.spec.nameservers`, the names published at the apex - converge the live apex NS RRset every reconcile, adding and retracting per record - fold a `spec.records` apex NS into that path, so it replaces instead of appending - read the live RRset with a TSIG-signed query, so a view-scoped zone is visible - publish without retracting when the RRset cannot be read - apply records before the apex NS, so an in-zone nameserver has its address first - default to the primary stable in-cluster name, so a seed needs no pod-IP glue Closes #21
unkin-agent added 1 commit 2026-09-26 18:27:33 +10:00
reconcile zone apex NS off the pod IP placeholder
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
4a41cbc427
Add BindZone.spec.nameservers and sync the apex NS RRset on every reconcile.
Author
Member
  • internal/controller/bindzone_controller.go:128 — apexNSUpdates has no live-state input (pure function of spec) and bind.NSUpdate always emits update delete+update add unconditionally; every primary zone with resolvable creds gets a real dynamic-update transaction every reconcile (requeueLong = 2m, internal/controller/helpers.go:20) forever, whether or not the apex NS RRset actually changed → bump the SOA serial / send the delete+add only when the desired values differ from what is currently published (query current RRset, or hash-compare against a status field) before calling NSUpdate.
  • internal/controller/zone_helpers.go:161 — ns1 seed-glue cleanup is gated on len(zone.Spec.Nameservers) > 0; a zone that never sets spec.nameservers (the exact acme.unkin.net scenario in the PR description, using only the primary-address fallback) gets its apex NS repointed but the stale ns1 A <pod-ip> record is never deleted → also run the glue cleanup when nameservers falls back to the default, not only when explicitly declared.
  • internal/controller/bindzone_controller.go:128-131 — apex NS sync failures are swallowed at V(1) log level with no condition/event/status field; a zone can reach Ready/Provisioned while the delegation bug this PR closes is still unfixed for it, with no operator-visible signal → surface at least a status condition or count of failed apex syncs.
  • nit: internal/controller/zone_helpers.go:141 — owns() compares record/nameserver names via fqdn() with no case-folding (unlike the type comparison, which uses EqualFold); mixed-case zone or record names in spec.nameservers/spec.records will fail to match and reintroduce the ns1/@ NS churn.
- internal/controller/bindzone_controller.go:128 — `apexNSUpdates` has no live-state input (pure function of spec) and `bind.NSUpdate` always emits `update delete`+`update add` unconditionally; every primary zone with resolvable creds gets a real dynamic-update transaction every reconcile (requeueLong = 2m, `internal/controller/helpers.go:20`) forever, whether or not the apex NS RRset actually changed → bump the SOA serial / send the delete+add only when the desired values differ from what is currently published (query current RRset, or hash-compare against a status field) before calling NSUpdate. - internal/controller/zone_helpers.go:161 — ns1 seed-glue cleanup is gated on `len(zone.Spec.Nameservers) > 0`; a zone that never sets `spec.nameservers` (the exact acme.unkin.net scenario in the PR description, using only the primary-address fallback) gets its apex NS repointed but the stale `ns1 A <pod-ip>` record is never deleted → also run the glue cleanup when nameservers falls back to the default, not only when explicitly declared. - internal/controller/bindzone_controller.go:128-131 — apex NS sync failures are swallowed at V(1) log level with no condition/event/status field; a zone can reach `Ready`/`Provisioned` while the delegation bug this PR closes is still unfixed for it, with no operator-visible signal → surface at least a status condition or count of failed apex syncs. - nit: internal/controller/zone_helpers.go:141 — `owns()` compares record/nameserver names via `fqdn()` with no case-folding (unlike the type comparison, which uses `EqualFold`); mixed-case zone or record names in `spec.nameservers`/`spec.records` will fail to match and reintroduce the ns1/@ NS churn.
unkin-agent added 1 commit 2026-09-26 18:50:59 +10:00
converge apex NS per record, not by RRset replace
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
e4ed6c8052
BIND ignores an RRset-wide delete of apex NS, so the previous replace only
appended to the seed placeholder.
Author
Member
  • internal/bind/rndc.go:90 — ApexNS queries dig +short @127.0.0.1 <zone> NS with no view selector, unlike every other rndc-based call in this file (ZoneSerial/AddZone/DelZone/ZoneExists all take/thread view). For a zone bound to a BindView whose match-clients does not route the pods own loopback query into that view (exactly config/samples/04-view.yamls split-horizon setup), the unsigned dig gets REFUSED with exit 0 and empty output — verified live against named 9.20 with a two-view config matching that sample. ApexNS returns (nil, nil), so apexNSUpdates (zone_helpers.go:158) always computes live=[], meaning del := missing(live, desired) is always empty: the stale placeholder apex NS is never deleted and the ns1 glue A never retires (zone_helpers.go:176-179 never fires). Confirmed the signed NSUpdate (TSIG key in match-clients) does land in the right view while the unsigned read never sees it — the two paths are on different sides of the view ACL, so convergence never completes for any BindView-scoped zone. → thread zone.Spec.ViewRef into ApexNS the same way the other rndc.go calls do (rndc has a zonestatus ... in <view> equivalent for NS; dig has none, so this likely needs to go through rndc/an authenticated query path instead of bare dig).
  • internal/controller/zone_helpers.go:141-146 — zoneNameservers folding an apex NS from spec.Records into the declared set copies only rec.Values, dropping rec.TTL. apexNSUpdates is always called with zone.Spec.DefaultTTL (bindzone_controller.go:135), so a user-set TTL on that record (a supported, tested combination per TestApexNSUpdatesLeavesRecordOwnedGlue) is silently overridden by the zone default once the record is routed through the apex path. → carry the records TTL through to apexNSUpdates, or document that an apex NS entry in spec.Records cannot set its own TTL.
  • nit: internal/controller/zone_helpers.go:176 — glue retirement is hardcoded to the name "ns1", but renderSeedZone (seed.go) glues every declared nameserver that resolves in-zone, not just one named ns1. A zone ever seeded/declared with more than one in-zone nameserver leaves any non-ns1 glue A record permanently orphaned once that name drops out of spec.nameservers.
- internal/bind/rndc.go:90 — `ApexNS` queries `dig +short @127.0.0.1 <zone> NS` with no view selector, unlike every other rndc-based call in this file (`ZoneSerial`/`AddZone`/`DelZone`/`ZoneExists` all take/thread `view`). For a zone bound to a BindView whose match-clients does not route the pods own loopback query into that view (exactly `config/samples/04-view.yaml`s split-horizon setup), the unsigned dig gets REFUSED with exit 0 and empty output — verified live against named 9.20 with a two-view config matching that sample. `ApexNS` returns `(nil, nil)`, so `apexNSUpdates` (zone_helpers.go:158) always computes `live=[]`, meaning `del := missing(live, desired)` is always empty: the stale placeholder apex NS is never deleted and the ns1 glue A never retires (zone_helpers.go:176-179 never fires). Confirmed the signed `NSUpdate` (TSIG key in match-clients) does land in the right view while the unsigned read never sees it — the two paths are on different sides of the view ACL, so convergence never completes for any BindView-scoped zone. → thread `zone.Spec.ViewRef` into `ApexNS` the same way the other rndc.go calls do (rndc has a `zonestatus ... in <view>` equivalent for NS; dig has none, so this likely needs to go through rndc/an authenticated query path instead of bare dig). - internal/controller/zone_helpers.go:141-146 — `zoneNameservers` folding an apex NS from `spec.Records` into the declared set copies only `rec.Values`, dropping `rec.TTL`. `apexNSUpdates` is always called with `zone.Spec.DefaultTTL` (bindzone_controller.go:135), so a user-set TTL on that record (a supported, tested combination per `TestApexNSUpdatesLeavesRecordOwnedGlue`) is silently overridden by the zone default once the record is routed through the apex path. → carry the records TTL through to `apexNSUpdates`, or document that an apex NS entry in spec.Records cannot set its own TTL. - nit: internal/controller/zone_helpers.go:176 — glue retirement is hardcoded to the name "ns1", but `renderSeedZone` (seed.go) glues every declared nameserver that resolves in-zone, not just one named ns1. A zone ever seeded/declared with more than one in-zone nameserver leaves any non-ns1 glue A record permanently orphaned once that name drops out of `spec.nameservers`.
unkin-agent added 1 commit 2026-09-26 19:06:51 +10:00
retract apex NS from recorded state, not a live query
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
dc57ac1b2d
An unsigned localhost query silently returns nothing for a zone behind a
BindView, which would strand the placeholder. Record what was published instead.
Author
Member
  • internal/controller/zone_helpers.go:166-174 (publishedNameservers) — unrecorded-status guess always assumes both the ns1 placeholder and the cluster-stable name are already live. For a zone actually seeded with only the ns1 placeholder, declaring spec.nameservers = the cluster-stable name (the documented migration target) computes zero adds and only a delete of the sole existing NS value. Verified against real named 9.20: BIND refuses the transaction ("attempt to delete last NS ignored" / "post update name server sanity check failed"), NSUpdate errors, and the zone is stuck in Error/ApexNSSyncFailed permanently — status.Nameservers is never written on this path so the same broken diff recomputes every reconcile → only guess a name as already-published when it's also in desired, or otherwise never let a delete outrun an add when the live RRset can't be confirmed.
  • internal/controller/bindzone_controller.go:128, zone_helpers.go:134-159 (nsDeclared gate) — apex convergence and the status.Nameservers write are both skipped whenever nsDeclared is false. Consequences: (1) a pre-PR zone seeded with the ns1+pod-IP placeholder whose spec sets neither nameservers nor an apex NS record is never migrated to the fallback, contradicting the Nameservers field doc ("When empty the apex NS is the primary's stable in-cluster DNS name"); (2) a zone that had spec.nameservers set and then clears it reverts to declared=false and its old NS values are left live forever with no retraction. Confirmed by mutation: replacing if nsDeclared with if true in Reconcile fails no existing test — this transition has zero coverage → converge toward the fallback (or previously-recorded status) even when undeclared, and add a Reconcile-level test for both transitions.
  • internal/bind/rndc.go:54-63 (DelZone) + internal/bind/seed.go (EnsureSeedZone/PlanSeed) — rndc delzone omits -clean, so the on-disk zone file/journal survive CR deletion. A BindZone CR deleted and recreated with the same zone name (PVC intact) gets created=true again but PlanSeed preserves the surviving valid file (WriteSeed=false) instead of reseeding; the new CR's status.Nameservers starts empty, so publishedNameservers's guess has no reason to match whatever the surviving file actually publishes, and any NS name outside that guess can never be retracted by apexNSUpdates.
  • nit: internal/controller/zone_helpers.go:134-152 (zoneNameservers) — the spec.records apex-NS scan is skipped once names is non-empty from spec.Nameservers (line 140 break), fine today, but the two sources are otherwise unordered/undocumented as to precedence beyond "nameservers wins" — worth a one-line doc note given a later refactor could invert the loop order silently.
- internal/controller/zone_helpers.go:166-174 (publishedNameservers) — unrecorded-status guess always assumes both the ns1 placeholder and the cluster-stable name are already live. For a zone actually seeded with only the ns1 placeholder, declaring spec.nameservers = the cluster-stable name (the documented migration target) computes zero adds and only a delete of the sole existing NS value. Verified against real named 9.20: BIND refuses the transaction ("attempt to delete last NS ignored" / "post update name server sanity check failed"), NSUpdate errors, and the zone is stuck in Error/ApexNSSyncFailed permanently — status.Nameservers is never written on this path so the same broken diff recomputes every reconcile → only guess a name as already-published when it's also in desired, or otherwise never let a delete outrun an add when the live RRset can't be confirmed. - internal/controller/bindzone_controller.go:128, zone_helpers.go:134-159 (nsDeclared gate) — apex convergence and the status.Nameservers write are both skipped whenever nsDeclared is false. Consequences: (1) a pre-PR zone seeded with the ns1+pod-IP placeholder whose spec sets neither nameservers nor an apex NS record is never migrated to the fallback, contradicting the Nameservers field doc ("When empty the apex NS is the primary's stable in-cluster DNS name"); (2) a zone that had spec.nameservers set and then clears it reverts to declared=false and its old NS values are left live forever with no retraction. Confirmed by mutation: replacing `if nsDeclared` with `if true` in Reconcile fails no existing test — this transition has zero coverage → converge toward the fallback (or previously-recorded status) even when undeclared, and add a Reconcile-level test for both transitions. - internal/bind/rndc.go:54-63 (DelZone) + internal/bind/seed.go (EnsureSeedZone/PlanSeed) — rndc delzone omits -clean, so the on-disk zone file/journal survive CR deletion. A BindZone CR deleted and recreated with the same zone name (PVC intact) gets created=true again but PlanSeed preserves the surviving valid file (WriteSeed=false) instead of reseeding; the new CR's status.Nameservers starts empty, so publishedNameservers's guess has no reason to match whatever the surviving file actually publishes, and any NS name outside that guess can never be retracted by apexNSUpdates. - nit: internal/controller/zone_helpers.go:134-152 (zoneNameservers) — the spec.records apex-NS scan is skipped once `names` is non-empty from spec.Nameservers (line 140 break), fine today, but the two sources are otherwise unordered/undocumented as to precedence beyond "nameservers wins" — worth a one-line doc note given a later refactor could invert the loop order silently.
unkin-agent added 1 commit 2026-09-26 19:25:10 +10:00
read the live apex NS with a signed query, retract nothing when unreadable
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
08d46ccce0
An unsigned localhost query is REFUSED for a zone behind a key-matched view and
answers empty with exit 0; signing it with the update creds reaches the view.
Author
Member
  • internal/controller/zone_helpers.go:172-195 (apexNSUpdates) — adding a new in-zone name to spec.nameservers on an already-existing zone (created=false) emits update add <apex> NS <newname> with no glue A record — glue is only ever written by seed.go's renderSeedZone, at zone creation only. named rejects the add outright ("post update name server sanity check failed" / REFUSED, verified live against bind9 9.20) → reconcile fails with ApexNSSyncFailed on every subsequent pass, permanently. The apex-sync transaction (bindzone_controller.go:126-145) runs alone, before and separately from the records-sync transaction (bindzone_controller.go:146-155), so adding the matching spec.records A record in the same edit does not help: the apex transaction still fails first and the records transaction never runs. Have apexNSUpdates seed/add glue for any newly-declared in-zone nameserver in the same transaction as the NS add.
  • internal/bind/nsupdate.go:55-62 (ApexNS parsing) — no test exercises this parser. Deleting the entire "ends with a dot" guard (replacing it with "any non-empty line is an answer") leaves go test ./internal/... ./api/... fully green. Add a unit test that feeds the parser real dig output shapes directly (REFUSED/NXDOMAIN/SERVFAIL empty output, a real NS answer, a ;;-comment line) without going through Exec/k8s.
  • nit: internal/bind/nsupdate.go:59 — the filter is "line ends with .", not "line is a dig comment"; strings /usr/bin/dig (9.20) turns up at least one comment that itself ends in a period (;; WARNING -- TSIG key was not used.), which this would append to the NS list as if it were an answer. Filter on the actual dig comment marker (leading ;) instead of inferring it from message wording.
  • nit: internal/controller/zone_helpers.go:181 — len(live) == 0 || is dead: del := missing(live, desired) ranges over live, so del is already nil whenever live is empty; len(del) == 0 alone is equivalent. Confirmed by deleting the clause: no test fails. Drop it, or add a test that actually needs it.
  • nit: internal/controller/bindzone_controller.go:126-127 — zoneUpdateCreds (a Secret GET) now runs unconditionally on every primary-zone reconcile, including zones with neither nsDeclared nor any spec.records (previously gated behind len(zone.Spec.Records) > 0). Gate it behind nsDeclared || len(zone.Spec.Records) > 0 to skip the wasted API call on the common no-op zone.
  • nit: internal/controller/zone_helpers.go:172-195 — apex NS TTL is never converged for names that stay in the desired set: apexNSUpdates diffs desired/live by name only, and ApexNS cannot observe TTL at all (dig +short prints bare names, no TTL). Changing spec.defaultTTL or a spec.records apex-NS TTL override on an already-converged zone silently never republishes the new TTL, despite the field doc's "kept in sync on every reconcile."
- internal/controller/zone_helpers.go:172-195 (apexNSUpdates) — adding a *new* in-zone name to `spec.nameservers` on an already-existing zone (`created=false`) emits `update add <apex> NS <newname>` with no glue A record — glue is only ever written by seed.go's `renderSeedZone`, at zone creation only. named rejects the add outright ("post update name server sanity check failed" / REFUSED, verified live against bind9 9.20) → reconcile fails with `ApexNSSyncFailed` on every subsequent pass, permanently. The apex-sync transaction (bindzone_controller.go:126-145) runs alone, before and separately from the records-sync transaction (bindzone_controller.go:146-155), so adding the matching `spec.records` A record in the same edit does not help: the apex transaction still fails first and the records transaction never runs. Have apexNSUpdates seed/add glue for any newly-declared in-zone nameserver in the same transaction as the NS add. - internal/bind/nsupdate.go:55-62 (ApexNS parsing) — no test exercises this parser. Deleting the entire "ends with a dot" guard (replacing it with "any non-empty line is an answer") leaves `go test ./internal/... ./api/...` fully green. Add a unit test that feeds the parser real dig output shapes directly (REFUSED/NXDOMAIN/SERVFAIL empty output, a real NS answer, a `;;`-comment line) without going through Exec/k8s. - nit: internal/bind/nsupdate.go:59 — the filter is "line ends with `.`", not "line is a dig comment"; `strings /usr/bin/dig` (9.20) turns up at least one comment that itself ends in a period (`;; WARNING -- TSIG key was not used.`), which this would append to the NS list as if it were an answer. Filter on the actual dig comment marker (leading `;`) instead of inferring it from message wording. - nit: internal/controller/zone_helpers.go:181 — `len(live) == 0 ||` is dead: `del := missing(live, desired)` ranges over `live`, so `del` is already nil whenever `live` is empty; `len(del) == 0` alone is equivalent. Confirmed by deleting the clause: no test fails. Drop it, or add a test that actually needs it. - nit: internal/controller/bindzone_controller.go:126-127 — `zoneUpdateCreds` (a Secret GET) now runs unconditionally on every primary-zone reconcile, including zones with neither `nsDeclared` nor any `spec.records` (previously gated behind `len(zone.Spec.Records) > 0`). Gate it behind `nsDeclared || len(zone.Spec.Records) > 0` to skip the wasted API call on the common no-op zone. - nit: internal/controller/zone_helpers.go:172-195 — apex NS TTL is never converged for names that stay in the desired set: `apexNSUpdates` diffs `desired`/`live` by name only, and `ApexNS` cannot observe TTL at all (`dig +short` prints bare names, no TTL). Changing `spec.defaultTTL` or a `spec.records` apex-NS TTL override on an already-converged zone silently never republishes the new TTL, despite the field doc's "kept in sync on every reconcile."
unkin-agent added 1 commit 2026-09-26 19:41:55 +10:00
apply records before the apex NS, so in-zone glue exists first
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
cc714dc2b4
named rejects an apex NS pointing at an in-zone name with no address record,
so the glue has to land in an earlier transaction.
benvin merged commit 057575b9b5 into main 2026-09-26 23:49:01 +10:00
benvin deleted branch benvin/zone-nameservers 2026-09-26 23:49:01 +10:00
Sign in to join this conversation.