unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

dns: stop exporting legacy DNS fragments (manage_export false) #495

Merged
benvin merged 1 commits from benvin/dns-export-off into develop 2026-07-25 12:29:21 +10:00
Conversation 0 Commits 1 Files Changed 1
+7 -1
unkinben commented 2026-07-25 00:27:37 +10:00
Owner
Copy Link
Copy Source

Why

During the k8s DNS cutover the dns module dual-writes every host's records two
ways: (a) LEGACY exported @@concat::fragments collected onto the VM puppet DNS
master into /var/named/sources/<zone>.conf, and (b) NEW nsupdate/RFC2136 to the
k8s bind-authoritative cluster (manage_nsupdate). Both were gated on so the
legacy path stayed as a safety net.

The legacy path is no longer in any client's resolution path:

  • #490 repointed profiles::dns::base::nameservers from the VM resolver
    anycast 198.18.19.16 to the k8s bind-resolvers LB 198.18.200.7. Every
    puppet client now resolves there.
  • The k8s bind-resolvers forward unkin.net, main.unkin.net, and all
    18.198.in-addr.arpa reverse zones to the k8s bind-authoritative cluster
    (198.18.200.6), which is populated purely by the nsupdate path — the same
    profiles::dns::record defines, manage_nsupdate branch.
  • The VM master's assembled /var/named/sources files are therefore orphaned:
    nothing resolves against them anymore.

This flips profiles::dns::updater::manage_export: false to stop nodes exporting
the legacy fragments. manage_nsupdate stays on.

What changes

  • hieradata/common.yaml: profiles::dns::updater::manage_export true -> false.

Verification evidence

Pulled all 498 exported Concat::Fragment resources from PuppetDB (239 A,
232 PTR, 27 CNAME; 490 unique fqdn/type/value) and dig'd each against the k8s
authoritative read endpoint 198.18.200.7 (the dynamic path's serving side):

Result Count
MATCH 447
MISSING-from-dynamic 38
DIFFERENT-value 5

Both non-match buckets are accounted for and are not coverage gaps:

  • 38 MISSING are all records the dynamic path deliberately filters via
    profiles::dns::updater::deny_ranges
    (198.18.200.0/24, 10.42.0.0/16, ...) plus reverse zones the authoritative
    cluster isn't even authoritative for: *-kube-lb0 (198.18.200.x),
    *-flannel.1 (10.42.x.x pod CIDR), and *-nomad/*-brcom1
    (172.26.64.x / 10.255.255.x). These are exactly the k8s/LB noise the deny
    ranges exist to keep out of the authoritative zones. Verified every one falls
    in a denied/non-authoritative range.
  • 5 DIFFERENT are stale legacy duplicates that the dynamic path holds more
    correctly:
    • 4x au-syd1-prod-halb.main.unkin.net A: four ceph nodes (2069-2073) each
      self-register the shared name with their own primary IP (haproxy::dns
      exports networking.ip per host, a legacy collision). Dynamic resolves it to
      the single VIP 198.18.24.60.
    • 1x dashboard.ceph.unkin.net CNAME: the ceph node self-publishes this to
      lb1.unkin.net. via its own nsupdate (per the k8s zones.yaml note); the
      exported VRRP-name fragment is stale.

Conclusion: the dynamic path fully covers the served record set, and the serving
path no longer depends on the exported files.

Collector-behavior note (what the flip actually does)

profiles::dns::zone realizes the fragments with an exported-resource collector
(Concat::Fragment <<| target == ... |>>). Turning off export purges the
fragments from PuppetDB, so on the VM master concat rebuilds each
/var/named/sources/<zone>.conf down to just the header (SOA+NS+glue) as nodes
converge. That is cosmetic: no client resolves against the VM master anymore
(#490), and the k8s authoritative zones are unaffected (fed by nsupdate, not
these files).

Rollback

Set profiles::dns::updater::manage_export: true again and let puppet converge;
nodes re-export their fragments and the VM-master zone files repopulate.

Follow-up (after soak)

Deleting the @@concat::fragment / Concat::Fragment <<||>> collector code in
profiles::dns::record / profiles::dns::zone (and the VM profiles::dns::master
zone assembly) is a separate cleanup PR once this has soaked and the VM master is
confirmed decommissionable.

## Why During the k8s DNS cutover the `dns` module dual-writes every host's records two ways: (a) LEGACY exported `@@concat::fragment`s collected onto the VM puppet DNS master into `/var/named/sources/<zone>.conf`, and (b) NEW nsupdate/RFC2136 to the k8s bind-authoritative cluster (`manage_nsupdate`). Both were gated on so the legacy path stayed as a safety net. The legacy path is no longer in any client's resolution path: - **#490** repointed `profiles::dns::base::nameservers` from the VM resolver anycast `198.18.19.16` to the k8s **bind-resolvers** LB `198.18.200.7`. Every puppet client now resolves there. - The k8s bind-resolvers forward `unkin.net`, `main.unkin.net`, and all `18.198.in-addr.arpa` reverse zones to the k8s **bind-authoritative** cluster (`198.18.200.6`), which is populated purely by the nsupdate path — the same `profiles::dns::record` defines, `manage_nsupdate` branch. - The VM master's assembled `/var/named/sources` files are therefore orphaned: nothing resolves against them anymore. This flips `profiles::dns::updater::manage_export: false` to stop nodes exporting the legacy fragments. `manage_nsupdate` stays on. ## What changes - `hieradata/common.yaml`: `profiles::dns::updater::manage_export` `true` -> `false`. ## Verification evidence Pulled all **498** exported `Concat::Fragment` resources from PuppetDB (239 A, 232 PTR, 27 CNAME; 490 unique fqdn/type/value) and `dig`'d each against the k8s authoritative read endpoint `198.18.200.7` (the dynamic path's serving side): | Result | Count | |---|---| | MATCH | 447 | | MISSING-from-dynamic | 38 | | DIFFERENT-value | 5 | Both non-match buckets are accounted for and are **not coverage gaps**: - **38 MISSING** are all records the dynamic path deliberately filters via `profiles::dns::updater::deny_ranges` (`198.18.200.0/24`, `10.42.0.0/16`, ...) plus reverse zones the authoritative cluster isn't even authoritative for: `*-kube-lb0` (198.18.200.x), `*-flannel.1` (10.42.x.x pod CIDR), and `*-nomad`/`*-brcom1` (172.26.64.x / 10.255.255.x). These are exactly the k8s/LB noise the deny ranges exist to keep out of the authoritative zones. Verified every one falls in a denied/non-authoritative range. - **5 DIFFERENT** are stale legacy duplicates that the dynamic path holds *more* correctly: - 4x `au-syd1-prod-halb.main.unkin.net` A: four ceph nodes (2069-2073) each self-register the shared name with their own primary IP (haproxy::dns exports `networking.ip` per host, a legacy collision). Dynamic resolves it to the single VIP `198.18.24.60`. - 1x `dashboard.ceph.unkin.net` CNAME: the ceph node self-publishes this to `lb1.unkin.net.` via its own nsupdate (per the k8s zones.yaml note); the exported VRRP-name fragment is stale. Conclusion: the dynamic path fully covers the served record set, and the serving path no longer depends on the exported files. ## Collector-behavior note (what the flip actually does) `profiles::dns::zone` realizes the fragments with an exported-resource collector (`Concat::Fragment <<| target == ... |>>`). Turning off export purges the fragments from PuppetDB, so on the VM master concat rebuilds each `/var/named/sources/<zone>.conf` down to just the header (SOA+NS+glue) as nodes converge. That is **cosmetic**: no client resolves against the VM master anymore (#490), and the k8s authoritative zones are unaffected (fed by nsupdate, not these files). ## Rollback Set `profiles::dns::updater::manage_export: true` again and let puppet converge; nodes re-export their fragments and the VM-master zone files repopulate. ## Follow-up (after soak) Deleting the `@@concat::fragment` / `Concat::Fragment <<||>>` collector code in `profiles::dns::record` / `profiles::dns::zone` (and the VM `profiles::dns::master` zone assembly) is a separate cleanup PR once this has soaked and the VM master is confirmed decommissionable.
unkinben added 1 commit 2026-07-25 00:27:37 +10:00
dns: stop exporting legacy DNS fragments (manage_export false)
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
c104360c1a 
Clients now resolve via the k8s bind (198.18.200.7) which serves the
authoritative zones fed by the nsupdate path, so the legacy
exported-resource -> VM-master /var/named/sources files are no longer in
the serving path. Turn off the export half of the dual-write; the
nsupdate (manage_nsupdate) half stays on.
benvin merged commit 82a5b2ee1a into develop 2026-07-25 12:29:21 +10:00
benvin deleted branch benvin/dns-export-off 2026-07-25 12:29:22 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/puppet-prod#495
Delete Branch "benvin/dns-export-off"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?