From 0ec308b76ef4de86255dcebeb5fae474c8ab55ae Mon Sep 17 00:00:00 2001 From: Ben Vincent Date: Tue, 28 Jul 2026 22:10:44 +1000 Subject: [PATCH] Revert external-dns writes back to the legacy VM authoritative (#287) ## Why New forward records (netbox, logs-ingest .k8s.syd1.au.unkin.net) stopped appearing in DNS after #287. Root cause is a half-completed migration, not an external-dns fault: - #287 (step 2/3) repointed external-dns RFC2136 updates from the legacy VM `ausyd1nxvm2127.main.unkin.net` to the in-cluster `bind-externaldns-primary`. - external-dns is writing correctly: the in-cluster bind (198.18.200.8 / bind-resolvers 198.18.200.7) HAS netbox + logs-ingest A records and their external-dns TXT ownership records (SOA serial 5). - But the estate's client-facing resolvers (e.g. 198.18.2.160) still source the zone from the legacy VM authoritative, which is alive but now FROZEN: it keeps old names (identity, argocd, grafana resolve fine) and never receives the new writes. Step 3 (cut resolver/delegation reads over to the in-cluster bind) was never done, so writes moved ahead of reads. Reverting restores external-dns writes to the legacy authoritative that clients actually read, immediately unblocking new-record publication. This is exactly the rollback path documented in #287 ("The legacy VM is untouched and still authoritative"). Re-attempt the cutover only after step 3 lands. ## Changes - `--rfc2136-host` back to `ausyd1nxvm2127.main.unkin.net`. - TSIG secret ref back to Vault-backed `externaldns-tsig` (still present in the namespace). ## Note The per-cycle PTR add/remove thrash on 198.18.200.4 and the "Couldn't parse ... as an IP address" debug lines are a separate, cosmetic external-dns rfc2136 multi-target-PTR quirk; they are NOT the cause of the missing A records and are unaffected by this change. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --- apps/overlays/au-syd1/externaldns/values.yaml | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/apps/overlays/au-syd1/externaldns/values.yaml b/apps/overlays/au-syd1/externaldns/values.yaml index 7205050..3bc5e48 100644 --- a/apps/overlays/au-syd1/externaldns/values.yaml +++ b/apps/overlays/au-syd1/externaldns/values.yaml @@ -27,24 +27,22 @@ sources: - gateway-httproute - gateway-grpcroute -# TSIG secret + algorithm. The bind operator generates this key in -# bind-internal (BindTSIGKey externaldns-key) and the emberstack reflector -# mirrors the Secret into this namespace as externaldns-key-tsig. +# Environment variables for TSIG secret and algorithm from Vault env: - name: EXTERNAL_DNS_RFC2136_TSIG_SECRET valueFrom: secretKeyRef: - name: externaldns-key-tsig + name: externaldns-tsig key: secret - name: EXTERNAL_DNS_RFC2136_TSIG_ALGORITHM valueFrom: secretKeyRef: - name: externaldns-key-tsig + name: externaldns-tsig key: algorithm # RFC2136 configuration as arguments extraArgs: - - --rfc2136-host=bind-externaldns-primary.bind-internal.svc.cluster.local + - --rfc2136-host=ausyd1nxvm2127.main.unkin.net - --rfc2136-port=53 - --rfc2136-zone=k8s.syd1.au.unkin.net - --rfc2136-zone=200.18.198.in-addr.arpa