unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Revert external-dns writes back to the legacy VM authoritative (#287) #302

Closed
unkinben wants to merge 1 commits from benvin/externaldns-fix into main
pull from: benvin/externaldns-fix
merge into: unkin:main
Conversation 1 Commits 1 Files Changed 1
+4 -6
unkinben commented 2026-07-28 22:11:10 +10:00
Owner
Copy Link
Copy Source

Why

New forward records (netbox, logs-ingest.k8s.syd1.au.unkin.net) stopped resolving after #287. Root cause is a half-completed DNS migration, not an external-dns fault:

  • #287 (step 2/3) repointed external-dns RFC2136 updates from the legacy VM ausyd1nxvm2127.main.unkin.net to the in-cluster bind-externaldns-primary.
  • external-dns is writing correctly: the in-cluster bind (dig @198.18.200.8, and bind-resolvers 198.18.200.7) HAS netbox/logs-ingest A records and their external-dns TXT ownership records. Its ApplyChanges shows Create: 0 because the records already exist there.
  • But the estate's client-facing resolvers (e.g. 198.18.2.160) still source the zone from the legacy VM authoritative, which is alive but now frozen — it keeps old names (identity, argocd, grafana still resolve) and never receives the new writes. Step 3 (cutting resolver/delegation reads over to the in-cluster bind) was never done, so writes moved ahead of reads.

Reverting restores external-dns writes to the legacy authoritative that clients actually read, immediately unblocking new-record publication. This is exactly the rollback documented in #287 ("The legacy VM is untouched and still authoritative"). The in-cluster cutover should be re-attempted only after step 3 (resolver/delegation move) lands.

Changes

  • --rfc2136-host back to ausyd1nxvm2127.main.unkin.net.
  • TSIG secret ref back to Vault-backed externaldns-tsig (verified still present in the namespace, 183d old).

Verification (after merge + external-dns re-sync)

dig +short netbox.k8s.syd1.au.unkin.net       # expect 198.18.200.4
dig +short logs-ingest.k8s.syd1.au.unkin.net  # expect 198.18.200.4

Note

The per-cycle PTR add/remove thrash on 198.18.200.4 and the Couldn't parse ... as an IP address debug lines are a separate, cosmetic external-dns rfc2136 multi-target-PTR quirk; they are NOT the cause of the missing A records and are unaffected by this change.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why New forward records (`netbox`, `logs-ingest`.k8s.syd1.au.unkin.net) stopped resolving after #287. Root cause is a **half-completed DNS migration, not an external-dns fault**: - #287 (step 2/3) repointed external-dns RFC2136 updates from the legacy VM `ausyd1nxvm2127.main.unkin.net` to the in-cluster `bind-externaldns-primary`. - external-dns is writing **correctly**: the in-cluster bind (`dig @198.18.200.8`, and bind-resolvers `198.18.200.7`) HAS `netbox`/`logs-ingest` A records **and** their external-dns TXT ownership records. Its `ApplyChanges` shows `Create: 0` because the records already exist there. - But the estate's client-facing resolvers (e.g. `198.18.2.160`) still source the zone from the **legacy VM authoritative**, which is alive but now **frozen** — it keeps old names (identity, argocd, grafana still resolve) and never receives the new writes. Step 3 (cutting resolver/delegation reads over to the in-cluster bind) was never done, so writes moved ahead of reads. Reverting restores external-dns writes to the legacy authoritative that clients actually read, immediately unblocking new-record publication. This is exactly the rollback documented in #287 ("The legacy VM is untouched and still authoritative"). The in-cluster cutover should be re-attempted only after step 3 (resolver/delegation move) lands. ## Changes - `--rfc2136-host` back to `ausyd1nxvm2127.main.unkin.net`. - TSIG secret ref back to Vault-backed `externaldns-tsig` (verified still present in the namespace, 183d old). ## Verification (after merge + external-dns re-sync) ``` dig +short netbox.k8s.syd1.au.unkin.net # expect 198.18.200.4 dig +short logs-ingest.k8s.syd1.au.unkin.net # expect 198.18.200.4 ``` ## Note The per-cycle PTR add/remove thrash on `198.18.200.4` and the `Couldn't parse ... as an IP address` debug lines are a separate, cosmetic external-dns rfc2136 multi-target-PTR quirk; they are NOT the cause of the missing A records and are unaffected by this change. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-28 22:11:11 +10:00
Revert external-dns writes back to the legacy VM authoritative (#287)
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
0ec308b76e 
## Why

New forward records (netbox, logs-ingest .k8s.syd1.au.unkin.net) stopped
appearing in DNS after #287. Root cause is a half-completed migration, not
an external-dns fault:

- #287 (step 2/3) repointed external-dns RFC2136 updates from the legacy VM
  `ausyd1nxvm2127.main.unkin.net` to the in-cluster `bind-externaldns-primary`.
- external-dns is writing correctly: the in-cluster bind (198.18.200.8 /
  bind-resolvers 198.18.200.7) HAS netbox + logs-ingest A records and their
  external-dns TXT ownership records (SOA serial 5).
- But the estate's client-facing resolvers (e.g. 198.18.2.160) still source
  the zone from the legacy VM authoritative, which is alive but now FROZEN:
  it keeps old names (identity, argocd, grafana resolve fine) and never
  receives the new writes. Step 3 (cut resolver/delegation reads over to the
  in-cluster bind) was never done, so writes moved ahead of reads.

Reverting restores external-dns writes to the legacy authoritative that
clients actually read, immediately unblocking new-record publication. This
is exactly the rollback path documented in #287 ("The legacy VM is untouched
and still authoritative"). Re-attempt the cutover only after step 3 lands.

## Changes

- `--rfc2136-host` back to `ausyd1nxvm2127.main.unkin.net`.
- TSIG secret ref back to Vault-backed `externaldns-tsig` (still present in
  the namespace).

## Note

The per-cycle PTR add/remove thrash on 198.18.200.4 and the "Couldn't parse
... as an IP address" debug lines are a separate, cosmetic external-dns
rfc2136 multi-target-PTR quirk; they are NOT the cause of the missing A
records and are unaffected by this change.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben commented 2026-07-28 22:22:00 +10:00
Author
Owner
Copy Link
Copy Source

Superseded: the resolver upstream for k8s.syd1.au.unkin.net now points at the in-cluster bind (198.18.200.7), which is where external-dns already writes — reads and writes are aligned without the revert. Verified netbox/logs-ingest/identity/argocd all resolve to 198.18.200.4 on the client path. Merging this revert NOW would recreate the read/write split in the opposite direction, so closing unmerged. Remaining follow-ups tracked separately: retire/freeze the legacy VM zone copy, and the SOA localhost shadow on some resolvers.

Superseded: the resolver upstream for k8s.syd1.au.unkin.net now points at the in-cluster bind (198.18.200.7), which is where external-dns already writes — reads and writes are aligned without the revert. Verified netbox/logs-ingest/identity/argocd all resolve to 198.18.200.4 on the client path. Merging this revert NOW would recreate the read/write split in the opposite direction, so closing unmerged. Remaining follow-ups tracked separately: retire/freeze the legacy VM zone copy, and the SOA localhost shadow on some resolvers.
unkinben closed this pull request 2026-07-28 22:22:00 +10:00
All checks were successful
ci/woodpecker/pr/vector-test Pipeline was successful
Required
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Required
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#302
Delete Branch "benvin/externaldns-fix"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?