Zone fails to load permanently when a skeleton zone file is written over an existing journal #19

New Issue

2026-09-19T22:21:47+10:00

unkin-agent commented

2026-09-19 22:21:47 +10:00

Problem

A zone fails to load permanently when the operator writes a fresh skeleton zone file at serial 1 over a zone whose existing journal (.jnl) is still on disk at a higher serial. BIND refuses the inconsistent file/journal pair:

addzone failed: out of range
dns_zone_loadnew failed: out of range

Observed in production on bind-externaldns-0: db.k8s.syd1.au.unkin.net rewritten to 249 bytes at serial 1, while db.k8s.syd1.au.unkin.net.jnl (7777 bytes, dated Aug 30, serial ~16) remained on the PVC. db.200.18.198.in-addr.arpa broke identically, so this affects the forward and reverse paths alike.

The failure is persistent and self-reinforcing:

Each replica has its own PVC, so the stale journal survives a pod restart — restarting does not recover the zone.
The operator retries addzone every reconcile and rewrites the skeleton each time, so it never converges.
BindZone k8s-syd1-au-unkin-net sat in phase Error / AddZoneFailed.

Blast radius

The bind-externaldns Service round-robins three pods, so roughly 1 in 3 authoritative answers became SERVFAIL. bind-resolvers cached those failures and resolution for the whole k8s.syd1.au.unkin.net zone collapsed estate-wide — measured 0/20 success via resolver 198.18.200.7, against 30/30 for a control name outside the zone.

The visible symptom was not a DNS alarm. Gitea could not resolve ci.k8s.syd1.au.unkin.net, so webhook deliveries to Woodpecker failed silently — no pipeline, no commit status, no log line. Three pushes/PRs across terraform-artifactapi, encapic and jellyfin-plugin-sso were dropped and did not replay. It presented as "CI intermittently stopped firing".

Remediation applied

Deleting the two stale .jnl files was sufficient; the operator's next reconcile ran addzone successfully and both zones returned to Ready. A subsequent pod restart came back clean, confirming the file/journal pair is consistent again.

Proposal

Make zone-file writes safe when a journal already exists. Decide the mechanism against the real code — candidates:

Remove or quarantine the stale .jnl whenever the zone file is rewritten; a journal is only valid against the file it was derived from.
Do not rewrite an existing zone file with a skeleton at all — leave it and reconcile via dynamic update.
Read the existing serial (from zone file or journal) and write a serial strictly greater, rather than resetting to 1.
Detect the out of range load failure and self-heal rather than retrying the same write.

Requirements:

Fix in the shared path so forward and in-addr.arpa zones are both covered.
Safe on a fresh install with no zone file and no journal present.
Must not destroy live zone data on healthy replicas.
Reconcile must be idempotent — repeated reconciles must not reset or corrupt the zone.

Acceptance

A zone whose file is rewritten while a higher-serial journal exists loads successfully, with a serial not lower than the journal's.
Fresh install with nothing on disk still works.
Running reconcile twice leaves the zone intact and Ready.
Unit tests cover all three, and fail against the unfixed code.

Separate bug found during the same investigation, worth its own issue: the zone's ns1 glue is wrong. It currently serves A 10.42.2.159, a dead pod IP, and the skeleton writes the replica's own pod IP; it should be the LB VIP 198.18.200.8. Masked today only because bind-resolvers uses a forward zone.

## Problem A zone fails to load permanently when the operator writes a fresh skeleton zone file at **serial 1** over a zone whose existing journal (`.jnl`) is still on disk at a higher serial. BIND refuses the inconsistent file/journal pair: ``` addzone failed: out of range dns_zone_loadnew failed: out of range ``` Observed in production on `bind-externaldns-0`: `db.k8s.syd1.au.unkin.net` rewritten to 249 bytes at serial 1, while `db.k8s.syd1.au.unkin.net.jnl` (7777 bytes, dated Aug 30, serial ~16) remained on the PVC. `db.200.18.198.in-addr.arpa` broke identically, so this affects the forward and reverse paths alike. The failure is persistent and self-reinforcing: - Each replica has its own PVC, so the stale journal survives a pod restart — restarting does not recover the zone. - The operator retries `addzone` every reconcile and rewrites the skeleton each time, so it never converges. - `BindZone k8s-syd1-au-unkin-net` sat in phase `Error` / `AddZoneFailed`. ### Blast radius The `bind-externaldns` Service round-robins three pods, so roughly 1 in 3 authoritative answers became SERVFAIL. `bind-resolvers` cached those failures and resolution for the whole `k8s.syd1.au.unkin.net` zone collapsed estate-wide — measured 0/20 success via resolver `198.18.200.7`, against 30/30 for a control name outside the zone. The visible symptom was not a DNS alarm. Gitea could not resolve `ci.k8s.syd1.au.unkin.net`, so webhook deliveries to Woodpecker failed silently — no pipeline, no commit status, no log line. Three pushes/PRs across `terraform-artifactapi`, `encapic` and `jellyfin-plugin-sso` were dropped and did not replay. It presented as "CI intermittently stopped firing". ### Remediation applied Deleting the two stale `.jnl` files was sufficient; the operator's next reconcile ran `addzone` successfully and both zones returned to `Ready`. A subsequent pod restart came back clean, confirming the file/journal pair is consistent again. ## Proposal Make zone-file writes safe when a journal already exists. Decide the mechanism against the real code — candidates: - Remove or quarantine the stale `.jnl` whenever the zone file is rewritten; a journal is only valid against the file it was derived from. - Do not rewrite an existing zone file with a skeleton at all — leave it and reconcile via dynamic update. - Read the existing serial (from zone file or journal) and write a serial strictly greater, rather than resetting to 1. - Detect the `out of range` load failure and self-heal rather than retrying the same write. Requirements: - Fix in the shared path so forward and `in-addr.arpa` zones are both covered. - Safe on a fresh install with no zone file and no journal present. - Must not destroy live zone data on healthy replicas. - Reconcile must be idempotent — repeated reconciles must not reset or corrupt the zone. ## Acceptance - A zone whose file is rewritten while a higher-serial journal exists loads successfully, with a serial not lower than the journal's. - Fresh install with nothing on disk still works. - Running reconcile twice leaves the zone intact and `Ready`. - Unit tests cover all three, and fail against the unfixed code. ## Related Separate bug found during the same investigation, worth its own issue: the zone's `ns1` glue is wrong. It currently serves `A 10.42.2.159`, a dead pod IP, and the skeleton writes the replica's own pod IP; it should be the LB VIP `198.18.200.8`. Masked today only because `bind-resolvers` uses a forward zone.

unkin-agent referenced this issue from a commit

2026-09-19 22:33:50 +10:00

Inspect zone file and journal before seeding a zone

~~unkin-agent referenced this issue 2026-09-19 22:33:57 +10:00~~

Inspect zone file and journal before seeding a zone #20

benvin closed this issue

2026-09-19 23:42:04 +10:00

Sign in to join this conversation.