Fix authoritative secondary replication (TSIG transfer + stable primary) #9

Merged
benvin merged 1 commits from benvin/fix-secondary-replication into main 2026-07-12 21:33:39 +10:00
Owner

Why

Replication to the authoritative secondaries never worked — the secondaries carried only 2 zones vs the master's 20, so the .6 read Service has been answering purely from the master (pod-0 is in its selector). Diagnosed on-cluster: the master REFUSES the catalog AXFR.

Three bugs in the transfer path:

  1. Secondary presents no TSIG key. The master's catalog zone is allow-transfer { key "transfer-key"; }, but the rendered secondary config transfers unkeyed (default-primaries { <ip>; }) → REFUSED → no catalog → no member zones.
  2. Member zones have no allow-transfer. e.g. main.unkin.net only had allow-update, so even once the catalog transfers, members couldn't be pulled.
  3. Secondaries point at the primary pod IP, which changes on every primary restart, leaving them aimed at a dead address (this is what first surfaced the problem).

Changes

  • Render the catalog transfer key into the secondary catalog-zones default-primaries and the secondary catalog zone primaries (<ip> key "transfer-key"), so key-authenticated AXFR is accepted.
  • Add allow-transfer { key "<transfer-key>"; } to catalog member primary zones when no explicit allow-transfer is set; applied to existing zones via modzone on reconcile.
  • Point secondaries at the stable primary Service ClusterIP instead of the pod IP, so replication survives primary pod restarts (falls back to pod IP when no primary Service exists).

Tests

  • New TestRenderCatalogPrimariesCarryTransferKey asserts the key appears in both secondary catalog clauses; existing catalog/keyless tests still pass. go build/vet/test clean.

Deploy note

Needs a version bump + redeploy; folding the image/CRD bump into the in-flight argocd PR. Once the new operator reconciles, existing member zones get allow-transfer via modzone and secondaries re-point at the ClusterIP with the key, restoring replication.

## Why Replication to the authoritative secondaries **never worked** — the secondaries carried only 2 zones vs the master's 20, so the `.6` read Service has been answering purely from the master (pod-0 is in its selector). Diagnosed on-cluster: the master REFUSES the catalog AXFR. Three bugs in the transfer path: 1. **Secondary presents no TSIG key.** The master's catalog zone is `allow-transfer { key "transfer-key"; }`, but the rendered secondary config transfers unkeyed (`default-primaries { <ip>; }`) → **REFUSED** → no catalog → no member zones. 2. **Member zones have no `allow-transfer`.** e.g. `main.unkin.net` only had `allow-update`, so even once the catalog transfers, members couldn't be pulled. 3. **Secondaries point at the primary pod IP**, which changes on every primary restart, leaving them aimed at a dead address (this is what first surfaced the problem). ## Changes - Render the catalog transfer key into the secondary `catalog-zones` default-primaries and the secondary catalog zone `primaries` (`<ip> key "transfer-key"`), so key-authenticated AXFR is accepted. - Add `allow-transfer { key "<transfer-key>"; }` to catalog **member** primary zones when no explicit `allow-transfer` is set; applied to existing zones via `modzone` on reconcile. - Point secondaries at the **stable primary Service ClusterIP** instead of the pod IP, so replication survives primary pod restarts (falls back to pod IP when no primary Service exists). ## Tests - New `TestRenderCatalogPrimariesCarryTransferKey` asserts the key appears in both secondary catalog clauses; existing catalog/keyless tests still pass. `go build/vet/test` clean. ## Deploy note Needs a version bump + redeploy; folding the image/CRD bump into the in-flight argocd PR. Once the new operator reconciles, existing member zones get `allow-transfer` via `modzone` and secondaries re-point at the ClusterIP with the key, restoring replication.
unkinben added 1 commit 2026-07-12 19:42:57 +10:00
Fix authoritative secondary replication (TSIG transfer + stable primary)
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
ea330bd767
Secondaries never replicated any member zone: the master's catalog zone
requires key-authenticated AXFR (allow-transfer { key "transfer-key"; }),
but the rendered secondary config transferred without presenting the key,
so every catalog transfer was REFUSED and no member zones provisioned.
Two further gaps compounded it: member zones had no allow-transfer at all,
and secondaries pointed at the primary's pod IP, which dies on restart.

- Render the catalog transfer key into the secondary catalog-zones
  default-primaries and the secondary catalog zone primaries, so
  key-authenticated AXFR from the primary is accepted.
- Add allow-transfer { key "<transfer-key>"; } to catalog member primary
  zones (when the zone does not set an explicit allow-transfer), so
  secondaries can pull them; applied to existing zones via modzone.
- Point secondaries at the stable primary Service ClusterIP instead of the
  primary pod IP, so replication survives primary pod restarts (falls back
  to the pod IP when no primary Service exists).
benvin merged commit ef8c41cb0f into main 2026-07-12 21:33:39 +10:00
benvin deleted branch benvin/fix-secondary-replication 2026-07-12 21:33:39 +10:00
Sign in to join this conversation.