bind-operator v0.4.0 renders a forward BindZone without forwarders as `forwarders { };`, so the RBL direct-resolution zones no longer need raw zone clauses in the openforwarder view extraOptions.
- add rbl-zones.yaml with one forwarder-less forward BindZone per RBL domain
- remove the raw RBL zone lines from the openforwarder view
Reviewed-on: #569
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Recursing public names in-cluster SERVFAILs a few percent of cold lookups and loses the cache on every restart, which breaks CI. DNSBLs refuse or poison queries relayed via public resolvers, so rspamd's RBL lookups must keep resolving directly.
- add global forwarders 1.1.1.1 and 8.8.8.8 (forward first) to bind-resolvers
- add empty-forwarder zones to the openforwarder view for the RBL domains rspamd queries, so they resolve iteratively
Reviewed-on: #555
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The second mail edge site moves to BinaryLane Brisbane (VPC 172.18.32.0/24), whose hosts resolve through the internal resolvers over the vpn.
- add 172.18.32.0/24 to acl-dmz
Reviewed-on: #554
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The in-cluster BIND resolvers have no IPv6 route but still try every AAAA nameserver. Each attempt fails with "network unreachable" yet counts toward the per-query recursion budget, so cold lookups of names with many dual-stack NS (registry.opentofu.org, cloudfront, crates.io, gcr.io) hit `quota reached` and SERVFAIL, surfacing as transient DNS errors in CI.
- mark all IPv6 servers bogus in the openforwarder view so recursion is IPv4-only
Reviewed-on: #553
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
BinaryLane only accepts RFC 1918 VPC ranges, so the syd and mel mail DMZ VPCs move from 198.18.30.0/24 and 198.18.31.0/24 to 172.18.30.0/24 and 172.18.31.0/24.
- replace the 198.18.30-31.0/24 entries in acl-dmz with 172.18.30.0/24 and 172.18.31.0/24
Reviewed-on: #551
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The mail edge VMs in the BinaryLane DMZ VPCs query the internal resolvers over the VPN, but the resolver match-clients ACLs do not include their ranges.
- add 198.18.30.0/24 (syd) and 198.18.31.0/24 (mel) to `acl-dmz`
Reviewed-on: #545
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
DNSBLs such as Spamhaus refuse queries that arrive via public resolvers, and forwarding everything to 8.8.8.8/1.1.1.1 leaves the estate dependent on them. The lab resolvers now do full recursion themselves (DNSSEC-validated via the operator default `dnssec-validation auto` and BIND built-in root hints).
- drop the global forwarders from the bind-resolvers BindCluster
- keep the per-zone forwarders for unkin.net, consul, k8s and reverse zones unchanged
Reviewed-on: #546
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. **Stacks on #318 — merge after it.**
How:
- 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: **bind_query** (k8s bind-* + VM named), **rancher_audit** (cattle-system sidecar JSON), **cnpg_pg** (ONE transform for all 10 CNPG clusters via the `.postgres` container), **gitea** (router+access, k8s+VM), **puppet** (openvoxserver/openvoxdb logback + access), **litellm** (JSON request logs), **postfix** (per-line maillog).
- Carve `.postgres` out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact.
- Companion k8s flips in this PR: litellm `JSON_LOGS=True`; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on.
- 15 new `vector test` cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing `fields Map(String,String)` — no DDL change.
Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with `.file`/`.SYSLOG_IDENTIFIER` tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #320
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 3 of 3 in the external-dns → in-cluster bind migration: the client-visible cutover. The `openforwarder` resolvers currently forward `k8s.syd1.au.unkin.net` to the legacy VM anycast `198.18.19.20` (a temporary measure — commit 7ee5dfb) and have NO forwarder at all for the reverse zone `200.18.198.in-addr.arpa`. Once external-dns publishes to the in-cluster `bind-externaldns` (PR 2), resolvers must read from it.
## Changes
- Repoints the `fwd-k8s-syd1-au-unkin-net` forwarder from `198.18.19.20` (legacy VM) to `198.18.200.8` (in-cluster `bind-externaldns` VIP).
- Adds `fwd-200-18-198-in-addr-arpa` forwarding `200.18.198.in-addr.arpa` → `198.18.200.8`, closing the reverse-zone gap so PTR lookups for the k8s LB range keep resolving after cutover. Modeled exactly on the existing forward-zone entries.
- Refreshes the header comment to describe the in-cluster upstream.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean; both zones render with forwarder `198.18.200.8` and there is no residual `198.18.19.20`.
## Merge gate
- PR 2 (`benvin/externaldns-incluster`) merged, AND
- record parity confirmed between legacy and in-cluster for the forward zone. Spot-check (repeat for each name):
```
for n in puppetca puppet puppetdb encapi pdbmux artifactapi consul; do
echo "$n:"
dig +short @198.18.19.20 A $n.k8s.syd1.au.unkin.net
dig +short @198.18.200.8 A $n.k8s.syd1.au.unkin.net
done
# plus 2-3 PTRs in the reverse zone:
dig +short @198.18.19.20 -x 198.18.200.8
dig +short @198.18.200.8 -x 198.18.200.8
```
A/PTR answers from `198.18.200.8` must match those from `198.18.19.20` before merging.
## Verification (after merge)
```
dig +short @198.18.200.7 A puppet.k8s.syd1.au.unkin.net # resolvers VIP
dig +short @198.18.200.7 -x 198.18.200.8 # reverse via resolvers
```
Resolution through the `bind-resolvers` VIP should now answer for both the forward and reverse k8s zones.
## Rollback
Revert this PR — the `fwd-k8s-syd1-au-unkin-net` forwarder returns to `198.18.19.20` and the reverse forward is removed. The legacy VM is untouched and remains authoritative until decommission.
Reviewed-on: #288
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Kubernetes nodes querying the bind-resolvers LoadBalancer VIP (198.18.200.7) get REFUSED (EDE 18 Prohibited).
The service is `externalTrafficPolicy: Local`, which preserves the client source IP for traffic entering the cluster from outside — but a node querying the VIP never leaves via OSPF. Its own kube-proxy DNATs the LB IP in the OUTPUT chain and masquerades the source to a cluster-internal address (the node's flannel.1, e.g. 10.42.x.x). That address is not in `acl-main.unkin.net`, so the openforwarder view's match-clients rejects the query.
External clients preserve their real source IP and match acl-main, which is why only in-cluster hosts were affected.
Add `10.42.0.0/16` to `acl-main.unkin.net` so node-originated (masqueraded) resolver queries are permitted. This mirrors the authoritative cluster, which already allows the pod network (`allow-query { ...; 10.42.0.0/16; }`).
Reviewed-on: #271
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The bind-resolvers `openforwarder` view forwarded `k8s.syd1.au.unkin.net` to the in-cluster bind-externaldns (198.18.200.8), which is not reliably serving those records yet, so lookups return NXDOMAIN.
Concrete impact: Gitea cannot resolve the k8s-hosted CI host, so its outbound webhook fails and tagged releases never trigger CI (e.g. cutting a new bind-operator release from a tag).
This points the `fwd-k8s-syd1-au-unkin-net` forwarder at the existing external external-dns bind service anycast **198.18.19.20** (puppet `roles::infra::dns::externaldns` — master `ausyd1nxvm2127` + slaves `2128`/`2129`, advertised via OSPF), which still holds the working `k8s.syd1.au.unkin.net` records. It is in the same 198.18.19.0/24 anycast family as the consul forwarder (198.18.19.14) the resolvers already use, so it is reachable from the pods.
Temporary measure. Revert to 198.18.200.8 once external-dns publishes to the in-cluster bind-externaldns service. Only the forward target changes; no CRD/schema changes.
Reviewed-on: #269
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to #258 (which added workstation+router to the *authoritative* ACL). The **resolver** (bind-resolvers, 198.18.200.7) has its own `acl-main.unkin.net` gating its `openforwarder` view; the workstation is not in it, so recursive queries return REFUSED. This lets the workstation use the resolver as its normal nameserver.
## Changes
- Add `10.10.12.200/32` (workstation, wireguard) to resolver `acl-main.unkin.net`
- Add `198.18.21.160/32` (router) explicitly for documentation; already covered by existing `198.18.21.160/27` and `198.18.21.0/24` (no functional change for the router)
Reviewed-on: #259
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why — urgent
PR #251 added `validate-except` to the resolver `BindCluster` but the list omitted the trailing semicolon after the final entry. `named` requires **every** entry in a list to be semicolon-terminated, including the last before the closing brace, so it fails config parse and the resolver pods crash-loop:
```
/run/named/named.conf:18: missing ';' before '}'
loading configuration: failure
exiting (due to fatal error)
```
The resolvers (`.7`) are down until this lands; the authoritative (`.6`/`.9`) and externaldns (`.8`) are unaffected.
## Fix
```diff
- validate-except { unkin.net; 18.198.in-addr.arpa; consul }
+ validate-except { unkin.net; 18.198.in-addr.arpa; consul; }
```
Renders to `validate-except { unkin.net; 18.198.in-addr.arpa; consul; };` — valid.
## Recovery
On merge + ArgoCD sync, the operator re-renders the ConfigMap with valid config and the crash-looping pods self-heal on their next restart (no manual `rollout restart` needed). Validated: `bind-internal` renders and pre-commit clean.
Reviewed-on: #252
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Resolving any `unkin.net` record through the resolver (`.7`) returns **SERVFAIL**, while the authoritative (`.6`) answers fine. Confirmed from the resolver's querylog:
```
view openforwarder: validating unkin.net/SOA: got insecure response; parent indicates it should be secure
broken trust chain resolving 'ausyd1nxvm2120.main.unkin.net/A/IN': 198.18.200.6#53
query failed (broken trust chain)
```
The resolver runs `dnssec-validation auto`. The public `unkin.net` is DNSSEC-signed (the `.net` parent publishes a DS), but the in-cluster split-horizon authoritative serves `unkin.net` **unsigned**. The validator sees "parent says secure" + an insecure answer → treats it as spoofing → SERVFAIL. The authoritative works directly because it does no validation.
## Fix
Add `validate-except` (via `spec.extraOptions`) for the forwarded internal domains, so the resolver treats them as insecure and skips validation:
```
validate-except { unkin.net; 18.198.in-addr.arpa; consul }
```
- `unkin.net` covers all `*.unkin.net` (incl. `main.unkin.net`, `k8s.syd1.au.unkin.net`)
- `18.198.in-addr.arpa` covers every `NN.18.198.in-addr.arpa` reverse zone (subtree)
- `consul` covers the consul TLD
This also makes internal resolution independent of Internet egress (no DNSSEC chain-walk needed). External-name validation is unchanged. No operator change required.
## Validation
`bind-internal` renders and passes `kubeconform` (56/56); pre-commit clean.
## Activation
After merge + operator reconcile, the resolver ConfigMap re-renders; the running pods hold a startup snapshot, so they need a reload: `kubectl -n bind-internal rollout restart statefulset/bind-resolvers`.
Reviewed-on: #251
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**Stacked on #228** (needs operator v0.1.5). Merge #228 first; the diff collapses to just this after.
## Why
Writes (RFC2136/nsupdate) must go to pod-0 — the round-robin read Service would land them on a secondary (rejected). Adds a dedicated write endpoint per cluster (operator v0.1.5 `primaryService`).
## Changes
- `bind-authoritative`: LoadBalancer write endpoint on **198.18.200.9** (`bind-authoritative-primary`)
- `bind-externaldns`: ClusterIP write endpoint (`bind-externaldns-primary`, for in-cluster writers)
- regenerate the bindcluster kubeconform schema (primaryService + externalTrafficPolicy)
## Deferred
external-dns is **not** repointed at `bind-externaldns-primary` yet: it authenticates with the existing TSIG key, which the operator-generated key won't match until the planned Vault-sync + secret-reflection features exist. Until then external-dns keeps writing to the puppet externaldns.
## Validated
kustomize build + kubeconform (3 BindClusters valid against the v0.1.5 schema).
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #229
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`dig google.com @198.18.200.7` was refused: the resolver never set allow-recursion, so BIND defaulted to localnets/localhost. This mirrors the puppet resolver (/etc/named/views.conf + acls.conf) exactly.
## Changes
- `openforwarder` BindView: `match-clients` = the 4 internal ACLs, recursion yes, allow-recursion/allow-query `any` (match-clients gates)
- 4 BindACLs from puppet acls.conf (acl-main.unkin.net/acl-dmz/acl-common/acl-nomad-jobs)
- 26 conditional forward zones in the view (unkin→198.18.19.15, consul→.14, k8s→.20, dmz/network/prod + 10.10.x reverse → 10.10.16.32/33)
- global forwarders 8.8.8.8/1.1.1.1
- operator image → v0.1.4
## Note
Forward-zone upstreams point at the **puppet anycast** servers (still authoritative during migration); flip to the in-cluster authoritative/externaldns LBs once zone data is migrated.
## Validated
kustomize build (59 docs), kubeconform clean.
Reviewed-on: #226
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**HOLD until v0.1.3 is tagged/built** (operator #4 merged + tagged) — this PR bumps the operator to v0.1.3, whose CRD adds the `clusterRef` field these keys use.
## Why
Put all BIND DNS services in one `bind-internal` namespace and name the StatefulSets clearly.
## Changes
- 3 clusters consolidated into `bind-internal`, StatefulSets renamed **bind-authoritative** / **bind-resolvers** / **bind-externaldns**; LBs kept on 198.18.200.6/.7/.8; external-dns hostnames renamed to match
- `clusterRef` added to `transfer-key` (→ bind-authoritative) and `externaldns-key` (→ bind-externaldns) so keys are scoped per cluster
- removed the old `ns-auth`/`ns-resolver`/`ns-externaldns` apps; ApplicationSet + AppProject now list `bind-internal`
- bumped `bind-system` operator to **v0.1.3** (CRD link + image)
- operator stays in `bind-system`
## Deploy impact
ArgoCD prunes the old ns-* namespaces (StatefulSets/PVCs — data is only seed SOA+NS, no migrated records yet) and creates the renamed clusters in bind-internal.
## Validated
`kustomize build` → 28 docs (3 BindCluster, 20 BindZone, 2 catalog, 2 keys, ns); kubeconform clean.
Reviewed-on: #225
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben