## Why
#417 stood up the central Let's Encrypt `*.unkin.net` wildcard (Certificate `wildcard-unkin-net` in `cert-manager`, secret `wildcard-unkin-net-tls`, emberstack-reflected). This wires the remaining single-label public hosts onto that publicly-trusted leaf instead of the internal Vault-PKI CA.
A `*.unkin.net` wildcard matches **single-label** subdomains only, so cluster hostnames under `*.k8s.syd1.au.unkin.net` are deliberately left on `vault-issuer`.
> Note: PR #417 already merged to `main`, so this is no longer stacked — it targets `main` directly.
## How
- **cert-manager**: reflect `wildcard-unkin-net-tls` into `arrstack`, `authentik`, `gitea` (appended to both reflector namespace lists → `cheeztv,arrstack,authentik,gitea`).
- **arrstack**: repoint the public `arrproxy` gateway listener (`arrstack.unkin.net`) `certificateRefs` → `wildcard-unkin-net-tls`; drop its `cert-manager.io/*` vault-issuer annotations.
- **authentik**: repoint the `identity.unkin.net` https listener → `wildcard-unkin-net-tls`. The `identity.k8s.syd1.au.unkin.net` listeners keep `authentik-tls` (vault-issuer), so the gateway `common-name` is retargeted to that cluster host. `authentik-ldap` gateway untouched.
- **gitea**: secret **reflected only**; `git.unkin.net` cutover deferred (no manifest change), same deferred pattern as fafflix.
## Validation
- `kustomize build` OK for cert-manager / arrstack / authentik / gitea overlays.
- `kubeconform` all valid (0 invalid, 0 errors) across the four overlays.
- Verified no `*.k8s.syd1.au.unkin.net` listener lost its vault-issuer wiring.
Reviewed-on: #418
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
logviewer.unkin.net does not resolve, so the log viewer UI is unreachable even though its Gateway and certificate already exist (from #404).
## Changes
- Adds DNSRecord `logviewer-dns-internal` (A, ttl 600 → 198.18.200.4, the internal traefik gateway VIP) to bind-internal authoritative records, mirroring the existing ghp/identity records
Reviewed-on: #405
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Deploys arrproxy v0.1.0 (task 9 final integration) into the existing `arrstack` app: the external, Authentik-gated front door for sonarr/radarr/prowlarr with per-user API-key brokering. Manifests are wired to match the arrproxy v0.1.0 code (ports, routes, identity headers, keys dir), mirroring the ghp/artifactapi estate patterns.
## Topology (`apps/base/arrstack/arrproxy/`)
- **oauth2-proxy** Deployment + Service `arrproxy` is the single front (OIDC against the Authentik `arrstack` app). It path-routes via `--upstreams` to:
- `arrproxy-ui` for `/` and static SPA assets,
- `arrproxy-api` for `/api/*` (token API) and `/sonarr|/radarr|/prowlarr` (the *arr proxy).
- `--skip-auth-regex=^/[^/]+/api` bypasses oauth **only** for the *arr proxy API (`/sonarr/api...`), while `/api/tokens`, `/api/me` and the UI stay authenticated.
- Identity+groups reach the api via `--pass-user-headers` (`X-Forwarded-{User,Email,Groups}`). The api reads `ARRPROXY_GROUPS_HEADER=X-Forwarded-Groups` — `--set-xauthrequest` only populates auth_request *response* headers and never reaches an `--upstreams` backend, so it is intentionally not used.
- oauth2-proxy trusts `identity.unkin.net` (internal Vault-PKI CA) via a `combine-certs` initContainer (system roots + `vault-ca-cert`), same pattern as artifactapi.
## Data + secrets
- **CNPG** token store `arrproxy-db` (2 instances, cephrgw S3 backups + nightly ScheduledBackup). arrproxy-api does **not** self-migrate, so a wave-1 ArgoCD `Sync` hook Job applies `migrations/0001_init.sql` (mirrored into a ConfigMap) as the CNPG app user. DSN built from the generated `arrproxy-db-app` Secret.
- **VaultStaticSecrets** (arrstack `default` SA / templated role — no terraform-vault change): `arrproxy-pepper` (seeded) and `oauth-credentials`. The three real *arr keys are **reused** from the existing `<app>-apikey` Secrets, projected one file per app into `/etc/arrproxy/keys`.
## Exposure
- **Gateway** `traefik-external`, hostname `arrstack.unkin.net`, cert-manager `vault-issuer` TLS (`arrproxy-gateway-tls`); HTTPRoute → the oauth2-proxy entry Service (+ HTTP→HTTPS redirect).
- Adds the `arrstack.unkin.net` apex A record → external DMZ VIP `198.18.199.0` to the bind-operator `unkin.net` zone (`apps/base/bind-internal/authoritative/records.yaml`), mirroring the ghp/identity apex records.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly; `kubeconform` = 51/51 valid; `pre-commit` passes (no plain Secrets).
## Prerequisites / flags for end-to-end function (NOT in this PR)
- **(a)** Each *arr must run with `UrlBase=/<app>` (path-based hosting) — arrproxy preserves the `/<app>` prefix upstream.
- **(b)** The Authentik `arrstack` OIDC app (terraform-authentik PR #18) must be applied, with redirect URI `https://arrstack.unkin.net/oauth2/callback` registered and the `ak_groups` scope emitted. Confirm the discovery slug matches `OAUTH2_PROXY_OIDC_ISSUER_URL` (`https://identity.unkin.net/application/o/arrstack/`).
- **(c)** Browser XHR to `/<app>/api` is oauth-**bypassed** (skip-auth-regex), so the SPA-driven proxy calls that rely on the session (no per-user token) will not carry identity on that path — expected per the arrproxy trust model; token-based *arr clients are unaffected.
- **NetworkPolicy:** the arrproxy-api Service is not exposed via any HTTPRoute and the api strips forged identity headers in-code, but the README recommends a NetworkPolicy restricting api ingress to oauth2-proxy. The estate currently has **zero** NetworkPolicies, so one was not added here to avoid being the first (and risking probe breakage) before the CNI’s policy/probe behavior is confirmed — flagged as follow-up hardening.
- Please confirm: CNPG size (2 instances / 10Gi), backups on, hostname/VIP, and UI theme.
ARRPROXY_PEPPER was seeded via the agents AppRole at `kv/kubernetes/namespace/arrstack/default/arrproxy-pepper` (key `pepper`).
Reviewed-on: #377
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Make `ghp.unkin.net` the primary URL that the ghp app responds to. Until now ghp has only been reachable at the internal admin name `ghp.k8s.syd1.au.unkin.net`; this promotes the apex `unkin.net` name to canonical (mirroring the gitea apex pattern) while keeping the k8s name as the admin/internal route.
## How
- **`apps/base/ghp/gateway.yaml`** — add `ghp.unkin.net` as the PRIMARY http/https listeners (`http-primary`/`https-primary`) and keep `ghp.k8s.syd1.au.unkin.net` as the admin route (`http-admin`/`https-admin`), mirroring gitea. Add the cert-manager gateway annotations (`cluster-issuer: vault-issuer`, `common-name: ghp.unkin.net`, `private-key-size: "4096"`) so cert-manager mints a NEW `ghp-gateway-tls` Secret with CN `ghp.unkin.net` + a SAN per TLS listener hostname (so it also covers the k8s host). Both https listeners' `certificateRefs` point at `ghp-gateway-tls`. This is a separate Secret from `ghp-tls` (ghp's own GitHub-impersonation cert), so cert-manager never fights over it. external-dns annotation stays scoped to the k8s host only — the apex is not published by external-dns.
- **`apps/base/ghp/httproute.yaml`** — add `ghp.unkin.net` to `hostnames`; update `parentRefs` to attach to the renamed `http-primary`/`http-admin`/`https-primary`/`https-admin` sections.
- **`apps/base/ghp/configmap.yaml`** — set `GHP_SERVER_MANAGEMENT_HOST: ghp.unkin.net` and `GHP_SERVER_BASE_URL: https://ghp.unkin.net`, so ghp serves its mgmt UI on that Host and OAuth callbacks use the new base.
- **`apps/base/bind-internal/authoritative/records.yaml`** — add a bind-operator `DNSRecord` `ghp-dns-internal` for `ghp.unkin.net` -> **A 198.18.200.4** (traefik-internal gateway VIP), mirroring the active `identity-dns-internal` record in the same file.
## ⚠️ NOTE — GitHub App OAuth callback must be updated on github.com
The GitHub App's OAuth callback URL must be changed on github.com to **`https://ghp.unkin.net/auth/github/callback`**. It was set to the k8s host in the runbook; OAuth login will break until this is updated.
## Note on DNS authority
The bind-operator apex `unkin.net` zone comments warn that some apex names may still be served by the puppet DNS master (records from PuppetDB), with the k8s bind cluster not yet confirmed as the live authority for `unkin.net`. This DNSRecord mirrors the already-active `identity`/`lb1` apex records in the same file, so it is consistent with them — but if the k8s bind cluster is not the live authority for `unkin.net`, `ghp.unkin.net` will not resolve from this Record alone and the puppet master record must be added instead.
## Validation
`kustomize build` of the ghp and bind-internal overlays render clean; kubeconform (strict) valid; yamllint + pre-commit clean. Not applied.
Reviewed-on: #360
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. **Stacks on #318 — merge after it.**
How:
- 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: **bind_query** (k8s bind-* + VM named), **rancher_audit** (cattle-system sidecar JSON), **cnpg_pg** (ONE transform for all 10 CNPG clusters via the `.postgres` container), **gitea** (router+access, k8s+VM), **puppet** (openvoxserver/openvoxdb logback + access), **litellm** (JSON request logs), **postfix** (per-line maillog).
- Carve `.postgres` out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact.
- Companion k8s flips in this PR: litellm `JSON_LOGS=True`; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on.
- 15 new `vector test` cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing `fields Map(String,String)` — no DDL change.
Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with `.file`/`.SYSLOG_IDENTIFIER` tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #320
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Vault's kubernetes secret engine will mint scoped tokens for a static \`agent-dns\` service account instead of generating cluster-wide RBAC, so agent DNS access is confined to exactly the bind namespaces. This is the GitOps half of the terraform-vault agent-dns role rework (PR unkin/terraform-vault#109). Ordering: this must sync before the Vault \`agent-dns\` creds are usable — Vault mints tokens for an SA that must already exist.
## How
- Add ServiceAccount \`agent-dns\` + ClusterRole \`agent-dns\` (definition only, no ClusterRoleBinding) in \`bind-system\`: full verbs on \`bind.unkin.net\` CRDs, get/list/watch pods/services/configmaps/events, get pods/log.
- Add RoleBinding \`agent-dns\` in each of \`bind-system\`, \`bind-internal\`, \`bind-external\`, \`externaldns\`, binding the SA to the ClusterRole in that namespace — confining all access (reads included) to those four namespaces.
Whitelist note: the platform AppProject already permits ClusterRole/ClusterRoleBinding and all four namespace destinations, so no project change is needed.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #332
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Move the estate's own service/operator image pulls off the Gitea container registry (git.unkin.net/unkin) to the artifactapi local docker registry (docker-internal), ahead of the git.unkin.net forge migration which disables Gitea's container registry. The images were copied digest-for-digest into docker-internal and pulls verified before this repoint.
- repoint age-api, bind-operator, bind-tsig-api, cephrgw-operator, encapi, logarchiver, pdbmux image pulls to artifactapi.k8s.syd1.au.unkin.net/docker-internal
Deliberately not repointed here: artifactapi's own api/ui images (circular — it can't pull itself from itself), and the almalinux9-* base/CI images (huge, and their per-repo Woodpecker push targets move in a batched follow-up). Forge raw-CRD URLs and git clone sources are unaffected (those stay on the forge).
Reviewed-on: #310
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md.
- add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute
- add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222
- register gitea in the platform ApplicationSet + AppProject
- add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback)
Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc.
Reviewed-on: #309
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 3 of 3 in the external-dns → in-cluster bind migration: the client-visible cutover. The `openforwarder` resolvers currently forward `k8s.syd1.au.unkin.net` to the legacy VM anycast `198.18.19.20` (a temporary measure — commit 7ee5dfb) and have NO forwarder at all for the reverse zone `200.18.198.in-addr.arpa`. Once external-dns publishes to the in-cluster `bind-externaldns` (PR 2), resolvers must read from it.
## Changes
- Repoints the `fwd-k8s-syd1-au-unkin-net` forwarder from `198.18.19.20` (legacy VM) to `198.18.200.8` (in-cluster `bind-externaldns` VIP).
- Adds `fwd-200-18-198-in-addr-arpa` forwarding `200.18.198.in-addr.arpa` → `198.18.200.8`, closing the reverse-zone gap so PTR lookups for the k8s LB range keep resolving after cutover. Modeled exactly on the existing forward-zone entries.
- Refreshes the header comment to describe the in-cluster upstream.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean; both zones render with forwarder `198.18.200.8` and there is no residual `198.18.19.20`.
## Merge gate
- PR 2 (`benvin/externaldns-incluster`) merged, AND
- record parity confirmed between legacy and in-cluster for the forward zone. Spot-check (repeat for each name):
```
for n in puppetca puppet puppetdb encapi pdbmux artifactapi consul; do
echo "$n:"
dig +short @198.18.19.20 A $n.k8s.syd1.au.unkin.net
dig +short @198.18.200.8 A $n.k8s.syd1.au.unkin.net
done
# plus 2-3 PTRs in the reverse zone:
dig +short @198.18.19.20 -x 198.18.200.8
dig +short @198.18.200.8 -x 198.18.200.8
```
A/PTR answers from `198.18.200.8` must match those from `198.18.19.20` before merging.
## Verification (after merge)
```
dig +short @198.18.200.7 A puppet.k8s.syd1.au.unkin.net # resolvers VIP
dig +short @198.18.200.7 -x 198.18.200.8 # reverse via resolvers
```
Resolution through the `bind-resolvers` VIP should now answer for both the forward and reverse k8s zones.
## Rollback
Revert this PR — the `fwd-k8s-syd1-au-unkin-net` forwarder returns to `198.18.19.20` and the reverse forward is removed. The legacy VM is untouched and remains authoritative until decommission.
Reviewed-on: #288
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 1 of 3 in the external-dns → in-cluster bind migration. For external-dns to send RFC2136 updates to the in-cluster `bind-externaldns` primary, it must present the exact TSIG key the primary's `allow-update` accepts. The bind-operator generates that key material into Secret `externaldns-key-tsig` in `bind-internal`; reflecting it into the `externaldns` namespace removes the manual eyaml→Vault key sync and guarantees key parity.
## Changes
- Adds `spec.secretTemplate.annotations` to BindTSIGKey `externaldns-key` with the emberstack reflector hints: `reflection-allowed`, `reflection-allowed-namespaces: externaldns`, `reflection-auto-enabled`, `reflection-auto-namespaces: externaldns`.
- Regenerates `schemas/bind.unkin.net/bindtsigkey_v1alpha1.json` from the live CRD (deployed bind-operator v0.2.4 already exposes `secretTemplate` — the WIP branch's ">= v0.3.0" claim is stale) to add the `secretTemplate` property. Schema output is byte-identical to running `ci/generate-schemas.sh`.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean and renders the annotations onto the BindTSIGKey.
## Verification (after merge)
```
kubectl -n externaldns get secret externaldns-key-tsig \
-o jsonpath='{.data.secret} {.data.algorithm}{"\n"}'
```
Both `secret` and `algorithm` keys must be present (reflector mirrored the source Secret from bind-internal).
## Rollback
Revert this PR. The source Secret in bind-internal is unaffected; only the reflected mirror in `externaldns` is removed.
Merge order: this is PR 1/3. PR 2 (repoint external-dns) must not merge until the reflected secret is verified.
Reviewed-on: #285
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Kubernetes nodes querying the bind-resolvers LoadBalancer VIP (198.18.200.7) get REFUSED (EDE 18 Prohibited).
The service is `externalTrafficPolicy: Local`, which preserves the client source IP for traffic entering the cluster from outside — but a node querying the VIP never leaves via OSPF. Its own kube-proxy DNATs the LB IP in the OUTPUT chain and masquerades the source to a cluster-internal address (the node's flannel.1, e.g. 10.42.x.x). That address is not in `acl-main.unkin.net`, so the openforwarder view's match-clients rejects the query.
External clients preserve their real source IP and match acl-main, which is why only in-cluster hosts were affected.
Add `10.42.0.0/16` to `acl-main.unkin.net` so node-originated (masqueraded) resolver queries are permitted. This mirrors the authoritative cluster, which already allows the pod network (`allow-query { ...; 10.42.0.0/16; }`).
Reviewed-on: #271
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The bind-resolvers `openforwarder` view forwarded `k8s.syd1.au.unkin.net` to the in-cluster bind-externaldns (198.18.200.8), which is not reliably serving those records yet, so lookups return NXDOMAIN.
Concrete impact: Gitea cannot resolve the k8s-hosted CI host, so its outbound webhook fails and tagged releases never trigger CI (e.g. cutting a new bind-operator release from a tag).
This points the `fwd-k8s-syd1-au-unkin-net` forwarder at the existing external external-dns bind service anycast **198.18.19.20** (puppet `roles::infra::dns::externaldns` — master `ausyd1nxvm2127` + slaves `2128`/`2129`, advertised via OSPF), which still holds the working `k8s.syd1.au.unkin.net` records. It is in the same 198.18.19.0/24 anycast family as the consul forwarder (198.18.19.14) the resolvers already use, so it is reachable from the pods.
Temporary measure. Revert to 198.18.200.8 once external-dns publishes to the in-cluster bind-externaldns service. Only the forward target changes; no CRD/schema changes.
Reviewed-on: #269
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Every bind-operator dynamic update is refused (`update ... denied due to allow-query`) because the operator execs `nsupdate` against `127.0.0.1` inside the primary pod, and the BindCluster `allow-query` listed only the client subnets (`auth-acl-main`) and the pod net (`10.42.0.0/16`) — not loopback. This blocked ALL DNSRecords (identity, s3, dashboard, lb1) from ever applying.
## Change
- Add `localhost` to the BindCluster `allow-query` in `apps/base/bind-internal/authoritative/cluster.yaml`. The `client-update` TSIG key still gates the actual update.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #267
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The cephrgw-operator (in-cluster) fails to reach the Ceph dashboard because CoreDNS/bind-internal has no record for `dashboard.ceph.unkin.net` (`no such host`). Publish it authoritatively so in-cluster clients can resolve it.
## Changes (apps/base/bind-internal/authoritative/records.yaml)
- `DNSRecord dashboard-ceph-cname`: CNAME `dashboard.ceph.unkin.net` -> `lb1.unkin.net.` (zone `ceph.unkin.net`, zoneRef `ceph-unkin-net`).
- `DNSRecord lb1-unkin-net`: A `lb1.unkin.net` -> `103.216.191.185` (zone `unkin.net`, zoneRef `unkin-net`).
Once applied, the operator's `sandbox-user`/`sandbox-bucket` (currently Error/Pending on DNS) will reconcile to Ready.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #266
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Publish the RGW S3 endpoint name (`s3.ceph.unkin.net`) that cephrgw-operator consumers use and that the radosgw hosts will carry as a cert SAN. For now it points at the Consul service; the real target will be changed later.
## Changes
- Add a `DNSRecord` in the `ceph.unkin.net` authoritative zone: `s3` CNAME `radosgw.service.consul.` (`apps/base/bind-internal/authoritative/records.yaml`, zoneRef `ceph-unkin-net`, TTL 600).
A companion puppet-prod change adds `s3.ceph.unkin.net` to the radosgw cert SANs and nginx server names.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #265
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The ceph/halb host (`ausyd1nxvm2069`) publishes `dashboard.ceph.unkin.net` via nsupdate to a dedicated `zone ceph.unkin.net.`, which `bind-authoritative` was not authoritative for (NOTZONE). This adds the zone so that record has a home.
## Changes
- Add `ceph.unkin.net` BindZone (primary, dynamicUpdate, updateKeyRef client-update) to bind-authoritative, matching the unkin.net/main.unkin.net pattern.
## Note — not the root cause of the 6 missing hosts
Log evidence (VictoriaLogs, dns-update-apply on 2069/2070) shows the actual failure is a **host-side bug in the puppet `dns-update` script**: `fqdn()` appends the zone even to records whose name is already fully-qualified (e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`), producing a `..` empty label → `invalid owner name: empty label` → the whole main.unkin.net update `send` fails (reverse PTR, sent first, still lands). That script fix (puppet-prod) is the real blocker; this zone is still needed so the ceph record does not hit NOTZONE afterward.
Reviewed-on: #260
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to #258 (which added workstation+router to the *authoritative* ACL). The **resolver** (bind-resolvers, 198.18.200.7) has its own `acl-main.unkin.net` gating its `openforwarder` view; the workstation is not in it, so recursive queries return REFUSED. This lets the workstation use the resolver as its normal nameserver.
## Changes
- Add `10.10.12.200/32` (workstation, wireguard) to resolver `acl-main.unkin.net`
- Add `198.18.21.160/32` (router) explicitly for documentation; already covered by existing `198.18.21.160/27` and `198.18.21.0/24` (no functional change for the router)
Reviewed-on: #259
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Allow the operator's workstation and router to query the bind-authoritative servers directly. Their source addresses are outside the existing `auth-acl-main` client subnets, so named returns REFUSED to them today. The router sits on 198.18.21.0/24 which is not in the ACL at all.
## Changes
- Add `10.10.12.200/32` (workstation, over wireguard) to `auth-acl-main`
- Add `198.18.21.160/32` (router) to `auth-acl-main`
## Note
This grants query permission only. Reaching the LoadBalancer VIP (198.18.200.6) from off-datacenter paths is separately gated by `externalTrafficPolicy: Local`; the workstation-over-wireguard path still needs its L4 routing addressed to actually land on a node with a ready endpoint.
Reviewed-on: #258
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`identity.unkin.net` is configured as the Authentik OIDC issuer but has **no DNS record anywhere**, so in-cluster OIDC discovery fails (e.g. `argocd-server` → `lookup identity.unkin.net ... no such host`). Add an authoritative A record served by the internal bind system (bind-operator) so it resolves.
## Change
- New `DNSRecord/identity-dns-internal` → `198.18.200.4` (the traefik-internal gateway VIP, where the Authentik Gateway serves the `identity.unkin.net` hostname), in the `unkin-net` zone.
- Lives in the **`bind-internal` namespace** alongside the `BindZone`: the operator resolves `zoneRef`/`clusterRef`/`updateKeyRef` within the record's own namespace, so it can't live in the app (authentik) namespace.
- Wired into `apps/base/bind-internal/authoritative/kustomization.yaml`.
- `identity-dns-internal` name distinguishes this from the external DNS that Authentik will manage its own records from later.
## Validation
`kustomize build apps/base/bind-internal` + kubeconform (validates against the `dnsrecord_v1alpha1` schema): 57 valid, 0 invalid. pre-commit clean.
Reviewed-on: #256
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2's config-hash rolling restart exposed a latent non-determinism: `client.List` returns cache-ordered results, so the resolver's forward zones reshuffled every reconcile, flipping the config hash and rolling `bind-resolvers-2` endlessly. v0.2.3 (bind-operator #11) sorts every rendered list so `named.conf` is byte-identical across reconciles and the hash is stable.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.3`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.3`.
CRDs are unchanged from v0.2.2 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.3 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.
## Deploy note
On deploy the deterministic operator stops churning the ConfigMap; the config hash stabilizes and the stuck resolver rolling update completes, leaving all three pods Ready on one revision.
Reviewed-on: #255
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2 (bind-operator #10) stamps a `bind.unkin.net/config-hash` on the pod template, so a ConfigMap or `keys.conf` change flips the hash and triggers an operator-driven rolling restart. This fixes the class of bug where config edits (ACLs, forwarders, `validate-except`, primary address, TSIG rotation) never reached running pods — they held a startup snapshot and needed manual pod deletes.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.2`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.2`.
CRDs are unchanged from v0.2.1 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.2 CRD bundle; pre-commit clean.
## Deploy note
When the v0.2.2 operator first reconciles it stamps the config-hash annotation, triggering **one rolling restart per bind StatefulSet** — expected, and it also pulls in any already-pending config. From then on, config changes roll pods automatically.
Reviewed-on: #254
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why — urgent
PR #251 added `validate-except` to the resolver `BindCluster` but the list omitted the trailing semicolon after the final entry. `named` requires **every** entry in a list to be semicolon-terminated, including the last before the closing brace, so it fails config parse and the resolver pods crash-loop:
```
/run/named/named.conf:18: missing ';' before '}'
loading configuration: failure
exiting (due to fatal error)
```
The resolvers (`.7`) are down until this lands; the authoritative (`.6`/`.9`) and externaldns (`.8`) are unaffected.
## Fix
```diff
- validate-except { unkin.net; 18.198.in-addr.arpa; consul }
+ validate-except { unkin.net; 18.198.in-addr.arpa; consul; }
```
Renders to `validate-except { unkin.net; 18.198.in-addr.arpa; consul; };` — valid.
## Recovery
On merge + ArgoCD sync, the operator re-renders the ConfigMap with valid config and the crash-looping pods self-heal on their next restart (no manual `rollout restart` needed). Validated: `bind-internal` renders and pre-commit clean.
Reviewed-on: #252
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Resolving any `unkin.net` record through the resolver (`.7`) returns **SERVFAIL**, while the authoritative (`.6`) answers fine. Confirmed from the resolver's querylog:
```
view openforwarder: validating unkin.net/SOA: got insecure response; parent indicates it should be secure
broken trust chain resolving 'ausyd1nxvm2120.main.unkin.net/A/IN': 198.18.200.6#53
query failed (broken trust chain)
```
The resolver runs `dnssec-validation auto`. The public `unkin.net` is DNSSEC-signed (the `.net` parent publishes a DS), but the in-cluster split-horizon authoritative serves `unkin.net` **unsigned**. The validator sees "parent says secure" + an insecure answer → treats it as spoofing → SERVFAIL. The authoritative works directly because it does no validation.
## Fix
Add `validate-except` (via `spec.extraOptions`) for the forwarded internal domains, so the resolver treats them as insecure and skips validation:
```
validate-except { unkin.net; 18.198.in-addr.arpa; consul }
```
- `unkin.net` covers all `*.unkin.net` (incl. `main.unkin.net`, `k8s.syd1.au.unkin.net`)
- `18.198.in-addr.arpa` covers every `NN.18.198.in-addr.arpa` reverse zone (subtree)
- `consul` covers the consul TLD
This also makes internal resolution independent of Internet egress (no DNSSEC chain-walk needed). External-name validation is unchanged. No operator change required.
## Validation
`bind-internal` renders and passes `kubeconform` (56/56); pre-commit clean.
## Activation
After merge + operator reconcile, the resolver ConfigMap re-renders; the running pods hold a startup snapshot, so they need a reload: `kubectl -n bind-internal rollout restart statefulset/bind-resolvers`.
Reviewed-on: #251
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.1 fixes authoritative **secondary replication**, which never actually worked — the master REFUSED the catalog AXFR. Root causes (bind-operator #9): secondaries presented no TSIG key, member zones had no `allow-transfer`, and secondaries pointed at the primary's ephemeral pod IP.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.1`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.1`.
CRDs are unchanged from v0.2.0, so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.1 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.
## Deploy note
Existing member zones pick up `allow-transfer` via `modzone`, and secondaries re-point at the stable primary Service ClusterIP with the transfer key, restoring replication without manual BIND surgery. A from-scratch namespace recreate also comes up clean (OrderedReady startup means secondaries snapshot a ClusterIP-correct config). Caveat for a full recreate: the operator regenerates the `client-update` TSIG key, so its new material must be re-synced into puppet eyaml before clients can nsupdate.
Reviewed-on: #250
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
bind-operator v0.2.0 adds the `BindTSIGAPI` CRD and a companion API that `vault-plugin-secrets-bind-tsig` calls to create, rotate and delete TSIG keys (it does so by managing `BindTSIGKey` resources, which the operator reconciles into key material). This rolls the operator forward and deploys an API instance so Vault never talks to the Kubernetes API directly.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.0`.
- Broaden the operator ClusterRole (`bind-system/rbac.yaml`) with `deployments`, `serviceaccounts` and `roles`/`rolebindings`, so the `BindTSIGAPI` reconciler can create the API Deployment and its namespaced Role/RoleBinding.
- Add a `BindTSIGAPI` (`bind-tsig-api`) in `bind-internal`; the operator reconciles it into a Deployment, Service, ConfigMap, master-token Secret and RBAC. Keys are created in `bind-internal`, alongside the authoritative cluster and its existing keys.
- Add the generated kubeconform schema for `BindTSIGAPI`.
## Notes
- The master access token Secret (`bind-tsig-api-token`) is generated by the operator when absent; the operator does not own it, so a `VaultStaticSecret` can later pre-seed/overwrite it to source the token from Vault.
- Validated: both overlays render (`kubectl kustomize`) and pass `kubeconform` (bind-internal 56/56 valid); pre-commit clean.
## Follow-up
- Point `vault-plugin-secrets-bind-tsig` config at `http://bind-tsig-api.bind-internal.svc:8443`.
Reviewed-on: #249
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Enables per-host RFC2136 updates from puppet (puppet-prod #475 profiles::dns::updater) to the bind-authoritative zones, via the .9 write endpoint.
## Changes
- add **client-update** BindTSIGKey (clusterRef bind-authoritative; operator generates the material into Secret client-update-tsig)
- set `dynamicUpdate: true` + `updateKeyRef: client-update` on all **18** authoritative zones → the operator renders `allow-update { key "client-update"; }`
## Key bridge (manual, per the TSIG plan)
The operator generates the client-update key value; it must reach puppet eyaml (`profiles::dns::updater::key_secret`) for clients to authenticate — until the planned Vault-sync/secret-reflection operator features exist. Get it with:
`kubectl -n bind-internal get secret client-update-tsig -o jsonpath='{.data.secret}' | base64 -d`
## Validated
kustomize build + kubeconform.
Reviewed-on: #244
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**Stacked on #228** (needs operator v0.1.5). Merge #228 first; the diff collapses to just this after.
## Why
Writes (RFC2136/nsupdate) must go to pod-0 — the round-robin read Service would land them on a secondary (rejected). Adds a dedicated write endpoint per cluster (operator v0.1.5 `primaryService`).
## Changes
- `bind-authoritative`: LoadBalancer write endpoint on **198.18.200.9** (`bind-authoritative-primary`)
- `bind-externaldns`: ClusterIP write endpoint (`bind-externaldns-primary`, for in-cluster writers)
- regenerate the bindcluster kubeconform schema (primaryService + externalTrafficPolicy)
## Deferred
external-dns is **not** repointed at `bind-externaldns-primary` yet: it authenticates with the existing TSIG key, which the operator-generated key won't match until the planned Vault-sync + secret-reflection features exist. Until then external-dns keeps writing to the puppet externaldns.
## Validated
kustomize build + kubeconform (3 BindClusters valid against the v0.1.5 schema).
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #229
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Mirrors the puppet authoritative `master-zones` view (match-clients `acl-main.unkin.net`, recursion no) — restricting who can query bind-authoritative.
## Changes
- add `auth-acl-main` BindACL with the puppet authoritative acl-main.unkin.net networks (13-17,19,20,24-29)
- `allow-query { auth-acl-main; 10.42.0.0/16; }` on bind-authoritative via extraOptions
## Notes
- Implemented as a global `allow-query` rather than a BindView: dynamic *primary* zones inside a view would need per-view `allow-new-zones` (an operator gap). Functionally equivalent for the single master-zones view.
- `10.42.0.0/16` (pod network) is included so secondaries can SOA-refresh from the primary during catalog replication — without it, replication breaks.
- Works on the current operator (no HOLD).
## Caveat
The DNS Services use externalTrafficPolicy: Cluster, which SNATs external clients to node IPs (198.18.19.x, already in acl-main), so this ACL doesn't truly restrict *external* clients yet. True source-IP restriction needs externalTrafficPolicy: Local — happy to switch if wanted.
Reviewed-on: #227
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`dig google.com @198.18.200.7` was refused: the resolver never set allow-recursion, so BIND defaulted to localnets/localhost. This mirrors the puppet resolver (/etc/named/views.conf + acls.conf) exactly.
## Changes
- `openforwarder` BindView: `match-clients` = the 4 internal ACLs, recursion yes, allow-recursion/allow-query `any` (match-clients gates)
- 4 BindACLs from puppet acls.conf (acl-main.unkin.net/acl-dmz/acl-common/acl-nomad-jobs)
- 26 conditional forward zones in the view (unkin→198.18.19.15, consul→.14, k8s→.20, dmz/network/prod + 10.10.x reverse → 10.10.16.32/33)
- global forwarders 8.8.8.8/1.1.1.1
- operator image → v0.1.4
## Note
Forward-zone upstreams point at the **puppet anycast** servers (still authoritative during migration); flip to the in-cluster authoritative/externaldns LBs once zone data is migrated.
## Validated
kustomize build (59 docs), kubeconform clean.
Reviewed-on: #226
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**HOLD until v0.1.3 is tagged/built** (operator #4 merged + tagged) — this PR bumps the operator to v0.1.3, whose CRD adds the `clusterRef` field these keys use.
## Why
Put all BIND DNS services in one `bind-internal` namespace and name the StatefulSets clearly.
## Changes
- 3 clusters consolidated into `bind-internal`, StatefulSets renamed **bind-authoritative** / **bind-resolvers** / **bind-externaldns**; LBs kept on 198.18.200.6/.7/.8; external-dns hostnames renamed to match
- `clusterRef` added to `transfer-key` (→ bind-authoritative) and `externaldns-key` (→ bind-externaldns) so keys are scoped per cluster
- removed the old `ns-auth`/`ns-resolver`/`ns-externaldns` apps; ApplicationSet + AppProject now list `bind-internal`
- bumped `bind-system` operator to **v0.1.3** (CRD link + image)
- operator stays in `bind-system`
## Deploy impact
ArgoCD prunes the old ns-* namespaces (StatefulSets/PVCs — data is only seed SOA+NS, no migrated records yet) and creates the renamed clusters in bind-internal.
## Validated
`kustomize build` → 28 docs (3 BindCluster, 20 BindZone, 2 catalog, 2 keys, ns); kubeconform clean.
Reviewed-on: #225
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben