## Why
During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move
between them as they migrate. `node-lookup` (and `pblastreport`) need a single,
consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy
that provides exactly that. Per the all-in-kubernetes estate direction it runs
as an in-cluster service, not a per-VM systemd unit.
pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) —
split out of the earlier node-lookup prototype — and is released as a container
image on its own `v*` tags.
## Changes
- Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway,
httproute), modeled directly on the encapi app.
- Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080,
`/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap.
- Backends: `old=http://puppetdbapi.service.consul:8080`,
`new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified
against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the
in-cluster address is preferred over the external gateway). `new` is
primary/prefer, merge = freshness.
- Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal`
Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a
port-80 Service — same shape as the puppetdb/encapi gateways — so
VM/workstation `node-lookup` can reach it.
- Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform
ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject
(`pdbmux` namespace destination), exactly as encapi is wired.
No new woodpecker ServiceAccount is required: the pdbmux image push uses the
`docker-buildx` plugin against the Gitea registry with the `default` SA (same as
encapi), not artifactapi.
## Verification
- `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves
to `git.unkin.net/unkin/pdbmux:v0.1.0`).
- ApplicationSet + AppProject YAML validated.
## Merge gates
1. The pdbmux repo initial-content PR
(unkin/pdbmux#1) must merge first.
2. `v0.1.0` must then be tagged on the pdbmux repo so the image
`git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s
`.woodpecker/docker.yaml`.
3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update
the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.)
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #275
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator v0.2.0 talks to radosgw over HTTPS (`radosgw.service.consul:443`, fronted by nginx presenting the internal `unkin.net` Vault-PKI cert). With no CA configured the operator fails:
```
Get "https://radosgw.service.consul:443/admin/user?...": tls: failed to verify certificate: x509: certificate signed by unknown authority
```
The `vault-ca-cert` Secret (the `unkin.net` intermediate+root) is already reflected into every namespace — including `cephrgw-system` — so the fix is deployment-only.
## Changes
- mount the `vault-ca-cert` Secret (key `ca.crt`) read-only at `/etc/vault-ca/ca.crt`, following the puppet/artifactapi pattern
- set `CEPH_RGW_CA_FILE=/etc/vault-ca/ca.crt` so the operator adds the CA to its TLS trust
No image change (still `v0.2.0`); `reloader` + a normal reconcile pick it up. Validated with `kustomize build` on the au-syd1 overlay.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #276
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The new **terragrunt-enc** repo (single source of truth for encapi ENC data) runs its Terraform apply/plan in Woodpecker and authenticates to Vault via kubernetes auth. The Vault k8s role `woodpecker_terraform_enc` (terraform-vault PR #98) binds to a ServiceAccount named `terraform-enc` in the `woodpecker` namespace, which must exist for that auth to work.
Changes:
- Add `apps/base/woodpecker/serviceaccount_terraform_enc.yaml` (SA `terraform-enc` in namespace `woodpecker`).
- Register it in the woodpecker kustomization.
Reviewed-on: #274
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator **v0.2.0** rebuilds the Ceph integration to talk directly to radosgw via **go-ceph** (Admin Ops API) + **aws-sdk-go-v2** (S3), replacing the manager-dashboard client, and adds **fine-grained bucket-access policies** (paths / actions / conditions / rawStatements). The operator now authenticates with an **RGW admin user's access/secret key** instead of a dashboard login.
Operator repo PRs: unkin/cephrgw-operator #3 (rebuild) and #4 (fine-grained), both merged; tag `v0.2.0`.
## Changes
- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.2.0`
- update the `envFrom` / `VaultStaticSecret` comments to the `CEPH_RGW_*` credential keys the new image consumes
## Required manual step (runtime)
The VaultStaticSecret copies the KV secret's keys **verbatim**, so the seed must be re-put with the new keys before/with rollout — otherwise the operator fails auth:
```
vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
CEPH_RGW_ACCESS_KEY=<key> CEPH_RGW_SECRET_KEY=<secret>
```
(The old `CEPH_DASHBOARD_*` keys are ignored by v0.2.0.) VSO refreshes within 5m and the `reloader` annotation restarts the operator.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #273
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Kubernetes nodes querying the bind-resolvers LoadBalancer VIP (198.18.200.7) get REFUSED (EDE 18 Prohibited).
The service is `externalTrafficPolicy: Local`, which preserves the client source IP for traffic entering the cluster from outside — but a node querying the VIP never leaves via OSPF. Its own kube-proxy DNATs the LB IP in the OUTPUT chain and masquerades the source to a cluster-internal address (the node's flannel.1, e.g. 10.42.x.x). That address is not in `acl-main.unkin.net`, so the openforwarder view's match-clients rejects the query.
External clients preserve their real source IP and match acl-main, which is why only in-cluster hosts were affected.
Add `10.42.0.0/16` to `acl-main.unkin.net` so node-originated (masqueraded) resolver queries are permitted. This mirrors the authoritative cluster, which already allows the pod network (`allow-query { ...; 10.42.0.0/16; }`).
Reviewed-on: #271
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Picks up immediate NOTIFY of secondaries on primary zone changes (also-notify
+ shorter seed SOA timers), so dynamic updates / CRD records replicate across
the authoritative replicas in seconds instead of waiting up to the SOA refresh.
Bumps both the operator image and the CRD install.yaml tag.
The bind-resolvers `openforwarder` view forwarded `k8s.syd1.au.unkin.net` to the in-cluster bind-externaldns (198.18.200.8), which is not reliably serving those records yet, so lookups return NXDOMAIN.
Concrete impact: Gitea cannot resolve the k8s-hosted CI host, so its outbound webhook fails and tagged releases never trigger CI (e.g. cutting a new bind-operator release from a tag).
This points the `fwd-k8s-syd1-au-unkin-net` forwarder at the existing external external-dns bind service anycast **198.18.19.20** (puppet `roles::infra::dns::externaldns` — master `ausyd1nxvm2127` + slaves `2128`/`2129`, advertised via OSPF), which still holds the working `k8s.syd1.au.unkin.net` records. It is in the same 198.18.19.0/24 anycast family as the consul forwarder (198.18.19.14) the resolvers already use, so it is reachable from the pods.
Temporary measure. Revert to 198.18.200.8 once external-dns publishes to the in-cluster bind-externaldns service. Only the forward target changes; no CRD/schema changes.
Reviewed-on: #269
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Pairs with terraform-authentik#10: request the litellm_role scope (Authentik
emits the computed role claim) and read it via GENERIC_USER_ROLE_ATTRIBUTE so
akP-litellm-admin -> proxy_admin, akP-litellm-user -> internal_user.
App-side of the LiteLLM Authentik onboarding (terraform-authentik#8). Configures
LiteLLM's generic OIDC SSO against Authentik.
- VaultStaticSecret oauth-credentials: surfaces the OIDC client secret (same
secret Authentik sets on the provider) as a k8s Secret.
- Deployment: GENERIC_CLIENT_SECRET from that Secret.
- litellm-env: GENERIC_CLIENT_ID, authorization/token/userinfo endpoints, scope,
and PROXY_BASE_URL (required for SSO). reloader restarts on secret/config change.
## Why
Every bind-operator dynamic update is refused (`update ... denied due to allow-query`) because the operator execs `nsupdate` against `127.0.0.1` inside the primary pod, and the BindCluster `allow-query` listed only the client subnets (`auth-acl-main`) and the pod net (`10.42.0.0/16`) — not loopback. This blocked ALL DNSRecords (identity, s3, dashboard, lb1) from ever applying.
## Change
- Add `localhost` to the BindCluster `allow-query` in `apps/base/bind-internal/authoritative/cluster.yaml`. The `client-update` TSIG key still gates the actual update.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #267
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The cephrgw-operator (in-cluster) fails to reach the Ceph dashboard because CoreDNS/bind-internal has no record for `dashboard.ceph.unkin.net` (`no such host`). Publish it authoritatively so in-cluster clients can resolve it.
## Changes (apps/base/bind-internal/authoritative/records.yaml)
- `DNSRecord dashboard-ceph-cname`: CNAME `dashboard.ceph.unkin.net` -> `lb1.unkin.net.` (zone `ceph.unkin.net`, zoneRef `ceph-unkin-net`).
- `DNSRecord lb1-unkin-net`: A `lb1.unkin.net` -> `103.216.191.185` (zone `unkin.net`, zoneRef `unkin-net`).
Once applied, the operator's `sandbox-user`/`sandbox-bucket` (currently Error/Pending on DNS) will reconcile to Ready.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #266
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to the cephrgw-operator deploy (#261): source the operator's Ceph dashboard credentials from Vault via VSO instead of a hand-created Secret.
## Changes
- Add `apps/base/cephrgw-system/vaultauth.yaml`: `VaultAuth` (mount `k8s/au/syd1`, role `cephrgw-operator`, SA `cephrgw-operator`, `vaultConnectionRef: vso-system/default`).
- Add `apps/base/cephrgw-system/vaultstaticsecret.yaml`: renders KV `service/cephrgw/dashboard-credentials` into the `cephrgw-credentials` Secret (keys copied verbatim → consumed by the Deployment via `envFrom`; the reloader annotation restarts the operator on rotation).
- Reference both from the base kustomization.
## Dependencies / ordering
- Requires the Vault role + policy from **terraform-vault #95** (merge/apply first), and the KV values to be seeded out-of-band:
```
vault kv put kv/service/cephrgw/dashboard-credentials \
CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
CEPH_DASHBOARD_PASSWORD=... CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net
```
- Until VSO auth succeeds the `cephrgw-credentials` Secret won't exist and the operator pod stays in `CreateContainerConfigError` (expected).
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #262
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Publish the RGW S3 endpoint name (`s3.ceph.unkin.net`) that cephrgw-operator consumers use and that the radosgw hosts will carry as a cert SAN. For now it points at the Consul service; the real target will be changed later.
## Changes
- Add a `DNSRecord` in the `ceph.unkin.net` authoritative zone: `s3` CNAME `radosgw.service.consul.` (`apps/base/bind-internal/authoritative/records.yaml`, zoneRef `ceph-unkin-net`, TTL 600).
A companion puppet-prod change adds `s3.ceph.unkin.net` to the radosgw cert SANs and nginx server names.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #265
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Consume the two-tier Authentik RBAC from terraform-authentik#7. Grafana should grant Admin to the `akP-grafana-admin` permission group, which `akR-global-admin` members inherit.
## Change
- **grafana.yaml** (`auth.generic_oauth`): add `ak_groups` to `scopes`; `role_attribute_path` now keys off `ak_groups` and `akP-grafana-admin` (replaces the flat `grafana-admins`). Non-admins who can log in (gated to `akP-grafana-*` by the Authentik access policy) get Viewer; `role_attribute_strict: false` retained.
## Depends on
terraform-authentik#7 (creates `akP-grafana-admin`, the access binding, and the `ak_groups` mapping).
## Validation
`kustomize build` (base + overlay) renders; pre-commit clean.
Reviewed-on: #264
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new `cephrgw-operator` provisions Ceph RGW (S3) buckets and access keys (RW/RO) from Kubernetes CRDs via the Ceph manager dashboard API. This deploys it as a platform app.
## Changes
- Add `apps/base/cephrgw-system`: namespace, ServiceAccount + ClusterRole/Binding (manage `ceph.unkin.net` CRDs, Secrets, leader-election leases), and the operator Deployment. CRDs are pulled from the operator repo at tag `v0.1.0`; the Deployment sources dashboard credentials from the `cephrgw-credentials` Secret via `envFrom` and carries the reloader annotation.
- Add `apps/overlays/au-syd1/cephrgw-system` referencing the base.
- Register `apps/overlays/*/cephrgw-system` in the platform ApplicationSet.
The platform AppProject already permits `*-system` namespaces and the Namespace/ClusterRole/CRD cluster resources, so no project change is needed.
## Ordering / dependencies
- Depends on the Gitea repo from terraform-git #34 and on the operator being pushed + tagged **v0.1.0** (image `git.unkin.net/unkin/cephrgw-operator:v0.1.0` and the raw CRD `install.yaml` at that tag). The `kubeconform` check will stay red until v0.1.0 exists, then go green.
- The `cephrgw-credentials` Secret must be created out-of-band in `cephrgw-system` (see the operator's `docs/ceph-setup.md`); it is intentionally **not** managed in GitOps.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #261
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The ceph/halb host (`ausyd1nxvm2069`) publishes `dashboard.ceph.unkin.net` via nsupdate to a dedicated `zone ceph.unkin.net.`, which `bind-authoritative` was not authoritative for (NOTZONE). This adds the zone so that record has a home.
## Changes
- Add `ceph.unkin.net` BindZone (primary, dynamicUpdate, updateKeyRef client-update) to bind-authoritative, matching the unkin.net/main.unkin.net pattern.
## Note — not the root cause of the 6 missing hosts
Log evidence (VictoriaLogs, dns-update-apply on 2069/2070) shows the actual failure is a **host-side bug in the puppet `dns-update` script**: `fqdn()` appends the zone even to records whose name is already fully-qualified (e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`), producing a `..` empty label → `invalid owner name: empty label` → the whole main.unkin.net update `send` fails (reverse PTR, sent first, still lands). That script fix (puppet-prod) is the real blocker; this zone is still needed so the ceph record does not hit NOTZONE afterward.
Reviewed-on: #260
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to #258 (which added workstation+router to the *authoritative* ACL). The **resolver** (bind-resolvers, 198.18.200.7) has its own `acl-main.unkin.net` gating its `openforwarder` view; the workstation is not in it, so recursive queries return REFUSED. This lets the workstation use the resolver as its normal nameserver.
## Changes
- Add `10.10.12.200/32` (workstation, wireguard) to resolver `acl-main.unkin.net`
- Add `198.18.21.160/32` (router) explicitly for documentation; already covered by existing `198.18.21.160/27` and `198.18.21.0/24` (no functional change for the router)
Reviewed-on: #259
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Allow the operator's workstation and router to query the bind-authoritative servers directly. Their source addresses are outside the existing `auth-acl-main` client subnets, so named returns REFUSED to them today. The router sits on 198.18.21.0/24 which is not in the ACL at all.
## Changes
- Add `10.10.12.200/32` (workstation, over wireguard) to `auth-acl-main`
- Add `198.18.21.160/32` (router) to `auth-acl-main`
## Note
This grants query permission only. Reaching the LoadBalancer VIP (198.18.200.6) from off-datacenter paths is separately gated by `externalTrafficPolicy: Local`; the workstation-over-wireguard path still needs its L4 routing addressed to actually land on a node with a ready endpoint.
Reviewed-on: #258
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new terraform-rancher CI pipeline runs as a pod in the woodpecker namespace; its ServiceAccount token is what Vault binds to the `woodpecker_terraform_rancher` k8s auth role (see terraform-vault#86) for rancher2 provider auth + Consul state.
## Change
- Add `ServiceAccount/terraform-rancher` (woodpecker ns) and wire it into the woodpecker kustomization, mirroring the other terraform-* runner SAs.
## Validation
`kustomize build apps/base/woodpecker` renders it; pre-commit clean.
Reviewed-on: #257
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`identity.unkin.net` is configured as the Authentik OIDC issuer but has **no DNS record anywhere**, so in-cluster OIDC discovery fails (e.g. `argocd-server` → `lookup identity.unkin.net ... no such host`). Add an authoritative A record served by the internal bind system (bind-operator) so it resolves.
## Change
- New `DNSRecord/identity-dns-internal` → `198.18.200.4` (the traefik-internal gateway VIP, where the Authentik Gateway serves the `identity.unkin.net` hostname), in the `unkin-net` zone.
- Lives in the **`bind-internal` namespace** alongside the `BindZone`: the operator resolves `zoneRef`/`clusterRef`/`updateKeyRef` within the record's own namespace, so it can't live in the app (authentik) namespace.
- Wired into `apps/base/bind-internal/authoritative/kustomization.yaml`.
- `identity-dns-internal` name distinguishes this from the external DNS that Authentik will manage its own records from later.
## Validation
`kustomize build apps/base/bind-internal` + kubeconform (validates against the `dnsrecord_v1alpha1` schema): 57 valid, 0 invalid. pre-commit clean.
Reviewed-on: #256
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2's config-hash rolling restart exposed a latent non-determinism: `client.List` returns cache-ordered results, so the resolver's forward zones reshuffled every reconcile, flipping the config hash and rolling `bind-resolvers-2` endlessly. v0.2.3 (bind-operator #11) sorts every rendered list so `named.conf` is byte-identical across reconciles and the hash is stable.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.3`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.3`.
CRDs are unchanged from v0.2.2 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.3 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.
## Deploy note
On deploy the deterministic operator stops churning the ConfigMap; the config hash stabilizes and the stuck resolver rolling update completes, leaving all three pods Ready on one revision.
Reviewed-on: #255
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2 (bind-operator #10) stamps a `bind.unkin.net/config-hash` on the pod template, so a ConfigMap or `keys.conf` change flips the hash and triggers an operator-driven rolling restart. This fixes the class of bug where config edits (ACLs, forwarders, `validate-except`, primary address, TSIG rotation) never reached running pods — they held a startup snapshot and needed manual pod deletes.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.2`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.2`.
CRDs are unchanged from v0.2.1 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.2 CRD bundle; pre-commit clean.
## Deploy note
When the v0.2.2 operator first reconciles it stamps the config-hash annotation, triggering **one rolling restart per bind StatefulSet** — expected, and it also pulls in any already-pending config. From then on, config changes roll pods automatically.
Reviewed-on: #254
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why — urgent
PR #251 added `validate-except` to the resolver `BindCluster` but the list omitted the trailing semicolon after the final entry. `named` requires **every** entry in a list to be semicolon-terminated, including the last before the closing brace, so it fails config parse and the resolver pods crash-loop:
```
/run/named/named.conf:18: missing ';' before '}'
loading configuration: failure
exiting (due to fatal error)
```
The resolvers (`.7`) are down until this lands; the authoritative (`.6`/`.9`) and externaldns (`.8`) are unaffected.
## Fix
```diff
- validate-except { unkin.net; 18.198.in-addr.arpa; consul }
+ validate-except { unkin.net; 18.198.in-addr.arpa; consul; }
```
Renders to `validate-except { unkin.net; 18.198.in-addr.arpa; consul; };` — valid.
## Recovery
On merge + ArgoCD sync, the operator re-renders the ConfigMap with valid config and the crash-looping pods self-heal on their next restart (no manual `rollout restart` needed). Validated: `bind-internal` renders and pre-commit clean.
Reviewed-on: #252
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Resolving any `unkin.net` record through the resolver (`.7`) returns **SERVFAIL**, while the authoritative (`.6`) answers fine. Confirmed from the resolver's querylog:
```
view openforwarder: validating unkin.net/SOA: got insecure response; parent indicates it should be secure
broken trust chain resolving 'ausyd1nxvm2120.main.unkin.net/A/IN': 198.18.200.6#53
query failed (broken trust chain)
```
The resolver runs `dnssec-validation auto`. The public `unkin.net` is DNSSEC-signed (the `.net` parent publishes a DS), but the in-cluster split-horizon authoritative serves `unkin.net` **unsigned**. The validator sees "parent says secure" + an insecure answer → treats it as spoofing → SERVFAIL. The authoritative works directly because it does no validation.
## Fix
Add `validate-except` (via `spec.extraOptions`) for the forwarded internal domains, so the resolver treats them as insecure and skips validation:
```
validate-except { unkin.net; 18.198.in-addr.arpa; consul }
```
- `unkin.net` covers all `*.unkin.net` (incl. `main.unkin.net`, `k8s.syd1.au.unkin.net`)
- `18.198.in-addr.arpa` covers every `NN.18.198.in-addr.arpa` reverse zone (subtree)
- `consul` covers the consul TLD
This also makes internal resolution independent of Internet egress (no DNSSEC chain-walk needed). External-name validation is unchanged. No operator change required.
## Validation
`bind-internal` renders and passes `kubeconform` (56/56); pre-commit clean.
## Activation
After merge + operator reconcile, the resolver ConfigMap re-renders; the running pods hold a startup snapshot, so they need a reload: `kubectl -n bind-internal rollout restart statefulset/bind-resolvers`.
Reviewed-on: #251
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.1 fixes authoritative **secondary replication**, which never actually worked — the master REFUSED the catalog AXFR. Root causes (bind-operator #9): secondaries presented no TSIG key, member zones had no `allow-transfer`, and secondaries pointed at the primary's ephemeral pod IP.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.1`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.1`.
CRDs are unchanged from v0.2.0, so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.1 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.
## Deploy note
Existing member zones pick up `allow-transfer` via `modzone`, and secondaries re-point at the stable primary Service ClusterIP with the transfer key, restoring replication without manual BIND surgery. A from-scratch namespace recreate also comes up clean (OrderedReady startup means secondaries snapshot a ClusterIP-correct config). Caveat for a full recreate: the operator regenerates the `client-update` TSIG key, so its new material must be re-synced into puppet eyaml before clients can nsupdate.
Reviewed-on: #250
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
bind-operator v0.2.0 adds the `BindTSIGAPI` CRD and a companion API that `vault-plugin-secrets-bind-tsig` calls to create, rotate and delete TSIG keys (it does so by managing `BindTSIGKey` resources, which the operator reconciles into key material). This rolls the operator forward and deploys an API instance so Vault never talks to the Kubernetes API directly.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.0`.
- Broaden the operator ClusterRole (`bind-system/rbac.yaml`) with `deployments`, `serviceaccounts` and `roles`/`rolebindings`, so the `BindTSIGAPI` reconciler can create the API Deployment and its namespaced Role/RoleBinding.
- Add a `BindTSIGAPI` (`bind-tsig-api`) in `bind-internal`; the operator reconciles it into a Deployment, Service, ConfigMap, master-token Secret and RBAC. Keys are created in `bind-internal`, alongside the authoritative cluster and its existing keys.
- Add the generated kubeconform schema for `BindTSIGAPI`.
## Notes
- The master access token Secret (`bind-tsig-api-token`) is generated by the operator when absent; the operator does not own it, so a `VaultStaticSecret` can later pre-seed/overwrite it to source the token from Vault.
- Validated: both overlays render (`kubectl kustomize`) and pass `kubeconform` (bind-internal 56/56 valid); pre-commit clean.
## Follow-up
- Point `vault-plugin-secrets-bind-tsig` config at `http://bind-tsig-api.bind-internal.svc:8443`.
Reviewed-on: #249
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Enables per-host RFC2136 updates from puppet (puppet-prod #475 profiles::dns::updater) to the bind-authoritative zones, via the .9 write endpoint.
## Changes
- add **client-update** BindTSIGKey (clusterRef bind-authoritative; operator generates the material into Secret client-update-tsig)
- set `dynamicUpdate: true` + `updateKeyRef: client-update` on all **18** authoritative zones → the operator renders `allow-update { key "client-update"; }`
## Key bridge (manual, per the TSIG plan)
The operator generates the client-update key value; it must reach puppet eyaml (`profiles::dns::updater::key_secret`) for clients to authenticate — until the planned Vault-sync/secret-reflection operator features exist. Get it with:
`kubectl -n bind-internal get secret client-update-tsig -o jsonpath='{.data.secret}' | base64 -d`
## Validated
kustomize build + kubeconform.
Reviewed-on: #244
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #239, pairing with the CNPG VMPodScrape (#242). Imports the **CloudNativePG** dashboard (grafana.com 20417) as a `GrafanaDashboard` (gzipJson, datasources resolved to the in-cluster VictoriaMetrics uid). Now that #242 collects the postgres metrics, this dashboard renders real data for all CNPG clusters.
Reviewed-on: #243
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #239. Every CNPG instance pod already exposes Prometheus metrics on `:9187` (`metrics` port), but nothing scraped them. Adds a single namespace-wide `VMPodScrape` (`namespaceSelector.any`, `selector cnpg.io/podRole=instance`) so the observability VMAgent collects postgres metrics for **all ~23 CNPG clusters** across the estate (authentik, grafana, woodpecker, artifactapi, puppet, litellm, …). No chart changes. Pairs well with the CNPG grafana.com dashboard (20417) as a further follow-up.
Reviewed-on: #242
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Follow-up to #239. Adds a `VMServiceScrape` for cert-manager's existing webhook (`metrics` :9402) and cainjector (`http-metrics` :9402) services so the observability VMAgent collects them. No chart change needed. (The controller's own metrics need `prometheus.enabled` in the chart to expose a metrics service — separate follow-up.)
Reviewed-on: #241
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Consul SD job discovered **0 targets**. vmagent logs showed it reaching the puppet Consul fine (TLS/connectivity OK) but getting `403` on `GET /v1/agent/self`: the anonymous token `lacks permission 'agent:read'`. VictoriaMetrics calls `/v1/agent/self` only to auto-detect the datacenter; catalog/health reads (what SD actually needs) work anonymously.
## Fix
Set `datacenter: au-syd1` on the consul_sd_config so VM skips the `agent/self` call. No consul token needed.
## Verify after sync
vmagent `/targets` → `consul` job shows the puppet targets (haproxy/ceph/gitea/node/…) up.
Reviewed-on: #240
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Deploys Grafana in-cluster (observability project) via the grafana-operator, mirroring the puppet Grafana but modernised — **CNPG** for state, **Authentik OIDC** for auth — and ports the live datasource + dashboards in as CRs.
Depends on: grafana-operator (#235, merged), grafana schemas (#236, merged), Authentik OIDC (terraform-authentik #2), Vault seeds (done), and `^grafana/` image proxy (terraform-artifactapi #5).
## Changes (`apps/base/grafana`)
- **CNPG** postgres Cluster + rw Pooler (db `grafana`); **VaultAuth** + **VaultStaticSecrets** pulling `postgres`/`oauth` credentials from `kv/kubernetes/namespace/grafana/default/*`.
- **Grafana CR**: postgres backend via the pooler; Authentik `generic_oauth` (client id/secret from the Vault-synced secret, openid/email/profile scopes, group→role mapping); `root_url` grafana.k8s.syd1.au.unkin.net.
- **1 GrafanaDatasource** — k8s VictoriaMetrics via the operator `vmselect-main` service; reuses the previous default datasource uid so the imported dashboards resolve unedited.
- **13 GrafanaDashboards** (gzipJson) exported from the current grafana.
- **Gateway API** (traefik-internal) + HTTPRoute for grafana.k8s.syd1.au.unkin.net.
- Registered in the observability ApplicationSet + project.
## Review notes
- OAuth `role_attribute_path` maps Authentik group `grafana-admins` → Admin, else Viewer — **confirm the group name**.
- `database.ssl_mode: require` against the CNPG pooler — adjust if the pooler isn't serving TLS.
- The `VictoriaLogs - cluster` dashboard has no in-cluster logs datasource yet (no VictoriaLogs in k8s) — included for completeness, will be empty until one exists.
- `make kubeconform` clean (24 resources, validated against the strict grafana schemas).
Reviewed-on: #238
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The k8s au-syd1 VictoriaMetrics stack ran as two helm charts and only scraped in-cluster targets. The victoria-metrics-operator already runs in vm-system, so this moves the stack onto operator-managed CRDs. That unlocks VMServiceScrape/VMPodScrape (auto-converted from Prometheus ServiceMonitors, used by a follow-up PR) and adds Consul service discovery so the cluster scrapes the **same puppet-prod targets** as the puppet vmagent. Also shrinks vmstorage 3 → 2 (Ceph-backed, replicationFactor 2).
## Changes
- Add **VMCluster `main`**: vmstorage 2 replicas (cephrbd-fast-delete 200Gi, 180d retention, replicationFactor 2), vminsert/vmselect 2 replicas + HPA (2–10, 60% cpu).
- Add **VMAgent `main`**: retains the kubernetes SD jobs (apiservers/nodes/cadvisor), `selectAllByDefault` for VMServiceScrape/VMPodScrape, and a **Consul SD job** against `consul.service.consul` (resolves to the puppet Consul from pods) replicating the puppet vmagent relabels — keep tag `metrics`, `__scheme__` from `metrics_scheme`, `job` from `metrics_job`. TLS is **verified against the reflected `vault-ca-cert`** (no insecure skip-verify).
- Expose vmselect/vminsert/vmagent via **Gateway API** (traefik-internal Gateway + HTTPRoute, http→https redirect), same hostnames.
- Remove the two helm charts, their values files, and vendored charts.
## Notes
- Data wipe on cutover is acceptable (confirmed) — old helm PVCs can be deleted.
- Verify at rollout: pods resolve `*.main.unkin.net` node FQDNs (needed for CA SAN match on scrape targets); `/targets` shows `job=consul`.
Reviewed-on: #234
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Adds the grafana-operator (grafana.integreatly.org CRDs + controller) so
Grafana and its dashboards/datasources can be managed declaratively as
CRs in a follow-up PR. Sits in the platform project like the other
operators (vm-system, cnpg-system).
Changes:
- Add grafana-system namespace + grafana-operator helm chart v5.24.0
(watches all namespaces).
- Render CRDs inline (crds.immutable: false) so ArgoCD installs/manages
the 13 grafana.integreatly.org CRDs instead of the skipped helm crds/
subchart.
- Register apps/overlays/*/grafana-system in the platform ApplicationSet.
## Why
encapi is the new Postgres-backed Puppet ENC that replaces Cobbler (Go API + encapi-cli + terraform provider). It needs to run somewhere reachable by the puppet masters (`encapi-cli classify`) and every node's `enc_direct_facts` fact. Deploy it in k8s alongside artifactapi, exposed at `encapi.k8s.syd1.au.unkin.net`.
## Changes
- add `apps/base/encapi/`: namespace, deployment (`git.unkin.net/unkin/encapi`, port 8000, `/healthz` probes), service, gateway + httproute (`encapi.k8s.syd1.au.unkin.net`, traefik-internal), configmap (DB coordinates), CNPG cluster + pooler (database `encapi`), and VaultAuth + VaultStaticSecrets (`postgres-credentials`, `environment`)
- add `apps/overlays/au-syd1/encapi` overlay referencing the base
- register `apps/overlays/*/encapi` in the platform ApplicationSet so ArgoCD picks it up
## Notes
- Mirrors the artifactapi pattern (VaultAuth role `default`, namespace-scoped VSO paths `kv/kubernetes/namespace/encapi/default/*`).
- Before first sync, seed the Vault KV secrets: `environment` must carry `DBPASS` (matching the CNPG owner password) and `ENCAPI_WRITE_TOKEN`; `postgres-credentials` carries the CNPG owner username/password.
- `kustomize build apps/overlays/au-syd1/encapi` validates clean (11 resources).
---------
Co-authored-by: unkinben <neotheo@gmail.com>
Reviewed-on: #230
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Roll out artifactapi `v3.7.5`, which ships the local docker registry (artifactapi#103): local `docker` repos now serve the Docker Registry HTTP API V2 for push and pull.
## Changes
- `apps/base/artifactapi/api-deployment.yaml`: `artifactapi` image `v3.7.4` → `v3.7.5`
- `apps/base/artifactapi/ui-deployment.yaml`: `artifactapi-ui` image `v3.7.4` → `v3.7.5`
## Heads-up (follow-up needed)
The API HPA runs `minReplicas: 2`. Local-docker **chunked** blob uploads keep the upload session in-memory per replica, so a real `docker push` (POST → PATCH → PUT across replicas, no session affinity) can intermittently 404 with `BLOB_UPLOAD_UNKNOWN`. Monolithic pushes are unaffected. Recommend a follow-up to make upload sessions replica-independent (S3-backed) or add session affinity for `/v2/*/blobs/uploads/` before relying on pushes in anger.
Reviewed-on: #231
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**Stacked on #228** (needs operator v0.1.5). Merge #228 first; the diff collapses to just this after.
## Why
Writes (RFC2136/nsupdate) must go to pod-0 — the round-robin read Service would land them on a secondary (rejected). Adds a dedicated write endpoint per cluster (operator v0.1.5 `primaryService`).
## Changes
- `bind-authoritative`: LoadBalancer write endpoint on **198.18.200.9** (`bind-authoritative-primary`)
- `bind-externaldns`: ClusterIP write endpoint (`bind-externaldns-primary`, for in-cluster writers)
- regenerate the bindcluster kubeconform schema (primaryService + externalTrafficPolicy)
## Deferred
external-dns is **not** repointed at `bind-externaldns-primary` yet: it authenticates with the existing TSIG key, which the operator-generated key won't match until the planned Vault-sync + secret-reflection features exist. Until then external-dns keeps writing to the puppet externaldns.
## Validated
kustomize build + kubeconform (3 BindClusters valid against the v0.1.5 schema).
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #229
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- bump operator to v0.1.5 (CRD link + image)
Reviewed-on: #228
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Mirrors the puppet authoritative `master-zones` view (match-clients `acl-main.unkin.net`, recursion no) — restricting who can query bind-authoritative.
## Changes
- add `auth-acl-main` BindACL with the puppet authoritative acl-main.unkin.net networks (13-17,19,20,24-29)
- `allow-query { auth-acl-main; 10.42.0.0/16; }` on bind-authoritative via extraOptions
## Notes
- Implemented as a global `allow-query` rather than a BindView: dynamic *primary* zones inside a view would need per-view `allow-new-zones` (an operator gap). Functionally equivalent for the single master-zones view.
- `10.42.0.0/16` (pod network) is included so secondaries can SOA-refresh from the primary during catalog replication — without it, replication breaks.
- Works on the current operator (no HOLD).
## Caveat
The DNS Services use externalTrafficPolicy: Cluster, which SNATs external clients to node IPs (198.18.19.x, already in acl-main), so this ACL doesn't truly restrict *external* clients yet. True source-IP restriction needs externalTrafficPolicy: Local — happy to switch if wanted.
Reviewed-on: #227
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`dig google.com @198.18.200.7` was refused: the resolver never set allow-recursion, so BIND defaulted to localnets/localhost. This mirrors the puppet resolver (/etc/named/views.conf + acls.conf) exactly.
## Changes
- `openforwarder` BindView: `match-clients` = the 4 internal ACLs, recursion yes, allow-recursion/allow-query `any` (match-clients gates)
- 4 BindACLs from puppet acls.conf (acl-main.unkin.net/acl-dmz/acl-common/acl-nomad-jobs)
- 26 conditional forward zones in the view (unkin→198.18.19.15, consul→.14, k8s→.20, dmz/network/prod + 10.10.x reverse → 10.10.16.32/33)
- global forwarders 8.8.8.8/1.1.1.1
- operator image → v0.1.4
## Note
Forward-zone upstreams point at the **puppet anycast** servers (still authoritative during migration); flip to the in-cluster authoritative/externaldns LBs once zone data is migrated.
## Validated
kustomize build (59 docs), kubeconform clean.
Reviewed-on: #226
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**HOLD until v0.1.3 is tagged/built** (operator #4 merged + tagged) — this PR bumps the operator to v0.1.3, whose CRD adds the `clusterRef` field these keys use.
## Why
Put all BIND DNS services in one `bind-internal` namespace and name the StatefulSets clearly.
## Changes
- 3 clusters consolidated into `bind-internal`, StatefulSets renamed **bind-authoritative** / **bind-resolvers** / **bind-externaldns**; LBs kept on 198.18.200.6/.7/.8; external-dns hostnames renamed to match
- `clusterRef` added to `transfer-key` (→ bind-authoritative) and `externaldns-key` (→ bind-externaldns) so keys are scoped per cluster
- removed the old `ns-auth`/`ns-resolver`/`ns-externaldns` apps; ApplicationSet + AppProject now list `bind-internal`
- bumped `bind-system` operator to **v0.1.3** (CRD link + image)
- operator stays in `bind-system`
## Deploy impact
ArgoCD prunes the old ns-* namespaces (StatefulSets/PVCs — data is only seed SOA+NS, no migrated records yet) and creates the renamed clusters in bind-internal.
## Validated
`kustomize build` → 28 docs (3 BindCluster, 20 BindZone, 2 catalog, 2 keys, ns); kubeconform clean.
Reviewed-on: #225
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Part of the bind rollout split. **Merge #219 (bind-operator) first** — stacked on it; diff reduces to the binddns-externaldns files once #219 merges.
## Why
The external-dns tier (replaces 3x Puppet external-dns servers): an authoritative cluster whose zones accept RFC2136 TSIG updates from external-dns.
## Changes
- `apps/base/binddns-externaldns`: authoritative `BindCluster` (3 replicas, LoadBalancer/PureLB), `BindTSIGKey` for RFC2136, namespace
- au-syd1 `binddns-externaldns` overlay
## Deploy impact
Creates the `binddns-externaldns` StatefulSet + LoadBalancer once merged.
Reviewed-on: #222
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkinben
benvin