Three changes from review:
1. Pull every container image through the artifactapi dockerhub remote instead
of direct upstream: clickhouse-server, altinity operator + metrics-exporter,
bitnami/kubectl (crdHook), nats + nats-server-config-reloader, nats-box
(bootstrap Job), and vector (all tiers + the CI image). Requires
terraform-artifactapi#16 (dockerhub allowlist patterns) merged first.
2. Keep upstream official images (no Docker Hardened Images). DHI exists for
clickhouse-server and vector but is subscription-gated and served from a
private org namespace not reachable via the anonymous artifactapi dockerhub
proxy; its shell-less images would also break the bash bootstrap Jobs and the
shell-based vector-test CI step. Use vector's distroless-libc for runtime
pods (near-hardened) and the debian variant only for CI.
3. Make the transform tier a stateless Deployment (was a StatefulSet): no PVC,
no disk buffer — JetStream is the sole durability layer. The ClickHouse sink
uses an in-memory block buffer so a ClickHouse outage back-pressures the
JetStream pull source (unpulled messages are retained/redelivered). Add a CPU
HPA (2-8) — safe because JetStream pull consumers distribute work across N
replicas on the one durable consumer. Caveat documented: vector's NATS source
has no end-to-end acks (acks on receipt), so a pod killed mid-outage can lose
its in-memory buffer window; accepted trade for a stateless autoscaling tier.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Rework the logging pipeline around a durable message bus so logs survive a
ClickHouse outage, can be replayed after a bad transform, and fan out to
multiple independent consumers. Add long-term raw-log backup to S3.
Topology becomes edge -> JetStream -> consumers -> sinks:
- Dedicated JetStream NATS cluster (3 replicas, file storage) in the logging
namespace. Deliberately separate from app messaging (streamstack) for
blast-radius isolation. Stream LOGS (subjects logs.>, retention=limits, 40GiB
/ 72h) is the outage buffer; durable consumers give independent offsets.
- Edge publishers (thin): the k8s DaemonSet and a new VM-ingest Deployment
(HTTP NDJSON front door behind the logs-ingest Gateway) publish into JetStream
(logs.k8s.<ns>.<container> / logs.vm.<host>). No parsing on the edge.
- Transform tier (StatefulSet): pulls the whole stream via the durable
`transform` consumer, routes by subject, shapes, and remains the sole
ClickHouse writer. Its disk buffer shrinks (JetStream is the outage buffer).
- Archiver (Deployment): its OWN durable `archiver` consumer (independent
offsets — archive lag never affects the ClickHouse path) writes RAW,
pre-transform events to a Ceph RGW S3 bucket (cephrgw-operator ObjectStoreUser
+ Bucket + BucketAccess) as gzipped NDJSON keyed by raw/<subject>/YYYY/MM/DD/.
Default subject filter is Vault audit (logs.k8s.vault.>), configurable.
Auth: distinct NATS users (producer publish-only, consumer pull+ack, admin for
the stream/consumer bootstrap Job) with passwords from Vault (nats-auth Secret);
S3 creds from the BucketAccess Secret. Streams/consumers are provisioned by an
idempotent PostSync bootstrap Job.
Add local kubeconform schemas for the ceph.unkin.net CRDs (datreeio lacks them)
and extend the vector-test CI to cover the agent, VM-ingest and archiver
configs. Verified end-to-end locally: NATS ACLs, vector JetStream publish, and
durable-consumer pull+ack (at-least-once) all work.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Stand up a centralized logging estate that captures ALL logs from k8s pods and
(via a reachable ingestion endpoint) puppet-managed VMs, storing them in
ClickHouse for query/retention. Metrics already live in VictoriaMetrics; this
adds the logs pillar under a dedicated `logging` ArgoCD project.
Deploy the Altinity clickhouse-operator (clickhouse-system) and a single-shard
ClickHouseInstallation (logging) on cephrbd-fast-delete with a MergeTree
logs.raw table (30d TTL) bootstrapped by an idempotent PostSync Job.
Deploy Vector as an explicit two tiers:
- Edge (thin): a DaemonSet tails every node's pod logs and forwards over the
Vector-native protocol to the aggregator; no parsing at the edge. Future VM
agents follow the same thin pattern.
- Aggregator (brain): HA StatefulSet that is the sole ClickHouse writer, holds
the only ClickHouse credentials, owns all transforms, batches into few fat
inserts (avoid too-many-parts), and buffers to disk (PVC) to ride out a
ClickHouse outage. Its pipeline is a single source-of-truth config validated
by `vector test` in CI; per-app pipelines become aggregator-only changes.
Expose the VM ingestion endpoint at logs-ingest.k8s.syd1.au.unkin.net via the
internal Traefik gateway (cert-manager + external-dns), routing to the
aggregator's HTTP source so puppet VMs can reach it over TLS.
Source ClickHouse credentials from Vault via the existing VaultStaticSecret
pattern (templated k8s auth policy already grants the logging namespace);
password hash never lands in git.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Follow-up to #294: --create-ptr validates that PTR is in --managed-record-types, and setting that flag replaces the default list, so A/AAAA/CNAME are re-stated alongside PTR (verified against the v0.21.0 binary's --help: default A,AAAA,CNAME). Pod is crash-looping on config validation until this merges.
- Adds --managed-record-types=A,AAAA,CNAME,PTR to the external-dns args
MERGE ASAP.
Reviewed-on: #295
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
external-dns v0.21.0 rejects --rfc2136-create-ptr (my error in #292) and the pod is crash-looping on flag parsing. The correct flag in this version is the generic --create-ptr, which synthesizes PTRs for A records whose reverse zone is in the domain filter (200.18.198.in-addr.arpa already is).
- Replaces --rfc2136-create-ptr with --create-ptr
Record reconciliation is stalled until this merges (serving unaffected — bind answers normally). MERGE ASAP.
Reviewed-on: #294
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Post-cutover verification (argocd-apps#288) found the reverse zone 200.18.198.in-addr.arpa empty: external-dns only writes PTRs when --rfc2136-create-ptr is set, and nothing else feeds that zone. Reverse resolution for the k8s LB range was already absent for clients pre-migration (no forwarder existed), so this completes the reverse path rather than fixing a regression.
- Adds --rfc2136-create-ptr to the external-dns rfc2136 args
Verification after merge: dig -x 198.18.200.2 @198.18.200.7 returns puppetca.k8s.syd1.au.unkin.net (allow a reconcile cycle + negative-cache expiry).
Reviewed-on: #292
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy bind-operator v0.2.6 (bind-operator#15): NOTIFYs are now TSIG-signed via the catalog transfer key and secondaries accept by key, with no pod IPs in restart-scoped config — a regression test asserts the config-hash is invariant under pod IP churn, making the v0.2.5 roll-loop class impossible. Restores seconds-fast dynamic-zone propagation on bind-externaldns and bind-authoritative.
- Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.6 (confirmed in registry)
- Bumps the CRD install pin to the v0.2.6 tag
Expect exactly ONE settling roll of the bind statefulsets when the new config lands, then stability.
Reviewed-on: #293
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
v0.2.5 renders the primary POD IP into the options-scope allow-notify. Options changes are restart-scoped (config-hash annotation), and every roll gives the primary a new pod IP, so the operator re-renders and rolls all bind clusters in an endless loop (externaldns, authoritative, resolvers all cycling ~45s pods right now).
- Reverts the operator image and CRD pin to v0.2.4
A v0.2.6 will re-do the NOTIFY fix loop-free (allow-notify via zone-scope/rndc-applied config or a TSIG-keyed notify instead of pod-IP-in-options). MERGE ASAP to stabilize DNS.
Reviewed-on: #291
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 3 of 3 in the external-dns → in-cluster bind migration: the client-visible cutover. The `openforwarder` resolvers currently forward `k8s.syd1.au.unkin.net` to the legacy VM anycast `198.18.19.20` (a temporary measure — commit 7ee5dfb) and have NO forwarder at all for the reverse zone `200.18.198.in-addr.arpa`. Once external-dns publishes to the in-cluster `bind-externaldns` (PR 2), resolvers must read from it.
## Changes
- Repoints the `fwd-k8s-syd1-au-unkin-net` forwarder from `198.18.19.20` (legacy VM) to `198.18.200.8` (in-cluster `bind-externaldns` VIP).
- Adds `fwd-200-18-198-in-addr-arpa` forwarding `200.18.198.in-addr.arpa` → `198.18.200.8`, closing the reverse-zone gap so PTR lookups for the k8s LB range keep resolving after cutover. Modeled exactly on the existing forward-zone entries.
- Refreshes the header comment to describe the in-cluster upstream.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean; both zones render with forwarder `198.18.200.8` and there is no residual `198.18.19.20`.
## Merge gate
- PR 2 (`benvin/externaldns-incluster`) merged, AND
- record parity confirmed between legacy and in-cluster for the forward zone. Spot-check (repeat for each name):
```
for n in puppetca puppet puppetdb encapi pdbmux artifactapi consul; do
echo "$n:"
dig +short @198.18.19.20 A $n.k8s.syd1.au.unkin.net
dig +short @198.18.200.8 A $n.k8s.syd1.au.unkin.net
done
# plus 2-3 PTRs in the reverse zone:
dig +short @198.18.19.20 -x 198.18.200.8
dig +short @198.18.200.8 -x 198.18.200.8
```
A/PTR answers from `198.18.200.8` must match those from `198.18.19.20` before merging.
## Verification (after merge)
```
dig +short @198.18.200.7 A puppet.k8s.syd1.au.unkin.net # resolvers VIP
dig +short @198.18.200.7 -x 198.18.200.8 # reverse via resolvers
```
Resolution through the `bind-resolvers` VIP should now answer for both the forward and reverse k8s zones.
## Rollback
Revert this PR — the `fwd-k8s-syd1-au-unkin-net` forwarder returns to `198.18.19.20` and the reverse forward is removed. The legacy VM is untouched and remains authoritative until decommission.
Reviewed-on: #288
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator **v0.3.1** logs a startup WARNING when its installed CRDs are missing or older than the operator (operator repo #6, merged) — added precisely because the CRD tag drifted behind the image before. The new startup check reads the `CustomResourceDefinition` objects, so it needs a small RBAC grant.
## Changes
- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.1`
- bump the CRD `install.yaml` tag → `v0.3.1` (keep CRDs in step with the image — the invariant the v0.3.1 warning enforces)
- add `apiextensions.k8s.io/customresourcedefinitions: [get, list]` to the operator ClusterRole so the startup check is not RBAC-denied
Validated with `kustomize build` on the au-syd1 overlay; the v0.3.1 CRD URL resolves. Supersedes nothing outstanding (the earlier CRD-tag PR #286 to v0.3.0 already merged).
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #290
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy bind-operator v0.2.5 (bind-operator#14): secondaries now carry an explicit allow-notify for the primary pod IP, so dynamic-zone updates propagate in seconds instead of the ~1h SOA refresh. Unblocks the external-dns migration parity gate (argocd-apps#288) and speeds up the dns-updater zones on bind-authoritative.
- Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.5 (confirmed in registry)
- Bumps the CRD install pin to the v0.2.5 tag
On sync the operator re-renders cluster ConfigMaps; the config-hash change rolls the bind secondaries, which then accept the primary's NOTIFYs.
Reviewed-on: #289
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 2 of 3 in the external-dns → in-cluster bind migration. external-dns currently pushes RFC2136 updates to the legacy VM `ausyd1nxvm2127.main.unkin.net`. This repoints it at the in-cluster `bind-externaldns` primary so the in-cluster zone becomes the live source of truth for `k8s.syd1.au.unkin.net` + `200.18.198.in-addr.arpa`.
## Changes
- Points `--rfc2136-host` at `bind-externaldns-primary.bind-internal.svc.cluster.local` (verified live: ClusterIP Service `bind-externaldns-primary` exists in `bind-internal`).
- Reads TSIG `secret` + `algorithm` from Secret `externaldns-key-tsig` (reflected by PR 1) instead of the Vault-backed `externaldns-tsig`.
- Keeps port, zones, keyname, and `txtOwnerId: k8s` unchanged.
- Leaves the old Vault manifests (`apps/base/externaldns/{vaultauth,vaultstaticsecret}.yaml`) in place as the rollback path; their removal is a later cleanup PR.
## Merge gate
- PR 1 (`benvin/externaldns-tsig-reflect`) merged, AND
- `kubectl -n externaldns get secret externaldns-key-tsig` returns keys `secret` + `algorithm`.
## TXT-registry note
`policy: sync` + `registry: txt`: on first reconcile against the (currently empty) in-cluster zone, external-dns re-creates all managed A/CNAME records and their ownership TXTs from scratch. This is expected and populates the zone.
## Verification (after merge)
```
kubectl -n externaldns logs deploy/externaldns --tail=100 | grep -Ei 'rfc2136|BADKEY|NOTAUTH|added|update'
dig +short @198.18.200.8 A puppet.k8s.syd1.au.unkin.net
```
Logs should show updates to `bind-externaldns-primary...` with NO `BADKEY`/`NOTAUTH`; sampled A records should start resolving against the in-cluster server (198.18.200.8) as the sync populates the zone.
## Rollback
Revert this PR (host + secret name back to `ausyd1nxvm2127.main.unkin.net` / `externaldns-tsig`). The legacy VM is untouched and still authoritative.
Reviewed-on: #287
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Step 1 of 3 in the external-dns → in-cluster bind migration. For external-dns to send RFC2136 updates to the in-cluster `bind-externaldns` primary, it must present the exact TSIG key the primary's `allow-update` accepts. The bind-operator generates that key material into Secret `externaldns-key-tsig` in `bind-internal`; reflecting it into the `externaldns` namespace removes the manual eyaml→Vault key sync and guarantees key parity.
## Changes
- Adds `spec.secretTemplate.annotations` to BindTSIGKey `externaldns-key` with the emberstack reflector hints: `reflection-allowed`, `reflection-allowed-namespaces: externaldns`, `reflection-auto-enabled`, `reflection-auto-namespaces: externaldns`.
- Regenerates `schemas/bind.unkin.net/bindtsigkey_v1alpha1.json` from the live CRD (deployed bind-operator v0.2.4 already exposes `secretTemplate` — the WIP branch's ">= v0.3.0" claim is stale) to add the `secretTemplate` property. Schema output is byte-identical to running `ci/generate-schemas.sh`.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean and renders the annotations onto the BindTSIGKey.
## Verification (after merge)
```
kubectl -n externaldns get secret externaldns-key-tsig \
-o jsonpath='{.data.secret} {.data.algorithm}{"\n"}'
```
Both `secret` and `algorithm` keys must be present (reflector mirrored the source Secret from bind-internal).
## Rollback
Revert this PR. The source Secret in bind-internal is unaffected; only the reflected mirror in `externaldns` is removed.
Merge order: this is PR 1/3. PR 2 (repoint external-dns) must not merge until the reflected secret is verified.
Reviewed-on: #285
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The `cephrgw-system` kustomization pinned the CRD source to **`raw/tag/v0.1.0/config/crd/install.yaml`**, so the in-cluster CRDs never gained the fields added since v0.1.0 — v0.2.0's fine-grained BucketAccess policy fields (`paths`/`actions`/`conditions`/`rawStatements`) and v0.3.0's adoption fields (`retainOnDelete`, `managePolicy`, `status.adopted`). The running operator is v0.3.0, so applying those specs fails with `strict decoding error: unknown field`. The image bumps (#273, #279) should have moved this tag too.
## Changes
- point the CRD `install.yaml` at `raw/tag/v0.3.0`
The tag must track the operator image tag on future bumps. Verified the v0.3.0 URL serves the new schema and `kustomize build` renders all 3 CRDs.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #286
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Turns on right-sizing telemetry for the whole estate. Adds a `VerticalPodAutoscaler` with `updateMode: "Off"` (recommendation-only, advise mode) for every Deployment and StatefulSet in `apps/base`. Off mode never evicts or mutates pods, so this is purely observational: the VPA recommender (added in the vpa-system PR) publishes suggested requests/limits in each VPA's status, and nothing acts on them until someone deliberately flips a mode.
## Changes
- Add one `vpa.yaml` per app under `apps/base/<app>/` containing a `<workload>-vpa` VerticalPodAutoscaler for each workload, and register it in that app's `kustomization.yaml`.
- Coverage: 17 workloads across 11 apps.
- age-api (age-api), artifactapi (api, redis, ui), authentik (redis), bind-system (bind-operator), cephrgw-system (cephrgw-operator), encapi (encapi), kanidm (kanidm StatefulSet), litellm (litellm, redis), paperclip (paperclip), pdbmux (pdbmux), puppet (puppetboard, puppetdb, puppetserver-compiler, puppetserver-master).
## Skipped (intentionally)
- **CNPG `Cluster` objects** (artifactapi, authentik, encapi, grafana, litellm, paperclip, puppet, woodpecker) — Postgres is managed by CloudNativePG, not a VPA target.
- **CronJobs** (puppet g10k/generate-types, reposync x4) — not VPA-able.
## HPA / VPA caveat
api, ui (artifactapi), litellm, and all four puppet deployments also carry an HPA. With `updateMode: "Off"` there is no conflict today (VPA only recommends). VPA objects targeting these carry an inline comment: do **not** flip to `Auto`/`Initial` while the HPA still autoscales on CPU/memory, or the two controllers will fight over the same resource. Move the HPA to a custom/non-resource metric first.
## Verification
- `kubectl kustomize` over every touched overlay: 9/11 overlays PASS rendering all their VPAs; 2 failures are pre-existing and unrelated to this change — `authentik` fails on a local helm-tooling flag error identically on origin/main, and `paperclip` has no `apps/overlays/au-syd1/paperclip` directory yet. Both apps' base kustomizations build clean and render their VPA.
## Merge gate
- **Requires the vpa-system CRDs PR (argocd-apps #281) to merge first.** These manifests use `autoscaling.k8s.io/v1 VerticalPodAutoscaler`; without the CRDs installed, ArgoCD sync fails on an unknown kind.
Reviewed-on: #283
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Rolls out the Vertical Pod Autoscaler control plane so the estate can gather right-sizing recommendations for every workload (advise mode, follow-up PR adds the per-workload VPA objects). Deploys the **recommender only**: advise mode never mutates pods, so the updater and admission-controller (and its mutating webhook) are intentionally omitted — fewer moving parts, no webhook in the admission path.
## Changes
- Add `apps/base/vpa-system/`: namespace, VPA CRDs (verticalpodautoscalers + verticalpodautoscalercheckpoints) pulled from the kubernetes/autoscaler repo at the pinned tag (same upstream-raw pattern node-feature-discovery uses), recommender-scoped RBAC (SA + metrics-reader/actor/status-actor/checkpoint-actor/target-reader), and the recommender Deployment.
- Add `apps/overlays/au-syd1/vpa-system/` referencing the base.
- Register `apps/overlays/*/vpa-system` in the platform ApplicationSet.
## Notes
- Pins upstream **vertical-pod-autoscaler-1.7.0** (latest stable, 2026-05-29) for both CRDs and the `registry.k8s.io/autoscaling/vpa-recommender:1.7.0` image.
- No platform AppProject change needed: `*-system` namespace destination plus CustomResourceDefinition/ClusterRole/ClusterRoleBinding are already whitelisted.
- The recommender image is pull-through-cached via terraform-artifactapi PR #13 (merge gate below).
## Merge gate
- Requires terraform-artifactapi PR #13 (adds the `autoscaling/vpa-` pattern to the k8s-registry docker remote) to land first so the recommender image is served through the cache.
Reviewed-on: #281
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Phase 2 of the consul migration: expose the HTTP API (not just the UI) at consul.k8s.syd1.au.unkin.net, now rebased onto main post-#280 (ACLs enabled).
- Adds a consul-http ClusterIP service targeting the server pods on 8500 (API + UI share the port, so the UI stays reachable at /ui/)
- Repoints the consul and consul-svc HTTPRoutes from consul-ui:80 to consul-http:8500
- Documents ACL-authenticated access in apps/base/consul/README.md: token from kv/kubernetes/namespace/consul/default/bootstrap-acl-token (VSO-synced), X-Consul-Token curl and consul CLI usage, UI token login, and the prefer-vault-minted-tokens note
Verification post-merge (ACLs are live, so authenticated): CONSUL_HTTP_TOKEN=$(vault kv get -field=token kv/kubernetes/namespace/consul/default/bootstrap-acl-token) && curl -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" https://consul.k8s.syd1.au.unkin.net/v1/status/leader
Rollback: revert the HTTPRoute backends to consul-ui:80.
Reviewed-on: #282
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Split out of #281 per review: the VerticalPodAutoscaler CRDs land first, together with their generated kubeconform schemas, so CI can validate the VPA objects that follow.
- Adds apps/base/vpa-system with the v1.7.0 VPA CRDs served via the artifactapi github remote (terraform-artifactapi#14, merged)
- Adds schemas/autoscaling.k8s.io/ (verticalpodautoscaler + checkpoint, v1 and v1beta2) generated with the same transform as ci/generate-schemas.sh (from the CRD manifest rather than the live cluster, since the CRDs are not installed yet)
- Wires the vpa-system overlay into the platform applicationset
Verified: kustomize renders both CRDs; a sample updateMode Off VPA passes kubeconform against the new schemas. Merge before #281 (recommender) and #283 (per-workload VPAs).
Reviewed-on: #284
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with `default_policy: deny` and `down_policy: extend-cache`; the k8s cluster currently runs with **ACLs disabled**. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the **same** `initial_management` token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git.
## Changes
- Enable `global.acls.manageSystemACLs` so the chart manages system ACL tokens/policies for consul components.
- Point `global.acls.bootstrapToken` at a pre-existing Kubernetes secret `consul-bootstrap-acl-token` (key `token`); chart 1.9.7 supports this, and when the secret is populated the `server-acl-init` job **skips bootstrapping** and adopts that token as the management token (renders `-bootstrap-token-secret-name`/`-bootstrap-token-secret-key`, verified in the kustomize output).
- Add a `VaultAuth` (mount `k8s/au/syd1`, role `default`, SA `default`) and `VaultStaticSecret` in the `consul` namespace that sync `kv/kubernetes/namespace/consul/default/bootstrap-acl-token` into the `consul-bootstrap-acl-token` k8s secret via VSO (mirrors the encapi pattern).
- Merge the `acl` block (`enabled`, `default_policy: deny`, `down_policy: extend-cache`, `enable_token_persistence`) into the server `extraConfig` to match the VM posture.
## OPERATIONAL NOTE — required BEFORE merge
The user MUST place the VM cluster's `initial_management` token in Vault first, or ACL bootstrapping will generate a *different* token and break the mirror:
```
vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token>
```
VSO then syncs it into the `consul-bootstrap-acl-token` secret before the `server-acl-init` job runs. No terraform-vault change is needed: the wildcard `default` k8s-auth role (`bound_service_account_namespaces: ['*']`) plus the templated `kv/kubernetes/default` policy already grant the `consul` namespace `default` SA read on `kv/kubernetes/namespace/consul/default/*`.
## Risk / expected behavior
- Enabling ACLs **rolls the 5 servers** (StatefulSet update) and runs a `server-acl-init` job.
- With `default_policy: deny`, previously-anonymous operations are denied. The `vault` service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace `vault`) via their `service_registration "consul"` stanza (catalog entry has `ServiceMeta.external-source: vault`, port 8200). After the flip this registration will be **denied** unless Vault is given a Consul ACL token with `service:write` on `vault` (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration.
- Anonymous HTTP API reads (e.g. `/v1/status/leader`) will also be denied post-merge unless a token is supplied — see PR 2 verification note.
## Ordering
Independent of the API-route PR (`benvin/consul-api-route`). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done.
Reviewed-on: #280
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator **v0.3.0** makes adopting pre-existing radosgw buckets/users safe (operator repo #5, merged): `retainOnDelete` on `ObjectStoreUser`/`BucketAccess`, non-destructive bucket-policy **merge** (+ `managePolicy`), non-destructive user attributes, and `status.adopted`.
## Changes
- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.0`
No credential or manifest change beyond the tag. Validated with `kustomize build` on the au-syd1 overlay.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #279
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the two new UI features released in v3.7.7: per-repo usage-instruction panels (artifactapi#105) and direct-download links for local repo files (artifactapi#106).
- Bumps artifactapi api image to git.unkin.net/unkin/artifactapi:v3.7.7
- Bumps artifactapi ui image to git.unkin.net/unkin/artifactapi-ui:v3.7.7
Both v3.7.7 images are confirmed published to the registry; kustomize builds clean.
Reviewed-on: #278
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The puppet-on-k8s compilers classify nodes with a uv/python ENC script (`encapi-enc`). Each fresh compiler pod resolves the script's python dependencies on first invocation, and that resolution fails on cold pods (observed exits 135/2), breaking puppet agent catalog compilation. `encapic` (git.unkin.net/unkin/encapic) is a stdlib-only Go replacement with no runtime dependency resolution — a behavioural drop-in whose output matches the python script byte-for-byte.
## Changes
- Points the compiler `external_nodes` at `/opt/bin/encapic`.
- Reworks the `setup-shared-bins` init container to `curl` the encapic `v0.1.0` `encapic_linux_amd64` release binary (sha256-verified against the published `.sha256`, installed mode 0755) into the shared bins dir, instead of copying the python script and installing uv.
- Removes the `puppet-encapi-enc` configmap generator, its volume and mount, and the `resources/encapi-enc` script. uv was consumed solely by that script (grep of `apps/base/puppet` confirms no other consumer), so its installation is removed too.
`kubectl kustomize apps/overlays/au-syd1/puppet` builds clean.
## Merge gate
Do not merge until the encapic `v0.1.0` release assets exist:
`https://git.unkin.net/unkin/encapic/releases/download/v0.1.0/encapic_linux_amd64` (+ `.sha256`). The init container pulls them at pod start.
## Rollback
Revert this PR to restore the `encapi-enc` configmap script + uv install and repoint `external_nodes`.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #277
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The k8s puppetserver compilers classify nodes via an exec ENC that today queries legacy Cobbler (`https://cobbler.main.unkin.net`) over TLS. `encapi` now runs in-cluster and exposes a cobbler-wire-compatible endpoint (`GET /cblr/svc/op/puppet/hostname/<certname>`), a drop-in for the Cobbler URL. This cuts the puppet-on-k8s ENC over from Cobbler to encapi — a prerequisite for migrating VM agents onto puppet-on-k8s.
## Changes
- Rename the ENC script `resources/cobbler-enc` -> `resources/encapi-enc`, and its configmap `puppet-cobbler-enc` -> `puppet-encapi-enc` (kustomization configMapGenerator + deployment volume, initContainer copy path, and volumeMount subPath).
- Point `external_nodes` in the compiler `puppet.conf` at `/opt/bin/encapi-enc`.
- Target the in-cluster encapi service `http://encapi.encapi.svc.cluster.local` (plain HTTP), overridable via the `ENCAPI_URL` env var.
- Drop the `/opt/vault-ca-cert.crt` verify for the ENC request (no TLS needed in-cluster).
- Leave the response normalization identical: classes coerced to a list, `enc_role`/`enc_env` params set, `environment` stripped when it equals `testing`.
Verified with `kubectl kustomize apps/overlays/au-syd1/puppet` (builds clean, exit 0); the generated `puppet-encapi-enc` configmap contains the new URL and env var.
## 🚨 Merge gate
**Do not merge until encapi is seeded** (terraform-incus `benvin/encapi-seed` PR applied). An empty encapi means every node resolves to a 404. On 404 the ENC script exits non-zero, so puppet fails the compile rather than classifying the node with zero classes — nodes will fail to run until they exist in encapi. Seed encapi first so real nodes classify correctly; only unknown nodes should 404.
Reviewed-on: #272
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move
between them as they migrate. `node-lookup` (and `pblastreport`) need a single,
consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy
that provides exactly that. Per the all-in-kubernetes estate direction it runs
as an in-cluster service, not a per-VM systemd unit.
pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) —
split out of the earlier node-lookup prototype — and is released as a container
image on its own `v*` tags.
## Changes
- Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway,
httproute), modeled directly on the encapi app.
- Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080,
`/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap.
- Backends: `old=http://puppetdbapi.service.consul:8080`,
`new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified
against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the
in-cluster address is preferred over the external gateway). `new` is
primary/prefer, merge = freshness.
- Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal`
Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a
port-80 Service — same shape as the puppetdb/encapi gateways — so
VM/workstation `node-lookup` can reach it.
- Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform
ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject
(`pdbmux` namespace destination), exactly as encapi is wired.
No new woodpecker ServiceAccount is required: the pdbmux image push uses the
`docker-buildx` plugin against the Gitea registry with the `default` SA (same as
encapi), not artifactapi.
## Verification
- `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves
to `git.unkin.net/unkin/pdbmux:v0.1.0`).
- ApplicationSet + AppProject YAML validated.
## Merge gates
1. The pdbmux repo initial-content PR
(unkin/pdbmux#1) must merge first.
2. `v0.1.0` must then be tagged on the pdbmux repo so the image
`git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s
`.woodpecker/docker.yaml`.
3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update
the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.)
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #275
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator v0.2.0 talks to radosgw over HTTPS (`radosgw.service.consul:443`, fronted by nginx presenting the internal `unkin.net` Vault-PKI cert). With no CA configured the operator fails:
```
Get "https://radosgw.service.consul:443/admin/user?...": tls: failed to verify certificate: x509: certificate signed by unknown authority
```
The `vault-ca-cert` Secret (the `unkin.net` intermediate+root) is already reflected into every namespace — including `cephrgw-system` — so the fix is deployment-only.
## Changes
- mount the `vault-ca-cert` Secret (key `ca.crt`) read-only at `/etc/vault-ca/ca.crt`, following the puppet/artifactapi pattern
- set `CEPH_RGW_CA_FILE=/etc/vault-ca/ca.crt` so the operator adds the CA to its TLS trust
No image change (still `v0.2.0`); `reloader` + a normal reconcile pick it up. Validated with `kustomize build` on the au-syd1 overlay.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #276
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The new **terragrunt-enc** repo (single source of truth for encapi ENC data) runs its Terraform apply/plan in Woodpecker and authenticates to Vault via kubernetes auth. The Vault k8s role `woodpecker_terraform_enc` (terraform-vault PR #98) binds to a ServiceAccount named `terraform-enc` in the `woodpecker` namespace, which must exist for that auth to work.
Changes:
- Add `apps/base/woodpecker/serviceaccount_terraform_enc.yaml` (SA `terraform-enc` in namespace `woodpecker`).
- Register it in the woodpecker kustomization.
Reviewed-on: #274
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
cephrgw-operator **v0.2.0** rebuilds the Ceph integration to talk directly to radosgw via **go-ceph** (Admin Ops API) + **aws-sdk-go-v2** (S3), replacing the manager-dashboard client, and adds **fine-grained bucket-access policies** (paths / actions / conditions / rawStatements). The operator now authenticates with an **RGW admin user's access/secret key** instead of a dashboard login.
Operator repo PRs: unkin/cephrgw-operator #3 (rebuild) and #4 (fine-grained), both merged; tag `v0.2.0`.
## Changes
- bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.2.0`
- update the `envFrom` / `VaultStaticSecret` comments to the `CEPH_RGW_*` credential keys the new image consumes
## Required manual step (runtime)
The VaultStaticSecret copies the KV secret's keys **verbatim**, so the seed must be re-put with the new keys before/with rollout — otherwise the operator fails auth:
```
vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
CEPH_RGW_ACCESS_KEY=<key> CEPH_RGW_SECRET_KEY=<secret>
```
(The old `CEPH_DASHBOARD_*` keys are ignored by v0.2.0.) VSO refreshes within 5m and the `reloader` annotation restarts the operator.
https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
Reviewed-on: #273
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Kubernetes nodes querying the bind-resolvers LoadBalancer VIP (198.18.200.7) get REFUSED (EDE 18 Prohibited).
The service is `externalTrafficPolicy: Local`, which preserves the client source IP for traffic entering the cluster from outside — but a node querying the VIP never leaves via OSPF. Its own kube-proxy DNATs the LB IP in the OUTPUT chain and masquerades the source to a cluster-internal address (the node's flannel.1, e.g. 10.42.x.x). That address is not in `acl-main.unkin.net`, so the openforwarder view's match-clients rejects the query.
External clients preserve their real source IP and match acl-main, which is why only in-cluster hosts were affected.
Add `10.42.0.0/16` to `acl-main.unkin.net` so node-originated (masqueraded) resolver queries are permitted. This mirrors the authoritative cluster, which already allows the pod network (`allow-query { ...; 10.42.0.0/16; }`).
Reviewed-on: #271
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Picks up immediate NOTIFY of secondaries on primary zone changes (also-notify
+ shorter seed SOA timers), so dynamic updates / CRD records replicate across
the authoritative replicas in seconds instead of waiting up to the SOA refresh.
Bumps both the operator image and the CRD install.yaml tag.
The bind-resolvers `openforwarder` view forwarded `k8s.syd1.au.unkin.net` to the in-cluster bind-externaldns (198.18.200.8), which is not reliably serving those records yet, so lookups return NXDOMAIN.
Concrete impact: Gitea cannot resolve the k8s-hosted CI host, so its outbound webhook fails and tagged releases never trigger CI (e.g. cutting a new bind-operator release from a tag).
This points the `fwd-k8s-syd1-au-unkin-net` forwarder at the existing external external-dns bind service anycast **198.18.19.20** (puppet `roles::infra::dns::externaldns` — master `ausyd1nxvm2127` + slaves `2128`/`2129`, advertised via OSPF), which still holds the working `k8s.syd1.au.unkin.net` records. It is in the same 198.18.19.0/24 anycast family as the consul forwarder (198.18.19.14) the resolvers already use, so it is reachable from the pods.
Temporary measure. Revert to 198.18.200.8 once external-dns publishes to the in-cluster bind-externaldns service. Only the forward target changes; no CRD/schema changes.
Reviewed-on: #269
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Pairs with terraform-authentik#10: request the litellm_role scope (Authentik
emits the computed role claim) and read it via GENERIC_USER_ROLE_ATTRIBUTE so
akP-litellm-admin -> proxy_admin, akP-litellm-user -> internal_user.
App-side of the LiteLLM Authentik onboarding (terraform-authentik#8). Configures
LiteLLM's generic OIDC SSO against Authentik.
- VaultStaticSecret oauth-credentials: surfaces the OIDC client secret (same
secret Authentik sets on the provider) as a k8s Secret.
- Deployment: GENERIC_CLIENT_SECRET from that Secret.
- litellm-env: GENERIC_CLIENT_ID, authorization/token/userinfo endpoints, scope,
and PROXY_BASE_URL (required for SSO). reloader restarts on secret/config change.
## Why
Every bind-operator dynamic update is refused (`update ... denied due to allow-query`) because the operator execs `nsupdate` against `127.0.0.1` inside the primary pod, and the BindCluster `allow-query` listed only the client subnets (`auth-acl-main`) and the pod net (`10.42.0.0/16`) — not loopback. This blocked ALL DNSRecords (identity, s3, dashboard, lb1) from ever applying.
## Change
- Add `localhost` to the BindCluster `allow-query` in `apps/base/bind-internal/authoritative/cluster.yaml`. The `client-update` TSIG key still gates the actual update.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #267
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The cephrgw-operator (in-cluster) fails to reach the Ceph dashboard because CoreDNS/bind-internal has no record for `dashboard.ceph.unkin.net` (`no such host`). Publish it authoritatively so in-cluster clients can resolve it.
## Changes (apps/base/bind-internal/authoritative/records.yaml)
- `DNSRecord dashboard-ceph-cname`: CNAME `dashboard.ceph.unkin.net` -> `lb1.unkin.net.` (zone `ceph.unkin.net`, zoneRef `ceph-unkin-net`).
- `DNSRecord lb1-unkin-net`: A `lb1.unkin.net` -> `103.216.191.185` (zone `unkin.net`, zoneRef `unkin-net`).
Once applied, the operator's `sandbox-user`/`sandbox-bucket` (currently Error/Pending on DNS) will reconcile to Ready.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #266
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to the cephrgw-operator deploy (#261): source the operator's Ceph dashboard credentials from Vault via VSO instead of a hand-created Secret.
## Changes
- Add `apps/base/cephrgw-system/vaultauth.yaml`: `VaultAuth` (mount `k8s/au/syd1`, role `cephrgw-operator`, SA `cephrgw-operator`, `vaultConnectionRef: vso-system/default`).
- Add `apps/base/cephrgw-system/vaultstaticsecret.yaml`: renders KV `service/cephrgw/dashboard-credentials` into the `cephrgw-credentials` Secret (keys copied verbatim → consumed by the Deployment via `envFrom`; the reloader annotation restarts the operator on rotation).
- Reference both from the base kustomization.
## Dependencies / ordering
- Requires the Vault role + policy from **terraform-vault #95** (merge/apply first), and the KV values to be seeded out-of-band:
```
vault kv put kv/service/cephrgw/dashboard-credentials \
CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
CEPH_DASHBOARD_PASSWORD=... CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net
```
- Until VSO auth succeeds the `cephrgw-credentials` Secret won't exist and the operator pod stays in `CreateContainerConfigError` (expected).
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #262
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Publish the RGW S3 endpoint name (`s3.ceph.unkin.net`) that cephrgw-operator consumers use and that the radosgw hosts will carry as a cert SAN. For now it points at the Consul service; the real target will be changed later.
## Changes
- Add a `DNSRecord` in the `ceph.unkin.net` authoritative zone: `s3` CNAME `radosgw.service.consul.` (`apps/base/bind-internal/authoritative/records.yaml`, zoneRef `ceph-unkin-net`, TTL 600).
A companion puppet-prod change adds `s3.ceph.unkin.net` to the radosgw cert SANs and nginx server names.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #265
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Consume the two-tier Authentik RBAC from terraform-authentik#7. Grafana should grant Admin to the `akP-grafana-admin` permission group, which `akR-global-admin` members inherit.
## Change
- **grafana.yaml** (`auth.generic_oauth`): add `ak_groups` to `scopes`; `role_attribute_path` now keys off `ak_groups` and `akP-grafana-admin` (replaces the flat `grafana-admins`). Non-admins who can log in (gated to `akP-grafana-*` by the Authentik access policy) get Viewer; `role_attribute_strict: false` retained.
## Depends on
terraform-authentik#7 (creates `akP-grafana-admin`, the access binding, and the `ak_groups` mapping).
## Validation
`kustomize build` (base + overlay) renders; pre-commit clean.
Reviewed-on: #264
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Consume the two-tier Authentik RBAC from terraform-authentik#7 (user → role → permissions). ArgoCD should grant admin to the `akP-argocd-admin` permission group, which `akR-global-admin` members inherit.
## Change
- **argocd-cm**: request the hierarchical `ak_groups` scope + id-token claim (carries permission groups inherited via role groups; distinct from the default `groups` claim to avoid collision).
- **argocd-rbac-cm**: `scopes: [ak_groups]`; `policy.csv`: `g, akP-argocd-admin, role:admin` (replaces the flat `argocd-admins`). Default stays `role:readonly`.
## Depends on
terraform-authentik#7 (creates `akP-argocd-admin`, the access binding, and the `ak_groups` mapping). Merge/apply that first; then add yourself to `akR-global-admin` in Authentik.
## Validation
`kustomize build` renders the patched configmaps; pre-commit clean. Note: argocd-server picks up argocd-cm/rbac-cm live.
Reviewed-on: #263
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new `cephrgw-operator` provisions Ceph RGW (S3) buckets and access keys (RW/RO) from Kubernetes CRDs via the Ceph manager dashboard API. This deploys it as a platform app.
## Changes
- Add `apps/base/cephrgw-system`: namespace, ServiceAccount + ClusterRole/Binding (manage `ceph.unkin.net` CRDs, Secrets, leader-election leases), and the operator Deployment. CRDs are pulled from the operator repo at tag `v0.1.0`; the Deployment sources dashboard credentials from the `cephrgw-credentials` Secret via `envFrom` and carries the reloader annotation.
- Add `apps/overlays/au-syd1/cephrgw-system` referencing the base.
- Register `apps/overlays/*/cephrgw-system` in the platform ApplicationSet.
The platform AppProject already permits `*-system` namespaces and the Namespace/ClusterRole/CRD cluster resources, so no project change is needed.
## Ordering / dependencies
- Depends on the Gitea repo from terraform-git #34 and on the operator being pushed + tagged **v0.1.0** (image `git.unkin.net/unkin/cephrgw-operator:v0.1.0` and the raw CRD `install.yaml` at that tag). The `kubeconform` check will stay red until v0.1.0 exists, then go green.
- The `cephrgw-credentials` Secret must be created out-of-band in `cephrgw-system` (see the operator's `docs/ceph-setup.md`); it is intentionally **not** managed in GitOps.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #261
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The ceph/halb host (`ausyd1nxvm2069`) publishes `dashboard.ceph.unkin.net` via nsupdate to a dedicated `zone ceph.unkin.net.`, which `bind-authoritative` was not authoritative for (NOTZONE). This adds the zone so that record has a home.
## Changes
- Add `ceph.unkin.net` BindZone (primary, dynamicUpdate, updateKeyRef client-update) to bind-authoritative, matching the unkin.net/main.unkin.net pattern.
## Note — not the root cause of the 6 missing hosts
Log evidence (VictoriaLogs, dns-update-apply on 2069/2070) shows the actual failure is a **host-side bug in the puppet `dns-update` script**: `fqdn()` appends the zone even to records whose name is already fully-qualified (e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`), producing a `..` empty label → `invalid owner name: empty label` → the whole main.unkin.net update `send` fails (reverse PTR, sent first, still lands). That script fix (puppet-prod) is the real blocker; this zone is still needed so the ceph record does not hit NOTZONE afterward.
Reviewed-on: #260
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Follow-up to #258 (which added workstation+router to the *authoritative* ACL). The **resolver** (bind-resolvers, 198.18.200.7) has its own `acl-main.unkin.net` gating its `openforwarder` view; the workstation is not in it, so recursive queries return REFUSED. This lets the workstation use the resolver as its normal nameserver.
## Changes
- Add `10.10.12.200/32` (workstation, wireguard) to resolver `acl-main.unkin.net`
- Add `198.18.21.160/32` (router) explicitly for documentation; already covered by existing `198.18.21.160/27` and `198.18.21.0/24` (no functional change for the router)
Reviewed-on: #259
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Allow the operator's workstation and router to query the bind-authoritative servers directly. Their source addresses are outside the existing `auth-acl-main` client subnets, so named returns REFUSED to them today. The router sits on 198.18.21.0/24 which is not in the ACL at all.
## Changes
- Add `10.10.12.200/32` (workstation, over wireguard) to `auth-acl-main`
- Add `198.18.21.160/32` (router) to `auth-acl-main`
## Note
This grants query permission only. Reaching the LoadBalancer VIP (198.18.200.6) from off-datacenter paths is separately gated by `externalTrafficPolicy: Local`; the workstation-over-wireguard path still needs its L4 routing addressed to actually land on a node with a ready endpoint.
Reviewed-on: #258
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new terraform-rancher CI pipeline runs as a pod in the woodpecker namespace; its ServiceAccount token is what Vault binds to the `woodpecker_terraform_rancher` k8s auth role (see terraform-vault#86) for rancher2 provider auth + Consul state.
## Change
- Add `ServiceAccount/terraform-rancher` (woodpecker ns) and wire it into the woodpecker kustomization, mirroring the other terraform-* runner SAs.
## Validation
`kustomize build apps/base/woodpecker` renders it; pre-commit clean.
Reviewed-on: #257
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`identity.unkin.net` is configured as the Authentik OIDC issuer but has **no DNS record anywhere**, so in-cluster OIDC discovery fails (e.g. `argocd-server` → `lookup identity.unkin.net ... no such host`). Add an authoritative A record served by the internal bind system (bind-operator) so it resolves.
## Change
- New `DNSRecord/identity-dns-internal` → `198.18.200.4` (the traefik-internal gateway VIP, where the Authentik Gateway serves the `identity.unkin.net` hostname), in the `unkin-net` zone.
- Lives in the **`bind-internal` namespace** alongside the `BindZone`: the operator resolves `zoneRef`/`clusterRef`/`updateKeyRef` within the record's own namespace, so it can't live in the app (authentik) namespace.
- Wired into `apps/base/bind-internal/authoritative/kustomization.yaml`.
- `identity-dns-internal` name distinguishes this from the external DNS that Authentik will manage its own records from later.
## Validation
`kustomize build apps/base/bind-internal` + kubeconform (validates against the `dnsrecord_v1alpha1` schema): 57 valid, 0 invalid. pre-commit clean.
Reviewed-on: #256
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2's config-hash rolling restart exposed a latent non-determinism: `client.List` returns cache-ordered results, so the resolver's forward zones reshuffled every reconcile, flipping the config hash and rolling `bind-resolvers-2` endlessly. v0.2.3 (bind-operator #11) sorts every rendered list so `named.conf` is byte-identical across reconciles and the hash is stable.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.3`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.3`.
CRDs are unchanged from v0.2.2 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.3 CRD bundle; `bind-internal` passes `kubeconform` (56/56); pre-commit clean.
## Deploy note
On deploy the deterministic operator stops churning the ConfigMap; the config hash stabilizes and the stuck resolver rolling update completes, leaving all three pods Ready on one revision.
Reviewed-on: #255
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
ArgoCD had no external ingress and only local admin auth. This exposes `argocd-server` behind the traefik-internal gateway and enables Authentik SSO, so operators log in with their Authentik identity and group membership. Pairs with unkin/terraform-authentik#3 (creates the OAuth2 provider).
## Changes
- **argocd-cm**: set `url` and `oidc.config` (Authentik issuer `identity.unkin.net/application/o/argocd/`, `argocd` client, openid/profile/email scopes). Client secret resolved from the `argocd-oidc` Secret via `$argocd-oidc:client_secret`.
- **argocd-rbac-cm**: match RBAC on the `groups` claim; default `role:readonly`; map the `argocd-admins` Authentik group to `role:admin`.
- **argocd-cmd-params-cm**: `server.insecure=true` so `argocd-server` serves HTTP behind the TLS-terminating gateway.
- Add **Gateway + HTTPRoutes** for `argocd.k8s.syd1.au.unkin.net` (mirrors the grafana pattern: traefik-internal, vault-issuer cert, external-dns).
- Add **VaultAuth + VaultStaticSecret** sourcing the OIDC client secret from `kv/kubernetes/namespace/argocd/default/oauth-credentials` into the `argocd-oidc` Secret (labelled `part-of=argocd` so ArgoCD will resolve the `$` reference).
## Notes / rollout
- Seed the client secret in Vault out of band (same path terraform-authentik reads).
- The argocd namespace `default` SA already has Vault read access via the `default` k8s role, so no terraform-vault change is needed.
- `argocd-server` needs a one-time rollout restart to pick up `server.insecure`.
Validated with `kustomize build --enable-helm clusters/au-syd1/bootstrap`, `make kubeconform`, and pre-commit (yamllint + no-plain-secrets).
Reviewed-on: #253
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
v0.2.2 (bind-operator #10) stamps a `bind.unkin.net/config-hash` on the pod template, so a ConfigMap or `keys.conf` change flips the hash and triggers an operator-driven rolling restart. This fixes the class of bug where config edits (ACLs, forwarders, `validate-except`, primary address, TSIG rotation) never reached running pods — they held a startup snapshot and needed manual pod deletes.
## Changes
- Bump the operator image (`bind-system/deployment.yaml`) and the pulled CRD bundle URL (`bind-system/kustomization.yaml`) to `v0.2.2`.
- Bump the `bind-tsig-api` image (`bind-internal/tsig-api/tsig-api.yaml`) to `v0.2.2`.
CRDs are unchanged from v0.2.1 (controller-only change), so the generated kubeconform schemas need no update.
## Validation
- `bind-system` renders with the v0.2.2 CRD bundle; pre-commit clean.
## Deploy note
When the v0.2.2 operator first reconciles it stamps the config-hash annotation, triggering **one rolling restart per bind StatefulSet** — expected, and it also pulls in any already-pending config. From then on, config changes roll pods automatically.
Reviewed-on: #254
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why — urgent
PR #251 added `validate-except` to the resolver `BindCluster` but the list omitted the trailing semicolon after the final entry. `named` requires **every** entry in a list to be semicolon-terminated, including the last before the closing brace, so it fails config parse and the resolver pods crash-loop:
```
/run/named/named.conf:18: missing ';' before '}'
loading configuration: failure
exiting (due to fatal error)
```
The resolvers (`.7`) are down until this lands; the authoritative (`.6`/`.9`) and externaldns (`.8`) are unaffected.
## Fix
```diff
- validate-except { unkin.net; 18.198.in-addr.arpa; consul }
+ validate-except { unkin.net; 18.198.in-addr.arpa; consul; }
```
Renders to `validate-except { unkin.net; 18.198.in-addr.arpa; consul; };` — valid.
## Recovery
On merge + ArgoCD sync, the operator re-renders the ConfigMap with valid config and the crash-looping pods self-heal on their next restart (no manual `rollout restart` needed). Validated: `bind-internal` renders and pre-commit clean.
Reviewed-on: #252
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkinben