9c10b9096a463c2c30df20e3336884376196330e
267 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 unkinben
|
9c10b9096a |
Replace vector-archiver with logarchiver (#308)
## Why The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed. ## Changes - Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring. - Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch. - Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`). - Remove the vector-archiver Helm release, values and pipeline ConfigMap. Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #308 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
96afbcf5e1 |
Fix NATS auth: wrap env-var passwords in << >> so the server expands them (#306)
## Why After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient). ### Evidence - Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped. - The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected. - **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**. ### Root cause The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them. ## What Wrap all three user passwords in `<< >>`: ``` password: << $NATS_ADMIN_PASSWORD >> # (+ producer, consumer) ``` Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted). This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password. ## Verified end-to-end nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully. ## Expected recovery after merge + sync 1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords. 2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config. 3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes. 4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3. 5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises. ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix. ## Validation kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #306 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
7dddf8c5aa |
Fix logging deploy: enable Vector env interpolation; operator watches logging ns (#301)
## Why PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here. ## Root cause 1 — Vector env-var interpolation is off by default in 0.57 `vector-aggregator` and `vector-archiver` crash-looped with `async_nats::connector: authorization violation`; `vector-agent` / `vector-vm-ingest` were "Running" but silently failing to publish. Diagnosis (evidence): - The NATS server config **does** expand `$NATS_*_PASSWORD` (a `nats` CLI login with the real secret value authenticated fine), and the ACL was **not** the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation). - The failure reproduces locally: a hardcoded password connects; the **same value via `${NATS_CONSUMER_PASSWORD}` fails**. Configuring the server to expect the literal string `${NATS_CONSUMER_PASSWORD}` makes Vector connect — proving **Vector sends the literal, un-interpolated string**. - `vector --help` shows `--dangerously-allow-env-var-interpolation` — in 0.57 `${VAR}` interpolation is **opt-in**. An unset-var test confirms interpolation is off (no "unknown env var" error). - Verified fix: with `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` → **connects and authenticates**. Every tier uses `${...}` for auth (`${NATS_*_PASSWORD}`, `${CLICKHOUSE_*}`), so the env var is added to **all four** vector deployments. (This slipped past CI because `vector test` never opens the NATS connection.) ## Root cause 2 — operator watches only its own namespace `kubectl get chi -n logging` showed the `logs` CHI existed but with **empty status / no finalizer** — the operator never touched it, so the `logging-logging` Argo sync was stuck `Progressing` on *"waiting for healthy state of ClickHouseInstallation/logs"*, and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed). Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states `watchNamespaces: []` (our value) makes the operator **watch only its own namespace** (`clickhouse-system`). The CHI is in `logging`. Fix: `watchNamespaces: ["logging"]` → operator config `watch.namespaces.include: [logging]`. ## Changes - `apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml`: add `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true`. - `apps/overlays/au-syd1/clickhouse-system/values.yaml`: `watchNamespaces: ["logging"]`. No NATS ACL change (the original narrow ACL is correct). No secret/base changes. ## Expected recovery after merge + sync 1. clickhouse-system syncs → operator config gains `logging` → operator restarts → reconciles the `logs` CHI → CHI pod comes up healthy. 2. `logging-logging` sync unblocks → PostSync hooks run → JetStream `LOGS` stream + `transform`/`archiver` consumers created; `logs.raw` table created. 3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3. 4. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` increases. ## Validation kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders `watch.namespaces.include: [logging]`; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #301 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
57691ef1d5 |
certificates: restore the validly-signed intermediate in vault-ca-cert (#305)
CNPG WAL archiving to Ceph RGW fails with CERTIFICATE_VERIFY_FAILED on six clusters because the reflected vault-ca-cert bundle carries a corrupt intermediate: the genuinely-signed cert has a typo'd AIA URL (vault.servuce.consul), and the committed copy was text-edited at some point to fix the typo — flipping one byte of signed data and invalidating the signature (openssl verify: error 7 certificate signature failure). Only radosgw surfaces it because it serves a bare leaf, forcing clients to verify the stored intermediate against the root; services presenting their own intermediate never exercised the corrupt copy. terraform-k8s's copy is defunct per Ben — this file is the authoritative source. - restore the original signed intermediate (one base64 character; sha256 E0:13:1B..., verified against the root, and the resulting bundle validates the live s3.ceph.unkin.net leaf) - add an explicit allow-plain-secret marker mechanism to ci/validate-no-secrets.sh for public-data bootstrap secrets, and mark vault-ca-cert.yaml with it (a CA bundle is public and cannot be Vault-sourced since it establishes Vault trust) After merge+sync the reflector propagates to all namespaces and barman's next retry (~1min) succeeds; base backups run on tonight's schedule. Follow-ups worth considering: re-issue the intermediate in Vault with a corrected AIA URL, and/or configure radosgw to serve its intermediate. Reviewed-on: #305 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
e0eeeb6b04 |
Use full postgres image on minimal CNPG clusters so WAL archiving works (#304)
## Why The CNPG buckets are empty after PR #298. Live diagnosis found **two** independent causes; this PR fixes one of them. `litellm`, `puppet` and `paperclip` run the CNPG `17-minimal-trixie` image, which omits the `barman-cloud` CLI. In-tree `barmanObjectStore` archiving shells out to `barman-cloud-wal-archive`, so their archiver dies immediately: ``` ContinuousArchiving=False :: unexpected failure invoking barman-cloud-wal-archive: exec: "barman-cloud-wal-archive": executable file not found in $PATH ``` (verified on the live `puppet-postgres` primary: `which barman-cloud-wal-archive` → not found; on a `-system` cluster it resolves to `/usr/local/bin/barman-cloud-wal-archive`). ## How Switch those three clusters from `17-minimal-trixie` to `17-system-trixie` — the `-system` variant already used by the other six clusters, which bundles the barman-cloud tools. Tag confirmed present upstream (ghcr manifest HTTP 200). ``` - imageName: ghcr.io/cloudnative-pg/postgresql:17-minimal-trixie + imageName: ghcr.io/cloudnative-pg/postgresql:17-system-trixie ``` CNPG applies this as a rolling image update (switchover, no data change). ## Not fixed here (separate, primary blocker) The other six clusters (full image, barman present) fail with a **TLS trust** error — the reflected `vault-ca-cert` bundle carries a **stale intermediate CA**, so barman can't verify `s3.ceph.unkin.net`: ``` SSL: CERTIFICATE_VERIFY_FAILED ... certificate signature failure ``` That is a shared trust-anchor refresh (likely owned by terraform-k8s / `config/certificates/secret.yaml`, `managed-by: terragrunt`), handled separately — it also gates litellm/puppet once they have barman. See the investigation report. ## Validation - `kustomize build --enable-helm` + `kubeconform` pass on `litellm`, `puppet` overlays and the `paperclip` base (paperclip has no overlay yet). - `pre-commit run` passes on all changed files. ## Follow-ups - Longer term, the Barman Cloud Plugin (sidecar) would let minimal images keep their size while still archiving — track with the plugin migration. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #304 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
d1a7646e09 |
Bump Woodpecker CI to v3.16.0 (#297)
## Why Upgrade the Woodpecker CI instance from v3.14.1 to v3.16.0 to pick up upstream fixes and the k8s-backend privilege-escalation hardening (GHSA-qf34-295c-26v8). Server and agent must move together. The chart is pulled at build time (`kustomize build --enable-helm`); the image tag defaults to the chart `appVersion`, so bumping the chart moves both images. Chart 3.6.5 -> appVersion 3.16.0 (verified against upstream `helm/charts/woodpecker/Chart.yaml`); subchart deps are unchanged from 3.6.3 (server 3.0.1, agent 2.0.1), so there is no values-schema migration. ## What - Bump the woodpecker helmChart from `3.6.3` (appVersion 3.14.1) to `3.6.5` (appVersion 3.16.0) in the au-syd1 overlay. Server + agent both render as `v3.16.0`. - Set `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true"` on the agent. ## CRITICAL: k8s backend serviceAccountName gating (required change) v3.16.0 (PR #6792, GHSA-qf34-295c-26v8) gates step-level `serviceAccountName` behind a new agent flag `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP`, **default `false`**. When disabled, any `backend_options.kubernetes.serviceAccountName` set by a pipeline is **silently ignored** and the namespace `default` SA is used instead. Every terraform pipeline in the estate sets `backend_options.kubernetes.serviceAccountName` (e.g. `terraform-git`, `terraform-vault`, `terraform-artifactapi`) and relies on that SA for Vault k8s auth / Consul state. Without this flag those jobs would run as `default` and lose their Vault identity. This PR sets the flag to `true` to preserve current behaviour. No other newly-gated backend_options keys (pod labels/annotations from step, native secrets) are used by the estate. ## Migration / rollback - **DB migration:** Woodpecker auto-migrates the schema (xorm) on server start; migrations are forward-only and NOT reversible. The 3.14 -> 3.16 changelogs do not call out a data-destructive migration, but a **DB backup (CNPG cluster `woodpecker`) should be taken before merge**. - **In-flight pipelines:** merging rolls the server StatefulSet and agents; any running pipelines are interrupted and will need re-running. - **Rollback:** re-pin chart `3.6.5` -> `3.6.3` reverts the images to v3.14.1, but because migrations are one-way, a clean rollback requires **restoring the CNPG DB from the pre-merge backup**, not just pinning the old image. ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/woodpecker` renders `woodpecker-server:v3.16.0` and `woodpecker-agent:v3.16.0`; agent carries the new env var. - `kubeconform` (k8s 1.33.7): 24/24 resources valid. - pre-commit (yamllint + checks): all pass. ## Follow-up (not in this PR) Woodpecker images are pulled from `docker.io` / `ghcr.io` directly, not the artifactapi proxy. Proxying them via artifactapi is a possible follow-up but out of scope for this version bump. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #297 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
f45cb6989f |
Add ClickHouse + Vector + NATS JetStream centralized logging (with S3 raw archive) (#296)
## Why Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists. ## Topology `edge (publishers) → JetStream → consumers → sinks` - **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets. - **Edge publishers (thin)** — no parsing, just a routing subject: - `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`. - `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.) - **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now). - **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`. - **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job. ## Streams / consumers / auth - Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment. - **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret. ## S3 / retention `ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here. ## Replay runbook - **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time. - **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window. ## Validation - `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**. - `pre-commit` (yamllint, check-json, no-plain-secrets) clean. - **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs. - **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets). ## Known upstream caveat Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook. ## Prerequisites (manual, one-time) ``` # ClickHouse PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1) vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \ username=vector password="$PW" password_sha256_hex="$HASH" # NATS vault kv put kv/kubernetes/namespace/logging/default/nats-auth \ admin_password=$(openssl rand -base64 24) \ producer_password=$(openssl rand -base64 24) \ consumer_password=$(openssl rand -base64 24) ``` No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR. ## Open decisions (defaults chosen, flag to change) - **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.** - **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side). - **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer. - **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA. - **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs). - **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --- ## Update: images via artifactapi, DHI, stateless transform tier **Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull. ### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`) | Image | Upstream | artifactapi path | DHI? | |---|---|---|---| | clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job | | altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI | | altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI | | bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI | | nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats | | natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI | | natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI | | timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened | | timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step | **DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace. ### Transform tier is now a stateless Deployment Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**. **Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.) --- ## Update: 7d retention, tunable limits ConfigMap, honest sizing - **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict. - **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3. - **Honest 7d sizing (stated assumption — please sanity-check against real volume):** - Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica. - Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica. - **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.** - ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob. - Replay window in the runbook is now **7d** (beyond that → the S3 archive). --- ## Update: retention cut to 3 days (both stores), PVCs shrunk Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design. | Store | Retention | Byte cap | PVC/node | Replicas | Total | |---|---|---|---|---|---| | NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) | | ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) | **NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC. **ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table. The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL. ⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit). Reviewed-on: #296 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
e6f2cbc363 |
authentik: raise CNPG postgres memory to stop replica OOMKills (#300)
Authentik's intermittent API 500s (which failed two terraform-authentik CI runs today) were traced to its CNPG postgres replicas being OOMKilled: 512Mi limits leave no headroom over shared_buffers 128MB + max_connections 200, both OOM events matched the 500 bursts to the second, and the session-pinned RO pooler turns each replica death into a batch of severed read connections. The primary is at 84% of its limit and is next. - raise the authentik CNPG memory limit from 512Mi to 1Gi and request from 256Mi to 512Mi Follow-up candidates (not in this PR): RO pooler poolMode session→transaction to shrink the blast radius of a replica loss; revisit max_connections/shared_buffers sizing. Reviewed-on: #300 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
2ee4c9518c |
Deploy NetBox (CNPG Postgres, Valkey, Authentik SSO) (#299)
## Why NetBox becomes the source of truth for host/interface/IPAM data as part of retiring Cobbler. The ENC role already moved to encapi; a kickstart-generation microservice that consumes NetBox comes later and is intentionally **out of scope** here. ## Change Adds the `netbox` app (namespace `netbox`, platform project) using the netbox-community helm chart via the OCI helm-through-kustomize pattern, plus estate-native data stores: - **NetBox** — chart `8.3.40` (appVersion `v4.6.5`), `oci://ghcr.io/netbox-community/netbox-chart`. 2 web replicas, 1 worker; bundled Postgres/Valkey subcharts disabled. Media on RWX CephFS so replicas share uploads. Chart `helm.sh/hook: test` Pod dropped via a kustomize delete patch (we deploy through ArgoCD, not `helm test`). - **CNPG Postgres** — `netbox-postgres`, 2 instances, PG 18, `cephrbd-fast-delete`. Standard per-cluster S3 backup: `barmanObjectStore -> s3://cnpg-netbox`, cephrgw `ObjectStoreUser` + `Bucket`, nightly `ScheduledBackup` at **03:40** (`0 40 3 * * *` — next free slot after grafana's 03:20), 30d retention. A pgbouncer `Pooler` (session mode) fronts it; NetBox connects via `netbox-postgres-pooler-rw`. - **Valkey** — standalone Deployment (`valkey/valkey:8-alpine`), AOF-persistent PVC on `cephrbd-fast-delete`. One instance: DB 0 = RQ task queue, DB 1 = cache. No auth (in-cluster, namespace-isolated). Chosen over the bundled Bitnami subchart to keep image control in-estate and avoid Bitnami's legacy-image churn; mirrors the litellm standalone-cache pattern. - **Ingress** — `Gateway` + `HTTPRoute` at `netbox.k8s.syd1.au.unkin.net` (`traefik-internal`, `vault-issuer` cert into `netbox-tls`, external-dns to the internal VIP), HTTP->HTTPS 301. - **Secrets** — all via VSO `VaultStaticSecret` (`postgres-credentials`, `netbox-secret-key`, `netbox-superuser`, `oauth-credentials`); no plain Secrets committed. The shared `default` k8s-auth role already binds `*` namespaces with a namespace-templated KV policy, so **no terraform-vault change is needed**. - **Authentik OIDC SSO** — `remoteAuth` wires `REMOTE_AUTH_ENABLED` + the `OpenIdConnectAuth` backend via chart values; `SOCIAL_AUTH_OIDC_*` via `extraConfig` (the chart's config loader globs `/run/config/extra/*/*.yaml`). The client secret is injected as a YAML fragment mounted from the Vault-synced `oauth-credentials` secret. New users auto-provision on first login. - Registers `netbox` in the platform ApplicationSet and AppProject (destination namespace + chart sourceRepo). ## Image table (source -> artifactapi -> DHI decision) All images flow through the estate's containerd registry mirrors; the allowlist patterns gate them. DHI (Docker Hardened Images) require authenticated pulls from a Docker Hub `dhi/` org and are **not** reachable through the anonymous mirror, so upstream official is used throughout. | Image | Upstream | Mirror / allowlist | Allowlisted? | DHI decision | |---|---|---|---|---| | NetBox app/worker/housekeeping | `ghcr.io/netbox-community/netbox:v4.6.5` | ghcr remote, `^netbox-community/` | **added in terraform-artifactapi #17** | No DHI published; upstream official | | CNPG Postgres | `ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie` | ghcr, `^cloudnative-pg/` | already | estate-standard CNPG image | | Valkey | `docker.io/valkey/valkey:8-alpine` | dockerhub, `^valkey/valkey` | already | DHI not anon-pullable; upstream official | | NetBox init (perms) | `docker.io/busybox:1.38.0` | dockerhub, `^library/busybox` | already | upstream official | | Worker wait-for-backend | `docker.io/rancher/kubectl:v1.36.2` | dockerhub, `^rancher/` | already | upstream official | ## Cross-repo PRs (merge order) 1. **terraform-artifactapi #17** (`^netbox-community/` ghcr allowlist) — **merge before** this PR so the NetBox image pulls on first sync. 2. **terraform-authentik #11** (OIDC provider/application) — independent; SSO works once applied + the secret is seeded. 3. This PR. > Note: the CNPG S3-backup stanza pattern here is identical to PR #298 (already in main); it merges cleanly regardless of ordering. ## One-time Vault seeds (before/at first sync) ``` # App DB user (CNPG bootstrap + NetBox both consume this) vault kv put kv/kubernetes/namespace/netbox/default/postgres-credentials \ username=netbox password="$(openssl rand -base64 30)" # Django SECRET_KEY vault kv put kv/kubernetes/namespace/netbox/default/netbox-secret-key \ secret_key="$(python3 -c 'import secrets;print(secrets.token_urlsafe(60))')" # Bootstrap superuser vault kv put kv/kubernetes/namespace/netbox/default/netbox-superuser \ username=admin email=admin@unkin.net \ password="$(openssl rand -base64 24)" api_token="$(openssl rand -hex 20)" # OIDC client secret — ONE value stored two ways (raw for Authentik, YAML for NetBox) CS="$(openssl rand -base64 30)" vault kv put kv/kubernetes/namespace/netbox/default/oauth-credentials \ client_secret="$CS" oidc.yaml="SOCIAL_AUTH_OIDC_SECRET: \"$CS\"" ``` Seed `oauth-credentials` **before** applying terraform-authentik #11 (that apply reads `client_secret`). ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/netbox` — clean. - `kubeconform` (CI args, k8s 1.33.7): **27/27 valid, 0 invalid**. - `pre-commit` (yamllint + no-plain-secrets) on all changed files — pass. ## Out of scope The PXE/kickstart microservice that will consume NetBox — not scaffolded here. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #299 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
3c2bdf307a |
Add S3 backups to all CNPG Postgres clusters (#298)
## Why None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for every cluster, plus code-forward restore docs. ## What - **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2, 30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected `vault-ca-cert` (`endpointCA`). - **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is hardcoded**) and a staggered nightly `ScheduledBackup`. - **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can validate the new CRs. - **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`). ## Design decisions (answers to the open questions) **One bucket for all, or per-database?** → **Per-database (one bucket + owner user per cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only *within the same namespace*), and CNPG reads its S3 credential Secret from its *own* namespace. A single shared bucket would require either cross-namespace bucket refs (unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the simplest correct topology and needs zero manual seeding. Each user owns exactly one bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess` grant needed. **Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart `cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree `barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`). ## Schedule / retention (defaults — Ben to adjust) | App | Cluster | Bucket | Nightly base backup | | --- | --- | --- | --- | | authentik | postgres | cnpg-authentik | 01:00 | | litellm | litellm-postgres | cnpg-litellm | 01:20 | | artifactapi | postgres | cnpg-artifactapi | 01:40 | | woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 | | puppet | puppet-postgres | cnpg-puppet | 02:20 | | encapi | postgres | cnpg-encapi | 02:40 | | paperclip | paperclip-postgres | cnpg-paperclip | 03:00 | | grafana | postgres | cnpg-grafana | 03:20 | Retention is **30d** across the board — flagged as a default to tune per cluster. Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once. ## Validation - `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph schemas) pass on all 8 affected overlays (paperclip validated at base — it has no overlay yet). ceph CRs resolve their schemas (`Skipped: 0`). - `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.). - Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated `cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) — pre-existing, reproduces on `origin/main`, unrelated to this change. ## Notes / caveats - No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json` files, which merge cleanly whichever lands first. - Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The only prerequisite is the operator being healthy (it is, in `cephrgw-system`). ## Follow-ups - Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs). - Tune per-cluster retention / schedule if the defaults don't fit. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #298 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
b725bf7dcf |
Add PTR to external-dns managed record types (#295)
Follow-up to #294: --create-ptr validates that PTR is in --managed-record-types, and setting that flag replaces the default list, so A/AAAA/CNAME are re-stated alongside PTR (verified against the v0.21.0 binary's --help: default A,AAAA,CNAME). Pod is crash-looping on config validation until this merges. - Adds --managed-record-types=A,AAAA,CNAME,PTR to the external-dns args MERGE ASAP. Reviewed-on: #295 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
faa256c764 |
Fix external-dns PTR flag (--create-ptr) (#294)
external-dns v0.21.0 rejects --rfc2136-create-ptr (my error in #292) and the pod is crash-looping on flag parsing. The correct flag in this version is the generic --create-ptr, which synthesizes PTRs for A records whose reverse zone is in the domain filter (200.18.198.in-addr.arpa already is). - Replaces --rfc2136-create-ptr with --create-ptr Record reconciliation is stalled until this merges (serving unaffected — bind answers normally). MERGE ASAP. Reviewed-on: #294 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
eff97b8d03 |
Enable PTR synthesis in external-dns (#292)
Post-cutover verification (argocd-apps#288) found the reverse zone 200.18.198.in-addr.arpa empty: external-dns only writes PTRs when --rfc2136-create-ptr is set, and nothing else feeds that zone. Reverse resolution for the k8s LB range was already absent for clients pre-migration (no forwarder existed), so this completes the reverse path rather than fixing a regression. - Adds --rfc2136-create-ptr to the external-dns rfc2136 args Verification after merge: dig -x 198.18.200.2 @198.18.200.7 returns puppetca.k8s.syd1.au.unkin.net (allow a reconcile cycle + negative-cache expiry). Reviewed-on: #292 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
aff06a633a |
Roll bind-operator to v0.2.6 (loop-free TSIG-keyed NOTIFY) (#293)
Deploy bind-operator v0.2.6 (bind-operator#15): NOTIFYs are now TSIG-signed via the catalog transfer key and secondaries accept by key, with no pod IPs in restart-scoped config — a regression test asserts the config-hash is invariant under pod IP churn, making the v0.2.5 roll-loop class impossible. Restores seconds-fast dynamic-zone propagation on bind-externaldns and bind-authoritative. - Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.6 (confirmed in registry) - Bumps the CRD install pin to the v0.2.6 tag Expect exactly ONE settling roll of the bind statefulsets when the new config lands, then stability. Reviewed-on: #293 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
0687c097d6 |
Revert bind-operator to v0.2.4 (v0.2.5 rolling-restart loop) (#291)
v0.2.5 renders the primary POD IP into the options-scope allow-notify. Options changes are restart-scoped (config-hash annotation), and every roll gives the primary a new pod IP, so the operator re-renders and rolls all bind clusters in an endless loop (externaldns, authoritative, resolvers all cycling ~45s pods right now). - Reverts the operator image and CRD pin to v0.2.4 A v0.2.6 will re-do the NOTIFY fix loop-free (allow-notify via zone-scope/rndc-applied config or a TSIG-keyed notify instead of pod-IP-in-options). MERGE ASAP to stabilize DNS. Reviewed-on: #291 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
7a889ca326 |
Flip resolver forwarding for k8s zones to the in-cluster bind (#288)
## Why
Step 3 of 3 in the external-dns → in-cluster bind migration: the client-visible cutover. The `openforwarder` resolvers currently forward `k8s.syd1.au.unkin.net` to the legacy VM anycast `198.18.19.20` (a temporary measure — commit
|
||
|
unkinben
|
9452473bf6 |
Roll cephrgw-operator to v0.3.1 (CRD staleness warning) (#290)
## Why cephrgw-operator **v0.3.1** logs a startup WARNING when its installed CRDs are missing or older than the operator (operator repo #6, merged) — added precisely because the CRD tag drifted behind the image before. The new startup check reads the `CustomResourceDefinition` objects, so it needs a small RBAC grant. ## Changes - bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.1` - bump the CRD `install.yaml` tag → `v0.3.1` (keep CRDs in step with the image — the invariant the v0.3.1 warning enforces) - add `apiextensions.k8s.io/customresourcedefinitions: [get, list]` to the operator ClusterRole so the startup check is not RBAC-denied Validated with `kustomize build` on the au-syd1 overlay; the v0.3.1 CRD URL resolves. Supersedes nothing outstanding (the earlier CRD-tag PR #286 to v0.3.0 already merged). https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM Reviewed-on: #290 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
9abb82afb4 |
Roll bind-operator to v0.2.5 (intra-cluster NOTIFY fix) (#289)
Deploy bind-operator v0.2.5 (bind-operator#14): secondaries now carry an explicit allow-notify for the primary pod IP, so dynamic-zone updates propagate in seconds instead of the ~1h SOA refresh. Unblocks the external-dns migration parity gate (argocd-apps#288) and speeds up the dns-updater zones on bind-authoritative. - Bumps the operator image to git.unkin.net/unkin/bind-operator:v0.2.5 (confirmed in registry) - Bumps the CRD install pin to the v0.2.5 tag On sync the operator re-renders cluster ConfigMaps; the config-hash change rolls the bind secondaries, which then accept the primary's NOTIFYs. Reviewed-on: #289 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
8dc65779a5 |
Repoint external-dns at the in-cluster bind-externaldns primary (#287)
## Why
Step 2 of 3 in the external-dns → in-cluster bind migration. external-dns currently pushes RFC2136 updates to the legacy VM `ausyd1nxvm2127.main.unkin.net`. This repoints it at the in-cluster `bind-externaldns` primary so the in-cluster zone becomes the live source of truth for `k8s.syd1.au.unkin.net` + `200.18.198.in-addr.arpa`.
## Changes
- Points `--rfc2136-host` at `bind-externaldns-primary.bind-internal.svc.cluster.local` (verified live: ClusterIP Service `bind-externaldns-primary` exists in `bind-internal`).
- Reads TSIG `secret` + `algorithm` from Secret `externaldns-key-tsig` (reflected by PR 1) instead of the Vault-backed `externaldns-tsig`.
- Keeps port, zones, keyname, and `txtOwnerId: k8s` unchanged.
- Leaves the old Vault manifests (`apps/base/externaldns/{vaultauth,vaultstaticsecret}.yaml`) in place as the rollback path; their removal is a later cleanup PR.
## Merge gate
- PR 1 (`benvin/externaldns-tsig-reflect`) merged, AND
- `kubectl -n externaldns get secret externaldns-key-tsig` returns keys `secret` + `algorithm`.
## TXT-registry note
`policy: sync` + `registry: txt`: on first reconcile against the (currently empty) in-cluster zone, external-dns re-creates all managed A/CNAME records and their ownership TXTs from scratch. This is expected and populates the zone.
## Verification (after merge)
```
kubectl -n externaldns logs deploy/externaldns --tail=100 | grep -Ei 'rfc2136|BADKEY|NOTAUTH|added|update'
dig +short @198.18.200.8 A puppet.k8s.syd1.au.unkin.net
```
Logs should show updates to `bind-externaldns-primary...` with NO `BADKEY`/`NOTAUTH`; sampled A records should start resolving against the in-cluster server (198.18.200.8) as the sync populates the zone.
## Rollback
Revert this PR (host + secret name back to `ausyd1nxvm2127.main.unkin.net` / `externaldns-tsig`). The legacy VM is untouched and still authoritative.
Reviewed-on: #287
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
|
||
|
unkinben
|
8eeca2d1fb |
Reflect externaldns TSIG key into the externaldns namespace (#285)
## Why
Step 1 of 3 in the external-dns → in-cluster bind migration. For external-dns to send RFC2136 updates to the in-cluster `bind-externaldns` primary, it must present the exact TSIG key the primary's `allow-update` accepts. The bind-operator generates that key material into Secret `externaldns-key-tsig` in `bind-internal`; reflecting it into the `externaldns` namespace removes the manual eyaml→Vault key sync and guarantees key parity.
## Changes
- Adds `spec.secretTemplate.annotations` to BindTSIGKey `externaldns-key` with the emberstack reflector hints: `reflection-allowed`, `reflection-allowed-namespaces: externaldns`, `reflection-auto-enabled`, `reflection-auto-namespaces: externaldns`.
- Regenerates `schemas/bind.unkin.net/bindtsigkey_v1alpha1.json` from the live CRD (deployed bind-operator v0.2.4 already exposes `secretTemplate` — the WIP branch's ">= v0.3.0" claim is stale) to add the `secretTemplate` property. Schema output is byte-identical to running `ci/generate-schemas.sh`.
`kubectl kustomize apps/overlays/au-syd1/bind-internal` builds clean and renders the annotations onto the BindTSIGKey.
## Verification (after merge)
```
kubectl -n externaldns get secret externaldns-key-tsig \
-o jsonpath='{.data.secret} {.data.algorithm}{"\n"}'
```
Both `secret` and `algorithm` keys must be present (reflector mirrored the source Secret from bind-internal).
## Rollback
Revert this PR. The source Secret in bind-internal is unaffected; only the reflected mirror in `externaldns` is removed.
Merge order: this is PR 1/3. PR 2 (repoint external-dns) must not merge until the reflected secret is verified.
Reviewed-on: #285
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
|
||
|
unkinben
|
e4afdf59ba |
Bump cephrgw-operator CRDs to v0.3.0 (#286)
## Why The `cephrgw-system` kustomization pinned the CRD source to **`raw/tag/v0.1.0/config/crd/install.yaml`**, so the in-cluster CRDs never gained the fields added since v0.1.0 — v0.2.0's fine-grained BucketAccess policy fields (`paths`/`actions`/`conditions`/`rawStatements`) and v0.3.0's adoption fields (`retainOnDelete`, `managePolicy`, `status.adopted`). The running operator is v0.3.0, so applying those specs fails with `strict decoding error: unknown field`. The image bumps (#273, #279) should have moved this tag too. ## Changes - point the CRD `install.yaml` at `raw/tag/v0.3.0` The tag must track the operator image tag on future bumps. Verified the v0.3.0 URL serves the new schema and `kustomize build` renders all 3 CRDs. https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM Reviewed-on: #286 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
1d87c42fc1 |
Add advise-mode VPA for every workload (#283)
## Why Turns on right-sizing telemetry for the whole estate. Adds a `VerticalPodAutoscaler` with `updateMode: "Off"` (recommendation-only, advise mode) for every Deployment and StatefulSet in `apps/base`. Off mode never evicts or mutates pods, so this is purely observational: the VPA recommender (added in the vpa-system PR) publishes suggested requests/limits in each VPA's status, and nothing acts on them until someone deliberately flips a mode. ## Changes - Add one `vpa.yaml` per app under `apps/base/<app>/` containing a `<workload>-vpa` VerticalPodAutoscaler for each workload, and register it in that app's `kustomization.yaml`. - Coverage: 17 workloads across 11 apps. - age-api (age-api), artifactapi (api, redis, ui), authentik (redis), bind-system (bind-operator), cephrgw-system (cephrgw-operator), encapi (encapi), kanidm (kanidm StatefulSet), litellm (litellm, redis), paperclip (paperclip), pdbmux (pdbmux), puppet (puppetboard, puppetdb, puppetserver-compiler, puppetserver-master). ## Skipped (intentionally) - **CNPG `Cluster` objects** (artifactapi, authentik, encapi, grafana, litellm, paperclip, puppet, woodpecker) — Postgres is managed by CloudNativePG, not a VPA target. - **CronJobs** (puppet g10k/generate-types, reposync x4) — not VPA-able. ## HPA / VPA caveat api, ui (artifactapi), litellm, and all four puppet deployments also carry an HPA. With `updateMode: "Off"` there is no conflict today (VPA only recommends). VPA objects targeting these carry an inline comment: do **not** flip to `Auto`/`Initial` while the HPA still autoscales on CPU/memory, or the two controllers will fight over the same resource. Move the HPA to a custom/non-resource metric first. ## Verification - `kubectl kustomize` over every touched overlay: 9/11 overlays PASS rendering all their VPAs; 2 failures are pre-existing and unrelated to this change — `authentik` fails on a local helm-tooling flag error identically on origin/main, and `paperclip` has no `apps/overlays/au-syd1/paperclip` directory yet. Both apps' base kustomizations build clean and render their VPA. ## Merge gate - **Requires the vpa-system CRDs PR (argocd-apps #281) to merge first.** These manifests use `autoscaling.k8s.io/v1 VerticalPodAutoscaler`; without the CRDs installed, ArgoCD sync fails on an unknown kind. Reviewed-on: #283 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
4d877531b0 |
Add vpa-system: VPA CRDs + recommender (advise mode) (#281)
## Why Rolls out the Vertical Pod Autoscaler control plane so the estate can gather right-sizing recommendations for every workload (advise mode, follow-up PR adds the per-workload VPA objects). Deploys the **recommender only**: advise mode never mutates pods, so the updater and admission-controller (and its mutating webhook) are intentionally omitted — fewer moving parts, no webhook in the admission path. ## Changes - Add `apps/base/vpa-system/`: namespace, VPA CRDs (verticalpodautoscalers + verticalpodautoscalercheckpoints) pulled from the kubernetes/autoscaler repo at the pinned tag (same upstream-raw pattern node-feature-discovery uses), recommender-scoped RBAC (SA + metrics-reader/actor/status-actor/checkpoint-actor/target-reader), and the recommender Deployment. - Add `apps/overlays/au-syd1/vpa-system/` referencing the base. - Register `apps/overlays/*/vpa-system` in the platform ApplicationSet. ## Notes - Pins upstream **vertical-pod-autoscaler-1.7.0** (latest stable, 2026-05-29) for both CRDs and the `registry.k8s.io/autoscaling/vpa-recommender:1.7.0` image. - No platform AppProject change needed: `*-system` namespace destination plus CustomResourceDefinition/ClusterRole/ClusterRoleBinding are already whitelisted. - The recommender image is pull-through-cached via terraform-artifactapi PR #13 (merge gate below). ## Merge gate - Requires terraform-artifactapi PR #13 (adds the `autoscaling/vpa-` pattern to the k8s-registry docker remote) to land first so the recommender image is served through the cache. Reviewed-on: #281 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
614c4b1aa6 |
Route consul API hostname to the HTTP API (8500) (#282)
Phase 2 of the consul migration: expose the HTTP API (not just the UI) at consul.k8s.syd1.au.unkin.net, now rebased onto main post-#280 (ACLs enabled). - Adds a consul-http ClusterIP service targeting the server pods on 8500 (API + UI share the port, so the UI stays reachable at /ui/) - Repoints the consul and consul-svc HTTPRoutes from consul-ui:80 to consul-http:8500 - Documents ACL-authenticated access in apps/base/consul/README.md: token from kv/kubernetes/namespace/consul/default/bootstrap-acl-token (VSO-synced), X-Consul-Token curl and consul CLI usage, UI token login, and the prefer-vault-minted-tokens note Verification post-merge (ACLs are live, so authenticated): CONSUL_HTTP_TOKEN=$(vault kv get -field=token kv/kubernetes/namespace/consul/default/bootstrap-acl-token) && curl -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" https://consul.k8s.syd1.au.unkin.net/v1/status/leader Rollback: revert the HTTPRoute backends to consul-ui:80. Reviewed-on: #282 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
3920d76ba2 |
Add VPA CRDs and kubeconform schemas (#284)
Split out of #281 per review: the VerticalPodAutoscaler CRDs land first, together with their generated kubeconform schemas, so CI can validate the VPA objects that follow. - Adds apps/base/vpa-system with the v1.7.0 VPA CRDs served via the artifactapi github remote (terraform-artifactapi#14, merged) - Adds schemas/autoscaling.k8s.io/ (verticalpodautoscaler + checkpoint, v1 and v1beta2) generated with the same transform as ci/generate-schemas.sh (from the CRD manifest rather than the live cluster, since the CRDs are not installed yet) - Wires the vpa-system overlay into the platform applicationset Verified: kustomize renders both CRDs; a sample updateMode Off VPA passes kubeconform against the new schemas. Merge before #281 (recommender) and #283 (per-workload VPAs). Reviewed-on: #284 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
a633835550 |
Enable ACLs on k8s consul with Vault-sourced bootstrap token (#280)
## Why Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with `default_policy: deny` and `down_policy: extend-cache`; the k8s cluster currently runs with **ACLs disabled**. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the **same** `initial_management` token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git. ## Changes - Enable `global.acls.manageSystemACLs` so the chart manages system ACL tokens/policies for consul components. - Point `global.acls.bootstrapToken` at a pre-existing Kubernetes secret `consul-bootstrap-acl-token` (key `token`); chart 1.9.7 supports this, and when the secret is populated the `server-acl-init` job **skips bootstrapping** and adopts that token as the management token (renders `-bootstrap-token-secret-name`/`-bootstrap-token-secret-key`, verified in the kustomize output). - Add a `VaultAuth` (mount `k8s/au/syd1`, role `default`, SA `default`) and `VaultStaticSecret` in the `consul` namespace that sync `kv/kubernetes/namespace/consul/default/bootstrap-acl-token` into the `consul-bootstrap-acl-token` k8s secret via VSO (mirrors the encapi pattern). - Merge the `acl` block (`enabled`, `default_policy: deny`, `down_policy: extend-cache`, `enable_token_persistence`) into the server `extraConfig` to match the VM posture. ## OPERATIONAL NOTE — required BEFORE merge The user MUST place the VM cluster's `initial_management` token in Vault first, or ACL bootstrapping will generate a *different* token and break the mirror: ``` vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token> ``` VSO then syncs it into the `consul-bootstrap-acl-token` secret before the `server-acl-init` job runs. No terraform-vault change is needed: the wildcard `default` k8s-auth role (`bound_service_account_namespaces: ['*']`) plus the templated `kv/kubernetes/default` policy already grant the `consul` namespace `default` SA read on `kv/kubernetes/namespace/consul/default/*`. ## Risk / expected behavior - Enabling ACLs **rolls the 5 servers** (StatefulSet update) and runs a `server-acl-init` job. - With `default_policy: deny`, previously-anonymous operations are denied. The `vault` service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace `vault`) via their `service_registration "consul"` stanza (catalog entry has `ServiceMeta.external-source: vault`, port 8200). After the flip this registration will be **denied** unless Vault is given a Consul ACL token with `service:write` on `vault` (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration. - Anonymous HTTP API reads (e.g. `/v1/status/leader`) will also be denied post-merge unless a token is supplied — see PR 2 verification note. ## Ordering Independent of the API-route PR (`benvin/consul-api-route`). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done. Reviewed-on: #280 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
69f304f60d |
Roll cephrgw-operator to v0.3.0 (safe adoption) (#279)
## Why cephrgw-operator **v0.3.0** makes adopting pre-existing radosgw buckets/users safe (operator repo #5, merged): `retainOnDelete` on `ObjectStoreUser`/`BucketAccess`, non-destructive bucket-policy **merge** (+ `managePolicy`), non-destructive user attributes, and `status.adopted`. ## Changes - bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.3.0` No credential or manifest change beyond the tag. Validated with `kustomize build` on the au-syd1 overlay. https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM Reviewed-on: #279 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
cc81e90a75 |
Roll artifactapi to v3.7.7 (#278)
Deploy the two new UI features released in v3.7.7: per-repo usage-instruction panels (artifactapi#105) and direct-download links for local repo files (artifactapi#106). - Bumps artifactapi api image to git.unkin.net/unkin/artifactapi:v3.7.7 - Bumps artifactapi ui image to git.unkin.net/unkin/artifactapi-ui:v3.7.7 Both v3.7.7 images are confirmed published to the registry; kustomize builds clean. Reviewed-on: #278 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
05318553d6 |
Swap puppet compiler ENC to the encapic Go binary (#277)
## Why The puppet-on-k8s compilers classify nodes with a uv/python ENC script (`encapi-enc`). Each fresh compiler pod resolves the script's python dependencies on first invocation, and that resolution fails on cold pods (observed exits 135/2), breaking puppet agent catalog compilation. `encapic` (git.unkin.net/unkin/encapic) is a stdlib-only Go replacement with no runtime dependency resolution — a behavioural drop-in whose output matches the python script byte-for-byte. ## Changes - Points the compiler `external_nodes` at `/opt/bin/encapic`. - Reworks the `setup-shared-bins` init container to `curl` the encapic `v0.1.0` `encapic_linux_amd64` release binary (sha256-verified against the published `.sha256`, installed mode 0755) into the shared bins dir, instead of copying the python script and installing uv. - Removes the `puppet-encapi-enc` configmap generator, its volume and mount, and the `resources/encapi-enc` script. uv was consumed solely by that script (grep of `apps/base/puppet` confirms no other consumer), so its installation is removed too. `kubectl kustomize apps/overlays/au-syd1/puppet` builds clean. ## Merge gate Do not merge until the encapic `v0.1.0` release assets exist: `https://git.unkin.net/unkin/encapic/releases/download/v0.1.0/encapic_linux_amd64` (+ `.sha256`). The init container pulls them at pod start. ## Rollback Revert this PR to restore the `encapi-enc` configmap script + uv install and repoint `external_nodes`. --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #277 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
3af12180fd |
Switch puppet compiler ENC from Cobbler to encapi (#272)
## Why The k8s puppetserver compilers classify nodes via an exec ENC that today queries legacy Cobbler (`https://cobbler.main.unkin.net`) over TLS. `encapi` now runs in-cluster and exposes a cobbler-wire-compatible endpoint (`GET /cblr/svc/op/puppet/hostname/<certname>`), a drop-in for the Cobbler URL. This cuts the puppet-on-k8s ENC over from Cobbler to encapi — a prerequisite for migrating VM agents onto puppet-on-k8s. ## Changes - Rename the ENC script `resources/cobbler-enc` -> `resources/encapi-enc`, and its configmap `puppet-cobbler-enc` -> `puppet-encapi-enc` (kustomization configMapGenerator + deployment volume, initContainer copy path, and volumeMount subPath). - Point `external_nodes` in the compiler `puppet.conf` at `/opt/bin/encapi-enc`. - Target the in-cluster encapi service `http://encapi.encapi.svc.cluster.local` (plain HTTP), overridable via the `ENCAPI_URL` env var. - Drop the `/opt/vault-ca-cert.crt` verify for the ENC request (no TLS needed in-cluster). - Leave the response normalization identical: classes coerced to a list, `enc_role`/`enc_env` params set, `environment` stripped when it equals `testing`. Verified with `kubectl kustomize apps/overlays/au-syd1/puppet` (builds clean, exit 0); the generated `puppet-encapi-enc` configmap contains the new URL and env var. ## 🚨 Merge gate **Do not merge until encapi is seeded** (terraform-incus `benvin/encapi-seed` PR applied). An empty encapi means every node resolves to a 404. On 404 the ENC script exits non-zero, so puppet fails the compile rather than classifying the node with zero classes — nodes will fail to run until they exist in encapi. Seed encapi first so real nodes classify correctly; only unknown nodes should 404. Reviewed-on: #272 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
6d7cdd59e3 |
pdbmux: deploy the merging PuppetDB proxy in-cluster (#275)
## Why During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move between them as they migrate. `node-lookup` (and `pblastreport`) need a single, consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy that provides exactly that. Per the all-in-kubernetes estate direction it runs as an in-cluster service, not a per-VM systemd unit. pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) — split out of the earlier node-lookup prototype — and is released as a container image on its own `v*` tags. ## Changes - Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway, httproute), modeled directly on the encapi app. - Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080, `/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap. - Backends: `old=http://puppetdbapi.service.consul:8080`, `new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the in-cluster address is preferred over the external gateway). `new` is primary/prefer, merge = freshness. - Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal` Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a port-80 Service — same shape as the puppetdb/encapi gateways — so VM/workstation `node-lookup` can reach it. - Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject (`pdbmux` namespace destination), exactly as encapi is wired. No new woodpecker ServiceAccount is required: the pdbmux image push uses the `docker-buildx` plugin against the Gitea registry with the `default` SA (same as encapi), not artifactapi. ## Verification - `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves to `git.unkin.net/unkin/pdbmux:v0.1.0`). - ApplicationSet + AppProject YAML validated. ## Merge gates 1. The pdbmux repo initial-content PR (unkin/pdbmux#1) must merge first. 2. `v0.1.0` must then be tagged on the pdbmux repo so the image `git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s `.woodpecker/docker.yaml`. 3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.) --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #275 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
4580a5f6a4 |
Trust internal CA in cephrgw-operator (fix radosgw TLS) (#276)
## Why cephrgw-operator v0.2.0 talks to radosgw over HTTPS (`radosgw.service.consul:443`, fronted by nginx presenting the internal `unkin.net` Vault-PKI cert). With no CA configured the operator fails: ``` Get "https://radosgw.service.consul:443/admin/user?...": tls: failed to verify certificate: x509: certificate signed by unknown authority ``` The `vault-ca-cert` Secret (the `unkin.net` intermediate+root) is already reflected into every namespace — including `cephrgw-system` — so the fix is deployment-only. ## Changes - mount the `vault-ca-cert` Secret (key `ca.crt`) read-only at `/etc/vault-ca/ca.crt`, following the puppet/artifactapi pattern - set `CEPH_RGW_CA_FILE=/etc/vault-ca/ca.crt` so the operator adds the CA to its TLS trust No image change (still `v0.2.0`); `reloader` + a normal reconcile pick it up. Validated with `kustomize build` on the au-syd1 overlay. https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM Reviewed-on: #276 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
739de129e1 |
Add terraform-enc woodpecker ServiceAccount (#274)
The new **terragrunt-enc** repo (single source of truth for encapi ENC data) runs its Terraform apply/plan in Woodpecker and authenticates to Vault via kubernetes auth. The Vault k8s role `woodpecker_terraform_enc` (terraform-vault PR #98) binds to a ServiceAccount named `terraform-enc` in the `woodpecker` namespace, which must exist for that auth to work. Changes: - Add `apps/base/woodpecker/serviceaccount_terraform_enc.yaml` (SA `terraform-enc` in namespace `woodpecker`). - Register it in the woodpecker kustomization. Reviewed-on: #274 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
887a8d34cb |
Roll cephrgw-operator to v0.2.0 (radosgw-native) (#273)
## Why cephrgw-operator **v0.2.0** rebuilds the Ceph integration to talk directly to radosgw via **go-ceph** (Admin Ops API) + **aws-sdk-go-v2** (S3), replacing the manager-dashboard client, and adds **fine-grained bucket-access policies** (paths / actions / conditions / rawStatements). The operator now authenticates with an **RGW admin user's access/secret key** instead of a dashboard login. Operator repo PRs: unkin/cephrgw-operator #3 (rebuild) and #4 (fine-grained), both merged; tag `v0.2.0`. ## Changes - bump the operator image `git.unkin.net/unkin/cephrgw-operator` → `v0.2.0` - update the `envFrom` / `VaultStaticSecret` comments to the `CEPH_RGW_*` credential keys the new image consumes ## Required manual step (runtime) The VaultStaticSecret copies the KV secret's keys **verbatim**, so the seed must be re-put with the new keys before/with rollout — otherwise the operator fails auth: ``` vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \ CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \ CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \ CEPH_RGW_ACCESS_KEY=<key> CEPH_RGW_SECRET_KEY=<secret> ``` (The old `CEPH_DASHBOARD_*` keys are ignored by v0.2.0.) VSO refreshes within 5m and the `reloader` annotation restarts the operator. https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM Reviewed-on: #273 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
982bf1c837 | Merge pull request 'Wire LiteLLM SSO to Authentik (generic OIDC)' (#268) from benvin/litellm-sso into main | ||
|
unkinben
|
39b180aec9 |
bind-internal: allow k8s pod network to query the resolvers (#271)
Kubernetes nodes querying the bind-resolvers LoadBalancer VIP (198.18.200.7) get REFUSED (EDE 18 Prohibited).
The service is `externalTrafficPolicy: Local`, which preserves the client source IP for traffic entering the cluster from outside — but a node querying the VIP never leaves via OSPF. Its own kube-proxy DNATs the LB IP in the OUTPUT chain and masquerades the source to a cluster-internal address (the node's flannel.1, e.g. 10.42.x.x). That address is not in `acl-main.unkin.net`, so the openforwarder view's match-clients rejects the query.
External clients preserve their real source IP and match acl-main, which is why only in-cluster hosts were affected.
Add `10.42.0.0/16` to `acl-main.unkin.net` so node-originated (masqueraded) resolver queries are permitted. This mirrors the authoritative cluster, which already allows the pod network (`allow-query { ...; 10.42.0.0/16; }`).
Reviewed-on: #271
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
|
||
|
unkinben
|
76db3bbfe0 |
Bump bind-operator to v0.2.4
Picks up immediate NOTIFY of secondaries on primary zone changes (also-notify + shorter seed SOA timers), so dynamic updates / CRD records replicate across the authoritative replicas in seconds instead of waiting up to the SOA refresh. Bumps both the operator image and the CRD install.yaml tag. |
||
|
unkinben
|
7ee5dfbbcd |
TEMP: forward k8s.syd1.au.unkin.net to external external-dns service (#269)
The bind-resolvers `openforwarder` view forwarded `k8s.syd1.au.unkin.net` to the in-cluster bind-externaldns (198.18.200.8), which is not reliably serving those records yet, so lookups return NXDOMAIN. Concrete impact: Gitea cannot resolve the k8s-hosted CI host, so its outbound webhook fails and tagged releases never trigger CI (e.g. cutting a new bind-operator release from a tag). This points the `fwd-k8s-syd1-au-unkin-net` forwarder at the existing external external-dns bind service anycast **198.18.19.20** (puppet `roles::infra::dns::externaldns` — master `ausyd1nxvm2127` + slaves `2128`/`2129`, advertised via OSPF), which still holds the working `k8s.syd1.au.unkin.net` records. It is in the same 198.18.19.0/24 anycast family as the consul forwarder (198.18.19.14) the resolvers already use, so it is reachable from the pods. Temporary measure. Revert to 198.18.200.8 once external-dns publishes to the in-cluster bind-externaldns service. Only the forward target changes; no CRD/schema changes. Reviewed-on: #269 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
91080c1504 |
LiteLLM SSO: request litellm_role scope + map to role
Pairs with terraform-authentik#10: request the litellm_role scope (Authentik emits the computed role claim) and read it via GENERIC_USER_ROLE_ATTRIBUTE so akP-litellm-admin -> proxy_admin, akP-litellm-user -> internal_user. |
||
|
unkinben
|
5380a9fbf4 |
Wire LiteLLM SSO to Authentik (generic OIDC)
App-side of the LiteLLM Authentik onboarding (terraform-authentik#8). Configures LiteLLM's generic OIDC SSO against Authentik. - VaultStaticSecret oauth-credentials: surfaces the OIDC client secret (same secret Authentik sets on the provider) as a k8s Secret. - Deployment: GENERIC_CLIENT_SECRET from that Secret. - litellm-env: GENERIC_CLIENT_ID, authorization/token/userinfo endpoints, scope, and PROXY_BASE_URL (required for SSO). reloader restarts on secret/config change. |
||
|
unkinben
|
0c1156282f |
bind-internal: allow localhost to query the authoritative cluster (#267)
## Why Every bind-operator dynamic update is refused (`update ... denied due to allow-query`) because the operator execs `nsupdate` against `127.0.0.1` inside the primary pod, and the BindCluster `allow-query` listed only the client subnets (`auth-acl-main`) and the pod net (`10.42.0.0/16`) — not loopback. This blocked ALL DNSRecords (identity, s3, dashboard, lb1) from ever applying. ## Change - Add `localhost` to the BindCluster `allow-query` in `apps/base/bind-internal/authoritative/cluster.yaml`. The `client-update` TSIG key still gates the actual update. --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #267 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
f61d5ba16d |
bind-internal: add dashboard.ceph.unkin.net + lb1.unkin.net records (#266)
## Why The cephrgw-operator (in-cluster) fails to reach the Ceph dashboard because CoreDNS/bind-internal has no record for `dashboard.ceph.unkin.net` (`no such host`). Publish it authoritatively so in-cluster clients can resolve it. ## Changes (apps/base/bind-internal/authoritative/records.yaml) - `DNSRecord dashboard-ceph-cname`: CNAME `dashboard.ceph.unkin.net` -> `lb1.unkin.net.` (zone `ceph.unkin.net`, zoneRef `ceph-unkin-net`). - `DNSRecord lb1-unkin-net`: A `lb1.unkin.net` -> `103.216.191.185` (zone `unkin.net`, zoneRef `unkin-net`). Once applied, the operator's `sandbox-user`/`sandbox-bucket` (currently Error/Pending on DNS) will reconcile to Ready. --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #266 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
5a22dd95ef |
cephrgw-system: source cephrgw-credentials from Vault via VSO (#262)
## Why Follow-up to the cephrgw-operator deploy (#261): source the operator's Ceph dashboard credentials from Vault via VSO instead of a hand-created Secret. ## Changes - Add `apps/base/cephrgw-system/vaultauth.yaml`: `VaultAuth` (mount `k8s/au/syd1`, role `cephrgw-operator`, SA `cephrgw-operator`, `vaultConnectionRef: vso-system/default`). - Add `apps/base/cephrgw-system/vaultstaticsecret.yaml`: renders KV `service/cephrgw/dashboard-credentials` into the `cephrgw-credentials` Secret (keys copied verbatim → consumed by the Deployment via `envFrom`; the reloader annotation restarts the operator on rotation). - Reference both from the base kustomization. ## Dependencies / ordering - Requires the Vault role + policy from **terraform-vault #95** (merge/apply first), and the KV values to be seeded out-of-band: ``` vault kv put kv/service/cephrgw/dashboard-credentials \ CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \ CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \ CEPH_DASHBOARD_PASSWORD=... CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net ``` - Until VSO auth succeeds the `cephrgw-credentials` Secret won't exist and the operator pod stays in `CreateContainerConfigError` (expected). --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #262 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
513c60492b |
bind-internal: add s3.ceph.unkin.net CNAME -> radosgw.service.consul (#265)
## Why Publish the RGW S3 endpoint name (`s3.ceph.unkin.net`) that cephrgw-operator consumers use and that the radosgw hosts will carry as a cert SAN. For now it points at the Consul service; the real target will be changed later. ## Changes - Add a `DNSRecord` in the `ceph.unkin.net` authoritative zone: `s3` CNAME `radosgw.service.consul.` (`apps/base/bind-internal/authoritative/records.yaml`, zoneRef `ceph-unkin-net`, TTL 600). A companion puppet-prod change adds `s3.ceph.unkin.net` to the radosgw cert SANs and nginx server names. --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #265 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
6f62b7873f |
Wire Grafana OAuth roles to Authentik ak_groups / akP-grafana-admin (#264)
## Why Consume the two-tier Authentik RBAC from terraform-authentik#7. Grafana should grant Admin to the `akP-grafana-admin` permission group, which `akR-global-admin` members inherit. ## Change - **grafana.yaml** (`auth.generic_oauth`): add `ak_groups` to `scopes`; `role_attribute_path` now keys off `ak_groups` and `akP-grafana-admin` (replaces the flat `grafana-admins`). Non-admins who can log in (gated to `akP-grafana-*` by the Authentik access policy) get Viewer; `role_attribute_strict: false` retained. ## Depends on terraform-authentik#7 (creates `akP-grafana-admin`, the access binding, and the `ak_groups` mapping). ## Validation `kustomize build` (base + overlay) renders; pre-commit clean. Reviewed-on: #264 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
19f8055144 |
Deploy cephrgw-operator to cephrgw-system (#261)
## Why The new `cephrgw-operator` provisions Ceph RGW (S3) buckets and access keys (RW/RO) from Kubernetes CRDs via the Ceph manager dashboard API. This deploys it as a platform app. ## Changes - Add `apps/base/cephrgw-system`: namespace, ServiceAccount + ClusterRole/Binding (manage `ceph.unkin.net` CRDs, Secrets, leader-election leases), and the operator Deployment. CRDs are pulled from the operator repo at tag `v0.1.0`; the Deployment sources dashboard credentials from the `cephrgw-credentials` Secret via `envFrom` and carries the reloader annotation. - Add `apps/overlays/au-syd1/cephrgw-system` referencing the base. - Register `apps/overlays/*/cephrgw-system` in the platform ApplicationSet. The platform AppProject already permits `*-system` namespaces and the Namespace/ClusterRole/CRD cluster resources, so no project change is needed. ## Ordering / dependencies - Depends on the Gitea repo from terraform-git #34 and on the operator being pushed + tagged **v0.1.0** (image `git.unkin.net/unkin/cephrgw-operator:v0.1.0` and the raw CRD `install.yaml` at that tag). The `kubeconform` check will stay red until v0.1.0 exists, then go green. - The `cephrgw-credentials` Secret must be created out-of-band in `cephrgw-system` (see the operator's `docs/ceph-setup.md`); it is intentionally **not** managed in GitOps. --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #261 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
706e30a202 |
bind-internal: add ceph.unkin.net authoritative zone (#260)
## Why The ceph/halb host (`ausyd1nxvm2069`) publishes `dashboard.ceph.unkin.net` via nsupdate to a dedicated `zone ceph.unkin.net.`, which `bind-authoritative` was not authoritative for (NOTZONE). This adds the zone so that record has a home. ## Changes - Add `ceph.unkin.net` BindZone (primary, dynamicUpdate, updateKeyRef client-update) to bind-authoritative, matching the unkin.net/main.unkin.net pattern. ## Note — not the root cause of the 6 missing hosts Log evidence (VictoriaLogs, dns-update-apply on 2069/2070) shows the actual failure is a **host-side bug in the puppet `dns-update` script**: `fqdn()` appends the zone even to records whose name is already fully-qualified (e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`), producing a `..` empty label → `invalid owner name: empty label` → the whole main.unkin.net update `send` fails (reverse PTR, sent first, still lands). That script fix (puppet-prod) is the real blocker; this zone is still needed so the ceph record does not hit NOTZONE afterward. Reviewed-on: #260 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
197991a6ca |
bind-internal: allow admin workstation to query the resolver (#259)
## Why Follow-up to #258 (which added workstation+router to the *authoritative* ACL). The **resolver** (bind-resolvers, 198.18.200.7) has its own `acl-main.unkin.net` gating its `openforwarder` view; the workstation is not in it, so recursive queries return REFUSED. This lets the workstation use the resolver as its normal nameserver. ## Changes - Add `10.10.12.200/32` (workstation, wireguard) to resolver `acl-main.unkin.net` - Add `198.18.21.160/32` (router) explicitly for documentation; already covered by existing `198.18.21.160/27` and `198.18.21.0/24` (no functional change for the router) Reviewed-on: #259 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
8f164873e0 |
bind-internal: allow admin workstation + router to query authoritative (#258)
## Why Allow the operator's workstation and router to query the bind-authoritative servers directly. Their source addresses are outside the existing `auth-acl-main` client subnets, so named returns REFUSED to them today. The router sits on 198.18.21.0/24 which is not in the ACL at all. ## Changes - Add `10.10.12.200/32` (workstation, over wireguard) to `auth-acl-main` - Add `198.18.21.160/32` (router) to `auth-acl-main` ## Note This grants query permission only. Reaching the LoadBalancer VIP (198.18.200.6) from off-datacenter paths is separately gated by `externalTrafficPolicy: Local`; the workstation-over-wireguard path still needs its L4 routing addressed to actually land on a node with a ready endpoint. Reviewed-on: #258 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
b1b2112c66 |
Add terraform-rancher woodpecker ServiceAccount (#257)
## Why The new terraform-rancher CI pipeline runs as a pod in the woodpecker namespace; its ServiceAccount token is what Vault binds to the `woodpecker_terraform_rancher` k8s auth role (see terraform-vault#86) for rancher2 provider auth + Consul state. ## Change - Add `ServiceAccount/terraform-rancher` (woodpecker ns) and wire it into the woodpecker kustomization, mirroring the other terraform-* runner SAs. ## Validation `kustomize build apps/base/woodpecker` renders it; pre-commit clean. Reviewed-on: #257 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |