4e3c8288e2f01a805d95f9ff2d5b84bbcf9eb0de
98 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 unkin-agent
|
ba71bd1a35 |
Scrape traefik gateway metrics into VictoriaMetrics (#397)
## Why Both traefik instances (traefik-internal/traefik-external, chart 40.0.0, image v3.7.0) already expose prometheus metrics on container port `metrics` (9100), but nothing scrapes them — VictoriaMetrics has no traefik series (verified empty). The observability VMAgent runs with selectAllByDefault and consumes VM*Scrape CRs only, so a VMPodScrape is all that is needed. ## Changes - Adds `apps/base/observability/vmpodscrape-traefik.yaml`: scrapes port `metrics` /metrics on `app.kubernetes.io/name: traefik` pods in traefik-system, relabeling `app.kubernetes.io/instance` to a stable `traefik_instance` label to split internal vs external series - Registers the new CR in the observability base kustomization - Enables `metrics.prometheus.addEntryPointsLabels` and `addRoutersLabels` in both au-syd1 traefik values files for per-entrypoint/per-route dashboards (pods restart on rollout — acceptable) Validated with `kustomize build` (observability base + overlay, traefik-system overlay with --enable-helm) and kubeconform (unknown CRDs skipped); rendered deployments carry the new metric flags and the VMPodScrape renders in the overlay. Reviewed-on: #397 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
7556d08d06 |
woodpecker: allow CA-baked docker-buildx plugin as privileged (#371)
Why: pushing in-house images to the internal-CA artifactapi registry fails `x509: certificate signed by unknown authority` with the upstream buildx plugin (and `:latest-insecure` does not fix it — buildkit still verifies the registry cert). The fix is the CA-baked plugin image now published at `artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx` (unkin/plugin-docker-buildx v0.1.0). How: add that image (bare + `:latest`) to `WOODPECKER_PLUGINS_PRIVILEGED` in the au-syd1 overlay so it may run privileged. Existing upstream entries kept so current repos are unaffected until they switch over. Next: ghp `.woodpecker` switches its buildx step to this image, then ghp re-tags and its push succeeds. Reviewed-on: #371 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
fc08f1fecc |
arrstack: sonarr/radarr/prowlarr/nzbget on shared CephFS media (#366)
## Why Stands up the download-automation stack (sonarr, radarr, prowlarr, nzbget) in a new `arrstack` namespace, sharing the same CephFS TV/movies subvolumes that jellyfin serves read-only (jellyfin static-PV rework is on the base branch, PR #237). Downloads and libraries live on one filesystem per media type so the arrs import with atomic hardlink moves. ## Changes - Add `apps/base/arrstack/` namespace with its own static CephFS PVs (`arrstack-media-tv`, `arrstack-media-movies`) + RWX PVCs (`media-tv`, `media-movies`) bound to the same rootPaths jellyfin uses; each namespace gets a unique PV name/volumeHandle pinned by claimRef. - Deploy sonarr (mounts media-tv RW), radarr (media-movies RW), prowlarr (config only), nzbget (both media PVCs RW); single replica, Recreate strategy, LinuxServer images via artifactapi dockerhub remote, PUID/PGID/TZ, probes, fsGroup for CephFS writability. - Per-app config PVCs on `cephrbd-fast-retain` (RWO, 5Gi) for the SQLite state; internal-only Gateway + HTTPRoute per app (traefik-internal, vault-issuer, external-dns `<app>.k8s.syd1.au.unkin.net`), no oauth yet. - nzbget: initContainer seeds `/config/nzbget.conf` (copy image template if absent, append managed path/category block once) and creates the media dirs; category `tv` -> `/media/tv/downloads`, `movies` -> `/media/movies/downloads`. - Wire ArgoCD: add `apps/overlays/*/arrstack` to the media ApplicationSet. (media AppProject already gains the arrstack destination + PersistentVolume whitelist on the base branch.) ## Depends on terraform-artifactapi PR #31 (allowlist `^linuxserver/` on the dockerhub remote) must be merged + applied before these pods can pull. ## Follow-up (terraform) Root folders (/media/tv/series, /media/movies/films), download-client wiring (host `nzbget`, port 6789, categories tv/movies) and API keys are configured later via terraform-arr. --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Co-authored-by: Ben Vin <neotheo@gmail.com> Reviewed-on: #366 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
 unkinben
|
6c8c0dd9e4 |
Deploy jellyfin-ha as a true-HA StatefulSet under a new media project (#237)
## Why Deploy the jellyfin-ha fork as a genuine high-availability service rather than a single replica, so its two headline capabilities can actually be exercised: the Redis-coordinated distributed transcoding (a surviving pod resumes the HLS segments of a pod that dies mid-stream) and the experimental PostgreSQL main database (which is what lets more than one replica share the same library). It lands in its own `jellyfin` namespace under a new `media` ArgoCD project. ## How **Workload — StatefulSet, 2 replicas.** The Deployment becomes a StatefulSet so each pod has a stable name. The fork's Redis transcode lease keys ownership on `JELLYFIN_INSTANCE_ID`, which is set from the downward-API pod name, giving each replica a unique, stable lease identity. Soft `podAntiAffinity` spreads the two pods across nodes and a `PodDisruptionBudget` keeps `minAvailable: 1` through drains and rollouts. **Main database — CloudNativePG.** A CNPG trio in-namespace mirrors the litellm pattern: a 3-instance `Cluster`, a PgBouncer `Pooler`, and Ceph RGW (barman) S3 backups to a dedicated `cnpg-jellyfin` bucket owned by a `cephrgw` `ObjectStoreUser`. An init container writes `/config/config/database.xml` selecting the fork's `Jellyfin-PostgreSQL` provider, and the connection string is composed from the CNPG-generated `jellyfin-postgres-app` secret (username / password / dbname) pointed at the pooler service — the password is never rendered into the manifest. Library-item metadata therefore moves off SQLite; metadata images, plugins, subtitles and config XML stay on `/config`. **Storage.** - `/config` is now a shared `ReadWriteMany` cephfs PVC (raid5, retain) so every replica reads/writes the same metadata and config. - `/config/transcodes` — the fork's real transcode temp path — is a shared RWX PVC (raid5, delete). This is the load-bearing fix: takeover reads the dead pod's in-flight `.ts`/`.m3u8` segments off shared storage, so per-pod scratch would silently break it. - `/cache` is per-pod via a `volumeClaimTemplate` (RWO). - The media library stays a fresh, empty RWX PVC mounted read-only; populating it is out of scope. **Hardware transcoding.** The container requests the `gpu.intel.com/i915` Intel device-plugin resource (which pins the pod to a GPU-labelled node and injects the DRI render node — no `/dev/dri` hostPath or privileged container) plus the render/video supplemental groups. VA-API hardware acceleration is now on by default: the `inject-config` init container seeds `/config/config/encoding.xml` with `HardwareAccelerationType` `vaapi`, `EnableHardwareEncoding`, the injected render node (`/dev/dri/renderD128`) and h264/hevc hardware decode, so transcodes use the iGPU on first boot with no manual admin-UI step. Both seed files (`database.xml`, `encoding.xml`) are written only when absent, so later admin changes persisted to the shared RWX `/config` are never clobbered on restart. **Networking.** The Gateway/HTTPRoute (traefik-internal, `jellyfin.k8s.syd1.au.unkin.net`) is unchanged; the Service gains `sessionAffinity: ClientIP` to keep a client pinned to one replica and reduce transcode-session churn. **Redis.** The in-namespace single-replica Redis stays as the transcode lease store. ## Follow-up UDP auto-discovery is disabled, but scheduled library scans still run on every replica (redundant scans). Single-scanner leader election is a planned follow-up pending a fork feature, tracked separately. --------- Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-authored-by: Ben Vincent <neotheo@gmail.com> Co-authored-by: Ben Vin <neotheo@gmail.com> Reviewed-on: #237 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkin-agent
|
b759f4beea |
Install valkey-operator (#365)
Makes the valkey-operator available in the cluster so a later change can replace the self-managed redis Deployment with a Valkey CR. This PR only installs the operator; it does not create any Valkey CR or touch existing apps. Changes: - Add `apps/base/valkey-operator-system/` (namespace `valkey-operator-system`, mirroring the cnpg-system pattern). - Add `apps/overlays/au-syd1/valkey-operator-system/` which inflates the `valkey-operator` 0.5.0 Helm chart from the artifactapi virtual helm repo with `includeCRDs: true` (the chart ships CRDs in the Helm-native `crds/` dir, which `helm template` skips otherwise). - Route the operator image through artifactapi: `artifactapi.k8s.syd1.au.unkin.net/ghcr/valkey-io/valkey-operator:v0.5.0`. - Register `apps/overlays/*/valkey-operator-system` in the platform ApplicationSet. No AppProject change needed: the `*-system` destination and the virtual-helm sourceRepo already cover it (same as cnpg-system/cephrgw-system). Depends on unkin/terraform-artifactapi#29 (adds the valkey helm remote + virtual member and allowlists the operator image). That PR must merge and apply before this one syncs, or the chart pull / image pull will 404/403. Validated with `kustomize build --enable-helm` against the upstream chart (identical version/values): renders the Namespace, 2 CRDs, operator Deployment (image = the artifactapi ghcr path), RBAC, SA, and metrics Service, all in `valkey-operator-system`. Reviewed-on: #365 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
dc02b3eebd |
Deploy k8up backup operator (platform, k8up-system) (#363)
The k8up CRDs already exist in the cluster (owned by a separate `k8up-crds` Helm release) but the k8up operator itself was never deployed, so no k8up backup/restore custom resources are reconciled. This installs the operator following the cnpg-system pattern (Helm inflated via kustomize `helmCharts`) in the `platform` ArgoCD project, namespace `k8up-system`. - Add `apps/base/k8up-system/` (Namespace + kustomization). - Add `apps/overlays/au-syd1/k8up-system/` inflating the **k8up chart `4.10.0`** (pinned) from the artifactapi virtual-helm proxy: `replicaCount: 2` (leader election with hot-standby follower), operator resource requests/limits (20m/128Mi req, 500m/256Mi lim), and ServiceMonitor/PrometheusRule disabled (require Prometheus Operator CRDs). - Add a `PodDisruptionBudget` (`minAvailable: 1`) in the overlay, since the k8up chart exposes no native PDB value. - Register `apps/overlays/*/k8up-system` in the platform ApplicationSet. **CRD management:** The k8up chart ships its CRDs via the Helm `crds/` directory (there is no `installCRDs`/`skipCrds`/`enableCRDCreation` value). The kustomize helm inflator does not emit `crds/`, so `kustomize build` renders only the operator (Deployment, RBAC, SA, Service) and no CRDs. ArgoCD therefore does not fight the existing `k8up-crds` Helm release over CRD ownership. **platform.yaml (AppProject):** no change needed. The virtual-helm sourceRepo, the `*-system` destination, and the ClusterRole/ClusterRoleBinding/CRD/webhook clusterResourceWhitelist entries are already present and cover everything this app renders. **Validation:** `kustomize build --enable-helm apps/overlays/au-syd1/k8up-system` succeeds (Deployment replicas 2, PDB, Namespace, RBAC; 0 CRDs). `make kubeconform` reports k8up-system 14/14 valid; the only failure is the pre-existing, unrelated `cattle-system` rancher chart kubeVersion issue. pre-commit passes. Reviewed-on: #363 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
fa1f3e7756 |
add ghp (GitHub proxy) deployment (#357)
## Why Deploy `goodtune/ghp` (a GitHub proxy) so estate agents have a controlled GitHub access path. Mirrors the artifactapi app structure and the estate's templated `default` Vault convention. ## How (this PR) - Namespace `ghp`. Runs as the namespace's auto-created `default` ServiceAccount (no custom SA). - CNPG `Cluster` (named `postgres`, db/owner `ghp`) + dedicated `cnpg-ghp` Ceph RGW backup bucket + nightly ScheduledBackup. - Deployment `command: ["/ghp","serve"]`; `GHP_DATABASE_DSN` assembled from the CNPG-generated `postgres-app` Secret via `$(VAR)` expansion pointing at `postgres-rw.ghp.svc`. - **Sync-phase migrate hook Job** `ghp-migrate` (`/ghp migrate`) at sync-wave 1 so serve replicas never race migrations. - VSO wiring on the templated `default` role: `VaultAuth` role/SA `default`; `VaultStaticSecret`s source `kv/kubernetes/namespace/ghp/default/github-app` → Secret `ghp-github-app` and `kv/kubernetes/namespace/ghp/default/app` → Secret `ghp-app`. `private_key` mounted as a file; `encryption_key` → `GHP_ENCRYPTION_KEY`. - cert-manager `Certificate` `ghp-tls` off ClusterIssuer `vault-issuer`, SANs github.com/api.github.com/codeload.github.com/*.githubcopilot.com/ghp.k8s.syd1.au.unkin.net; mounted for `GHP_TLS_CERT_FILE`/`GHP_TLS_KEY_FILE`. - Service (443->8443, 80->8080, metrics 9136), PDB, VMServiceScrape, internal Traefik Gateway + HTTPRoute for `ghp.k8s.syd1.au.unkin.net`. - Sync-wave ordering: DB + VSO/cert (wave 0, wait Healthy) -> migrate Job (wave 1) -> serve Deployment + Service/Gateway/HTTPRoute (wave 2). Writable `/tmp` emptyDir on both pods (root FS is read-only). - Registered in the `platform` ApplicationSet + AppProject. - Validation: `kustomize build` clean, kubeconform 17/17 valid (0 skipped), yamllint + pre-commit clean. ## NOTE — prerequisites before ghp goes Healthy - **`GHP_ADMINS` placeholder**: `apps/base/ghp/configmap.yaml` has `GHP_ADMINS: "REPLACE_ME_ben_github_username"` — set it to Ben's GitHub username. - Ben's GitHub-App runbook + kv population at the templated paths `kv/kubernetes/namespace/ghp/default/github-app` (app_id/client_id/client_secret/private_key) and `kv/kubernetes/namespace/ghp/default/app` (encryption_key). - tf-artifactapi #25: the `ghcr` remote pattern for pulling `ghcr/goodtune/ghp`. - **No terraform-vault change needed** — the estate's templated `default` policy already grants read on `kv/kubernetes/namespace/<ns>/default/*` (the bespoke ghp role PR terraform-vault#120 is closed). - DNS overrides (pointing github.com etc. at ghp) are intentionally **out of scope** — separate deferred bind-operator PR. Assumptions flagged: metrics env var `GHP_METRICS_LISTEN`; two Secrets (`ghp-github-app` + `ghp-app`) rather than one merged Secret; DSN assumes the CNPG-generated password is URL-safe. Reviewed-on: #357 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkinben
|
4d58f37ea5 |
Fix cert-manager recursive-nameserver ControllerConfiguration field (#347)
## Why - The cert-manager v1.20.2 controller crashloops: strict decoding of its ControllerConfiguration rejects the unknown field `acmeDNS01` (`failed to load config file ... strict decoding error: unknown field "acmeDNS01"`), so `/var/cert-manager/config/config.yaml` fails to load and the controller never starts. The rollout is stuck with only the old pod running. - PR #337 placed the DNS-01 recursive-nameserver settings under `acmeDNS01`, but the field in the `controller.config.cert-manager.io/v1alpha1` schema is `acmeDNS01Config` (`ACMEDNS01Config`, with `recursiveNameservers` / `recursiveNameserversOnly`). The recursive-ns settings belong in the config file, not `extraArgs`; the CLI flags feed the same struct but the chart already renders a `--config` ControllerConfiguration, so the correct fix is the correct field name. ## How - Rename the `config:` block `acmeDNS01` to `acmeDNS01Config`, keeping `recursiveNameservers` (`8.8.8.8:53`, `1.1.1.1:53`) and `recursiveNameserversOnly: true` so DNS-01 resolution and self-checks still use the public DNS view for the split-horizon delegation. Rendered `kustomize build --enable-helm` confirms the ConfigMap `config.yaml` now carries a valid `acmeDNS01Config` block and no longer contains the invalid `acmeDNS01`; the cert-manager overlay is kubeconform-clean (55 valid, 0 invalid). --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Reviewed-on: #347 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
07bd94c55a |
Wire API_TOKEN_PEPPERS into NetBox config (#346)
## Why
NetBox 4.6.5 refuses to save v2 API tokens without `API_TOKEN_PEPPERS` ("Unable to save v2 tokens: API_TOKEN_PEPPERS is not defined"), which blocks creating the superuser token the NetBox Vault engine needs (it defaults to v2 tokens). The chart only auto-generates a pepper when it creates the config secret itself; it does not do that while `existingSecret` (`netbox-secret-key`) is set, so the config secret carries no `api_token_peppers` key.
## Changes
- Document `api_token_peppers` on the `netbox-secret-key` VaultStaticSecret: a JSON pepper map `{"1": "<random>"}` seeded once into Vault alongside `secret_key`. VSO syncs every key at the path into the config secret, which the chart already mounts as an optional file into `API_TOKEN_PEPPERS`.
- Add a reloader annotation via `commonAnnotations` so the `netbox` and `netbox-worker` Deployments roll when `netbox-secret-key` changes, picking up the seeded pepper (and any rotated `secret_key`) without a manual restart.
## Follow-up (out of band)
seed the pepper once (rotating it invalidates existing v2 tokens):
```
PEP=$(openssl rand -base64 48 | tr -d '\n')
vault kv patch kv/kubernetes/namespace/netbox/default/netbox-secret-key \
api_token_peppers="{\"1\": \"$PEP\"}"
```
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #346
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
|
||
|
unkinben
|
5e409f5f06 | traefik-internal: add ldaps :636 entrypoint for authentik LDAPS (#345) | ||
|
unkinben
|
da4a66046a |
Add Tier-2 per-app Vector transform pipelines (structured logs) (#320)
Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. **Stacks on #318 — merge after it.** How: - 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: **bind_query** (k8s bind-* + VM named), **rancher_audit** (cattle-system sidecar JSON), **cnpg_pg** (ONE transform for all 10 CNPG clusters via the `.postgres` container), **gitea** (router+access, k8s+VM), **puppet** (openvoxserver/openvoxdb logback + access), **litellm** (JSON request logs), **postfix** (per-line maillog). - Carve `.postgres` out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact. - Companion k8s flips in this PR: litellm `JSON_LOGS=True`; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on. - 15 new `vector test` cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing `fields Map(String,String)` — no DDL change. Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with `.file`/`.SYSLOG_IDENTIFIER` tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Reviewed-on: #320 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
ba7a1a9509 |
Enable Reloader secret watching, scope existing auto to configmap-only (#326) (#339)
## Why The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret. ## Changes - Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched. - Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads. - Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`. - Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf. - Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`). ## Safety review (secret-only / CA workloads) `vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded. - `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated. - `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`). - `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only. - `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added. CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`). ## Notes / coordination - Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression). - Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here. ## Validation - `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains. - `uvx pre-commit run --all-files` — all hooks pass. Closes #326 --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Reviewed-on: #339 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
b99682861b |
Point cert-manager DNS-01 at public recursive nameservers (#337)
unkin.net is split-horizon: the `_acme-challenge.unkin.net -> _acme-challenge.acme.unkin.net` delegation CNAME exists only in the public Google Cloud DNS view. cert-manager's CNAME following (`cnameStrategy: Follow`) resolves via in-cluster CoreDNS to the nodes' internal resolver, which serves an internal view of unkin.net lacking that CNAME; Follow therefore finds no delegation and still sends the rfc2136 UPDATE to zone unkin.net on bind-external (only authoritative for acme.unkin.net), returning NOTAUTH. Follow needs a public-view resolver for both the CNAME chase and the propagation self-check. TSIG is proven fine. - Set `acmeDNS01.recursiveNameservers` to `8.8.8.8:53` and `1.1.1.1:53` with `acmeDNS01.recursiveNameserversOnly: true` in the cert-manager ControllerConfiguration so DNS-01 resolution and self-checks use the public DNS view. - Keeps `cnameStrategy: Follow` on the ClusterIssuers (merged in #331); this PR gives that following a resolver that can see the delegation. https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Reviewed-on: #337 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
f31552e192 |
Deploy kea DHCP operator to dhcp-system (#333)
Replaces the isc-dhcpd PXE-boot VM with the kea-operator + an HA kea pair, GitOps-managed. Deploys on a new, unused anycast IP so nothing is cut over yet; the production cutover off the current dhcpd address (198.18.19.18) is a separate later task. - Add `apps/base/dhcp-system`: namespace, kea-operator RBAC + Deployment (v0.1.0), VPA, and the 4 kea.unkin.net CRDs pulled from the operator repo at tag v0.1.0. - Add CRs translating the legacy dhcpd config (source: puppet `roles/infra/dhcp/server.yaml`): KeaCluster `kea` (2 replicas, hot-standby HA, main.unkin.net, 1200/86400 leases, AU ntp pool); five KeaSubnets 198.18.13-17.0/24 with .200-.220 pools, gateways .254 except .17->.1, next-server 198.18.19.19; Legacy/UEFI-64 PXE client classes; KeaAPI. - DHCP-advertised DNS points at the in-cluster bind-resolvers cluster (PureLB 198.18.200.7), not the legacy 198.18.19.15 forwarder. - Pin the DHCP LoadBalancer Service to the free common-pool IP 198.18.200.10 via PureLB. - KeaAPI bearer token is operator-generated (no plain Secret committed). - Commit generated kea.unkin.net JSON schemas for kubeconform; register dhcp-system in the platform ApplicationSet + AppProject. Client-class object names are lowercased (`legacy`/`uefi-64`) to satisfy RFC1123 since the operator renders the kea class name from metadata.name. https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Reviewed-on: #333 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
131b4e1695 |
Add bind-external namespace for externally-reachable zones (#329)
We self-delegate `_acme-challenge.unkin.net` into an `acme.unkin.net` zone we serve ourselves, so cert-manager can solve Let's Encrypt DNS-01 over RFC2136/TSIG. That needs a publicly-reachable authoritative BIND, separate from the internal estate. - Add app `bind-external` (base + au-syd1 overlay); register it in the platform ApplicationSet and AppProject destinations (bind-operator already watches all namespaces). - Add BindCluster `bind-external`: authoritative-only, recursion off, no forwarding, transfers denied except the keyed catalog/zone AXFR; 2 replicas; primaryService is a dmz-pinned PureLB LoadBalancer at `198.18.199.53`. - Add BindZone `acme.unkin.net` (primary, dynamicUpdate) and BindTSIGKey `certmanager` (hmac-sha256), whose Secret `certmanager-tsig` reflects into the `cert-manager` namespace for the rfc2136 solver. Pairs with argocd-apps #327 (the ClusterIssuers) and a one-time Google Cloud DNS delegation + NAT of the public IP :53 to `198.18.199.53`. --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Reviewed-on: #329 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
8d70149467 |
Add Tier-1 per-app Vector transform pipelines (structured logs) (#318)
Why: the logging aggregator wrote every event through the generic catch-all shape. The Tier-1 survey picked six high-value log sources that warrant structured parsing into logs.raw columns/fields for real querying. How: - Two-stage routing in `aggregator.yaml`: `app_route` peels off the six Tier-1 streams by subject / VM source tag (mutually exclusive — no double-insert); everything else falls through `app_route._unmatched` to the unchanged generic k8s/vm catch-all. - Six parse transforms emit the full `logs.raw` shape plus structured `.fields` (Map(String,String) — no DDL change): authentik (JSON), traefik (JSON access), vault audit (JSON), nginx access+error (regex), haproxy httplog (regex), glauth (JSON). - Companion flip: traefik-system access logs to `format: json` (both overlays) so `traefik_parse` has structured input. - 15 new `vector test` cases (routing + field extraction) in `aggregator-tests.yaml`; all green locally (vector 0.57). Live now: authentik + traefik (k8s). Awaiting the puppet-side vector rollout (logs.vm.* with `.file`/`.SYSLOG_IDENTIFIER` tags per the documented convention): vault-file, nginx, haproxy, glauth — transforms are present and unit-tested so they light up automatically. Note: geoip enrichment for nginx/traefik client IPs is a separate prerequisite — no enrichment table exists in the aggregator yet; these transforms extract `client_ip` ready for it. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #318 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
5f87d0c96d |
gitea: use git.k8s ROOT_URL for testing (#317)
Points the new gitea at its own k8s route so it can be exercised (login, browse, OIDC callback) before the git.unkin.net data cutover — ROOT_URL currently resolves to the live VM forge, which would break links on the k8s route. Flips back to git.unkin.net at cutover. - set gitea DOMAIN/ROOT_URL/SSH_DOMAIN to git.k8s.syd1.au.unkin.net Reviewed-on: #317 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
a04dcc2975 |
Add k8s Gitea deployment (migration target for git.unkin.net) (#309)
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md. - add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute - add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222 - register gitea in the platform ApplicationSet + AppProject - add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback) Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc. Reviewed-on: #309 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
dfb495d771 |
Trust internal CA for Authentik SSO; canonical identity.unkin.net for NetBox (#314)
Authentik is canonical at https://identity.unkin.net, served by the internal unkin.net CA. Grafana, LiteLLM and NetBox failed OIDC discovery because their images don't trust that CA (x509: unknown authority); NetBox also still pointed at the secondary admin host. - grafana: mount the reflected vault-ca-cert; set generic_oauth `tls_client_ca`. - litellm: `combine-certs` init builds public+internal CA bundle; `SSL_CERT_FILE` + `REQUESTS_CA_BUNDLE` point at it. - netbox: flip OIDC issuer to identity.unkin.net; same combine bundle for python-social-auth (`requests`). - docs: record the Rancher manual runtime step (issuer + CA in the auth config). Validated: kustomize build + kubeconform + pre-commit. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #314 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
9c10b9096a |
Replace vector-archiver with logarchiver (#308)
## Why The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed. ## Changes - Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring. - Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch. - Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`). - Remove the vector-archiver Helm release, values and pipeline ConfigMap. Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #308 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
96afbcf5e1 |
Fix NATS auth: wrap env-var passwords in << >> so the server expands them (#306)
## Why After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient). ### Evidence - Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped. - The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected. - **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**. ### Root cause The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them. ## What Wrap all three user passwords in `<< >>`: ``` password: << $NATS_ADMIN_PASSWORD >> # (+ producer, consumer) ``` Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted). This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password. ## Verified end-to-end nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully. ## Expected recovery after merge + sync 1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords. 2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config. 3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes. 4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3. 5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises. ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix. ## Validation kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #306 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
7dddf8c5aa |
Fix logging deploy: enable Vector env interpolation; operator watches logging ns (#301)
## Why PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here. ## Root cause 1 — Vector env-var interpolation is off by default in 0.57 `vector-aggregator` and `vector-archiver` crash-looped with `async_nats::connector: authorization violation`; `vector-agent` / `vector-vm-ingest` were "Running" but silently failing to publish. Diagnosis (evidence): - The NATS server config **does** expand `$NATS_*_PASSWORD` (a `nats` CLI login with the real secret value authenticated fine), and the ACL was **not** the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation). - The failure reproduces locally: a hardcoded password connects; the **same value via `${NATS_CONSUMER_PASSWORD}` fails**. Configuring the server to expect the literal string `${NATS_CONSUMER_PASSWORD}` makes Vector connect — proving **Vector sends the literal, un-interpolated string**. - `vector --help` shows `--dangerously-allow-env-var-interpolation` — in 0.57 `${VAR}` interpolation is **opt-in**. An unset-var test confirms interpolation is off (no "unknown env var" error). - Verified fix: with `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` → **connects and authenticates**. Every tier uses `${...}` for auth (`${NATS_*_PASSWORD}`, `${CLICKHOUSE_*}`), so the env var is added to **all four** vector deployments. (This slipped past CI because `vector test` never opens the NATS connection.) ## Root cause 2 — operator watches only its own namespace `kubectl get chi -n logging` showed the `logs` CHI existed but with **empty status / no finalizer** — the operator never touched it, so the `logging-logging` Argo sync was stuck `Progressing` on *"waiting for healthy state of ClickHouseInstallation/logs"*, and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed). Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states `watchNamespaces: []` (our value) makes the operator **watch only its own namespace** (`clickhouse-system`). The CHI is in `logging`. Fix: `watchNamespaces: ["logging"]` → operator config `watch.namespaces.include: [logging]`. ## Changes - `apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml`: add `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true`. - `apps/overlays/au-syd1/clickhouse-system/values.yaml`: `watchNamespaces: ["logging"]`. No NATS ACL change (the original narrow ACL is correct). No secret/base changes. ## Expected recovery after merge + sync 1. clickhouse-system syncs → operator config gains `logging` → operator restarts → reconciles the `logs` CHI → CHI pod comes up healthy. 2. `logging-logging` sync unblocks → PostSync hooks run → JetStream `LOGS` stream + `transform`/`archiver` consumers created; `logs.raw` table created. 3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3. 4. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` increases. ## Validation kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders `watch.namespaces.include: [logging]`; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #301 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
d1a7646e09 |
Bump Woodpecker CI to v3.16.0 (#297)
## Why Upgrade the Woodpecker CI instance from v3.14.1 to v3.16.0 to pick up upstream fixes and the k8s-backend privilege-escalation hardening (GHSA-qf34-295c-26v8). Server and agent must move together. The chart is pulled at build time (`kustomize build --enable-helm`); the image tag defaults to the chart `appVersion`, so bumping the chart moves both images. Chart 3.6.5 -> appVersion 3.16.0 (verified against upstream `helm/charts/woodpecker/Chart.yaml`); subchart deps are unchanged from 3.6.3 (server 3.0.1, agent 2.0.1), so there is no values-schema migration. ## What - Bump the woodpecker helmChart from `3.6.3` (appVersion 3.14.1) to `3.6.5` (appVersion 3.16.0) in the au-syd1 overlay. Server + agent both render as `v3.16.0`. - Set `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true"` on the agent. ## CRITICAL: k8s backend serviceAccountName gating (required change) v3.16.0 (PR #6792, GHSA-qf34-295c-26v8) gates step-level `serviceAccountName` behind a new agent flag `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP`, **default `false`**. When disabled, any `backend_options.kubernetes.serviceAccountName` set by a pipeline is **silently ignored** and the namespace `default` SA is used instead. Every terraform pipeline in the estate sets `backend_options.kubernetes.serviceAccountName` (e.g. `terraform-git`, `terraform-vault`, `terraform-artifactapi`) and relies on that SA for Vault k8s auth / Consul state. Without this flag those jobs would run as `default` and lose their Vault identity. This PR sets the flag to `true` to preserve current behaviour. No other newly-gated backend_options keys (pod labels/annotations from step, native secrets) are used by the estate. ## Migration / rollback - **DB migration:** Woodpecker auto-migrates the schema (xorm) on server start; migrations are forward-only and NOT reversible. The 3.14 -> 3.16 changelogs do not call out a data-destructive migration, but a **DB backup (CNPG cluster `woodpecker`) should be taken before merge**. - **In-flight pipelines:** merging rolls the server StatefulSet and agents; any running pipelines are interrupted and will need re-running. - **Rollback:** re-pin chart `3.6.5` -> `3.6.3` reverts the images to v3.14.1, but because migrations are one-way, a clean rollback requires **restoring the CNPG DB from the pre-merge backup**, not just pinning the old image. ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/woodpecker` renders `woodpecker-server:v3.16.0` and `woodpecker-agent:v3.16.0`; agent carries the new env var. - `kubeconform` (k8s 1.33.7): 24/24 resources valid. - pre-commit (yamllint + checks): all pass. ## Follow-up (not in this PR) Woodpecker images are pulled from `docker.io` / `ghcr.io` directly, not the artifactapi proxy. Proxying them via artifactapi is a possible follow-up but out of scope for this version bump. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #297 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
f45cb6989f |
Add ClickHouse + Vector + NATS JetStream centralized logging (with S3 raw archive) (#296)
## Why Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists. ## Topology `edge (publishers) → JetStream → consumers → sinks` - **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets. - **Edge publishers (thin)** — no parsing, just a routing subject: - `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`. - `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.) - **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now). - **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`. - **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job. ## Streams / consumers / auth - Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment. - **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret. ## S3 / retention `ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here. ## Replay runbook - **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time. - **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window. ## Validation - `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**. - `pre-commit` (yamllint, check-json, no-plain-secrets) clean. - **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs. - **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets). ## Known upstream caveat Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook. ## Prerequisites (manual, one-time) ``` # ClickHouse PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1) vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \ username=vector password="$PW" password_sha256_hex="$HASH" # NATS vault kv put kv/kubernetes/namespace/logging/default/nats-auth \ admin_password=$(openssl rand -base64 24) \ producer_password=$(openssl rand -base64 24) \ consumer_password=$(openssl rand -base64 24) ``` No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR. ## Open decisions (defaults chosen, flag to change) - **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.** - **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side). - **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer. - **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA. - **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs). - **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --- ## Update: images via artifactapi, DHI, stateless transform tier **Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull. ### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`) | Image | Upstream | artifactapi path | DHI? | |---|---|---|---| | clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job | | altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI | | altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI | | bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI | | nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats | | natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI | | natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI | | timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened | | timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step | **DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace. ### Transform tier is now a stateless Deployment Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**. **Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.) --- ## Update: 7d retention, tunable limits ConfigMap, honest sizing - **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict. - **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3. - **Honest 7d sizing (stated assumption — please sanity-check against real volume):** - Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica. - Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica. - **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.** - ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob. - Replay window in the runbook is now **7d** (beyond that → the S3 archive). --- ## Update: retention cut to 3 days (both stores), PVCs shrunk Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design. | Store | Retention | Byte cap | PVC/node | Replicas | Total | |---|---|---|---|---|---| | NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) | | ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) | **NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC. **ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table. The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL. ⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit). Reviewed-on: #296 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
2ee4c9518c |
Deploy NetBox (CNPG Postgres, Valkey, Authentik SSO) (#299)
## Why NetBox becomes the source of truth for host/interface/IPAM data as part of retiring Cobbler. The ENC role already moved to encapi; a kickstart-generation microservice that consumes NetBox comes later and is intentionally **out of scope** here. ## Change Adds the `netbox` app (namespace `netbox`, platform project) using the netbox-community helm chart via the OCI helm-through-kustomize pattern, plus estate-native data stores: - **NetBox** — chart `8.3.40` (appVersion `v4.6.5`), `oci://ghcr.io/netbox-community/netbox-chart`. 2 web replicas, 1 worker; bundled Postgres/Valkey subcharts disabled. Media on RWX CephFS so replicas share uploads. Chart `helm.sh/hook: test` Pod dropped via a kustomize delete patch (we deploy through ArgoCD, not `helm test`). - **CNPG Postgres** — `netbox-postgres`, 2 instances, PG 18, `cephrbd-fast-delete`. Standard per-cluster S3 backup: `barmanObjectStore -> s3://cnpg-netbox`, cephrgw `ObjectStoreUser` + `Bucket`, nightly `ScheduledBackup` at **03:40** (`0 40 3 * * *` — next free slot after grafana's 03:20), 30d retention. A pgbouncer `Pooler` (session mode) fronts it; NetBox connects via `netbox-postgres-pooler-rw`. - **Valkey** — standalone Deployment (`valkey/valkey:8-alpine`), AOF-persistent PVC on `cephrbd-fast-delete`. One instance: DB 0 = RQ task queue, DB 1 = cache. No auth (in-cluster, namespace-isolated). Chosen over the bundled Bitnami subchart to keep image control in-estate and avoid Bitnami's legacy-image churn; mirrors the litellm standalone-cache pattern. - **Ingress** — `Gateway` + `HTTPRoute` at `netbox.k8s.syd1.au.unkin.net` (`traefik-internal`, `vault-issuer` cert into `netbox-tls`, external-dns to the internal VIP), HTTP->HTTPS 301. - **Secrets** — all via VSO `VaultStaticSecret` (`postgres-credentials`, `netbox-secret-key`, `netbox-superuser`, `oauth-credentials`); no plain Secrets committed. The shared `default` k8s-auth role already binds `*` namespaces with a namespace-templated KV policy, so **no terraform-vault change is needed**. - **Authentik OIDC SSO** — `remoteAuth` wires `REMOTE_AUTH_ENABLED` + the `OpenIdConnectAuth` backend via chart values; `SOCIAL_AUTH_OIDC_*` via `extraConfig` (the chart's config loader globs `/run/config/extra/*/*.yaml`). The client secret is injected as a YAML fragment mounted from the Vault-synced `oauth-credentials` secret. New users auto-provision on first login. - Registers `netbox` in the platform ApplicationSet and AppProject (destination namespace + chart sourceRepo). ## Image table (source -> artifactapi -> DHI decision) All images flow through the estate's containerd registry mirrors; the allowlist patterns gate them. DHI (Docker Hardened Images) require authenticated pulls from a Docker Hub `dhi/` org and are **not** reachable through the anonymous mirror, so upstream official is used throughout. | Image | Upstream | Mirror / allowlist | Allowlisted? | DHI decision | |---|---|---|---|---| | NetBox app/worker/housekeeping | `ghcr.io/netbox-community/netbox:v4.6.5` | ghcr remote, `^netbox-community/` | **added in terraform-artifactapi #17** | No DHI published; upstream official | | CNPG Postgres | `ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie` | ghcr, `^cloudnative-pg/` | already | estate-standard CNPG image | | Valkey | `docker.io/valkey/valkey:8-alpine` | dockerhub, `^valkey/valkey` | already | DHI not anon-pullable; upstream official | | NetBox init (perms) | `docker.io/busybox:1.38.0` | dockerhub, `^library/busybox` | already | upstream official | | Worker wait-for-backend | `docker.io/rancher/kubectl:v1.36.2` | dockerhub, `^rancher/` | already | upstream official | ## Cross-repo PRs (merge order) 1. **terraform-artifactapi #17** (`^netbox-community/` ghcr allowlist) — **merge before** this PR so the NetBox image pulls on first sync. 2. **terraform-authentik #11** (OIDC provider/application) — independent; SSO works once applied + the secret is seeded. 3. This PR. > Note: the CNPG S3-backup stanza pattern here is identical to PR #298 (already in main); it merges cleanly regardless of ordering. ## One-time Vault seeds (before/at first sync) ``` # App DB user (CNPG bootstrap + NetBox both consume this) vault kv put kv/kubernetes/namespace/netbox/default/postgres-credentials \ username=netbox password="$(openssl rand -base64 30)" # Django SECRET_KEY vault kv put kv/kubernetes/namespace/netbox/default/netbox-secret-key \ secret_key="$(python3 -c 'import secrets;print(secrets.token_urlsafe(60))')" # Bootstrap superuser vault kv put kv/kubernetes/namespace/netbox/default/netbox-superuser \ username=admin email=admin@unkin.net \ password="$(openssl rand -base64 24)" api_token="$(openssl rand -hex 20)" # OIDC client secret — ONE value stored two ways (raw for Authentik, YAML for NetBox) CS="$(openssl rand -base64 30)" vault kv put kv/kubernetes/namespace/netbox/default/oauth-credentials \ client_secret="$CS" oidc.yaml="SOCIAL_AUTH_OIDC_SECRET: \"$CS\"" ``` Seed `oauth-credentials` **before** applying terraform-authentik #11 (that apply reads `client_secret`). ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/netbox` — clean. - `kubeconform` (CI args, k8s 1.33.7): **27/27 valid, 0 invalid**. - `pre-commit` (yamllint + no-plain-secrets) on all changed files — pass. ## Out of scope The PXE/kickstart microservice that will consume NetBox — not scaffolded here. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #299 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
b725bf7dcf |
Add PTR to external-dns managed record types (#295)
Follow-up to #294: --create-ptr validates that PTR is in --managed-record-types, and setting that flag replaces the default list, so A/AAAA/CNAME are re-stated alongside PTR (verified against the v0.21.0 binary's --help: default A,AAAA,CNAME). Pod is crash-looping on config validation until this merges. - Adds --managed-record-types=A,AAAA,CNAME,PTR to the external-dns args MERGE ASAP. Reviewed-on: #295 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
faa256c764 |
Fix external-dns PTR flag (--create-ptr) (#294)
external-dns v0.21.0 rejects --rfc2136-create-ptr (my error in #292) and the pod is crash-looping on flag parsing. The correct flag in this version is the generic --create-ptr, which synthesizes PTRs for A records whose reverse zone is in the domain filter (200.18.198.in-addr.arpa already is). - Replaces --rfc2136-create-ptr with --create-ptr Record reconciliation is stalled until this merges (serving unaffected — bind answers normally). MERGE ASAP. Reviewed-on: #294 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
eff97b8d03 |
Enable PTR synthesis in external-dns (#292)
Post-cutover verification (argocd-apps#288) found the reverse zone 200.18.198.in-addr.arpa empty: external-dns only writes PTRs when --rfc2136-create-ptr is set, and nothing else feeds that zone. Reverse resolution for the k8s LB range was already absent for clients pre-migration (no forwarder existed), so this completes the reverse path rather than fixing a regression. - Adds --rfc2136-create-ptr to the external-dns rfc2136 args Verification after merge: dig -x 198.18.200.2 @198.18.200.7 returns puppetca.k8s.syd1.au.unkin.net (allow a reconcile cycle + negative-cache expiry). Reviewed-on: #292 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
8dc65779a5 |
Repoint external-dns at the in-cluster bind-externaldns primary (#287)
## Why
Step 2 of 3 in the external-dns → in-cluster bind migration. external-dns currently pushes RFC2136 updates to the legacy VM `ausyd1nxvm2127.main.unkin.net`. This repoints it at the in-cluster `bind-externaldns` primary so the in-cluster zone becomes the live source of truth for `k8s.syd1.au.unkin.net` + `200.18.198.in-addr.arpa`.
## Changes
- Points `--rfc2136-host` at `bind-externaldns-primary.bind-internal.svc.cluster.local` (verified live: ClusterIP Service `bind-externaldns-primary` exists in `bind-internal`).
- Reads TSIG `secret` + `algorithm` from Secret `externaldns-key-tsig` (reflected by PR 1) instead of the Vault-backed `externaldns-tsig`.
- Keeps port, zones, keyname, and `txtOwnerId: k8s` unchanged.
- Leaves the old Vault manifests (`apps/base/externaldns/{vaultauth,vaultstaticsecret}.yaml`) in place as the rollback path; their removal is a later cleanup PR.
## Merge gate
- PR 1 (`benvin/externaldns-tsig-reflect`) merged, AND
- `kubectl -n externaldns get secret externaldns-key-tsig` returns keys `secret` + `algorithm`.
## TXT-registry note
`policy: sync` + `registry: txt`: on first reconcile against the (currently empty) in-cluster zone, external-dns re-creates all managed A/CNAME records and their ownership TXTs from scratch. This is expected and populates the zone.
## Verification (after merge)
```
kubectl -n externaldns logs deploy/externaldns --tail=100 | grep -Ei 'rfc2136|BADKEY|NOTAUTH|added|update'
dig +short @198.18.200.8 A puppet.k8s.syd1.au.unkin.net
```
Logs should show updates to `bind-externaldns-primary...` with NO `BADKEY`/`NOTAUTH`; sampled A records should start resolving against the in-cluster server (198.18.200.8) as the sync populates the zone.
## Rollback
Revert this PR (host + secret name back to `ausyd1nxvm2127.main.unkin.net` / `externaldns-tsig`). The legacy VM is untouched and still authoritative.
Reviewed-on: #287
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
|
||
|
unkinben
|
3920d76ba2 |
Add VPA CRDs and kubeconform schemas (#284)
Split out of #281 per review: the VerticalPodAutoscaler CRDs land first, together with their generated kubeconform schemas, so CI can validate the VPA objects that follow. - Adds apps/base/vpa-system with the v1.7.0 VPA CRDs served via the artifactapi github remote (terraform-artifactapi#14, merged) - Adds schemas/autoscaling.k8s.io/ (verticalpodautoscaler + checkpoint, v1 and v1beta2) generated with the same transform as ci/generate-schemas.sh (from the CRD manifest rather than the live cluster, since the CRDs are not installed yet) - Wires the vpa-system overlay into the platform applicationset Verified: kustomize renders both CRDs; a sample updateMode Off VPA passes kubeconform against the new schemas. Merge before #281 (recommender) and #283 (per-workload VPAs). Reviewed-on: #284 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
a633835550 |
Enable ACLs on k8s consul with Vault-sourced bootstrap token (#280)
## Why Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with `default_policy: deny` and `down_policy: extend-cache`; the k8s cluster currently runs with **ACLs disabled**. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the **same** `initial_management` token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git. ## Changes - Enable `global.acls.manageSystemACLs` so the chart manages system ACL tokens/policies for consul components. - Point `global.acls.bootstrapToken` at a pre-existing Kubernetes secret `consul-bootstrap-acl-token` (key `token`); chart 1.9.7 supports this, and when the secret is populated the `server-acl-init` job **skips bootstrapping** and adopts that token as the management token (renders `-bootstrap-token-secret-name`/`-bootstrap-token-secret-key`, verified in the kustomize output). - Add a `VaultAuth` (mount `k8s/au/syd1`, role `default`, SA `default`) and `VaultStaticSecret` in the `consul` namespace that sync `kv/kubernetes/namespace/consul/default/bootstrap-acl-token` into the `consul-bootstrap-acl-token` k8s secret via VSO (mirrors the encapi pattern). - Merge the `acl` block (`enabled`, `default_policy: deny`, `down_policy: extend-cache`, `enable_token_persistence`) into the server `extraConfig` to match the VM posture. ## OPERATIONAL NOTE — required BEFORE merge The user MUST place the VM cluster's `initial_management` token in Vault first, or ACL bootstrapping will generate a *different* token and break the mirror: ``` vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token> ``` VSO then syncs it into the `consul-bootstrap-acl-token` secret before the `server-acl-init` job runs. No terraform-vault change is needed: the wildcard `default` k8s-auth role (`bound_service_account_namespaces: ['*']`) plus the templated `kv/kubernetes/default` policy already grant the `consul` namespace `default` SA read on `kv/kubernetes/namespace/consul/default/*`. ## Risk / expected behavior - Enabling ACLs **rolls the 5 servers** (StatefulSet update) and runs a `server-acl-init` job. - With `default_policy: deny`, previously-anonymous operations are denied. The `vault` service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace `vault`) via their `service_registration "consul"` stanza (catalog entry has `ServiceMeta.external-source: vault`, port 8200). After the flip this registration will be **denied** unless Vault is given a Consul ACL token with `service:write` on `vault` (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration. - Anonymous HTTP API reads (e.g. `/v1/status/leader`) will also be denied post-merge unless a token is supplied — see PR 2 verification note. ## Ordering Independent of the API-route PR (`benvin/consul-api-route`). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done. Reviewed-on: #280 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
6d7cdd59e3 |
pdbmux: deploy the merging PuppetDB proxy in-cluster (#275)
## Why During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move between them as they migrate. `node-lookup` (and `pblastreport`) need a single, consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy that provides exactly that. Per the all-in-kubernetes estate direction it runs as an in-cluster service, not a per-VM systemd unit. pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) — split out of the earlier node-lookup prototype — and is released as a container image on its own `v*` tags. ## Changes - Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway, httproute), modeled directly on the encapi app. - Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080, `/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap. - Backends: `old=http://puppetdbapi.service.consul:8080`, `new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the in-cluster address is preferred over the external gateway). `new` is primary/prefer, merge = freshness. - Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal` Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a port-80 Service — same shape as the puppetdb/encapi gateways — so VM/workstation `node-lookup` can reach it. - Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject (`pdbmux` namespace destination), exactly as encapi is wired. No new woodpecker ServiceAccount is required: the pdbmux image push uses the `docker-buildx` plugin against the Gitea registry with the `default` SA (same as encapi), not artifactapi. ## Verification - `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves to `git.unkin.net/unkin/pdbmux:v0.1.0`). - ApplicationSet + AppProject YAML validated. ## Merge gates 1. The pdbmux repo initial-content PR (unkin/pdbmux#1) must merge first. 2. `v0.1.0` must then be tagged on the pdbmux repo so the image `git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s `.woodpecker/docker.yaml`. 3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.) --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #275 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
19f8055144 |
Deploy cephrgw-operator to cephrgw-system (#261)
## Why The new `cephrgw-operator` provisions Ceph RGW (S3) buckets and access keys (RW/RO) from Kubernetes CRDs via the Ceph manager dashboard API. This deploys it as a platform app. ## Changes - Add `apps/base/cephrgw-system`: namespace, ServiceAccount + ClusterRole/Binding (manage `ceph.unkin.net` CRDs, Secrets, leader-election leases), and the operator Deployment. CRDs are pulled from the operator repo at tag `v0.1.0`; the Deployment sources dashboard credentials from the `cephrgw-credentials` Secret via `envFrom` and carries the reloader annotation. - Add `apps/overlays/au-syd1/cephrgw-system` referencing the base. - Register `apps/overlays/*/cephrgw-system` in the platform ApplicationSet. The platform AppProject already permits `*-system` namespaces and the Namespace/ClusterRole/CRD cluster resources, so no project change is needed. ## Ordering / dependencies - Depends on the Gitea repo from terraform-git #34 and on the operator being pushed + tagged **v0.1.0** (image `git.unkin.net/unkin/cephrgw-operator:v0.1.0` and the raw CRD `install.yaml` at that tag). The `kubeconform` check will stay red until v0.1.0 exists, then go green. - The `cephrgw-credentials` Secret must be created out-of-band in `cephrgw-system` (see the operator's `docs/ceph-setup.md`); it is intentionally **not** managed in GitOps. --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #261 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
2409a4d3a0 |
reduce CPU requests across observability, authentik, bind, woodpecker (#248)
Reduce reserved-but-unused CPU requests across several workloads (limits unchanged). | Workload | CPU request | |---|---| | vmstorage (vmcluster) | 1 → 250m | | vmagent | 500m → 250m | | authentik server | 250m → 50m | | authentik worker | 250m → 100m | | bind authoritative | 100m → 20m | | bind externaldns | 100m → 20m | | bind resolvers | 100m → 20m | | woodpecker agent | 100m → 50m | Reviewed-on: #248 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
 benvin
|
3320c823b3 |
chore: remove paperclip (#245)
no longer using paperclip, remove it from au-syd1 cluster --------- Co-authored-by: Ben Vincent <ben@unkin.net> Reviewed-on: #245 |
||
|
unkinben
|
95c5d71222 |
grafana: deploy Grafana instance, datasource and dashboards (#238)
## Why Deploys Grafana in-cluster (observability project) via the grafana-operator, mirroring the puppet Grafana but modernised — **CNPG** for state, **Authentik OIDC** for auth — and ports the live datasource + dashboards in as CRs. Depends on: grafana-operator (#235, merged), grafana schemas (#236, merged), Authentik OIDC (terraform-authentik #2), Vault seeds (done), and `^grafana/` image proxy (terraform-artifactapi #5). ## Changes (`apps/base/grafana`) - **CNPG** postgres Cluster + rw Pooler (db `grafana`); **VaultAuth** + **VaultStaticSecrets** pulling `postgres`/`oauth` credentials from `kv/kubernetes/namespace/grafana/default/*`. - **Grafana CR**: postgres backend via the pooler; Authentik `generic_oauth` (client id/secret from the Vault-synced secret, openid/email/profile scopes, group→role mapping); `root_url` grafana.k8s.syd1.au.unkin.net. - **1 GrafanaDatasource** — k8s VictoriaMetrics via the operator `vmselect-main` service; reuses the previous default datasource uid so the imported dashboards resolve unedited. - **13 GrafanaDashboards** (gzipJson) exported from the current grafana. - **Gateway API** (traefik-internal) + HTTPRoute for grafana.k8s.syd1.au.unkin.net. - Registered in the observability ApplicationSet + project. ## Review notes - OAuth `role_attribute_path` maps Authentik group `grafana-admins` → Admin, else Viewer — **confirm the group name**. - `database.ssl_mode: require` against the CNPG pooler — adjust if the pooler isn't serving TLS. - The `VictoriaLogs - cluster` dashboard has no in-cluster logs datasource yet (no VictoriaLogs in k8s) — included for completeness, will be empty until one exists. - `make kubeconform` clean (24 resources, validated against the strict grafana schemas). Reviewed-on: #238 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
0f6c9e5502 | Merge pull request 'grafana-system: deploy grafana-operator' (#235) from benvin/grafana-operator into main | ||
|
unkinben
|
5f8f2b1c70 |
observability: migrate VictoriaMetrics to operator CRDs + Consul SD (#234)
## Why The k8s au-syd1 VictoriaMetrics stack ran as two helm charts and only scraped in-cluster targets. The victoria-metrics-operator already runs in vm-system, so this moves the stack onto operator-managed CRDs. That unlocks VMServiceScrape/VMPodScrape (auto-converted from Prometheus ServiceMonitors, used by a follow-up PR) and adds Consul service discovery so the cluster scrapes the **same puppet-prod targets** as the puppet vmagent. Also shrinks vmstorage 3 → 2 (Ceph-backed, replicationFactor 2). ## Changes - Add **VMCluster `main`**: vmstorage 2 replicas (cephrbd-fast-delete 200Gi, 180d retention, replicationFactor 2), vminsert/vmselect 2 replicas + HPA (2–10, 60% cpu). - Add **VMAgent `main`**: retains the kubernetes SD jobs (apiservers/nodes/cadvisor), `selectAllByDefault` for VMServiceScrape/VMPodScrape, and a **Consul SD job** against `consul.service.consul` (resolves to the puppet Consul from pods) replicating the puppet vmagent relabels — keep tag `metrics`, `__scheme__` from `metrics_scheme`, `job` from `metrics_job`. TLS is **verified against the reflected `vault-ca-cert`** (no insecure skip-verify). - Expose vmselect/vminsert/vmagent via **Gateway API** (traefik-internal Gateway + HTTPRoute, http→https redirect), same hostnames. - Remove the two helm charts, their values files, and vendored charts. ## Notes - Data wipe on cutover is acceptable (confirmed) — old helm PVCs can be deleted. - Verify at rollout: pods resolve `*.main.unkin.net` node FQDNs (needed for CA SAN match on scrape targets); `/targets` shows `job=consul`. Reviewed-on: #234 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
3df8ac2779 |
grafana-system: deploy grafana-operator
Adds the grafana-operator (grafana.integreatly.org CRDs + controller) so Grafana and its dashboards/datasources can be managed declaratively as CRs in a follow-up PR. Sits in the platform project like the other operators (vm-system, cnpg-system). Changes: - Add grafana-system namespace + grafana-operator helm chart v5.24.0 (watches all namespaces). - Render CRDs inline (crds.immutable: false) so ArgoCD installs/manages the 13 grafana.integreatly.org CRDs instead of the skipped helm crds/ subchart. - Register apps/overlays/*/grafana-system in the platform ApplicationSet. |
||
|
unkinben
|
333e638e24 |
deploy encapi to au-syd1 (#230)
## Why encapi is the new Postgres-backed Puppet ENC that replaces Cobbler (Go API + encapi-cli + terraform provider). It needs to run somewhere reachable by the puppet masters (`encapi-cli classify`) and every node's `enc_direct_facts` fact. Deploy it in k8s alongside artifactapi, exposed at `encapi.k8s.syd1.au.unkin.net`. ## Changes - add `apps/base/encapi/`: namespace, deployment (`git.unkin.net/unkin/encapi`, port 8000, `/healthz` probes), service, gateway + httproute (`encapi.k8s.syd1.au.unkin.net`, traefik-internal), configmap (DB coordinates), CNPG cluster + pooler (database `encapi`), and VaultAuth + VaultStaticSecrets (`postgres-credentials`, `environment`) - add `apps/overlays/au-syd1/encapi` overlay referencing the base - register `apps/overlays/*/encapi` in the platform ApplicationSet so ArgoCD picks it up ## Notes - Mirrors the artifactapi pattern (VaultAuth role `default`, namespace-scoped VSO paths `kv/kubernetes/namespace/encapi/default/*`). - Before first sync, seed the Vault KV secrets: `environment` must carry `DBPASS` (matching the CNPG owner password) and `ENCAPI_WRITE_TOKEN`; `postgres-credentials` carries the CNPG owner username/password. - `kustomize build apps/overlays/au-syd1/encapi` validates clean (11 resources). --------- Co-authored-by: unkinben <neotheo@gmail.com> Reviewed-on: #230 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
ce8ebc71ce |
Consolidate BIND DNS into one bind-internal namespace (#225)
**HOLD until v0.1.3 is tagged/built** (operator #4 merged + tagged) — this PR bumps the operator to v0.1.3, whose CRD adds the `clusterRef` field these keys use. ## Why Put all BIND DNS services in one `bind-internal` namespace and name the StatefulSets clearly. ## Changes - 3 clusters consolidated into `bind-internal`, StatefulSets renamed **bind-authoritative** / **bind-resolvers** / **bind-externaldns**; LBs kept on 198.18.200.6/.7/.8; external-dns hostnames renamed to match - `clusterRef` added to `transfer-key` (→ bind-authoritative) and `externaldns-key` (→ bind-externaldns) so keys are scoped per cluster - removed the old `ns-auth`/`ns-resolver`/`ns-externaldns` apps; ApplicationSet + AppProject now list `bind-internal` - bumped `bind-system` operator to **v0.1.3** (CRD link + image) - operator stays in `bind-system` ## Deploy impact ArgoCD prunes the old ns-* namespaces (StatefulSets/PVCs — data is only seed SOA+NS, no migrated records yet) and creates the renamed clusters in bind-internal. ## Validated `kustomize build` → 28 docs (3 BindCluster, 20 BindZone, 2 catalog, 2 keys, ns); kubeconform clean. Reviewed-on: #225 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
7c9a697452 |
Deploy binddns-externaldns (RFC2136 dynamic cluster) (#222)
Part of the bind rollout split. **Merge #219 (bind-operator) first** — stacked on it; diff reduces to the binddns-externaldns files once #219 merges. ## Why The external-dns tier (replaces 3x Puppet external-dns servers): an authoritative cluster whose zones accept RFC2136 TSIG updates from external-dns. ## Changes - `apps/base/binddns-externaldns`: authoritative `BindCluster` (3 replicas, LoadBalancer/PureLB), `BindTSIGKey` for RFC2136, namespace - au-syd1 `binddns-externaldns` overlay ## Deploy impact Creates the `binddns-externaldns` StatefulSet + LoadBalancer once merged. Reviewed-on: #222 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
6affc5d8f4 |
Deploy binddns-resolver (recursive resolver cluster) (#221)
Part of the bind rollout split. **Merge #219 (bind-operator) first** — stacked on it; diff reduces to the binddns-resolver files once #219 merges. ## Why The recursive-resolver tier (replaces 3x Puppet only-resolver servers): 3 identical recursive servers with upstream forwarders. ## Changes - `apps/base/binddns-resolver`: resolver `BindCluster` (3 replicas, forwarders, LoadBalancer/PureLB), namespace - au-syd1 `binddns-resolver` overlay ## Deploy impact Creates the `binddns-resolver` StatefulSet + LoadBalancer once merged. Reviewed-on: #221 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
649ed07ab0 |
Deploy binddns-auth (authoritative BIND cluster) (#220)
Part of the bind rollout split. **Merge #219 (bind-operator) first** — this PR is stacked on it, so its diff will reduce to just the binddns-auth files once #219 merges. ## Why The authoritative masters tier (replaces 3x Puppet authoritative servers): pod-0 primary + 2 secondaries replicating via the catalog zone + AXFR/IXFR. ## Changes - `apps/base/binddns-auth`: authoritative `BindCluster` (3 replicas, LoadBalancer/PureLB), `BindCatalogZone`, transfer `BindTSIGKey`, namespace - au-syd1 `binddns-auth` overlay ## Deploy impact Creates the `binddns-auth` StatefulSet + LoadBalancer once merged. Reviewed-on: #220 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
4b8f9313c8 |
Deploy bind-operator (operator + CRDs) (#219)
First of a 4-PR split of the bind rollout (was #216). Deploys just the operator control plane so it can be verified before any DNS clusters exist. ## Why Roll out incrementally: operator + CRDs first, then each BIND tier as its own PR. ## Changes - `apps/base/bind-system`: operator Deployment (`git.unkin.net/unkin/bind-operator:v0.1.1`), RBAC, namespace; CRDs pulled from the operator repo by raw URL (`config/crd/install.yaml` @ v0.1.1) - au-syd1 `bind-system` overlay - register all four bind apps in `argocd/applicationsets/platform.yaml` (DNS overlays instantiate only when their dirs land in the follow-up PRs) - add `binddns-*` namespaces to `argocd/projects/platform.yaml` - add `schemas/bind.unkin.net/*.json` for kubeconform ## Deploy impact Operator pod + CRDs only. No DNS services yet — the operator is idle until BindClusters exist. ## Follow-ups (merge after this) binddns-auth, binddns-resolver, binddns-externaldns — one PR each. Reviewed-on: #219 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
7f1444fb38 |
Add Authentik identity provider deployment (#211)
## Summary - Deploy Authentik (identity.unkin.net) via Helm chart 2026.5.3 - CNPG PostgreSQL cluster (3 instances) with separate rw/ro poolers (2 instances each) - Redis with 5Gi persistent storage - Gateway API for HTTPS (identity.unkin.net) and LDAPS (ldap.k8s.syd1.au.unkin.net, ldap.main.unkin.net) - TLSRoute for LDAPS passthrough, HTTPRoute for external-dns record creation - Vault secrets for postgres credentials, authentik secret key, and S3 storage credentials - S3 storage via RadosGW (bucket: authentik) - 3 server replicas, 2 worker replicas - Woodpecker ServiceAccount for terraform-authentik CI - Platform applicationset and project updated ## Dependencies - terraform-git #15 (merged) — repo definition - terraform-vault #78 (merged) — auth roles and Consul ACL ## Vault secrets needed before deploy Write to `kv/kubernetes/namespace/authentik/default/`: - `postgres-credentials`: username + password - `authentik-credentials`: AUTHENTIK_SECRET_KEY - `s3-credentials`: S3 access key + secret key Reviewed-on: #211 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
cfca1e5278 |
Add age-api deployment (#210)
## Summary - Deploy age-api to the au-syd1 cluster - Uses configMapGenerator for people config with jaidi, ben, and sudaporn - Includes gateway, httproute, service, and deployment - Image: git.unkin.net/unkin/age-api:v0.1.0 Reviewed-on: #210 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
1b781e0885 |
feat(woodpecker): set workflow pod priority class to power (#175)
## Summary Sets `WOODPECKER_BACKEND_K8S_PRIORITY_CLASS: power` on the Woodpecker agent so all CI pipeline pods are scheduled with the `power` PriorityClass (value 100, preemptionPolicy: Never). This means pipeline pods can be evicted when the cluster is under pressure but won't preempt other workloads. ## Dependency Requires the `power` PriorityClass to exist on the cluster — deploy PR #174 (priority-classes app) first. ## Test plan - Trigger a pipeline run and confirm pods are created with `priorityClassName: power` - `kubectl get pod -n woodpecker -o jsonpath='{.items[*].spec.priorityClassName}'` Reviewed-on: #175 |
||
|
unkinben
|
ede25a3858 |
feat(platform): add priority-classes app with low/power/medium/high classes (#174)
## Summary - New `apps/base/priority-classes/` app with four `PriorityClass` objects managed via the `platform` ArgoCD project - Adds `apps/overlays/*/priority-classes` to the platform ApplicationSet generator - Adds `priority-classes` namespace to platform AppProject destinations (required even for cluster-scoped resources) | Class | Value | PreemptionPolicy | Intent | |---|---|---|---| | `low` | 100 | Never | Background work; evictable, won't preempt others | | `power` | 100 | Never | Compute-heavy but expendable (e.g. AI/ML workloads) | | `medium` | 10000 | PreemptLowerPriority | Standard services | | `high` | 100000 | PreemptLowerPriority | Critical services; preempts lower-priority pods | `PriorityClass` is already in the platform project's `clusterResourceWhitelist` so no project policy changes were needed. ## Test plan - ArgoCD syncs `platform-priority-classes` successfully - `kubectl get priorityclasses low power medium high` shows all four classes Reviewed-on: #174 |
||
|
unkinben
|
3990fbfe06 |
feat(vault): switch to Kubernetes service registration (#171)
Replaces Consul service registration with the native Kubernetes provider so Vault labels its own pods with active/standby/perf-standby status without requiring a Consul dependency.
## Changes
- `values.yaml`: swap `service_registration "consul"` for `service_registration "kubernetes" {}`, add `VAULT_K8S_NAMESPACE` and `VAULT_K8S_POD_NAME` env vars via downward API
- `role_k8s-service-registration.yaml`: Role + RoleBinding granting the `vault` service account `get`/`update`/`patch` on pods
- `kustomization.yaml`: include new RBAC file
Reviewed-on: #171
|