Why:
Jellyfin ships and gets validated first, ahead of the rest of the media stack.
Scoping this PR to jellyfin alone keeps the initial rollout small and lets the
HA fork prove out against the real library before the download and manager apps
follow.
How:
- Drop sonarr, radarr, prowlarr, bazarr, nzbget, and jellyseerr and their shared
media-apps foundation from this PR; they land in later PRs.
- Move jellyfin into its own jellyfin namespace and fold the namespace and the
static mediafs PV plus its RWX claim into the jellyfin base.
- Keep the static CephFS PV bound to the in-use mediafs library with
reclaimPolicy Retain and staticVolume true so nothing can reclaim it, mounted
into jellyfin by the movies and tvseries subPaths; keep redis, the fresh RWX
transcode scratch, the intel iGPU nodeSelector and i915 request, gateway, and
httproute.
- Scope the media AppProject and ApplicationSet to the single jellyfin
namespace and app, extensible as the remaining apps are added.
Why:
The media apps must serve and manage the actual media library, not empty
volumes. That library already exists on the puppet-managed CephFS filesystem
mediafs (mounted by the VM/incus instances at /shared/media) and is in active
use, so the k8s apps must mount it in place rather than provision fresh storage.
How:
- Replace the two fresh movies/tvseries PVCs with one static CephFS
PersistentVolume bound to mediafs and a single RWX media-library claim the
whole stack shares.
- Set the PV reclaim policy to Retain and mark it staticVolume so ceph-csi only
mounts the pre-existing storage and can never provision or reclaim it;
deleting the PVC or PV cannot destroy the underlying library.
- Reuse the live csi-cephfs cluster parameters (clusterID cephfs_csi_ssd_ec_4_1
for mon discovery, csi-cephfs/csi-cephfs-secret node-stage secret) with
fsName mediafs and rootPath / (the mediafs root that maps to /shared/media).
- Mount the library into each app by subPath so the tree matches the VM
layout: sonarr /mnt/tvseries (tvseries), radarr /mnt/movies (movies),
jellyfin and nzbget both subtrees; prowlarr keeps no library mount. The
jellyfin transcode PVC stays a fresh scratch volume.
- Whitelist PersistentVolume in the media AppProject so the cluster-scoped PV
can sync.
Why:
The media stack (jellyfin plus the sonarr/radarr/prowlarr/bazarr/nzbget/
jellyseerr apps) runs in the media-apps namespace but is deployed out-of-band
by terraform-k8s rather than GitOps. Bringing it under ArgoCD makes the stack
declarative, self-healing, and consistent with every other cluster workload,
and prepares terraform-k8s to drop the media-apps config.
How:
- Add a media AppProject scoped to the media-apps namespace and a media-apps
ApplicationSet that renders one Application per app plus a shared foundation.
- Add a shared media-apps foundation (namespace, media-apps-vault-reader
ServiceAccount, default VaultAuth on k8s/au/syd1, and the RWX movies/tvseries
library PVCs) that the whole stack mounts.
- Add per-app kustomize base and au-syd1 overlay for jellyfin and the six *arr
apps, using plain resource names (jellyfin, sonarr, ...) with fresh PVCs.
- Deploy jellyfin from the jellyfin-ha fork (Redis transcode store, RWX
transcode scratch) wired to the shared movies/tvseries library PVCs, keeping
the intel iGPU nodeSelector and gpu.intel.com/i915 request.
- Source API keys and nzbget credentials through VSO VaultStaticSecrets from
kv/service/media-apps/<app>; expose each app via a traefik-internal Gateway
and HTTPRoute at <app>.k8s.syd1.au.unkin.net.
- Register the media project and applicationset in the argocd bootstrap
kustomizations.
Replaces the isc-dhcpd PXE-boot VM with the kea-operator + an HA kea pair, GitOps-managed. Deploys on a new, unused anycast IP so nothing is cut over yet; the production cutover off the current dhcpd address (198.18.19.18) is a separate later task.
- Add `apps/base/dhcp-system`: namespace, kea-operator RBAC + Deployment (v0.1.0), VPA, and the 4 kea.unkin.net CRDs pulled from the operator repo at tag v0.1.0.
- Add CRs translating the legacy dhcpd config (source: puppet `roles/infra/dhcp/server.yaml`): KeaCluster `kea` (2 replicas, hot-standby HA, main.unkin.net, 1200/86400 leases, AU ntp pool); five KeaSubnets 198.18.13-17.0/24 with .200-.220 pools, gateways .254 except .17->.1, next-server 198.18.19.19; Legacy/UEFI-64 PXE client classes; KeaAPI.
- DHCP-advertised DNS points at the in-cluster bind-resolvers cluster (PureLB 198.18.200.7), not the legacy 198.18.19.15 forwarder.
- Pin the DHCP LoadBalancer Service to the free common-pool IP 198.18.200.10 via PureLB.
- KeaAPI bearer token is operator-generated (no plain Secret committed).
- Commit generated kea.unkin.net JSON schemas for kubeconform; register dhcp-system in the platform ApplicationSet + AppProject.
Client-class object names are lowercased (`legacy`/`uefi-64`) to satisfy RFC1123 since the operator renders the kea class name from metadata.name.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #333
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
We self-delegate `_acme-challenge.unkin.net` into an `acme.unkin.net` zone we serve ourselves, so cert-manager can solve Let's Encrypt DNS-01 over RFC2136/TSIG. That needs a publicly-reachable authoritative BIND, separate from the internal estate.
- Add app `bind-external` (base + au-syd1 overlay); register it in the platform ApplicationSet and AppProject destinations (bind-operator already watches all namespaces).
- Add BindCluster `bind-external`: authoritative-only, recursion off, no forwarding, transfers denied except the keyed catalog/zone AXFR; 2 replicas; primaryService is a dmz-pinned PureLB LoadBalancer at `198.18.199.53`.
- Add BindZone `acme.unkin.net` (primary, dynamicUpdate) and BindTSIGKey `certmanager` (hmac-sha256), whose Secret `certmanager-tsig` reflects into the `cert-manager` namespace for the rfc2136 solver.
Pairs with argocd-apps #327 (the ClusterIssuers) and a one-time Google Cloud DNS delegation + NAT of the public IP :53 to `198.18.199.53`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #329
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Publicly-trusted wildcard certs via Let's Encrypt DNS-01, solved over RFC2136/TSIG against our own BIND. A one-time CNAME self-delegates `_acme-challenge.unkin.net` into the `acme.unkin.net` zone served by bind-external; cert-manager writes the challenge TXT there. No GCP/clouddns and no Vault secret involved. The existing `vault-issuer` (internal PKI) is untouched.
- Add ClusterIssuers `letsencrypt` (prod) and `letsencrypt-staging`, both using a dns01 rfc2136 solver: nameserver `198.18.199.53:53`, key `certmanager`, HMACSHA256, `tsigSecretSecretRef` -> reflected Secret `certmanager-tsig` key `secret`.
- Whitelist `cert-manager.io ClusterIssuer` in the platform AppProject.
Depends on #329 (bind-external: the acme.unkin.net zone, the certmanager TSIG key reflected into cert-manager, and the 198.18.199.53 nameserver) and on the one-time Google Cloud DNS delegation + NAT of the public IP :53 to 198.18.199.53. Earlier clouddns/Vault commits on this branch are reverted.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #327
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md.
- add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute
- add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222
- register gitea in the platform ApplicationSet + AppProject
- add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback)
Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc.
Reviewed-on: #309
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists.
## Topology
`edge (publishers) → JetStream → consumers → sinks`
- **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets.
- **Edge publishers (thin)** — no parsing, just a routing subject:
- `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`.
- `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.)
- **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now).
- **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`.
- **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job.
## Streams / consumers / auth
- Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment.
- **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret.
## S3 / retention
`ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here.
## Replay runbook
- **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time.
- **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window.
## Validation
- `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**.
- `pre-commit` (yamllint, check-json, no-plain-secrets) clean.
- **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs.
- **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets).
## Known upstream caveat
Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook.
## Prerequisites (manual, one-time)
```
# ClickHouse
PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1)
vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \
username=vector password="$PW" password_sha256_hex="$HASH"
# NATS
vault kv put kv/kubernetes/namespace/logging/default/nats-auth \
admin_password=$(openssl rand -base64 24) \
producer_password=$(openssl rand -base64 24) \
consumer_password=$(openssl rand -base64 24)
```
No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR.
## Open decisions (defaults chosen, flag to change)
- **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.**
- **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side).
- **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer.
- **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA.
- **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs).
- **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---
## Update: images via artifactapi, DHI, stateless transform tier
**Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull.
### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`)
| Image | Upstream | artifactapi path | DHI? |
|---|---|---|---|
| clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job |
| altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI |
| altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI |
| bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI |
| nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats |
| natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI |
| natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI |
| timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened |
| timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step |
**DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace.
### Transform tier is now a stateless Deployment
Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**.
**Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.)
---
## Update: 7d retention, tunable limits ConfigMap, honest sizing
- **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict.
- **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3.
- **Honest 7d sizing (stated assumption — please sanity-check against real volume):**
- Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica.
- Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica.
- **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.**
- ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob.
- Replay window in the runbook is now **7d** (beyond that → the S3 archive).
---
## Update: retention cut to 3 days (both stores), PVCs shrunk
Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design.
| Store | Retention | Byte cap | PVC/node | Replicas | Total |
|---|---|---|---|---|---|
| NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) |
| ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) |
**NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC.
**ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table.
The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL.
⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit).
Reviewed-on: #296
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
NetBox becomes the source of truth for host/interface/IPAM data as part of retiring Cobbler. The ENC role already moved to encapi; a kickstart-generation microservice that consumes NetBox comes later and is intentionally **out of scope** here.
## Change
Adds the `netbox` app (namespace `netbox`, platform project) using the netbox-community helm chart via the OCI helm-through-kustomize pattern, plus estate-native data stores:
- **NetBox** — chart `8.3.40` (appVersion `v4.6.5`), `oci://ghcr.io/netbox-community/netbox-chart`. 2 web replicas, 1 worker; bundled Postgres/Valkey subcharts disabled. Media on RWX CephFS so replicas share uploads. Chart `helm.sh/hook: test` Pod dropped via a kustomize delete patch (we deploy through ArgoCD, not `helm test`).
- **CNPG Postgres** — `netbox-postgres`, 2 instances, PG 18, `cephrbd-fast-delete`. Standard per-cluster S3 backup: `barmanObjectStore -> s3://cnpg-netbox`, cephrgw `ObjectStoreUser` + `Bucket`, nightly `ScheduledBackup` at **03:40** (`0 40 3 * * *` — next free slot after grafana's 03:20), 30d retention. A pgbouncer `Pooler` (session mode) fronts it; NetBox connects via `netbox-postgres-pooler-rw`.
- **Valkey** — standalone Deployment (`valkey/valkey:8-alpine`), AOF-persistent PVC on `cephrbd-fast-delete`. One instance: DB 0 = RQ task queue, DB 1 = cache. No auth (in-cluster, namespace-isolated). Chosen over the bundled Bitnami subchart to keep image control in-estate and avoid Bitnami's legacy-image churn; mirrors the litellm standalone-cache pattern.
- **Ingress** — `Gateway` + `HTTPRoute` at `netbox.k8s.syd1.au.unkin.net` (`traefik-internal`, `vault-issuer` cert into `netbox-tls`, external-dns to the internal VIP), HTTP->HTTPS 301.
- **Secrets** — all via VSO `VaultStaticSecret` (`postgres-credentials`, `netbox-secret-key`, `netbox-superuser`, `oauth-credentials`); no plain Secrets committed. The shared `default` k8s-auth role already binds `*` namespaces with a namespace-templated KV policy, so **no terraform-vault change is needed**.
- **Authentik OIDC SSO** — `remoteAuth` wires `REMOTE_AUTH_ENABLED` + the `OpenIdConnectAuth` backend via chart values; `SOCIAL_AUTH_OIDC_*` via `extraConfig` (the chart's config loader globs `/run/config/extra/*/*.yaml`). The client secret is injected as a YAML fragment mounted from the Vault-synced `oauth-credentials` secret. New users auto-provision on first login.
- Registers `netbox` in the platform ApplicationSet and AppProject (destination namespace + chart sourceRepo).
## Image table (source -> artifactapi -> DHI decision)
All images flow through the estate's containerd registry mirrors; the allowlist patterns gate them. DHI (Docker Hardened Images) require authenticated pulls from a Docker Hub `dhi/` org and are **not** reachable through the anonymous mirror, so upstream official is used throughout.
| Image | Upstream | Mirror / allowlist | Allowlisted? | DHI decision |
|---|---|---|---|---|
| NetBox app/worker/housekeeping | `ghcr.io/netbox-community/netbox:v4.6.5` | ghcr remote, `^netbox-community/` | **added in terraform-artifactapi #17** | No DHI published; upstream official |
| CNPG Postgres | `ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie` | ghcr, `^cloudnative-pg/` | already | estate-standard CNPG image |
| Valkey | `docker.io/valkey/valkey:8-alpine` | dockerhub, `^valkey/valkey` | already | DHI not anon-pullable; upstream official |
| NetBox init (perms) | `docker.io/busybox:1.38.0` | dockerhub, `^library/busybox` | already | upstream official |
| Worker wait-for-backend | `docker.io/rancher/kubectl:v1.36.2` | dockerhub, `^rancher/` | already | upstream official |
## Cross-repo PRs (merge order)
1. **terraform-artifactapi #17** (`^netbox-community/` ghcr allowlist) — **merge before** this PR so the NetBox image pulls on first sync.
2. **terraform-authentik #11** (OIDC provider/application) — independent; SSO works once applied + the secret is seeded.
3. This PR.
> Note: the CNPG S3-backup stanza pattern here is identical to PR #298 (already in main); it merges cleanly regardless of ordering.
## One-time Vault seeds (before/at first sync)
```
# App DB user (CNPG bootstrap + NetBox both consume this)
vault kv put kv/kubernetes/namespace/netbox/default/postgres-credentials \
username=netbox password="$(openssl rand -base64 30)"
# Django SECRET_KEY
vault kv put kv/kubernetes/namespace/netbox/default/netbox-secret-key \
secret_key="$(python3 -c 'import secrets;print(secrets.token_urlsafe(60))')"
# Bootstrap superuser
vault kv put kv/kubernetes/namespace/netbox/default/netbox-superuser \
username=admin email=admin@unkin.net \
password="$(openssl rand -base64 24)" api_token="$(openssl rand -hex 20)"
# OIDC client secret — ONE value stored two ways (raw for Authentik, YAML for NetBox)
CS="$(openssl rand -base64 30)"
vault kv put kv/kubernetes/namespace/netbox/default/oauth-credentials \
client_secret="$CS" oidc.yaml="SOCIAL_AUTH_OIDC_SECRET: \"$CS\""
```
Seed `oauth-credentials` **before** applying terraform-authentik #11 (that apply reads `client_secret`).
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/netbox` — clean.
- `kubeconform` (CI args, k8s 1.33.7): **27/27 valid, 0 invalid**.
- `pre-commit` (yamllint + no-plain-secrets) on all changed files — pass.
## Out of scope
The PXE/kickstart microservice that will consume NetBox — not scaffolded here.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #299
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move
between them as they migrate. `node-lookup` (and `pblastreport`) need a single,
consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy
that provides exactly that. Per the all-in-kubernetes estate direction it runs
as an in-cluster service, not a per-VM systemd unit.
pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) —
split out of the earlier node-lookup prototype — and is released as a container
image on its own `v*` tags.
## Changes
- Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway,
httproute), modeled directly on the encapi app.
- Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080,
`/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap.
- Backends: `old=http://puppetdbapi.service.consul:8080`,
`new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified
against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the
in-cluster address is preferred over the external gateway). `new` is
primary/prefer, merge = freshness.
- Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal`
Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a
port-80 Service — same shape as the puppetdb/encapi gateways — so
VM/workstation `node-lookup` can reach it.
- Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform
ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject
(`pdbmux` namespace destination), exactly as encapi is wired.
No new woodpecker ServiceAccount is required: the pdbmux image push uses the
`docker-buildx` plugin against the Gitea registry with the `default` SA (same as
encapi), not artifactapi.
## Verification
- `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves
to `git.unkin.net/unkin/pdbmux:v0.1.0`).
- ApplicationSet + AppProject YAML validated.
## Merge gates
1. The pdbmux repo initial-content PR
(unkin/pdbmux#1) must merge first.
2. `v0.1.0` must then be tagged on the pdbmux repo so the image
`git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s
`.woodpecker/docker.yaml`.
3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update
the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.)
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #275
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Deploys Grafana in-cluster (observability project) via the grafana-operator, mirroring the puppet Grafana but modernised — **CNPG** for state, **Authentik OIDC** for auth — and ports the live datasource + dashboards in as CRs.
Depends on: grafana-operator (#235, merged), grafana schemas (#236, merged), Authentik OIDC (terraform-authentik #2), Vault seeds (done), and `^grafana/` image proxy (terraform-artifactapi #5).
## Changes (`apps/base/grafana`)
- **CNPG** postgres Cluster + rw Pooler (db `grafana`); **VaultAuth** + **VaultStaticSecrets** pulling `postgres`/`oauth` credentials from `kv/kubernetes/namespace/grafana/default/*`.
- **Grafana CR**: postgres backend via the pooler; Authentik `generic_oauth` (client id/secret from the Vault-synced secret, openid/email/profile scopes, group→role mapping); `root_url` grafana.k8s.syd1.au.unkin.net.
- **1 GrafanaDatasource** — k8s VictoriaMetrics via the operator `vmselect-main` service; reuses the previous default datasource uid so the imported dashboards resolve unedited.
- **13 GrafanaDashboards** (gzipJson) exported from the current grafana.
- **Gateway API** (traefik-internal) + HTTPRoute for grafana.k8s.syd1.au.unkin.net.
- Registered in the observability ApplicationSet + project.
## Review notes
- OAuth `role_attribute_path` maps Authentik group `grafana-admins` → Admin, else Viewer — **confirm the group name**.
- `database.ssl_mode: require` against the CNPG pooler — adjust if the pooler isn't serving TLS.
- The `VictoriaLogs - cluster` dashboard has no in-cluster logs datasource yet (no VictoriaLogs in k8s) — included for completeness, will be empty until one exists.
- `make kubeconform` clean (24 resources, validated against the strict grafana schemas).
Reviewed-on: #238
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**HOLD until v0.1.3 is tagged/built** (operator #4 merged + tagged) — this PR bumps the operator to v0.1.3, whose CRD adds the `clusterRef` field these keys use.
## Why
Put all BIND DNS services in one `bind-internal` namespace and name the StatefulSets clearly.
## Changes
- 3 clusters consolidated into `bind-internal`, StatefulSets renamed **bind-authoritative** / **bind-resolvers** / **bind-externaldns**; LBs kept on 198.18.200.6/.7/.8; external-dns hostnames renamed to match
- `clusterRef` added to `transfer-key` (→ bind-authoritative) and `externaldns-key` (→ bind-externaldns) so keys are scoped per cluster
- removed the old `ns-auth`/`ns-resolver`/`ns-externaldns` apps; ApplicationSet + AppProject now list `bind-internal`
- bumped `bind-system` operator to **v0.1.3** (CRD link + image)
- operator stays in `bind-system`
## Deploy impact
ArgoCD prunes the old ns-* namespaces (StatefulSets/PVCs — data is only seed SOA+NS, no migrated records yet) and creates the renamed clusters in bind-internal.
## Validated
`kustomize build` → 28 docs (3 BindCluster, 20 BindZone, 2 catalog, 2 keys, ns); kubeconform clean.
Reviewed-on: #225
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Renames the three BIND DNS app namespaces `binddns-{auth,resolver,externaldns}` -> `ns-{auth,resolver,externaldns}`.
## Why
Shorter, clearer namespace names for the DNS tiers.
## Changes
- `argocd/applicationsets/platform.yaml`: overlay path registrations renamed (the ApplicationSet derives each app's namespace from its overlay dir name)
- `argocd/projects/platform.yaml`: destination namespaces renamed
## Coupled with
The per-tier PRs (#220/#221/#222) rename the overlay dirs + namespaces + external-dns hostnames to match. No app deploys to a renamed namespace until both this and the tier PR are merged (harmless before then — the ApplicationSet only instantiates apps for existing dirs).
Reviewed-on: #223
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
First of a 4-PR split of the bind rollout (was #216). Deploys just the operator control plane so it can be verified before any DNS clusters exist.
## Why
Roll out incrementally: operator + CRDs first, then each BIND tier as its own PR.
## Changes
- `apps/base/bind-system`: operator Deployment (`git.unkin.net/unkin/bind-operator:v0.1.1`), RBAC, namespace; CRDs pulled from the operator repo by raw URL (`config/crd/install.yaml` @ v0.1.1)
- au-syd1 `bind-system` overlay
- register all four bind apps in `argocd/applicationsets/platform.yaml` (DNS overlays instantiate only when their dirs land in the follow-up PRs)
- add `binddns-*` namespaces to `argocd/projects/platform.yaml`
- add `schemas/bind.unkin.net/*.json` for kubeconform
## Deploy impact
Operator pod + CRDs only. No DNS services yet — the operator is idle until BindClusters exist.
## Follow-ups (merge after this)
binddns-auth, binddns-resolver, binddns-externaldns — one PR each.
Reviewed-on: #219
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- Deploy age-api to the au-syd1 cluster
- Uses configMapGenerator for people config with jaidi, ben, and sudaporn
- Includes gateway, httproute, service, and deployment
- Image: git.unkin.net/unkin/age-api:v0.1.0
Reviewed-on: #210
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- New `apps/base/priority-classes/` app with four `PriorityClass` objects managed via the `platform` ArgoCD project
- Adds `apps/overlays/*/priority-classes` to the platform ApplicationSet generator
- Adds `priority-classes` namespace to platform AppProject destinations (required even for cluster-scoped resources)
| Class | Value | PreemptionPolicy | Intent |
|---|---|---|---|
| `low` | 100 | Never | Background work; evictable, won't preempt others |
| `power` | 100 | Never | Compute-heavy but expendable (e.g. AI/ML workloads) |
| `medium` | 10000 | PreemptLowerPriority | Standard services |
| `high` | 100000 | PreemptLowerPriority | Critical services; preempts lower-priority pods |
`PriorityClass` is already in the platform project's `clusterResourceWhitelist` so no project policy changes were needed.
## Test plan
- ArgoCD syncs `platform-priority-classes` successfully
- `kubectl get priorityclasses low power medium high` shows all four classes
Reviewed-on: #174
Vault and consul namespaces were missing from the platform AppProject
allowed destinations, causing ArgoCD sync failures with:
destination server 'https://kubernetes.default.svc' and namespace
'vault' do not match any of the allowed destinations in project 'platform'
Reviewed-on: #152
Deploys LiteLLM proxy with CNPG PostgreSQL (3-instance HA), PgBouncer
pooler, and Redis cache. Introduces a dedicated aitooling AppProject and
ApplicationSet to keep AI tooling services separate from platform infra.
Reviewed-on: #94
Migrate PureLB load balancer from Terragrunt to ArgoCD/Kustomize.
Deploys purelb v0.13.0 with two LBNodeAgent and two ServiceGroup CRs
(common: 198.18.200.0/24, dmz: 198.18.199.0/24).
Adds LBNodeAgent and ServiceGroup to kubeconform skip list (no CRD catalog schema).
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #84
Migrate Vault Secrets Operator from Terragrunt to ArgoCD/Kustomize.
Deploys vault-secrets-operator v1.2.0 with 3 replicas, plus ClusterRole,
ClusterRoleBindings, and vault-admin ServiceAccount.
Note: static service account tokens (kubernetes.io/service-account-token)
cannot be stored in git; create manually or via Vault after deployment.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #81
Migrate Victoria Metrics cluster and agent from Terragrunt to ArgoCD/Kustomize.
Creates new observability AppProject and ApplicationSet.
Deploys victoria-metrics-cluster v0.33.0 (vmselect/vminsert/vmstorage with
HPA, PDB, ingress) and victoria-metrics-agent v0.30.0 (3 replicas, k8s scrape
configs) in the observability namespace.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #82
Migrate Victoria Metrics operator from Terragrunt to ArgoCD/Kustomize.
Deploys victoria-metrics-operator v0.57.1 with 2 replicas in vm-system.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #80
Migrate ECK operator from Terragrunt to ArgoCD/Kustomize.
Deploys eck-operator v3.2.0 with 2 replicas and PodDisruptionBudget
in the elastic-system namespace.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #79
Migrate repository sync cronjobs from Terragrunt to ArgoCD/Kustomize.
Adds four daily CronJobs (almalinux9-baseos, almalinux9-appstream, epel9,
openvox7) with associated PVCs and ConfigMaps in the reposync namespace.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #78
- Add node-feature-discovery and inteldeviceplugins-system to platform project
- Convert intel-nfd-rules from local Helm chart to static NodeFeatureRule manifests
- Add required Helm repositories (NFD OCI registry and Intel charts)
- Create base configurations with Helm charts and overlay structures
- Update platform ApplicationSet and project permissions
Reviewed-on: #48
- Migrate csi-cephfs from Terraform to ArgoCD
- Migrate csi-cephrbd from Terraform to ArgoCD
- Create dedicated storage project and ApplicationSet for CSI drivers
- Add csi-* pattern matching in storage ApplicationSet
- Remove CSI apps from platform project to separate concerns
Reviewed-on: #45
- Add cnpg-system base ArgoCD application with namespace
- Create cnpg-system overlay for au-syd1 with CloudNativePG Helm chart
- Update platform ApplicationSet to include cnpg-system deployment
- Configure cloudnative-pg operator v0.27.0 with HA and resource limits
- Maintain one-to-one migration from Terraform configuration
Reviewed-on: #44
- Add externaldns base ArgoCD application with namespace and Vault integration
- Create externaldns overlay for au-syd1 with Helm chart configuration
- Update platform ApplicationSet to include externaldns deployment
- Configure external-dns v1.19.0 with RFC2136 provider for DNS updates
- Maintain one-to-one migration from Terraform configuration including TSIG secrets
Reviewed-on: #43
- Add cattle-system base ArgoCD application with namespace, Vault integration, and ingress
- Create cattle-system overlay for au-syd1 with Rancher Helm chart configuration
- Update platform ApplicationSet to include cattle-system deployment
- Update platform project to include Rancher Helm repository as source
- Configure Rancher v2.13.1 with HA, TLS, audit logging, and bootstrap secret from Vault
- Maintain one-to-one migration from Terraform configuration
Reviewed-on: #39
working towards a larger, redundant, autoscaling and simple puppet
implementation in kubernetes. this was originally based on the openvox
helm chart with several improvements (not all in this pr)
- use of cnpg instead of single bitnamilegacy postgres container
- use for g10k instead of r10k
- run one instance of g10k per namespace, instead of per-pod
- store only keep one copy of the environments/branches (instead of per-pod)
- change g10k to native cronjob instead of hacky implementation
- use vault secrets
part one adds:
- cnpg puppetdb pgsql cluster
- cnpg puppetdb pgpooler
- persistent volume claims for puppet, puppetdb, the code repository, etc
Reviewed-on: #25
Ben Vincent
unkinben
benvin