6c8c0dd9e4b8a469f24cb14b078cf7b2d21fe60f
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 unkinben
|
6c8c0dd9e4 |
Deploy jellyfin-ha as a true-HA StatefulSet under a new media project (#237)
## Why Deploy the jellyfin-ha fork as a genuine high-availability service rather than a single replica, so its two headline capabilities can actually be exercised: the Redis-coordinated distributed transcoding (a surviving pod resumes the HLS segments of a pod that dies mid-stream) and the experimental PostgreSQL main database (which is what lets more than one replica share the same library). It lands in its own `jellyfin` namespace under a new `media` ArgoCD project. ## How **Workload — StatefulSet, 2 replicas.** The Deployment becomes a StatefulSet so each pod has a stable name. The fork's Redis transcode lease keys ownership on `JELLYFIN_INSTANCE_ID`, which is set from the downward-API pod name, giving each replica a unique, stable lease identity. Soft `podAntiAffinity` spreads the two pods across nodes and a `PodDisruptionBudget` keeps `minAvailable: 1` through drains and rollouts. **Main database — CloudNativePG.** A CNPG trio in-namespace mirrors the litellm pattern: a 3-instance `Cluster`, a PgBouncer `Pooler`, and Ceph RGW (barman) S3 backups to a dedicated `cnpg-jellyfin` bucket owned by a `cephrgw` `ObjectStoreUser`. An init container writes `/config/config/database.xml` selecting the fork's `Jellyfin-PostgreSQL` provider, and the connection string is composed from the CNPG-generated `jellyfin-postgres-app` secret (username / password / dbname) pointed at the pooler service — the password is never rendered into the manifest. Library-item metadata therefore moves off SQLite; metadata images, plugins, subtitles and config XML stay on `/config`. **Storage.** - `/config` is now a shared `ReadWriteMany` cephfs PVC (raid5, retain) so every replica reads/writes the same metadata and config. - `/config/transcodes` — the fork's real transcode temp path — is a shared RWX PVC (raid5, delete). This is the load-bearing fix: takeover reads the dead pod's in-flight `.ts`/`.m3u8` segments off shared storage, so per-pod scratch would silently break it. - `/cache` is per-pod via a `volumeClaimTemplate` (RWO). - The media library stays a fresh, empty RWX PVC mounted read-only; populating it is out of scope. **Hardware transcoding.** The container requests the `gpu.intel.com/i915` Intel device-plugin resource (which pins the pod to a GPU-labelled node and injects the DRI render node — no `/dev/dri` hostPath or privileged container) plus the render/video supplemental groups. VA-API hardware acceleration is now on by default: the `inject-config` init container seeds `/config/config/encoding.xml` with `HardwareAccelerationType` `vaapi`, `EnableHardwareEncoding`, the injected render node (`/dev/dri/renderD128`) and h264/hevc hardware decode, so transcodes use the iGPU on first boot with no manual admin-UI step. Both seed files (`database.xml`, `encoding.xml`) are written only when absent, so later admin changes persisted to the shared RWX `/config` are never clobbered on restart. **Networking.** The Gateway/HTTPRoute (traefik-internal, `jellyfin.k8s.syd1.au.unkin.net`) is unchanged; the Service gains `sessionAffinity: ClientIP` to keep a client pinned to one replica and reduce transcode-session churn. **Redis.** The in-namespace single-replica Redis stays as the transcode lease store. ## Follow-up UDP auto-discovery is disabled, but scheduled library scans still run on every replica (redundant scans). Single-scanner leader election is a planned follow-up pending a fork feature, tracked separately. --------- Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-authored-by: Ben Vincent <neotheo@gmail.com> Co-authored-by: Ben Vin <neotheo@gmail.com> Reviewed-on: #237 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
f45cb6989f |
Add ClickHouse + Vector + NATS JetStream centralized logging (with S3 raw archive) (#296)
## Why Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists. ## Topology `edge (publishers) → JetStream → consumers → sinks` - **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets. - **Edge publishers (thin)** — no parsing, just a routing subject: - `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`. - `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.) - **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now). - **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`. - **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job. ## Streams / consumers / auth - Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment. - **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret. ## S3 / retention `ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here. ## Replay runbook - **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time. - **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window. ## Validation - `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**. - `pre-commit` (yamllint, check-json, no-plain-secrets) clean. - **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs. - **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets). ## Known upstream caveat Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook. ## Prerequisites (manual, one-time) ``` # ClickHouse PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1) vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \ username=vector password="$PW" password_sha256_hex="$HASH" # NATS vault kv put kv/kubernetes/namespace/logging/default/nats-auth \ admin_password=$(openssl rand -base64 24) \ producer_password=$(openssl rand -base64 24) \ consumer_password=$(openssl rand -base64 24) ``` No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR. ## Open decisions (defaults chosen, flag to change) - **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.** - **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side). - **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer. - **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA. - **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs). - **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --- ## Update: images via artifactapi, DHI, stateless transform tier **Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull. ### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`) | Image | Upstream | artifactapi path | DHI? | |---|---|---|---| | clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job | | altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI | | altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI | | bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI | | nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats | | natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI | | natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI | | timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened | | timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step | **DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace. ### Transform tier is now a stateless Deployment Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**. **Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.) --- ## Update: 7d retention, tunable limits ConfigMap, honest sizing - **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict. - **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3. - **Honest 7d sizing (stated assumption — please sanity-check against real volume):** - Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica. - Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica. - **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.** - ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob. - Replay window in the runbook is now **7d** (beyond that → the S3 archive). --- ## Update: retention cut to 3 days (both stores), PVCs shrunk Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design. | Store | Retention | Byte cap | PVC/node | Replicas | Total | |---|---|---|---|---|---| | NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) | | ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) | **NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC. **ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table. The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL. ⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit). Reviewed-on: #296 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkinben
|
5372914803 |
feat: add litellm to new aitooling ArgoCD project (#94)
Deploys LiteLLM proxy with CNPG PostgreSQL (3-instance HA), PgBouncer pooler, and Redis cache. Introduces a dedicated aitooling AppProject and ApplicationSet to keep AI tooling services separate from platform infra. Reviewed-on: #94 |
||
|
unkinben
|
b100f3034e |
feat: migrate observability to ArgoCD (#82)
Migrate Victoria Metrics cluster and agent from Terragrunt to ArgoCD/Kustomize. Creates new observability AppProject and ApplicationSet. Deploys victoria-metrics-cluster v0.33.0 (vmselect/vminsert/vmstorage with HPA, PDB, ingress) and victoria-metrics-agent v0.30.0 (3 replicas, k8s scrape configs) in the observability namespace. 💘 Generated with Crush Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land> Reviewed-on: #82 |
||
|
unkinben
|
c157774033 |
fix: enable ServerSideApply for ArgoCD ApplicationSets (#46)
- resolve CRD annotation size limit errors by enabling server-side apply - add storage ApplicationSet and project to kustomization files Reviewed-on: #46 |
||
|
unkinben
|
0daa026f01 |
feat: add pre-commit workflow (#10)
- enforce pre-commit is run for all pull-requests Reviewed-on: #10 |
||
|
unkinben
|
971835f845 |
feat: initial commit
- add structure to clusters, apps and argocd objects - add bootstrapping features |