unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add ClickHouse + Vector + NATS JetStream centralized logging (with S3 raw archive) #296

Merged
benvin merged 6 commits from benvin/clickhouse-vector-logging into main 2026-07-28 19:54:27 +10:00
Conversation 0 Commits 6 Files Changed 34
+2032
unkinben commented 2026-07-27 19:49:59 +10:00
Owner
Copy Link
Copy Source

Why

Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture all logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a durable NATS JetStream bus in the middle so logs survive a ClickHouse outage, can be replayed after a bad transform, and fan out to independent consumers. A third consumer archives selected raw logs to S3 (Ceph RGW) for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists.

Topology

edge (publishers) → JetStream → consumers → sinks

  • NATS JetStream (dedicated, logging ns): 3-replica cluster, file storage on cephrbd-fast-delete (50Gi/node). Deliberately separate from app messaging (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream LOGS (subjects logs.>, retention=limits, S2-compressed, 3d / 130 GiB). Durable consumers = independent offsets.
  • Edge publishers (thin) — no parsing, just a routing subject:
    • vector-agent (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream logs.k8s.<ns>.<container>.
    • vector-vm-ingest (Deployment): HTTPS/NDJSON front door behind the logs-ingest Gateway → JetStream logs.vm.<host>. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.)
  • Transform tier vector-aggregator (StatefulSet): pulls the whole stream via durable consumer transform, routes by subject, normalises into logs.raw, and is the sole ClickHouse writer. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now).
  • Archiver vector-archiver (Deployment): its own durable consumer archiver (independent offsets — archive lag can never stall ClickHouse) writes raw, pre-transform events to a Ceph RGW bucket as gzipped NDJSON, keyed raw/<subject>/YYYY/MM/DD/. Default subject filter logs.k8s.vault.> (Vault audit) — configurable via the bootstrap Job's ARCHIVE_SUBJECTS.
  • ClickHouse: Altinity operator + single-shard ClickHouseInstallation (200Gi RBD), logs.raw MergeTree, 30d TTL, idempotent PostSync schema Job.

Streams / consumers / auth

  • Stream + both durable consumers provisioned by an idempotent PostSync bootstrap Job (nats CLI). Runbook lines for both replay directions are in the Job's header comment.
  • Distinct NATS users: log-producer (publish logs.> only), log-consumer (pull + ack only), log-admin (bootstrap). Passwords from Vault (nats-auth Secret, env-var expansion in the server config). S3 creds from the cephrgw-operator BucketAccess Secret.

S3 / retention

ObjectStoreUser + Bucket (logs-archive, retainOnDelete) + BucketAccess (read-write) via the in-estate cephrgw-operator. aws_s3 sink → https://s3.ceph.unkin.net (path-style, trusts the reflected vault-ca-cert). Object retention is an RGW-side bucket lifecycle policy (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here.

Replay runbook

  • Within 3d (JetStream): scale the transform tier to 0, nats consumer rm LOGS transform, re-run the bootstrap Job (recreates at DeliverAll) — or nats consumer edit/--replay from a seq/time.
  • Long-horizon (S3): re-ingest archived objects through the transform tier (vector aws_s3 source or a one-shot Job); the archive is the replay source beyond JetStream's window.

Validation

  • kustomize build --enable-helm clean; kubeconform (k8s 1.33.7) all valid — clickhouse-system 22, logging 38 (incl. ClickHouseInstallation via datreeio and the ceph.unkin.net CRDs via local schemas added under schemas/), apps/base 10.
  • pre-commit (yamllint, check-json, no-plain-secrets) clean.
  • vector test passes the transform-tier + VM-ingest unit tests; vector validate passes the agent + archiver configs.
  • End-to-end integration test (local docker): ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and acked (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets).

Known upstream caveat

Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook.

Prerequisites (manual, one-time)

# ClickHouse
PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1)
vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \
  username=vector password="$PW" password_sha256_hex="$HASH"
# NATS
vault kv put kv/kubernetes/namespace/logging/default/nats-auth \
  admin_password=$(openssl rand -base64 24) \
  producer_password=$(openssl rand -base64 24) \
  consumer_password=$(openssl rand -base64 24)

No terraform-vault change needed (templated default k8s auth policy already grants the logging namespace KV path). The vault-ca-cert Secret is reflected into logging by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR.

Open decisions (defaults chosen, flag to change)

  • Archive subject filter: default logs.k8s.vault.> (Vault audit). Candidates to add: logs.k8s.authentik.>, logs.k8s.kanidm.>, VM auth roles — please confirm the exact security set.
  • Retention: ClickHouse 3d TTL; JetStream 3d (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side).
  • Sizing: NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer.
  • HA: ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA.
  • VM front door: HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs).
  • CI image: timberio/vector:0.57.0-debian + natsio/nats-box:0.18.0 (Docker Hub) — mirror if runners restrict egress.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

Update: images via artifactapi, DHI, stateless transform tier

Depends on unkin/terraform-artifactapi#16 (dockerhub allowlist patterns) — merge that first or images won't pull.

Image table (all pulled through artifactapi.k8s.syd1.au.unkin.net/dockerhub/…)

Image Upstream artifactapi path DHI?
clickhouse/clickhouse-server:24.8 Docker Hub dockerhub/clickhouse/clickhouse-server DHI exists — not used: subscription/private-namespace + shell-less breaks the bash schema Job
altinity/clickhouse-operator:0.27.2 Docker Hub dockerhub/altinity/clickhouse-operator No DHI
altinity/metrics-exporter:0.27.2 Docker Hub dockerhub/altinity/metrics-exporter No DHI
bitnami/kubectl:latest (crdHook) Docker Hub dockerhub/bitnami/kubectl No DHI
nats:2.14.2-alpine Docker Hub dockerhub/library/nats No DHI for nats
natsio/nats-server-config-reloader:0.23.0 Docker Hub dockerhub/natsio/nats-server-config-reloader No DHI
natsio/nats-box:0.18.0 (bootstrap Job) Docker Hub dockerhub/natsio/nats-box No DHI
timberio/vector:0.57.0-distroless-libc (runtime) Docker Hub dockerhub/timberio/vector DHI exists — not used (subscription/private-namespace); distroless-libc is already near-hardened
timberio/vector:0.57.0-debian (CI only) Docker Hub dockerhub/timberio/vector shell needed for the CI step

DHI decision: Docker Hardened Images exist for clickhouse-server and vector, but they're subscription-gated and served from a private Docker org namespace (authenticated pull) — not reachable via the estate's anonymous artifactapi dockerhub proxy, and no DHI org/remote exists here. Their shell-less nature would also break the bash heredoc in the ClickHouse schema Job and the shell-based vector-test CI step. So: upstream official through artifactapi, using vector distroless-libc for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace.

Transform tier is now a stateless Deployment

Was a StatefulSet with a disk buffer/PVC; now a Deployment with no PVC and an in-memory buffer — JetStream is the sole durability layer. Added a CPU HPA (min 2 / max 8).

Ack / backpressure design (important caveat): Vector's NATS source has acknowledgements: no — it acks the JetStream message on receipt, not after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses buffer.when_full=block, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and unpulled messages stay in JetStream and are redelivered. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed mid-outage. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer transform. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.)

Update: 7d retention, tunable limits ConfigMap, honest sizing

  • Retention → 7 days (max_age=168h), still retention=limits / discard=old: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict.

  • Stream limits live in a ConfigMap (nats-stream-limits: max_age, max_bytes, dupe_window). The nats-bootstrap PostSync Job reads them and does an idempotent create-or-UPDATE (nats stream add || nats stream edit). How a change propagates: the ConfigMap keeps its kustomize content-hash suffix, so editing a value renames the ConfigMap and rewrites the Job's configMapKeyRefs → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with hook-delete-policy=BeforeHookCreation) → nats stream edit applies the new limits. No manual nats surgery. Verified against a real nats-server: create (7d), idempotent re-run, and a max_age change (168h→24h) all applied; all flags incl. --compression=s2 accepted by nats CLI v0.2.3.

  • Honest 7d sizing (stated assumption — please sanity-check against real volume):

    • Assume ~1,500 events/s average @ ~1 KiB/event stored JSON ⇒ ~130 GiB/day raw, ~910 GiB/7d raw per replica.
    • Enable JetStream S2 compression (logs ~4× conservative) ⇒ ~33 GiB/day, ~230 GiB/7d compressed per replica.
    • max_bytes = 300 GiB (headroom over the 230 GiB estimate). PVC = 400Gi/node on cephrbd-fast-delete (max_bytes + file-store WAL/index/overhead, safely under). 3 replicas ⇒ 1.2 TiB provisioned.
    • ⚠️ This is a large, prominent number by design. If real volume exceeds the assumption, discard=old truncates retention below 7d rather than silently overflowing. Raising retention/volume requires bumping both max_bytes (ConfigMap) and the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob.

  • Replay window in the runbook is now 7d (beyond that → the S3 archive).

Update: retention cut to 3 days (both stores), PVCs shrunk

Ben: 1.2 TiB is too much. Both stores now retain 3 days; long-term retention lives exclusively in the encrypted S3 archive (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design.

Store Retention Byte cap PVC/node Replicas Total
NATS JetStream LOGS max_age=72h (3d) max_bytes=130 GiB 180Gi 3 ~0.5 TiB (was 1.2 TiB)
ClickHouse logs.raw TTL 3 DAY — 150Gi 1 150Gi (was 200Gi)

NATS math: ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → max_bytes 130 GiB (headroom) under a 180Gi PVC.
ClickHouse math: ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. logs.raw is the only table.

The retention knobs remain in the nats-stream-limits ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL.

⚠️ PVC-shrink caveat: this is a plan-time change — the stack is not deployed yet, so shrinking PVCs is clean. If it were already deployed, PVCs cannot shrink in place (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit).

## Why Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists. ## Topology `edge (publishers) → JetStream → consumers → sinks` - **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets. - **Edge publishers (thin)** — no parsing, just a routing subject: - `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`. - `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.) - **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now). - **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`. - **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job. ## Streams / consumers / auth - Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment. - **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret. ## S3 / retention `ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here. ## Replay runbook - **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time. - **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window. ## Validation - `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**. - `pre-commit` (yamllint, check-json, no-plain-secrets) clean. - **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs. - **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets). ## Known upstream caveat Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook. ## Prerequisites (manual, one-time) ``` # ClickHouse PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1) vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \ username=vector password="$PW" password_sha256_hex="$HASH" # NATS vault kv put kv/kubernetes/namespace/logging/default/nats-auth \ admin_password=$(openssl rand -base64 24) \ producer_password=$(openssl rand -base64 24) \ consumer_password=$(openssl rand -base64 24) ``` No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR. ## Open decisions (defaults chosen, flag to change) - **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.** - **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side). - **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer. - **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA. - **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs). - **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --- ## Update: images via artifactapi, DHI, stateless transform tier **Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull. ### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`) | Image | Upstream | artifactapi path | DHI? | |---|---|---|---| | clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job | | altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI | | altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI | | bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI | | nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats | | natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI | | natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI | | timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened | | timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step | **DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace. ### Transform tier is now a stateless Deployment Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**. **Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.) --- ## Update: 7d retention, tunable limits ConfigMap, honest sizing - **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict. - **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3. - **Honest 7d sizing (stated assumption — please sanity-check against real volume):** - Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica. - Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica. - **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.** - ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob. - Replay window in the runbook is now **7d** (beyond that → the S3 archive). --- ## Update: retention cut to 3 days (both stores), PVCs shrunk Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design. | Store | Retention | Byte cap | PVC/node | Replicas | Total | |---|---|---|---|---|---| | NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) | | ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) | **NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC. **ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table. The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL. ⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit).
unkinben added 1 commit 2026-07-27 19:49:59 +10:00
Add ClickHouse + Vector centralized logging stack
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
10020033d9 
Stand up a centralized logging estate that captures ALL logs from k8s pods and
(via a reachable ingestion endpoint) puppet-managed VMs, storing them in
ClickHouse for query/retention. Metrics already live in VictoriaMetrics; this
adds the logs pillar under a dedicated `logging` ArgoCD project.

Deploy the Altinity clickhouse-operator (clickhouse-system) and a single-shard
ClickHouseInstallation (logging) on cephrbd-fast-delete with a MergeTree
logs.raw table (30d TTL) bootstrapped by an idempotent PostSync Job.

Deploy Vector as an explicit two tiers:
- Edge (thin): a DaemonSet tails every node's pod logs and forwards over the
  Vector-native protocol to the aggregator; no parsing at the edge. Future VM
  agents follow the same thin pattern.
- Aggregator (brain): HA StatefulSet that is the sole ClickHouse writer, holds
  the only ClickHouse credentials, owns all transforms, batches into few fat
  inserts (avoid too-many-parts), and buffers to disk (PVC) to ride out a
  ClickHouse outage. Its pipeline is a single source-of-truth config validated
  by `vector test` in CI; per-app pipelines become aggregator-only changes.

Expose the VM ingestion endpoint at logs-ingest.k8s.syd1.au.unkin.net via the
internal Traefik gateway (cert-manager + external-dns), routing to the
aggregator's HTTP source so puppet VMs can reach it over TLS.

Source ClickHouse credentials from Vault via the existing VaultStaticSecret
pattern (templated k8s auth policy already grants the logging namespace);
password hash never lands in git.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-27 20:24:00 +10:00
Insert NATS JetStream log bus + S3 raw archive
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
c39af2f9c3 
Rework the logging pipeline around a durable message bus so logs survive a
ClickHouse outage, can be replayed after a bad transform, and fan out to
multiple independent consumers. Add long-term raw-log backup to S3.

Topology becomes edge -> JetStream -> consumers -> sinks:
- Dedicated JetStream NATS cluster (3 replicas, file storage) in the logging
  namespace. Deliberately separate from app messaging (streamstack) for
  blast-radius isolation. Stream LOGS (subjects logs.>, retention=limits, 40GiB
  / 72h) is the outage buffer; durable consumers give independent offsets.
- Edge publishers (thin): the k8s DaemonSet and a new VM-ingest Deployment
  (HTTP NDJSON front door behind the logs-ingest Gateway) publish into JetStream
  (logs.k8s.<ns>.<container> / logs.vm.<host>). No parsing on the edge.
- Transform tier (StatefulSet): pulls the whole stream via the durable
  `transform` consumer, routes by subject, shapes, and remains the sole
  ClickHouse writer. Its disk buffer shrinks (JetStream is the outage buffer).
- Archiver (Deployment): its OWN durable `archiver` consumer (independent
  offsets — archive lag never affects the ClickHouse path) writes RAW,
  pre-transform events to a Ceph RGW S3 bucket (cephrgw-operator ObjectStoreUser
  + Bucket + BucketAccess) as gzipped NDJSON keyed by raw/<subject>/YYYY/MM/DD/.
  Default subject filter is Vault audit (logs.k8s.vault.>), configurable.

Auth: distinct NATS users (producer publish-only, consumer pull+ack, admin for
the stream/consumer bootstrap Job) with passwords from Vault (nats-auth Secret);
S3 creds from the BucketAccess Secret. Streams/consumers are provisioned by an
idempotent PostSync bootstrap Job.

Add local kubeconform schemas for the ceph.unkin.net CRDs (datreeio lacks them)
and extend the vector-test CI to cover the agent, VM-ingest and archiver
configs. Verified end-to-end locally: NATS ACLs, vector JetStream publish, and
durable-consumer pull+ack (at-least-once) all work.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben changed title from Add ClickHouse + Vector centralized logging stack to Add ClickHouse + Vector + NATS JetStream centralized logging (with S3 raw archive) 2026-07-27 20:24:43 +10:00
unkinben added 1 commit 2026-07-27 21:19:47 +10:00
Pull images via artifactapi; make transform tier stateless
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/vector-test Pipeline failed
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
1202aae06f 
Three changes from review:

1. Pull every container image through the artifactapi dockerhub remote instead
   of direct upstream: clickhouse-server, altinity operator + metrics-exporter,
   bitnami/kubectl (crdHook), nats + nats-server-config-reloader, nats-box
   (bootstrap Job), and vector (all tiers + the CI image). Requires
   terraform-artifactapi#16 (dockerhub allowlist patterns) merged first.

2. Keep upstream official images (no Docker Hardened Images). DHI exists for
   clickhouse-server and vector but is subscription-gated and served from a
   private org namespace not reachable via the anonymous artifactapi dockerhub
   proxy; its shell-less images would also break the bash bootstrap Jobs and the
   shell-based vector-test CI step. Use vector's distroless-libc for runtime
   pods (near-hardened) and the debian variant only for CI.

3. Make the transform tier a stateless Deployment (was a StatefulSet): no PVC,
   no disk buffer — JetStream is the sole durability layer. The ClickHouse sink
   uses an in-memory block buffer so a ClickHouse outage back-pressures the
   JetStream pull source (unpulled messages are retained/redelivered). Add a CPU
   HPA (2-8) — safe because JetStream pull consumers distribute work across N
   replicas on the one durable consumer. Caveat documented: vector's NATS source
   has no end-to-end acks (acks on receipt), so a pod killed mid-outage can lose
   its in-memory buffer window; accepted trade for a stateless autoscaling tier.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-27 21:40:14 +10:00
Move JetStream limits to a ConfigMap; 7d retention, honest sizing
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/vector-test Pipeline failed
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
b18e9a669f 
- Retention -> 7 days (max_age=168h), still retention=limits/discard=old so the
  transform tier and the archiver each independently see every message; reading
  never deletes.
- Put the tunable stream limits (max_age, max_bytes, dupe_window) in the
  nats-stream-limits ConfigMap. The bootstrap Job reads them and does an
  idempotent create-or-UPDATE (nats stream add || nats stream edit), so changing
  the ConfigMap + re-sync applies new limits with no manual surgery. The
  ConfigMap keeps its kustomize content-hash suffix, so an edit renames it and
  rewrites the Job's env refs -> the PostSync hook Job's spec changes and Argo
  re-runs it (on top of hooks running each sync). Verified end-to-end against a
  real nats-server: create, idempotent re-run, and a max_age change all apply.
- Honest 7d sizing: assume ~1500 events/s avg @ ~1 KiB/event; with S2 stream
  compression (~4x) that's ~33 GiB/day -> ~230 GiB/7d per replica. Enable S2
  compression on the stream, set max_bytes=300 GiB, and raise the file-store PVC
  to 400Gi/node (3 replicas = 1.2 TiB) so the byte cap can't silently truncate
  retention below 7d. Numbers + assumptions flagged in the PR body.
- Update runbook/comments: replay window is now 7d.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-27 22:13:29 +10:00
ci: re-run vector-test after artifactapi allowlist applied
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
c60a639563 
terraform-artifactapi #16 is merged and the dockerhub allowlist now serves
timberio/vector (verified 200); this empty commit re-triggers CI.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-28 18:16:43 +10:00
Reduce retention to 3 days (NATS + ClickHouse); shrink PVCs
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
5ae14b3a38 
Ben: 1.2 TiB is too much. Cut both stores to 3d and resize honestly.

- NATS LOGS stream max_age 168h -> 72h in the nats-stream-limits ConfigMap.
  At the stated ~33 GiB/day compressed (S2), 3d ~= 100 GiB, so max_bytes ->
  130 GiB and the file-store PVC -> 180Gi/node. 3 replicas = ~0.5 TiB total
  (down from 1.2 TiB).
- ClickHouse logs.raw TTL 30d -> 3d in the bootstrap DDL. At ~130 GiB/day raw,
  LZ4/ZSTD (~6x) stores ~20-25 GiB/day => ~60-75 GiB/3d; with merge headroom the
  CHI PVC -> 150Gi (from 200Gi). logs.raw is the only table.
- Long-term retention now lives EXCLUSIVELY in the S3 archive (logarchiver) for
  the configured subjects; everything else is gone after 3 days — accepted
  design, documented in the runbook + PR body.
- Update all 7d/30d comments and the replay window (now 3d).

PVC shrink is a plan-time change: this stack is not deployed yet, so it is
clean. Once deployed, PVCs cannot shrink in place (would need recreate).

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
benvin merged commit f45cb6989f into main 2026-07-28 19:54:27 +10:00
benvin deleted branch benvin/clickhouse-vector-logging 2026-07-28 19:54:28 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#296
Delete Branch "benvin/clickhouse-vector-logging"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?