unkinben
f45cb6989f
## Why Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists. ## Topology `edge (publishers) → JetStream → consumers → sinks` - **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets. - **Edge publishers (thin)** — no parsing, just a routing subject: - `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`. - `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.) - **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now). - **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`. - **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job. ## Streams / consumers / auth - Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment. - **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret. ## S3 / retention `ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here. ## Replay runbook - **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time. - **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window. ## Validation - `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**. - `pre-commit` (yamllint, check-json, no-plain-secrets) clean. - **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs. - **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets). ## Known upstream caveat Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook. ## Prerequisites (manual, one-time) ``` # ClickHouse PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1) vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \ username=vector password="$PW" password_sha256_hex="$HASH" # NATS vault kv put kv/kubernetes/namespace/logging/default/nats-auth \ admin_password=$(openssl rand -base64 24) \ producer_password=$(openssl rand -base64 24) \ consumer_password=$(openssl rand -base64 24) ``` No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR. ## Open decisions (defaults chosen, flag to change) - **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.** - **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side). - **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer. - **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA. - **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs). - **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --- ## Update: images via artifactapi, DHI, stateless transform tier **Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull. ### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`) | Image | Upstream | artifactapi path | DHI? | |---|---|---|---| | clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job | | altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI | | altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI | | bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI | | nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats | | natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI | | natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI | | timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened | | timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step | **DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace. ### Transform tier is now a stateless Deployment Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**. **Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.) --- ## Update: 7d retention, tunable limits ConfigMap, honest sizing - **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict. - **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3. - **Honest 7d sizing (stated assumption — please sanity-check against real volume):** - Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica. - Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica. - **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.** - ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob. - Replay window in the runbook is now **7d** (beyond that → the S3 archive). --- ## Update: retention cut to 3 days (both stores), PVCs shrunk Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design. | Store | Retention | Byte cap | PVC/node | Replicas | Total | |---|---|---|---|---|---| | NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) | | ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) | **NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC. **ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table. The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL. ⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit). Reviewed-on: #296 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
52 lines
1.7 KiB
YAML
52 lines
1.7 KiB
YAML
---
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
resources:
|
|
- namespace.yaml
|
|
- vaultauth.yaml
|
|
- vaultstaticsecret.yaml
|
|
- clickhouseinstallation.yaml
|
|
- job_clickhouse-schema.yaml
|
|
- nats-bootstrap-job.yaml
|
|
- cephrgw.yaml
|
|
- gateway.yaml
|
|
- httproute.yaml
|
|
|
|
# Vector pipelines are the single source of truth (also validated by
|
|
# `vector test` in CI). Mounted into each tier via `existingConfigMaps`.
|
|
configMapGenerator:
|
|
# Tunable JetStream stream limits (the nats-bootstrap Job reads these and does
|
|
# create-or-update). Hash suffix is INTENTIONALLY left on: editing a value
|
|
# renames the ConfigMap, which rewrites the Job's env reference, which changes
|
|
# the PostSync hook Job's spec and forces Argo to re-run it -> new limits apply.
|
|
# Sizing assumes ~1500 events/s avg @ ~1 KiB/event with S2 compression (~4x):
|
|
# ~33 GiB/day compressed -> ~100 GiB/3d per replica. max_bytes 130 GiB sits
|
|
# under the 180Gi/node PVC (see values-nats.yaml). Raising retention beyond the
|
|
# PVC requires bumping BOTH max_bytes here and fileStore PVC size in values.
|
|
- name: nats-stream-limits
|
|
literals:
|
|
- max_age=72h
|
|
- max_bytes=139586437120
|
|
- dupe_window=2m
|
|
- name: vector-agent-config
|
|
files:
|
|
- agent.yaml=vector/agent.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|
|
- name: vector-aggregator-config
|
|
files:
|
|
- aggregator.yaml=vector/aggregator.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|
|
- name: vector-vm-ingest-config
|
|
files:
|
|
- vm-ingest.yaml=vector/vm-ingest.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|
|
- name: vector-archiver-config
|
|
files:
|
|
- archiver.yaml=vector/archiver.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|