b18e9a669f49226a9c6c7e6d3438decda710b550
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 unkinben
|
b18e9a669f |
Move JetStream limits to a ConfigMap; 7d retention, honest sizing
- Retention -> 7 days (max_age=168h), still retention=limits/discard=old so the transform tier and the archiver each independently see every message; reading never deletes. - Put the tunable stream limits (max_age, max_bytes, dupe_window) in the nats-stream-limits ConfigMap. The bootstrap Job reads them and does an idempotent create-or-UPDATE (nats stream add || nats stream edit), so changing the ConfigMap + re-sync applies new limits with no manual surgery. The ConfigMap keeps its kustomize content-hash suffix, so an edit renames it and rewrites the Job's env refs -> the PostSync hook Job's spec changes and Argo re-runs it (on top of hooks running each sync). Verified end-to-end against a real nats-server: create, idempotent re-run, and a max_age change all apply. - Honest 7d sizing: assume ~1500 events/s avg @ ~1 KiB/event; with S2 stream compression (~4x) that's ~33 GiB/day -> ~230 GiB/7d per replica. Enable S2 compression on the stream, set max_bytes=300 GiB, and raise the file-store PVC to 400Gi/node (3 replicas = 1.2 TiB) so the byte cap can't silently truncate retention below 7d. Numbers + assumptions flagged in the PR body. - Update runbook/comments: replay window is now 7d. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv |
||
|
unkinben
|
1202aae06f |
Pull images via artifactapi; make transform tier stateless
Three changes from review: 1. Pull every container image through the artifactapi dockerhub remote instead of direct upstream: clickhouse-server, altinity operator + metrics-exporter, bitnami/kubectl (crdHook), nats + nats-server-config-reloader, nats-box (bootstrap Job), and vector (all tiers + the CI image). Requires terraform-artifactapi#16 (dockerhub allowlist patterns) merged first. 2. Keep upstream official images (no Docker Hardened Images). DHI exists for clickhouse-server and vector but is subscription-gated and served from a private org namespace not reachable via the anonymous artifactapi dockerhub proxy; its shell-less images would also break the bash bootstrap Jobs and the shell-based vector-test CI step. Use vector's distroless-libc for runtime pods (near-hardened) and the debian variant only for CI. 3. Make the transform tier a stateless Deployment (was a StatefulSet): no PVC, no disk buffer — JetStream is the sole durability layer. The ClickHouse sink uses an in-memory block buffer so a ClickHouse outage back-pressures the JetStream pull source (unpulled messages are retained/redelivered). Add a CPU HPA (2-8) — safe because JetStream pull consumers distribute work across N replicas on the one durable consumer. Caveat documented: vector's NATS source has no end-to-end acks (acks on receipt), so a pod killed mid-outage can lose its in-memory buffer window; accepted trade for a stateless autoscaling tier. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv |
||
|
unkinben
|
c39af2f9c3 |
Insert NATS JetStream log bus + S3 raw archive
Rework the logging pipeline around a durable message bus so logs survive a ClickHouse outage, can be replayed after a bad transform, and fan out to multiple independent consumers. Add long-term raw-log backup to S3. Topology becomes edge -> JetStream -> consumers -> sinks: - Dedicated JetStream NATS cluster (3 replicas, file storage) in the logging namespace. Deliberately separate from app messaging (streamstack) for blast-radius isolation. Stream LOGS (subjects logs.>, retention=limits, 40GiB / 72h) is the outage buffer; durable consumers give independent offsets. - Edge publishers (thin): the k8s DaemonSet and a new VM-ingest Deployment (HTTP NDJSON front door behind the logs-ingest Gateway) publish into JetStream (logs.k8s.<ns>.<container> / logs.vm.<host>). No parsing on the edge. - Transform tier (StatefulSet): pulls the whole stream via the durable `transform` consumer, routes by subject, shapes, and remains the sole ClickHouse writer. Its disk buffer shrinks (JetStream is the outage buffer). - Archiver (Deployment): its OWN durable `archiver` consumer (independent offsets — archive lag never affects the ClickHouse path) writes RAW, pre-transform events to a Ceph RGW S3 bucket (cephrgw-operator ObjectStoreUser + Bucket + BucketAccess) as gzipped NDJSON keyed by raw/<subject>/YYYY/MM/DD/. Default subject filter is Vault audit (logs.k8s.vault.>), configurable. Auth: distinct NATS users (producer publish-only, consumer pull+ack, admin for the stream/consumer bootstrap Job) with passwords from Vault (nats-auth Secret); S3 creds from the BucketAccess Secret. Streams/consumers are provisioned by an idempotent PostSync bootstrap Job. Add local kubeconform schemas for the ceph.unkin.net CRDs (datreeio lacks them) and extend the vector-test CI to cover the agent, VM-ingest and archiver configs. Verified end-to-end locally: NATS ACLs, vector JetStream publish, and durable-consumer pull+ack (at-least-once) all work. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv |
||
|
unkinben
|
10020033d9 |
Add ClickHouse + Vector centralized logging stack
Stand up a centralized logging estate that captures ALL logs from k8s pods and (via a reachable ingestion endpoint) puppet-managed VMs, storing them in ClickHouse for query/retention. Metrics already live in VictoriaMetrics; this adds the logs pillar under a dedicated `logging` ArgoCD project. Deploy the Altinity clickhouse-operator (clickhouse-system) and a single-shard ClickHouseInstallation (logging) on cephrbd-fast-delete with a MergeTree logs.raw table (30d TTL) bootstrapped by an idempotent PostSync Job. Deploy Vector as an explicit two tiers: - Edge (thin): a DaemonSet tails every node's pod logs and forwards over the Vector-native protocol to the aggregator; no parsing at the edge. Future VM agents follow the same thin pattern. - Aggregator (brain): HA StatefulSet that is the sole ClickHouse writer, holds the only ClickHouse credentials, owns all transforms, batches into few fat inserts (avoid too-many-parts), and buffers to disk (PVC) to ride out a ClickHouse outage. Its pipeline is a single source-of-truth config validated by `vector test` in CI; per-app pipelines become aggregator-only changes. Expose the VM ingestion endpoint at logs-ingest.k8s.syd1.au.unkin.net via the internal Traefik gateway (cert-manager + external-dns), routing to the aggregator's HTTP source so puppet VMs can reach it over TLS. Source ClickHouse credentials from Vault via the existing VaultStaticSecret pattern (templated k8s auth policy already grants the logging namespace); password hash never lands in git. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv |