Ben: 1.2 TiB is too much. Cut both stores to 3d and resize honestly.
- NATS LOGS stream max_age 168h -> 72h in the nats-stream-limits ConfigMap.
At the stated ~33 GiB/day compressed (S2), 3d ~= 100 GiB, so max_bytes ->
130 GiB and the file-store PVC -> 180Gi/node. 3 replicas = ~0.5 TiB total
(down from 1.2 TiB).
- ClickHouse logs.raw TTL 30d -> 3d in the bootstrap DDL. At ~130 GiB/day raw,
LZ4/ZSTD (~6x) stores ~20-25 GiB/day => ~60-75 GiB/3d; with merge headroom the
CHI PVC -> 150Gi (from 200Gi). logs.raw is the only table.
- Long-term retention now lives EXCLUSIVELY in the S3 archive (logarchiver) for
the configured subjects; everything else is gone after 3 days — accepted
design, documented in the runbook + PR body.
- Update all 7d/30d comments and the replay window (now 3d).
PVC shrink is a plan-time change: this stack is not deployed yet, so it is
clean. Once deployed, PVCs cannot shrink in place (would need recreate).
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
- Retention -> 7 days (max_age=168h), still retention=limits/discard=old so the
transform tier and the archiver each independently see every message; reading
never deletes.
- Put the tunable stream limits (max_age, max_bytes, dupe_window) in the
nats-stream-limits ConfigMap. The bootstrap Job reads them and does an
idempotent create-or-UPDATE (nats stream add || nats stream edit), so changing
the ConfigMap + re-sync applies new limits with no manual surgery. The
ConfigMap keeps its kustomize content-hash suffix, so an edit renames it and
rewrites the Job's env refs -> the PostSync hook Job's spec changes and Argo
re-runs it (on top of hooks running each sync). Verified end-to-end against a
real nats-server: create, idempotent re-run, and a max_age change all apply.
- Honest 7d sizing: assume ~1500 events/s avg @ ~1 KiB/event; with S2 stream
compression (~4x) that's ~33 GiB/day -> ~230 GiB/7d per replica. Enable S2
compression on the stream, set max_bytes=300 GiB, and raise the file-store PVC
to 400Gi/node (3 replicas = 1.2 TiB) so the byte cap can't silently truncate
retention below 7d. Numbers + assumptions flagged in the PR body.
- Update runbook/comments: replay window is now 7d.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Three changes from review:
1. Pull every container image through the artifactapi dockerhub remote instead
of direct upstream: clickhouse-server, altinity operator + metrics-exporter,
bitnami/kubectl (crdHook), nats + nats-server-config-reloader, nats-box
(bootstrap Job), and vector (all tiers + the CI image). Requires
terraform-artifactapi#16 (dockerhub allowlist patterns) merged first.
2. Keep upstream official images (no Docker Hardened Images). DHI exists for
clickhouse-server and vector but is subscription-gated and served from a
private org namespace not reachable via the anonymous artifactapi dockerhub
proxy; its shell-less images would also break the bash bootstrap Jobs and the
shell-based vector-test CI step. Use vector's distroless-libc for runtime
pods (near-hardened) and the debian variant only for CI.
3. Make the transform tier a stateless Deployment (was a StatefulSet): no PVC,
no disk buffer — JetStream is the sole durability layer. The ClickHouse sink
uses an in-memory block buffer so a ClickHouse outage back-pressures the
JetStream pull source (unpulled messages are retained/redelivered). Add a CPU
HPA (2-8) — safe because JetStream pull consumers distribute work across N
replicas on the one durable consumer. Caveat documented: vector's NATS source
has no end-to-end acks (acks on receipt), so a pod killed mid-outage can lose
its in-memory buffer window; accepted trade for a stateless autoscaling tier.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Rework the logging pipeline around a durable message bus so logs survive a
ClickHouse outage, can be replayed after a bad transform, and fan out to
multiple independent consumers. Add long-term raw-log backup to S3.
Topology becomes edge -> JetStream -> consumers -> sinks:
- Dedicated JetStream NATS cluster (3 replicas, file storage) in the logging
namespace. Deliberately separate from app messaging (streamstack) for
blast-radius isolation. Stream LOGS (subjects logs.>, retention=limits, 40GiB
/ 72h) is the outage buffer; durable consumers give independent offsets.
- Edge publishers (thin): the k8s DaemonSet and a new VM-ingest Deployment
(HTTP NDJSON front door behind the logs-ingest Gateway) publish into JetStream
(logs.k8s.<ns>.<container> / logs.vm.<host>). No parsing on the edge.
- Transform tier (StatefulSet): pulls the whole stream via the durable
`transform` consumer, routes by subject, shapes, and remains the sole
ClickHouse writer. Its disk buffer shrinks (JetStream is the outage buffer).
- Archiver (Deployment): its OWN durable `archiver` consumer (independent
offsets — archive lag never affects the ClickHouse path) writes RAW,
pre-transform events to a Ceph RGW S3 bucket (cephrgw-operator ObjectStoreUser
+ Bucket + BucketAccess) as gzipped NDJSON keyed by raw/<subject>/YYYY/MM/DD/.
Default subject filter is Vault audit (logs.k8s.vault.>), configurable.
Auth: distinct NATS users (producer publish-only, consumer pull+ack, admin for
the stream/consumer bootstrap Job) with passwords from Vault (nats-auth Secret);
S3 creds from the BucketAccess Secret. Streams/consumers are provisioned by an
idempotent PostSync bootstrap Job.
Add local kubeconform schemas for the ceph.unkin.net CRDs (datreeio lacks them)
and extend the vector-test CI to cover the agent, VM-ingest and archiver
configs. Verified end-to-end locally: NATS ACLs, vector JetStream publish, and
durable-consumer pull+ack (at-least-once) all work.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Stand up a centralized logging estate that captures ALL logs from k8s pods and
(via a reachable ingestion endpoint) puppet-managed VMs, storing them in
ClickHouse for query/retention. Metrics already live in VictoriaMetrics; this
adds the logs pillar under a dedicated `logging` ArgoCD project.
Deploy the Altinity clickhouse-operator (clickhouse-system) and a single-shard
ClickHouseInstallation (logging) on cephrbd-fast-delete with a MergeTree
logs.raw table (30d TTL) bootstrapped by an idempotent PostSync Job.
Deploy Vector as an explicit two tiers:
- Edge (thin): a DaemonSet tails every node's pod logs and forwards over the
Vector-native protocol to the aggregator; no parsing at the edge. Future VM
agents follow the same thin pattern.
- Aggregator (brain): HA StatefulSet that is the sole ClickHouse writer, holds
the only ClickHouse credentials, owns all transforms, batches into few fat
inserts (avoid too-many-parts), and buffers to disk (PVC) to ride out a
ClickHouse outage. Its pipeline is a single source-of-truth config validated
by `vector test` in CI; per-app pipelines become aggregator-only changes.
Expose the VM ingestion endpoint at logs-ingest.k8s.syd1.au.unkin.net via the
internal Traefik gateway (cert-manager + external-dns), routing to the
aggregator's HTTP source so puppet VMs can reach it over TLS.
Source ClickHouse credentials from Vault via the existing VaultStaticSecret
pattern (templated k8s auth policy already grants the logging namespace);
password hash never lands in git.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben