Rework the logging pipeline around a durable message bus so logs survive a
ClickHouse outage, can be replayed after a bad transform, and fan out to
multiple independent consumers. Add long-term raw-log backup to S3.
Topology becomes edge -> JetStream -> consumers -> sinks:
- Dedicated JetStream NATS cluster (3 replicas, file storage) in the logging
namespace. Deliberately separate from app messaging (streamstack) for
blast-radius isolation. Stream LOGS (subjects logs.>, retention=limits, 40GiB
/ 72h) is the outage buffer; durable consumers give independent offsets.
- Edge publishers (thin): the k8s DaemonSet and a new VM-ingest Deployment
(HTTP NDJSON front door behind the logs-ingest Gateway) publish into JetStream
(logs.k8s.<ns>.<container> / logs.vm.<host>). No parsing on the edge.
- Transform tier (StatefulSet): pulls the whole stream via the durable
`transform` consumer, routes by subject, shapes, and remains the sole
ClickHouse writer. Its disk buffer shrinks (JetStream is the outage buffer).
- Archiver (Deployment): its OWN durable `archiver` consumer (independent
offsets — archive lag never affects the ClickHouse path) writes RAW,
pre-transform events to a Ceph RGW S3 bucket (cephrgw-operator ObjectStoreUser
+ Bucket + BucketAccess) as gzipped NDJSON keyed by raw/<subject>/YYYY/MM/DD/.
Default subject filter is Vault audit (logs.k8s.vault.>), configurable.
Auth: distinct NATS users (producer publish-only, consumer pull+ack, admin for
the stream/consumer bootstrap Job) with passwords from Vault (nats-auth Secret);
S3 creds from the BucketAccess Secret. Streams/consumers are provisioned by an
idempotent PostSync bootstrap Job.
Add local kubeconform schemas for the ceph.unkin.net CRDs (datreeio lacks them)
and extend the vector-test CI to cover the agent, VM-ingest and archiver
configs. Verified end-to-end locally: NATS ACLs, vector JetStream publish, and
durable-consumer pull+ack (at-least-once) all work.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Stand up a centralized logging estate that captures ALL logs from k8s pods and
(via a reachable ingestion endpoint) puppet-managed VMs, storing them in
ClickHouse for query/retention. Metrics already live in VictoriaMetrics; this
adds the logs pillar under a dedicated `logging` ArgoCD project.
Deploy the Altinity clickhouse-operator (clickhouse-system) and a single-shard
ClickHouseInstallation (logging) on cephrbd-fast-delete with a MergeTree
logs.raw table (30d TTL) bootstrapped by an idempotent PostSync Job.
Deploy Vector as an explicit two tiers:
- Edge (thin): a DaemonSet tails every node's pod logs and forwards over the
Vector-native protocol to the aggregator; no parsing at the edge. Future VM
agents follow the same thin pattern.
- Aggregator (brain): HA StatefulSet that is the sole ClickHouse writer, holds
the only ClickHouse credentials, owns all transforms, batches into few fat
inserts (avoid too-many-parts), and buffers to disk (PVC) to ride out a
ClickHouse outage. Its pipeline is a single source-of-truth config validated
by `vector test` in CI; per-app pipelines become aggregator-only changes.
Expose the VM ingestion endpoint at logs-ingest.k8s.syd1.au.unkin.net via the
internal Traefik gateway (cert-manager + external-dns), routing to the
aggregator's HTTP source so puppet VMs can reach it over TLS.
Source ClickHouse credentials from Vault via the existing VaultStaticSecret
pattern (templated k8s auth policy already grants the logging namespace);
password hash never lands in git.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben