Three changes from review:
1. Pull every container image through the artifactapi dockerhub remote instead
of direct upstream: clickhouse-server, altinity operator + metrics-exporter,
bitnami/kubectl (crdHook), nats + nats-server-config-reloader, nats-box
(bootstrap Job), and vector (all tiers + the CI image). Requires
terraform-artifactapi#16 (dockerhub allowlist patterns) merged first.
2. Keep upstream official images (no Docker Hardened Images). DHI exists for
clickhouse-server and vector but is subscription-gated and served from a
private org namespace not reachable via the anonymous artifactapi dockerhub
proxy; its shell-less images would also break the bash bootstrap Jobs and the
shell-based vector-test CI step. Use vector's distroless-libc for runtime
pods (near-hardened) and the debian variant only for CI.
3. Make the transform tier a stateless Deployment (was a StatefulSet): no PVC,
no disk buffer — JetStream is the sole durability layer. The ClickHouse sink
uses an in-memory block buffer so a ClickHouse outage back-pressures the
JetStream pull source (unpulled messages are retained/redelivered). Add a CPU
HPA (2-8) — safe because JetStream pull consumers distribute work across N
replicas on the one durable consumer. Caveat documented: vector's NATS source
has no end-to-end acks (acks on receipt), so a pod killed mid-outage can lose
its in-memory buffer window; accepted trade for a stateless autoscaling tier.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Stand up a centralized logging estate that captures ALL logs from k8s pods and
(via a reachable ingestion endpoint) puppet-managed VMs, storing them in
ClickHouse for query/retention. Metrics already live in VictoriaMetrics; this
adds the logs pillar under a dedicated `logging` ArgoCD project.
Deploy the Altinity clickhouse-operator (clickhouse-system) and a single-shard
ClickHouseInstallation (logging) on cephrbd-fast-delete with a MergeTree
logs.raw table (30d TTL) bootstrapped by an idempotent PostSync Job.
Deploy Vector as an explicit two tiers:
- Edge (thin): a DaemonSet tails every node's pod logs and forwards over the
Vector-native protocol to the aggregator; no parsing at the edge. Future VM
agents follow the same thin pattern.
- Aggregator (brain): HA StatefulSet that is the sole ClickHouse writer, holds
the only ClickHouse credentials, owns all transforms, batches into few fat
inserts (avoid too-many-parts), and buffers to disk (PVC) to ride out a
ClickHouse outage. Its pipeline is a single source-of-truth config validated
by `vector test` in CI; per-app pipelines become aggregator-only changes.
Expose the VM ingestion endpoint at logs-ingest.k8s.syd1.au.unkin.net via the
internal Traefik gateway (cert-manager + external-dns), routing to the
aggregator's HTTP source so puppet VMs can reach it over TLS.
Source ClickHouse credentials from Vault via the existing VaultStaticSecret
pattern (templated k8s auth policy already grants the logging namespace);
password hash never lands in git.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben