Add ClickHouse + Vector + NATS JetStream centralized logging (with S3 raw archive) (#296)

## Why Metrics already land in VictoriaMetrics, but there is no centralized log store. This stands up the logs pillar: capture **all** logs from (a) k8s pods and (b) puppet-managed VMs into ClickHouse, with a **durable NATS JetStream bus** in the middle so logs survive a ClickHouse outage, can be **replayed** after a bad transform, and **fan out** to independent consumers. A third consumer archives selected raw logs to **S3 (Ceph RGW)** for long-horizon replay beyond the JetStream window. The puppet-side Vector rollout is a later task — this PR makes sure a reachable VM ingestion endpoint exists. ## Topology `edge (publishers) → JetStream → consumers → sinks` - **NATS JetStream** (dedicated, `logging` ns): 3-replica cluster, file storage on `cephrbd-fast-delete` (50Gi/node). Deliberately **separate from app messaging** (streamstack runs its own NATS in its own repo) for blast-radius isolation. Stream `LOGS` (subjects `logs.>`, `retention=limits`, S2-compressed, **3d / 130 GiB**). Durable consumers = independent offsets. - **Edge publishers (thin)** — no parsing, just a routing subject: - `vector-agent` (DaemonSet): tails every node's pod logs (incl. control-plane) → JetStream `logs.k8s.<ns>.<container>`. - `vector-vm-ingest` (Deployment): HTTPS/NDJSON front door behind the `logs-ingest` Gateway → JetStream `logs.vm.<host>`. (Chosen over exposing NATS TCP to ~143 VMs: keeps VM shipping to a simple TLS POST while still gaining JetStream durability; direct-NATS-for-VMs noted as an alternative.) - **Transform tier** `vector-aggregator` (StatefulSet): pulls the whole stream via durable consumer `transform`, routes by subject, normalises into `logs.raw`, and is the **sole ClickHouse writer**. Disk buffer shrunk to 2GiB/5Gi PVC (JetStream is the real outage buffer now). - **Archiver** `vector-archiver` (Deployment): its **own** durable consumer `archiver` (independent offsets — archive lag can never stall ClickHouse) writes **raw, pre-transform** events to a Ceph RGW bucket as gzipped NDJSON, keyed `raw/<subject>/YYYY/MM/DD/`. Default subject filter **`logs.k8s.vault.>`** (Vault audit) — configurable via the bootstrap Job's `ARCHIVE_SUBJECTS`. - **ClickHouse**: Altinity operator + single-shard `ClickHouseInstallation` (200Gi RBD), `logs.raw` MergeTree, 30d TTL, idempotent PostSync schema Job. ## Streams / consumers / auth - Stream + both durable consumers provisioned by an **idempotent PostSync bootstrap Job** (`nats` CLI). Runbook lines for both replay directions are in the Job's header comment. - **Distinct NATS users**: `log-producer` (publish `logs.>` only), `log-consumer` (pull + ack only), `log-admin` (bootstrap). Passwords from Vault (`nats-auth` Secret, env-var expansion in the server config). S3 creds from the `cephrgw-operator` `BucketAccess` Secret. ## S3 / retention `ObjectStoreUser` + `Bucket` (`logs-archive`, retainOnDelete) + `BucketAccess` (read-write) via the in-estate cephrgw-operator. aws_s3 sink → `https://s3.ceph.unkin.net` (path-style, trusts the reflected `vault-ca-cert`). **Object retention is an RGW-side bucket lifecycle policy** (the operator doesn't manage lifecycle) — flagged as an operational knob, not invented here. ## Replay runbook - **Within 3d (JetStream):** scale the transform tier to 0, `nats consumer rm LOGS transform`, re-run the bootstrap Job (recreates at DeliverAll) — or `nats consumer edit`/`--replay` from a seq/time. - **Long-horizon (S3):** re-ingest archived objects through the transform tier (vector `aws_s3` source or a one-shot Job); the archive is the replay source beyond JetStream's window. ## Validation - `kustomize build --enable-helm` clean; `kubeconform` (k8s 1.33.7) all valid — clickhouse-system **22**, logging **38** (incl. `ClickHouseInstallation` via datreeio and the `ceph.unkin.net` CRDs via **local schemas added under `schemas/`**), apps/base **10**. - `pre-commit` (yamllint, check-json, no-plain-secrets) clean. - **`vector test`** passes the transform-tier + VM-ingest unit tests; `vector validate` passes the agent + archiver configs. - **End-to-end integration test (local docker):** ran nats-server (JetStream) with the exact auth block, created the stream + durable consumer, published via Vector (producer ACL), and consumed via Vector's JetStream durable consumer (consumer ACL) — all 3 events pulled, routed, shaped, and **acked** (Outstanding Acks: 0). Confirms the NATS ACLs, Vector JetStream publish, and durable-consumer pull+ack (at-least-once + durable offsets). ## Known upstream caveat Vector's NATS JetStream source has an open reliability issue (vectordotdev/vector#24932: consumer can stall after a NATS "lame duck"/reconnect). Recovery is a pod restart of the affected consumer; noted for the runbook. ## Prerequisites (manual, one-time) ``` # ClickHouse PW=$(openssl rand -base64 24); HASH=$(printf '%s' "$PW" | sha256sum | cut -d' ' -f1) vault kv put kv/kubernetes/namespace/logging/default/clickhouse-credentials \ username=vector password="$PW" password_sha256_hex="$HASH" # NATS vault kv put kv/kubernetes/namespace/logging/default/nats-auth \ admin_password=$(openssl rand -base64 24) \ producer_password=$(openssl rand -base64 24) \ consumer_password=$(openssl rand -base64 24) ``` No terraform-vault change needed (templated `default` k8s auth policy already grants the `logging` namespace KV path). The `vault-ca-cert` Secret is reflected into `logging` by the existing reflector. RGW bucket + creds are provisioned by cephrgw-operator from the CRs in this PR. ## Open decisions (defaults chosen, flag to change) - **Archive subject filter:** default `logs.k8s.vault.>` (Vault audit). Candidates to add: `logs.k8s.authentik.>`, `logs.k8s.kanidm.>`, VM auth roles — **please confirm the exact security set.** - **Retention:** ClickHouse **3d** TTL; JetStream **3d** (130 GiB cap, 180Gi/node PVC, S2 compression); S3 lifecycle TBD (RGW-side). - **Sizing:** NATS 50Gi/node; ClickHouse 200Gi; aggregator 5Gi/2GiB buffer. - **HA:** ClickHouse single-replica (no Keeper) initially; NATS + transform tier are HA. - **VM front door:** HTTPS/NDJSON → vm-ingest → JetStream (vs. direct NATS TCP to VMs). - **CI image:** `timberio/vector:0.57.0-debian` + `natsio/nats-box:0.18.0` (Docker Hub) — mirror if runners restrict egress. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --- ## Update: images via artifactapi, DHI, stateless transform tier **Depends on unkin/terraform-artifactapi#16** (dockerhub allowlist patterns) — merge that first or images won't pull. ### Image table (all pulled through `artifactapi.k8s.syd1.au.unkin.net/dockerhub/…`) | Image | Upstream | artifactapi path | DHI? | |---|---|---|---| | clickhouse/clickhouse-server:24.8 | Docker Hub | dockerhub/clickhouse/clickhouse-server | DHI exists — **not used**: subscription/private-namespace + shell-less breaks the bash schema Job | | altinity/clickhouse-operator:0.27.2 | Docker Hub | dockerhub/altinity/clickhouse-operator | No DHI | | altinity/metrics-exporter:0.27.2 | Docker Hub | dockerhub/altinity/metrics-exporter | No DHI | | bitnami/kubectl:latest (crdHook) | Docker Hub | dockerhub/bitnami/kubectl | No DHI | | nats:2.14.2-alpine | Docker Hub | dockerhub/library/nats | No DHI for nats | | natsio/nats-server-config-reloader:0.23.0 | Docker Hub | dockerhub/natsio/nats-server-config-reloader | No DHI | | natsio/nats-box:0.18.0 (bootstrap Job) | Docker Hub | dockerhub/natsio/nats-box | No DHI | | timberio/vector:0.57.0-distroless-libc (runtime) | Docker Hub | dockerhub/timberio/vector | DHI exists — **not used** (subscription/private-namespace); distroless-libc is already near-hardened | | timberio/vector:0.57.0-debian (CI only) | Docker Hub | dockerhub/timberio/vector | shell needed for the CI step | **DHI decision:** Docker Hardened Images exist for clickhouse-server and vector, but they're **subscription-gated and served from a private Docker org namespace** (authenticated pull) — not reachable via the estate's anonymous artifactapi `dockerhub` proxy, and no DHI org/remote exists here. Their shell-less nature would also break the `bash` heredoc in the ClickHouse schema Job and the shell-based `vector-test` CI step. So: **upstream official through artifactapi**, using vector `distroless-libc` for runtime pods. Adopting DHI later would need a Docker Business subscription + an authenticated artifactapi remote for the DHI namespace. ### Transform tier is now a stateless Deployment Was a StatefulSet with a disk buffer/PVC; now a **Deployment with no PVC and an in-memory buffer** — **JetStream is the sole durability layer**. Added a **CPU HPA (min 2 / max 8)**. **Ack / backpressure design (important caveat):** Vector's NATS source has **`acknowledgements: no`** — it acks the JetStream message on receipt, *not* after the ClickHouse sink confirms. So end-to-end "sink-failure-must-not-ack" isn't achievable with the current source. What we get instead: the ClickHouse sink uses `buffer.when_full=block`, so on a ClickHouse outage the memory buffer fills, back-pressure stops the pull source, and **unpulled messages stay in JetStream and are redelivered**. The only at-risk window is the in-memory buffer (2000 events) of already-pulled events if a pod is killed *mid-outage*. This is the accepted trade for a stateless, autoscalable tier. HPA is safe because JetStream pull consumers distribute work across N replicas on the single durable consumer `transform`. (If stronger delivery is needed later: reintroduce a StatefulSet+disk buffer, or wait for upstream end-to-end-ack support on the nats source — vectordotdev/vector.) --- ## Update: 7d retention, tunable limits ConfigMap, honest sizing - **Retention → 7 days** (`max_age=168h`), still `retention=limits` / `discard=old`: the transform tier and the archiver each have their own durable consumer and independently see every message — reading never deletes; only max_age/max_bytes evict. - **Stream limits live in a ConfigMap** (`nats-stream-limits`: `max_age`, `max_bytes`, `dupe_window`). The `nats-bootstrap` PostSync Job reads them and does an idempotent **create-or-UPDATE** (`nats stream add` || `nats stream edit`). **How a change propagates:** the ConfigMap keeps its kustomize **content-hash suffix**, so editing a value renames the ConfigMap *and* rewrites the Job's `configMapKeyRef`s → the hook Job's spec changes → Argo re-runs it (on top of PostSync hooks running every sync with `hook-delete-policy=BeforeHookCreation`) → `nats stream edit` applies the new limits. No manual `nats` surgery. **Verified against a real nats-server:** create (7d), idempotent re-run, and a `max_age` change (168h→24h) all applied; all flags incl. `--compression=s2` accepted by nats CLI v0.2.3. - **Honest 7d sizing (stated assumption — please sanity-check against real volume):** - Assume **~1,500 events/s** average @ **~1 KiB/event** stored JSON ⇒ **~130 GiB/day raw**, ~910 GiB/7d raw per replica. - Enable **JetStream S2 compression** (logs ~4× conservative) ⇒ **~33 GiB/day**, **~230 GiB/7d** compressed per replica. - **`max_bytes = 300 GiB`** (headroom over the 230 GiB estimate). **PVC = 400Gi/node** on `cephrbd-fast-delete` (max_bytes + file-store WAL/index/overhead, safely under). **3 replicas ⇒ 1.2 TiB provisioned.** - ⚠️ **This is a large, prominent number by design.** If real volume exceeds the assumption, `discard=old` truncates retention **below 7d** rather than silently overflowing. Raising retention/volume requires bumping **both** `max_bytes` (ConfigMap) **and** the file-store PVC (values-nats.yaml) together — the PVC is not a live-tunable knob. - Replay window in the runbook is now **7d** (beyond that → the S3 archive). --- ## Update: retention cut to 3 days (both stores), PVCs shrunk Ben: 1.2 TiB is too much. Both stores now retain **3 days**; long-term retention lives **exclusively in the encrypted S3 archive** (the archiver's configured subjects) — everything else is gone after 3d. That's the accepted design. | Store | Retention | Byte cap | PVC/node | Replicas | Total | |---|---|---|---|---|---| | NATS JetStream `LOGS` | `max_age=72h` (3d) | `max_bytes=130 GiB` | 180Gi | 3 | **~0.5 TiB** (was 1.2 TiB) | | ClickHouse `logs.raw` | `TTL 3 DAY` | — | 150Gi | 1 | 150Gi (was 200Gi) | **NATS math:** ~33 GiB/day compressed (S2) × 3d ≈ 100 GiB → `max_bytes` 130 GiB (headroom) under a 180Gi PVC. **ClickHouse math:** ~130 GiB/day raw, LZ4/ZSTD ~6× ⇒ ~20-25 GiB/day ⇒ ~60-75 GiB/3d; +merge headroom ⇒ 150Gi PVC. `logs.raw` is the only table. The retention knobs remain in the `nats-stream-limits` ConfigMap (max_age/max_bytes/dupe_window) — tunable without redeploy; the ClickHouse TTL is in the bootstrap DDL. ⚠️ **PVC-shrink caveat:** this is a **plan-time** change — the stack **is not deployed yet**, so shrinking PVCs is clean. If it were already deployed, PVCs **cannot shrink in place** (a StatefulSet/CHI PVC resize-down needs a recreate/migration, not an edit). Reviewed-on: #296 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 19:54:27 +10:00
parent e6f2cbc363
commit f45cb6989f
31 changed files with 1533 additions and 0 deletions
@@ -0,0 +1,16 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: clickhouse-system
+
+resources:
+  - ../../../base/clickhouse-system
+
+helmCharts:
+  - name: altinity-clickhouse-operator
+    repo: https://helm.altinity.com
+    version: "0.27.2"
+    releaseName: clickhouse-operator
+    namespace: clickhouse-system
+    valuesFile: values.yaml
@@ -0,0 +1,40 @@
+# Altinity ClickHouse operator. Cluster-scoped: watches ClickHouseInstallation
+# resources in all namespaces (the logs cluster lives in the `logging` namespace).
+# CRDs are installed at runtime by the chart's crdHook Job.
+#
+# All images are pulled through the artifactapi dockerhub remote (no direct
+# upstream). Upstream official images are used; no Docker Hardened Image variant
+# is adopted (DHI is subscription-gated and served from a private org namespace
+# not reachable via the anonymous artifactapi dockerhub proxy).
+crdHook:
+  image:
+    repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/bitnami/kubectl
+  resources:
+    requests:
+      cpu: 50m
+      memory: 64Mi
+    limits:
+      cpu: 250m
+      memory: 128Mi
+
+operator:
+  image:
+    repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/altinity/clickhouse-operator
+  resources:
+    requests:
+      cpu: 100m
+      memory: 256Mi
+    limits:
+      cpu: 500m
+      memory: 512Mi
+
+metrics:
+  image:
+    repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/altinity/metrics-exporter
+  resources:
+    requests:
+      cpu: 50m
+      memory: 64Mi
+    limits:
+      cpu: 250m
+      memory: 256Mi
@@ -0,0 +1,45 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: logging
+
+resources:
+  - ../../../base/logging
+
+helmCharts:
+  # Dedicated JetStream NATS cluster — the durable log bus.
+  - name: nats
+    repo: https://nats-io.github.io/k8s/helm/charts
+    version: "2.14.2"
+    releaseName: nats
+    namespace: logging
+    valuesFile: values-nats.yaml
+  # Edge agent (DaemonSet): tails every pod's logs, publishes to JetStream.
+  - name: vector
+    repo: https://helm.vector.dev
+    version: "0.57.0"
+    releaseName: vector-agent
+    namespace: logging
+    valuesFile: values-vector-agent.yaml
+  # VM ingest (Deployment): HTTP NDJSON front door -> JetStream.
+  - name: vector
+    repo: https://helm.vector.dev
+    version: "0.57.0"
+    releaseName: vector-vm-ingest
+    namespace: logging
+    valuesFile: values-vector-vm-ingest.yaml
+  # Transform tier (StatefulSet): JetStream consumer -> shape -> ClickHouse.
+  - name: vector
+    repo: https://helm.vector.dev
+    version: "0.57.0"
+    releaseName: vector-aggregator
+    namespace: logging
+    valuesFile: values-vector-aggregator.yaml
+  # Archiver (Deployment): independent JetStream consumer -> raw logs to S3.
+  - name: vector
+    repo: https://helm.vector.dev
+    version: "0.57.0"
+    releaseName: vector-archiver
+    namespace: logging
+    valuesFile: values-vector-archiver.yaml
@@ -0,0 +1,100 @@
+# Dedicated JetStream-enabled NATS cluster for the log bus. Deliberately NOT
+# shared with app messaging (streamstack et al. run their own NATS in their own
+# repo) — a separate cluster isolates logging blast-radius from app messaging
+# and lets us size retention/storage purely for the log outage-buffer + replay
+# use-case.
+fullnameOverride: nats
+
+config:
+  cluster:
+    enabled: true
+    replicas: 3
+  jetstream:
+    enabled: true
+    fileStore:
+      pvc:
+        # Sized for 3d retention: ~100 GiB/3d compressed (see
+        # nats-stream-limits ConfigMap) + file-store WAL/index/overhead, kept
+        # safely above the 130 GiB max_bytes cap. 3 replicas => ~0.5 TiB total
+        # provisioned on cephrbd-fast-delete. NB: this is the honest number for
+        # the assumed ~1500 events/s; higher real volume needs a bigger PVC +
+        # max_bytes together, else discard=old truncates retention below 3d.
+        size: 180Gi
+        storageClassName: cephrbd-fast-delete
+  # Per-user auth with publish/subscribe separation. Passwords are injected as
+  # env vars from the Vault-synced nats-auth Secret (NATS expands $VAR in config).
+  merge:
+    authorization:
+      users:
+        # Bootstrap Job (stream/consumer management) — full JetStream API.
+        - user: log-admin
+          password: $NATS_ADMIN_PASSWORD
+        # Edge publishers (k8s DaemonSet + VM ingest) — publish only.
+        - user: log-producer
+          password: $NATS_PRODUCER_PASSWORD
+          permissions:
+            publish:
+              allow:
+                - "logs.>"
+            subscribe:
+              allow:
+                - "_INBOX.>"
+        # Consumers (transform tier + archiver) — pull + ack only, no publish
+        # to log subjects.
+        - user: log-consumer
+          password: $NATS_CONSUMER_PASSWORD
+          permissions:
+            publish:
+              allow:
+                - "$JS.API.CONSUMER.>"
+                - "$JS.API.STREAM.INFO.>"
+                - "$JS.ACK.LOGS.>"
+            subscribe:
+              allow:
+                - "_INBOX.>"
+
+container:
+  # Pulled through the artifactapi dockerhub remote (upstream official nats;
+  # no DHI variant available for nats).
+  image:
+    repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/library/nats
+    tag: 2.14.2-alpine
+  env:
+    NATS_ADMIN_PASSWORD:
+      valueFrom:
+        secretKeyRef:
+          name: nats-auth
+          key: admin_password
+    NATS_PRODUCER_PASSWORD:
+      valueFrom:
+        secretKeyRef:
+          name: nats-auth
+          key: producer_password
+    NATS_CONSUMER_PASSWORD:
+      valueFrom:
+        secretKeyRef:
+          name: nats-auth
+          key: consumer_password
+  resources:
+    requests:
+      cpu: 250m
+      memory: 512Mi
+    limits:
+      cpu: "2"
+      memory: 4Gi
+
+# Roll the StatefulSet when nats-auth changes.
+podTemplate:
+  merge:
+    metadata:
+      annotations:
+        reloader.stakater.com/auto: "true"
+
+# Config-reloader sidecar image, also through artifactapi.
+reloader:
+  image:
+    repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/natsio/nats-server-config-reloader
+    tag: "0.23.0"
+
+natsBox:
+  enabled: false
@@ -0,0 +1,50 @@
+# Vector EDGE agent (DaemonSet) — thin publisher. Tails every node's pod logs
+# (incl. control-plane via the blanket toleration) and publishes them into
+# JetStream over the Vector NATS sink. No parsing; only a routing subject token
+# is attached. Shaping happens in the transform tier after JetStream.
+role: Agent
+fullnameOverride: vector-agent
+
+# Pulled through the artifactapi dockerhub remote; distroless-libc (no DHI —
+# subscription-gated/private-namespace, not reachable via the anon proxy).
+image:
+  repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector
+  tag: 0.57.0-distroless-libc
+
+rbac:
+  create: true
+serviceAccount:
+  create: true
+
+podLabels:
+  vector.dev/exclude: "true"
+
+tolerations:
+  - operator: Exists
+
+env:
+  - name: NATS_PRODUCER_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: nats-auth
+        key: producer_password
+
+resources:
+  requests:
+    cpu: 100m
+    memory: 256Mi
+  limits:
+    cpu: "1"
+    memory: 1Gi
+
+service:
+  enabled: false
+
+# Pipeline is the single source of truth in apps/base/logging/vector/agent.yaml,
+# mounted via existingConfigMaps (avoids the chart's customConfig Helm-tpl pass).
+dataDir: /vector-data-dir
+existingConfigMaps:
+  - vector-agent-config
+
+workloadResourceAnnotations:
+  reloader.stakater.com/auto: "true"
@@ -0,0 +1,80 @@
+# Vector TRANSFORM tier (STATELESS Deployment) — the "brain": sole ClickHouse
+# writer, owns all transforms, holds the only ClickHouse + NATS-consumer creds.
+#
+# Stateless by design: a JetStream pull consumer with NO PVC and NO disk buffer.
+# JetStream is the sole durability layer. On a ClickHouse outage the clickhouse
+# sink blocks (buffer when_full=block), back-pressure stops the source pulling,
+# and unpulled messages stay in JetStream for redelivery. Because Vector's NATS
+# source does NOT support end-to-end acknowledgements (it acks on receipt, not
+# after the sink), the only at-risk window is the in-memory buffer's worth of
+# already-pulled events if a pod is killed mid-outage — the accepted trade for a
+# horizontally-autoscalable stateless tier. Multiple replicas share the one
+# durable consumer `transform` (JetStream pull consumers distribute work), so
+# HPA is safe.
+role: Stateless-Aggregator
+fullnameOverride: vector-aggregator
+
+image:
+  repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector
+  tag: 0.57.0-distroless-libc
+
+# Horizontal autoscaling on CPU — safe with N replicas on one durable consumer.
+autoscaling:
+  enabled: true
+  minReplicas: 2
+  maxReplicas: 8
+  targetCPUUtilizationPercentage: 70
+
+workloadResourceAnnotations:
+  reloader.stakater.com/auto: "true"
+
+podLabels:
+  vector.dev/exclude: "true"
+
+# Pipeline is the single source of truth in apps/base/logging/vector/
+# aggregator.yaml (unit-tested by `vector test` in CI), mounted via
+# existingConfigMaps. No persistence — stateless.
+dataDir: /vector-data-dir
+existingConfigMaps:
+  - vector-aggregator-config
+
+# The ONLY place ClickHouse + NATS-consumer creds are consumed.
+env:
+  - name: CLICKHOUSE_USER
+    valueFrom:
+      secretKeyRef:
+        name: clickhouse-credentials
+        key: username
+  - name: CLICKHOUSE_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: clickhouse-credentials
+        key: password
+  - name: NATS_CONSUMER_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: nats-auth
+        key: consumer_password
+
+# Pure consumer: expose only the Vector API for debugging.
+containerPorts:
+  - name: api
+    containerPort: 8686
+    protocol: TCP
+
+service:
+  enabled: true
+  type: ClusterIP
+  ports:
+    - name: api
+      port: 8686
+      targetPort: 8686
+      protocol: TCP
+
+resources:
+  requests:
+    cpu: 250m
+    memory: 512Mi
+  limits:
+    cpu: "2"
+    memory: 2Gi
@@ -0,0 +1,56 @@
+# Vector ARCHIVER tier (Deployment) — independent JetStream consumer writing raw
+# logs to S3 (Ceph RGW). Isolated from the ClickHouse path (own durable
+# consumer). Pipeline: apps/base/logging/vector/archiver.yaml.
+role: Stateless-Aggregator
+fullnameOverride: vector-archiver
+replicas: 1
+
+image:
+  repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector
+  tag: 0.57.0-distroless-libc
+
+workloadResourceAnnotations:
+  reloader.stakater.com/auto: "true"
+
+podLabels:
+  vector.dev/exclude: "true"
+
+dataDir: /vector-data-dir
+existingConfigMaps:
+  - vector-archiver-config
+
+env:
+  - name: NATS_CONSUMER_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: nats-auth
+        key: consumer_password
+
+# S3 creds (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) from the cephrgw-operator
+# BucketAccess Secret.
+envFrom:
+  - secretRef:
+      name: logs-archive-s3
+
+# Trust the internal unkin.net Vault-PKI CA to verify s3.ceph.unkin.net.
+# vault-ca-cert is reflected into every namespace from the certificates ns.
+extraVolumes:
+  - name: vault-ca-cert
+    secret:
+      secretName: vault-ca-cert
+extraVolumeMounts:
+  - name: vault-ca-cert
+    mountPath: /etc/vault-ca/ca.crt
+    subPath: ca.crt
+    readOnly: true
+
+service:
+  enabled: false
+
+resources:
+  requests:
+    cpu: 100m
+    memory: 256Mi
+  limits:
+    cpu: "1"
+    memory: 1Gi
@@ -0,0 +1,56 @@
+# Vector VM-INGEST tier (Deployment) — VM front door. HTTP NDJSON in (behind the
+# logs-ingest Gateway), publishes into JetStream. Stateless publisher.
+# Pipeline: apps/base/logging/vector/vm-ingest.yaml (unit-tested in CI).
+role: Stateless-Aggregator
+fullnameOverride: vector-vm-ingest
+replicas: 2
+
+image:
+  repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector
+  tag: 0.57.0-distroless-libc
+
+workloadResourceAnnotations:
+  reloader.stakater.com/auto: "true"
+
+podLabels:
+  vector.dev/exclude: "true"
+
+dataDir: /vector-data-dir
+existingConfigMaps:
+  - vector-vm-ingest-config
+
+env:
+  - name: NATS_PRODUCER_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: nats-auth
+        key: producer_password
+
+containerPorts:
+  - name: http-ingest
+    containerPort: 8080
+    protocol: TCP
+  - name: api
+    containerPort: 8686
+    protocol: TCP
+
+service:
+  enabled: true
+  type: ClusterIP
+  ports:
+    - name: http-ingest
+      port: 8080
+      targetPort: 8080
+      protocol: TCP
+    - name: api
+      port: 8686
+      targetPort: 8686
+      protocol: TCP
+
+resources:
+  requests:
+    cpu: 100m
+    memory: 256Mi
+  limits:
+    cpu: "1"
+    memory: 1Gi