unkinben
96afbcf5e1
## Why After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient). ### Evidence - Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped. - The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected. - **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**. ### Root cause The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them. ## What Wrap all three user passwords in `<< >>`: ``` password: << $NATS_ADMIN_PASSWORD >> # (+ producer, consumer) ``` Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted). This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password. ## Verified end-to-end nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully. ## Expected recovery after merge + sync 1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords. 2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config. 3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes. 4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3. 5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises. ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix. ## Validation kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #306 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
104 lines
3.3 KiB
YAML
104 lines
3.3 KiB
YAML
# Dedicated JetStream-enabled NATS cluster for the log bus. Deliberately NOT
|
|
# shared with app messaging (streamstack et al. run their own NATS in their own
|
|
# repo) — a separate cluster isolates logging blast-radius from app messaging
|
|
# and lets us size retention/storage purely for the log outage-buffer + replay
|
|
# use-case.
|
|
fullnameOverride: nats
|
|
|
|
config:
|
|
cluster:
|
|
enabled: true
|
|
replicas: 3
|
|
jetstream:
|
|
enabled: true
|
|
fileStore:
|
|
pvc:
|
|
# Sized for 3d retention: ~100 GiB/3d compressed (see
|
|
# nats-stream-limits ConfigMap) + file-store WAL/index/overhead, kept
|
|
# safely above the 130 GiB max_bytes cap. 3 replicas => ~0.5 TiB total
|
|
# provisioned on cephrbd-fast-delete. NB: this is the honest number for
|
|
# the assumed ~1500 events/s; higher real volume needs a bigger PVC +
|
|
# max_bytes together, else discard=old truncates retention below 3d.
|
|
size: 180Gi
|
|
storageClassName: cephrbd-fast-delete
|
|
# Per-user auth with publish/subscribe separation. Passwords are injected as
|
|
# env vars from the Vault-synced nats-auth Secret. The `<< $VAR >>` wrapping is
|
|
# REQUIRED by this chart: it renders the value UNQUOTED in nats.conf so the
|
|
# NATS server expands the env var. A plain `$VAR` is JSON-quoted ("$VAR") and
|
|
# NATS then treats it as a literal string — which broke auth for every client.
|
|
merge:
|
|
authorization:
|
|
users:
|
|
# Bootstrap Job (stream/consumer management) — full JetStream API.
|
|
- user: log-admin
|
|
password: << $NATS_ADMIN_PASSWORD >>
|
|
# Edge publishers (k8s DaemonSet + VM ingest) — publish only.
|
|
- user: log-producer
|
|
password: << $NATS_PRODUCER_PASSWORD >>
|
|
permissions:
|
|
publish:
|
|
allow:
|
|
- "logs.>"
|
|
subscribe:
|
|
allow:
|
|
- "_INBOX.>"
|
|
# Consumers (transform tier + archiver) — pull + ack only, no publish
|
|
# to log subjects.
|
|
- user: log-consumer
|
|
password: << $NATS_CONSUMER_PASSWORD >>
|
|
permissions:
|
|
publish:
|
|
allow:
|
|
- "$JS.API.CONSUMER.>"
|
|
- "$JS.API.STREAM.INFO.>"
|
|
- "$JS.ACK.LOGS.>"
|
|
subscribe:
|
|
allow:
|
|
- "_INBOX.>"
|
|
|
|
container:
|
|
# Pulled through the artifactapi dockerhub remote (upstream official nats;
|
|
# no DHI variant available for nats).
|
|
image:
|
|
repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/library/nats
|
|
tag: 2.14.2-alpine
|
|
env:
|
|
NATS_ADMIN_PASSWORD:
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nats-auth
|
|
key: admin_password
|
|
NATS_PRODUCER_PASSWORD:
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nats-auth
|
|
key: producer_password
|
|
NATS_CONSUMER_PASSWORD:
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nats-auth
|
|
key: consumer_password
|
|
resources:
|
|
requests:
|
|
cpu: 250m
|
|
memory: 512Mi
|
|
limits:
|
|
cpu: "2"
|
|
memory: 4Gi
|
|
|
|
# Roll the StatefulSet when nats-auth changes.
|
|
podTemplate:
|
|
merge:
|
|
metadata:
|
|
annotations:
|
|
reloader.stakater.com/auto: "true"
|
|
|
|
# Config-reloader sidecar image, also through artifactapi.
|
|
reloader:
|
|
image:
|
|
repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/natsio/nats-server-config-reloader
|
|
tag: "0.23.0"
|
|
|
|
natsBox:
|
|
enabled: false
|