Fix NATS auth: wrap env-var passwords in << >> so the server expands them
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful

Live diagnosis of the still-broken stack found the real NATS auth bug. The
server was rejecting ALL clients (bootstrap admin, and every vector tier) with
"authorization violation" even though the pod env and the Vault secret matched:
authenticating as log-admin with the LITERAL string "$NATS_ADMIN_PASSWORD"
succeeded, proving the server stored the passwords un-expanded.

The nats chart renders config.merge as JSON, so a plain `$VAR` becomes the
quoted literal "$VAR" in nats.conf, and NATS does not expand variables inside
quoted strings. Per the chart README, env vars must be wrapped in `<< $VAR >>`
so they render UNQUOTED and NATS expands them. Wrap all three user passwords.

This is the server-side half; #301 fixed the client side (Vector 0.57 needs
VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION to send the real password).
Both are required: with both, server-expanded == vector-interpolated. Verified
end-to-end (nats-server with unquoted $VAR + env, vector with the flag): admin
auth ok, stream/consumer created, vector consumer connects.

Once this syncs, the config-reloader reloads NATS with the real passwords, the
nats-bootstrap PostSync hook succeeds (creating the LOGS stream + consumers),
the sync completes and applies #301's vector env, and the aggregator/archiver
recover on their next restart.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
2026-07-29 00:56:56 +10:00
parent 7dddf8c5aa
commit 6ac73a25e3
@@ -22,16 +22,19 @@ config:
size: 180Gi
storageClassName: cephrbd-fast-delete
# Per-user auth with publish/subscribe separation. Passwords are injected as
# env vars from the Vault-synced nats-auth Secret (NATS expands $VAR in config).
# env vars from the Vault-synced nats-auth Secret. The `<< $VAR >>` wrapping is
# REQUIRED by this chart: it renders the value UNQUOTED in nats.conf so the
# NATS server expands the env var. A plain `$VAR` is JSON-quoted ("$VAR") and
# NATS then treats it as a literal string — which broke auth for every client.
merge:
authorization:
users:
# Bootstrap Job (stream/consumer management) — full JetStream API.
- user: log-admin
password: $NATS_ADMIN_PASSWORD
password: << $NATS_ADMIN_PASSWORD >>
# Edge publishers (k8s DaemonSet + VM ingest) — publish only.
- user: log-producer
password: $NATS_PRODUCER_PASSWORD
password: << $NATS_PRODUCER_PASSWORD >>
permissions:
publish:
allow:
@@ -42,7 +45,7 @@ config:
# Consumers (transform tier + archiver) — pull + ack only, no publish
# to log subjects.
- user: log-consumer
password: $NATS_CONSUMER_PASSWORD
password: << $NATS_CONSUMER_PASSWORD >>
permissions:
publish:
allow: