Live diagnosis of the still-broken stack found the real NATS auth bug. The
server was rejecting ALL clients (bootstrap admin, and every vector tier) with
"authorization violation" even though the pod env and the Vault secret matched:
authenticating as log-admin with the LITERAL string "$NATS_ADMIN_PASSWORD"
succeeded, proving the server stored the passwords un-expanded.
The nats chart renders config.merge as JSON, so a plain `$VAR` becomes the
quoted literal "$VAR" in nats.conf, and NATS does not expand variables inside
quoted strings. Per the chart README, env vars must be wrapped in `<< $VAR >>`
so they render UNQUOTED and NATS expands them. Wrap all three user passwords.
This is the server-side half; #301 fixed the client side (Vector 0.57 needs
VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION to send the real password).
Both are required: with both, server-expanded == vector-interpolated. Verified
end-to-end (nats-server with unquoted $VAR + env, vector with the flag): admin
auth ok, stream/consumer created, vector consumer connects.
Once this syncs, the config-reloader reloads NATS with the real passwords, the
nats-bootstrap PostSync hook succeeds (creating the LOGS stream + consumers),
the sync completes and applies #301's vector env, and the aggregator/archiver
recover on their next restart.
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben