unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Fix NATS auth: wrap env-var passwords in << >> so the server expands them #306

Merged
benvin merged 1 commits from benvin/logging-deploy-fixes-2 into main 2026-07-29 19:28:27 +10:00
Conversation 0 Commits 1 Files Changed 1
+7 -4
unkinben commented 2026-07-29 00:57:19 +10:00
Owner
Copy Link
Copy Source

Why

After #301 merged, the stack was still broken. Live diagnosis found the actual NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient).

Evidence

  • Every NATS client failed with authorization violation: the nats-bootstrap PostSync Job hung 30 min as log-admin then failed DeadlineExceeded (its until nats account info loop never authenticated), and vector-aggregator/vector-archiver crash-looped.
  • The nats-0 container env matched the Vault secret exactly (all three password SHAs), yet auth was rejected.
  • Decisive test: authenticating as log-admin with the literal string $NATS_ADMIN_PASSWORD succeeded — proving the server stored the passwords un-expanded.

Root cause

The nats chart renders config.merge as JSON, so a plain password: $NATS_ADMIN_PASSWORD becomes the quoted literal "$NATS_ADMIN_PASSWORD" in nats.conf, and NATS does not expand variables inside quoted strings. Per the chart README, env vars must be wrapped in << $VAR >> to render unquoted so NATS expands them.

What

Wrap all three user passwords in << >>:

password: << $NATS_ADMIN_PASSWORD >>      # (+ producer, consumer)

Rendered nats.conf now emits "password": $NATS_ADMIN_PASSWORD (unquoted).

This is the server-side half; #301 (merged) fixed the client-side half (Vector 0.57 needs VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION to send the real password). Both are required — with both, server-expanded password == vector-interpolated password.

Verified end-to-end

nats-server with unquoted $VAR config + env, plus vector with the interpolation flag: admin account info OK, LOGS stream + transform consumer created, and the vector consumer connects successfully.

Expected recovery after merge + sync

  1. nats-config CM updates → the config-reloader reloads NATS with the real (expanded) passwords.
  2. The stuck logging-logging sync retries; the Sync phase applies #301's vector env + this config.
  3. nats-bootstrap PostSync hook now authenticates as admin → creates the LOGS stream + transform/archiver consumers → sync completes.
  4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3.
  5. Verify: nats stream info LOGS shows messages; SELECT count() FROM logs.raw rises.

ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix.

Validation

kustomize + kubeconform clean (logging 40); rendered nats.conf shows unquoted $VAR; pre-commit clean.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient). ### Evidence - Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped. - The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected. - **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**. ### Root cause The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them. ## What Wrap all three user passwords in `<< >>`: ``` password: << $NATS_ADMIN_PASSWORD >> # (+ producer, consumer) ``` Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted). This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password. ## Verified end-to-end nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully. ## Expected recovery after merge + sync 1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords. 2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config. 3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes. 4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3. 5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises. ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix. ## Validation kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-29 00:57:19 +10:00
Fix NATS auth: wrap env-var passwords in << >> so the server expands them
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
6ac73a25e3 
Live diagnosis of the still-broken stack found the real NATS auth bug. The
server was rejecting ALL clients (bootstrap admin, and every vector tier) with
"authorization violation" even though the pod env and the Vault secret matched:
authenticating as log-admin with the LITERAL string "$NATS_ADMIN_PASSWORD"
succeeded, proving the server stored the passwords un-expanded.

The nats chart renders config.merge as JSON, so a plain `$VAR` becomes the
quoted literal "$VAR" in nats.conf, and NATS does not expand variables inside
quoted strings. Per the chart README, env vars must be wrapped in `<< $VAR >>`
so they render UNQUOTED and NATS expands them. Wrap all three user passwords.

This is the server-side half; #301 fixed the client side (Vector 0.57 needs
VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION to send the real password).
Both are required: with both, server-expanded == vector-interpolated. Verified
end-to-end (nats-server with unquoted $VAR + env, vector with the flag): admin
auth ok, stream/consumer created, vector consumer connects.

Once this syncs, the config-reloader reloads NATS with the real passwords, the
nats-bootstrap PostSync hook succeeds (creating the LOGS stream + consumers),
the sync completes and applies #301's vector env, and the aggregator/archiver
recover on their next restart.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
benvin merged commit 96afbcf5e1 into main 2026-07-29 19:28:27 +10:00
benvin deleted branch benvin/logging-deploy-fixes-2 2026-07-29 19:28:28 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#306
Delete Branch "benvin/logging-deploy-fixes-2"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?