From 96afbcf5e17809bdb97ea961f872c1b30620439a Mon Sep 17 00:00:00 2001
From: Ben Vincent <ben@unkin.net>
Date: Wed, 29 Jul 2026 19:28:27 +1000
Subject: [PATCH] Fix NATS auth: wrap env-var passwords in << >> so the server
 expands them (#306)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Why

After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient).

### Evidence
- Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped.
- The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected.
- **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**.

### Root cause
The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them.

## What

Wrap all three user passwords in `<< >>`:
```
password: << $NATS_ADMIN_PASSWORD >>      # (+ producer, consumer)
```
Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted).

This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password.

## Verified end-to-end
nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully.

## Expected recovery after merge + sync

1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords.
2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config.
3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes.
4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3.
5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises.

ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix.

## Validation
kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: https://git.unkin.net/unkin/argocd-apps/pulls/306
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
---
 apps/overlays/au-syd1/logging/values-nats.yaml | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/apps/overlays/au-syd1/logging/values-nats.yaml b/apps/overlays/au-syd1/logging/values-nats.yaml
index 956ebd2..fc16bd2 100644
--- a/apps/overlays/au-syd1/logging/values-nats.yaml
+++ b/apps/overlays/au-syd1/logging/values-nats.yaml
@@ -22,16 +22,19 @@ config:
         size: 180Gi
         storageClassName: cephrbd-fast-delete
   # Per-user auth with publish/subscribe separation. Passwords are injected as
-  # env vars from the Vault-synced nats-auth Secret (NATS expands $VAR in config).
+  # env vars from the Vault-synced nats-auth Secret. The `<< $VAR >>` wrapping is
+  # REQUIRED by this chart: it renders the value UNQUOTED in nats.conf so the
+  # NATS server expands the env var. A plain `$VAR` is JSON-quoted ("$VAR") and
+  # NATS then treats it as a literal string — which broke auth for every client.
   merge:
     authorization:
       users:
         # Bootstrap Job (stream/consumer management) — full JetStream API.
         - user: log-admin
-          password: $NATS_ADMIN_PASSWORD
+          password: << $NATS_ADMIN_PASSWORD >>
         # Edge publishers (k8s DaemonSet + VM ingest) — publish only.
         - user: log-producer
-          password: $NATS_PRODUCER_PASSWORD
+          password: << $NATS_PRODUCER_PASSWORD >>
           permissions:
             publish:
               allow:
@@ -42,7 +45,7 @@ config:
         # Consumers (transform tier + archiver) — pull + ack only, no publish
         # to log subjects.
         - user: log-consumer
-          password: $NATS_CONSUMER_PASSWORD
+          password: << $NATS_CONSUMER_PASSWORD >>
           permissions:
             publish:
               allow: