Fix logging deploy: enable Vector env interpolation; operator watches logging ns #301

Merged

benvin merged 1 commits from benvin/logging-deploy-fixes into main 2026-07-29 00:11:34 +10:00

Conversation 0 Commits 1 Files Changed 5

+19

unkinben commented 2026-07-28 21:34:39 +10:00

Owner

Copy Link

Copy Source

Why

PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here.

Root cause 1 — Vector env-var interpolation is off by default in 0.57

vector-aggregator and vector-archiver crash-looped with async_nats::connector: authorization violation; vector-agent / vector-vm-ingest were "Running" but silently failing to publish.

Diagnosis (evidence):

• The NATS server config does expand $NATS_*_PASSWORD (a nats CLI login with the real secret value authenticated fine), and the ACL was not the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation).
• The failure reproduces locally: a hardcoded password connects; the same value via ${NATS_CONSUMER_PASSWORD} fails. Configuring the server to expect the literal string ${NATS_CONSUMER_PASSWORD} makes Vector connect — proving Vector sends the literal, un-interpolated string.
• vector --help shows --dangerously-allow-env-var-interpolation — in 0.57 ${VAR} interpolation is opt-in. An unset-var test confirms interpolation is off (no "unknown env var" error).
• Verified fix: with VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true → connects and authenticates.

Every tier uses ${...} for auth (${NATS_*_PASSWORD}, ${CLICKHOUSE_*}), so the env var is added to all four vector deployments. (This slipped past CI because vector test never opens the NATS connection.)

Root cause 2 — operator watches only its own namespace

kubectl get chi -n logging showed the logs CHI existed but with empty status / no finalizer — the operator never touched it, so the logging-logging Argo sync was stuck Progressing on "waiting for healthy state of ClickHouseInstallation/logs", and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed).

Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states watchNamespaces: [] (our value) makes the operator watch only its own namespace (clickhouse-system). The CHI is in logging.

Fix: watchNamespaces: ["logging"] → operator config watch.namespaces.include: [logging].

Changes

• apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml: add VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true.
• apps/overlays/au-syd1/clickhouse-system/values.yaml: watchNamespaces: ["logging"].

No NATS ACL change (the original narrow ACL is correct). No secret/base changes.

Expected recovery after merge + sync

1. clickhouse-system syncs → operator config gains logging → operator restarts → reconciles the logs CHI → CHI pod comes up healthy.
2. logging-logging sync unblocks → PostSync hooks run → JetStream LOGS stream + transform/archiver consumers created; logs.raw table created.
3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3.
4. Verify: nats stream info LOGS shows messages; SELECT count() FROM logs.raw increases.

Validation

kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders watch.namespaces.include: [logging]; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it).

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here. ## Root cause 1 — Vector env-var interpolation is off by default in 0.57 `vector-aggregator` and `vector-archiver` crash-looped with `async_nats::connector: authorization violation`; `vector-agent` / `vector-vm-ingest` were "Running" but silently failing to publish. Diagnosis (evidence): - The NATS server config **does** expand `$NATS_*_PASSWORD` (a `nats` CLI login with the real secret value authenticated fine), and the ACL was **not** the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation). - The failure reproduces locally: a hardcoded password connects; the **same value via `${NATS_CONSUMER_PASSWORD}` fails**. Configuring the server to expect the literal string `${NATS_CONSUMER_PASSWORD}` makes Vector connect — proving **Vector sends the literal, un-interpolated string**. - `vector --help` shows `--dangerously-allow-env-var-interpolation` — in 0.57 `${VAR}` interpolation is **opt-in**. An unset-var test confirms interpolation is off (no "unknown env var" error). - Verified fix: with `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` → **connects and authenticates**. Every tier uses `${...}` for auth (`${NATS_*_PASSWORD}`, `${CLICKHOUSE_*}`), so the env var is added to **all four** vector deployments. (This slipped past CI because `vector test` never opens the NATS connection.) ## Root cause 2 — operator watches only its own namespace `kubectl get chi -n logging` showed the `logs` CHI existed but with **empty status / no finalizer** — the operator never touched it, so the `logging-logging` Argo sync was stuck `Progressing` on *"waiting for healthy state of ClickHouseInstallation/logs"*, and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed). Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states `watchNamespaces: []` (our value) makes the operator **watch only its own namespace** (`clickhouse-system`). The CHI is in `logging`. Fix: `watchNamespaces: ["logging"]` → operator config `watch.namespaces.include: [logging]`. ## Changes - `apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml`: add `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true`. - `apps/overlays/au-syd1/clickhouse-system/values.yaml`: `watchNamespaces: ["logging"]`. No NATS ACL change (the original narrow ACL is correct). No secret/base changes. ## Expected recovery after merge + sync 1. clickhouse-system syncs → operator config gains `logging` → operator restarts → reconciles the `logs` CHI → CHI pod comes up healthy. 2. `logging-logging` sync unblocks → PostSync hooks run → JetStream `LOGS` stream + `transform`/`archiver` consumers created; `logs.raw` table created. 3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3. 4. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` increases. ## Validation kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders `watch.namespaces.include: [logging]`; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

unkinben added 1 commit 2026-07-28 21:34:39 +10:00

Fix logging deploy: enable Vector env interpolation; operator watches logging ns ... ab76fb8420

Two root causes found by diagnosing the live cluster after #296 deployed.

1. Vector 0.57 disables ${VAR} config interpolation by default (it now requires
   --dangerously-allow-env-var-interpolation). So every ${...PASSWORD} in the
   pipelines was sent as a literal string, breaking NATS auth for ALL vector
   tiers: the aggregator/archiver crash-looped with "authorization violation",
   and the agent/vm-ingest producers silently failed to publish (ClickHouse
   creds would have failed the same way). Enable interpolation via
   VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true on all four vector
   deployments. Verified end-to-end: without the flag -> authorization
   violation; with it -> connects and authenticates (narrow ACL unchanged, so
   the earlier ACL theory was wrong and nothing there needed changing).

2. The Altinity operator's watchNamespaces defaulted to [] which, per the chart,
   makes it watch ONLY its own namespace (clickhouse-system). The
   ClickHouseInstallation lives in `logging`, so it was never reconciled (no
   finalizer, empty status) -> the logging-logging Argo sync blocked forever on
   "waiting for healthy CHI" -> the PostSync hooks (nats-bootstrap stream/
   consumers, clickhouse-schema) never ran. Set watchNamespaces: [logging] so
   the operator reconciles the CHI, the sync completes, and the hooks run.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

benvin merged commit 7dddf8c5aa into main 2026-07-29 00:11:34 +10:00

benvin deleted branch benvin/logging-deploy-fixes 2026-07-29 00:11:34 +10:00

unkinben referenced this issue from a commit 2026-07-29 00:56:59 +10:00

Fix NATS auth: wrap env-var passwords in << >> so the server expands them

unkinben referenced this pull request 2026-07-29 00:57:19 +10:00

Fix NATS auth: wrap env-var passwords in << >> so the server expands them #306

benvin referenced this issue from a commit 2026-07-29 19:28:29 +10:00

Fix NATS auth: wrap env-var passwords in << >> so the server expands them (#306)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/argocd-apps#301