unkinben
7dddf8c5aa
## Why PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here. ## Root cause 1 — Vector env-var interpolation is off by default in 0.57 `vector-aggregator` and `vector-archiver` crash-looped with `async_nats::connector: authorization violation`; `vector-agent` / `vector-vm-ingest` were "Running" but silently failing to publish. Diagnosis (evidence): - The NATS server config **does** expand `$NATS_*_PASSWORD` (a `nats` CLI login with the real secret value authenticated fine), and the ACL was **not** the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation). - The failure reproduces locally: a hardcoded password connects; the **same value via `${NATS_CONSUMER_PASSWORD}` fails**. Configuring the server to expect the literal string `${NATS_CONSUMER_PASSWORD}` makes Vector connect — proving **Vector sends the literal, un-interpolated string**. - `vector --help` shows `--dangerously-allow-env-var-interpolation` — in 0.57 `${VAR}` interpolation is **opt-in**. An unset-var test confirms interpolation is off (no "unknown env var" error). - Verified fix: with `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` → **connects and authenticates**. Every tier uses `${...}` for auth (`${NATS_*_PASSWORD}`, `${CLICKHOUSE_*}`), so the env var is added to **all four** vector deployments. (This slipped past CI because `vector test` never opens the NATS connection.) ## Root cause 2 — operator watches only its own namespace `kubectl get chi -n logging` showed the `logs` CHI existed but with **empty status / no finalizer** — the operator never touched it, so the `logging-logging` Argo sync was stuck `Progressing` on *"waiting for healthy state of ClickHouseInstallation/logs"*, and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed). Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states `watchNamespaces: []` (our value) makes the operator **watch only its own namespace** (`clickhouse-system`). The CHI is in `logging`. Fix: `watchNamespaces: ["logging"]` → operator config `watch.namespaces.include: [logging]`. ## Changes - `apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml`: add `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true`. - `apps/overlays/au-syd1/clickhouse-system/values.yaml`: `watchNamespaces: ["logging"]`. No NATS ACL change (the original narrow ACL is correct). No secret/base changes. ## Expected recovery after merge + sync 1. clickhouse-system syncs → operator config gains `logging` → operator restarts → reconciles the `logs` CHI → CHI pod comes up healthy. 2. `logging-logging` sync unblocks → PostSync hooks run → JetStream `LOGS` stream + `transform`/`archiver` consumers created; `logs.raw` table created. 3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3. 4. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` increases. ## Validation kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders `watch.namespaces.include: [logging]`; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #301 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
60 lines
1.5 KiB
YAML
60 lines
1.5 KiB
YAML
# Vector ARCHIVER tier (Deployment) — independent JetStream consumer writing raw
|
|
# logs to S3 (Ceph RGW). Isolated from the ClickHouse path (own durable
|
|
# consumer). Pipeline: apps/base/logging/vector/archiver.yaml.
|
|
role: Stateless-Aggregator
|
|
fullnameOverride: vector-archiver
|
|
replicas: 1
|
|
|
|
image:
|
|
repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector
|
|
tag: 0.57.0-distroless-libc
|
|
|
|
workloadResourceAnnotations:
|
|
reloader.stakater.com/auto: "true"
|
|
|
|
podLabels:
|
|
vector.dev/exclude: "true"
|
|
|
|
dataDir: /vector-data-dir
|
|
existingConfigMaps:
|
|
- vector-archiver-config
|
|
|
|
env:
|
|
# Vector 0.57 disables ${VAR} config interpolation by default; auth needs it.
|
|
- name: VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION
|
|
value: "true"
|
|
- name: NATS_CONSUMER_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nats-auth
|
|
key: consumer_password
|
|
|
|
# S3 creds (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) from the cephrgw-operator
|
|
# BucketAccess Secret.
|
|
envFrom:
|
|
- secretRef:
|
|
name: logs-archive-s3
|
|
|
|
# Trust the internal unkin.net Vault-PKI CA to verify s3.ceph.unkin.net.
|
|
# vault-ca-cert is reflected into every namespace from the certificates ns.
|
|
extraVolumes:
|
|
- name: vault-ca-cert
|
|
secret:
|
|
secretName: vault-ca-cert
|
|
extraVolumeMounts:
|
|
- name: vault-ca-cert
|
|
mountPath: /etc/vault-ca/ca.crt
|
|
subPath: ca.crt
|
|
readOnly: true
|
|
|
|
service:
|
|
enabled: false
|
|
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "1"
|
|
memory: 1Gi
|