unkinben
ab76fb8420
Two root causes found by diagnosing the live cluster after #296 deployed. 1. Vector 0.57 disables ${VAR} config interpolation by default (it now requires --dangerously-allow-env-var-interpolation). So every ${...PASSWORD} in the pipelines was sent as a literal string, breaking NATS auth for ALL vector tiers: the aggregator/archiver crash-looped with "authorization violation", and the agent/vm-ingest producers silently failed to publish (ClickHouse creds would have failed the same way). Enable interpolation via VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true on all four vector deployments. Verified end-to-end: without the flag -> authorization violation; with it -> connects and authenticates (narrow ACL unchanged, so the earlier ACL theory was wrong and nothing there needed changing). 2. The Altinity operator's watchNamespaces defaulted to [] which, per the chart, makes it watch ONLY its own namespace (clickhouse-system). The ClickHouseInstallation lives in `logging`, so it was never reconciled (no finalizer, empty status) -> the logging-logging Argo sync blocked forever on "waiting for healthy CHI" -> the PostSync hooks (nats-bootstrap stream/ consumers, clickhouse-schema) never ran. Set watchNamespaces: [logging] so the operator reconciles the CHI, the sync completes, and the hooks run. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
54 lines
1.4 KiB
YAML
54 lines
1.4 KiB
YAML
# Vector EDGE agent (DaemonSet) — thin publisher. Tails every node's pod logs
|
|
# (incl. control-plane via the blanket toleration) and publishes them into
|
|
# JetStream over the Vector NATS sink. No parsing; only a routing subject token
|
|
# is attached. Shaping happens in the transform tier after JetStream.
|
|
role: Agent
|
|
fullnameOverride: vector-agent
|
|
|
|
# Pulled through the artifactapi dockerhub remote; distroless-libc (no DHI —
|
|
# subscription-gated/private-namespace, not reachable via the anon proxy).
|
|
image:
|
|
repository: artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector
|
|
tag: 0.57.0-distroless-libc
|
|
|
|
rbac:
|
|
create: true
|
|
serviceAccount:
|
|
create: true
|
|
|
|
podLabels:
|
|
vector.dev/exclude: "true"
|
|
|
|
tolerations:
|
|
- operator: Exists
|
|
|
|
env:
|
|
# Vector 0.57 disables ${VAR} config interpolation by default; auth needs it.
|
|
- name: VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION
|
|
value: "true"
|
|
- name: NATS_PRODUCER_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nats-auth
|
|
key: producer_password
|
|
|
|
resources:
|
|
requests:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
limits:
|
|
cpu: "1"
|
|
memory: 1Gi
|
|
|
|
service:
|
|
enabled: false
|
|
|
|
# Pipeline is the single source of truth in apps/base/logging/vector/agent.yaml,
|
|
# mounted via existingConfigMaps (avoids the chart's customConfig Helm-tpl pass).
|
|
dataDir: /vector-data-dir
|
|
existingConfigMaps:
|
|
- vector-agent-config
|
|
|
|
workloadResourceAnnotations:
|
|
reloader.stakater.com/auto: "true"
|