Add Tier-2 per-app Vector transform pipelines (structured logs) #320

Merged
benvin merged 1 commits from benvin/vector-tier2-pipelines into main 2026-08-08 22:12:46 +10:00

1 Commits

Author SHA1 Message Date
Ben Vincent 595663bae9 Add Tier-2 per-app Vector transform pipelines (structured logs)
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
Why: extend the Tier-1 structured-parsing survey with seven more
high-value log sources so they land in logs.raw columns/.fields for real
querying instead of the generic catch-all. Stacks on #318.

How:
- Seven mutually-exclusive app_route conditions + parse transforms wired
  into the ClickHouse sink: bind_query (k8s bind-* + VM named),
  rancher_audit (cattle-system sidecar JSON), cnpg_pg (ONE transform for
  all 10 CNPG clusters via the `.postgres` container), gitea (router +
  access, k8s + VM), puppet (openvoxserver/openvoxdb logback + access),
  litellm (JSON request logs), postfix (per-line maillog).
- Carve `.postgres` out of the Tier-1 authentik route and the new
  puppet/gitea/litellm routes so the single cnpg_pg route claims every
  CNPG pod without double-insert (keeps app_route mutually exclusive).
- Companion k8s config flips in this PR: litellm JSON_LOGS=True; bind
  `querylog yes` on both bind-internal BindClusters; gitea router+access
  logging to stdout. Rancher auditLog was already on.
- 15 new vector test cases (routing + field extraction + the
  authentik-postgres->cnpg exclusivity proof); all 35 green (vector 0.57).
- Structured fields go into the existing fields Map(String,String) — no
  DDL change.

Puppet-side follow-ups (out of scope for argocd): enable named query
logging (profiles/dns/server.pp); ship the VM vector rollout with the
.file/.SYSLOG_IDENTIFIER tags for named/gitea/puppetserver(+multiline
logback join)/postfix maillog.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-08-01 01:32:49 +10:00