Add Tier-1 per-app Vector transform pipelines (structured logs) #318

Merged
benvin merged 1 commits from benvin/vector-tier1-pipelines into main 2026-08-01 00:22:32 +10:00

1 Commits

Author SHA1 Message Date
unkinben 084e454954 Add Tier-1 per-app Vector transform pipelines (structured logs)
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
Why: the logging aggregator wrote every event through the generic catch-all
shape. The Tier-1 survey (session 15) picked six high-value log sources that
warrant structured parsing into logs.raw columns/fields for real querying.

How:
- Two-stage routing in aggregator.yaml: app_route peels off the six Tier-1
  streams by subject / VM source tag (mutually exclusive, no double-insert),
  everything else falls through app_route._unmatched to the unchanged generic
  k8s/vm shape catch-all.
- Six parse transforms emit the full logs.raw shape plus structured .fields
  (Map(String,String), no DDL change): authentik (JSON), traefik (JSON access),
  vault audit (JSON), nginx access+error (regex), haproxy httplog (regex),
  glauth (JSON).
- Flip traefik-system access logs to format: json (companion change) so
  traefik_parse has structured input.
- 15 new vector test cases (routing + field extraction) in aggregator-tests.yaml.

VM apps (vault-file/nginx/haproxy/glauth) light up when the puppet-side vector
rollout ships logs.vm.* with .file / .SYSLOG_IDENTIFIER tags per the documented
convention. authentik and traefik are live now.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-31 22:17:47 +10:00