unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add Tier-1 per-app Vector transform pipelines (structured logs) #318

Merged
benvin merged 1 commits from benvin/vector-tier1-pipelines into main 2026-08-01 00:22:32 +10:00
Conversation 0 Commits 1 Files Changed 4
+660 -4
unkinben commented 2026-07-31 22:18:10 +10:00
Owner
Copy Link
Copy Source

Why: the logging aggregator wrote every event through the generic catch-all shape. The Tier-1 survey picked six high-value log sources that warrant structured parsing into logs.raw columns/fields for real querying.

How:

  • Two-stage routing in aggregator.yaml: app_route peels off the six Tier-1 streams by subject / VM source tag (mutually exclusive — no double-insert); everything else falls through app_route._unmatched to the unchanged generic k8s/vm catch-all.
  • Six parse transforms emit the full logs.raw shape plus structured .fields (Map(String,String) — no DDL change): authentik (JSON), traefik (JSON access), vault audit (JSON), nginx access+error (regex), haproxy httplog (regex), glauth (JSON).
  • Companion flip: traefik-system access logs to format: json (both overlays) so traefik_parse has structured input.
  • 15 new vector test cases (routing + field extraction) in aggregator-tests.yaml; all green locally (vector 0.57).

Live now: authentik + traefik (k8s). Awaiting the puppet-side vector rollout (logs.vm.* with .file/.SYSLOG_IDENTIFIER tags per the documented convention): vault-file, nginx, haproxy, glauth — transforms are present and unit-tested so they light up automatically.

Note: geoip enrichment for nginx/traefik client IPs is a separate prerequisite — no enrichment table exists in the aggregator yet; these transforms extract client_ip ready for it.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

Why: the logging aggregator wrote every event through the generic catch-all shape. The Tier-1 survey picked six high-value log sources that warrant structured parsing into logs.raw columns/fields for real querying. How: - Two-stage routing in `aggregator.yaml`: `app_route` peels off the six Tier-1 streams by subject / VM source tag (mutually exclusive — no double-insert); everything else falls through `app_route._unmatched` to the unchanged generic k8s/vm catch-all. - Six parse transforms emit the full `logs.raw` shape plus structured `.fields` (Map(String,String) — no DDL change): authentik (JSON), traefik (JSON access), vault audit (JSON), nginx access+error (regex), haproxy httplog (regex), glauth (JSON). - Companion flip: traefik-system access logs to `format: json` (both overlays) so `traefik_parse` has structured input. - 15 new `vector test` cases (routing + field extraction) in `aggregator-tests.yaml`; all green locally (vector 0.57). Live now: authentik + traefik (k8s). Awaiting the puppet-side vector rollout (logs.vm.* with `.file`/`.SYSLOG_IDENTIFIER` tags per the documented convention): vault-file, nginx, haproxy, glauth — transforms are present and unit-tested so they light up automatically. Note: geoip enrichment for nginx/traefik client IPs is a separate prerequisite — no enrichment table exists in the aggregator yet; these transforms extract `client_ip` ready for it. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-07-31 22:18:11 +10:00
Add Tier-1 per-app Vector transform pipelines (structured logs)
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
084e454954 
Why: the logging aggregator wrote every event through the generic catch-all
shape. The Tier-1 survey (session 15) picked six high-value log sources that
warrant structured parsing into logs.raw columns/fields for real querying.

How:
- Two-stage routing in aggregator.yaml: app_route peels off the six Tier-1
  streams by subject / VM source tag (mutually exclusive, no double-insert),
  everything else falls through app_route._unmatched to the unchanged generic
  k8s/vm shape catch-all.
- Six parse transforms emit the full logs.raw shape plus structured .fields
  (Map(String,String), no DDL change): authentik (JSON), traefik (JSON access),
  vault audit (JSON), nginx access+error (regex), haproxy httplog (regex),
  glauth (JSON).
- Flip traefik-system access logs to format: json (companion change) so
  traefik_parse has structured input.
- 15 new vector test cases (routing + field extraction) in aggregator-tests.yaml.

VM apps (vault-file/nginx/haproxy/glauth) light up when the puppet-side vector
rollout ships logs.vm.* with .file / .SYSLOG_IDENTIFIER tags per the documented
convention. authentik and traefik are live now.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
benvin merged commit 8d70149467 into main 2026-08-01 00:22:32 +10:00
benvin deleted branch benvin/vector-tier1-pipelines 2026-08-01 00:22:33 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#318
Delete Branch "benvin/vector-tier1-pipelines"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?