unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add Tier-2 per-app Vector transform pipelines (structured logs) #320

Merged
benvin merged 1 commits from benvin/vector-tier2-pipelines into main 2026-08-08 22:12:46 +10:00
Conversation 0 Commits 1 Files Changed 6
+748 -2
unkinben commented 2026-08-01 01:33:05 +10:00
Owner
Copy Link
Copy Source

Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. Stacks on #318 — merge after it.

How:

  • 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: bind_query (k8s bind-* + VM named), rancher_audit (cattle-system sidecar JSON), cnpg_pg (ONE transform for all 10 CNPG clusters via the .postgres container), gitea (router+access, k8s+VM), puppet (openvoxserver/openvoxdb logback + access), litellm (JSON request logs), postfix (per-line maillog).
  • Carve .postgres out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact.
  • Companion k8s flips in this PR: litellm JSON_LOGS=True; bind querylog yes on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on.
  • 15 new vector test cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing fields Map(String,String) — no DDL change.

Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with .file/.SYSLOG_IDENTIFIER tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. **Stacks on #318 — merge after it.** How: - 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: **bind_query** (k8s bind-* + VM named), **rancher_audit** (cattle-system sidecar JSON), **cnpg_pg** (ONE transform for all 10 CNPG clusters via the `.postgres` container), **gitea** (router+access, k8s+VM), **puppet** (openvoxserver/openvoxdb logback + access), **litellm** (JSON request logs), **postfix** (per-line maillog). - Carve `.postgres` out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact. - Companion k8s flips in this PR: litellm `JSON_LOGS=True`; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on. - 15 new `vector test` cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing `fields Map(String,String)` — no DDL change. Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with `.file`/`.SYSLOG_IDENTIFIER` tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
unkinben added 1 commit 2026-08-01 01:33:06 +10:00
Add Tier-2 per-app Vector transform pipelines (structured logs)
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
595663bae9 
Why: extend the Tier-1 structured-parsing survey with seven more
high-value log sources so they land in logs.raw columns/.fields for real
querying instead of the generic catch-all. Stacks on #318.

How:
- Seven mutually-exclusive app_route conditions + parse transforms wired
  into the ClickHouse sink: bind_query (k8s bind-* + VM named),
  rancher_audit (cattle-system sidecar JSON), cnpg_pg (ONE transform for
  all 10 CNPG clusters via the `.postgres` container), gitea (router +
  access, k8s + VM), puppet (openvoxserver/openvoxdb logback + access),
  litellm (JSON request logs), postfix (per-line maillog).
- Carve `.postgres` out of the Tier-1 authentik route and the new
  puppet/gitea/litellm routes so the single cnpg_pg route claims every
  CNPG pod without double-insert (keeps app_route mutually exclusive).
- Companion k8s config flips in this PR: litellm JSON_LOGS=True; bind
  `querylog yes` on both bind-internal BindClusters; gitea router+access
  logging to stdout. Rancher auditLog was already on.
- 15 new vector test cases (routing + field extraction + the
  authentik-postgres->cnpg exclusivity proof); all 35 green (vector 0.57).
- Structured fields go into the existing fields Map(String,String) — no
  DDL change.

Puppet-side follow-ups (out of scope for argocd): enable named query
logging (profiles/dns/server.pp); ship the VM vector rollout with the
.file/.SYSLOG_IDENTIFIER tags for named/gitea/puppetserver(+multiline
logback join)/postfix maillog.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
benvin merged commit da4a66046a into main 2026-08-08 22:12:46 +10:00
benvin deleted branch benvin/vector-tier2-pipelines 2026-08-08 22:12:47 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#320
Delete Branch "benvin/vector-tier2-pipelines"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?