Add Tier-2 per-app Vector transform pipelines (structured logs)
Why: extend the Tier-1 structured-parsing survey with seven more high-value log sources so they land in logs.raw columns/.fields for real querying instead of the generic catch-all. Stacks on #318. How: - Seven mutually-exclusive app_route conditions + parse transforms wired into the ClickHouse sink: bind_query (k8s bind-* + VM named), rancher_audit (cattle-system sidecar JSON), cnpg_pg (ONE transform for all 10 CNPG clusters via the `.postgres` container), gitea (router + access, k8s + VM), puppet (openvoxserver/openvoxdb logback + access), litellm (JSON request logs), postfix (per-line maillog). - Carve `.postgres` out of the Tier-1 authentik route and the new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). - Companion k8s config flips in this PR: litellm JSON_LOGS=True; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on. - 15 new vector test cases (routing + field extraction + the authentik-postgres->cnpg exclusivity proof); all 35 green (vector 0.57). - Structured fields go into the existing fields Map(String,String) — no DDL change. Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with the .file/.SYSLOG_IDENTIFIER tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
@@ -18,6 +18,9 @@ spec:
|
||||
# without it every dynamic update is "denied due to allow-query".
|
||||
extraOptions:
|
||||
- "allow-query { localhost; auth-acl-main; 10.42.0.0/16; }"
|
||||
# Enable query logging for the Tier-2 vector bind_query pipeline (see the
|
||||
# resolvers cluster for the routing rationale).
|
||||
- "querylog yes"
|
||||
service:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Local
|
||||
|
||||
@@ -30,6 +30,11 @@ spec:
|
||||
# (incl. k8s.syd1.au.unkin.net); 18.198.in-addr.arpa covers every reverse zone.
|
||||
extraOptions:
|
||||
- "validate-except { unkin.net; 18.198.in-addr.arpa; consul; }"
|
||||
# Enable query logging so the Tier-2 vector bind_query pipeline can parse
|
||||
# client/qname/qtype. Routes to the `queries` category which, with no explicit
|
||||
# logging{} clause, follows the default category to the named foreground
|
||||
# stderr channel -> pod stdout -> vector (subject logs.k8s.bind-internal.*).
|
||||
- "querylog yes"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
|
||||
Reference in New Issue
Block a user