unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

VM-side Vector rollout: ship host logs to the central logging stack #500

New Issue
Open
opened 2026-08-02 16:23:11 +10:00 by unkinben · 0 comments
unkinben commented 2026-08-02 16:23:11 +10:00
Owner
Copy Link
Copy Source

Problem
The centralized logging platform (Vector→JetStream→ClickHouse) is live for k8s, and Tier-1/Tier-2 transforms for VM apps (nginx, HAProxy, glauth, BIND, puppetserver, postfix, Vault file-audit) are implemented but dormant — nothing on the ~143-VM fleet ships logs yet, and the transforms match on .file/.SYSLOG_IDENTIFIER source tags the VMs must set.

Proposal

  • add a puppet profile deploying Vector (edge/thin) on VMs, shipping journald + tailed files to https://logs-ingest.k8s.syd1.au.unkin.net (vector-native/HTTPS), publishing to logs.vm.<host> subjects
  • set the source tags the aggregator expects: .file on file sources, .SYSLOG_IDENTIFIER on journald, for named/nginx/haproxy/glauth/postfix/puppetserver/vault_audit
  • enable BIND querylog yes in profiles/dns/server.pp
  • do puppetserver multiline logback stacktrace join at the edge (stateful) + tail puppetserver-access.log

Acceptance

  • VM logs appear in logs.raw under logs.vm.*; the dormant Tier-1/2 transforms populate structured fields; SELECT count() FROM logs.raw WHERE ...vm... rising; Vault file-audit reaches the logarchiver archive subject.
**Problem** The centralized logging platform (Vector→JetStream→ClickHouse) is live for k8s, and Tier-1/Tier-2 transforms for VM apps (nginx, HAProxy, glauth, BIND, puppetserver, postfix, Vault file-audit) are implemented but dormant — nothing on the ~143-VM fleet ships logs yet, and the transforms match on `.file`/`.SYSLOG_IDENTIFIER` source tags the VMs must set. **Proposal** - add a puppet profile deploying Vector (edge/thin) on VMs, shipping journald + tailed files to `https://logs-ingest.k8s.syd1.au.unkin.net` (vector-native/HTTPS), publishing to `logs.vm.<host>` subjects - set the source tags the aggregator expects: `.file` on file sources, `.SYSLOG_IDENTIFIER` on journald, for named/nginx/haproxy/glauth/postfix/puppetserver/vault_audit - enable BIND `querylog yes` in `profiles/dns/server.pp` - do puppetserver multiline logback stacktrace join at the edge (stateful) + tail `puppetserver-access.log` **Acceptance** - VM logs appear in `logs.raw` under `logs.vm.*`; the dormant Tier-1/2 transforms populate structured fields; `SELECT count() FROM logs.raw WHERE ...vm...` rising; Vault file-audit reaches the logarchiver archive subject.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/puppet-prod#500
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?