unkinben
10020033d9
Stand up a centralized logging estate that captures ALL logs from k8s pods and (via a reachable ingestion endpoint) puppet-managed VMs, storing them in ClickHouse for query/retention. Metrics already live in VictoriaMetrics; this adds the logs pillar under a dedicated `logging` ArgoCD project. Deploy the Altinity clickhouse-operator (clickhouse-system) and a single-shard ClickHouseInstallation (logging) on cephrbd-fast-delete with a MergeTree logs.raw table (30d TTL) bootstrapped by an idempotent PostSync Job. Deploy Vector as an explicit two tiers: - Edge (thin): a DaemonSet tails every node's pod logs and forwards over the Vector-native protocol to the aggregator; no parsing at the edge. Future VM agents follow the same thin pattern. - Aggregator (brain): HA StatefulSet that is the sole ClickHouse writer, holds the only ClickHouse credentials, owns all transforms, batches into few fat inserts (avoid too-many-parts), and buffers to disk (PVC) to ride out a ClickHouse outage. Its pipeline is a single source-of-truth config validated by `vector test` in CI; per-app pipelines become aggregator-only changes. Expose the VM ingestion endpoint at logs-ingest.k8s.syd1.au.unkin.net via the internal Traefik gateway (cert-manager + external-dns), routing to the aggregator's HTTP source so puppet VMs can reach it over TLS. Source ClickHouse credentials from Vault via the existing VaultStaticSecret pattern (templated k8s auth policy already grants the logging namespace); password hash never lands in git. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
45 lines
1.4 KiB
YAML
45 lines
1.4 KiB
YAML
---
|
|
# Log ingestion endpoint for puppet-managed VMs (and any non-k8s client).
|
|
# Reuses the internal Traefik gateway + cert-manager + external-dns pattern so
|
|
# VMs reach the Vector aggregator's HTTP source over TLS at a DNS name they can
|
|
# resolve. The puppet-side Vector rollout ships NDJSON to
|
|
# https://logs-ingest.k8s.syd1.au.unkin.net/ (a later task).
|
|
apiVersion: gateway.networking.k8s.io/v1
|
|
kind: Gateway
|
|
metadata:
|
|
name: logs-ingest
|
|
namespace: logging
|
|
labels:
|
|
app.kubernetes.io/name: vector-aggregator
|
|
app.kubernetes.io/component: ingest
|
|
traefik.io/instance: internal
|
|
annotations:
|
|
cert-manager.io/cluster-issuer: vault-issuer
|
|
cert-manager.io/common-name: logs-ingest.k8s.syd1.au.unkin.net
|
|
cert-manager.io/private-key-size: "4096"
|
|
external-dns.alpha.kubernetes.io/hostname: logs-ingest.k8s.syd1.au.unkin.net
|
|
external-dns.alpha.kubernetes.io/target: 198.18.200.4
|
|
spec:
|
|
gatewayClassName: traefik-internal
|
|
listeners:
|
|
- name: http
|
|
port: 80
|
|
protocol: HTTP
|
|
hostname: logs-ingest.k8s.syd1.au.unkin.net
|
|
allowedRoutes:
|
|
namespaces:
|
|
from: Same
|
|
- name: https
|
|
port: 443
|
|
protocol: HTTPS
|
|
hostname: logs-ingest.k8s.syd1.au.unkin.net
|
|
allowedRoutes:
|
|
namespaces:
|
|
from: Same
|
|
tls:
|
|
mode: Terminate
|
|
certificateRefs:
|
|
- group: ""
|
|
kind: Secret
|
|
name: logs-ingest-tls
|