benvin
a96c46fd6e
The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (logs.archive_index), acking JetStream only after the object is stored and indexed. - Add logarchiver Deployment (image git.unkin.net/unkin/logarchiver:v0.1.0), ConfigMap, and dedicated ServiceAccount. Reuses the archiver's NATS (log-consumer / durable archiver / ARCHIVE_SUBJECTS=logs.k8s.vault.>), S3 (logs-archive-s3 BucketAccess), ClickHouse (clickhouse-credentials) and vault-ca wiring. Encrypts to the logarchive gpg public key, fetched from the gpg engine via k8s auth (role logging_logarchiver, projected vault-audience token). ack_wait > batch max_age so messages are not redelivered mid-batch. - Add logs.archive_index DDL to the clickhouse-schema bootstrap Job (no TTL). - Remove the vector-archiver Helm release, values and pipeline ConfigMap. Cross-repo: apply terraform-vault #106 (gpg key + role/policy) before this syncs, or the pod cannot fetch the public key. Sequencing: apply after argocd-apps #306. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
40 lines
1.2 KiB
YAML
40 lines
1.2 KiB
YAML
---
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
namespace: logging
|
|
|
|
resources:
|
|
- ../../../base/logging
|
|
|
|
helmCharts:
|
|
# Dedicated JetStream NATS cluster — the durable log bus.
|
|
- name: nats
|
|
repo: https://nats-io.github.io/k8s/helm/charts
|
|
version: "2.14.2"
|
|
releaseName: nats
|
|
namespace: logging
|
|
valuesFile: values-nats.yaml
|
|
# Edge agent (DaemonSet): tails every pod's logs, publishes to JetStream.
|
|
- name: vector
|
|
repo: https://helm.vector.dev
|
|
version: "0.57.0"
|
|
releaseName: vector-agent
|
|
namespace: logging
|
|
valuesFile: values-vector-agent.yaml
|
|
# VM ingest (Deployment): HTTP NDJSON front door -> JetStream.
|
|
- name: vector
|
|
repo: https://helm.vector.dev
|
|
version: "0.57.0"
|
|
releaseName: vector-vm-ingest
|
|
namespace: logging
|
|
valuesFile: values-vector-vm-ingest.yaml
|
|
# Transform tier (StatefulSet): JetStream consumer -> shape -> ClickHouse.
|
|
- name: vector
|
|
repo: https://helm.vector.dev
|
|
version: "0.57.0"
|
|
releaseName: vector-aggregator
|
|
namespace: logging
|
|
valuesFile: values-vector-aggregator.yaml
|
|
# Archiver: replaced by the logarchiver Deployment (apps/base/logging).
|