Replace vector-archiver with logarchiver #308

Merged
benvin merged 2 commits from benvin/logarchiver-swap into main 2026-07-29 21:07:20 +10:00

2 Commits

Author SHA1 Message Date
benvin 130fea71a1 Pin logarchiver S3 endpoint; drop archiver from vector-test
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
- Pin the RGW endpoint/bucket in the logarchiver config to the proven
  s3.ceph.unkin.net (ignore the secret's S3_ENDPOINT/BUCKET_NAME; AWS creds still
  come from the secret) to match the Vector archiver exactly.
- Remove the deleted vector/archiver.yaml from the vector-test validate step.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-29 20:36:32 +10:00
benvin a96c46fd6e Replace vector-archiver with logarchiver
ci/woodpecker/pr/vector-test Pipeline failed
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption.
logarchiver replaces it with a Go service that seals raw logs to S3 as zstd +
OpenPGP objects and indexes each object in ClickHouse (logs.archive_index),
acking JetStream only after the object is stored and indexed.

- Add logarchiver Deployment (image git.unkin.net/unkin/logarchiver:v0.1.0),
  ConfigMap, and dedicated ServiceAccount. Reuses the archiver's NATS
  (log-consumer / durable archiver / ARCHIVE_SUBJECTS=logs.k8s.vault.>), S3
  (logs-archive-s3 BucketAccess), ClickHouse (clickhouse-credentials) and
  vault-ca wiring. Encrypts to the logarchive gpg public key, fetched from the
  gpg engine via k8s auth (role logging_logarchiver, projected vault-audience
  token). ack_wait > batch max_age so messages are not redelivered mid-batch.
- Add logs.archive_index DDL to the clickhouse-schema bootstrap Job (no TTL).
- Remove the vector-archiver Helm release, values and pipeline ConfigMap.

Cross-repo: apply terraform-vault #106 (gpg key + role/policy) before this syncs,
or the pod cannot fetch the public key. Sequencing: apply after argocd-apps #306.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-29 20:33:51 +10:00