Replace vector-archiver with logarchiver
ci/woodpecker/pr/vector-test Pipeline failed
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful

The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption.
logarchiver replaces it with a Go service that seals raw logs to S3 as zstd +
OpenPGP objects and indexes each object in ClickHouse (logs.archive_index),
acking JetStream only after the object is stored and indexed.

- Add logarchiver Deployment (image git.unkin.net/unkin/logarchiver:v0.1.0),
  ConfigMap, and dedicated ServiceAccount. Reuses the archiver's NATS
  (log-consumer / durable archiver / ARCHIVE_SUBJECTS=logs.k8s.vault.>), S3
  (logs-archive-s3 BucketAccess), ClickHouse (clickhouse-credentials) and
  vault-ca wiring. Encrypts to the logarchive gpg public key, fetched from the
  gpg engine via k8s auth (role logging_logarchiver, projected vault-audience
  token). ack_wait > batch max_age so messages are not redelivered mid-batch.
- Add logs.archive_index DDL to the clickhouse-schema bootstrap Job (no TTL).
- Remove the vector-archiver Helm release, values and pipeline ConfigMap.

Cross-repo: apply terraform-vault #106 (gpg key + role/policy) before this syncs,
or the pod cannot fetch the public key. Sequencing: apply after argocd-apps #306.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
benvin
2026-07-29 20:33:51 +10:00
parent 96afbcf5e1
commit a96c46fd6e
8 changed files with 202 additions and 133 deletions
-62
View File
@@ -1,62 +0,0 @@
---
# Vector ARCHIVER tier — long-term raw-log backup to S3 (Ceph RGW). Independent
# durable JetStream consumer (`archiver`) so its offsets/lag are fully isolated
# from the ClickHouse transform path (archive lag can never stall ingest — true
# fan-out). Writes RAW, pre-transform events (as they sit in JetStream) as
# gzipped NDJSON, partitioned by subject + date. This is the long-horizon replay
# source beyond JetStream's 3d retention window.
data_dir: /vector-data-dir
api:
enabled: true
address: 0.0.0.0:8686
sources:
js_archive:
type: nats
url: nats://nats.logging.svc.cluster.local:4222
connection_name: vector-archiver
subject: "logs.>"
jetstream:
stream: LOGS
consumer: archiver
auth:
strategy: user_password
user_password:
user: log-consumer
password: ${NATS_CONSUMER_PASSWORD}
decoding:
codec: json
sinks:
s3:
type: aws_s3
inputs:
- js_archive
bucket: logs-archive
endpoint: https://s3.ceph.unkin.net
region: us-east-1
force_path_style: true
tls:
ca_file: /etc/vault-ca/ca.crt
# AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY come from the logs-archive-s3
# Secret (cephrgw-operator) via envFrom on the deployment.
key_prefix: "raw/{{ subject }}/%Y/%m/%d/"
compression: gzip
encoding:
codec: json
framing:
method: newline_delimited
filename_time_format: "%Y%m%dT%H%M%SZ"
filename_append_uuid: true
batch:
max_bytes: 134217728
timeout_secs: 300
buffer:
type: memory
max_events: 5000
when_full: block
# Disabled so slow BucketAccess credential propagation doesn't crash-loop
# the pod; RGW reachability is proven by the operator's own health.
healthcheck:
enabled: false