Replace vector-archiver with logarchiver
The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (logs.archive_index), acking JetStream only after the object is stored and indexed. - Add logarchiver Deployment (image git.unkin.net/unkin/logarchiver:v0.1.0), ConfigMap, and dedicated ServiceAccount. Reuses the archiver's NATS (log-consumer / durable archiver / ARCHIVE_SUBJECTS=logs.k8s.vault.>), S3 (logs-archive-s3 BucketAccess), ClickHouse (clickhouse-credentials) and vault-ca wiring. Encrypts to the logarchive gpg public key, fetched from the gpg engine via k8s auth (role logging_logarchiver, projected vault-audience token). ack_wait > batch max_age so messages are not redelivered mid-batch. - Add logs.archive_index DDL to the clickhouse-schema bootstrap Job (no TTL). - Remove the vector-archiver Helm release, values and pipeline ConfigMap. Cross-repo: apply terraform-vault #106 (gpg key + role/policy) before this syncs, or the pod cannot fetch the public key. Sequencing: apply after argocd-apps #306. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
@@ -1,62 +0,0 @@
|
||||
---
|
||||
# Vector ARCHIVER tier — long-term raw-log backup to S3 (Ceph RGW). Independent
|
||||
# durable JetStream consumer (`archiver`) so its offsets/lag are fully isolated
|
||||
# from the ClickHouse transform path (archive lag can never stall ingest — true
|
||||
# fan-out). Writes RAW, pre-transform events (as they sit in JetStream) as
|
||||
# gzipped NDJSON, partitioned by subject + date. This is the long-horizon replay
|
||||
# source beyond JetStream's 3d retention window.
|
||||
data_dir: /vector-data-dir
|
||||
|
||||
api:
|
||||
enabled: true
|
||||
address: 0.0.0.0:8686
|
||||
|
||||
sources:
|
||||
js_archive:
|
||||
type: nats
|
||||
url: nats://nats.logging.svc.cluster.local:4222
|
||||
connection_name: vector-archiver
|
||||
subject: "logs.>"
|
||||
jetstream:
|
||||
stream: LOGS
|
||||
consumer: archiver
|
||||
auth:
|
||||
strategy: user_password
|
||||
user_password:
|
||||
user: log-consumer
|
||||
password: ${NATS_CONSUMER_PASSWORD}
|
||||
decoding:
|
||||
codec: json
|
||||
|
||||
sinks:
|
||||
s3:
|
||||
type: aws_s3
|
||||
inputs:
|
||||
- js_archive
|
||||
bucket: logs-archive
|
||||
endpoint: https://s3.ceph.unkin.net
|
||||
region: us-east-1
|
||||
force_path_style: true
|
||||
tls:
|
||||
ca_file: /etc/vault-ca/ca.crt
|
||||
# AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY come from the logs-archive-s3
|
||||
# Secret (cephrgw-operator) via envFrom on the deployment.
|
||||
key_prefix: "raw/{{ subject }}/%Y/%m/%d/"
|
||||
compression: gzip
|
||||
encoding:
|
||||
codec: json
|
||||
framing:
|
||||
method: newline_delimited
|
||||
filename_time_format: "%Y%m%dT%H%M%SZ"
|
||||
filename_append_uuid: true
|
||||
batch:
|
||||
max_bytes: 134217728
|
||||
timeout_secs: 300
|
||||
buffer:
|
||||
type: memory
|
||||
max_events: 5000
|
||||
when_full: block
|
||||
# Disabled so slow BucketAccess credential propagation doesn't crash-loop
|
||||
# the pod; RGW reachability is proven by the operator's own health.
|
||||
healthcheck:
|
||||
enabled: false
|
||||
Reference in New Issue
Block a user