Replace vector-archiver with logarchiver (#308)

## Why The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed. ## Changes - Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring. - Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch. - Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`). - Remove the vector-archiver Helm release, values and pipeline ConfigMap. Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #308 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-29 21:07:20 +10:00
parent 96afbcf5e1
commit 9c10b9096a
9 changed files with 214 additions and 135 deletions
@@ -1,62 +0,0 @@
---
-# Vector ARCHIVER tier — long-term raw-log backup to S3 (Ceph RGW). Independent
-# durable JetStream consumer (`archiver`) so its offsets/lag are fully isolated
-# from the ClickHouse transform path (archive lag can never stall ingest — true
-# fan-out). Writes RAW, pre-transform events (as they sit in JetStream) as
-# gzipped NDJSON, partitioned by subject + date. This is the long-horizon replay
-# source beyond JetStream's 3d retention window.
-data_dir: /vector-data-dir
-
-api:
-  enabled: true
-  address: 0.0.0.0:8686
-
-sources:
-  js_archive:
-    type: nats
-    url: nats://nats.logging.svc.cluster.local:4222
-    connection_name: vector-archiver
-    subject: "logs.>"
-    jetstream:
-      stream: LOGS
-      consumer: archiver
-    auth:
-      strategy: user_password
-      user_password:
-        user: log-consumer
-        password: ${NATS_CONSUMER_PASSWORD}
-    decoding:
-      codec: json
-
-sinks:
-  s3:
-    type: aws_s3
-    inputs:
-      - js_archive
-    bucket: logs-archive
-    endpoint: https://s3.ceph.unkin.net
-    region: us-east-1
-    force_path_style: true
-    tls:
-      ca_file: /etc/vault-ca/ca.crt
-    # AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY come from the logs-archive-s3
-    # Secret (cephrgw-operator) via envFrom on the deployment.
-    key_prefix: "raw/{{ subject }}/%Y/%m/%d/"
-    compression: gzip
-    encoding:
-      codec: json
-    framing:
-      method: newline_delimited
-    filename_time_format: "%Y%m%dT%H%M%SZ"
-    filename_append_uuid: true
-    batch:
-      max_bytes: 134217728
-      timeout_secs: 300
-    buffer:
-      type: memory
-      max_events: 5000
-      when_full: block
-    # Disabled so slow BucketAccess credential propagation doesn't crash-loop
-    # the pod; RGW reachability is proven by the operator's own health.
-    healthcheck:
-      enabled: false