unkinben
9c10b9096a
## Why The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed. ## Changes - Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring. - Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch. - Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`). - Remove the vector-archiver Helm release, values and pipeline ConfigMap. Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged). https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv --------- Co-authored-by: benvin <neotheo@gmail.com> Reviewed-on: #308 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
41 lines
1.2 KiB
YAML
41 lines
1.2 KiB
YAML
---
|
|
# logarchiver non-secret config. Secrets (NATS/S3/ClickHouse creds) and the
|
|
# subject filter come from env; everything else uses the binary's built-in
|
|
# defaults, which already target this stack. ack_wait MUST exceed batch.max_age
|
|
# so unacked messages in an open batch are not redelivered mid-batch.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: logarchiver-config
|
|
namespace: logging
|
|
data:
|
|
config.yaml: |
|
|
nats:
|
|
ack_wait: 5m
|
|
fetch_batch: 512
|
|
batch:
|
|
max_bytes: 67108864
|
|
max_events: 200000
|
|
max_age: 2m
|
|
# Pin the proven RGW endpoint/bucket; ignore the secret's S3_ENDPOINT/BUCKET_NAME
|
|
# (AWS creds still come from the secret env). endpoint_env/bucket_env off.
|
|
s3:
|
|
endpoint: "https://s3.ceph.unkin.net"
|
|
bucket: "logs-archive"
|
|
region: "us-east-1"
|
|
path_style: true
|
|
ca_file: /etc/vault-ca/ca.crt
|
|
endpoint_env: ""
|
|
bucket_env: ""
|
|
crypto:
|
|
key_name: logarchive
|
|
pubkey_source: vault
|
|
vault:
|
|
address: "https://vault.service.consul:8200"
|
|
mount: gpg
|
|
auth_method: kubernetes
|
|
k8s_mount: k8s/au/syd1
|
|
k8s_role: logging_logarchiver
|
|
k8s_jwt_path: /var/run/secrets/vault/token
|
|
ca_file: /etc/vault-ca/ca.crt
|