benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
3.8 KiB
3.8 KiB
logarchiver
Archive raw logs from the centralized logging NATS JetStream stream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects — and a CLI to search the index and retrieve/decrypt archived logs.
logarchiver replaces the plain Vector archiver leg of the logging stack (argocd-apps #296). Where Vector wrote gzipped NDJSON to Ceph RGW with no index and no encryption, logarchiver adds:
- zstd compression (
github.com/klauspost/compress/zstd); - OpenPGP encryption with a key held in Ben's Vault GPG secrets engine
(
vault-plugin-secrets-gpg) — the private key never leaves Vault; - a searchable ClickHouse index so you can answer "vault logs from host X between dates Y and Z" without scanning S3;
- sink-conditional acks: a batch's JetStream messages are acknowledged only after the object is durably in S3 and indexed — something a stock Vector NATS consumer cannot do.
It is a single Go binary that is both the service (logarchiver run) and the
operator CLI (search / fetch / init-schema).
Quick start
# Service (in-cluster): drains the JetStream 'archiver' consumer to S3 + index.
logarchiver run # defaults suit the logging stack; env supplies secrets
# Operator CLI (laptop, with VAULT_TOKEN / ~/.vault-token and S3/ClickHouse creds):
logarchiver search --subject 'logs.k8s.vault.>' --host node-1 --from -24h
logarchiver fetch --subject 'logs.vm.*' --host db-1 --from -24h -o ./out
logarchiver fetch archive/logs.k8s.vault._/2026/07/27/….ndjson.zst.larc -o -
# Schema (local/dev; in-cluster the argocd bootstrap Job owns this):
logarchiver init-schema # or: init-schema --print to emit the DDL
How it works (short version)
JetStream LOGS / durable "archiver" ──▶ batch by subject (size/count/time)
│ │
│ NDJSON ─▶ zstd ─▶ AES-256-GCM frames
│ │ (random per-object data key)
│ data key ─▶ OpenPGP-encrypt to Vault pubkey
│ ▼
└── ack ONLY after ───────────────── S3 PUT (LARC1 object) + ClickHouse index row
Retrieval sends only the tiny wrapped data key to the Vault GPG engine (which does whole-payload decrypt only), recovers it, and streams the bulk locally. See docs/architecture.md and the retrieval runbook for the full design and the honest account of what the GPG engine supports.
Documentation
- docs/architecture.md — components, data flow, the LARC1 container format, index schema, delivery guarantees.
- docs/retrieval-runbook.md — crypto/decrypt design and step-by-step retrieval, key setup, failure modes.
- docs/deployment.md — drop-in fit with the logging stack (NATS/S3/ClickHouse/Vault wiring, secrets, k8s).
- Subcommands: run · search · fetch · init-schema
- config.example.yaml — every config knob with defaults.
- schema/archive_index.sql — the index DDL.
Build / test / lint
make build # -> dist/logarchiver
make test # go test -race ./...
make lint # golangci-lint run ./...
make rpm # build + package the CLI RPM (nfpm)
make patch|minor|major# tag + push a release (triggers the Woodpecker pipeline)
Releases (on a v* tag): a container image git.unkin.net/unkin/logarchiver
for the service, and a Gitea binary release + rpm-internal RPM for the CLI.