logarchiver

Archive raw logs from the centralized logging NATS JetStream stream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects — and a CLI to search the index and retrieve/decrypt archived logs.

logarchiver replaces the plain Vector archiver leg of the logging stack (argocd-apps #296). Where Vector wrote gzipped NDJSON to Ceph RGW with no index and no encryption, logarchiver adds:

• zstd compression (github.com/klauspost/compress/zstd);
• OpenPGP encryption with a key held in Ben's Vault GPG secrets engine (vault-plugin-secrets-gpg) — the private key never leaves Vault;
• a searchable ClickHouse index so you can answer "vault logs from host X between dates Y and Z" without scanning S3;
• sink-conditional acks: a batch's JetStream messages are acknowledged only after the object is durably in S3 and indexed — something a stock Vector NATS consumer cannot do.

It is a single Go binary that is both the service (logarchiver run) and the operator CLI (search / fetch / init-schema).

Quick start

# Service (in-cluster): drains the JetStream 'archiver' consumer to S3 + index.
logarchiver run                       # defaults suit the logging stack; env supplies secrets

# Operator CLI (laptop, with VAULT_TOKEN / ~/.vault-token and S3/ClickHouse creds):
logarchiver search --subject 'logs.k8s.vault.>' --host node-1 --from -24h
logarchiver fetch  --subject 'logs.vm.*' --host db-1 --from -24h -o ./out
logarchiver fetch  archive/logs.k8s.vault._/2026/07/27/…​.ndjson.zst.larc -o -

# Schema (local/dev; in-cluster the argocd bootstrap Job owns this):
logarchiver init-schema            # or: init-schema --print  to emit the DDL

How it works (short version)

JetStream LOGS / durable "archiver"  ──▶  batch by subject (size/count/time)
   │                                            │
   │                                       NDJSON ─▶ zstd ─▶ AES-256-GCM frames
   │                                            │        (random per-object data key)
   │                                       data key ─▶ OpenPGP-encrypt to Vault pubkey
   │                                            ▼
   └── ack ONLY after ─────────────────  S3 PUT (LARC1 object)  +  ClickHouse index row

Retrieval sends only the tiny wrapped data key to the Vault GPG engine (which does whole-payload decrypt only), recovers it, and streams the bulk locally. See docs/architecture.md and the retrieval runbook for the full design and the honest account of what the GPG engine supports.

Documentation

• docs/architecture.md — components, data flow, the LARC1 container format, index schema, delivery guarantees.
• docs/retrieval-runbook.md — crypto/decrypt design and step-by-step retrieval, key setup, failure modes.
• docs/deployment.md — drop-in fit with the logging stack (NATS/S3/ClickHouse/Vault wiring, secrets, k8s).
• Subcommands: run · search · fetch · init-schema
• config.example.yaml — every config knob with defaults.
• schema/archive_index.sql — the index DDL.

Build / test / lint

make build            # -> dist/logarchiver
make test             # go test -race ./...
make lint             # golangci-lint run ./...
make rpm              # build + package the CLI RPM (nfpm)
make patch|minor|major# tag + push a release (triggers the Woodpecker pipeline)

Releases (on a v* tag): a container image git.unkin.net/unkin/logarchiver for the service, and a Gitea binary release + rpm-internal RPM for the CLI.