benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
81 lines
3.8 KiB
Markdown
81 lines
3.8 KiB
Markdown
# logarchiver
|
|
|
|
Archive raw logs from the centralized logging NATS JetStream stream to S3 as
|
|
**zstd-compressed, OpenPGP-encrypted, indexed** objects — and a CLI to **search**
|
|
the index and **retrieve/decrypt** archived logs.
|
|
|
|
logarchiver replaces the plain Vector archiver leg of the logging stack
|
|
(argocd-apps #296). Where Vector wrote gzipped NDJSON to Ceph RGW with no index
|
|
and no encryption, logarchiver adds:
|
|
|
|
- **zstd** compression (`github.com/klauspost/compress/zstd`);
|
|
- **OpenPGP encryption** with a key held in Ben's Vault GPG secrets engine
|
|
(`vault-plugin-secrets-gpg`) — the private key never leaves Vault;
|
|
- a **searchable ClickHouse index** so you can answer *"vault logs from host X
|
|
between dates Y and Z"* without scanning S3;
|
|
- **sink-conditional acks**: a batch's JetStream messages are acknowledged only
|
|
after the object is durably in S3 **and** indexed — something a stock Vector
|
|
NATS consumer cannot do.
|
|
|
|
It is a single Go binary that is both the service (`logarchiver run`) and the
|
|
operator CLI (`search` / `fetch` / `init-schema`).
|
|
|
|
## Quick start
|
|
|
|
```sh
|
|
# Service (in-cluster): drains the JetStream 'archiver' consumer to S3 + index.
|
|
logarchiver run # defaults suit the logging stack; env supplies secrets
|
|
|
|
# Operator CLI (laptop, with VAULT_TOKEN / ~/.vault-token and S3/ClickHouse creds):
|
|
logarchiver search --subject 'logs.k8s.vault.>' --host node-1 --from -24h
|
|
logarchiver fetch --subject 'logs.vm.*' --host db-1 --from -24h -o ./out
|
|
logarchiver fetch archive/logs.k8s.vault._/2026/07/27/….ndjson.zst.larc -o -
|
|
|
|
# Schema (local/dev; in-cluster the argocd bootstrap Job owns this):
|
|
logarchiver init-schema # or: init-schema --print to emit the DDL
|
|
```
|
|
|
|
## How it works (short version)
|
|
|
|
```
|
|
JetStream LOGS / durable "archiver" ──▶ batch by subject (size/count/time)
|
|
│ │
|
|
│ NDJSON ─▶ zstd ─▶ AES-256-GCM frames
|
|
│ │ (random per-object data key)
|
|
│ data key ─▶ OpenPGP-encrypt to Vault pubkey
|
|
│ ▼
|
|
└── ack ONLY after ───────────────── S3 PUT (LARC1 object) + ClickHouse index row
|
|
```
|
|
|
|
Retrieval sends only the tiny wrapped data key to the Vault GPG engine (which
|
|
does whole-payload decrypt only), recovers it, and streams the bulk locally.
|
|
See [docs/architecture.md](docs/architecture.md) and the
|
|
[retrieval runbook](docs/retrieval-runbook.md) for the full design and the
|
|
honest account of what the GPG engine supports.
|
|
|
|
## Documentation
|
|
|
|
- [docs/architecture.md](docs/architecture.md) — components, data flow, the
|
|
LARC1 container format, index schema, delivery guarantees.
|
|
- [docs/retrieval-runbook.md](docs/retrieval-runbook.md) — crypto/decrypt design
|
|
and step-by-step retrieval, key setup, failure modes.
|
|
- [docs/deployment.md](docs/deployment.md) — drop-in fit with the logging stack
|
|
(NATS/S3/ClickHouse/Vault wiring, secrets, k8s).
|
|
- Subcommands: [run](docs/run.md) · [search](docs/search.md) ·
|
|
[fetch](docs/fetch.md) · [init-schema](docs/init-schema.md)
|
|
- [config.example.yaml](config.example.yaml) — every config knob with defaults.
|
|
- [schema/archive_index.sql](schema/archive_index.sql) — the index DDL.
|
|
|
|
## Build / test / lint
|
|
|
|
```sh
|
|
make build # -> dist/logarchiver
|
|
make test # go test -race ./...
|
|
make lint # golangci-lint run ./...
|
|
make rpm # build + package the CLI RPM (nfpm)
|
|
make patch|minor|major# tag + push a release (triggers the Woodpecker pipeline)
|
|
```
|
|
|
|
Releases (on a `v*` tag): a container image `git.unkin.net/unkin/logarchiver`
|
|
for the service, and a Gitea binary release + `rpm-internal` RPM for the CLI.
|