Initial implementation: NATS->S3 archiver + search/retrieve CLI #1

2026-07-27T22:12:21+10:00

unkinben commented

2026-07-27 22:12:21 +10:00

Why

The centralized logging stack (argocd-apps #296) archives raw logs with a plain Vector consumer (gzip NDJSON to Ceph RGW) — no index, no encryption. logarchiver replaces that archiver leg with the capabilities that outgrew Vector: zstd compression, OpenPGP encryption keyed from Ben's Vault GPG secrets engine (private key never leaves Vault), a searchable ClickHouse index ("vault logs from host X between dates Y and Z"), and sink-conditional acks — a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed, which a stock Vector NATS consumer cannot do. It is one Go binary that is both the service and the operator CLI. The argocd manifest swap is a separate later task; this is designed for drop-in fit.

What changed

Service (logarchiver run): durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>); per-subject batching by size/count/time; NDJSON → zstd → encrypt → S3 PUT → ClickHouse index row → ack; Nak + redelivery on any failure; Prometheus metrics, structured slog, graceful drain.
Crypto (container LARC1): the Vault GPG engine only does whole-payload decrypt, so objects use a wrapped-DEK envelope — bulk is AES-256-GCM framed under a random data key and only that key is OpenPGP-encrypted to the engine's public key. Retrieval round-trips just the tiny wrapped key regardless of object size. Public key from the engine or a mounted file (configurable); fingerprint recorded per object; periodic refresh for rotation. The honest account of engine support and a possible follow-up enhancement are in docs/retrieval-runbook.md.
CLI: search (query the index), fetch (download + engine-decrypt + unzstd → NDJSON, optionally re-filtered by host/time), init-schema (create/print the DDL), cobra completion.
Index: ClickHouse logs.archive_index, one row per object (subject, hosts[], min/max ts, counts, sizes, zstd/pgp metadata, key fingerprint); DDL shipped as schema/archive_index.sql + internal/index/ddl.go.
Config via file+env (secrets from env), all external boundaries behind interfaces.
Tests: config, batching boundaries, host/subject extraction, crypto roundtrip with a generated test key (+ tamper/empty/multi-frame), ack-after-persist with fakes (archiver + runner ack-vs-nak), search-SQL building. go build/go vet/go test -race clean; golangci-lint v2 (docker) clean.
Release: .woodpecker/ build/test/pre-commit on PR; on v* tag a container image git.unkin.net/unkin/logarchiver (buildx) plus a Gitea binary release + rpm-internal RPM (nfpm) for the CLI. All steps set k8s resources + serviceAccountName: default (no new SA required).
Docs: README + per-subcommand docs + architecture + retrieval runbook + deployment drop-in; config.example.yaml.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why The centralized logging stack (argocd-apps #296) archives raw logs with a plain Vector consumer (gzip NDJSON to Ceph RGW) — no index, no encryption. logarchiver replaces that archiver leg with the capabilities that outgrew Vector: zstd compression, OpenPGP encryption keyed from Ben's Vault GPG secrets engine (private key never leaves Vault), a searchable ClickHouse index ("vault logs from host X between dates Y and Z"), and sink-conditional acks — a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed, which a stock Vector NATS consumer cannot do. It is one Go binary that is both the service and the operator CLI. The argocd manifest swap is a separate later task; this is designed for drop-in fit. ## What changed - **Service (`logarchiver run`):** durable JetStream pull consumer (stream `LOGS`, durable `archiver`, subject filter default `logs.k8s.vault.>`); per-subject batching by size/count/time; NDJSON → zstd → encrypt → S3 PUT → ClickHouse index row → ack; Nak + redelivery on any failure; Prometheus metrics, structured slog, graceful drain. - **Crypto (container `LARC1`):** the Vault GPG engine only does whole-payload decrypt, so objects use a wrapped-DEK envelope — bulk is AES-256-GCM framed under a random data key and only that key is OpenPGP-encrypted to the engine's public key. Retrieval round-trips just the tiny wrapped key regardless of object size. Public key from the engine or a mounted file (configurable); fingerprint recorded per object; periodic refresh for rotation. The honest account of engine support and a possible follow-up enhancement are in `docs/retrieval-runbook.md`. - **CLI:** `search` (query the index), `fetch` (download + engine-decrypt + unzstd → NDJSON, optionally re-filtered by host/time), `init-schema` (create/print the DDL), cobra `completion`. - **Index:** ClickHouse `logs.archive_index`, one row per object (subject, hosts[], min/max ts, counts, sizes, zstd/pgp metadata, key fingerprint); DDL shipped as `schema/archive_index.sql` + `internal/index/ddl.go`. - **Config** via file+env (secrets from env), all external boundaries behind interfaces. - **Tests:** config, batching boundaries, host/subject extraction, crypto roundtrip with a generated test key (+ tamper/empty/multi-frame), ack-after-persist with fakes (archiver + runner ack-vs-nak), search-SQL building. `go build`/`go vet`/`go test -race` clean; golangci-lint v2 (docker) clean. - **Release:** `.woodpecker/` build/test/pre-commit on PR; on `v*` tag a container image `git.unkin.net/unkin/logarchiver` (buildx) plus a Gitea binary release + `rpm-internal` RPM (nfpm) for the CLI. All steps set k8s resources + `serviceAccountName: default` (no new SA required). - **Docs:** README + per-subcommand docs + architecture + retrieval runbook + deployment drop-in; `config.example.yaml`. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

unkinben added 1 commit 2026-07-27 23:22:41 +10:00

Initial implementation: NATS->S3 archiver + search/retrieve CLI

ci/woodpecker/pr/build Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

c05ccfcb5d

logarchiver replaces the plain Vector archiver leg of the centralized
logging stack (argocd-apps #296) with a Go service that archives raw logs
from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed
objects, plus an operator CLI to search the index and retrieve/decrypt
archived logs. It adds the things that outgrew Vector: zstd compression,
encryption keyed from Ben's Vault GPG secrets engine, a searchable
ClickHouse index, and sink-conditional acks (a batch is acknowledged to
JetStream only after the object is durably in S3 AND indexed).

Service (`logarchiver run`):
- Durable JetStream pull consumer (stream LOGS, durable archiver, subject
  filter default logs.k8s.vault.>), explicit acks, independent offsets.
- Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3
  PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and
  redelivered, so nothing is lost on a sink outage.
- Encryption is a wrapped-DEK envelope (container LARC1): the bulk is
  AES-256-GCM framed under a random data key, and only that 32-byte key is
  OpenPGP-encrypted to the engine's public key. This is because the Vault
  GPG engine does whole-payload decrypt only; retrieval round-trips just the
  tiny wrapped key regardless of object size. Public key fetched from the
  engine or a mounted file (configurable); key fingerprint recorded per
  object; periodic pubkey refresh for rotation.
- Prometheus metrics, structured slog, graceful drain on shutdown.

CLI:
- `search` queries the index (subject/host/time) and lists matching objects.
- `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits
  NDJSON (optionally re-filtered by host/time).
- `init-schema` creates/prints the ClickHouse archive_index DDL.
- cobra `completion` subcommands.

Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/
ClickHouse/Vault) behind interfaces with unit tests (config, batching,
host/subject extraction, crypto roundtrip with a test key, ack-after-persist
with fakes, search query building). go build/vet/test -race clean;
golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v*
tag a container image plus a Gitea binary release + rpm-internal RPM. Docs
per subcommand + architecture + retrieval runbook + deployment drop-in.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

unkinben force-pushed benvin/logarchiver-initial from 5ec89b0028 to c05ccfcb5d

2026-07-27 23:22:41 +10:00

Compare

benvin merged commit 06e057ece3 into main

2026-07-28 07:22:20 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/logarchiver#1