benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
42 lines
1.6 KiB
Markdown
42 lines
1.6 KiB
Markdown
# `logarchiver fetch`
|
|
|
|
Download, decrypt, and decompress archived objects to plain NDJSON.
|
|
|
|
```sh
|
|
logarchiver fetch [object-key ...] [flags]
|
|
```
|
|
|
|
Objects are selected either by explicit object-key arguments, or — when no keys
|
|
are given — by the same `--subject/--host/--from/--to` query used by
|
|
[`search`](search.md). When `--host/--from/--to` are supplied they ALSO
|
|
re-filter the emitted events to just the matching lines.
|
|
|
|
## Flags
|
|
|
|
| Flag | Meaning |
|
|
|---|---|
|
|
| `-o, --output` | `-` for stdout (default), or a directory to write one NDJSON file per object. |
|
|
| `--subject`/`--host`/`--from`/`--to`/`--limit` | Object selection (same as `search`) when no keys are given; host/time also re-filter emitted lines. |
|
|
|
|
## How decryption works
|
|
|
|
For each object, `fetch` reads the object header to learn which Vault GPG key
|
|
wrapped it, sends only the small wrapped data key to the engine's
|
|
`gpg/decrypt/<name>` endpoint, recovers the data key, and streams the AES-GCM
|
|
frames through local decryption + zstd to emit NDJSON. See the
|
|
[retrieval runbook](retrieval-runbook.md) for the full design and required creds
|
|
(S3 read, internal CA, ambient `VAULT_TOKEN`/`~/.vault-token`).
|
|
|
|
## Examples
|
|
|
|
```sh
|
|
# Straight to stdout by key:
|
|
logarchiver fetch archive/logs.k8s.vault._/2026/07/27/20260727T101500Z-ab12cd34.ndjson.zst.larc -o -
|
|
|
|
# From a query, re-filtered to host db-1 in the last 24h, into ./out:
|
|
logarchiver fetch --subject 'logs.vm.*' --host db-1 --from -24h -o ./out
|
|
```
|
|
|
|
Exit status is non-zero if any selected object fails; per-object errors are
|
|
reported on stderr and the remaining objects are still processed.
|