benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
41 lines
1.7 KiB
Markdown
41 lines
1.7 KiB
Markdown
# `logarchiver run`
|
|
|
|
Runs the archiver service: binds the JetStream pull consumer and drains it to
|
|
S3 + the ClickHouse index, acking only after each object is durably persisted.
|
|
|
|
```sh
|
|
logarchiver run [-c config.yaml]
|
|
```
|
|
|
|
## Behaviour
|
|
|
|
- Creates/updates the durable consumer (`nats.stream` / `nats.durable`) with the
|
|
configured subject filters, explicit acks, and `nats.ack_wait`.
|
|
- Batches events per subject and flushes on `batch.max_bytes` / `max_events` /
|
|
`max_age`. Each flush seals a LARC1 object, PUTs it to S3, writes one index
|
|
row, then acks the batch. On any failure the batch is Nak'd for redelivery.
|
|
- Loads the OpenPGP public key from `crypto.pubkey_source` (`file` or `vault`)
|
|
and refreshes it every `crypto.refresh_interval`.
|
|
- Serves Prometheus metrics and `/healthz` on `metrics.address` (default `:9090`)
|
|
when `metrics.enabled`.
|
|
- Handles SIGINT/SIGTERM: stops fetching, drains open batches (bounded), exits.
|
|
|
|
## Key env vars
|
|
|
|
| Env | Purpose |
|
|
|---|---|
|
|
| `NATS_CONSUMER_PASSWORD` | NATS `log-consumer` password (nats-auth secret). |
|
|
| `ARCHIVE_SUBJECTS` | Space-separated subject filters (overrides `nats.subjects`). |
|
|
| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | S3 creds (cephrgw `logs-archive-s3`). |
|
|
| `S3_ENDPOINT` / `BUCKET_NAME` | Optional S3 endpoint/bucket from the cephrgw secret. |
|
|
| `CLICKHOUSE_PASSWORD` | ClickHouse `vector` user password. |
|
|
| `VAULT_ADDR` | Vault address when `pubkey_source: vault`. |
|
|
| `LOGARCHIVER_CONFIG` | Config file path (same as `-c`). |
|
|
|
|
## Metrics
|
|
|
|
`logarchiver_objects_stored_total`, `_events_archived_total`,
|
|
`_raw_bytes_total`, `_stored_bytes_total`, `_store_failures_total`,
|
|
`_index_failures_total`, `_messages_fetched_total`, `_acks_total`,
|
|
`_batches_flushed_total{trigger}`, `logarchiver_pending_events`.
|