benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
3.5 KiB
3.5 KiB
Deployment (drop-in fit with the logging stack)
This describes how logarchiver slots into the centralized logging stack (argocd-apps #296). The actual argocd manifest swap is a separate later task; this documents the wiring logarchiver is built for so that swap is mechanical.
Where it runs
- Namespace
logging, ServiceAccountdefault(reuses the stack'sVaultAuthdefault, k8s auth mountk8s/au/syd1, roledefault). - Single-replica Deployment (independent JetStream offsets; one archiver is enough — scale by subject-sharding into multiple durables if ever needed).
- Pod label
vector.dev/exclude: "true"so the Vector agent does not scrape logarchiver's own logs (matches the Vector archiver it replaces). - Startup healthcheck against RGW should be lenient (RGW cred propagation is slow), like the Vector archiver.
What it binds to (all already provided by #296)
| Dependency | Wiring |
|---|---|
| NATS | nats://nats.logging.svc.cluster.local:4222, user log-consumer, password from secret nats-auth key consumer_password → env NATS_CONSUMER_PASSWORD. Stream LOGS, durable archiver. |
| S3 (Ceph RGW) | Bucket logs-archive, endpoint https://s3.ceph.unkin.net (path-style, region us-east-1), CA /etc/vault-ca/ca.crt (mount the vault-ca-cert secret). Creds from cephrgw BucketAccess secret logs-archive-s3 via envFrom (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY; optional S3_ENDPOINT/BUCKET_NAME). |
| ClickHouse | clickhouse-logs.logging.svc.cluster.local:9000 (native), database logs, table archive_index, user vector, password from secret clickhouse-credentials key password → env CLICKHOUSE_PASSWORD. |
| Vault GPG engine | Public key: default pubkey_source: file from a mounted armored key (secret/ConfigMap at /etc/logarchiver/pubkey.asc) — no Vault dependency on the hot path. Or pubkey_source: vault with k8s auth to read gpg/keys/logarchive. |
Secrets to seed (Vault KV, reusing the logging/default path)
Nothing new is strictly required if you reuse the existing nats-auth,
logs-archive-s3, and clickhouse-credentials secrets. For the file pubkey,
add an armored public key as a mounted secret (e.g.
kv/kubernetes/namespace/logging/default/logarchiver-pubkey → VaultStaticSecret
→ file mount).
Cross-repo notes
- No new ServiceAccount (reuses
default), so no argocd-apps SA PR is needed for CI; the Woodpecker steps already useserviceAccountName: default. - terraform-vault: only needed if you (a) use
pubkey_source: vaultand thelogging/defaultpolicy doesn't already permitread gpg/keys/logarchive, or (b) want a dedicated operator policy forupdate gpg/decrypt/logarchive. Operators today use their own human Vault tokens forfetch, so this is optional — track as a follow-up, not a blocker. - ClickHouse table: created by the argocd bootstrap Job (embed the output of
logarchiver init-schema --print), not by the service at runtime.
Migrating off the Vector archiver
The Vector archiver binds the same LOGS/archiver durable. To cut over
safely: deploy logarchiver with a distinct durable (e.g.
durable: archiver-canary) and a narrow ARCHIVE_SUBJECTS to validate objects
- index rows land, then repoint it to the
archiverdurable and scale the Vector archiver to zero. logarchiver writes a different object prefix (archive/….ndjson.zst.larc) than Vector (raw/….log.gz), so the two never collide in the bucket.