benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
56 lines
3.5 KiB
Markdown
56 lines
3.5 KiB
Markdown
# Deployment (drop-in fit with the logging stack)
|
|
|
|
This describes how logarchiver slots into the centralized logging stack
|
|
(argocd-apps #296). The actual argocd manifest swap is a **separate later task**;
|
|
this documents the wiring logarchiver is built for so that swap is mechanical.
|
|
|
|
## Where it runs
|
|
|
|
- Namespace **`logging`**, ServiceAccount **`default`** (reuses the stack's
|
|
`VaultAuth` `default`, k8s auth mount `k8s/au/syd1`, role `default`).
|
|
- Single-replica Deployment (independent JetStream offsets; one archiver is
|
|
enough — scale by subject-sharding into multiple durables if ever needed).
|
|
- Pod label **`vector.dev/exclude: "true"`** so the Vector agent does not scrape
|
|
logarchiver's own logs (matches the Vector archiver it replaces).
|
|
- Startup healthcheck against RGW should be lenient (RGW cred propagation is
|
|
slow), like the Vector archiver.
|
|
|
|
## What it binds to (all already provided by #296)
|
|
|
|
| Dependency | Wiring |
|
|
|---|---|
|
|
| **NATS** | `nats://nats.logging.svc.cluster.local:4222`, user `log-consumer`, password from secret **`nats-auth`** key `consumer_password` → env `NATS_CONSUMER_PASSWORD`. Stream `LOGS`, durable `archiver`. |
|
|
| **S3 (Ceph RGW)** | Bucket `logs-archive`, endpoint `https://s3.ceph.unkin.net` (path-style, region `us-east-1`), CA `/etc/vault-ca/ca.crt` (mount the `vault-ca-cert` secret). Creds from cephrgw BucketAccess secret **`logs-archive-s3`** via `envFrom` (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`; optional `S3_ENDPOINT`/`BUCKET_NAME`). |
|
|
| **ClickHouse** | `clickhouse-logs.logging.svc.cluster.local:9000` (native), database `logs`, table `archive_index`, user `vector`, password from secret **`clickhouse-credentials`** key `password` → env `CLICKHOUSE_PASSWORD`. |
|
|
| **Vault GPG engine** | Public key: default `pubkey_source: file` from a mounted armored key (secret/ConfigMap at `/etc/logarchiver/pubkey.asc`) — no Vault dependency on the hot path. Or `pubkey_source: vault` with k8s auth to read `gpg/keys/logarchive`. |
|
|
|
|
## Secrets to seed (Vault KV, reusing the `logging/default` path)
|
|
|
|
Nothing new is strictly required if you reuse the existing `nats-auth`,
|
|
`logs-archive-s3`, and `clickhouse-credentials` secrets. For the file pubkey,
|
|
add an armored public key as a mounted secret (e.g.
|
|
`kv/kubernetes/namespace/logging/default/logarchiver-pubkey` → VaultStaticSecret
|
|
→ file mount).
|
|
|
|
## Cross-repo notes
|
|
|
|
- **No new ServiceAccount** (reuses `default`), so no argocd-apps SA PR is needed
|
|
for CI; the Woodpecker steps already use `serviceAccountName: default`.
|
|
- **terraform-vault:** only needed if you (a) use `pubkey_source: vault` and the
|
|
`logging/default` policy doesn't already permit `read gpg/keys/logarchive`, or
|
|
(b) want a dedicated operator policy for `update gpg/decrypt/logarchive`.
|
|
Operators today use their own human Vault tokens for `fetch`, so this is
|
|
optional — track as a follow-up, not a blocker.
|
|
- **ClickHouse table:** created by the argocd bootstrap Job (embed the output of
|
|
`logarchiver init-schema --print`), not by the service at runtime.
|
|
|
|
## Migrating off the Vector archiver
|
|
|
|
The Vector archiver binds the same `LOGS`/`archiver` durable. To cut over
|
|
safely: deploy logarchiver with a **distinct** durable (e.g.
|
|
`durable: archiver-canary`) and a narrow `ARCHIVE_SUBJECTS` to validate objects
|
|
+ index rows land, then repoint it to the `archiver` durable and scale the
|
|
Vector archiver to zero. logarchiver writes a different object prefix
|
|
(`archive/….ndjson.zst.larc`) than Vector (`raw/….log.gz`), so the two never
|
|
collide in the bucket.
|