# logarchiver

Archive raw logs from the centralized logging NATS JetStream stream to S3 as
**zstd-compressed, OpenPGP-encrypted, indexed** objects — and a CLI to **search**
the index and **retrieve/decrypt** archived logs.

logarchiver replaces the plain Vector archiver leg of the logging stack
(argocd-apps #296). Where Vector wrote gzipped NDJSON to Ceph RGW with no index
and no encryption, logarchiver adds:

- **zstd** compression (`github.com/klauspost/compress/zstd`);
- **OpenPGP encryption** with a key held in Ben's Vault GPG secrets engine
  (`vault-plugin-secrets-gpg`) — the private key never leaves Vault;
- a **searchable ClickHouse index** so you can answer *"vault logs from host X
  between dates Y and Z"* without scanning S3;
- **sink-conditional acks**: a batch's JetStream messages are acknowledged only
  after the object is durably in S3 **and** indexed — something a stock Vector
  NATS consumer cannot do.

It is a single Go binary that is both the service (`logarchiver run`) and the
operator CLI (`search` / `fetch` / `init-schema`).

## Quick start

```sh
# Service (in-cluster): drains the JetStream 'archiver' consumer to S3 + index.
logarchiver run                       # defaults suit the logging stack; env supplies secrets

# Operator CLI (laptop, with VAULT_TOKEN / ~/.vault-token and S3/ClickHouse creds):
logarchiver search --subject 'logs.k8s.vault.>' --host node-1 --from -24h
logarchiver fetch  --subject 'logs.vm.*' --host db-1 --from -24h -o ./out
logarchiver fetch  archive/logs.k8s.vault._/2026/07/27/…​.ndjson.zst.larc -o -

# Schema (local/dev; in-cluster the argocd bootstrap Job owns this):
logarchiver init-schema            # or: init-schema --print  to emit the DDL
```

## How it works (short version)

```
JetStream LOGS / durable "archiver"  ──▶  batch by subject (size/count/time)
   │                                            │
   │                                       NDJSON ─▶ zstd ─▶ AES-256-GCM frames
   │                                            │        (random per-object data key)
   │                                       data key ─▶ OpenPGP-encrypt to Vault pubkey
   │                                            ▼
   └── ack ONLY after ─────────────────  S3 PUT (LARC1 object)  +  ClickHouse index row
```

Retrieval sends only the tiny wrapped data key to the Vault GPG engine (which
does whole-payload decrypt only), recovers it, and streams the bulk locally.
See [docs/architecture.md](docs/architecture.md) and the
[retrieval runbook](docs/retrieval-runbook.md) for the full design and the
honest account of what the GPG engine supports.

## Documentation

- [docs/architecture.md](docs/architecture.md) — components, data flow, the
  LARC1 container format, index schema, delivery guarantees.
- [docs/retrieval-runbook.md](docs/retrieval-runbook.md) — crypto/decrypt design
  and step-by-step retrieval, key setup, failure modes.
- [docs/deployment.md](docs/deployment.md) — drop-in fit with the logging stack
  (NATS/S3/ClickHouse/Vault wiring, secrets, k8s).
- Subcommands: [run](docs/run.md) · [search](docs/search.md) ·
  [fetch](docs/fetch.md) · [init-schema](docs/init-schema.md)
- [config.example.yaml](config.example.yaml) — every config knob with defaults.
- [schema/archive_index.sql](schema/archive_index.sql) — the index DDL.

## Build / test / lint

```sh
make build            # -> dist/logarchiver
make test             # go test -race ./...
make lint             # golangci-lint run ./...
make rpm              # build + package the CLI RPM (nfpm)
make patch|minor|major# tag + push a release (triggers the Woodpecker pipeline)
```

Releases (on a `v*` tag): a container image `git.unkin.net/unkin/logarchiver`
for the service, and a Gitea binary release + `rpm-internal` RPM for the CLI.