logarchiver/docs/retrieval-runbook.md

# Retrieval runbook & crypto design

This document is the honest account of how encryption and retrieval work, what
the Vault GPG engine actually supports, and the operational steps to get plain
NDJSON back out of an archived object.

## What the Vault GPG engine supports (and doesn't)

logarchiver encrypts to a key in Ben's `vault-plugin-secrets-gpg` engine
(transit-style OpenPGP, mounted at `gpg/`). Relevant endpoints:

- `GET  gpg/keys/<name>` → returns `data.public_key` (armored) and
  `data.fingerprint` (40-hex uppercase, no spaces). Used to encrypt locally.
- `POST gpg/decrypt/<name>` with `{ "ciphertext": "<base64 of a whole OpenPGP
  message>" }` → returns `{ "plaintext": "<base64>" }`.

**Crucial limitation:** the decrypt endpoint does **whole-payload inline
decryption only**. It reads the entire OpenPGP message and returns the entire
plaintext. There is **no** session-key (PKESK) extraction, **no** chunking, and
**no** streaming. The whole ciphertext must fit in a base64 JSON request body
(Vault's default `max_request_size` is 32 MiB), and the whole plaintext comes
back base64 in the response. So you cannot feed a large archive object straight
to the engine, and you cannot ask it to decrypt only the session key.

## The design logarchiver chose

Rather than bound object sizes to the engine's request limit, logarchiver does
hybrid encryption itself and sends the engine only a tiny key blob:

1. **Encrypt (service, local):** generate a random 256-bit **DEK**; encrypt the
   zstd-compressed NDJSON locally with AES-256-GCM in frames; OpenPGP-encrypt
   just the 32-byte DEK to the engine's **public** key. The wrapped DEK is a
   small standard OpenPGP message. The private key is never present.
2. **Decrypt (CLI, retrieval):** read the object header, send **only the wrapped
   DEK** (a few hundred bytes) to `gpg/decrypt/<name>`, get the DEK back, then
   stream-decrypt the AES-GCM frames locally and zstd-decompress to NDJSON.

The Vault round-trip is tiny and constant regardless of object size, the engine
needs no changes, and the private key stays in Vault. See
[architecture.md](architecture.md#the-larc1-container-format) for the container
layout.

> **Possible engine enhancement (follow-up, not required):** if the engine ever
> grows a "decrypt session key only / return PKESK plaintext" operation, objects
> could be plain standard OpenPGP messages while retaining tiny Vault
> round-trips. logarchiver's self-managed DEK envelope already achieves the
> operational goal without it, so this is optional.

## One-time key setup

Create the archive key in the engine (if it doesn't already exist) and export
the public key for the service:

```sh
export VAULT_ADDR=https://vault.service.consul:8200

# Create the key (idempotent; rsa-3072 default, or ed25519).
vault write gpg/keys/logarchive identity="logarchiver <ops@unkin.net>" algorithm=ed25519

# Fingerprint the service records per object (sanity check):
vault read -field=fingerprint gpg/keys/logarchive

# Export the armored public key for the service's file-mounted pubkey source:
vault read -field=public_key gpg/keys/logarchive > pubkey.asc
```

In k8s the service can instead read the public key directly from Vault
(`pubkey_source: vault`, k8s auth) — no mounted file needed. The file source is
the default because it removes any Vault dependency from the hot path.

## Retrieve archived logs

The CLI needs: S3 read creds (`AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` for the
`logs-archive` bucket), the internal CA (`s3.ca_file`), and a Vault token with
decrypt on the key (ambient `VAULT_TOKEN` or `~/.vault-token`, exactly like
`passv`).

```sh
export VAULT_ADDR=https://vault.service.consul:8200
vault login ...            # or have ~/.vault-token
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...

# 1. Find objects via the index.
logarchiver search --subject 'logs.k8s.vault.>' --host node-1 \
  --from 2026-07-01 --to 2026-07-27

# 2a. Retrieve by the object keys search printed, to stdout:
logarchiver fetch <object-key> [<object-key> ...] -o -

# 2b. Or retrieve straight from a query, re-filtered to just the matching
#     host/time events, into a directory (one NDJSON file per object):
logarchiver fetch --subject 'logs.vm.*' --host db-1 --from -24h -o ./out
```

`fetch` downloads each object, reads its header to learn which Vault key wrapped
it (`key_name`), decrypts the wrapped DEK via the engine, streams the frames
through local AES-GCM + zstd, and emits NDJSON. When `--host`/`--from`/`--to`
are supplied they also re-filter the emitted lines (events lacking a parseable
timestamp are kept, never silently dropped on time grounds).

## Failure modes

| Symptom | Likely cause | Action |
|---|---|---|
| `decrypt via gpg/decrypt/<name>: permission denied` | Token lacks the decrypt policy | Grant `update` on `gpg/decrypt/<name>` (and `read` on `gpg/keys/<name>`). |
| `unwrap dek: … no plaintext` | Wrong key name / key rotated below min_decryption_version | Confirm object `key_name`; the engine tries versions ≥ min_decryption_version. |
| `bad magic: not a logarchiver (LARC1) object` | Fetching a non-logarchiver object (e.g. an old Vector `.log.gz`) | Use the right prefix; Vector objects are plain gzip, decrypt them with gzip. |
| `decrypt frame N: cipher: message authentication failed` | Object corruption/tampering | Object integrity is broken; re-archive from JetStream if still within retention. |
| `pubkey fingerprint mismatch` (service) | Armored pubkey doesn't match the engine's reported fingerprint | Re-export the pubkey; the service refuses to encrypt to a mismatched key. |

## Key rotation

Rotating the engine key (`vault write gpg/keys/logarchive/rotate ...`) starts
encrypting new objects to the new version; the service picks up the new public
key within `crypto.refresh_interval` (default 1h) or on restart. Old objects
remain decryptable because the engine tries every key version from
`min_decryption_version` up. Each object records the fingerprint it was sealed
with, so you can always tell which key version applies.