logarchiver/docs/retrieval-runbook.md

Retrieval runbook & crypto design

This document is the honest account of how encryption and retrieval work, what the Vault GPG engine actually supports, and the operational steps to get plain NDJSON back out of an archived object.

What the Vault GPG engine supports (and doesn't)

logarchiver encrypts to a key in Ben's vault-plugin-secrets-gpg engine (transit-style OpenPGP, mounted at gpg/). Relevant endpoints:

• GET gpg/keys/<name> → returns data.public_key (armored) and data.fingerprint (40-hex uppercase, no spaces). Used to encrypt locally.
• POST gpg/decrypt/<name> with { "ciphertext": "<base64 of a whole OpenPGP message>" } → returns { "plaintext": "<base64>" }.

Crucial limitation: the decrypt endpoint does whole-payload inline decryption only. It reads the entire OpenPGP message and returns the entire plaintext. There is no session-key (PKESK) extraction, no chunking, and no streaming. The whole ciphertext must fit in a base64 JSON request body (Vault's default max_request_size is 32 MiB), and the whole plaintext comes back base64 in the response. So you cannot feed a large archive object straight to the engine, and you cannot ask it to decrypt only the session key.

The design logarchiver chose

Rather than bound object sizes to the engine's request limit, logarchiver does hybrid encryption itself and sends the engine only a tiny key blob:

1. Encrypt (service, local): generate a random 256-bit DEK; encrypt the zstd-compressed NDJSON locally with AES-256-GCM in frames; OpenPGP-encrypt just the 32-byte DEK to the engine's public key. The wrapped DEK is a small standard OpenPGP message. The private key is never present.
2. Decrypt (CLI, retrieval): read the object header, send only the wrapped DEK (a few hundred bytes) to gpg/decrypt/<name>, get the DEK back, then stream-decrypt the AES-GCM frames locally and zstd-decompress to NDJSON.

The Vault round-trip is tiny and constant regardless of object size, the engine needs no changes, and the private key stays in Vault. See architecture.md for the container layout.

Possible engine enhancement (follow-up, not required): if the engine ever grows a "decrypt session key only / return PKESK plaintext" operation, objects could be plain standard OpenPGP messages while retaining tiny Vault round-trips. logarchiver's self-managed DEK envelope already achieves the operational goal without it, so this is optional.

One-time key setup

Create the archive key in the engine (if it doesn't already exist) and export the public key for the service:

export VAULT_ADDR=https://vault.service.consul:8200

# Create the key (idempotent; rsa-3072 default, or ed25519).
vault write gpg/keys/logarchive identity="logarchiver <ops@unkin.net>" algorithm=ed25519

# Fingerprint the service records per object (sanity check):
vault read -field=fingerprint gpg/keys/logarchive

# Export the armored public key for the service's file-mounted pubkey source:
vault read -field=public_key gpg/keys/logarchive > pubkey.asc

In k8s the service can instead read the public key directly from Vault (pubkey_source: vault, k8s auth) — no mounted file needed. The file source is the default because it removes any Vault dependency from the hot path.

Retrieve archived logs

The CLI needs: S3 read creds (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY for the logs-archive bucket), the internal CA (s3.ca_file), and a Vault token with decrypt on the key (ambient VAULT_TOKEN or ~/.vault-token, exactly like passv).

export VAULT_ADDR=https://vault.service.consul:8200
vault login ...            # or have ~/.vault-token
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=...

# 1. Find objects via the index.
logarchiver search --subject 'logs.k8s.vault.>' --host node-1 \
  --from 2026-07-01 --to 2026-07-27

# 2a. Retrieve by the object keys search printed, to stdout:
logarchiver fetch <object-key> [<object-key> ...] -o -

# 2b. Or retrieve straight from a query, re-filtered to just the matching
#     host/time events, into a directory (one NDJSON file per object):
logarchiver fetch --subject 'logs.vm.*' --host db-1 --from -24h -o ./out

fetch downloads each object, reads its header to learn which Vault key wrapped it (key_name), decrypts the wrapped DEK via the engine, streams the frames through local AES-GCM + zstd, and emits NDJSON. When --host/--from/--to are supplied they also re-filter the emitted lines (events lacking a parseable timestamp are kept, never silently dropped on time grounds).

Failure modes

Symptom	Likely cause	Action
decrypt via gpg/decrypt/<name>: permission denied	Token lacks the decrypt policy	Grant update on gpg/decrypt/<name> (and read on gpg/keys/<name>).
unwrap dek: … no plaintext	Wrong key name / key rotated below min_decryption_version	Confirm object key_name; the engine tries versions ≥ min_decryption_version.
bad magic: not a logarchiver (LARC1) object	Fetching a non-logarchiver object (e.g. an old Vector .log.gz)	Use the right prefix; Vector objects are plain gzip, decrypt them with gzip.
decrypt frame N: cipher: message authentication failed	Object corruption/tampering	Object integrity is broken; re-archive from JetStream if still within retention.
pubkey fingerprint mismatch (service)	Armored pubkey doesn't match the engine's reported fingerprint	Re-export the pubkey; the service refuses to encrypt to a mismatched key.

Key rotation

Rotating the engine key (vault write gpg/keys/logarchive/rotate ...) starts encrypting new objects to the new version; the service picks up the new public key within crypto.refresh_interval (default 1h) or on restart. Old objects remain decryptable because the engine tries every key version from min_decryption_version up. Each object records the fingerprint it was sealed with, so you can always tell which key version applies.