Initial implementation: NATS->S3 archiver + search/retrieve CLI
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Package the (already built) logarchiver binary into an RPM with nfpm,
|
||||
# bundling generated bash/zsh/fish shell completions.
|
||||
# Usage: scripts/build-rpm.sh [version] (version defaults to $CI_COMMIT_TAG)
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
cd "${ROOT_DIR}"
|
||||
|
||||
VERSION="${1:-${CI_COMMIT_TAG:-0.0.0-dev}}"
|
||||
VERSION="${VERSION#v}" # strip a leading v
|
||||
BINARY="logarchiver"
|
||||
DIST="dist"
|
||||
|
||||
if [ ! -f "${DIST}/${BINARY}" ]; then
|
||||
echo "ERROR: ${DIST}/${BINARY} not found; run 'make build' first" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Generate shell completions from the freshly built binary so they always match
|
||||
# the shipped flags/subcommands.
|
||||
COMP_DIR="${DIST}/completions"
|
||||
mkdir -p "${COMP_DIR}"
|
||||
"./${DIST}/${BINARY}" completion bash >"${COMP_DIR}/${BINARY}.bash"
|
||||
"./${DIST}/${BINARY}" completion zsh >"${COMP_DIR}/_${BINARY}"
|
||||
"./${DIST}/${BINARY}" completion fish >"${COMP_DIR}/${BINARY}.fish"
|
||||
|
||||
export PACKAGE_NAME="${BINARY}"
|
||||
export PACKAGE_VERSION="${VERSION}"
|
||||
export PACKAGE_RELEASE="1"
|
||||
export PACKAGE_ARCH="amd64"
|
||||
export PACKAGE_PLATFORM="linux"
|
||||
export PACKAGE_DESCRIPTION="Archive NATS JetStream logs to S3 (zstd + OpenPGP, indexed) and search/retrieve them. Service + operator CLI."
|
||||
export PACKAGE_MAINTAINER="Ben Vincent <ben@unkin.net>"
|
||||
export PACKAGE_HOMEPAGE="https://git.unkin.net/unkin/logarchiver"
|
||||
export PACKAGE_LICENSE="MIT"
|
||||
|
||||
envsubst <packaging/nfpm.yaml >"${DIST}/nfpm.yaml"
|
||||
nfpm pkg --config "${DIST}/nfpm.yaml" --target "${DIST}" --packager rpm
|
||||
|
||||
echo "Built:"
|
||||
ls -1 "${DIST}"/*.rpm
|
||||
Reference in New Issue
Block a user