benvin
c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
82 lines
2.4 KiB
Makefile
82 lines
2.4 KiB
Makefile
BINARY := logarchiver
|
|
PKG := ./cmd/logarchiver
|
|
DIST := dist
|
|
MODULE := git.unkin.net/unkin/logarchiver
|
|
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
|
GOFLAGS := -ldflags="-s -w -X main.version=$(VERSION)"
|
|
OS ?= $(shell go env GOOS)
|
|
ARCH ?= $(shell go env GOARCH)
|
|
|
|
GO_VERSION_REQUIRED := 1.23
|
|
GO_VERSION_ACTUAL := $(shell go version | sed 's/go version go\([0-9]*\.[0-9]*\).*/\1/')
|
|
|
|
.PHONY: all build test test-race lint fmt vet tidy clean install completions rpm rpm-package check-go patch minor major _tag
|
|
|
|
all: build
|
|
|
|
check-go:
|
|
@if [ "$$(printf '%s\n%s' "$(GO_VERSION_REQUIRED)" "$(GO_VERSION_ACTUAL)" | sort -V | head -1)" != "$(GO_VERSION_REQUIRED)" ]; then \
|
|
echo "ERROR: Go >= $(GO_VERSION_REQUIRED) required, found $(GO_VERSION_ACTUAL)"; exit 1; \
|
|
fi
|
|
|
|
# Build the single logarchiver binary (service + CLI) into dist/.
|
|
build: check-go tidy
|
|
CGO_ENABLED=0 GOOS=$(OS) GOARCH=$(ARCH) go build $(GOFLAGS) -o $(DIST)/$(BINARY) $(PKG)
|
|
|
|
test: check-go
|
|
go test -race -count=1 ./...
|
|
|
|
test-race: test
|
|
|
|
lint:
|
|
golangci-lint run ./...
|
|
|
|
vet:
|
|
go vet ./...
|
|
|
|
fmt:
|
|
gofmt -w .
|
|
|
|
tidy:
|
|
go mod tidy
|
|
|
|
install:
|
|
go install $(GOFLAGS) $(PKG)
|
|
|
|
clean:
|
|
rm -rf $(DIST) $(BINARY) $(BINARY)-*
|
|
|
|
# Generate bash/zsh/fish completions from the freshly built binary.
|
|
completions: build
|
|
@mkdir -p $(DIST)/completions
|
|
$(DIST)/$(BINARY) completion bash > $(DIST)/completions/$(BINARY).bash
|
|
$(DIST)/$(BINARY) completion zsh > $(DIST)/completions/_$(BINARY)
|
|
$(DIST)/$(BINARY) completion fish > $(DIST)/completions/$(BINARY).fish
|
|
|
|
# Build then package the CLI into an RPM (with completions) via nfpm.
|
|
rpm: build rpm-package
|
|
rpm-package:
|
|
./scripts/build-rpm.sh $(VERSION)
|
|
|
|
# Bump helpers — read the latest semver tag and create/push the next one.
|
|
_LATEST := $(shell git tag --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$$' | head -1)
|
|
_BASE := $(if $(_LATEST),$(_LATEST),v0.0.0)
|
|
_MAJ := $(shell echo $(_BASE) | sed 's/^v//' | cut -d. -f1)
|
|
_MIN := $(shell echo $(_BASE) | sed 's/^v//' | cut -d. -f2)
|
|
_PAT := $(shell echo $(_BASE) | sed 's/^v//' | cut -d. -f3)
|
|
|
|
patch:
|
|
@NEW=v$(_MAJ).$(_MIN).$(shell expr $(_PAT) + 1); \
|
|
git tag $$NEW && echo "Tagged $$NEW" && $(MAKE) _tag TAG=$$NEW
|
|
|
|
minor:
|
|
@NEW=v$(_MAJ).$(shell expr $(_MIN) + 1).0; \
|
|
git tag $$NEW && echo "Tagged $$NEW" && $(MAKE) _tag TAG=$$NEW
|
|
|
|
major:
|
|
@NEW=v$(shell expr $(_MAJ) + 1).0.0; \
|
|
git tag $$NEW && echo "Tagged $$NEW" && $(MAKE) _tag TAG=$$NEW
|
|
|
|
_tag:
|
|
git push origin $(TAG)
|