c05ccfcb5d
logarchiver replaces the plain Vector archiver leg of the centralized logging stack (argocd-apps #296) with a Go service that archives raw logs from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed objects, plus an operator CLI to search the index and retrieve/decrypt archived logs. It adds the things that outgrew Vector: zstd compression, encryption keyed from Ben's Vault GPG secrets engine, a searchable ClickHouse index, and sink-conditional acks (a batch is acknowledged to JetStream only after the object is durably in S3 AND indexed). Service (`logarchiver run`): - Durable JetStream pull consumer (stream LOGS, durable archiver, subject filter default logs.k8s.vault.>), explicit acks, independent offsets. - Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3 PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and redelivered, so nothing is lost on a sink outage. - Encryption is a wrapped-DEK envelope (container LARC1): the bulk is AES-256-GCM framed under a random data key, and only that 32-byte key is OpenPGP-encrypted to the engine's public key. This is because the Vault GPG engine does whole-payload decrypt only; retrieval round-trips just the tiny wrapped key regardless of object size. Public key fetched from the engine or a mounted file (configurable); key fingerprint recorded per object; periodic pubkey refresh for rotation. - Prometheus metrics, structured slog, graceful drain on shutdown. CLI: - `search` queries the index (subject/host/time) and lists matching objects. - `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits NDJSON (optionally re-filtered by host/time). - `init-schema` creates/prints the ClickHouse archive_index DDL. - cobra `completion` subcommands. Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/ ClickHouse/Vault) behind interfaces with unit tests (config, batching, host/subject extraction, crypto roundtrip with a test key, ack-after-persist with fakes, search query building). go build/vet/test -race clean; golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v* tag a container image plus a Gitea binary release + rpm-internal RPM. Docs per subcommand + architecture + retrieval runbook + deployment drop-in. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
143 lines
5.0 KiB
YAML
143 lines
5.0 KiB
YAML
when:
|
|
- event: tag
|
|
|
|
steps:
|
|
- name: test
|
|
image: golang:1.25
|
|
commands:
|
|
- go test -race ./...
|
|
backend_options:
|
|
kubernetes:
|
|
serviceAccountName: default
|
|
resources:
|
|
requests:
|
|
memory: 512Mi
|
|
cpu: 1
|
|
limits:
|
|
memory: 2Gi
|
|
cpu: 2
|
|
|
|
# Build the dist/ binary (consumed by the RPM step) plus the cross-platform
|
|
# CLI binaries attached to the Gitea release.
|
|
- name: build
|
|
image: git.unkin.net/unkin/almalinux9-gobuilder:20260606
|
|
commands:
|
|
- make build VERSION=${CI_COMMIT_TAG}
|
|
# $$ escapes shell vars so Woodpecker leaves them for the shell instead of
|
|
# substituting pipeline vars at parse time; ${CI_COMMIT_TAG} is a real var.
|
|
- |
|
|
for osarch in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64; do
|
|
os="$${osarch%/*}"; arch="$${osarch#*/}"
|
|
GOOS="$$os" GOARCH="$$arch" \
|
|
go build -ldflags="-s -w -X main.version=${CI_COMMIT_TAG}" \
|
|
-o "logarchiver-$${os}-$${arch}" ./cmd/logarchiver
|
|
done
|
|
depends_on: [test]
|
|
backend_options:
|
|
kubernetes:
|
|
serviceAccountName: default
|
|
resources:
|
|
requests:
|
|
memory: 512Mi
|
|
cpu: 1
|
|
limits:
|
|
memory: 2Gi
|
|
cpu: 2
|
|
|
|
# Package the built binary + generated shell completions into an RPM.
|
|
- name: package
|
|
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
|
|
commands:
|
|
- ./scripts/build-rpm.sh ${CI_COMMIT_TAG}
|
|
depends_on: [build]
|
|
backend_options:
|
|
kubernetes:
|
|
serviceAccountName: default
|
|
resources:
|
|
requests:
|
|
memory: 512Mi
|
|
cpu: 1
|
|
limits:
|
|
memory: 2Gi
|
|
cpu: 2
|
|
|
|
# Publish the RPM to the artifactapi local rpm repo (a real yum repo;
|
|
# repodata regenerates automatically).
|
|
- name: upload-rpm
|
|
image: git.unkin.net/unkin/almalinux9-base:20260606
|
|
commands:
|
|
- |
|
|
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
|
|
REPO="rpm-internal"
|
|
for rpm in dist/*.rpm; do
|
|
FILE=$$(basename "$$rpm")
|
|
# artifactapi has no HEAD route (405); probe with GET against the
|
|
# served path (RPMs are stored under Packages/) to avoid re-upload.
|
|
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
|
|
if [ "$$code" = "200" ]; then
|
|
echo "$$FILE already exists in $$REPO (HTTP $$code); skipping upload"
|
|
continue
|
|
fi
|
|
echo "Uploading $$FILE to $$REPO (existence probe returned $$code)"
|
|
curl -f -X PUT \
|
|
"$$HOST/api/v2/remotes/$$REPO/files/$$FILE" \
|
|
-H "Content-Type: application/x-rpm" \
|
|
--data-binary @"$$rpm"
|
|
done
|
|
depends_on: [package]
|
|
backend_options:
|
|
kubernetes:
|
|
serviceAccountName: default
|
|
resources:
|
|
requests:
|
|
memory: 128Mi
|
|
cpu: 100m
|
|
limits:
|
|
memory: 512Mi
|
|
cpu: 500m
|
|
|
|
# Cut a Gitea release with the cross-platform CLI binaries attached.
|
|
- name: release
|
|
image: git.unkin.net/unkin/almalinux9-base:20260606
|
|
environment:
|
|
RELEASER_TOKEN:
|
|
from_secret: RELEASER_TOKEN
|
|
commands:
|
|
- |
|
|
curl --output /usr/local/bin/tea https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/gitea-dl/tea/0.12.0/tea-0.12.0-linux-amd64 && chmod +x /usr/local/bin/tea
|
|
tea logins add --name gitea --url https://git.unkin.net --token "$${RELEASER_TOKEN}" --no-version-check
|
|
# Find the previous release tag for the changelog range; skip tags on the
|
|
# current commit and pick the newest semver ancestor.
|
|
CUR_SHA=$$(git rev-list -n1 "${CI_COMMIT_TAG}")
|
|
PREV_TAG=""
|
|
for t in $$(git tag --sort=-v:refname); do
|
|
[ "$$t" = "${CI_COMMIT_TAG}" ] && continue
|
|
[ "$$(git rev-list -n1 "$$t")" = "$$CUR_SHA" ] && continue
|
|
if git merge-base --is-ancestor "$$t" "${CI_COMMIT_TAG}" 2>/dev/null; then
|
|
PREV_TAG="$$t"; break
|
|
fi
|
|
done
|
|
if [ -n "$$PREV_TAG" ]; then
|
|
NOTES=$$(git log "$${PREV_TAG}..${CI_COMMIT_TAG}" --pretty=format:"- %s")
|
|
else
|
|
NOTES=$$(git log --pretty=format:"- %s")
|
|
fi
|
|
tea releases create --tag "${CI_COMMIT_TAG}" --title "${CI_COMMIT_TAG}" --note "$${NOTES}" --login gitea --repo "${CI_REPO}"
|
|
RPM=$$(ls dist/*.rpm 2>/dev/null | head -1)
|
|
ASSETS="logarchiver-linux-amd64 logarchiver-linux-arm64 logarchiver-darwin-amd64 logarchiver-darwin-arm64"
|
|
[ -n "$$RPM" ] && ASSETS="$$ASSETS $$RPM"
|
|
sha256sum $$ASSETS > sha256sums.txt
|
|
tea releases assets create "${CI_COMMIT_TAG}" $$ASSETS sha256sums.txt \
|
|
--login gitea --repo "${CI_REPO}"
|
|
depends_on: [upload-rpm]
|
|
backend_options:
|
|
kubernetes:
|
|
serviceAccountName: default
|
|
resources:
|
|
requests:
|
|
memory: 128Mi
|
|
cpu: 100m
|
|
limits:
|
|
memory: 512Mi
|
|
cpu: 500m
|