Files
logarchiver/.woodpecker/release.yaml
T
benvin c05ccfcb5d
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
Initial implementation: NATS->S3 archiver + search/retrieve CLI
logarchiver replaces the plain Vector archiver leg of the centralized
logging stack (argocd-apps #296) with a Go service that archives raw logs
from NATS JetStream to S3 as zstd-compressed, OpenPGP-encrypted, indexed
objects, plus an operator CLI to search the index and retrieve/decrypt
archived logs. It adds the things that outgrew Vector: zstd compression,
encryption keyed from Ben's Vault GPG secrets engine, a searchable
ClickHouse index, and sink-conditional acks (a batch is acknowledged to
JetStream only after the object is durably in S3 AND indexed).

Service (`logarchiver run`):
- Durable JetStream pull consumer (stream LOGS, durable archiver, subject
  filter default logs.k8s.vault.>), explicit acks, independent offsets.
- Batch per subject by size/count/time -> NDJSON -> zstd -> encrypt -> S3
  PUT -> ClickHouse index row -> ack. On any failure the batch is Nak'd and
  redelivered, so nothing is lost on a sink outage.
- Encryption is a wrapped-DEK envelope (container LARC1): the bulk is
  AES-256-GCM framed under a random data key, and only that 32-byte key is
  OpenPGP-encrypted to the engine's public key. This is because the Vault
  GPG engine does whole-payload decrypt only; retrieval round-trips just the
  tiny wrapped key regardless of object size. Public key fetched from the
  engine or a mounted file (configurable); key fingerprint recorded per
  object; periodic pubkey refresh for rotation.
- Prometheus metrics, structured slog, graceful drain on shutdown.

CLI:
- `search` queries the index (subject/host/time) and lists matching objects.
- `fetch` downloads, decrypts via the Vault GPG engine, unzstds and emits
  NDJSON (optionally re-filtered by host/time).
- `init-schema` creates/prints the ClickHouse archive_index DDL.
- cobra `completion` subcommands.

Config via file+env (k8s-friendly, secrets from env), boundaries (NATS/S3/
ClickHouse/Vault) behind interfaces with unit tests (config, batching,
host/subject extraction, crypto roundtrip with a test key, ack-after-persist
with fakes, search query building). go build/vet/test -race clean;
golangci-lint v2 clean. Woodpecker CI: build/test/pre-commit on PR; on v*
tag a container image plus a Gitea binary release + rpm-internal RPM. Docs
per subcommand + architecture + retrieval runbook + deployment drop-in.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-27 23:22:40 +10:00

143 lines
5.0 KiB
YAML

when:
- event: tag
steps:
- name: test
image: golang:1.25
commands:
- go test -race ./...
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 512Mi
cpu: 1
limits:
memory: 2Gi
cpu: 2
# Build the dist/ binary (consumed by the RPM step) plus the cross-platform
# CLI binaries attached to the Gitea release.
- name: build
image: git.unkin.net/unkin/almalinux9-gobuilder:20260606
commands:
- make build VERSION=${CI_COMMIT_TAG}
# $$ escapes shell vars so Woodpecker leaves them for the shell instead of
# substituting pipeline vars at parse time; ${CI_COMMIT_TAG} is a real var.
- |
for osarch in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64; do
os="$${osarch%/*}"; arch="$${osarch#*/}"
GOOS="$$os" GOARCH="$$arch" \
go build -ldflags="-s -w -X main.version=${CI_COMMIT_TAG}" \
-o "logarchiver-$${os}-$${arch}" ./cmd/logarchiver
done
depends_on: [test]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 512Mi
cpu: 1
limits:
memory: 2Gi
cpu: 2
# Package the built binary + generated shell completions into an RPM.
- name: package
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
commands:
- ./scripts/build-rpm.sh ${CI_COMMIT_TAG}
depends_on: [build]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 512Mi
cpu: 1
limits:
memory: 2Gi
cpu: 2
# Publish the RPM to the artifactapi local rpm repo (a real yum repo;
# repodata regenerates automatically).
- name: upload-rpm
image: git.unkin.net/unkin/almalinux9-base:20260606
commands:
- |
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
REPO="rpm-internal"
for rpm in dist/*.rpm; do
FILE=$$(basename "$$rpm")
# artifactapi has no HEAD route (405); probe with GET against the
# served path (RPMs are stored under Packages/) to avoid re-upload.
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
if [ "$$code" = "200" ]; then
echo "$$FILE already exists in $$REPO (HTTP $$code); skipping upload"
continue
fi
echo "Uploading $$FILE to $$REPO (existence probe returned $$code)"
curl -f -X PUT \
"$$HOST/api/v2/remotes/$$REPO/files/$$FILE" \
-H "Content-Type: application/x-rpm" \
--data-binary @"$$rpm"
done
depends_on: [package]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 500m
# Cut a Gitea release with the cross-platform CLI binaries attached.
- name: release
image: git.unkin.net/unkin/almalinux9-base:20260606
environment:
RELEASER_TOKEN:
from_secret: RELEASER_TOKEN
commands:
- |
curl --output /usr/local/bin/tea https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/gitea-dl/tea/0.12.0/tea-0.12.0-linux-amd64 && chmod +x /usr/local/bin/tea
tea logins add --name gitea --url https://git.unkin.net --token "$${RELEASER_TOKEN}" --no-version-check
# Find the previous release tag for the changelog range; skip tags on the
# current commit and pick the newest semver ancestor.
CUR_SHA=$$(git rev-list -n1 "${CI_COMMIT_TAG}")
PREV_TAG=""
for t in $$(git tag --sort=-v:refname); do
[ "$$t" = "${CI_COMMIT_TAG}" ] && continue
[ "$$(git rev-list -n1 "$$t")" = "$$CUR_SHA" ] && continue
if git merge-base --is-ancestor "$$t" "${CI_COMMIT_TAG}" 2>/dev/null; then
PREV_TAG="$$t"; break
fi
done
if [ -n "$$PREV_TAG" ]; then
NOTES=$$(git log "$${PREV_TAG}..${CI_COMMIT_TAG}" --pretty=format:"- %s")
else
NOTES=$$(git log --pretty=format:"- %s")
fi
tea releases create --tag "${CI_COMMIT_TAG}" --title "${CI_COMMIT_TAG}" --note "$${NOTES}" --login gitea --repo "${CI_REPO}"
RPM=$$(ls dist/*.rpm 2>/dev/null | head -1)
ASSETS="logarchiver-linux-amd64 logarchiver-linux-arm64 logarchiver-darwin-amd64 logarchiver-darwin-arm64"
[ -n "$$RPM" ] && ASSETS="$$ASSETS $$RPM"
sha256sum $$ASSETS > sha256sums.txt
tea releases assets create "${CI_COMMIT_TAG}" $$ASSETS sha256sums.txt \
--login gitea --repo "${CI_REPO}"
depends_on: [upload-rpm]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 500m