Files
node-lookup/.woodpecker/release.yaml
T
Ben Vincent 2aa94f0de7
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Add pdbmux: a merging PuppetDB proxy for the VM->k8s migration
During the VM->k8s Puppet migration there are two PuppetDBs - the legacy
Consul-registered one (http://puppetdbapi.service.consul:8080) and the new
k8s one (https://puppetdb.k8s.syd1.au.unkin.net) - and nodes move between
them as they migrate. node-lookup and pblastreport need a single, consistent
merged view without knowing which PuppetDB a node currently lives in.

This adds pdbmux, a small HTTP daemon that fronts both backends:

- Adds cmd/pdbmux/ (config.go, merge.go, server.go, main.go): a cobra tool
  whose default action (also `serve`) starts the proxy, plus config init/show
  and version subcommands, following the repo's config precedence pattern
  (defaults < config file < env PDBMUX_* < flags).
- Merges GET /pdb/query/v4/nodes: dedupes by certname, keeping the record with
  the newer report_timestamp.
- Merges GET /pdb/query/v4/facts at node granularity: keeps all facts from the
  backend owning each certname, chosen by the freshness strategy (per-certname
  report_timestamp map from /nodes, cached for freshness_ttl) or a static
  prefer-backend fallback.
- Fans out to both backends concurrently, serves the survivor if one fails, and
  returns 502 only when both fail; passes records through as raw JSON so unknown
  fields survive.
- Transparently proxies any other /pdb/query/v4/* path to the configurable
  primary, and exposes /healthz with per-backend reachability (200 ok /
  200 degraded / 503 down).
- Adds table-driven tests (go test -race, no network) covering merge logic,
  handler behaviour with httptest backends, query passthrough, one/both backend
  down, and config precedence/validation.
- Wires pdbmux into the build/release: Makefile BINARIES, scripts/build-rpm.sh,
  nfpm packaging (binary + completions + a systemd unit), and the release
  pipeline's cross-platform build + Gitea asset list.
- Documents pdbmux (what/why/endpoints/merge-semantics/config/deployment) in a
  new README.md and updates AGENTS.md.
2026-07-24 22:50:18 +10:00

152 lines
5.3 KiB
YAML

when:
- event: tag
steps:
- name: test
image: golang:1.25
commands:
- go test -race ./...
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 512Mi
cpu: 1
limits:
memory: 2Gi
cpu: 2
# Build all binaries into dist/ (consumed by the RPM step) plus the
# cross-platform binaries attached to the Gitea release. Each tool is a
# separate main package, so they are built individually per os/arch.
- name: build
image: git.unkin.net/unkin/almalinux9-gobuilder:20260606
commands:
- make build VERSION=${CI_COMMIT_TAG}
# Shell variables/expansions are escaped as $$ so Woodpecker leaves them
# for the shell instead of substituting them (as pipeline vars) at parse
# time. ${CI_COMMIT_TAG} is a real Woodpecker var and stays single-$.
- |
for entry in "node-lookup:." "pburl:./cmd/pburl" "pblastreport:./cmd/pblastreport" "pdbmux:./cmd/pdbmux"; do
name="$${entry%%:*}"; pkg="$${entry##*:}"
for osarch in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64; do
os="$${osarch%/*}"; arch="$${osarch#*/}"
GOOS="$$os" GOARCH="$$arch" \
go build -ldflags="-s -w -X main.version=${CI_COMMIT_TAG}" \
-o "$${name}-$${os}-$${arch}" "$$pkg"
done
done
depends_on: [test]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 512Mi
cpu: 1
limits:
memory: 2Gi
cpu: 2
# Package the built binary + generated shell completions into an RPM.
- name: package
image: git.unkin.net/unkin/almalinux9-rpmbuilder:latest
commands:
- ./scripts/build-rpm.sh ${CI_COMMIT_TAG}
depends_on: [build]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 512Mi
cpu: 1
limits:
memory: 2Gi
cpu: 2
# Publish the RPM to the artifactapi local rpm repo (a real yum repo;
# repodata regenerates automatically).
- name: upload-rpm
image: git.unkin.net/unkin/almalinux9-base:20260606
commands:
- |
HOST="https://artifactapi.k8s.syd1.au.unkin.net"
REPO="rpm-internal"
for rpm in dist/*.rpm; do
FILE=$$(basename "$$rpm")
# artifactapi has no HEAD route (returns 405); probe with GET against
# the served path (RPMs are stored under Packages/) to avoid re-upload.
code=$$(curl -s -o /dev/null -w '%{http_code}' "$$HOST/api/v2/remotes/$$REPO/files/Packages/$$FILE" || true)
if [ "$$code" = "200" ]; then
echo "$$FILE already exists in $$REPO (HTTP $$code); skipping upload"
continue
fi
echo "Uploading $$FILE to $$REPO (existence probe returned $$code)"
curl -f -X PUT \
"$$HOST/api/v2/remotes/$$REPO/files/$$FILE" \
-H "Content-Type: application/x-rpm" \
--data-binary @"$$rpm"
done
depends_on: [package]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 500m
# Cut a Gitea release with the cross-platform binaries attached.
- name: release
image: git.unkin.net/unkin/almalinux9-base:20260606
environment:
RELEASER_TOKEN:
from_secret: RELEASER_TOKEN
commands:
- |
curl --output /usr/local/bin/tea https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/gitea-dl/tea/0.12.0/tea-0.12.0-linux-amd64 && chmod +x /usr/local/bin/tea
tea logins add --name gitea --url https://git.unkin.net --token "$${RELEASER_TOKEN}" --no-version-check
# $$ escapes shell vars/substitutions so Woodpecker doesn't blank them
# at parse time; ${CI_COMMIT_TAG}/${CI_REPO} are real Woodpecker vars.
PREV_TAG=$$(git describe --tags --abbrev=0 HEAD^ 2>/dev/null || echo "")
if [ -n "$$PREV_TAG" ]; then
NOTES=$$(git log "$${PREV_TAG}..${CI_COMMIT_TAG}" --pretty=format:"- %s")
else
NOTES=$$(git log --pretty=format:"- %s")
fi
tea releases create --tag "${CI_COMMIT_TAG}" --title "${CI_COMMIT_TAG}" --note "$${NOTES}" --login gitea --repo "${CI_REPO}"
tea releases assets create "${CI_COMMIT_TAG}" \
node-lookup-linux-amd64 \
node-lookup-linux-arm64 \
node-lookup-darwin-amd64 \
node-lookup-darwin-arm64 \
pburl-linux-amd64 \
pburl-linux-arm64 \
pburl-darwin-amd64 \
pburl-darwin-arm64 \
pblastreport-linux-amd64 \
pblastreport-linux-arm64 \
pblastreport-darwin-amd64 \
pblastreport-darwin-arm64 \
pdbmux-linux-amd64 \
pdbmux-linux-arm64 \
pdbmux-darwin-amd64 \
pdbmux-darwin-arm64 \
--login gitea --repo "${CI_REPO}"
depends_on: [upload-rpm]
backend_options:
kubernetes:
serviceAccountName: default
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 500m