From 2aa94f0de7137934bebbd0c374223372cba7d3d9 Mon Sep 17 00:00:00 2001 From: Ben Vincent Date: Fri, 24 Jul 2026 22:50:18 +1000 Subject: [PATCH 1/2] Add pdbmux: a merging PuppetDB proxy for the VM->k8s migration During the VM->k8s Puppet migration there are two PuppetDBs - the legacy Consul-registered one (http://puppetdbapi.service.consul:8080) and the new k8s one (https://puppetdb.k8s.syd1.au.unkin.net) - and nodes move between them as they migrate. node-lookup and pblastreport need a single, consistent merged view without knowing which PuppetDB a node currently lives in. This adds pdbmux, a small HTTP daemon that fronts both backends: - Adds cmd/pdbmux/ (config.go, merge.go, server.go, main.go): a cobra tool whose default action (also `serve`) starts the proxy, plus config init/show and version subcommands, following the repo's config precedence pattern (defaults < config file < env PDBMUX_* < flags). - Merges GET /pdb/query/v4/nodes: dedupes by certname, keeping the record with the newer report_timestamp. - Merges GET /pdb/query/v4/facts at node granularity: keeps all facts from the backend owning each certname, chosen by the freshness strategy (per-certname report_timestamp map from /nodes, cached for freshness_ttl) or a static prefer-backend fallback. - Fans out to both backends concurrently, serves the survivor if one fails, and returns 502 only when both fail; passes records through as raw JSON so unknown fields survive. - Transparently proxies any other /pdb/query/v4/* path to the configurable primary, and exposes /healthz with per-backend reachability (200 ok / 200 degraded / 503 down). - Adds table-driven tests (go test -race, no network) covering merge logic, handler behaviour with httptest backends, query passthrough, one/both backend down, and config precedence/validation. - Wires pdbmux into the build/release: Makefile BINARIES, scripts/build-rpm.sh, nfpm packaging (binary + completions + a systemd unit), and the release pipeline's cross-platform build + Gitea asset list. - Documents pdbmux (what/why/endpoints/merge-semantics/config/deployment) in a new README.md and updates AGENTS.md. --- .gitignore | 2 + .woodpecker/release.yaml | 6 +- AGENTS.md | 18 ++- Makefile | 2 +- README.md | 137 ++++++++++++++++++ cmd/pdbmux/config.go | 250 ++++++++++++++++++++++++++++++++ cmd/pdbmux/config_test.go | 126 ++++++++++++++++ cmd/pdbmux/main.go | 171 ++++++++++++++++++++++ cmd/pdbmux/merge.go | 173 ++++++++++++++++++++++ cmd/pdbmux/merge_test.go | 228 +++++++++++++++++++++++++++++ cmd/pdbmux/server.go | 294 ++++++++++++++++++++++++++++++++++++++ cmd/pdbmux/server_test.go | 286 ++++++++++++++++++++++++++++++++++++ packaging/nfpm.yaml | 24 ++++ packaging/pdbmux.service | 31 ++++ scripts/build-rpm.sh | 4 +- 15 files changed, 1743 insertions(+), 9 deletions(-) create mode 100644 README.md create mode 100644 cmd/pdbmux/config.go create mode 100644 cmd/pdbmux/config_test.go create mode 100644 cmd/pdbmux/main.go create mode 100644 cmd/pdbmux/merge.go create mode 100644 cmd/pdbmux/merge_test.go create mode 100644 cmd/pdbmux/server.go create mode 100644 cmd/pdbmux/server_test.go create mode 100644 packaging/pdbmux.service diff --git a/.gitignore b/.gitignore index 2ad83ab..0c15e09 100644 --- a/.gitignore +++ b/.gitignore @@ -2,8 +2,10 @@ /node-lookup /pburl /pblastreport +/pdbmux # cross-compiled release artifacts (e.g. node-lookup-linux-amd64) /node-lookup-* /pburl-* /pblastreport-* +/pdbmux-* dist/ diff --git a/.woodpecker/release.yaml b/.woodpecker/release.yaml index 47b7826..3043916 100644 --- a/.woodpecker/release.yaml +++ b/.woodpecker/release.yaml @@ -28,7 +28,7 @@ steps: # for the shell instead of substituting them (as pipeline vars) at parse # time. ${CI_COMMIT_TAG} is a real Woodpecker var and stays single-$. - | - for entry in "node-lookup:." "pburl:./cmd/pburl" "pblastreport:./cmd/pblastreport"; do + for entry in "node-lookup:." "pburl:./cmd/pburl" "pblastreport:./cmd/pblastreport" "pdbmux:./cmd/pdbmux"; do name="$${entry%%:*}"; pkg="$${entry##*:}" for osarch in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64; do os="$${osarch%/*}"; arch="$${osarch#*/}" @@ -133,6 +133,10 @@ steps: pblastreport-linux-arm64 \ pblastreport-darwin-amd64 \ pblastreport-darwin-arm64 \ + pdbmux-linux-amd64 \ + pdbmux-linux-arm64 \ + pdbmux-darwin-amd64 \ + pdbmux-darwin-arm64 \ --login gitea --repo "${CI_REPO}" depends_on: [upload-rpm] backend_options: diff --git a/AGENTS.md b/AGENTS.md index 57321bf..6780a52 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Project Overview -This repo ships three related Puppet CLIs in one RPM: +This repo ships four related Puppet tools in one RPM: - **`node-lookup`** — queries the PuppetDB API to retrieve and filter node facts. - **`pburl`** — prints the Puppetboard node-page URL for each host (reads hosts @@ -10,10 +10,16 @@ This repo ships three related Puppet CLIs in one RPM: - **`pblastreport`** — prints each host's last Puppet report time and its Puppetboard URL. Output: `\t