Add pdbmux: a merging PuppetDB proxy for the VM->k8s migration
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful

During the VM->k8s Puppet migration there are two PuppetDBs - the legacy
Consul-registered one (http://puppetdbapi.service.consul:8080) and the new
k8s one (https://puppetdb.k8s.syd1.au.unkin.net) - and nodes move between
them as they migrate. node-lookup and pblastreport need a single, consistent
merged view without knowing which PuppetDB a node currently lives in.

This adds pdbmux, a small HTTP daemon that fronts both backends:

- Adds cmd/pdbmux/ (config.go, merge.go, server.go, main.go): a cobra tool
  whose default action (also `serve`) starts the proxy, plus config init/show
  and version subcommands, following the repo's config precedence pattern
  (defaults < config file < env PDBMUX_* < flags).
- Merges GET /pdb/query/v4/nodes: dedupes by certname, keeping the record with
  the newer report_timestamp.
- Merges GET /pdb/query/v4/facts at node granularity: keeps all facts from the
  backend owning each certname, chosen by the freshness strategy (per-certname
  report_timestamp map from /nodes, cached for freshness_ttl) or a static
  prefer-backend fallback.
- Fans out to both backends concurrently, serves the survivor if one fails, and
  returns 502 only when both fail; passes records through as raw JSON so unknown
  fields survive.
- Transparently proxies any other /pdb/query/v4/* path to the configurable
  primary, and exposes /healthz with per-backend reachability (200 ok /
  200 degraded / 503 down).
- Adds table-driven tests (go test -race, no network) covering merge logic,
  handler behaviour with httptest backends, query passthrough, one/both backend
  down, and config precedence/validation.
- Wires pdbmux into the build/release: Makefile BINARIES, scripts/build-rpm.sh,
  nfpm packaging (binary + completions + a systemd unit), and the release
  pipeline's cross-platform build + Gitea asset list.
- Documents pdbmux (what/why/endpoints/merge-semantics/config/deployment) in a
  new README.md and updates AGENTS.md.
This commit is contained in:
Ben Vincent
2026-07-24 22:50:18 +10:00
parent 2aaffd7e31
commit 2aa94f0de7
15 changed files with 1743 additions and 9 deletions
+13 -5
View File
@@ -2,7 +2,7 @@
## Project Overview
This repo ships three related Puppet CLIs in one RPM:
This repo ships four related Puppet tools in one RPM:
- **`node-lookup`** — queries the PuppetDB API to retrieve and filter node facts.
- **`pburl`** — prints the Puppetboard node-page URL for each host (reads hosts
@@ -10,10 +10,16 @@ This repo ships three related Puppet CLIs in one RPM:
- **`pblastreport`** — prints each host's last Puppet report time and its
Puppetboard URL. Output: `<host>\t<time>\t<url>`. Supports `--relative`/`-r`
(relative age) and `--timezone`/`-z <IANA>` (default: local timezone).
- **`pdbmux`** — a long-running HTTP daemon that presents a single merged
PuppetDB v4 query surface over the old (Consul) and new (k8s) PuppetDBs during
the VM→k8s migration. Merges `/pdb/query/v4/{nodes,facts}`, transparently
proxies other v4 paths to the primary, and exposes `/healthz`. See README.md
for full config/merge semantics.
`node-lookup` is the module root; `pburl` and `pblastreport` live under `cmd/`
and share the `internal/puppet` package (config, PuppetDB `nodes` queries,
Puppetboard URL construction, stdin host reading).
`node-lookup` is the module root; `pburl`, `pblastreport` and `pdbmux` live
under `cmd/`. The three CLI tools share the `internal/puppet` package (config,
PuppetDB `nodes` queries, Puppetboard URL construction, stdin host reading);
`pdbmux` is self-contained (its own config + HTTP server).
## Structure
@@ -22,11 +28,13 @@ main.go # node-lookup CLI source (module root, package mai
main_test.go # node-lookup unit tests (mock PuppetDB via httptest)
cmd/pburl/main.go # pburl CLI
cmd/pblastreport/main.go # pblastreport CLI (report.go: report-time formatting)
cmd/pdbmux/ # pdbmux daemon: main.go, config.go, merge.go, server.go
internal/puppet/ # shared: config, puppetdb nodes query, board URLs, stdin
go.mod # Go module (module name: node-lookup)
go.sum # dependency checksums
Makefile # build / test / lint / completions / rpm / version-bump targets
packaging/nfpm.yaml # nfpm spec (envsubst-templated) for the RPM (all 3 binaries)
packaging/nfpm.yaml # nfpm spec (envsubst-templated) for the RPM (all 4 binaries)
packaging/pdbmux.service # systemd unit for the pdbmux daemon
scripts/build-rpm.sh # generates completions + packages the RPM with nfpm
.woodpecker/ # CI: build, test, pre-commit (PR) + release (tag)
dist/ # build output: binaries, completions, RPM (not committed)