pdbmux — merging PuppetDB proxy

pdbmux is a small HTTP daemon that fronts several PuppetDB backends and serves a single, merged PuppetDB v4 query surface on one address. Point Puppetboard, or any other PuppetDB API client, at pdbmux instead of a raw PuppetDB and it sees one consistent view spanning all of them.

Why

Running more than one PuppetDB — during a migration between two of them, or across regions — means a given node's current data lives in exactly one at any moment, and consumers have to know which, or query each in turn. pdbmux merges them all so consumers don't have to know (or query twice) which PuppetDB a node currently lives in.

All backends are equal — pdbmux is never told which one to favour. Backend names are arbitrary labels and there is no fixed number of them. The configured order is used only as a tie-break, so output is reproducible.

Endpoints

pdbmux proxies GET requests only. The query param (PuppetDB AST JSON, not PQL) is forwarded verbatim.

Path	Behaviour
GET /pdb/query/v4/nodes	Fan out to all backends, dedupe by certname, keep the record with the newer report_timestamp, stamped with the winning backend's name (see provenance). An extract query with a function column is combined instead.
GET /pdb/query/v4/facts	Fan out to all, and per certname keep all facts from the backend that owns that node (see merge semantics), plus a synthetic pdbmux_source fact naming it. An extract query with a function column is combined instead.
GET /pdb/query/v4/facts/<name>[/<value>]	Same fan-out and merge as /facts, and an extract query with a function column is combined the same way. The path segment is a name constraint, so no synthetic pdbmux_source record is added — except on the fact's own path, which is synthesised from the /facts merge (see provenance).
GET /pdb/query/v4/fact-names	Fan out to all and serve the union of the flat name arrays, deduped and re-sorted, re-paged across backends, plus the pdbmux_source name while injection is on. order_by is only valid on name.
GET /pdb/query/v4/resources	An extract query with a function column is fanned out and combined; any other query is an unmerged pass-through.
GET /pdb/query/v4/reports	Fan out to all and serve the union, deduped by report hash, re-ordered and re-paged across backends.
GET /pdb/query/v4/events	Fan out to all and serve the union, deduped by record identity, re-ordered and re-paged. An extract query with a function column is combined instead.
GET /pdb/query/v4/event-counts	Fan out to all and sum each subject's counts into one row per subject.
GET /pdb/query/v4/aggregate-event-counts	Fan out to all and sum the summary object's counts.
GET /pdb/query/v4/reports/<hash>/{events,logs,metrics}	Ask every backend; serve the answer from whichever backend actually holds that report. 404 when none does.
GET /pdb/query/v4/{nodes,factsets,catalogs}/<certname>[/...]	Ask the backend that owns that certname — the same owner the /facts merge attributes records to — and serve its reply verbatim, success or error. The remaining backends are tried after it, so a node only one backend holds is still served, and openvoxdb's own 404 body is replayed when none holds it.
GET /pdb/query/v4/* (any other)	No merge rule, so backends are tried in configured order and the first success is streamed back verbatim; if all reject it, the first upstream error response is replayed.
GET /pdb/meta/v1/version	Fan out to all and report the lowest version any backend runs.
GET /pdb/meta/v1/server-time	Fan out to all and serve the first reachable backend's clock.
GET /metrics/v2/read/<mbean>	Fan out to all and merge the Jolokia response; numeric attributes are summed by default (see merge semantics).
GET /metrics/v2/list	Fan out to all and serve the union of the backends' MBean trees.
GET /metrics/v1/mbeans[/<mbean>]	Same merge, applied to the legacy envelope-less body.
GET /healthz	Per-backend reachability, probe state and cache state. 200 {"status":"ok"} if all reachable, 200 degraded if some fail, 503 down if all fail.

Fan-out is concurrent, and goes only to the backends the health prober currently believes are up — see Backend health. If one backend errors or times out, pdbmux serves the surviving backends' results and logs a warning; a merged endpoint only returns 502 when every backend fails. Response records are passed through as raw JSON so unknown fields survive untouched, under openvoxdb's own application/json;charset=utf-8.

Every backend is asked the same question, so a status every backend answered with is the estate's own answer, not an outage: that status, openvoxdb's own explanation and its content type are replayed to the client instead of a 502, with any backend address stripped out of the body first. This holds for every status from 400 up — openvoxdb answers ["=","name"] with 400 on /nodes but 500 on /facts, and /metrics/v2 with a flat 403, so a rule drawn at 500 would replay one and swallow the other. 502 is kept for what it actually describes: backends disagreeing on the status, or a backend that answered nothing at all. The rule is the same on /pdb/query, /pdb/meta and /metrics, so no route answers a failure differently from any other.

Unanimity is asked for only where the backends are peers answering the same question. The per-certname routes are not: one backend owns the node and the others are fallbacks, so the owner's reply is the answer whatever they said — a backend that does not hold the certname answers 404 to say the node is not its own, not to disagree about the request. When the owner answers nothing at all the first fallback that did answer stands in, so an unreachable owner does not take the node with it, and 502 is left for a request no backend answered.

A unanimous 4xx blames the request, so it is not counted as a partial round on /healthz and nothing about it is cached. A unanimous 5xx is the backends reporting their own fault, so it is replayed just as faithfully but still counts as a failed round and still falls back to a stale cache entry where there is one.

Responses carry PuppetDB's X-Records when the query asked for a total, and on the merged paths X-Backends (see Backend health). Cached paths add two more headers pdbmux sets itself, X-Cache and Age — see Caching.

Merge semantics

• /nodes — dedupe by certname; the record with the strictly-newer report_timestamp wins. On a tie, the backend listed first in backends supplies the record — a tie-break only, so the merged output is deterministic.
• /facts — node-level granularity. For a certname present in more than one backend, pdbmux keeps all of that node's facts from one backend and drops the others', chosen by the merge strategy:
  • freshness (default) — attribute each certname to whichever backend holds its newer report_timestamp. pdbmux derives this from a per-certname freshness map built by querying /nodes from every backend, cached for freshness_ttl (default 30s).
  • static — skip the extra /nodes query and take each shared node's facts from the first backend in configured order that holds it.
  • A node present in only one backend always appears (falls back to whichever backend actually returned facts for it).
  • /facts/<name> and /facts/<name>/<value> are the same records with one more constraint applied upstream, so they take the same rule — and the same aggregate branch, since a count row has no certname there either. The pdbmux_source path is the exception: no backend holds that name, so it is synthesised from the /facts merge (see provenance).
• /fact-names — a flat array of strings, not records: union, deduped by the name and re-sorted, ascending unless order_by says otherwise. name is the only column the entity projects, so an order_by on any other field is rejected with 400, as the backends reject it. While injection is on the pdbmux_source name is listed too, exactly once (see provenance). include_total=true reports the deduped union's size, so the limit is applied to the merged list rather than pushed upstream.
• /reports, /events — union, not a per-node winner. Reports are immutable history, so a node's reports can legitimately exist in more than one backend and all of them belong in the merged view. Reports dedupe on hash; events, which carry no id of their own, dedupe on the verbatim record (a node reporting to more than one backend stores identical records in each).
• Aggregates — extract/group_by rows are counts, not records, so each backend returns a partial answer that has to be combined, not deduped. This covers /event-counts, /aggregate-event-counts, and any /reports, /events, /nodes, /resources, /facts or /facts/<name>[/<value>] query whose extract carries a ["function", ...] column.

  • The grouping key is the row's full set of non-aggregate columns: for /reports, /events, /nodes, /resources, /facts and /facts/<name> they come from the query — the plain extract fields, the row-function columns and any group_by clause — and for the event-count endpoints from the row itself (subject_type/subject, or summarize_by), whose remaining fields are all counts.

  • On /nodes, /facts and /facts/<name>[/<value>] this takes precedence over the certname merge: an aggregate row has no certname, so deduping would collapse every backend's rows into one backend's numbers. A query with no function column — including a plain extract projection — still merges by certname.

  • PuppetDB accepts count, sum, avg, min, max, to_string and jsonb_typeof as extract functions, and names each response column after the function itself. Each is combined by its own rule rather than by a blanket sum:

    function	merged across backends by
    count	adding
    sum	adding
    min	the smallest value any backend reported, on text columns as well as numeric ones
    max	the largest value any backend reported, likewise
    avg	rewriting the upstream query into sum + count of the same column and dividing the totals, so the answer is the estate's true weighted average, not an average of averages
    to_string	nothing — it is a row function, so it groups like a plain projected column
    jsonb_typeof	likewise

  • Because each column is named after its function, an extract that projects the same function twice — any of them — names one response column twice. openvoxdb aliases the repeat as <name>_2 (then _3, and so on), which is neither a grouping key nor an aggregate pdbmux knows to fold, so the first backend's value would freeze into the merged row. Such a query is refused with 400 naming the clashing column, as is one whose plain extract field takes the name a projected function would use. Repeating a plain field is not a clash: the copy holds the same value as the key it duplicates.

  • The avg rewrite is invisible to the client: the request still answers under the avg key. It needs the sum and count response columns for itself, so an extract that also projects a sum or a count is refused with 400 naming the clash rather than answered with a wrong number. An avg over no rows stays null, as upstream. An order_by on avg is applied to the merged rows here, not upstream.

  • avg is folded as sum / count in float64, while a single openvoxdb divides in Postgres numeric, which is arbitrary-precision. Whole-number averages round-trip exactly; a fractional one can differ from a single backend's answer in the low-order digits, as can a sum beyond 2^53.

  • An extract function pdbmux has no combiner for is refused with 400 rather than folded on a guess.

  • /resources has no cross-backend record identity to dedupe on, so only its aggregate queries merge; everything else stays an unmerged pass-through.

  • Rows sharing a key collapse into one with each aggregate column combined by its own rule. A key only one backend reported is passed through byte-for-byte. An aggregate column that is absent or null in a row is skipped, never zeroed or treated as an extreme, so the backends that did report a value still count.

  • to_string and jsonb_typeof compile to scalar expressions upstream, so they return one row per record rather than an aggregate. They form part of the grouping key alongside the plain extract fields and the group_by clause — including a group_by that names the function itself. An extract of nothing but row functions has no aggregate to fold, so every backend's rows are kept as they came — and, having one row per record rather than per group, they keep the upstream limit that bounds them.

  • limit and offset are not forwarded for an extract that folds: a backend's own first N groups are not the merged result's first N, and a group truncated away on one backend would fold to a wrong value. Every group is fetched and the window cut after the fold, which an aggregate's row count — one per distinct group value — keeps affordable. include_total still reports the merged group count.

  • A /reports or /events query with no function column is a projection of real records, not an aggregate, and stays on the union path — so an event stored identically in two backends is still served once.

  • distinct_resources=true on an /events extract with a function column is refused with 400 naming the incompatibility, before any fan-out. openvoxdb answers the distinct-resources form of /events from its legacy compiler, which supports neither function nor group_by, so every backend fails and the mistake would otherwise surface as a 502. A distinct_resources query with no function column is untouched and still fans out; no other endpoint pdbmux combines honours the parameter.

  • include_total=true on a combined endpoint reports the merged row count, not the sum of the backends' X-Records, since shared keys collapse.

Provenance: the pdbmux_source fact

Once several PuppetDBs sit behind one endpoint, a consumer can no longer tell which backend a node's data came from. pdbmux makes that visible in the response itself, so nothing has to query each backend to find out:

• /facts gains one extra fact record per certname, alongside the node's real facts, in the shape of a real fact record — certname, name, value, environment — with value set to the backend name from backends / PDBMUX_BACKENDS. environment is copied from that node's own facts (all four keys are always present, since clients index them directly).
• /nodes gains a pdbmux_source key on each merged node record. A node record carries no facts, so this is a synthetic field, not a fact — the one key outside PuppetDB's documented node schema. Clients read node fields by name, so an extra key is ignored by anything that doesn't want it.

The value always names the backend whose data won that endpoint's merge, not a backend that merely holds the node. The two endpoints resolve their winner separately, so under merge: static they can legitimately disagree: /facts attributes a shared node to the first backend in configured order, while /nodes always attributes it to the backend holding the newer report_timestamp. Each answer describes the record it is attached to.

While the fact is enabled, any /facts record whose own name field equals the configured name is dropped, on every query shape — including the shapes below, where nothing is injected in its place. The rule reads the record, not the query, so a projection that filters on name without returning it — say ["extract",["certname","value"],["=","name","pdbmux_source"]] — produces rows that no longer identify themselves, and an upstream value of that name comes through. Ask for the name column and the guarantee holds. Each request that drops a record logs it once. Only source_fact_enabled: false restores upstream records of that name; rename the synthetic fact via source_fact if the real one matters more.

Injection is skipped, and no synthetic record is added, when:

• the query contains an extract outside a subquery — it projects a column subset, and with a ["function", ...] column it aggregates. Injecting there would break the row shape or silently inflate a count(), so aggregate results are never changed. The whole query is walked, so an extract nested under and/or/not/from skips injection too; an extract under in, subquery, or select_<entity> projects that subquery rather than the response, so it does not;
• the query is not an AST array — every PQL-syntax query (facts { certname = "web1" }) lands here. pdbmux cannot tell what such a query projects, so it never injects into a PQL response. Use the AST form to get the fact;
• (/facts only) the query constrains name — ["=","name","osfamily"] and friends ask for specific facts, and the synthetic record is not one of them. Only the outer query is inspected: a name filter inside an in/select_facts subquery narrows which nodes match, not which facts come back, so injection still happens;
• the path is /facts/<name> for any other fact. The path segment is the same outer name constraint, so only the fact's own path carries the record;
• injection is turned off (see source_fact_enabled).

The fact's own path is synthesised. /pdb/query/v4/fact-names lists the name while injection is on, so a client that discovers names there can click through to it, and /pdb/query/v4/facts/pdbmux_source has to answer. No backend holds a record of that name, so the route does not serve its own fan-out: it takes the records from the /facts merge that produces them, which makes the certname set, the owner and the environment identical to the ones an unfiltered /facts response reports, and lets the request's own query narrow the result upstream. That costs one /facts fan-out per cache miss — the widest fan-out pdbmux makes — on a rare, user-initiated path.

/facts/pdbmux_source/<value> pins the backend name, so it answers with the nodes that backend owns. The <value> segment never reaches that fan-out: the synthetic record's value is always a backend name, so a value naming none is answered [] from the configured names alone, with no fan-out at all, and a value naming one filters a record set fetched under a key the value is not part of. The record set is a property of the estate rather than of the filter, so every value of it — and the unfiltered path — share one entry and one fetch. An extract query with a function column still takes the combining branch, and the gated query shapes above still answer [], as does every form while injection is off — with the name kept out of /fact-names, since nothing then produces it.

Not supported in v1: server-side filtering on the fact. A query that selects it — ["=","name","pdbmux_source"], or an extract naming it — is forwarded to the backends like any other, and they return nothing, because the fact does not exist upstream. pdbmux does not evaluate the AST itself, so it cannot answer such a query correctly for every operator (not, or, subqueries) and does not pretend to for some. Read the fact from an unfiltered (or certname-filtered) /facts response, from the path route above, and filter client-side.

Not covered: /factsets, /inventory and the per-certname routes. The first two carry facts but are not merged today — they take the unmerged pass-through path, where the answer comes from whichever backend replied first rather than from a merge winner, so there is no owner to attribute. The per-certname routes do resolve to an owner, but their bodies are passed through verbatim rather than rebuilt, so nothing is added to them either.

Metadata and metrics

• /pdb/meta/v1/version — when the backends agree, that version is served. When they differ, pdbmux reports the lowest: a client reads this as the feature level it may rely on, and the estate can only be relied on for what its oldest PuppetDB implements. Versions compare segment by segment, numerically where both segments are numbers (7.9.0 < 7.12.0), lexically otherwise. A backend whose body is unparseable is skipped rather than treated as lowest.
• /pdb/meta/v1/server-time — the clock of whichever PuppetDB answered is not estate state and has no meaningful merge, so the first reachable backend in configured order supplies it, the same tie-break used elsewhere.
• /metrics/... — the Jolokia envelope's value is merged and the rest of the envelope comes from the first backend (with the newest timestamp). Values merge recursively:
  • Objects merge over the union of their keys, so an MBean attribute only one backend exposes still survives.
  • Numbers combine by the attribute's own name. The default is a sum — almost everything here is a population count (num-nodes, num-resources, queue depth, command totals) whose estate-wide value is the total, and rates are additive throughput. The exceptions describe a distribution or a bound, where adding two servers' numbers yields a figure that was never true of either: Min takes the minimum; Max, Uptime and StartTime take the maximum; Mean, Median, StdDev and *Percentile take the unweighted arithmetic mean (pdbmux has no per-backend sample counts to weight by). Matching is case-insensitive.
  • Strings, booleans, arrays, nulls and mixed kinds keep the first backend's value — there is no sound way to add them.
  • Jolokia signals a bad MBean as a non-2xx status inside an HTTP 200. Such a backend is skipped; if every backend does so, the first one's error envelope is replayed verbatim so the client sees the real reason.
  • MBean names arrive percent-encoded over Jolokia's own !-escaping; the raw path is forwarded so neither layer is lost.

Paging and ordering on the merged endpoints

Each backend applies order_by/limit/offset to its own slice only, so pdbmux re-does all three over the union:

• order_by is parsed and the merged set re-sorted by those fields (ties keep the merged set's existing order). A record missing an ordered field sorts first.
• Backends are asked for the first offset + limit records — never an offset — and the requested window is then cut from the merged, re-sorted set.
• A folded extract aggregate is the exception: neither limit nor offset is forwarded, since a group truncated on one backend cannot be folded correctly.
• include_total=true on a union endpoint makes pdbmux sum each backend's X-Records header into one merged header. Deduped records are counted once per backend, so the total is an upper bound. Combined endpoints report the merged row count instead.
• A malformed limit, offset or order_by gets a 400 rather than being forwarded.

Backend health

A backend that is down otherwise costs a full timeout stall on every request, since fan-out has no way to know before it asks. pdbmux polls each backend's status endpoint in the background instead, and skips the ones that are not answering.

• Endpoint — health_probe_path, default /status/v1/services, PuppetDB's trapperkeeper status service (unauthenticated by default). A backend is healthy when it answers 200 and every service in the body reports "state": "running" — a 200 whose body says starting, stopping, error or unknown counts as a failure. A body that is not in that shape is judged on its status code alone, so pointing health_probe_path at some other endpoint still works.
• A refused probe is not a sick backend. Probe replies split in two. Evidence about the backend is a transport failure (connection refused, DNS, TLS, timeout), a 5xx — 503 included, since trapperkeeper answers 503 exactly when its services are not nominal — or a 429, which is the backend reporting its own capacity rather than judging the request, so an overloaded backend gets backed off instead of kept at full traffic. A reply that refuses the probe request is evidence about the probe: the other 4xx are the backend answering that our request is the problem (404/410 the path is not there, 405 it does not take a GET, 401/403 we are not allowed to ask), and 501 says it does not implement the endpoint.
• A backend is only gated on a probe that has worked for it. Each backend carries one latch: has its probe endpoint ever answered — replied with something readable as healthy or unhealthy — since pdbmux started? A rejection refused the request and a transport failure never reached the endpoint, so neither one sets it; a 200, a 503 or a degraded body does. The latch decides which rule applies, and it never clears, so no repeating pattern of failures can argue a backend back into service.
  • Never answered — there is no health signal for this backend, so nothing gates on one. It is left in service — still queried, still contributing records — permanently, reported as probe_unsupported rather than healthy so an operator can tell "verified healthy" from "not actually being checked". Real failures do not take it out either: no conclusion about a backend can be drawn from a probe that cannot run. This is the misconfigured-path case, and it degrades that backend to the behaviour from before health checks existed, which is the right floor. The misconfiguration is logged once, naming the backend, the probe path and the status. A backend that has been unreachable since pdbmux started has not answered either, so it is not gated until it answers once — reachable on /healthz is what reports it in the meantime.
  • Answered at least once — the path works, so the probe is trusted and the ordinary thresholds below apply. A later run of rejections counts as failure, not probe_unsupported: a path that answered before and refuses now has moved or changed its authorization, which is logged loudly when the run starts.
• Thresholds — a healthy backend leaves the pool after health_probe_failures (default 3) consecutive unsuccessful probes; a down one comes back after health_probe_successes (default 2) consecutive successes, and the same failure threshold debounces the probe_unsupported warning. One blip cannot flap a backend out, and one lucky reply cannot flap it back in. The run counts every unsuccessful probe whatever its kind, so a backend that fails every probe in mixed ways — a 503, then a 404, then a timeout — still trips the threshold; only a success resets the run. Probes from before the latch was set do not count toward it. A down backend keeps being probed, so recovery is automatic.
• Accepted trade-off: a probe path that is removed. If a backend's probe path works and later goes away — an upgrade, a proxy change — the latch is already set, so the refusals count as failures and that backend is excluded even though it is serving queries fine. Global fail-open still covers the case where this happens to every backend, /healthz shows the state, and the log line names the probe path: fix health_probe_path, or set health_probe_enabled: false. There is no machinery to detect this automatically — any rule that readmits a backend on "no real failure lately" flaps a genuinely dead backend into service on a periodic failure pattern.
• Fails open — if the prober has marked every backend down, pdbmux queries them all anyway. A wrong health_probe_path, a broken prober or a partition that only the prober sees can therefore never black-hole traffic; the worst case is today's behaviour.
• Serves immediately — the listener never waits for a first probe round, and a backend nobody has probed yet counts as healthy, so a restart drops nothing.
• Quiet — only transitions are logged, never individual probes: up→down, down→up, a probe that has never answered reaching its failure threshold, the first answer after that, and the start of a rejection run on a probe that used to answer.
• Partial responses stay partial. Health state changes which backends are asked, never what a merged answer means: a response built from a subset is still served, as before. Every merged response carries X-Backends: <contributed>/<configured> naming how many backends' records went into it, so a client can tell a full answer from a partial one. On a cache hit the header describes the stored body, not the current backend count.
• /healthz gives each backend a state (healthy, unhealthy, probe_unsupported, unprobed, or unmonitored when probing is off), consecutive_failures, consecutive_successes, last_probe and last_error, alongside the reachable check /healthz runs itself — which always asks every backend, so a backend queries are skipping is still reported. Read the two together: state is the prober's verdict and probe_unsupported means "not being verified", not "well", so reachable is the field that says whether the backend is answering right now. A proxy that 404s everything because the backend behind it is dead shows state: probe_unsupported with reachable carrying the query error, and the overall status drops to degraded or down accordingly. A query object reports the last merged fan-out: partial, contributed, configured, partial_rounds and last_partial.
• health_probe_enabled: false turns the whole thing off: no probing goroutines, no backend ever skipped, every backend queried on every request. X-Backends still reports how many answered.

Caching

pdbmux caches merged /nodes, /facts, /facts/<name>[/<value>] and /fact-names record sets in memory so a busy Puppetboard does not re-fan-out the same query every few seconds — its facts overview and fact drilldown are two of the pages that hit hardest. Everything else runs uncached — including extract aggregates on those paths, and the /pdb/meta/v1/* and /metrics/* endpoints, which are served live on every request. The cache is an interface, and /reports gets its own (S3-backed) backend later without further handler changes.

• Key — <path>?<params>, where the params are the ones that actually determine the response, URL-encoded with keys sorted ascending and a repeated param's values sorted ascending. Param order in the request is therefore irrelevant: one canonical key per distinct request. A request with no params keys on the bare path.
• TTL — facts_ttl, default 30s, hard cap 30s. A larger configured value is clamped down to the cap, not rejected, so a stray env var cannot crash-loop a container; pdbmux config show prints facts_ttl : 30s (clamped from 600s, cap 30s) when that happens. facts_ttl: 0 disables the cache entirely and the merged endpoints behave exactly as before.
• Stale on failure only — an expired entry is kept, not dropped. When the TTL has passed pdbmux always re-queries the backends; the expired copy is served only if every backend fails, which turns a 502 into slightly-old data. A healthy backend is never shadowed by a stale entry, and a query every backend refuses with a 4xx is answered with the refusal rather than the stale copy — a unanimous 5xx is an outage like any other and still takes the stale copy.
• Bounded — facts_cache_bytes (default 64 MiB) is a byte budget, evicted least-recently-used; reads count as use, so a stale entry that is still being asked for survives. A single response larger than the whole budget is not cached at all. The budget counts stored response bodies only — cache keys and the list/map bookkeeping are not accounted for, so it is a target for body bytes rather than a hard cap on process memory.
• Single-flight — concurrent requests for the same key collapse into one upstream fan-out; the rest wait for it and share the result. That fan-out runs on its own context, bounded by timeout, so a client that disconnects can neither cancel nor fail the requests sharing its flight; a waiter whose own client goes away leaves the flight running for the others. The flight is cancelled once its last participant leaves, so a lone client disconnecting releases the upstream connections straight away.
• Response headers — every response on a cached path carries X-Cache (hit served from a fresh entry, miss built by this request, stale the expired-entry fallback) and Age in whole seconds since the served copy was stored (0 on a miss). Uncached paths carry neither.
• Visibility — /healthz carries a cache object: backend (memory/none), ttl, entries, stale_entries, bytes, serving_stale, stale_served and last_stale_served. serving_stale is true from the moment a stale fallback is served until the next response comes from a live fan-out or a fresh entry.
• Provenance is stored, not re-applied — what a cache entry holds is the fully merged body, pdbmux_source already injected and upstream records of that name already dropped. Attribution names the backend that supplied the data, which is a property of that fetch, so it stays correct for as long as the body does and ages out with it — X-Cache and Age say how old both are. Two requests can only share an entry when they share a key, and the key is path plus query, which is exactly what decides whether injection applies; a name-filtered /facts query and a plain one therefore cache separately and neither is ever served the other's shape. The one path that keys on less than it is asked is the pdbmux_source drilldown, whose <value> is dropped from the key and applied to the shared entry instead. source_fact and source_fact_enabled are read once at startup, and the cache lives for the same process, so changing either cannot leave differently-shaped entries behind.

Config

Precedence (lowest → highest): defaults < config file < env vars (PDBMUX_*) < flags.

The config file is optional; a file, env vars, or both work equally well, including in a container.

Which file is read: --config <path>, else PDBMUX_CONFIG, else the first that exists of $XDG_CONFIG_HOME/pdbmux/config.yaml (or $HOME/.config/pdbmux/config.yaml), then /etc/pdbmux/config.yaml. A path given via --config/PDBMUX_CONFIG must exist — pdbmux fails rather than silently falling back — while a missing file on the default search path is fine. pdbmux config show prints the file it loaded, or the paths it searched.

listen: ":8080"
backends:                          # order is a tie-break only, not a ranking
  - name: pdb-a
    url: http://puppetdb1.example.com:8080
  - name: pdb-b
    url: https://puppetdb2.example.com
merge: freshness             # freshness | static
timeout: 10s                 # per-upstream request timeout
freshness_ttl: 30s           # freshness-map cache TTL (freshness merge only)
facts_ttl: 30s               # /facts + /nodes response cache TTL; 0 disables, capped at 30s
facts_cache_bytes: 67108864  # byte budget for that cache (64 MiB), LRU-evicted
source_fact: pdbmux_source   # name of the synthetic provenance fact
source_fact_enabled: true    # false serves backends' records untouched
health_probe_enabled: true          # false queries every backend on every request
health_probe_path: /status/v1/services  # backend health endpoint
health_probe_interval: 10s          # how often each backend is probed
health_probe_timeout: 5s            # per-probe timeout
health_probe_failures: 3            # consecutive failures before a backend is skipped
health_probe_successes: 2           # consecutive successes before it is used again

backends[*].url is a base URL (scheme://host[:port]); pdbmux appends the /pdb/query/v4/... path per request.

Env var	Overrides
PDBMUX_CONFIG	config file path (not a file key)
PDBMUX_LISTEN	listen
PDBMUX_MERGE	merge
PDBMUX_TIMEOUT	timeout (Go duration, e.g. 10s)
PDBMUX_FRESHNESS_TTL	freshness_ttl
PDBMUX_FACTS_TTL	facts_ttl (clamped to 30s)
PDBMUX_FACTS_CACHE_BYTES	facts_cache_bytes (plain integer bytes)
PDBMUX_BACKENDS	whole backend list, as name=url,name=url
PDBMUX_SOURCE_FACT	source_fact (default pdbmux_source)
PDBMUX_SOURCE_FACT_ENABLED	source_fact_enabled (default true); false disables injection
PDBMUX_HEALTH_PROBE_ENABLED	health_probe_enabled (default true)
PDBMUX_HEALTH_PROBE_PATH	health_probe_path (default /status/v1/services)
PDBMUX_HEALTH_PROBE_INTERVAL	health_probe_interval (Go duration)
PDBMUX_HEALTH_PROBE_TIMEOUT	health_probe_timeout (Go duration)
PDBMUX_HEALTH_PROBE_FAILURES	health_probe_failures (plain integer, minimum 1)
PDBMUX_HEALTH_PROBE_SUCCESSES	health_probe_successes (plain integer, minimum 1)

Flags: --config, --listen, --merge, --health-probe.

config init writes to --config/PDBMUX_CONFIG when set, else to $XDG_CONFIG_HOME/pdbmux/config.yaml.

Running

Subcommands: serve (default), config init, config show, version. Run pdbmux --help for details. Any PuppetDB v4 client works against the pdbmux base URL in place of a PuppetDB one.

PDBMUX_BACKENDS='pdb-a=http://puppetdb1.example.com:8080,pdb-b=http://puppetdb2.example.com:8080' pdbmux
curl -s --get http://localhost:8080/pdb/query/v4/nodes \
  --data-urlencode 'query=["=","certname","host1.example.com"]'

Build

make build (static binary into dist/), make test, make lint. Requires Go 1.25+.

End-to-end tests

make e2e runs the suite against real PuppetDB backends: two openvoxdb containers, each on its own PostgreSQL, loaded over the command API (replace facts v5, store report v8, replace catalog v9, deactivate node v3) and queried through pdbmux. Two real clients — Puppetboard and, when a binary is available, node-lookup — are pointed at pdbmux and asserted on. It needs a container runtime and takes a couple of minutes, so it sits behind the e2e build tag and never runs as part of make test or go test ./....

Commands are submitted with secondsToWaitForCompletion, so the harness waits on PuppetDB actually processing each one rather than sleeping, and the fixture load ends by polling queue_depth on /status/v1/services until both backends have drained.

Env var	Overrides
PDBMUX_E2E_OPENVOXDB_IMAGE	ghcr.io/openvoxproject/openvoxdb:8.15.0
PDBMUX_E2E_POSTGRES_IMAGE	docker.io/library/postgres:17-alpine
PDBMUX_E2E_PUPPETBOARD_IMAGE	ghcr.io/voxpupuli/puppetboard:latest
PDBMUX_E2E_TUNNEL_IMAGE	docker.io/library/alpine:3
PDBMUX_E2E_NODE_LOOKUP	path to a node-lookup binary (else PATH, else skipped)

Every test asserts; the suite records no known gaps.

Deployment

Container image only — no OS package. Every v* tag builds and pushes the image (.woodpecker/docker.yaml); registry and repository are pipeline settings. Tag with make patch / minor / major.

A static (CGO_ENABLED=0) binary on a distroless base. Configure it with PDBMUX_* env vars (at minimum PDBMUX_BACKENDS), or mount a config file — a configmap at /etc/pdbmux/config.yaml is picked up with no env var at all, and any other mount path works via PDBMUX_CONFIG. Env vars still override file values, so the two mix. Run as many replicas as you like — the only state is the in-memory cache, which is per-replica and bounded by facts_cache_bytes, so size the memory limit above it. Use /healthz for liveness/readiness probes.