Every backend gets the same query, so one they all refuse is the client's
mistake; flattening it into "all backends failed" threw openvoxdb's own
explanation away and logged a typo as an outage.
- Carry status, content type and body on a typed upstreamError
- Replay the status and explanation when every backend refuses alike
- Redact backend addresses from replayed bodies
- Keep a refused query out of the partial counters and the cache
distinct_resources sends /events to openvoxdb's legacy compiler, which has no
function or group_by, so every backend failed and the client saw a generic 502
instead of the reason.
- Refuse an aggregate carrying a truthy distinct_resources with 400, before any fan-out
- Read the param the way openvoxdb does, so any capitalisation of "true" counts
- Leave non-aggregate distinct_resources queries and every other route alone
- Document the refusal
An extract carrying a ["function", ...] column returns counts, not events, so
the union's verbatim-record key folded two backends' identical rows into one
number.
- Route /events through the central aggregate guard with its own fan-out path
- Drop its unsummed opt-out so the route-table property tests cover it
- Document the combined path on /events
A repeated extract function names one response column twice, which openvoxdb
aliases as <name>_2: unknown to the merge spec, it froze at the first backend's
value. A limit pushed upstream truncated each backend's groups before the
cross-backend fold, so a group could be partly counted or missed.
- Refuses any extract projecting one response column twice, naming the clash
- Fetches every group and applies limit/offset after the fold
- Documents the float64 avg divergence from Postgres numeric
sumRows folded every numeric column by addition, which is only correct
for count and sum, so min/max returned a sum, avg an average of
averages, and a to_string extract collapsed into one empty-key row.
- Combine count and sum by adding, min and max by the extreme, on text
columns as well as numeric ones
- Rewrite an avg extract into an upstream sum and count and divide the
totals, answering under the avg key the client asked for
- Refuse an aggregate pdbmux cannot merge with 400 naming the clash
- Treat to_string and jsonb_typeof as row functions that group rather
than fold, and key groups on every non-aggregate projected column
- Give the e2e fixture per-node resource line numbers and titles whose
extremes differ per backend
A merged route that forgets the parseAggregate check silently returns one
backend's rows: an aggregate row carries no certname, hash or name, so the
identity-keyed merges collapse every backend's numbers into one. That is how
/facts/<name> shipped broken.
- Resolve every /pdb/query/v4 request through one route table
- Sum extract/function queries in handleQuery, before any route's handler runs
- Make each route that is not summed name its reason; the zero value is guarded
- Assert the guard over the route table, so a new route inherits the assertion
The <value> path segment is client-supplied and reached a full unfiltered
/facts fan-out, keyed per value, so every distinct value was a fresh
whole-estate query and a fresh cache entry.
- validate <value> against the configured backend names, answering [] with
no fan-out when it names none
- key the drilldown's fetch on the fact name alone and apply <value> to the
shared record set, so all values share one entry and one fan-out
- report every configured backend on the no-fan-out empty response, which is
complete rather than partial
/fact-names advertises the fact, so its drilldown must not be a dead link.
- Serve the source fact's own path from the /facts merge that produces the
records, so certname set, owner and environment match /facts.
- Filter /facts/<source-fact>/<value> by the owning backend.
- Keep the aggregate, query-gate and disabled paths answering as before.
An aggregate row carries no certname, so the per-certname merge collapsed
every backend's row into one bucket and served a single backend's count as
the estate's — no error, no warning, X-Backends still 2/2.
- Route an aggregate query on /facts/<name>[/<value>] to the summing path.
- List the injected fact's name in /fact-names instead of hiding it.
- Reject an order_by on any field but name, as the backends do.
Both fell to the unmerged pass-through, so one backend's answer was served
as if it were the estate's: Puppetboard's fact drilldown lost the other
backend's nodes and its facts overview lost that backend's fact names.
- Serve /facts/<name> and /facts/<name>/<value> through the /facts merge.
- Serve /fact-names as a deduped, re-sorted, re-paged union of name arrays.
- Gate provenance on the path: only /facts/<source-fact> may be injected.
- Keep the owned fact name out of /fact-names while the feature is on.
- Cache both alongside the merged /facts and /nodes record sets.
- Turn the two recorded e2e gaps into positive assertions.
A `/facts` aggregate row carries no certname, so the per-certname fact
merge collapsed every backend's rows into one bucket and served a single
backend's numbers.
- Route a `/facts` query whose `extract` carries a `function` column to
serveSummed, as /nodes, /resources and /reports already do
- Document which extract functions combine correctly across backends
Resetting the run on every change of outcome kind meant only consecutive
same-kind probes accumulated, so a backend failing every probe but
alternating kinds — a fronting proxy flipping 404 and 503 — never reached
the failure threshold and kept taking every query.
- count one run of consecutive not-OK probes for the down decision, and use
the kinds only to pick which non-healthy state the run enters
- send a run containing any real failure to unhealthy; only a run of nothing
but refusals enters probe_unsupported and stays in service
- classify 429 as a real failure: it is the backend reporting its own
capacity, so an overloaded backend gets backed off
- document that probe_unsupported means "not verified" and that reachable is
the /healthz field carrying actual reachability
A backend that 404s on the health probe path but serves queries fine was
marked down and excluded from every fan-out for good, since the fail-open
only triggers when no backend is left healthy.
Classify a probe reply that refuses the request itself - any 4xx, plus 501 -
as evidence about the probe, not the backend. Such a backend keeps serving
queries and reports the distinct probe_unsupported state on /healthz.
Transport failures and 5xx, 503 included, still mark a backend down.
Log the misconfiguration once per transition with the backend, probe path
and status. Track failure runs per outcome kind so a 404 run and a 503 run
never add up to one threshold.
A down backend costs a full timeout stall on every request, since fan-out
has no way to know before it asks, and the client is never told the answer
came from fewer backends than are configured.
- poll each backend's status endpoint in the background, one goroutine per
backend, with failure/success thresholds so a blip cannot flap it
- skip backends the prober has down, and fall open to querying all of them
when none is left healthy
- treat a not-yet-probed backend as healthy so a restart drops no traffic
- log only up/down transitions
- stamp merged responses with X-Backends: <contributed>/<configured>
- report per-backend probe state and the last round's partiality on /healthz
- add health_probe_enabled, health_probe_path, health_probe_interval,
health_probe_timeout, health_probe_failures and health_probe_successes,
with matching PDBMUX_* env vars and a --health-probe flag
Reference-count flightCall so the fan-out context ends with the last
caller waiting on it, keeping cfg.Timeout as the upper bound. A leader
leaving with a follower still parked no longer disturbs the flight, and
a solo requester disconnecting releases the upstream sockets at once
instead of holding them for the whole timeout.
Return errFlightAbandoned from Do rather than inferring the abandon path
from the request context's sentinel, and read Age off the server's clock
so it matches the timestamp the cache stored.
A stale fallback is byte-identical to a fresh response, so a client has
no way to tell it is holding data pdbmux served only because every
backend was down; the sole signal is a log line and a /healthz counter.
Set X-Cache to hit, miss or stale and Age to whole seconds since the
served copy was stored on every response from a cached path. Neither
header is emitted by OpenVoxDB, so nothing upstream is shadowed.
A single flight is built by whichever request arrived first, but every
request on that key waits for it. Running the fan-out on the leader's
cancelable request context hands the leader's disconnect to followers
whose own connections are healthy: they get 502 all backends failed.
Waiters also parked on a WaitGroup, so a follower whose own client went
away stayed blocked until the leader finished.
Run the flight on a context detached from the leader's request and
bounded by the configured timeout, and pass that context into build so
the fan-out uses it. Give Do a context so a waiter can abandon a flight
it no longer needs; the leader ignores it and always runs fn to
completion, keeping the cache warm for the others. A caller that
abandons on its own cancellation writes no response.
A busy Puppetboard re-fans-out the same /facts query every few seconds, and a
502 is worse than 30-second-old facts when every PuppetDB is unreachable.
- Add a `Cache` interface (get reports fresh/stale/miss, put, stats) keyed on
`<path>?<params>` with keys and repeated values sorted, plus a no-op default
so uncached paths behave exactly as before.
- Route serveMerged/serveUnion/serveSummed through `serveCached`, so the
reports cache drops in at `cacheFor` without touching a handler.
- Back /facts and /nodes with a byte-bounded LRU: `facts_ttl` (default 30s,
clamped to a 30s cap) and `facts_cache_bytes` (default 64 MiB); expired
entries are kept and served only when every backend fails.
- Single-flight identical keys so N concurrent requests cause one fan-out.
- Surface `cache` state and `serving_stale` in /healthz and the cache settings
in `config show`.
- drop upstream facts of the configured name whenever the feature is enabled,
independent of the per-query injection gate, and log the drop once per request
- walk the whole AST for a nested extract and skip injection when one is found
outside an in subquery
- skip the environment scan on /facts when nothing is injected
- document the override rule and that PQL-syntax queries never get the fact
## Why
With several PuppetDBs behind one endpoint, consumers cannot tell which backend a node's data came from.
## How
- Add a synthetic `pdbmux_source` fact per certname on `/facts`, valued with the backend that won the facts merge, and stamp the same key on merged `/nodes` records.
- Emit all four fact keys including `environment`, which clients index directly.
- Skip injection for top-level `extract` queries, so `count()` and other aggregates keep the backends' own numbers, and for `/facts` queries constraining `name`; a `name` filter inside an `in` subquery still injects.
- Replace, never duplicate, an upstream fact of the configured name.
- Configure via `source_fact` / `source_fact_enabled` (`PDBMUX_SOURCE_FACT`, `PDBMUX_SOURCE_FACT_ENABLED`), defaulting to `pdbmux_source` enabled.
## Why
Puppetboard 7.0.1 cannot run against pdbmux: it exits at import when
/pdb/meta/v1/version 404s, and its landing page, metrics and radiator views
404 on the Jolokia surface.
## How
- Serve /pdb/meta/v1/version, reporting the lowest version any backend runs,
and /pdb/meta/v1/server-time from the first reachable backend.
- Merge the Jolokia surface (/metrics/v2/read, /metrics/v2/list,
/metrics/v1/mbeans): objects union, numeric attributes sum by default, and
Min/Max/Uptime/StartTime plus the distribution stats take a bound or a mean.
- Route /nodes extract-count queries to the summing path ahead of the certname
merge, and give /resources aggregates the same path.
- Document the endpoints and merge semantics in the README.
- Cover version disagreement, metric rules, escaped MBean names, count summing
and the non-aggregate /nodes merge with httptest backends.
An extract/count()/group_by query returns synthetic rows with no report
hash, which reportKey fell back to keying by verbatim raw bytes. Two
backends emitting a byte-identical aggregate row (e.g.
{"status":"changed","count":1}) therefore collapsed into one, silently
undercounting the merged result and contradicting the documented
guarantee that no backend's rows are dropped.
Give the mergeUnion key func an ok return: false means the record has no
dedupe identity and is always kept. reportKey returns ok=false for
hash-less rows; hash-keyed report dedupe and event verbatim-identity
dedupe are unchanged.
Add TestMergeUnion_IdenticalHashlessRowsAreNotCollapsed covering the
collision case, and reword the README line to say aggregate rows pass
through even when byte-identical.
Reports are immutable history, so a node that migrated has reports in the
old PuppetDB and the new one; serve the union rather than picking a single
owning backend as /facts does.
Re-apply order_by/limit/offset over the merged set and sum X-Records, since
each backend only orders and pages its own slice.
Split out from node-lookup PR #17 into its own repo. pdbmux presents a
single merged PuppetDB v4 query surface over the old (Consul) and new
(k8s) PuppetDBs during the VM to k8s migration, and is deployed in-cluster
via argocd-apps as a container image.
unkin-agent
benvin