openvoxdb does not reserve 5xx for its own faults: the same malformed
query is a 400 on /nodes and a 500 on /facts, and /metrics answers a flat
403, so a 4xx-only replay rule made pdbmux's behaviour depend on the
route. The meta and metrics handlers held their own copy of the gateway
error and bypassed the replay entirely.
- Replay any status from 400 up that every backend agreed on, with the
backend's own body and content type.
- Keep 502 for backends disagreeing on the status, or a backend that
answered nothing at all.
- Route /pdb/meta, /metrics and the pass-through path through the same
rule as the merged query handlers.
- Count a unanimous 5xx as a failed round and let it fall back to a stale
cache entry; only a unanimous 4xx stays exempt from both.
- Answer successful queries with openvoxdb's application/json;charset=utf-8.
The paths keyed on one certname took the pass-through, so a node both
backends hold answered from whichever was configured first while /facts
answered from whichever held its newer report.
- add a route claiming /pdb/query/v4/{nodes,factsets,catalogs}/<certname>
- order the backends for it by the freshness map the /facts merge uses
- try the remaining backends after the owner, replaying upstream's 404 when none holds the certname
- assert ownership, the upstream 404 body, a failed backend and query forwarding against captured openvoxdb shapes
DNS names are case-insensitive, so a backend named back in a different
case escaped redaction and reached the client.
- Fold ASCII case when replacing a backend's URL, host and hostname
- Pin the redaction vectors, the same-status rule and the bare-GET
premise behind treating 406 and 415 as replayable
Every backend gets the same query, so one they all refuse is the client's
mistake; flattening it into "all backends failed" threw openvoxdb's own
explanation away and logged a typo as an outage.
- Carry status, content type and body on a typed upstreamError
- Replay the status and explanation when every backend refuses alike
- Redact backend addresses from replayed bodies
- Keep a refused query out of the partial counters and the cache
distinct_resources sends /events to openvoxdb's legacy compiler, which has no
function or group_by, so every backend failed and the client saw a generic 502
instead of the reason.
- Refuse an aggregate carrying a truthy distinct_resources with 400, before any fan-out
- Read the param the way openvoxdb does, so any capitalisation of "true" counts
- Leave non-aggregate distinct_resources queries and every other route alone
- Document the refusal
An extract carrying a ["function", ...] column returns counts, not events, so
the union's verbatim-record key folded two backends' identical rows into one
number.
- Route /events through the central aggregate guard with its own fan-out path
- Drop its unsummed opt-out so the route-table property tests cover it
- Document the combined path on /events
Asserting an absolute runtime.NumGoroutine() fails roughly one run in
three under -tags e2e, where the harness keeps a live server, its prober
and testcontainers goroutines alive alongside these tests.
- Match goroutine stacks naming the flight group, fan-out or prober
- Derive those frame names from method expressions
- Compare against a baseline sampled the same way
- Poll the prober's settle check instead of sampling it once
An aggregate reaching a guarded route through the dispatch table folds each
column by its own operation, so max is a maximum rather than a blanket sum.
A group_by key that repeats a folded column made the column a grouping
key and an aggregate at once, and for avg it left the upstream query
grouping on a column the rewrite had removed, so the request failed as
an opaque 502.
- refuse a group_by field that names a folded aggregate or the avg column
- cover the function-then-field ordering of the existing clash check
A repeated extract function names one response column twice, which openvoxdb
aliases as <name>_2: unknown to the merge spec, it froze at the first backend's
value. A limit pushed upstream truncated each backend's groups before the
cross-backend fold, so a group could be partly counted or missed.
- Refuses any extract projecting one response column twice, naming the clash
- Fetches every group and applies limit/offset after the fold
- Documents the float64 avg divergence from Postgres numeric
sumRows folded every numeric column by addition, which is only correct
for count and sum, so min/max returned a sum, avg an average of
averages, and a to_string extract collapsed into one empty-key row.
- Combine count and sum by adding, min and max by the extreme, on text
columns as well as numeric ones
- Rewrite an avg extract into an upstream sum and count and divide the
totals, answering under the avg key the client asked for
- Refuse an aggregate pdbmux cannot merge with 400 naming the clash
- Treat to_string and jsonb_typeof as row functions that group rather
than fold, and key groups on every non-aggregate projected column
- Give the e2e fixture per-node resource line numbers and titles whose
extremes differ per backend
A merged route that forgets the parseAggregate check silently returns one
backend's rows: an aggregate row carries no certname, hash or name, so the
identity-keyed merges collapse every backend's numbers into one. That is how
/facts/<name> shipped broken.
- Resolve every /pdb/query/v4 request through one route table
- Sum extract/function queries in handleQuery, before any route's handler runs
- Make each route that is not summed name its reason; the zero value is guarded
- Assert the guard over the route table, so a new route inherits the assertion
The empty answer for an unknown value never reaches a backend, so it must
leave /healthz reporting a whole estate.
- assert partial_rounds stays 0 after an unknown-value drilldown
The <value> path segment is client-supplied and reached a full unfiltered
/facts fan-out, keyed per value, so every distinct value was a fresh
whole-estate query and a fresh cache entry.
- validate <value> against the configured backend names, answering [] with
no fan-out when it names none
- key the drilldown's fetch on the fact name alone and apply <value> to the
shared record set, so all values share one entry and one fan-out
- report every configured backend on the no-fan-out empty response, which is
complete rather than partial
/fact-names advertises the fact, so its drilldown must not be a dead link.
- Serve the source fact's own path from the /facts merge that produces the
records, so certname set, owner and environment match /facts.
- Filter /facts/<source-fact>/<value> by the owning backend.
- Keep the aggregate, query-gate and disabled paths answering as before.
An aggregate row carries no certname, so the per-certname merge collapsed
every backend's row into one bucket and served a single backend's count as
the estate's — no error, no warning, X-Backends still 2/2.
- Route an aggregate query on /facts/<name>[/<value>] to the summing path.
- List the injected fact's name in /fact-names instead of hiding it.
- Reject an order_by on any field but name, as the backends do.
Both fell to the unmerged pass-through, so one backend's answer was served
as if it were the estate's: Puppetboard's fact drilldown lost the other
backend's nodes and its facts overview lost that backend's fact names.
- Serve /facts/<name> and /facts/<name>/<value> through the /facts merge.
- Serve /fact-names as a deduped, re-sorted, re-paged union of name arrays.
- Gate provenance on the path: only /facts/<source-fact> may be injected.
- Keep the owned fact name out of /fact-names while the feature is on.
- Cache both alongside the merged /facts and /nodes record sets.
- Turn the two recorded e2e gaps into positive assertions.
A `/facts` aggregate row carries no certname, so the per-certname fact
merge collapsed every backend's rows into one bucket and served a single
backend's numbers.
- Route a `/facts` query whose `extract` carries a `function` column to
serveSummed, as /nodes, /resources and /reports already do
- Document which extract functions combine correctly across backends
Every merge rule, AST gate and provenance decision was derived from reading
upstream source and proven only against fake backends, so nothing had ever run
against a real PuppetDB.
- Add an e2e suite behind the `e2e` build tag and a `make e2e` target
- Stand up two openvoxdb backends on their own PostgreSQL with testcontainers
- Load facts, reports and catalogs over the command API, waiting on processing
- Assert the union, freshness dedupe, summed aggregates, provenance gating,
X-Backends, backend death and recovery, and the report paths
- Drive Puppetboard and node-lookup against pdbmux as real clients
- Record three known gaps as skips that fail once the gap closes
The decaying failure window let a repeating cycle of one failure and a run
of rejections readmit a dead backend once per cycle, forever: any rule that
readmits on "no real failure lately" flaps under a periodic pattern.
Replace it with a per-backend latch: has this probe endpoint ever answered
with a verdict we can read? Until it has, there is no health signal, so the
backend is never gated and stays in service as probe_unsupported. Once it
has, the path works and every unsuccessful probe counts, rejections
included. The latch never clears, so no pattern can argue a backend back in.
A backend whose health_probe_path is wrong rejects every probe, so no
success can ever arrive to clear the sticky run-failed flag. One transient
failure in the middle of those rejections excluded such a backend from the
pool for the life of the process, reintroducing the stranding bug.
Decide unhealthy vs probe_unsupported on whether a real failure landed
within the last health_probe_failures probes, floored at 3 so the window
always spans a reject/reject/failure cycle. Sustained alternation keeps a
failure in every window and stays unhealthy; an aged-out failure leaves a
pure-rejection run on probe_unsupported and back in service.
Resetting the run on every change of outcome kind meant only consecutive
same-kind probes accumulated, so a backend failing every probe but
alternating kinds — a fronting proxy flipping 404 and 503 — never reached
the failure threshold and kept taking every query.
- count one run of consecutive not-OK probes for the down decision, and use
the kinds only to pick which non-healthy state the run enters
- send a run containing any real failure to unhealthy; only a run of nothing
but refusals enters probe_unsupported and stays in service
- classify 429 as a real failure: it is the backend reporting its own
capacity, so an overloaded backend gets backed off
- document that probe_unsupported means "not verified" and that reachable is
the /healthz field carrying actual reachability
A backend that 404s on the health probe path but serves queries fine was
marked down and excluded from every fan-out for good, since the fail-open
only triggers when no backend is left healthy.
Classify a probe reply that refuses the request itself - any 4xx, plus 501 -
as evidence about the probe, not the backend. Such a backend keeps serving
queries and reports the distinct probe_unsupported state on /healthz.
Transport failures and 5xx, 503 included, still mark a backend down.
Log the misconfiguration once per transition with the backend, probe path
and status. Track failure runs per outcome kind so a 404 run and a 503 run
never add up to one threshold.
A down backend costs a full timeout stall on every request, since fan-out
has no way to know before it asks, and the client is never told the answer
came from fewer backends than are configured.
- poll each backend's status endpoint in the background, one goroutine per
backend, with failure/success thresholds so a blip cannot flap it
- skip backends the prober has down, and fall open to querying all of them
when none is left healthy
- treat a not-yet-probed backend as healthy so a restart drops no traffic
- log only up/down transitions
- stamp merged responses with X-Backends: <contributed>/<configured>
- report per-backend probe state and the last round's partiality on /healthz
- add health_probe_enabled, health_probe_path, health_probe_interval,
health_probe_timeout, health_probe_failures and health_probe_successes,
with matching PDBMUX_* env vars and a --health-probe flag
Rebasing onto main brought in the per-request sourceInjector, which now
runs inside the cached build: what a cache entry holds is the merged body
with pdbmux_source already stamped and upstream records of that name
already dropped.
Injecting on the way out instead would mean storing the un-injected
records plus a per-certname backend map and re-marshalling every record on
every hit, which is the work the cache exists to avoid. Baking it in stays
correct because the value names the backend that supplied the data — a
property of that fetch, not of the caller reading it — so it ages out with
the body it labels, and because the injection gate is a pure function of
path and query, both of which are already in the cache key.
Update the two cache tests whose byte-exact bodies predate the fact, and
add tests for the composition: attribution survives a cache hit on /facts
and /nodes, it ages with its entry rather than tracking a node that moved,
gated and ungated queries cache separately, and suppression of an upstream
fact of that name survives into the entry.
Reference-count flightCall so the fan-out context ends with the last
caller waiting on it, keeping cfg.Timeout as the upper bound. A leader
leaving with a follower still parked no longer disturbs the flight, and
a solo requester disconnecting releases the upstream sockets at once
instead of holding them for the whole timeout.
Return errFlightAbandoned from Do rather than inferring the abandon path
from the request context's sentinel, and read Age off the server's clock
so it matches the timestamp the cache stored.
A stale fallback is byte-identical to a fresh response, so a client has
no way to tell it is holding data pdbmux served only because every
backend was down; the sole signal is a log line and a /healthz counter.
Set X-Cache to hit, miss or stale and Age to whole seconds since the
served copy was stored on every response from a cached path. Neither
header is emitted by OpenVoxDB, so nothing upstream is shadowed.
A single flight is built by whichever request arrived first, but every
request on that key waits for it. Running the fan-out on the leader's
cancelable request context hands the leader's disconnect to followers
whose own connections are healthy: they get 502 all backends failed.
Waiters also parked on a WaitGroup, so a follower whose own client went
away stayed blocked until the leader finished.
Run the flight on a context detached from the leader's request and
bounded by the configured timeout, and pass that context into build so
the fan-out uses it. Give Do a context so a waiter can abandon a flight
it no longer needs; the leader ignores it and always runs fn to
completion, keeping the cache warm for the others. A caller that
abandons on its own cancellation writes no response.
- flightGroup.Do recovers a panicking fn so the leader and every waiter get a non-nil error instead of a zero-value success served as 200 []
- Note in the README that facts_cache_bytes budgets body bytes only
A busy Puppetboard re-fans-out the same /facts query every few seconds, and a
502 is worse than 30-second-old facts when every PuppetDB is unreachable.
- Add a `Cache` interface (get reports fresh/stale/miss, put, stats) keyed on
`<path>?<params>` with keys and repeated values sorted, plus a no-op default
so uncached paths behave exactly as before.
- Route serveMerged/serveUnion/serveSummed through `serveCached`, so the
reports cache drops in at `cacheFor` without touching a handler.
- Back /facts and /nodes with a byte-bounded LRU: `facts_ttl` (default 30s,
clamped to a 30s cap) and `facts_cache_bytes` (default 64 MiB); expired
entries are kept and served only when every backend fails.
- Single-flight identical keys so N concurrent requests cause one fan-out.
- Surface `cache` state and `serving_stale` in /healthz and the cache settings
in `config show`.
unkin-agent
benvin