Both sides added jsonContentType; keep one. The no-entry 502 test kills the
backends outright, since main replays a unanimous 500, and the cached
content-type test moves to /nodes, the only cached path left.
Unanimity is the right rule for a fan-out of peers, but the per-certname
routes are not one: a backend that does not hold the certname answers 404
to say so, not to disagree, so requiring it to agree turned the owner's
real 500 into a 502 that described neither backend.
- Add askOrder, which says whether a set of backends was asked as peers
or owner-first, and resolve each round's replies under its own rule.
- Serve the first backend that answered on owner-routed paths, so an
unreachable owner still falls back rather than collapsing to 502.
- Keep unanimity for the merged, meta, metrics and pass-through routes.
- Cover the owner routes: owner errors against a non-owner 404, both
erroring differently, an unreachable owner, and a non-owner error
behind the owner's 200.
- Record what clientRefusal's 4xx exemption assumes about client certs.
openvoxdb does not reserve 5xx for its own faults: the same malformed
query is a 400 on /nodes and a 500 on /facts, and /metrics answers a flat
403, so a 4xx-only replay rule made pdbmux's behaviour depend on the
route. The meta and metrics handlers held their own copy of the gateway
error and bypassed the replay entirely.
- Replay any status from 400 up that every backend agreed on, with the
backend's own body and content type.
- Keep 502 for backends disagreeing on the status, or a backend that
answered nothing at all.
- Route /pdb/meta, /metrics and the pass-through path through the same
rule as the merged query handlers.
- Count a unanimous 5xx as a failed round and let it fall back to a stale
cache entry; only a unanimous 4xx stays exempt from both.
- Answer successful queries with openvoxdb's application/json;charset=utf-8.
Fact answers must be as current as a backend's own, and X-Cache/Age are
headers PuppetDB never sends.
- serve /facts, /facts/<name>[/<value>] and /fact-names live on every request
- keep the in-memory cache on merged /nodes only
- drop X-Cache and Age everywhere; /healthz still reports cache state
- answer successful queries with PuppetDB's application/json;charset=utf-8
The paths keyed on one certname took the pass-through, so a node both
backends hold answered from whichever was configured first while /facts
answered from whichever held its newer report.
- add a route claiming /pdb/query/v4/{nodes,factsets,catalogs}/<certname>
- order the backends for it by the freshness map the /facts merge uses
- try the remaining backends after the owner, replaying upstream's 404 when none holds the certname
- assert ownership, the upstream 404 body, a failed backend and query forwarding against captured openvoxdb shapes
DNS names are case-insensitive, so a backend named back in a different
case escaped redaction and reached the client.
- Fold ASCII case when replacing a backend's URL, host and hostname
- Pin the redaction vectors, the same-status rule and the bare-GET
premise behind treating 406 and 415 as replayable
Every backend gets the same query, so one they all refuse is the client's
mistake; flattening it into "all backends failed" threw openvoxdb's own
explanation away and logged a typo as an outage.
- Carry status, content type and body on a typed upstreamError
- Replay the status and explanation when every backend refuses alike
- Redact backend addresses from replayed bodies
- Keep a refused query out of the partial counters and the cache
distinct_resources sends /events to openvoxdb's legacy compiler, which has no
function or group_by, so every backend failed and the client saw a generic 502
instead of the reason.
- Refuse an aggregate carrying a truthy distinct_resources with 400, before any fan-out
- Read the param the way openvoxdb does, so any capitalisation of "true" counts
- Leave non-aggregate distinct_resources queries and every other route alone
- Document the refusal
An extract carrying a ["function", ...] column returns counts, not events, so
the union's verbatim-record key folded two backends' identical rows into one
number.
- Route /events through the central aggregate guard with its own fan-out path
- Drop its unsummed opt-out so the route-table property tests cover it
- Document the combined path on /events
Asserting an absolute runtime.NumGoroutine() fails roughly one run in
three under -tags e2e, where the harness keeps a live server, its prober
and testcontainers goroutines alive alongside these tests.
- Match goroutine stacks naming the flight group, fan-out or prober
- Derive those frame names from method expressions
- Compare against a baseline sampled the same way
- Poll the prober's settle check instead of sampling it once
An aggregate reaching a guarded route through the dispatch table folds each
column by its own operation, so max is a maximum rather than a blanket sum.
A group_by key that repeats a folded column made the column a grouping
key and an aggregate at once, and for avg it left the upstream query
grouping on a column the rewrite had removed, so the request failed as
an opaque 502.
- refuse a group_by field that names a folded aggregate or the avg column
- cover the function-then-field ordering of the existing clash check
A repeated extract function names one response column twice, which openvoxdb
aliases as <name>_2: unknown to the merge spec, it froze at the first backend's
value. A limit pushed upstream truncated each backend's groups before the
cross-backend fold, so a group could be partly counted or missed.
- Refuses any extract projecting one response column twice, naming the clash
- Fetches every group and applies limit/offset after the fold
- Documents the float64 avg divergence from Postgres numeric
sumRows folded every numeric column by addition, which is only correct
for count and sum, so min/max returned a sum, avg an average of
averages, and a to_string extract collapsed into one empty-key row.
- Combine count and sum by adding, min and max by the extreme, on text
columns as well as numeric ones
- Rewrite an avg extract into an upstream sum and count and divide the
totals, answering under the avg key the client asked for
- Refuse an aggregate pdbmux cannot merge with 400 naming the clash
- Treat to_string and jsonb_typeof as row functions that group rather
than fold, and key groups on every non-aggregate projected column
- Give the e2e fixture per-node resource line numbers and titles whose
extremes differ per backend
A merged route that forgets the parseAggregate check silently returns one
backend's rows: an aggregate row carries no certname, hash or name, so the
identity-keyed merges collapse every backend's numbers into one. That is how
/facts/<name> shipped broken.
- Resolve every /pdb/query/v4 request through one route table
- Sum extract/function queries in handleQuery, before any route's handler runs
- Make each route that is not summed name its reason; the zero value is guarded
- Assert the guard over the route table, so a new route inherits the assertion
The empty answer for an unknown value never reaches a backend, so it must
leave /healthz reporting a whole estate.
- assert partial_rounds stays 0 after an unknown-value drilldown
The <value> path segment is client-supplied and reached a full unfiltered
/facts fan-out, keyed per value, so every distinct value was a fresh
whole-estate query and a fresh cache entry.
- validate <value> against the configured backend names, answering [] with
no fan-out when it names none
- key the drilldown's fetch on the fact name alone and apply <value> to the
shared record set, so all values share one entry and one fan-out
- report every configured backend on the no-fan-out empty response, which is
complete rather than partial
/fact-names advertises the fact, so its drilldown must not be a dead link.
- Serve the source fact's own path from the /facts merge that produces the
records, so certname set, owner and environment match /facts.
- Filter /facts/<source-fact>/<value> by the owning backend.
- Keep the aggregate, query-gate and disabled paths answering as before.
An aggregate row carries no certname, so the per-certname merge collapsed
every backend's row into one bucket and served a single backend's count as
the estate's — no error, no warning, X-Backends still 2/2.
- Route an aggregate query on /facts/<name>[/<value>] to the summing path.
- List the injected fact's name in /fact-names instead of hiding it.
- Reject an order_by on any field but name, as the backends do.
Both fell to the unmerged pass-through, so one backend's answer was served
as if it were the estate's: Puppetboard's fact drilldown lost the other
backend's nodes and its facts overview lost that backend's fact names.
- Serve /facts/<name> and /facts/<name>/<value> through the /facts merge.
- Serve /fact-names as a deduped, re-sorted, re-paged union of name arrays.
- Gate provenance on the path: only /facts/<source-fact> may be injected.
- Keep the owned fact name out of /fact-names while the feature is on.
- Cache both alongside the merged /facts and /nodes record sets.
- Turn the two recorded e2e gaps into positive assertions.
A `/facts` aggregate row carries no certname, so the per-certname fact
merge collapsed every backend's rows into one bucket and served a single
backend's numbers.
- Route a `/facts` query whose `extract` carries a `function` column to
serveSummed, as /nodes, /resources and /reports already do
- Document which extract functions combine correctly across backends
Every merge rule, AST gate and provenance decision was derived from reading
upstream source and proven only against fake backends, so nothing had ever run
against a real PuppetDB.
- Add an e2e suite behind the `e2e` build tag and a `make e2e` target
- Stand up two openvoxdb backends on their own PostgreSQL with testcontainers
- Load facts, reports and catalogs over the command API, waiting on processing
- Assert the union, freshness dedupe, summed aggregates, provenance gating,
X-Backends, backend death and recovery, and the report paths
- Drive Puppetboard and node-lookup against pdbmux as real clients
- Record three known gaps as skips that fail once the gap closes
The decaying failure window let a repeating cycle of one failure and a run
of rejections readmit a dead backend once per cycle, forever: any rule that
readmits on "no real failure lately" flaps under a periodic pattern.
Replace it with a per-backend latch: has this probe endpoint ever answered
with a verdict we can read? Until it has, there is no health signal, so the
backend is never gated and stays in service as probe_unsupported. Once it
has, the path works and every unsuccessful probe counts, rejections
included. The latch never clears, so no pattern can argue a backend back in.
A backend whose health_probe_path is wrong rejects every probe, so no
success can ever arrive to clear the sticky run-failed flag. One transient
failure in the middle of those rejections excluded such a backend from the
pool for the life of the process, reintroducing the stranding bug.
Decide unhealthy vs probe_unsupported on whether a real failure landed
within the last health_probe_failures probes, floored at 3 so the window
always spans a reject/reject/failure cycle. Sustained alternation keeps a
failure in every window and stays unhealthy; an aged-out failure leaves a
pure-rejection run on probe_unsupported and back in service.
Resetting the run on every change of outcome kind meant only consecutive
same-kind probes accumulated, so a backend failing every probe but
alternating kinds — a fronting proxy flipping 404 and 503 — never reached
the failure threshold and kept taking every query.
- count one run of consecutive not-OK probes for the down decision, and use
the kinds only to pick which non-healthy state the run enters
- send a run containing any real failure to unhealthy; only a run of nothing
but refusals enters probe_unsupported and stays in service
- classify 429 as a real failure: it is the backend reporting its own
capacity, so an overloaded backend gets backed off
- document that probe_unsupported means "not verified" and that reachable is
the /healthz field carrying actual reachability
A backend that 404s on the health probe path but serves queries fine was
marked down and excluded from every fan-out for good, since the fail-open
only triggers when no backend is left healthy.
Classify a probe reply that refuses the request itself - any 4xx, plus 501 -
as evidence about the probe, not the backend. Such a backend keeps serving
queries and reports the distinct probe_unsupported state on /healthz.
Transport failures and 5xx, 503 included, still mark a backend down.
Log the misconfiguration once per transition with the backend, probe path
and status. Track failure runs per outcome kind so a 404 run and a 503 run
never add up to one threshold.
A down backend costs a full timeout stall on every request, since fan-out
has no way to know before it asks, and the client is never told the answer
came from fewer backends than are configured.
- poll each backend's status endpoint in the background, one goroutine per
backend, with failure/success thresholds so a blip cannot flap it
- skip backends the prober has down, and fall open to querying all of them
when none is left healthy
- treat a not-yet-probed backend as healthy so a restart drops no traffic
- log only up/down transitions
- stamp merged responses with X-Backends: <contributed>/<configured>
- report per-backend probe state and the last round's partiality on /healthz
- add health_probe_enabled, health_probe_path, health_probe_interval,
health_probe_timeout, health_probe_failures and health_probe_successes,
with matching PDBMUX_* env vars and a --health-probe flag
Rebasing onto main brought in the per-request sourceInjector, which now
runs inside the cached build: what a cache entry holds is the merged body
with pdbmux_source already stamped and upstream records of that name
already dropped.
Injecting on the way out instead would mean storing the un-injected
records plus a per-certname backend map and re-marshalling every record on
every hit, which is the work the cache exists to avoid. Baking it in stays
correct because the value names the backend that supplied the data — a
property of that fetch, not of the caller reading it — so it ages out with
the body it labels, and because the injection gate is a pure function of
path and query, both of which are already in the cache key.
Update the two cache tests whose byte-exact bodies predate the fact, and
add tests for the composition: attribution survives a cache hit on /facts
and /nodes, it ages with its entry rather than tracking a node that moved,
gated and ungated queries cache separately, and suppression of an upstream
fact of that name survives into the entry.
benvin
unkin-agent