golang:1.25 and the stale almalinux9-gobuilder pin are replaced by
gobuilder:0.1.2-alma9 on every step that invokes the Go toolchain, with
GOCACHEPROG wired to the baked-in go-cache-plugin against the shared
S3 cache bucket.
- build, pre-commit, test: image -> gobuilder:0.1.2-alma9
- add GOCACHE_* env + AWS creds from org secrets, absolute cache-dir
- lint step (golangci-lint, no gobuilder binary) and docker buildx left untouched
Both sides added jsonContentType; keep one. The no-entry 502 test kills the
backends outright, since main replays a unanimous 500, and the cached
content-type test moves to /nodes, the only cached path left.
Unanimity is the right rule for a fan-out of peers, but the per-certname
routes are not one: a backend that does not hold the certname answers 404
to say so, not to disagree, so requiring it to agree turned the owner's
real 500 into a 502 that described neither backend.
- Add askOrder, which says whether a set of backends was asked as peers
or owner-first, and resolve each round's replies under its own rule.
- Serve the first backend that answered on owner-routed paths, so an
unreachable owner still falls back rather than collapsing to 502.
- Keep unanimity for the merged, meta, metrics and pass-through routes.
- Cover the owner routes: owner errors against a non-owner 404, both
erroring differently, an unreachable owner, and a non-owner error
behind the owner's 200.
- Record what clientRefusal's 4xx exemption assumes about client certs.
openvoxdb does not reserve 5xx for its own faults: the same malformed
query is a 400 on /nodes and a 500 on /facts, and /metrics answers a flat
403, so a 4xx-only replay rule made pdbmux's behaviour depend on the
route. The meta and metrics handlers held their own copy of the gateway
error and bypassed the replay entirely.
- Replay any status from 400 up that every backend agreed on, with the
backend's own body and content type.
- Keep 502 for backends disagreeing on the status, or a backend that
answered nothing at all.
- Route /pdb/meta, /metrics and the pass-through path through the same
rule as the merged query handlers.
- Count a unanimous 5xx as a failed round and let it fall back to a stale
cache entry; only a unanimous 4xx stays exempt from both.
- Answer successful queries with openvoxdb's application/json;charset=utf-8.
Fact answers must be as current as a backend's own, and X-Cache/Age are
headers PuppetDB never sends.
- serve /facts, /facts/<name>[/<value>] and /fact-names live on every request
- keep the in-memory cache on merged /nodes only
- drop X-Cache and Age everywhere; /healthz still reports cache state
- answer successful queries with PuppetDB's application/json;charset=utf-8
The paths keyed on one certname took the pass-through, so a node both
backends hold answered from whichever was configured first while /facts
answered from whichever held its newer report.
- add a route claiming /pdb/query/v4/{nodes,factsets,catalogs}/<certname>
- order the backends for it by the freshness map the /facts merge uses
- try the remaining backends after the owner, replaying upstream's 404 when none holds the certname
- assert ownership, the upstream 404 body, a failed backend and query forwarding against captured openvoxdb shapes
DNS names are case-insensitive, so a backend named back in a different
case escaped redaction and reached the client.
- Fold ASCII case when replacing a backend's URL, host and hostname
- Pin the redaction vectors, the same-status rule and the bare-GET
premise behind treating 406 and 415 as replayable
Every backend gets the same query, so one they all refuse is the client's
mistake; flattening it into "all backends failed" threw openvoxdb's own
explanation away and logged a typo as an outage.
- Carry status, content type and body on a typed upstreamError
- Replay the status and explanation when every backend refuses alike
- Redact backend addresses from replayed bodies
- Keep a refused query out of the partial counters and the cache
distinct_resources sends /events to openvoxdb's legacy compiler, which has no
function or group_by, so every backend failed and the client saw a generic 502
instead of the reason.
- Refuse an aggregate carrying a truthy distinct_resources with 400, before any fan-out
- Read the param the way openvoxdb does, so any capitalisation of "true" counts
- Leave non-aggregate distinct_resources queries and every other route alone
- Document the refusal
An extract carrying a ["function", ...] column returns counts, not events, so
the union's verbatim-record key folded two backends' identical rows into one
number.
- Route /events through the central aggregate guard with its own fan-out path
- Drop its unsummed opt-out so the route-table property tests cover it
- Document the combined path on /events
Asserting an absolute runtime.NumGoroutine() fails roughly one run in
three under -tags e2e, where the harness keeps a live server, its prober
and testcontainers goroutines alive alongside these tests.
- Match goroutine stacks naming the flight group, fan-out or prober
- Derive those frame names from method expressions
- Compare against a baseline sampled the same way
- Poll the prober's settle check instead of sampling it once
An aggregate reaching a guarded route through the dispatch table folds each
column by its own operation, so max is a maximum rather than a blanket sum.
A group_by key that repeats a folded column made the column a grouping
key and an aggregate at once, and for avg it left the upstream query
grouping on a column the rewrite had removed, so the request failed as
an opaque 502.
- refuse a group_by field that names a folded aggregate or the avg column
- cover the function-then-field ordering of the existing clash check
A repeated extract function names one response column twice, which openvoxdb
aliases as <name>_2: unknown to the merge spec, it froze at the first backend's
value. A limit pushed upstream truncated each backend's groups before the
cross-backend fold, so a group could be partly counted or missed.
- Refuses any extract projecting one response column twice, naming the clash
- Fetches every group and applies limit/offset after the fold
- Documents the float64 avg divergence from Postgres numeric
sumRows folded every numeric column by addition, which is only correct
for count and sum, so min/max returned a sum, avg an average of
averages, and a to_string extract collapsed into one empty-key row.
- Combine count and sum by adding, min and max by the extreme, on text
columns as well as numeric ones
- Rewrite an avg extract into an upstream sum and count and divide the
totals, answering under the avg key the client asked for
- Refuse an aggregate pdbmux cannot merge with 400 naming the clash
- Treat to_string and jsonb_typeof as row functions that group rather
than fold, and key groups on every non-aggregate projected column
- Give the e2e fixture per-node resource line numbers and titles whose
extremes differ per backend
A merged route that forgets the parseAggregate check silently returns one
backend's rows: an aggregate row carries no certname, hash or name, so the
identity-keyed merges collapse every backend's numbers into one. That is how
/facts/<name> shipped broken.
- Resolve every /pdb/query/v4 request through one route table
- Sum extract/function queries in handleQuery, before any route's handler runs
- Make each route that is not summed name its reason; the zero value is guarded
- Assert the guard over the route table, so a new route inherits the assertion
The empty answer for an unknown value never reaches a backend, so it must
leave /healthz reporting a whole estate.
- assert partial_rounds stays 0 after an unknown-value drilldown
The <value> path segment is client-supplied and reached a full unfiltered
/facts fan-out, keyed per value, so every distinct value was a fresh
whole-estate query and a fresh cache entry.
- validate <value> against the configured backend names, answering [] with
no fan-out when it names none
- key the drilldown's fetch on the fact name alone and apply <value> to the
shared record set, so all values share one entry and one fan-out
- report every configured backend on the no-fan-out empty response, which is
complete rather than partial
/fact-names advertises the fact, so its drilldown must not be a dead link.
- Serve the source fact's own path from the /facts merge that produces the
records, so certname set, owner and environment match /facts.
- Filter /facts/<source-fact>/<value> by the owning backend.
- Keep the aggregate, query-gate and disabled paths answering as before.
An aggregate row carries no certname, so the per-certname merge collapsed
every backend's row into one bucket and served a single backend's count as
the estate's — no error, no warning, X-Backends still 2/2.
- Route an aggregate query on /facts/<name>[/<value>] to the summing path.
- List the injected fact's name in /fact-names instead of hiding it.
- Reject an order_by on any field but name, as the backends do.
Both fell to the unmerged pass-through, so one backend's answer was served
as if it were the estate's: Puppetboard's fact drilldown lost the other
backend's nodes and its facts overview lost that backend's fact names.
- Serve /facts/<name> and /facts/<name>/<value> through the /facts merge.
- Serve /fact-names as a deduped, re-sorted, re-paged union of name arrays.
- Gate provenance on the path: only /facts/<source-fact> may be injected.
- Keep the owned fact name out of /fact-names while the feature is on.
- Cache both alongside the merged /facts and /nodes record sets.
- Turn the two recorded e2e gaps into positive assertions.
A `/facts` aggregate row carries no certname, so the per-certname fact
merge collapsed every backend's rows into one bucket and served a single
backend's numbers.
- Route a `/facts` query whose `extract` carries a `function` column to
serveSummed, as /nodes, /resources and /reports already do
- Document which extract functions combine correctly across backends
Every merge rule, AST gate and provenance decision was derived from reading
upstream source and proven only against fake backends, so nothing had ever run
against a real PuppetDB.
- Add an e2e suite behind the `e2e` build tag and a `make e2e` target
- Stand up two openvoxdb backends on their own PostgreSQL with testcontainers
- Load facts, reports and catalogs over the command API, waiting on processing
- Assert the union, freshness dedupe, summed aggregates, provenance gating,
X-Backends, backend death and recovery, and the report paths
- Drive Puppetboard and node-lookup against pdbmux as real clients
- Record three known gaps as skips that fail once the gap closes
benvin
unkin-agent