Merge pull request 'Serve fact queries live and drop the cache headers' (#26) from benvin/drop-fact-caching into main

Reviewed-on: #26

README.md

@@ -34,7 +34,7 @@ not PQL) is forwarded verbatim.
 | `GET /pdb/query/v4/event-counts` | Fan out to all and **sum** each subject's counts into one row per subject. |
 | `GET /pdb/query/v4/aggregate-event-counts` | Fan out to all and **sum** the summary object's counts. |
 | `GET /pdb/query/v4/reports/<hash>/{events,logs,metrics}` | Ask every backend; serve the answer from whichever backend actually holds that report. `404` when none does. |
-| `GET /pdb/query/v4/{nodes,factsets,catalogs}/<certname>[/...]` | Ask the backend that owns that `certname` — the same owner the `/facts` merge attributes records to — and serve its reply verbatim. The remaining backends are tried after it, so a node only one backend holds is still served, and openvoxdb's own `404` body is replayed when none holds it. |
+| `GET /pdb/query/v4/{nodes,factsets,catalogs}/<certname>[/...]` | Ask the backend that owns that `certname` — the same owner the `/facts` merge attributes records to — and serve its reply verbatim, success or error. The remaining backends are tried after it, so a node only one backend holds is still served, and openvoxdb's own `404` body is replayed when none holds it. |
 | `GET /pdb/query/v4/*` (any other) | No merge rule, so backends are tried in configured order and the first success is streamed back verbatim; if all reject it, the first upstream error response is replayed. |
 | `GET /pdb/meta/v1/version` | Fan out to all and report the **lowest** version any backend runs. |
 | `GET /pdb/meta/v1/server-time` | Fan out to all and serve the first reachable backend's clock. |
@@ -47,22 +47,38 @@ Fan-out is concurrent, and goes only to the backends the health prober currently
 believes are up — see [Backend health](#backend-health). If one backend errors or
 times out, `pdbmux` serves the surviving backends' results and logs a warning; a
 merged endpoint only returns `502` when **every** backend fails. Response records
-are passed through as raw JSON so unknown fields survive untouched.
+are passed through as raw JSON so unknown fields survive untouched, under
+openvoxdb's own `application/json;charset=utf-8`.
 
-Every backend is asked the same question, so a query all of them *refuse* with
+Every backend is asked the same question, so a status **every** backend answered
-the same client-shaped status — a `400` naming an unknown field, say — is the
+with is the estate's own answer, not an outage: that status, openvoxdb's own
-query's fault rather than an outage: that status and openvoxdb's own explanation
+explanation and its content type are replayed to the client instead of a `502`,
-are replayed to the client instead of a `502`, with any backend address stripped
+with any backend address stripped out of the body first. This holds for every
-out of the body first. Backends disagreeing on the status, a `403` (`pdbmux`'s
+status from `400` up — openvoxdb answers `["=","name"]` with `400` on `/nodes`
-own credentials, not the client's), a `404` (which records a backend holds is
+but `500` on `/facts`, and `/metrics/v2` with a flat `403`, so a rule drawn at
-exactly what backends disagree about), `408`, `429` and every `5xx` still return
+`500` would replay one and swallow the other. `502` is kept for what it actually
-`502`. A refused query is not counted as a partial round on `/healthz`, and
+describes: backends **disagreeing** on the status, or a backend that answered
-nothing about it is cached.
+nothing at all. The rule is the same on `/pdb/query`, `/pdb/meta` and `/metrics`,
+so no route answers a failure differently from any other.
+
+Unanimity is asked for only where the backends are peers answering the same
+question. The per-certname routes are not: one backend **owns** the node and the
+others are fallbacks, so the owner's reply is the answer whatever they said — a
+backend that does not hold the certname answers `404` to say the node is not its
+own, not to disagree about the request. When the owner answers nothing at all the
+first fallback that did answer stands in, so an unreachable owner does not take
+the node with it, and `502` is left for a request no backend answered.
+
+A unanimous `4xx` blames the request, so it is not counted as a partial round on
+`/healthz` and nothing about it is cached. A unanimous `5xx` is the backends
+reporting their own fault, so it is replayed just as faithfully but still counts
+as a failed round and still falls back to a stale cache entry where there is one.
 
 Responses carry PuppetDB's `X-Records` when the query asked for a total, and on
-the merged paths `X-Backends` (see [Backend health](#backend-health)). Cached
+the merged paths `X-Backends` (see [Backend health](#backend-health)). Successful
-paths add two more headers `pdbmux` sets itself, `X-Cache` and `Age` — see
+query responses carry PuppetDB's own `Content-Type: application/json;charset=utf-8`.
-[Caching](#caching).
+Nothing tells a client whether a response came from the cache: PuppetDB sets no
+`X-Cache` or `Age`, so neither does `pdbmux` — see [Caching](#caching).
 
 ## Merge semantics
 
@@ -243,7 +259,7 @@ record of that name, so the route does not serve its own fan-out: it takes the
 records from the `/facts` merge that produces them, which makes the `certname`
 set, the owner and the `environment` identical to the ones an unfiltered `/facts`
 response reports, and lets the request's own `query` narrow the result upstream.
-That costs one `/facts` fan-out per cache miss — the widest fan-out `pdbmux`
+That costs one `/facts` fan-out per request — the widest fan-out `pdbmux`
 makes — on a rare, user-initiated path.
 
 `/facts/pdbmux_source/<value>` pins the backend name, so it answers with the
@@ -404,8 +420,8 @@ not answering.
   asked, never what a merged answer means: a response built from a subset is
   still served, as before. Every merged response carries `X-Backends:
   <contributed>/<configured>` naming how many backends' records went into it, so
-  a client can tell a full answer from a partial one. On a cache hit the header
+  a client can tell a full answer from a partial one. On a `/nodes` cache hit the
-  describes the stored body, not the current backend count.
+  header describes the stored body, not the current backend count.
 - **`/healthz`** gives each backend a `state` (`healthy`, `unhealthy`,
   `probe_unsupported`, `unprobed`, or `unmonitored` when probing is off),
   `consecutive_failures`,
@@ -425,21 +441,27 @@ not answering.
 
 ## Caching
 
-`pdbmux` caches merged `/nodes`, `/facts`, `/facts/<name>[/<value>]` and
+`pdbmux` caches the merged `/nodes` record set **in memory** so a busy
-`/fact-names` record sets **in memory** so a busy Puppetboard does not re-fan-out
+Puppetboard does not re-fan-out the node list every few seconds. Its report
-the same query every few seconds — its facts overview and fact drilldown are two
+columns only move when a node finishes a run, so a `30s` answer is a `30s`-old
-of the pages that hit hardest. Everything else runs uncached — including
+report timestamp and nothing else.
-`extract` aggregates on those paths, and
+
-the `/pdb/meta/v1/*` and `/metrics/*` endpoints, which are served live on every
+**No fact-serving path is cached.** `/facts`, `/facts/<name>[/<value>]`,
-request. The cache is an interface, and `/reports` gets its own (S3-backed)
+`/fact-names`, `/factsets*`, `/nodes/<certname>/facts`, `/fact-contents`,
-backend later without further handler changes.
+`/fact-paths` and `/inventory` all fan out on every request, so a fact answer is
+exactly as current as the backend's own. Cache fact data client-side if you want
+it cached. Everything else runs uncached too — including `extract` aggregates on
+`/nodes`, and the `/pdb/meta/v1/*` and `/metrics/*` endpoints. The cache is an
+interface, and `/reports` gets its own (S3-backed) backend later without further
+handler changes.
 
 - **Key** — `<path>?<params>`, where the params are the ones that actually
   determine the response, URL-encoded with keys sorted ascending and a repeated
   param's values sorted ascending. Param order in the request is therefore
   irrelevant: one canonical key per distinct request. A request with no params
   keys on the bare path.
-- **TTL** — `facts_ttl`, default `30s`, **hard cap `30s`**. A larger configured
+- **TTL** — `facts_ttl` (the key predates the cache narrowing to `/nodes`),
+  default `30s`, **hard cap `30s`**. A larger configured
   value is **clamped** down to the cap, not rejected, so a stray env var cannot
   crash-loop a container; `pdbmux config show` prints
   `facts_ttl : 30s (clamped from 600s, cap 30s)` when that happens. `facts_ttl: 0`
@@ -448,7 +470,8 @@ backend later without further handler changes.
   has passed `pdbmux` always re-queries the backends; the expired copy is served
   **only** if every backend fails, which turns a `502` into slightly-old data. A
   healthy backend is never shadowed by a stale entry, and a query every backend
-  refuses is answered with the refusal rather than the stale copy.
+  *refuses* with a `4xx` is answered with the refusal rather than the stale copy —
+  a unanimous `5xx` is an outage like any other and still takes the stale copy.
 - **Bounded** — `facts_cache_bytes` (default 64 MiB) is a byte budget, evicted
   least-recently-used; reads count as use, so a stale entry that is still being
   asked for survives. A single response larger than the whole budget is not
@@ -462,29 +485,20 @@ backend later without further handler changes.
   client goes away leaves the flight running for the others. The flight is
   cancelled once its last participant leaves, so a lone client disconnecting
   releases the upstream connections straight away.
-- **Response headers** — every response on a cached path carries `X-Cache`
+- **Invisible to clients** — a cached response is byte-for-byte a live one, with
-  (`hit` served from a fresh entry, `miss` built by this request, `stale` the
+  the same headers. PuppetDB emits no `X-Cache` and no `Age`, so neither does
-  expired-entry fallback) and `Age` in whole seconds since the served copy was
+  `pdbmux`; read the cache state off `/healthz` instead.
-  stored (`0` on a `miss`). Uncached paths carry neither.
 - **Visibility** — `/healthz` carries a `cache` object: `backend`
   (`memory`/`none`), `ttl`, `entries`, `stale_entries`, `bytes`, `serving_stale`,
   `stale_served` and `last_stale_served`. `serving_stale` is `true` from the
   moment a stale fallback is served until the next response comes from a live
   fan-out or a fresh entry.
-- **Provenance is stored, not re-applied** — what a cache entry holds is the
+- **Merged bodies are stored, not re-merged** — an entry holds the fully merged
-  fully merged body, `pdbmux_source` already injected and upstream records of
+  `/nodes` body, `pdbmux_source` already stamped on, and the
-  that name already dropped. Attribution names the backend that supplied the
+  `X-Records`/`X-Backends` it was built with, so a hit reproduces all three.
-  data, which is a property of that fetch, so it stays correct for as long as the
+  Attribution is a property of that fetch, so it stays correct for as long as the
-  body does and ages out with it — `X-Cache` and `Age` say how old both are. Two
+  body does and ages out with it. Two requests share an entry only when they
-  requests can only share an entry when they share a key, and the key is path
+  share a key, and the key is path plus query.
-  plus query, which is exactly what decides whether injection applies; a
-  name-filtered `/facts` query and a plain one therefore cache separately and
-  neither is ever served the other's shape. The one path that keys on less than
-  it is asked is the `pdbmux_source` drilldown, whose `<value>` is dropped from
-  the key and applied to the shared entry instead. `source_fact` and
-  `source_fact_enabled` are read once at startup, and the cache lives for the
-  same process, so changing either cannot leave differently-shaped entries
-  behind.
 
 ## Config
 
@@ -510,7 +524,7 @@ backends:                          # order is a tie-break only, not a ranking
 merge: freshness             # freshness | static
 timeout: 10s                 # per-upstream request timeout
 freshness_ttl: 30s           # freshness-map cache TTL (freshness merge only)
-facts_ttl: 30s               # /facts + /nodes response cache TTL; 0 disables, capped at 30s
+facts_ttl: 30s               # merged /nodes response cache TTL; 0 disables, capped at 30s
 facts_cache_bytes: 67108864  # byte budget for that cache (64 MiB), LRU-evicted
 source_fact: pdbmux_source   # name of the synthetic provenance fact
 source_fact_enabled: true    # false serves backends' records untouched

cache_test.go

@@ -61,6 +61,7 @@ type countingBackend struct {
 	hits   map[string]int
 	bodies map[string]string
 	fail   bool
+	dead   bool
 	block  chan struct{}
 }
 
@@ -70,12 +71,15 @@ func newCountingBackend(t *testing.T, bodies map[string]string) *countingBackend
 	cb.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		cb.mu.Lock()
 		cb.hits[r.URL.Path]++
-		fail, block, body := cb.fail, cb.block, cb.bodies[r.URL.Path]
+		fail, dead, block, body := cb.fail, cb.dead, cb.block, cb.bodies[r.URL.Path]
 		cb.mu.Unlock()
 
 		if block != nil {
 			<-block
 		}
+		if dead {
+			panic(http.ErrAbortHandler)
+		}
 		if fail {
 			http.Error(w, "boom", http.StatusInternalServerError)
 			return
@@ -108,6 +112,12 @@ func (cb *countingBackend) totalHits() int {
 	return n
 }
 
+func (cb *countingBackend) setDead(v bool) {
+	cb.mu.Lock()
+	defer cb.mu.Unlock()
+	cb.dead = v
+}
+
 func (cb *countingBackend) setFail(v bool) {
 	cb.mu.Lock()
 	defer cb.mu.Unlock()
@@ -126,13 +136,13 @@ func (cb *countingBackend) setBlock(ch chan struct{}) {
 	cb.block = ch
 }
 
-// newCachedServer builds a server whose facts cache runs on a fake clock.
+// newCachedServer builds a server whose /nodes cache runs on a fake clock.
 func newCachedServer(t *testing.T, cfg Config) (*Server, *fakeClock) {
 	t.Helper()
 	srv := newTestServer(cfg)
-	mc, ok := srv.factsCache.(*memoryCache)
+	mc, ok := srv.nodeCache.(*memoryCache)
 	if !ok {
-		t.Fatalf("expected a memory cache, got %T", srv.factsCache)
+		t.Fatalf("expected a memory cache, got %T", srv.nodeCache)
 	}
 	clk := newFakeClock()
 	mc.now = clk.now
@@ -267,14 +277,14 @@ func TestNoopCache_AlwaysMisses(t *testing.T) {
 	}
 }
 
-func TestHandler_FactsCacheFreshHit(t *testing.T) {
+func TestHandler_NodesCacheFreshHit(t *testing.T) {
-	body := `[` + fact("h1", "role", "web", "") + `]`
+	body := `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`
-	a := newCountingBackend(t, map[string]string{factsPath: body})
+	a := newCountingBackend(t, map[string]string{nodesPath: body})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	first := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`)
+	first := doGet(t, srv.Handler(), nodesPath, `["=","certname","h1"]`)
-	second := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`)
+	second := doGet(t, srv.Handler(), nodesPath, `["=","certname","h1"]`)
 
 	if first.Code != http.StatusOK || second.Code != http.StatusOK {
 		t.Fatalf("statuses %d/%d", first.Code, second.Code)
@@ -282,40 +292,40 @@ func TestHandler_FactsCacheFreshHit(t *testing.T) {
 	if first.Body.String() != second.Body.String() {
 		t.Errorf("cache hit changed the body:\n %s\n %s", first.Body.String(), second.Body.String())
 	}
-	if got := a.hitCount(factsPath); got != 1 {
+	if got := a.hitCount(nodesPath); got != 1 {
 		t.Errorf("backend a saw %d requests, want 1 (second served from cache)", got)
 	}
-	if got := b.hitCount(factsPath); got != 1 {
+	if got := b.hitCount(nodesPath); got != 1 {
 		t.Errorf("backend b saw %d requests, want 1", got)
 	}
 
 	// A different query is a different key and must go upstream.
-	doGet(t, srv.Handler(), factsPath, `["=","name","osfamily"]`)
+	doGet(t, srv.Handler(), nodesPath, `["=","certname","h2"]`)
-	if got := a.hitCount(factsPath); got != 2 {
+	if got := a.hitCount(nodesPath); got != 2 {
 		t.Errorf("a different query should refetch: %d requests, want 2", got)
 	}
 }
 
-func TestHandler_FactsCacheExpires(t *testing.T) {
+func TestHandler_NodesCacheExpires(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	doGet(t, srv.Handler(), factsPath, "")
+	doGet(t, srv.Handler(), nodesPath, "")
 	clk.advance(31 * time.Second)
-	doGet(t, srv.Handler(), factsPath, "")
+	doGet(t, srv.Handler(), nodesPath, "")
 
-	if got := a.hitCount(factsPath); got != 2 {
+	if got := a.hitCount(nodesPath); got != 2 {
 		t.Errorf("an expired entry should refetch: %d requests, want 2", got)
 	}
 }
 
 func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "old", "") + `]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("old", "2026-01-01T00:00:00.000Z") + `]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	warm := doGet(t, srv.Handler(), factsPath, "")
+	warm := doGet(t, srv.Handler(), nodesPath, "")
 	if warm.Code != http.StatusOK {
 		t.Fatalf("warm-up status %d", warm.Code)
 	}
@@ -325,8 +335,8 @@ func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
 
 	// Backends healthy but the entry expired: fresh data wins, never the stale copy.
 	clk.advance(31 * time.Second)
-	a.setBody(factsPath, `[`+fact("h1", "role", "new", "")+`]`)
+	a.setBody(nodesPath, `[`+node("new", "2026-01-01T00:00:00.000Z")+`]`)
-	refetch := doGet(t, srv.Handler(), factsPath, "")
+	refetch := doGet(t, srv.Handler(), nodesPath, "")
 	if !strings.Contains(refetch.Body.String(), `"new"`) {
 		t.Errorf("a healthy backend must not be shadowed by the stale entry: %s", refetch.Body.String())
 	}
@@ -338,7 +348,7 @@ func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
 	clk.advance(31 * time.Second)
 	a.setFail(true)
 	b.setFail(true)
-	stale := doGet(t, srv.Handler(), factsPath, "")
+	stale := doGet(t, srv.Handler(), nodesPath, "")
 	if stale.Code != http.StatusOK {
 		t.Fatalf("stale fallback status %d: %s", stale.Code, stale.Body.String())
 	}
@@ -352,51 +362,14 @@ func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
 	}
 }
 
-// The stored entry is the whole estate's record set and the pinned value narrows
-// it per request, so the fallback has to keep narrowing: a client asking for one
-// backend's records must not be handed every backend's because the entry expired.
-func TestHandler_StaleSourceFactDrilldownStaysFilteredByOwner(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "osfamily", "RedHat", "") + `]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h2", "osfamily", "Debian", "") + `]`})
-	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
-
-	drilldown := sourceFactURL + "/a"
-	want := map[string]string{"h1": "a"}
-
-	warm := doGet(t, srv.Handler(), drilldown, "")
-	if warm.Code != http.StatusOK {
-		t.Fatalf("warm-up status %d: %s", warm.Code, warm.Body.String())
-	}
-	if got, n := sourceValues(t, warm.Body.Bytes(), defaultSourceFact); n != len(want) || !reflect.DeepEqual(got, want) {
-		t.Fatalf("warm-up %s = %v (%d records), want %v", drilldown, got, n, want)
-	}
-
-	// Every backend down and the entry expired: the stale copy is served.
-	clk.advance(31 * time.Second)
-	a.setFail(true)
-	b.setFail(true)
-
-	rec := doGet(t, srv.Handler(), drilldown, "")
-	if rec.Code != http.StatusOK {
-		t.Fatalf("stale fallback status %d: %s", rec.Code, rec.Body.String())
-	}
-	if got := rec.Header().Get(cacheStatusHeader); got != "stale" {
-		t.Fatalf("%s = %q, want stale: the request did not take the fallback path", cacheStatusHeader, got)
-	}
-	got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact)
-	if n != len(want) || !reflect.DeepEqual(got, want) {
-		t.Errorf("stale %s = %v (%d records), want only backend a's %v", drilldown, got, n, want)
-	}
-}
-
 func TestHandler_NoCacheEntryMeansBackendFailureIs502(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
-	a.setFail(true)
+	a.setDead(true)
-	b.setFail(true)
+	b.setDead(true)
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	rec := doGet(t, srv.Handler(), factsPath, "")
+	rec := doGet(t, srv.Handler(), nodesPath, "")
 	if rec.Code != http.StatusBadGateway {
 		t.Fatalf("status %d, want 502", rec.Code)
 	}
@@ -406,8 +379,8 @@ func TestHandler_NoCacheEntryMeansBackendFailureIs502(t *testing.T) {
 }
 
 func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 	h := srv.Handler()
 
@@ -422,13 +395,13 @@ func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
 		wg.Add(1)
 		go func(i int) {
 			defer wg.Done()
-			codes[i] = doGet(t, h, factsPath, "").Code
+			codes[i] = doGet(t, h, nodesPath, "").Code
 		}(i)
 	}
 
 	// The leader is parked inside the blocked backend, so every caller that
 	// reaches the handler before the release joins its flight.
-	waitFor(t, func() bool { return a.hitCount(factsPath) >= 1 })
+	waitFor(t, func() bool { return a.hitCount(nodesPath) >= 1 })
 	time.Sleep(250 * time.Millisecond)
 	close(release)
 	wg.Wait()
@@ -438,10 +411,10 @@ func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
 			t.Fatalf("caller %d got %d", i, code)
 		}
 	}
-	if got := a.hitCount(factsPath); got != 1 {
+	if got := a.hitCount(nodesPath); got != 1 {
 		t.Errorf("backend a saw %d requests, want 1 for %d concurrent callers", got, callers)
 	}
-	if got := b.hitCount(factsPath); got != 1 {
+	if got := b.hitCount(nodesPath); got != 1 {
 		t.Errorf("backend b saw %d requests, want 1 for %d concurrent callers", got, callers)
 	}
 }
@@ -557,15 +530,15 @@ func TestFlightGroup_LeaderPanicFailsLeaderAndWaiters(t *testing.T) {
 }
 
 func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
 	panicBuild := func(context.Context) (cachedResponse, error) { panic("build exploded") }
 	serve := func(build func(context.Context) (cachedResponse, error)) *httptest.ResponseRecorder {
 		rec := httptest.NewRecorder()
-		req := httptest.NewRequest(http.MethodGet, factsPath, nil)
+		req := httptest.NewRequest(http.MethodGet, nodesPath, nil)
-		srv.serveCached(rec, req, factsPath, nil, build)
+		srv.serveCached(rec, req, nodesPath, nil, build)
 		return rec
 	}
 
@@ -575,7 +548,7 @@ func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
 		t.Fatalf("status %d (%s), want 502", rec.Code, rec.Body.String())
 	}
 
-	stored := `[` + fact("h1", "role", "web", "") + `]`
+	stored := `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`
 	rec = serve(func(context.Context) (cachedResponse, error) {
 		return cachedResponse{Body: json.RawMessage(stored), Records: 1}, nil
 	})
@@ -600,16 +573,16 @@ func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
 }
 
 func TestServeCached_BuildPanicFailsConcurrentWaiters(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
 	entered := make(chan struct{})
 	release := make(chan struct{})
 	serve := func(build func(context.Context) (cachedResponse, error)) *httptest.ResponseRecorder {
 		rec := httptest.NewRecorder()
-		req := httptest.NewRequest(http.MethodGet, factsPath, nil)
+		req := httptest.NewRequest(http.MethodGet, nodesPath, nil)
-		srv.serveCached(rec, req, factsPath, nil, build)
+		srv.serveCached(rec, req, nodesPath, nil, build)
 		return rec
 	}
 
@@ -710,8 +683,8 @@ func TestHandler_LeaderDisconnectDoesNotFailFollowers(t *testing.T) {
 
 	// The flight the leader started still populated the cache.
 	warmed := doGet(t, h, nodesPath, "")
-	if got := warmed.Header().Get(cacheStatusHeader); got != "hit" {
+	if got := strings.TrimSpace(warmed.Body.String()); !sameJSON(t, got, want) {
-		t.Errorf("%s = %q, want hit: the abandoned leader's flight must still warm the cache", cacheStatusHeader, got)
+		t.Errorf("warmed body = %s, want %s: the abandoned leader's flight must still warm the cache", got, want)
 	}
 	if got := a.hitCount(nodesPath); got != 1 {
 		t.Errorf("backend a saw %d requests after the cached read, want 1", got)
@@ -750,7 +723,7 @@ func TestHandler_SoloDisconnectAbortsFanOutPromptly(t *testing.T) {
 	}()
 
 	start := time.Now()
-	req := httptest.NewRequest(http.MethodGet, factsPath, nil).WithContext(ctx)
+	req := httptest.NewRequest(http.MethodGet, nodesPath, nil).WithContext(ctx)
 	srv.Handler().ServeHTTP(httptest.NewRecorder(), req)
 	if elapsed := time.Since(start); elapsed >= cfg.Timeout/2 {
 		t.Errorf("handler returned after %s, want well under cfg.Timeout %s", elapsed, cfg.Timeout)
@@ -783,7 +756,7 @@ func TestHandler_DisconnectedRequestsDoNotPinBackends(t *testing.T) {
 				time.Sleep(disconnectAfter)
 				cancel()
 			}()
-			target := factsPath + "?query=" + url.QueryEscape(strconv.Itoa(i))
+			target := nodesPath + "?query=" + url.QueryEscape(strconv.Itoa(i))
 			req := httptest.NewRequest(http.MethodGet, target, nil).WithContext(ctx)
 			h.ServeHTTP(httptest.NewRecorder(), req)
 		}(i)
@@ -1116,47 +1089,39 @@ func assertGoroutinesSettle(t *testing.T, baseline int, frames ...string) {
 	}
 }
 
-func TestServeCached_CacheStatusHeaders(t *testing.T) {
+// PuppetDB emits no X-Cache and no Age, so a client cannot tell a cached
-	stored := `[` + fact("h1", "role", "web", "") + `]`
+// response from a live one. Covers the fresh hit, the miss and the stale
-	a := newCountingBackend(t, map[string]string{factsPath: stored})
+// fallback, which are the three ways a response can leave the cached path.
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+func TestHandler_CachedResponsesCarryNoCacheHeaders(t *testing.T) {
+	a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 	h := srv.Handler()
 
-	rec := doGet(t, h, factsPath, "")
+	assertNoCacheHeaders(t, doGet(t, h, nodesPath, ""), "miss")
-	if got := rec.Header().Get(cacheStatusHeader); got != "miss" {
-		t.Errorf("first request %s = %q, want miss", cacheStatusHeader, got)
-	}
-	if got := rec.Header().Get(ageHeader); got != "0" {
-		t.Errorf("first request %s = %q, want 0", ageHeader, got)
-	}
 
 	clk.advance(7 * time.Second)
-	rec = doGet(t, h, factsPath, "")
+	assertNoCacheHeaders(t, doGet(t, h, nodesPath, ""), "fresh hit")
-	if got := rec.Header().Get(cacheStatusHeader); got != "hit" {
-		t.Errorf("cached request %s = %q, want hit", cacheStatusHeader, got)
-	}
-	if got := rec.Header().Get(ageHeader); got != "7" {
-		t.Errorf("cached request %s = %q, want 7", ageHeader, got)
-	}
 
-	// Past the TTL with every backend down, the stale fallback must say so.
 	clk.advance(24 * time.Second)
 	a.setFail(true)
 	b.setFail(true)
-	rec = doGet(t, h, factsPath, "")
+	rec := doGet(t, h, nodesPath, "")
 	if rec.Code != http.StatusOK {
 		t.Fatalf("stale fallback status %d (%s)", rec.Code, rec.Body.String())
 	}
-	if got := rec.Header().Get(cacheStatusHeader); got != "stale" {
+	if serving, _, _ := srv.stale.snapshot(); !serving {
-		t.Errorf("stale fallback %s = %q, want stale", cacheStatusHeader, got)
+		t.Fatal("the third request did not take the stale fallback path")
 	}
-	if got := rec.Header().Get(ageHeader); got != "31" {
+	assertNoCacheHeaders(t, rec, "stale fallback")
-		t.Errorf("stale fallback %s = %q, want 31 seconds since the entry was stored", ageHeader, got)
+}
-	}
+
-	want := `[` + fact("h1", "role", "web", "") + `,` + factEnv("h1", defaultSourceFact, "a", "") + `]`
+func assertNoCacheHeaders(t *testing.T, rec *httptest.ResponseRecorder, what string) {
-	if got := strings.TrimSpace(rec.Body.String()); !sameJSON(t, got, want) {
+	t.Helper()
-		t.Errorf("stale body = %s, want %s", got, want)
+	for _, h := range []string{"X-Cache", "Age"} {
+		if got := rec.Header().Get(h); got != "" {
+			t.Errorf("%s response carries %s: %q, want no such header", what, h, got)
+		}
 	}
 }
 
@@ -1193,18 +1158,18 @@ func (c *recordingCache) Stats() CacheStats { return CacheStats{Backend: "record
 func TestServeCached_PutRunsOnDetachedContext(t *testing.T) {
 	srv := newTestServer(cacheTestConfig("http://backend.invalid", "http://backend.invalid"))
 	cache := newRecordingCache()
-	srv.factsCache = cache
+	srv.nodeCache = cache
 
 	entered := make(chan struct{})
 	release := make(chan struct{})
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
-	req := httptest.NewRequest(http.MethodGet, factsPath, nil).WithContext(ctx)
+	req := httptest.NewRequest(http.MethodGet, nodesPath, nil).WithContext(ctx)
 
 	served := make(chan struct{})
 	go func() {
 		defer close(served)
-		srv.serveCached(httptest.NewRecorder(), req, factsPath, nil, func(context.Context) (cachedResponse, error) {
+		srv.serveCached(httptest.NewRecorder(), req, nodesPath, nil, func(context.Context) (cachedResponse, error) {
 			close(entered)
 			<-release
 			return cachedResponse{Body: json.RawMessage(`[]`), Records: -1}, nil
@@ -1236,8 +1201,8 @@ func TestServeCached_PutRunsOnDetachedContext(t *testing.T) {
 }
 
 func TestHandler_HealthzReportsCacheState(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
 	hr := health(t, srv)
@@ -1248,7 +1213,7 @@ func TestHandler_HealthzReportsCacheState(t *testing.T) {
 		t.Fatalf("a cold cache should be empty and not stale: %+v", hr.Cache)
 	}
 
-	doGet(t, srv.Handler(), factsPath, "")
+	doGet(t, srv.Handler(), nodesPath, "")
 	hr = health(t, srv)
 	if hr.Cache.Entries != 1 || hr.Cache.StaleEntries != 0 || hr.Cache.Bytes == 0 {
 		t.Fatalf("after one request: %+v", hr.Cache)
@@ -1264,7 +1229,7 @@ func TestHandler_HealthzReportsCacheState(t *testing.T) {
 
 	a.setFail(true)
 	b.setFail(true)
-	doGet(t, srv.Handler(), factsPath, "")
+	doGet(t, srv.Handler(), nodesPath, "")
 	hr = health(t, srv)
 	if !hr.Cache.ServingStale || hr.Cache.StaleServed != 1 || hr.Cache.LastStale == "" {
 		t.Errorf("staleness not surfaced in /healthz: %+v", hr.Cache)
@@ -1444,9 +1409,9 @@ func TestNewServer_ClampsFactsTTL(t *testing.T) {
 	cfg.CacheBytes = 1 << 20
 	srv := newTestServer(cfg)
 
-	mc, ok := srv.factsCache.(*memoryCache)
+	mc, ok := srv.nodeCache.(*memoryCache)
 	if !ok {
-		t.Fatalf("expected a memory cache, got %T", srv.factsCache)
+		t.Fatalf("expected a memory cache, got %T", srv.nodeCache)
 	}
 	if mc.ttl != maxFactsTTL {
 		t.Errorf("cache ttl = %s, want %s", mc.ttl, maxFactsTTL)
@@ -1520,35 +1485,6 @@ func stamped(t *testing.T, raw, field, value string) string {
 // The injector is per-request but a cache entry is shared, so a second caller is
 // served a body built for the first. Provenance names the backend that supplied
 // the data, which is a property of that fetch, so the shared body stays correct.
-func TestHandler_CachedFactsKeepSourceAttribution(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
-	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
-	h := srv.Handler()
-
-	first := doGet(t, h, factsPath, "")
-	if got := first.Header().Get(cacheStatusHeader); got != "miss" {
-		t.Fatalf("first %s = %q, want miss", cacheStatusHeader, got)
-	}
-	srcs, n := sourceValues(t, first.Body.Bytes(), defaultSourceFact)
-	if n != 1 || srcs["h1"] != "a" {
-		t.Fatalf("first request sources = %v (%d records), want h1 -> a", srcs, n)
-	}
-
-	second := doGet(t, h, factsPath, "")
-	if got := second.Header().Get(cacheStatusHeader); got != "hit" {
-		t.Fatalf("second %s = %q, want hit", cacheStatusHeader, got)
-	}
-	srcs, n = sourceValues(t, second.Body.Bytes(), defaultSourceFact)
-	if n != 1 || srcs["h1"] != "a" {
-		t.Errorf("cached request sources = %v (%d records), want exactly one h1 -> a", srcs, n)
-	}
-	if got := a.hitCount(factsPath); got != 1 {
-		t.Errorf("backend a saw %d requests, want 1: the second read must come from the cache", got)
-	}
-}
-
-// The same, for the /nodes stamp rather than the synthetic /facts record.
 func TestHandler_CachedNodesKeepSourceStamp(t *testing.T) {
 	a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-02T00:00:00.000Z") + `]`})
 	b := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
@@ -1561,9 +1497,6 @@ func TestHandler_CachedNodesKeepSourceStamp(t *testing.T) {
 	}
 
 	second := doGet(t, h, nodesPath, "")
-	if got := second.Header().Get(cacheStatusHeader); got != "hit" {
-		t.Fatalf("second %s = %q, want hit", cacheStatusHeader, got)
-	}
 	want := `[` + stamped(t, node("h1", "2026-01-02T00:00:00.000Z"), defaultSourceFact, "a") + `]`
 	if got := strings.TrimSpace(second.Body.String()); !sameJSON(t, got, want) {
 		t.Errorf("cached body = %s, want %s", got, want)
@@ -1577,77 +1510,115 @@ func TestHandler_CachedNodesKeepSourceStamp(t *testing.T) {
 // while the entry is served the attribution is the one that fetch had, and the
 // rebuild after the TTL picks up the move.
 func TestHandler_CachedSourceAgesWithItsData(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
+	a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 	h := srv.Handler()
 
-	doGet(t, h, factsPath, "")
+	doGet(t, h, nodesPath, "")
 
 	// The node moves to b while the entry is still fresh.
-	a.setBody(factsPath, `[]`)
+	a.setBody(nodesPath, `[]`)
-	b.setBody(factsPath, `[`+fact("h1", "role", "web", "")+`]`)
+	b.setBody(nodesPath, `[`+node("h1", "2026-01-01T00:00:00.000Z")+`]`)
 
-	cached := doGet(t, h, factsPath, "")
+	cached := doGet(t, h, nodesPath, "")
-	if got := cached.Header().Get(cacheStatusHeader); got != "hit" {
+	if got := nodeSources(t, cached.Body.Bytes(), defaultSourceFact); got["h1"] != "a" {
-		t.Fatalf("%s = %q, want hit", cacheStatusHeader, got)
+		t.Errorf("cached stamp = %v, want h1 -> a: the body and its attribution come from the same fetch", got)
 	}
-	if srcs, _ := sourceValues(t, cached.Body.Bytes(), defaultSourceFact); srcs["h1"] != "a" {
+	if got := a.hitCount(nodesPath); got != 1 {
-		t.Errorf("cached sources = %v, want h1 -> a: the body and its attribution come from the same fetch", srcs)
+		t.Errorf("backend a saw %d requests, want 1 while the entry is fresh", got)
 	}
 
 	clk.advance(31 * time.Second)
-	rebuilt := doGet(t, h, factsPath, "")
+	rebuilt := doGet(t, h, nodesPath, "")
-	if srcs, _ := sourceValues(t, rebuilt.Body.Bytes(), defaultSourceFact); srcs["h1"] != "b" {
+	if got := nodeSources(t, rebuilt.Body.Bytes(), defaultSourceFact); got["h1"] != "b" {
-		t.Errorf("rebuilt sources = %v, want h1 -> b once the entry expired", srcs)
+		t.Errorf("rebuilt stamp = %v, want h1 -> b once the entry expired", got)
 	}
 }
 
-// The injection gate is a pure function of path and query, both of which are in
+// Fact answers must be as current as a backend's own, so every fact-serving
-// the cache key, so a gated request can never be served an injected body cached
+// route refetches: the second request sees data that changed between the two.
-// for an ungated one.
+func TestHandler_FactPathsAreNeverCached(t *testing.T) {
-func TestHandler_SourceGateIsPerCacheKey(t *testing.T) {
+	for _, path := range []string{
+		factsPath,
+		factsPath + "/role",
+		factsPath + "/role/web",
+		factNamesPath,
+		factsetsPath,
+		nodesPath + "/h1/facts",
+		"/pdb/query/v4/fact-contents",
+		"/pdb/query/v4/fact-paths",
+		"/pdb/query/v4/inventory",
+	} {
+		t.Run(path, func(t *testing.T) {
+			before := `[{"certname":"h1","name":"role","value":"before"}]`
+			after := `[{"certname":"h1","name":"role","value":"after"}]`
+			a := newCountingBackend(t, map[string]string{path: before})
+			b := newCountingBackend(t, map[string]string{path: `[]`})
+			srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
+			h := srv.Handler()
+
+			if first := doGet(t, h, path, ""); !strings.Contains(first.Body.String(), "before") {
+				t.Fatalf("first %s = %s, want the backend's own records", path, first.Body.String())
+			}
+			a.setBody(path, after)
+
+			second := doGet(t, h, path, "")
+			if strings.Contains(second.Body.String(), "before") {
+				t.Errorf("second %s = %s, want the changed data rather than a cached copy", path, second.Body.String())
+			}
+			if !strings.Contains(second.Body.String(), "after") {
+				t.Errorf("second %s = %s, want the changed data", path, second.Body.String())
+			}
+			if got := a.hitCount(path); got != 2 {
+				t.Errorf("backend a saw %d requests for %s, want one per request", got, path)
+			}
+		})
+	}
+}
+
+// No fact response may carry the cache headers PuppetDB never sends, whatever
+// else pdbmux is doing with the merged body.
+func TestHandler_FactResponsesCarryNoCacheHeaders(t *testing.T) {
 	body := `[` + fact("h1", "role", "web", "") + `]`
-	a := newCountingBackend(t, map[string]string{factsPath: body})
+	for _, path := range []string{factsPath, factsPath + "/role", factNamesPath, sourceFactURL} {
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+		t.Run(path, func(t *testing.T) {
-	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
+			a := newCountingBackend(t, map[string]string{path: body, factsPath: body})
-	h := srv.Handler()
+			b := newCountingBackend(t, map[string]string{path: `[]`, factsPath: `[]`})
+			srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
+			h := srv.Handler()
 
-	const nameFiltered = `["=","name","role"]`
+			assertNoCacheHeaders(t, doGet(t, h, path, ""), path)
-	for _, pass := range []string{"first", "cached"} {
+			assertNoCacheHeaders(t, doGet(t, h, path, ""), path+" repeated")
-		open := doGet(t, h, factsPath, "")
+		})
-		if srcs, n := sourceValues(t, open.Body.Bytes(), defaultSourceFact); n != 1 || srcs["h1"] != "a" {
-			t.Errorf("%s unfiltered request sources = %v (%d records), want h1 -> a", pass, srcs, n)
-		}
-		gated := doGet(t, h, factsPath, nameFiltered)
-		if _, n := sourceValues(t, gated.Body.Bytes(), defaultSourceFact); n != 0 {
-			t.Errorf("%s name-filtered request carried %d synthetic record(s), want none", pass, n)
-		}
-		if got := strings.TrimSpace(gated.Body.String()); !sameJSON(t, got, body) {
-			t.Errorf("%s name-filtered body = %s, want %s", pass, got, body)
-		}
-	}
-	if got := a.hitCount(factsPath); got != 2 {
-		t.Errorf("backend a saw %d requests, want 2: one per distinct cache key", got)
 	}
 }
 
-// Suppression of an upstream fact of the configured name is part of the merged
+// openvoxdb answers a successful query with application/json;charset=utf-8
-// body, so it survives into the cache rather than being re-applied per request.
+// (src/puppetlabs/puppetdb/query_eng.clj:555 through http.clj:80), so a merged
-func TestHandler_SuppressionSurvivesCacheHit(t *testing.T) {
+// response has to as well or a client can tell the proxy from the real thing.
-	upstream := `[` + fact("h1", defaultSourceFact, "somewhere-else", "") + `,` + fact("h1", "role", "web", "") + `]`
+func TestHandler_SuccessContentTypeMatchesPuppetDB(t *testing.T) {
-	a := newCountingBackend(t, map[string]string{factsPath: upstream})
+	bodies := map[string]string{
-	b := newCountingBackend(t, map[string]string{factsPath: `[]`})
+		nodesPath:     `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`,
+		factsPath:     `[` + fact("h1", "role", "web", "") + `]`,
+		factNamesPath: `["role"]`,
+		reportsPath:   `[` + report("h1", "abc", "2026-01-01T00:00:00.000Z") + `]`,
+	}
+	a := newCountingBackend(t, bodies)
+	b := newCountingBackend(t, map[string]string{nodesPath: `[]`, factsPath: `[]`, factNamesPath: `[]`, reportsPath: `[]`})
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 	h := srv.Handler()
 
-	doGet(t, h, factsPath, "")
+	for path := range bodies {
-	cached := doGet(t, h, factsPath, "")
+		// Twice, so a cache hit is held to the same content type as a live build.
-	if got := cached.Header().Get(cacheStatusHeader); got != "hit" {
+		for _, pass := range []string{"live", "repeated"} {
-		t.Fatalf("%s = %q, want hit", cacheStatusHeader, got)
+			rec := doGet(t, h, path, "")
-	}
+			if rec.Code != http.StatusOK {
-	srcs, n := sourceValues(t, cached.Body.Bytes(), defaultSourceFact)
+				t.Fatalf("%s %s status %d (%s)", pass, path, rec.Code, rec.Body.String())
-	if n != 1 || srcs["h1"] != "a" {
+			}
-		t.Errorf("cached sources = %v (%d records), want exactly one h1 -> a, the upstream value dropped", srcs, n)
+			if got := rec.Header().Get("Content-Type"); got != "application/json;charset=utf-8" {
+				t.Errorf("%s %s Content-Type = %q, want application/json;charset=utf-8", pass, path, got)
+			}
+		}
 	}
 }

certname.go

@@ -50,8 +50,12 @@ func isCertnameRoute(path string) bool { return certnameFor(path) != "" }
 // which is the record array the merges are built on. The remaining backends are
 // tried after it, so a node only one backend holds is still served and a
 // certname no backend holds still answers with openvoxdb's own 404 body.
+//
+// askOwnerFirst is what makes the owner's reply the answer even when it is an
+// error. A backend that does not hold the certname 404s to say so, so it has no
+// opinion to weigh against the owner's.
 func (s *Server) serveByOwner(w http.ResponseWriter, r *http.Request) {
-	s.proxyOrdered(w, r, s.ownerFirst(r.Context(), certnameFor(r.URL.Path)))
+	s.proxyOrdered(w, r, s.ownerFirst(r.Context(), certnameFor(r.URL.Path)), askOwnerFirst)
 }
 
 // ownerFirst puts the backend holding a certname's newest report ahead of the

certname_test.go

@@ -55,6 +55,13 @@ type voxBackend struct {
 	// nodes maps a certname it holds to that node's report_timestamp.
 	nodes map[string]string
 	fail  bool
+	// perNodeStatus and perNodeBody answer the per-certname routes with a fixed
+	// reply, leaving /nodes alone so the freshness map still resolves an owner.
+	perNodeStatus int
+	perNodeBody   string
+	// perNodeDead aborts the per-certname routes without a status, standing in
+	// for a backend that is unreachable only for that request.
+	perNodeDead bool
 
 	mu        sync.Mutex
 	gotParams map[string]url.Values
@@ -73,6 +80,17 @@ func newVoxBackend(t *testing.T, name string, nodes map[string]string) *voxBacke
 		}
 		w.Header().Set("Content-Type", "application/json;charset=utf-8")
 
+		if isCertnameRoute(r.URL.Path) {
+			if vb.perNodeDead {
+				panic(http.ErrAbortHandler)
+			}
+			if vb.perNodeStatus != 0 {
+				w.WriteHeader(vb.perNodeStatus)
+				_, _ = io.WriteString(w, vb.perNodeBody)
+				return
+			}
+		}
+
 		if certname, ok := strings.CutPrefix(r.URL.Path, factsetsPath+"/"); ok {
 			ts, held := vb.nodes[certname]
 			if !held {
@@ -342,3 +360,76 @@ func TestCertnameRoutes_ClaimedByTheRouteTable(t *testing.T) {
 		}
 	}
 }
+
+// The owner holds the node, so its answer is the answer: a non-owner's 404 only
+// says the node is not its own, and demanding it agree turns the owner's real
+// error into a 502 that describes neither backend.
+func TestCertnameRoutes_OwnerAnswerWins(t *testing.T) {
+	const certname = "h1.example.net"
+	for _, tc := range []struct {
+		name string
+		// setUp configures the owner (a, which reported later) and the backend
+		// that does not hold the node (b).
+		setUp    func(a, b *voxBackend)
+		wantCode int
+		wantBody string
+	}{
+		{
+			name:     "owner errors while the non-owner 404s",
+			setUp:    func(a, b *voxBackend) { a.perNodeStatus, a.perNodeBody = http.StatusInternalServerError, "boom" },
+			wantCode: http.StatusInternalServerError,
+			wantBody: "boom",
+		},
+		{
+			name: "owner and non-owner error differently",
+			setUp: func(a, b *voxBackend) {
+				a.perNodeStatus, a.perNodeBody = http.StatusInternalServerError, "boom"
+				b.perNodeStatus, b.perNodeBody = http.StatusServiceUnavailable, "busy"
+			},
+			wantCode: http.StatusInternalServerError,
+			wantBody: "boom",
+		},
+		{
+			// The fallback chain is there so an unreachable owner does not take the
+			// node with it; a backend that did answer explains more than a 502.
+			name: "owner unreachable, fallback answers",
+			setUp: func(a, b *voxBackend) {
+				a.perNodeDead = true
+				b.perNodeStatus, b.perNodeBody = http.StatusServiceUnavailable, "busy"
+			},
+			wantCode: http.StatusServiceUnavailable,
+			wantBody: "busy",
+		},
+		{
+			name: "owner succeeds while the non-owner errors",
+			setUp: func(a, b *voxBackend) {
+				b.perNodeStatus, b.perNodeBody = http.StatusInternalServerError, "boom"
+			},
+			wantCode: http.StatusOK,
+			wantBody: `"a"`,
+		},
+		{
+			// Nothing answered at all, so there is no upstream reply to replay.
+			name:     "every backend unreachable",
+			setUp:    func(a, b *voxBackend) { a.perNodeDead, b.perNodeDead = true, true },
+			wantCode: http.StatusBadGateway,
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			a := newVoxBackend(t, "a", map[string]string{certname: "2026-07-20T00:00:00Z"})
+			b := newVoxBackend(t, "b", map[string]string{})
+			tc.setUp(a, b)
+			srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
+
+			for _, rt := range perNodePaths {
+				rec := doGet(t, srv.Handler(), rt.path(certname), "")
+				if rec.Code != tc.wantCode {
+					t.Fatalf("%s: status = %d, want %d: %s", rt.name, rec.Code, tc.wantCode, rec.Body.String())
+				}
+				if tc.wantBody != "" && !strings.Contains(rec.Body.String(), tc.wantBody) {
+					t.Errorf("%s: body = %q, want it to contain %q", rt.name, rec.Body.String(), tc.wantBody)
+				}
+			}
+		})
+	}
+}

config.go

@@ -59,7 +59,7 @@ type Config struct {
 	Merge        string        `yaml:"merge"`
 	Timeout      time.Duration `yaml:"timeout"`
 	FreshnessTTL time.Duration `yaml:"freshness_ttl"`
-	FactsTTL     time.Duration `yaml:"facts_ttl"`         // 0 disables the /facts+/nodes cache
+	FactsTTL     time.Duration `yaml:"facts_ttl"`         // 0 disables the merged /nodes cache
 	CacheBytes   int64         `yaml:"facts_cache_bytes"` // byte budget for that cache
 
 	SourceFact        string `yaml:"source_fact"`
@@ -344,8 +344,8 @@ func (c Config) Validate() error {
 	return nil
 }
 
-// cacheEnabled reports whether a facts/nodes cache should be built: both a TTL
+// cacheEnabled reports whether the merged /nodes cache should be built: both a
-// and a byte budget are required.
+// TTL and a byte budget are required.
 func (c Config) cacheEnabled() bool { return c.FactsTTL > 0 && c.CacheBytes > 0 }
 
 func writeDefaultConfig(path string) error {
@@ -366,8 +366,8 @@ func writeDefaultConfig(path string) error {
 		"#   PDBMUX_HEALTH_PROBE_ENABLED, PDBMUX_HEALTH_PROBE_PATH,\n" +
 		"#   PDBMUX_HEALTH_PROBE_INTERVAL, PDBMUX_HEALTH_PROBE_TIMEOUT,\n" +
 		"#   PDBMUX_HEALTH_PROBE_FAILURES, PDBMUX_HEALTH_PROBE_SUCCESSES.\n" +
-		"# facts_ttl caches merged /facts and /nodes in memory; it is capped at 30s\n" +
+		"# facts_ttl caches merged /nodes in memory; it is capped at 30s (a larger\n" +
-		"# (a larger value is clamped) and 0 disables the cache.\n" +
+		"# value is clamped) and 0 disables the cache. Fact answers are never cached.\n" +
 		"# health_probe_* polls each backend's status endpoint so queries skip a\n" +
 		"# backend that is down; when every backend is down all are queried anyway.\n\n")
 	if err := os.WriteFile(path, append(header, data...), 0o644); err != nil {

e2e_upstream_test.go

@@ -4,6 +4,7 @@ package main
 
 import (
 	"context"
+	"net/url"
 	"strings"
 	"testing"
 )
@@ -22,7 +23,7 @@ func TestRejectedQuerySurfacesUpstreamStatus(t *testing.T) {
 		t.Run(tc.name, func(t *testing.T) {
 			params := query(tc.q)
 			status, upstream := h.a.queryRaw(ctx, t, tc.path, params)
-			if !clientShaped(status) {
+			if !replayableStatus(status) || status >= 500 {
 				t.Fatalf("backend %s answered HTTP %d for %s, which is not the client error this test needs: %s",
 					h.a.name, status, tc.q, upstream)
 			}
@@ -56,3 +57,68 @@ func TestRejectedQuerySurfacesUpstreamStatus(t *testing.T) {
 		})
 	}
 }
+
+// openvoxdb answers the same malformed clause with 400 on /nodes and 500 on
+// /facts — engine.clj's rewrite-fact-query runs an unguarded nth only for the
+// facts entity. Both are the backends' real answer, so both have to reach the
+// client rather than one of them becoming a 502.
+func TestUnanimousServerErrorSurfacesUpstreamStatus(t *testing.T) {
+	ctx := context.Background()
+	params := query(`["=","name"]`)
+
+	status, upstream := h.a.queryRaw(ctx, t, factsPath, params)
+	if status < 500 {
+		t.Skipf("backend %s answered HTTP %d for the arity bug this test needs", h.a.name, status)
+	}
+	if other, _ := h.b.queryRaw(ctx, t, factsPath, params); other != status {
+		t.Fatalf("backends answered %d and %d, so the failure is not unanimous", status, other)
+	}
+
+	resp := rawGet(t, factsPath, params)
+	if resp.status != status {
+		t.Fatalf("pdbmux answered HTTP %d, want the upstream %d: %s", resp.status, status, resp.body)
+	}
+	if got, want := strings.TrimSpace(string(resp.body)), strings.TrimSpace(string(upstream)); want != "" && got != want {
+		t.Errorf("pdbmux body = %q, want openvoxdb's own explanation %q", got, want)
+	}
+	for _, b := range []*backend{h.a, h.b} {
+		if strings.Contains(string(resp.body), b.url) {
+			t.Errorf("replayed body names backend %s: %q", b.name, resp.body)
+		}
+	}
+}
+
+// /metrics and /pdb/meta had their own copy of the gateway error, so a status
+// both backends agreed on never reached the client on those routes.
+func TestUnanimousMetricsStatusIsReplayed(t *testing.T) {
+	ctx := context.Background()
+	for _, path := range []string{"/metrics/v1/mbeans", "/metrics/v2/list"} {
+		t.Run(path, func(t *testing.T) {
+			status, _ := h.a.queryRaw(ctx, t, path, nil)
+			if status < 400 {
+				t.Skipf("%s answers HTTP %d on this estate, so there is nothing to replay", path, status)
+			}
+			if other, _ := h.b.queryRaw(ctx, t, path, nil); other != status {
+				t.Fatalf("backends answered %d and %d, so the status is not unanimous", status, other)
+			}
+			if resp := rawGet(t, path, nil); resp.status != status {
+				t.Fatalf("pdbmux answered HTTP %d, want the upstream %d: %s", resp.status, status, resp.body)
+			}
+		})
+	}
+}
+
+// A client cannot tell pdbmux from a PuppetDB by the content type either.
+func TestSuccessContentTypeMatchesUpstream(t *testing.T) {
+	ctx := context.Background()
+	params := url.Values{"limit": {"1"}}
+	_, _ = h.a.queryRaw(ctx, t, nodesPath, params)
+
+	resp := rawGet(t, nodesPath, params)
+	if resp.status != 200 {
+		t.Fatalf("status %d: %s", resp.status, resp.body)
+	}
+	if got := resp.header.Get("Content-Type"); got != "application/json;charset=utf-8" {
+		t.Errorf("Content-Type = %q, want openvoxdb's own %q", got, "application/json;charset=utf-8")
+	}
+}

factroutes_test.go

@@ -370,9 +370,9 @@ func TestHandler_FactsBySourceNameUnknownValueSkipsFanOut(t *testing.T) {
 	}
 }
 
-// The record set is the estate's, not the pinned value's, so every value shares
+// The record set is the estate's, not the pinned value's: the <value> segment is
-// one entry: two valid values must not cost two whole-estate fan-outs.
+// applied to the merged records rather than fanned out as a path of its own.
-func TestHandler_FactsBySourceNameValuesShareOneFetch(t *testing.T) {
+func TestHandler_FactsBySourceNameValuesFilterOneFetch(t *testing.T) {
 	a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "osfamily", "RedHat", "") + `]`})
 	b := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h2", "osfamily", "Debian", "") + `]`})
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
@@ -392,8 +392,8 @@ func TestHandler_FactsBySourceNameValuesShareOneFetch(t *testing.T) {
 		}
 	}
 	for _, cb := range []*countingBackend{a, b} {
-		if got := cb.hitCount(factsPath); got != 1 {
+		if got := cb.hitCount(factsPath); got != 3 {
-			t.Errorf("%s fetched %d times, want 1 shared fetch", factsPath, got)
+			t.Errorf("%s fetched %d times, want one uncached fetch per request", factsPath, got)
 		}
 		if got := cb.hitCount(sourceFactURL); got != 0 {
 			t.Errorf("%s was fanned out %d times, want 0", sourceFactURL, got)
@@ -678,21 +678,27 @@ func TestHandler_FactNamesSurvivesOneBackend(t *testing.T) {
 	}
 }
 
-// Both routes are merged record sets, so they use the same cache /facts does.
+// Every route that serves fact records fans out per request, even with the
-func TestHandler_FactRoutesAreCached(t *testing.T) {
+// cache configured: a fact answer is only ever as old as the backend's own.
+func TestHandler_FactRoutesAreNotCached(t *testing.T) {
 	for _, path := range []string{roleFactPath, roleFactPath + "/web", sourceFactURL, factNamesPath} {
 		t.Run(path, func(t *testing.T) {
+			// The source-fact drilldown is answered from the /facts merge, so
+			// that is the path its fan-out lands on.
+			fanOut := path
+			if path == sourceFactURL {
+				fanOut = factsPath
+			}
 			a := newFakeBackend(t, `[]`, `[]`)
-			a.bodies[path] = `[]`
+			a.bodies[fanOut] = `[]`
 			b := newFakeBackend(t, `[]`, `[]`)
-			b.bodies[path] = `[]`
+			b.bodies[fanOut] = `[]`
 			srv := newTestServer(cacheTestConfig(a.srv.URL, b.srv.URL))
 
-			if got := doGet(t, srv.Handler(), path, "").Header().Get(cacheStatusHeader); got != "miss" {
+			doGet(t, srv.Handler(), path, "")
-				t.Errorf("first %s = %q, want miss", cacheStatusHeader, got)
+			doGet(t, srv.Handler(), path, "")
-			}
+			if got := a.hits(fanOut); got != 2 {
-			if got := doGet(t, srv.Handler(), path, "").Header().Get(cacheStatusHeader); got != "hit" {
+				t.Errorf("backend a saw %d requests for %s, want one per request", got, fanOut)
-				t.Errorf("second %s = %q, want hit", cacheStatusHeader, got)
 			}
 		})
 	}

meta.go

@@ -72,8 +72,10 @@ func (s *Server) rawBackend(ctx context.Context, b Backend, path, rawQuery strin
 	}
 }
 
-// aliveRaw drops backends that errored or answered non-2xx, writing a 502 and
+// aliveRaw drops backends that errored or answered non-2xx, returning ok=false
-// returning ok=false only when none is left.
+// when none is left. The reply it writes then is the same one the merged query
+// handlers write, so /pdb/meta and /metrics answer a failure exactly as
+// /pdb/query does: a unanimous upstream error replayed, anything else a 502.
 func (s *Server) aliveRaw(w http.ResponseWriter, results []rawResult, path string) ([]rawResult, bool) {
 	var alive []rawResult
 	for _, res := range results {
@@ -84,12 +86,25 @@ func (s *Server) aliveRaw(w http.ResponseWriter, results []rawResult, path strin
 		alive = append(alive, res)
 	}
 	if len(alive) == 0 {
-		http.Error(w, "all backends failed", http.StatusBadGateway)
+		s.writeUpstreamError(w, peerOutcome(rawUpstreamErrors(results)))
 		return nil, false
 	}
 	return alive, true
 }
 
+// rawUpstreamErrors reduces a raw fan-out to one entry per backend, nil where
+// the backend answered 2xx or never answered at all.
+func rawUpstreamErrors(results []rawResult) []*upstreamError {
+	errs := make([]*upstreamError, len(results))
+	for i, res := range results {
+		if res.err != nil || res.ok() {
+			continue
+		}
+		errs[i] = newUpstreamError(res.status, res.contentType, res.body)
+	}
+	return errs
+}
+
 func (r rawResult) reason() string {
 	if r.err != nil {
 		return r.err.Error()

meta_test.go

@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"net/http"
 	"net/http/httptest"
+	"strings"
 	"testing"
 )
 
@@ -85,9 +86,9 @@ func TestMetaVersion_OneBackendDown(t *testing.T) {
 
 func TestMetaVersion_AllBackendsDown(t *testing.T) {
 	a := newFakeBackend(t, `[]`, `[]`)
-	a.fail = true
+	a.dead = true
 	b := newFakeBackend(t, `[]`, `[]`)
-	b.fail = true
+	b.dead = true
 	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
 
 	if rec := metaGet(t, srv.Handler(), metaVersionPath); rec.Code != http.StatusBadGateway {
@@ -95,6 +96,60 @@ func TestMetaVersion_AllBackendsDown(t *testing.T) {
 	}
 }
 
+// /pdb/meta had its own copy of the gateway error, so a status both backends
+// agreed on never reached the client. It answers failures the way the query
+// routes do now.
+func TestMeta_ReplaysUnanimousUpstreamStatus(t *testing.T) {
+	for _, path := range []string{metaVersionPath, metaServerTimePath} {
+		for _, status := range []int{http.StatusNotFound, http.StatusForbidden, http.StatusInternalServerError} {
+			t.Run(path+"/"+http.StatusText(status), func(t *testing.T) {
+				a := newFakeBackend(t, `[]`, `[]`)
+				b := newFakeBackend(t, `[]`, `[]`)
+				a.reject, a.rejectBody = status, "upstream said no"
+				b.reject, b.rejectBody = status, "upstream said no"
+				srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+				rec := metaGet(t, srv.Handler(), path)
+				if rec.Code != status {
+					t.Fatalf("status %d, want the upstream %d: %s", rec.Code, status, rec.Body.String())
+				}
+				if !strings.Contains(rec.Body.String(), "upstream said no") {
+					t.Errorf("body = %q, want the upstream explanation", rec.Body.String())
+				}
+			})
+		}
+	}
+}
+
+func TestMeta_DisagreeingStatusesStay502(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	a.reject, a.rejectBody = http.StatusNotFound, "gone"
+	b.reject, b.rejectBody = http.StatusInternalServerError, "boom"
+	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+	if rec := metaGet(t, srv.Handler(), metaVersionPath); rec.Code != http.StatusBadGateway {
+		t.Errorf("status %d, want 502 when backends disagree", rec.Code)
+	}
+}
+
+// A replayed body must not name a backend, on /pdb/meta as anywhere else.
+func TestMeta_ReplayedBodyIsRedacted(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	a.reject, a.rejectBody = http.StatusInternalServerError, "upstream "+a.srv.URL+" blew up"
+	b.reject, b.rejectBody = http.StatusInternalServerError, "upstream "+b.srv.URL+" blew up"
+	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+	rec := metaGet(t, srv.Handler(), metaVersionPath)
+	if rec.Code != http.StatusInternalServerError {
+		t.Fatalf("status %d, want the upstream 500", rec.Code)
+	}
+	if strings.Contains(rec.Body.String(), hostOf(t, a.srv.URL)) {
+		t.Errorf("replayed body names a backend: %q", rec.Body.String())
+	}
+}
+
 func TestMetaServerTime_FirstReachableBackend(t *testing.T) {
 	a := newFakeBackend(t, `[]`, `[]`)
 	a.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T01:00:00.000Z"}`

metrics.go

@@ -65,7 +65,13 @@ func (s *Server) handleMetrics(w http.ResponseWriter, r *http.Request) {
 		writeRaw(w, alive[0])
 		return
 	}
-	w.Header().Set("Content-Type", "application/json")
+	// The merged body is the backends' own payload reshaped, so it keeps their
+	// content type rather than announcing a different one.
+	contentType := alive[0].contentType
+	if contentType == "" {
+		contentType = jsonContentType
+	}
+	w.Header().Set("Content-Type", contentType)
 	_, _ = w.Write(body)
 }
 

metrics_test.go

@@ -6,6 +6,7 @@ import (
 	"net/http/httptest"
 	"reflect"
 	"strconv"
+	"strings"
 	"testing"
 )
 
@@ -229,9 +230,9 @@ func TestMetrics_OneBackendDown(t *testing.T) {
 
 func TestMetrics_AllBackendsDown(t *testing.T) {
 	a := newFakeBackend(t, `[]`, `[]`)
-	a.fail = true
+	a.dead = true
 	b := newFakeBackend(t, `[]`, `[]`)
-	b.fail = true
+	b.dead = true
 	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
 
 	if rec := metaGet(t, srv.Handler(), numNodesEscaped); rec.Code != http.StatusBadGateway {
@@ -239,6 +240,57 @@ func TestMetrics_AllBackendsDown(t *testing.T) {
 	}
 }
 
+// /metrics/v1/mbeans is a unanimous 404 on this estate and /metrics/v2 a
+// unanimous 403; both used to come back as 502.
+func TestMetrics_ReplaysUnanimousUpstreamStatus(t *testing.T) {
+	for _, status := range []int{http.StatusNotFound, http.StatusForbidden, http.StatusInternalServerError} {
+		t.Run(http.StatusText(status), func(t *testing.T) {
+			a := newFakeBackend(t, `[]`, `[]`)
+			b := newFakeBackend(t, `[]`, `[]`)
+			a.reject, a.rejectBody = status, "Forbidden request: /metrics/v2/list (method :get)."
+			b.reject, b.rejectBody = status, "Forbidden request: /metrics/v2/list (method :get)."
+			srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+			rec := metaGet(t, srv.Handler(), numNodesEscaped)
+			if rec.Code != status {
+				t.Fatalf("status %d, want the upstream %d: %s", rec.Code, status, rec.Body.String())
+			}
+			if !strings.Contains(rec.Body.String(), "Forbidden request") {
+				t.Errorf("body = %q, want the upstream explanation", rec.Body.String())
+			}
+		})
+	}
+}
+
+func TestMetrics_DisagreeingStatusesStay502(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	a.reject, a.rejectBody = http.StatusNotFound, "gone"
+	b.reject, b.rejectBody = http.StatusForbidden, "denied"
+	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+	if rec := metaGet(t, srv.Handler(), numNodesEscaped); rec.Code != http.StatusBadGateway {
+		t.Errorf("status %d, want 502 when backends disagree", rec.Code)
+	}
+}
+
+// One backend refusing is not the estate's answer, so the other still serves.
+func TestMetrics_OneRefusalStillServesTheOther(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	a.reject, a.rejectBody = http.StatusForbidden, "denied"
+	b.bodies[numNodesPath] = `{"request":{},"value":{"Value":7},"status":200}`
+	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+	rec := metaGet(t, srv.Handler(), numNodesEscaped)
+	if rec.Code != http.StatusOK {
+		t.Fatalf("status %d, want the surviving backend's 200: %s", rec.Code, rec.Body.String())
+	}
+	if got := metricValue(t, rec.Body.Bytes())["Value"]; got != float64(7) {
+		t.Errorf("Value = %v, want 7", got)
+	}
+}
+
 func TestMetrics_RejectsNonGET(t *testing.T) {
 	a := newFakeBackend(t, `[]`, `[]`)
 	b := newFakeBackend(t, `[]`, `[]`)

server.go

@@ -32,11 +32,6 @@ const (
 	// PuppetDB only sends this when the request carries include_total=true.
 	recordsHeader = "X-Records"
 
-	// Set by pdbmux, not by PuppetDB: how a cache-backed response was answered
-	// and how old the served copy is.
-	cacheStatusHeader = "X-Cache"
-	ageHeader         = "Age"
-
 	// Set by pdbmux: "<contributed>/<configured>" backends behind a merged response.
 	backendsHeader = "X-Backends"
 )
@@ -50,21 +45,28 @@ type backendResult struct {
 
 var errAllBackendsFailed = errors.New("all backends failed")
 
+// jsonContentType is the spelling openvoxdb puts on the wire for a query
+// response, charset and all. Its own reader treats the space after the
+// semicolon as insignificant (http.clj:55-66, simple-utf8-ctype?), but a client
+// comparing the raw header must see no difference between pdbmux and a PuppetDB.
+const jsonContentType = "application/json;charset=utf-8"
+
 type Server struct {
 	cfg    Config
 	client *http.Client
 	log    *log.Logger
 
-	// factsCache is nil when caching is disabled; cacheFor hands out a noop then.
+	// nodeCache is nil when caching is disabled; cacheFor hands out a noop then.
-	factsCache Cache
+	nodeCache Cache
-	flights    flightGroup
+	flights   flightGroup
-	stale      staleTracker
+	stale     staleTracker
 
 	// health is nil when probing is disabled, which makes every backend healthy.
 	health  *prober
 	partial partialTracker
 
-	// now is shared with the cache's clock so Age matches the stored timestamp.
+	// now is shared with the cache's clock so a stale fallback is reported
+	// against the same timestamps the cache stamps entries with.
 	now func() time.Time
 
 	// freshness cache (freshness merge only).
@@ -82,7 +84,7 @@ func NewServer(cfg Config, logger *log.Logger) *Server {
 		now:    time.Now,
 	}
 	if cfg.cacheEnabled() {
-		s.factsCache = newMemoryCache(cfg.FactsTTL, cfg.CacheBytes)
+		s.nodeCache = newMemoryCache(cfg.FactsTTL, cfg.CacheBytes)
 	}
 	s.health = newProber(cfg, logger)
 	return s
@@ -95,20 +97,21 @@ func (s *Server) StartProbes(ctx context.Context) { s.health.Start(ctx) }
 // StopProbes stops the probing goroutines and waits for them to exit.
 func (s *Server) StopProbes() { s.health.Stop() }
 
-// cacheFor picks the cache backing a request. The merged /nodes and fact record
+// cacheFor picks the cache backing a request. Only the merged /nodes record set
-// sets — /facts, /facts/<name>[/<value>] and /fact-names — share the in-memory
+// is cached: its report columns move once per node run, while fact answers must
-// cache; every other path is uncached until the reports cache lands, and a new
+// be as current as a backend's own, so no fact-serving path is cached here.
-// backend is a case here rather than a change to any handler.
+// Every other path is uncached until the reports cache lands, and a new backend
+// is a case here rather than a change to any handler.
 func (s *Server) cacheFor(path string, params url.Values) (Cache, bool) {
-	switch {
+	switch path {
-	case path == factsPath, path == nodesPath, path == factNamesPath, isFactsSubPath(path):
+	case nodesPath:
 		// An aggregate row is a combined count, not the merged record set the
 		// cache was built for, so it stays on the live path.
 		if spec, err := parseAggregate(params.Get("query")); spec != nil || err != nil {
 			return noopCache{}, false
 		}
-		if s.factsCache != nil {
+		if s.nodeCache != nil {
-			return s.factsCache, true
+			return s.nodeCache, true
 		}
 	}
 	return noopCache{}, false
@@ -527,11 +530,9 @@ func (s *Server) serveFirstHolder(w http.ResponseWriter, r *http.Request) {
 		alive = append(alive, res)
 	}
 	if len(alive) == 0 {
-		if ue := unanimousClientError(results); ue != nil {
+		// Unanimity is the whole answer here too: every backend saying 404 means
-			s.writeUpstreamError(w, ue)
+		// nobody holds the report, while one silent backend leaves that unknown.
-			return
+		s.writeUpstreamError(w, peerOutcome(backendUpstreamErrors(results)))
-		}
-		http.Error(w, "no backend holds this report", http.StatusNotFound)
 		return
 	}
 	for _, res := range alive {
@@ -543,8 +544,8 @@ func (s *Server) serveFirstHolder(w http.ResponseWriter, r *http.Request) {
 	writeJSON(w, nil)
 }
 
-// Returns an *upstreamError when every backend refused the query the same way,
+// Returns an *upstreamError when every backend answered the same way, and
-// and errAllBackendsFailed when every backend failed for any other reason.
+// errAllBackendsFailed when every backend failed for any other reason.
 func (s *Server) aliveResults(ctx context.Context, path string, params url.Values) ([]backendResult, error) {
 	results := s.fanOut(ctx, path, params)
 
@@ -554,13 +555,17 @@ func (s *Server) aliveResults(ctx context.Context, path string, params url.Value
 			alive = append(alive, res)
 		}
 	}
+	var unanimous *upstreamError
 	if len(alive) == 0 {
-		// A query every backend refuses identically is the client's mistake, not an
+		unanimous = unanimousUpstreamError(backendUpstreamErrors(results))
-		// outage, so it is neither logged as one nor counted as degraded service.
+	}
-		if ue := unanimousClientError(results); ue != nil {
+	// A query every backend refuses identically is the client's mistake, not an
-			s.log.Printf("info: every backend refused %s: %v", path, ue)
+	// outage, so it is neither logged as one nor counted as degraded service. A
-			return nil, ue
+	// unanimous 5xx is replayed just as faithfully, but it is the backends
-		}
+	// reporting their own fault, so it still counts against service health.
+	if clientRefusal(unanimous) {
+		s.log.Printf("info: every backend refused %s: %v", path, unanimous)
+		return nil, unanimous
 	}
 	for _, res := range results {
 		if res.err != nil {
@@ -569,6 +574,9 @@ func (s *Server) aliveResults(ctx context.Context, path string, params url.Value
 	}
 	s.partial.record(len(alive), len(s.cfg.Backends), s.now())
 	if len(alive) == 0 {
+		if unanimous != nil {
+			return nil, unanimous
+		}
 		return nil, errAllBackendsFailed
 	}
 	return alive, nil
@@ -584,10 +592,9 @@ type cachedResponse struct {
 	Configured int             `json:"configured"` // backends configured at build time
 }
 
-// recordFilter narrows a response's records after it has been built or read back
+// recordFilter narrows a response's records after it has been built, so a route
-// from the cache, so requests differing only in the filter share one stored entry
+// that answers from another path's merge can pin part of it per request. It
-// and one fan-out. It leaves Records alone, so it only suits responses that set
+// leaves Records alone, so it only suits responses that set no X-Records.
-// no X-Records.
 type recordFilter func([]json.RawMessage) []json.RawMessage
 
 func (f recordFilter) apply(resp cachedResponse) cachedResponse {
@@ -633,7 +640,7 @@ func (s *Server) serveFiltered(w http.ResponseWriter, r *http.Request, path stri
 		s.log.Printf("warning: cache lookup for %s failed: %v", key, err)
 	case status == CacheFresh:
 		s.stale.markFresh()
-		s.writeStored(w, ent, CacheFresh, filter)
+		s.writeStored(w, ent, filter)
 		return
 	case status == CacheStale:
 		stale = &ent
@@ -672,18 +679,17 @@ func (s *Server) serveFiltered(w http.ResponseWriter, r *http.Request, path stri
 		}
 		// A refused query is answered, not degraded, so stale records are no reply
 		// to it: the client has to see why the query was rejected.
-		if stale != nil && !errors.As(err, new(*upstreamError)) {
+		if stale != nil && !clientRefusal(err) {
 			s.stale.markStale(s.now())
 			s.log.Printf("warning: serving stale %s from cache (stored %s): %v",
 				path, stale.StoredAt.UTC().Format(time.RFC3339), err)
-			s.writeStored(w, *stale, CacheStale, filter)
+			s.writeStored(w, *stale, filter)
 			return
 		}
 		s.writeUpstreamError(w, err)
 		return
 	}
 	s.stale.markFresh()
-	s.setCacheHeaders(w, CacheMiss, time.Time{})
 	writeCached(w, filter.apply(resp))
 }
 
@@ -696,41 +702,21 @@ func (s *Server) flightTimeout() time.Duration {
 	return defaultTimeout
 }
 
-func (s *Server) writeStored(w http.ResponseWriter, ent CacheEntry, status CacheStatus, filter recordFilter) {
+// writeStored answers from a cache entry. Whether a response came from the cache
+// is deliberately invisible to the client: PuppetDB sets no X-Cache or Age, so
+// neither does pdbmux, and /healthz carries the cache state instead.
+func (s *Server) writeStored(w http.ResponseWriter, ent CacheEntry, filter recordFilter) {
 	var resp cachedResponse
 	if err := json.Unmarshal(ent.Body, &resp); err != nil {
 		s.log.Printf("warning: unreadable cache entry: %v", err)
 		http.Error(w, "unreadable cache entry", http.StatusBadGateway)
 		return
 	}
-	s.setCacheHeaders(w, status, ent.StoredAt)
 	writeCached(w, filter.apply(resp))
 }
 
-// setCacheHeaders labels a response from a cache-backed path: X-Cache is
-// hit/stale/miss and Age is whole seconds since the served copy was stored (0
-// for a response built by this request). It reads the same clock the cache
-// stamps entries with, so the two never disagree.
-func (s *Server) setCacheHeaders(w http.ResponseWriter, status CacheStatus, storedAt time.Time) {
-	label := "miss"
-	switch status {
-	case CacheFresh:
-		label = "hit"
-	case CacheStale:
-		label = "stale"
-	}
-	age := 0
-	if !storedAt.IsZero() {
-		if secs := int(s.now().Sub(storedAt).Seconds()); secs > 0 {
-			age = secs
-		}
-	}
-	w.Header().Set(cacheStatusHeader, label)
-	w.Header().Set(ageHeader, strconv.Itoa(age))
-}
-
 func writeCached(w http.ResponseWriter, resp cachedResponse) {
-	w.Header().Set("Content-Type", "application/json")
+	w.Header().Set("Content-Type", jsonContentType)
 	if resp.Records >= 0 {
 		w.Header().Set(recordsHeader, strconv.Itoa(resp.Records))
 	}
@@ -911,18 +897,21 @@ func (s *Server) queryBackend(ctx context.Context, b Backend, path string, param
 	return recs, total, err
 }
 
-// The record shape is unknown, so a union would be guesswork: the first 2xx wins and the first error response is replayed when none succeeds.
+// The record shape is unknown, so a union would be guesswork: the first 2xx wins, and every backend was asked the same question, so only a reply they all gave is replayed.
 func (s *Server) proxyUnmerged(w http.ResponseWriter, r *http.Request) {
-	s.proxyOrdered(w, r, s.cfg.Backends)
+	s.proxyOrdered(w, r, s.cfg.Backends, askPeers)
 }
 
-// proxyOrdered asks backends in the given order, which is what decides the answer when more than one of them holds the path.
+// proxyOrdered asks backends in the given order, which is what decides the
-func (s *Server) proxyOrdered(w http.ResponseWriter, r *http.Request, backends []Backend) {
+// answer when more than one of them holds the path. order says why they are in
-	var fallback *bufferedResponse
+// that order, which is what decides whose reply is served when none answers 2xx.
+func (s *Server) proxyOrdered(w http.ResponseWriter, r *http.Request, backends []Backend, order askOrder) {
+	replies := make([]*upstreamError, 0, len(backends))
 	for _, b := range backends {
 		resp, err := s.passThrough(r, b)
 		if err != nil {
 			s.log.Printf("warning: backend %q pass-through failed for %s: %v", b.Name, r.URL.Path, err)
+			replies = append(replies, nil)
 			continue
 		}
 		if resp.StatusCode >= 200 && resp.StatusCode < 300 {
@@ -934,27 +923,9 @@ func (s *Server) proxyOrdered(w http.ResponseWriter, r *http.Request, backends [
 		}
 		body, _ := io.ReadAll(resp.Body)
 		_ = resp.Body.Close()
-		if fallback == nil {
+		replies = append(replies, newUpstreamError(resp.StatusCode, resp.Header.Get("Content-Type"), body))
-			fallback = &bufferedResponse{
-				status:      resp.StatusCode,
-				contentType: resp.Header.Get("Content-Type"),
-				body:        body,
-			}
-		}
 	}
-	if fallback == nil {
+	s.writeUpstreamError(w, order.outcome(replies))
-		http.Error(w, "all backends failed", http.StatusBadGateway)
-		return
-	}
-	setContentType(w, fallback.contentType)
-	w.WriteHeader(fallback.status)
-	_, _ = w.Write(s.redactBackends(fallback.body))
-}
-
-type bufferedResponse struct {
-	status      int
-	contentType string
-	body        []byte
 }
 
 func (s *Server) passThrough(r *http.Request, b Backend) (*http.Response, error) {
@@ -1037,8 +1008,8 @@ type cacheHealth struct {
 func (s *Server) cacheHealth() cacheHealth {
 	stats := CacheStats{Backend: "none"}
 	ttl := time.Duration(0)
-	if s.factsCache != nil {
+	if s.nodeCache != nil {
-		stats = s.factsCache.Stats()
+		stats = s.nodeCache.Stats()
 		ttl = s.cfg.FactsTTL
 	}
 	serving, served, last := s.stale.snapshot()
@@ -1106,7 +1077,7 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
 }
 
 func writeJSON(w http.ResponseWriter, recs []json.RawMessage) {
-	w.Header().Set("Content-Type", "application/json")
+	w.Header().Set("Content-Type", jsonContentType)
 	if recs == nil {
 		recs = []json.RawMessage{}
 	}

server_test.go

@@ -29,8 +29,11 @@ type fakeBackend struct {
 	// totals is the X-Records count advertised per path when the request asks
 	// for include_total.
 	totals map[string]int
-	fail   bool          // return 500 for everything
+	fail   bool // answer 500 for everything, as a backend admitting a fault
-	delay  time.Duration // artificial latency
+	// dead aborts the connection instead of answering, standing in for a backend
+	// that is unreachable rather than one that replies badly.
+	dead  bool
+	delay time.Duration // artificial latency
 	// reject answers every request with this status and rejectBody, standing in
 	// for a PuppetDB refusing a query it cannot answer.
 	reject     int
@@ -60,6 +63,9 @@ func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend {
 		fb.gotParams[r.URL.Path] = r.URL.Query()
 		fb.rawPaths = append(fb.rawPaths, r.URL.EscapedPath())
 		fb.mu.Unlock()
+		if fb.dead {
+			panic(http.ErrAbortHandler)
+		}
 		if fb.fail {
 			http.Error(w, "boom", http.StatusInternalServerError)
 			return
@@ -285,7 +291,7 @@ func TestHandler_OneBackendDown(t *testing.T) {
 func TestHandler_BothBackendsDown(t *testing.T) {
 	a := newFakeBackend(t, `[]`, `[]`)
 	b := newFakeBackend(t, `[]`, `[]`)
-	a.fail, b.fail = true, true
+	a.dead, b.dead = true, true
 	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
 
 	rec := doGet(t, srv.Handler(), nodesPath, "")

upstream.go

@@ -35,45 +35,42 @@ func (e *upstreamError) Error() string {
 	return fmt.Sprintf("HTTP %d: %s", e.status, strings.TrimSpace(string(e.body)))
 }
 
-// clientShaped reports whether a status blames the request rather than the
+// replayableStatus reports whether a status is an error a backend explained and
-// backend or pdbmux itself. openvoxdb answers every bad query with 400 —
+// so can stand as the answer when every backend gave it. Everything from 400 up
-// src/puppetlabs/puppetdb/middleware.clj:98-116 and http.clj:115-129 — and its
+// qualifies, because openvoxdb does not reserve 5xx for its own faults: a
-// other 4xx say something a client of pdbmux neither caused nor can fix: 403 is
+// malformed query reaches the client as a 400 only when something on the way
-// pdbmux's own certificate being refused (middleware.clj:44-58); 404 is either
+// down happens to throw a class the query engine catches (query_eng.clj:571-583
-// an absent object or an unknown path (http.clj:238-242,
+// catches IllegalArgumentException and the :puppetlabs.puppetdb.query/invalid
-// middleware.clj:381-398), and which objects a backend holds is the one thing
+// ExceptionInfo), and as a 500 otherwise. ["=","name"] is a 400 on /nodes and a
-// backends are meant to disagree about; 408 and 429 report a backend's timing
+// 500 with an empty body on /facts, because only the facts entity runs
-// and capacity. A backend that times out mid-query answers 200 with a truncated
+// rewrite-fact-query, whose name-constraint does an unguarded (nth clause 2)
-// body rather than any 4xx (query_eng.clj:463-483), so no timeout reaches here.
+// (engine.clj:2988-3003); a bare string or number for query is a 500 reading
+// "Output of convert-query-params does not match schema" from the schema check
+// in front of the parser. Drawing the line at 500 would replay one of those and
+// swallow the other, which is the route-dependence clients notice.
 //
-// Every 4xx left over blames the request and so is safe to replay, though none
+// Below 400 nothing is replayed. A 3xx carries its meaning in Location, which
-// is reachable while the fan-out forwards no client header: 406 needs an Accept
+// the fan-out does not keep, and a non-200 2xx is not an error at all.
-// the query app refuses (http/server.clj:72; must-accept-type in
+func replayableStatus(status int) bool {
-// http.clj:136-145 is unwired), and 415 a Content-Encoding on a POST to
+	return status >= 400
-// /commands (middleware.clj:165-178), which pdbmux never proxies.
-func clientShaped(status int) bool {
-	switch status {
-	case http.StatusForbidden, http.StatusNotFound,
-		http.StatusRequestTimeout, http.StatusTooManyRequests:
-		return false
-	}
-	return status >= 400 && status < 500
 }
 
-// unanimousClientError returns the rejection to replay when every backend in a
+// unanimousUpstreamError returns the reply to replay when every backend answered
-// fan-out refused the same query with the same client-shaped status, and nil
+// the same request the same way, and nil otherwise. Every backend is asked the
-// otherwise. Every backend is asked the same question, so unanimity is what
+// same question, so unanimity is what separates the estate's real answer from a
-// distinguishes a bad query from a sick estate: a transport failure, a 5xx, or
+// sick estate: a transport failure, or two backends disagreeing on the status,
-// two backends disagreeing on the status all leave at least one backend whose
+// leaves at least one backend whose answer is evidence about the backend rather
-// answer is evidence about the backend rather than about the request.
+// than about the request, and that is what 502 describes.
+//
+// A nil entry stands for a backend that did not answer with an HTTP status at
+// all, which defeats unanimity.
 //
 // The reply returned is the first in configured order, so a client retrying a
 // rejected query is told the same thing every time.
-func unanimousClientError(results []backendResult) *upstreamError {
+func unanimousUpstreamError(errs []*upstreamError) *upstreamError {
 	var first *upstreamError
-	for _, res := range results {
+	for _, ue := range errs {
-		var ue *upstreamError
+		if ue == nil || !replayableStatus(ue.status) {
-		if !errors.As(res.err, &ue) || !clientShaped(ue.status) {
 			return nil
 		}
 		if first == nil {
@@ -85,9 +82,99 @@ func unanimousClientError(results []backendResult) *upstreamError {
 	return first
 }
 
-// writeUpstreamError answers a fan-out that produced no records. A unanimous
+// askOrder says why a set of backends was asked, which is what decides whose
-// client-shaped rejection is replayed with the backend's own status and
+// reply becomes the client's answer when none of them answered 2xx. The two
-// explanation; anything else is reported as a gateway failure.
+// cases are not settings on one rule, they are different questions, and reading
+// a reply under the wrong one is how a real answer turns into a 502.
+type askOrder int
+
+const (
+	// askPeers: every backend was asked the same question and any of them could
+	// have answered it, so each reply is an opinion about that question. Only a
+	// reply they all gave is the estate's answer; anything else leaves a backend
+	// whose reply is evidence about the backend, which is what 502 reports.
+	askPeers askOrder = iota
+	// askOwnerFirst: the path names one entity, so the backend holding it is
+	// asked first — or, where no owner is known, they are asked in configured
+	// order — and the rest only as fallbacks. A backend that does not hold the
+	// entity replies 404 to say exactly that, which is not a dissent from the
+	// holder's reply, so requiring the two to agree asks a question nobody was
+	// posed. The first backend that answered is the one that knows.
+	askOwnerFirst
+)
+
+// outcome turns a round's replies — one entry per backend in the order they were
+// asked, nil where a backend produced no HTTP status at all — into the error to
+// answer with.
+func (o askOrder) outcome(replies []*upstreamError) error {
+	if o == askOwnerFirst {
+		return ownerOutcome(replies)
+	}
+	return peerOutcome(replies)
+}
+
+// peerOutcome is the error to answer a fan-out of peers with when it produced
+// nothing: their unanimous reply where there is one, and pdbmux's gateway error
+// otherwise.
+func peerOutcome(errs []*upstreamError) error {
+	if ue := unanimousUpstreamError(errs); ue != nil {
+		return ue
+	}
+	return errAllBackendsFailed
+}
+
+// ownerOutcome is the error to answer an owner-routed path with: the reply of
+// the first backend that answered. The owner is asked first, so that is the
+// owner's own reply — it holds the entity, so its 500 is the truth about this
+// request whatever a backend that does not hold the entity said.
+//
+// An owner that produced no status at all leaves a nil entry and the next
+// backend's reply stands instead. The fallback chain exists so a node whose
+// owner is unreachable is still served, and a backend that did answer explains
+// more than a 502 that describes neither. Only a round where nothing answered is
+// pdbmux's own gateway error.
+func ownerOutcome(replies []*upstreamError) error {
+	for _, ue := range replies {
+		if ue != nil && replayableStatus(ue.status) {
+			return ue
+		}
+	}
+	return errAllBackendsFailed
+}
+
+// backendUpstreamErrors reduces a merged fan-out's results to one entry per
+// backend, nil where the backend answered or failed without a status.
+func backendUpstreamErrors(results []backendResult) []*upstreamError {
+	errs := make([]*upstreamError, len(results))
+	for i, res := range results {
+		var ue *upstreamError
+		if errors.As(res.err, &ue) {
+			errs[i] = ue
+		}
+	}
+	return errs
+}
+
+// clientRefusal reports whether err is a unanimous upstream rejection blaming
+// the request. Only those mean the estate is healthy and the query was wrong,
+// which is why they are neither counted as degraded service nor answered from
+// the cache; a unanimous 5xx is replayed just the same but is the backends
+// reporting their own fault.
+//
+// Every 4xx openvoxdb answers a fan-out with is about the request, including the
+// 403 on /metrics/v2/list, which is that backend's own policy rather than an
+// authentication failure. That holds only while pdbmux presents no client
+// certificate: authenticate to backends and a rejected certificate becomes a
+// 403 on every route at once, which this would read as a healthy estate refusing
+// a bad query and hide a total outage behind a replayed 403.
+func clientRefusal(err error) bool {
+	var ue *upstreamError
+	return errors.As(err, &ue) && ue != nil && ue.status < 500
+}
+
+// writeUpstreamError answers a round that produced no records. An upstream
+// error the askOrder resolved to is replayed with the backend's own status,
+// content type and explanation; anything else is reported as a gateway failure.
 func (s *Server) writeUpstreamError(w http.ResponseWriter, err error) {
 	var ue *upstreamError
 	if !errors.As(err, &ue) {
@@ -98,7 +185,7 @@ func (s *Server) writeUpstreamError(w http.ResponseWriter, err error) {
 	if len(strings.TrimSpace(string(body))) == 0 {
 		setContentType(w, "text/plain; charset=utf-8")
 		w.WriteHeader(ue.status)
-		_, _ = fmt.Fprintf(w, "upstream rejected the query: %s\n", http.StatusText(ue.status))
+		_, _ = fmt.Fprintf(w, "upstream answered %d %s\n", ue.status, http.StatusText(ue.status))
 		return
 	}
 	setContentType(w, ue.contentType)

upstream_test.go

@@ -14,11 +14,16 @@ import (
 
 const badOrderBy = `Unrecognized column 'bogus' specified in :order_by`
 
+// upstreamJSONContentType is spelled out rather than taken from jsonContentType,
+// so the assertion is against what openvoxdb answers and not against whatever
+// pdbmux happens to be configured with.
+const upstreamJSONContentType = "application/json;charset=utf-8"
+
 func upErr(status int, body string) error {
 	return newUpstreamError(status, "text/plain; charset=utf-8", []byte(body))
 }
 
-func TestUnanimousClientError_AgreementRule(t *testing.T) {
+func TestUnanimousUpstreamError_AgreementRule(t *testing.T) {
 	refused := errors.New("dial tcp: connection refused")
 	for _, tc := range []struct {
 		name string
@@ -29,18 +34,25 @@ func TestUnanimousClientError_AgreementRule(t *testing.T) {
 		{"all agree on 415", []error{upErr(415, "bad media"), upErr(415, "bad media")}, 415},
 		{"differing bodies still agree", []error{upErr(400, "one"), upErr(400, "two")}, 400},
 		{"disagreeing 4xx", []error{upErr(400, "bad"), upErr(404, "gone")}, 0},
-		// Both client-shaped, so only the same-status rule refuses these.
 		{"agree on shape, disagree on code", []error{upErr(400, "x"), upErr(415, "y")}, 0},
 		{"a majority agrees", []error{upErr(400, "x"), upErr(400, "y"), upErr(422, "z")}, 0},
 		{"the first differs", []error{upErr(422, "z"), upErr(400, "x"), upErr(400, "y")}, 0},
 		{"4xx with a 5xx", []error{upErr(400, "bad"), upErr(500, "boom")}, 0},
 		{"4xx with a transport failure", []error{upErr(400, "bad"), refused}, 0},
-		{"all 403", []error{upErr(403, "denied"), upErr(403, "denied")}, 0},
+		{"5xx with a transport failure", []error{upErr(500, "boom"), refused}, 0},
-		{"all 404", []error{upErr(404, "gone"), upErr(404, "gone")}, 0},
+		// Statuses the old 4xx-only rule swallowed. Every one of them is what
-		{"all 429", []error{upErr(429, "slow down"), upErr(429, "slow down")}, 0},
+		// both backends actually said, so every one of them is the answer.
-		{"all 408", []error{upErr(408, "too slow"), upErr(408, "too slow")}, 0},
+		{"all 403", []error{upErr(403, "denied"), upErr(403, "denied")}, 403},
-		{"all 500", []error{upErr(500, "boom"), upErr(500, "boom")}, 0},
+		{"all 404", []error{upErr(404, "gone"), upErr(404, "gone")}, 404},
-		{"all 503", []error{upErr(503, "unavailable"), upErr(503, "unavailable")}, 0},
+		{"all 429", []error{upErr(429, "slow down"), upErr(429, "slow down")}, 429},
+		{"all 408", []error{upErr(408, "too slow"), upErr(408, "too slow")}, 408},
+		{"all 500", []error{upErr(500, "boom"), upErr(500, "boom")}, 500},
+		{"all 503", []error{upErr(503, "unavailable"), upErr(503, "unavailable")}, 503},
+		{"disagreeing 5xx", []error{upErr(500, "boom"), upErr(503, "later")}, 0},
+		// Nothing below 400 is an error a backend explained, and a 3xx's meaning
+		// lives in a Location header the fan-out never kept.
+		{"all 302", []error{upErr(302, ""), upErr(302, "")}, 0},
+		{"all 204", []error{upErr(204, ""), upErr(204, "")}, 0},
 		{"no backends", nil, 0},
 		{"a backend succeeded", []error{upErr(400, "bad"), nil}, 0},
 	} {
@@ -49,7 +61,7 @@ func TestUnanimousClientError_AgreementRule(t *testing.T) {
 			for i, err := range tc.errs {
 				results[i] = backendResult{name: fmt.Sprintf("b%d", i), err: err}
 			}
-			got := unanimousClientError(results)
+			got := unanimousUpstreamError(backendUpstreamErrors(results))
 			switch {
 			case tc.want == 0 && got != nil:
 				t.Fatalf("want no replay, got HTTP %d", got.status)
@@ -64,17 +76,33 @@ func TestUnanimousClientError_AgreementRule(t *testing.T) {
 
 // Two backends can explain the same rejection differently; the reply is the
 // first in configured order so a retried query is answered the same way twice.
-func TestUnanimousClientError_PicksFirstInConfiguredOrder(t *testing.T) {
+func TestUnanimousUpstreamError_PicksFirstInConfiguredOrder(t *testing.T) {
 	results := []backendResult{
 		{name: "a", err: upErr(400, "from a")},
 		{name: "b", err: upErr(400, "from b")},
 	}
-	got := unanimousClientError(results)
+	got := unanimousUpstreamError(backendUpstreamErrors(results))
 	if got == nil || !strings.Contains(string(got.body), "from a") {
 		t.Fatalf("body = %q, want the first backend's", got)
 	}
 }
 
+// Only a 4xx says the estate is well and the query was wrong. A 5xx is replayed
+// too, but it must keep counting as a backend fault.
+func TestClientRefusal_OnlyFourXX(t *testing.T) {
+	for status, want := range map[int]bool{400: true, 404: true, 429: true, 499: true, 500: false, 503: false} {
+		if got := clientRefusal(upErr(status, "x")); got != want {
+			t.Errorf("clientRefusal(%d) = %v, want %v", status, got, want)
+		}
+	}
+	if clientRefusal(errAllBackendsFailed) {
+		t.Error("a transport failure is not a client refusal")
+	}
+	if clientRefusal(unanimousUpstreamError(nil)) {
+		t.Error("an absent unanimous error is not a client refusal")
+	}
+}
+
 func TestUpstreamError_BodyIsCapped(t *testing.T) {
 	ue := newUpstreamError(400, "text/plain", []byte(strings.Repeat("x", upstreamBodyLimit*2)))
 	if len(ue.body) != upstreamBodyLimit {
@@ -112,31 +140,78 @@ func TestHandler_MergedReplaysNonBadRequestStatus(t *testing.T) {
 	}
 }
 
-// A unanimous 403 is pdbmux's own credentials being refused, not the client's
+// A status every backend agreed on is the estate's own answer whatever it is,
-// query, so it must not be handed back as the client's fault.
+// so the statuses the 4xx-only rule used to swallow now reach the client.
-func TestHandler_MergedForbiddenStays502(t *testing.T) {
+func TestHandler_MergedReplaysEveryUnanimousStatus(t *testing.T) {
-	a := newFakeBackend(t, `[]`, `[]`)
+	for _, tc := range []struct {
-	b := newFakeBackend(t, `[]`, `[]`)
+		status int
-	a.reject, a.rejectBody = http.StatusForbidden, "certificate not allowed"
+		body   string
-	b.reject, b.rejectBody = http.StatusForbidden, "certificate not allowed"
+	}{
-	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+		{http.StatusForbidden, "certificate not allowed"},
+		{http.StatusNotFound, "Not Found"},
+		{http.StatusTooManyRequests, "slow down"},
+		{http.StatusInternalServerError, "Output of convert-query-params does not match schema"},
+		{http.StatusServiceUnavailable, "starting up"},
+	} {
+		t.Run(http.StatusText(tc.status), func(t *testing.T) {
+			a := newFakeBackend(t, `[]`, `[]`)
+			b := newFakeBackend(t, `[]`, `[]`)
+			a.reject, a.rejectBody = tc.status, tc.body
+			b.reject, b.rejectBody = tc.status, tc.body
+			srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
 
-	if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway {
+			rec := doGet(t, srv.Handler(), nodesPath, "")
-		t.Fatalf("status = %d, want 502", rec.Code)
+			if rec.Code != tc.status {
+				t.Fatalf("status = %d, want the upstream %d: %s", rec.Code, tc.status, rec.Body.String())
+			}
+			if !strings.Contains(rec.Body.String(), tc.body) {
+				t.Errorf("body = %q, want the upstream explanation", rec.Body.String())
+			}
+		})
 	}
 }
 
-// Every backend answering 404 is ambiguous between a bad path and a record
+// A backend that answered nothing at all leaves no unanimity to replay, whatever
-// nobody holds, so it stays a gateway error on a merged route.
+// the others said.
-func TestHandler_MergedNotFoundStays502(t *testing.T) {
+func TestHandler_MergedServerErrorWithUnreachableStays502(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
+	dead := newFakeBackend(t, `[]`, `[]`)
+	deadURL := dead.srv.URL
+	dead.srv.Close()
+	srv := newTestServer(testConfig(a.srv.URL, deadURL, mergeStatic))
+
+	if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway {
+		t.Fatalf("status = %d, want 502 when a backend never answered", rec.Code)
+	}
+}
+
+func TestHandler_MergedDisagreeingServerErrorsStay502(t *testing.T) {
 	a := newFakeBackend(t, `[]`, `[]`)
 	b := newFakeBackend(t, `[]`, `[]`)
-	a.reject, a.rejectBody = http.StatusNotFound, "Not Found"
+	a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
-	b.reject, b.rejectBody = http.StatusNotFound, "Not Found"
+	b.reject, b.rejectBody = http.StatusServiceUnavailable, "later"
 	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
 
 	if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway {
-		t.Fatalf("status = %d, want 502", rec.Code)
+		t.Fatalf("status = %d, want 502 when backends disagree", rec.Code)
+	}
+}
+
+// A unanimous 5xx is replayed, but it is the backends admitting a fault, so it
+// still has to read as degraded service.
+func TestHandler_UnanimousServerErrorIsStillADegradedRound(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
+	b.reject, b.rejectBody = http.StatusInternalServerError, "boom"
+	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+	if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusInternalServerError {
+		t.Fatalf("status = %d, want the upstream 500", rec.Code)
+	}
+	if hr := health(t, srv); hr.Query.PartialRounds == 0 {
+		t.Errorf("query health = %+v, want the 5xx counted as a failed round", hr.Query)
 	}
 }
 
@@ -255,7 +330,7 @@ func TestHandler_ReportSubResourceRejectionReplayedButNotItsAbsence(t *testing.T
 	b := newFakeBackend(t, `[]`, `[]`)
 	if rec := doGet(t, newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)).Handler(),
 		reportsPath+"/nope/events", ""); rec.Code != http.StatusNotFound {
-		t.Fatalf("status = %d, want pdbmux's own 404 when nobody holds the report", rec.Code)
+		t.Fatalf("status = %d, want 404 when nobody holds the report", rec.Code)
 	}
 
 	a.reject, a.rejectBody = http.StatusBadRequest, badOrderBy
@@ -296,18 +371,18 @@ func TestHandler_RejectedQueryIsNotCached(t *testing.T) {
 	b.reject, b.rejectBody = http.StatusBadRequest, badOrderBy
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	rec := doGet(t, srv.Handler(), factsPath, "")
+	rec := doGet(t, srv.Handler(), nodesPath, "")
 	if rec.Code != http.StatusBadRequest {
 		t.Fatalf("status = %d, want the upstream 400 replayed: %s", rec.Code, rec.Body.String())
 	}
-	if n := srv.factsCache.Stats().Entries; n != 0 {
+	if n := srv.nodeCache.Stats().Entries; n != 0 {
 		t.Fatalf("cache holds %d entries after a rejected query", n)
 	}
 
 	// The rejection left nothing behind, so the next good query is served fresh.
 	a.reject, b.reject = 0, 0
-	a.factsBody = `[` + fact("h1", "role", "web", "") + `]`
+	a.nodesBody = `[` + node("web", "2026-01-01T00:00:00.000Z") + `]`
-	ok := doGet(t, srv.Handler(), factsPath, "")
+	ok := doGet(t, srv.Handler(), nodesPath, "")
 	if ok.Code != http.StatusOK || !strings.Contains(ok.Body.String(), "web") {
 		t.Fatalf("follow-up = %d %s", ok.Code, ok.Body.String())
 	}
@@ -316,13 +391,13 @@ func TestHandler_RejectedQueryIsNotCached(t *testing.T) {
 func TestHandler_AllBackendsFailedIsNotCached(t *testing.T) {
 	a := newFakeBackend(t, `[]`, `[]`)
 	b := newFakeBackend(t, `[]`, `[]`)
-	a.fail, b.fail = true, true
+	a.dead, b.dead = true, true
 	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	if rec := doGet(t, srv.Handler(), factsPath, ""); rec.Code != http.StatusBadGateway {
+	if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway {
 		t.Fatalf("status = %d, want 502", rec.Code)
 	}
-	if n := srv.factsCache.Stats().Entries; n != 0 {
+	if n := srv.nodeCache.Stats().Entries; n != 0 {
 		t.Fatalf("cache holds %d entries after a failed fan-out", n)
 	}
 }
@@ -330,18 +405,18 @@ func TestHandler_AllBackendsFailedIsNotCached(t *testing.T) {
 // Stale records answer an outage. They do not answer a query the estate refused:
 // the client has to be told why, not handed data for a question it did not ask.
 func TestHandler_RejectedQueryIsNotAnsweredFromStale(t *testing.T) {
-	a := newFakeBackend(t, `[]`, `[`+fact("h1", "role", "old", "")+`]`)
+	a := newFakeBackend(t, `[`+node("old", "2026-01-01T00:00:00.000Z")+`]`, `[]`)
 	b := newFakeBackend(t, `[]`, `[]`)
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	if warm := doGet(t, srv.Handler(), factsPath, ""); warm.Code != http.StatusOK {
+	if warm := doGet(t, srv.Handler(), nodesPath, ""); warm.Code != http.StatusOK {
 		t.Fatalf("warm-up status %d", warm.Code)
 	}
 
 	clk.advance(31 * time.Second)
 	a.reject, a.rejectBody = http.StatusBadRequest, badOrderBy
 	b.reject, b.rejectBody = http.StatusBadRequest, badOrderBy
-	rec := doGet(t, srv.Handler(), factsPath, "")
+	rec := doGet(t, srv.Handler(), nodesPath, "")
 	if rec.Code != http.StatusBadRequest {
 		t.Fatalf("status = %d, want the rejection rather than the stale entry: %s", rec.Code, rec.Body.String())
 	}
@@ -355,16 +430,16 @@ func TestHandler_RejectedQueryIsNotAnsweredFromStale(t *testing.T) {
 
 // An outage still falls back to the stale copy, unchanged by the replay path.
 func TestHandler_OutageStillFallsBackToStale(t *testing.T) {
-	a := newFakeBackend(t, `[]`, `[`+fact("h1", "role", "old", "")+`]`)
+	a := newFakeBackend(t, `[`+node("old", "2026-01-01T00:00:00.000Z")+`]`, `[]`)
 	b := newFakeBackend(t, `[]`, `[]`)
 	srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
 
-	if warm := doGet(t, srv.Handler(), factsPath, ""); warm.Code != http.StatusOK {
+	if warm := doGet(t, srv.Handler(), nodesPath, ""); warm.Code != http.StatusOK {
 		t.Fatalf("warm-up status %d", warm.Code)
 	}
 	clk.advance(31 * time.Second)
 	a.fail, b.fail = true, true
-	rec := doGet(t, srv.Handler(), factsPath, "")
+	rec := doGet(t, srv.Handler(), nodesPath, "")
 	if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), "old") {
 		t.Fatalf("stale fallback = %d %s", rec.Code, rec.Body.String())
 	}
@@ -493,3 +568,100 @@ func hostOf(t *testing.T, raw string) string {
 	}
 	return u.Host
 }
+
+// Every backend answering 500 is the estate's answer on the first-holder route
+// as much as on a merged one.
+func TestHandler_ReportSubResourceReplaysUnanimousServerError(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
+	b.reject, b.rejectBody = http.StatusInternalServerError, "boom"
+
+	rec := doGet(t, newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)).Handler(),
+		reportsPath+"/nope/events", "")
+	if rec.Code != http.StatusInternalServerError {
+		t.Fatalf("status = %d, want the upstream 500: %s", rec.Code, rec.Body.String())
+	}
+}
+
+// One backend saying 404 and one saying nothing leaves it unknown whether the
+// report exists, which is the gateway error's job to say.
+func TestHandler_ReportSubResourceUnreachableBackendIs502(t *testing.T) {
+	a := newFakeBackend(t, `[]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	b.dead = true
+
+	rec := doGet(t, newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)).Handler(),
+		reportsPath+"/nope/events", "")
+	if rec.Code != http.StatusBadGateway {
+		t.Fatalf("status = %d, want 502 when a backend never answered", rec.Code)
+	}
+}
+
+// The pass-through route used to replay whichever backend answered first, even
+// when the others disagreed or never answered. It follows the same rule now.
+func TestProxyUnmerged_ReplayRule(t *testing.T) {
+	for _, tc := range []struct {
+		name    string
+		aStatus int
+		bStatus int
+		bDead   bool
+		want    int
+	}{
+		{"unanimous 500 is replayed", 500, 500, false, 500},
+		{"unanimous 404 is replayed", 404, 404, false, 404},
+		{"unanimous 403 is replayed", 403, 403, false, 403},
+		{"disagreement is a gateway error", 400, 404, false, http.StatusBadGateway},
+		{"a silent backend is a gateway error", 404, 0, true, http.StatusBadGateway},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			a := newFakeBackend(t, `[]`, `[]`)
+			b := newFakeBackend(t, `[]`, `[]`)
+			a.reject, a.rejectBody = tc.aStatus, "upstream said no"
+			b.reject, b.rejectBody = tc.bStatus, "upstream said no"
+			b.dead = tc.bDead
+			srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+
+			rec := doGet(t, srv.Handler(), resourcesPath, `["=","type","File"]`)
+			if rec.Code != tc.want {
+				t.Fatalf("status = %d, want %d: %s", rec.Code, tc.want, rec.Body.String())
+			}
+		})
+	}
+}
+
+// pdbmux has to be indistinguishable from a PuppetDB, and openvoxdb answers a
+// query with a charset on the content type.
+func TestHandler_SuccessContentTypeMatchesUpstream(t *testing.T) {
+	a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
+	h := srv.Handler()
+
+	for _, path := range []string{nodesPath, factsPath, factNamesPath, reportsPath} {
+		t.Run(path, func(t *testing.T) {
+			rec := doGet(t, h, path, "")
+			if rec.Code != http.StatusOK {
+				t.Fatalf("status = %d: %s", rec.Code, rec.Body.String())
+			}
+			if got := rec.Header().Get("Content-Type"); got != upstreamJSONContentType {
+				t.Errorf("Content-Type = %q, want %q", got, upstreamJSONContentType)
+			}
+		})
+	}
+}
+
+func TestHandler_CachedSuccessContentTypeMatchesUpstream(t *testing.T) {
+	a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`)
+	b := newFakeBackend(t, `[]`, `[]`)
+	srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
+	h := srv.Handler()
+
+	// The miss builds the entry and the hit replays it; both are the client's view.
+	for _, label := range []string{"miss", "hit"} {
+		rec := doGet(t, h, nodesPath, "")
+		if got := rec.Header().Get("Content-Type"); got != upstreamJSONContentType {
+			t.Errorf("%s Content-Type = %q, want %q", label, got, upstreamJSONContentType)
+		}
+	}
+}