21 Commits

Author SHA1 Message Date
benvin 9254396227 Merge pull request 'Report the backend host as the pdbmux_source value' (#31) from benvin/pdbmux-source-fact-fqdn into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #31
2026-10-10 01:08:14 +11:00
benvin c75d0db69e Merge pull request 'Answer AST name queries for pdbmux_source on /facts' (#30) from benvin/pdbmux-source-fact-ast into main
Reviewed-on: #30
2026-10-10 01:06:52 +11:00
unkin-agent 3fe8c5b19f Give e2e backends distinct hosts and assert the source fact host
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2026-10-09 23:55:24 +11:00
unkin-agent a47f907538 Merge remote-tracking branch 'origin/benvin/pdbmux-source-fact-ast' into benvin/pdbmux-source-fact-fqdn
# Conflicts:
#	source_test.go
2026-10-09 23:55:24 +11:00
unkin-agent b1d68f5c80 Check backend hosts only when the source fact is enabled, compare them case-insensitively
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2026-10-09 23:49:42 +11:00
unkin-agent 97ddc378ff Narrow the source-fact /facts fetch to the query's certnames
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2026-10-09 23:47:56 +11:00
unkin-agent a43e5481f0 Report the backend host as the source fact value
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2026-10-09 23:46:21 +11:00
unkin-agent 0aff1746e2 Answer AST name queries for the source fact on /facts
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2026-10-09 23:41:57 +11:00
benvin 7e2d5f7115 Merge pull request 'ci: switch Go steps to gobuilder, enable S3 build cache' (#29) from benvin/gocache into main
Reviewed-on: #29
2026-10-03 00:10:22 +10:00
unkin-agent 9aac54a4d6 ci: switch Go steps to gobuilder, enable S3 build cache
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
golang:1.25 and the stale almalinux9-gobuilder pin are replaced by
gobuilder:0.1.2-alma9 on every step that invokes the Go toolchain, with
GOCACHEPROG wired to the baked-in go-cache-plugin against the shared
S3 cache bucket.

- build, pre-commit, test: image -> gobuilder:0.1.2-alma9
- add GOCACHE_* env + AWS creds from org secrets, absolute cache-dir
- lint step (golangci-lint, no gobuilder binary) and docker buildx left untouched
2026-10-02 23:50:10 +10:00
benvin 2652778306 Merge pull request 'Serve fact queries live and drop the cache headers' (#26) from benvin/drop-fact-caching into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #26
2026-09-13 15:04:26 +10:00
unkin-agent 5c9d1e9055 Merge main into benvin/drop-fact-caching
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Both sides added jsonContentType; keep one. The no-entry 502 test kills the
backends outright, since main replays a unanimous 500, and the cached
content-type test moves to /nodes, the only cached path left.
2026-09-13 14:54:08 +10:00
benvin 3bea2e9b71 Merge pull request 'Replay every unanimous upstream status, not just 4xx' (#27) from benvin/replay-upstream-errors into main
Reviewed-on: #27
2026-09-13 14:31:56 +10:00
unkin-agent 85c9088293 Say that owner-first covers a certname with no resolved owner
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
2026-09-13 14:13:44 +10:00
unkin-agent 72380f27e6 Serve the owner's own answer on the per-certname routes
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was canceled
Unanimity is the right rule for a fan-out of peers, but the per-certname
routes are not one: a backend that does not hold the certname answers 404
to say so, not to disagree, so requiring it to agree turned the owner's
real 500 into a 502 that described neither backend.

- Add askOrder, which says whether a set of backends was asked as peers
  or owner-first, and resolve each round's replies under its own rule.
- Serve the first backend that answered on owner-routed paths, so an
  unreachable owner still falls back rather than collapsing to 502.
- Keep unanimity for the merged, meta, metrics and pass-through routes.
- Cover the owner routes: owner errors against a non-owner 404, both
  erroring differently, an unreachable owner, and a non-owner error
  behind the owner's 200.
- Record what clientRefusal's 4xx exemption assumes about client certs.
2026-09-13 14:11:44 +10:00
unkin-agent 121bfacc2f Replay every unanimous upstream status, not just 4xx
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
openvoxdb does not reserve 5xx for its own faults: the same malformed
query is a 400 on /nodes and a 500 on /facts, and /metrics answers a flat
403, so a 4xx-only replay rule made pdbmux's behaviour depend on the
route. The meta and metrics handlers held their own copy of the gateway
error and bypassed the replay entirely.

- Replay any status from 400 up that every backend agreed on, with the
  backend's own body and content type.
- Keep 502 for backends disagreeing on the status, or a backend that
  answered nothing at all.
- Route /pdb/meta, /metrics and the pass-through path through the same
  rule as the merged query handlers.
- Count a unanimous 5xx as a failed round and let it fall back to a stale
  cache entry; only a unanimous 4xx stays exempt from both.
- Answer successful queries with openvoxdb's application/json;charset=utf-8.
2026-09-13 13:35:21 +10:00
unkin-agent 24f6d73d5c Serve fact queries live and drop the cache headers
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Fact answers must be as current as a backend's own, and X-Cache/Age are
headers PuppetDB never sends.

- serve /facts, /facts/<name>[/<value>] and /fact-names live on every request
- keep the in-memory cache on merged /nodes only
- drop X-Cache and Age everywhere; /healthz still reports cache state
- answer successful queries with PuppetDB's application/json;charset=utf-8
2026-09-13 13:34:52 +10:00
benvin 5207a79ad4 Merge pull request 'Resolve the per-certname routes to the node's owning backend' (#25) from benvin/merge-per-node-fact-routes into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #25
2026-09-12 20:33:22 +10:00
unkin-agent eeb44057db Resolve the per-certname routes to the node's owning backend
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
The paths keyed on one certname took the pass-through, so a node both
backends hold answered from whichever was configured first while /facts
answered from whichever held its newer report.

- add a route claiming /pdb/query/v4/{nodes,factsets,catalogs}/<certname>
- order the backends for it by the freshness map the /facts merge uses
- try the remaining backends after the owner, replaying upstream's 404 when none holds the certname
- assert ownership, the upstream 404 body, a failed backend and query forwarding against captured openvoxdb shapes
2026-09-12 17:58:38 +10:00
benvin 94769a7864 Merge pull request 'Match backend addresses case-insensitively when redacting' (#24) from benvin/redact-backend-case into main
Reviewed-on: #24
2026-09-08 07:02:01 +10:00
benvin 6a3b277e5e Merge pull request 'Replay a unanimous upstream rejection instead of a 502' (#23) from benvin/upstream-errors into main
Reviewed-on: #23
2026-09-07 23:07:55 +10:00
27 changed files with 2092 additions and 541 deletions
+13 -1
View File
@@ -3,7 +3,19 @@ when:
steps: steps:
- name: build - name: build
image: golang:1.25 image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/gobuilder:0.1.2-alma9
environment:
GOCACHE_S3_BUCKET: gocache
GOCACHE_S3_REGION: us-east-1
GOCACHE_S3_ENDPOINT_URL: https://s3.ceph.unkin.net
GOCACHE_S3_PATH_STYLE: "true"
GOCACHE_KEY_PREFIX: ci-pdbmux
GOCACHE_METRICS: "true"
GOCACHEPROG: go-cache-plugin --cache-dir=/tmp/gocache
AWS_ACCESS_KEY_ID:
from_secret: GOCACHE_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY:
from_secret: GOCACHE_AWS_SECRET_ACCESS_KEY
commands: commands:
- make build - make build
backend_options: backend_options:
+13 -1
View File
@@ -3,7 +3,19 @@ when:
steps: steps:
- name: pre-commit - name: pre-commit
image: git.unkin.net/unkin/almalinux9-gobuilder:20260606 image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/gobuilder:0.1.2-alma9
environment:
GOCACHE_S3_BUCKET: gocache
GOCACHE_S3_REGION: us-east-1
GOCACHE_S3_ENDPOINT_URL: https://s3.ceph.unkin.net
GOCACHE_S3_PATH_STYLE: "true"
GOCACHE_KEY_PREFIX: ci-pdbmux
GOCACHE_METRICS: "true"
GOCACHEPROG: go-cache-plugin --cache-dir=/tmp/gocache
AWS_ACCESS_KEY_ID:
from_secret: GOCACHE_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY:
from_secret: GOCACHE_AWS_SECRET_ACCESS_KEY
commands: commands:
- uvx pre-commit run --all-files - uvx pre-commit run --all-files
backend_options: backend_options:
+13 -1
View File
@@ -18,7 +18,19 @@ steps:
cpu: 2 cpu: 2
- name: test - name: test
image: golang:1.25 image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/gobuilder:0.1.2-alma9
environment:
GOCACHE_S3_BUCKET: gocache
GOCACHE_S3_REGION: us-east-1
GOCACHE_S3_ENDPOINT_URL: https://s3.ceph.unkin.net
GOCACHE_S3_PATH_STYLE: "true"
GOCACHE_KEY_PREFIX: ci-pdbmux
GOCACHE_METRICS: "true"
GOCACHEPROG: go-cache-plugin --cache-dir=/tmp/gocache
AWS_ACCESS_KEY_ID:
from_secret: GOCACHE_AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY:
from_secret: GOCACHE_AWS_SECRET_ACCESS_KEY
commands: commands:
- go test -v -race ./... - go test -v -race ./...
backend_options: backend_options:
+81 -57
View File
@@ -24,7 +24,7 @@ not PQL) is forwarded verbatim.
| Path | Behaviour | | Path | Behaviour |
|---|---| |---|---|
| `GET /pdb/query/v4/nodes` | Fan out to all backends, dedupe by `certname`, keep the record with the newer `report_timestamp`, stamped with the winning backend's name (see provenance). An `extract` query with a `function` column is **combined** instead. | | `GET /pdb/query/v4/nodes` | Fan out to all backends, dedupe by `certname`, keep the record with the newer `report_timestamp`, stamped with the winning backend's host (see provenance). An `extract` query with a `function` column is **combined** instead. |
| `GET /pdb/query/v4/facts` | Fan out to all, and per `certname` keep **all** facts from the backend that owns that node (see merge semantics), plus a synthetic `pdbmux_source` fact naming it. An `extract` query with a `function` column is **combined** instead. | | `GET /pdb/query/v4/facts` | Fan out to all, and per `certname` keep **all** facts from the backend that owns that node (see merge semantics), plus a synthetic `pdbmux_source` fact naming it. An `extract` query with a `function` column is **combined** instead. |
| `GET /pdb/query/v4/facts/<name>[/<value>]` | Same fan-out and merge as `/facts`, and an `extract` query with a `function` column is **combined** the same way. The path segment is a `name` constraint, so no synthetic `pdbmux_source` record is added — except on the fact's own path, which is **synthesised** from the `/facts` merge (see provenance). | | `GET /pdb/query/v4/facts/<name>[/<value>]` | Same fan-out and merge as `/facts`, and an `extract` query with a `function` column is **combined** the same way. The path segment is a `name` constraint, so no synthetic `pdbmux_source` record is added — except on the fact's own path, which is **synthesised** from the `/facts` merge (see provenance). |
| `GET /pdb/query/v4/fact-names` | Fan out to all and serve the **union** of the flat name arrays, deduped and re-sorted, re-paged across backends, plus the `pdbmux_source` name while injection is on. `order_by` is only valid on `name`. | | `GET /pdb/query/v4/fact-names` | Fan out to all and serve the **union** of the flat name arrays, deduped and re-sorted, re-paged across backends, plus the `pdbmux_source` name while injection is on. `order_by` is only valid on `name`. |
@@ -34,6 +34,7 @@ not PQL) is forwarded verbatim.
| `GET /pdb/query/v4/event-counts` | Fan out to all and **sum** each subject's counts into one row per subject. | | `GET /pdb/query/v4/event-counts` | Fan out to all and **sum** each subject's counts into one row per subject. |
| `GET /pdb/query/v4/aggregate-event-counts` | Fan out to all and **sum** the summary object's counts. | | `GET /pdb/query/v4/aggregate-event-counts` | Fan out to all and **sum** the summary object's counts. |
| `GET /pdb/query/v4/reports/<hash>/{events,logs,metrics}` | Ask every backend; serve the answer from whichever backend actually holds that report. `404` when none does. | | `GET /pdb/query/v4/reports/<hash>/{events,logs,metrics}` | Ask every backend; serve the answer from whichever backend actually holds that report. `404` when none does. |
| `GET /pdb/query/v4/{nodes,factsets,catalogs}/<certname>[/...]` | Ask the backend that owns that `certname` — the same owner the `/facts` merge attributes records to — and serve its reply verbatim, success or error. The remaining backends are tried after it, so a node only one backend holds is still served, and openvoxdb's own `404` body is replayed when none holds it. |
| `GET /pdb/query/v4/*` (any other) | No merge rule, so backends are tried in configured order and the first success is streamed back verbatim; if all reject it, the first upstream error response is replayed. | | `GET /pdb/query/v4/*` (any other) | No merge rule, so backends are tried in configured order and the first success is streamed back verbatim; if all reject it, the first upstream error response is replayed. |
| `GET /pdb/meta/v1/version` | Fan out to all and report the **lowest** version any backend runs. | | `GET /pdb/meta/v1/version` | Fan out to all and report the **lowest** version any backend runs. |
| `GET /pdb/meta/v1/server-time` | Fan out to all and serve the first reachable backend's clock. | | `GET /pdb/meta/v1/server-time` | Fan out to all and serve the first reachable backend's clock. |
@@ -46,22 +47,38 @@ Fan-out is concurrent, and goes only to the backends the health prober currently
believes are up — see [Backend health](#backend-health). If one backend errors or believes are up — see [Backend health](#backend-health). If one backend errors or
times out, `pdbmux` serves the surviving backends' results and logs a warning; a times out, `pdbmux` serves the surviving backends' results and logs a warning; a
merged endpoint only returns `502` when **every** backend fails. Response records merged endpoint only returns `502` when **every** backend fails. Response records
are passed through as raw JSON so unknown fields survive untouched. are passed through as raw JSON so unknown fields survive untouched, under
openvoxdb's own `application/json;charset=utf-8`.
Every backend is asked the same question, so a query all of them *refuse* with Every backend is asked the same question, so a status **every** backend answered
the same client-shaped status — a `400` naming an unknown field, say — is the with is the estate's own answer, not an outage: that status, openvoxdb's own
query's fault rather than an outage: that status and openvoxdb's own explanation explanation and its content type are replayed to the client instead of a `502`,
are replayed to the client instead of a `502`, with any backend address stripped with any backend address stripped out of the body first. This holds for every
out of the body first. Backends disagreeing on the status, a `403` (`pdbmux`'s status from `400` up — openvoxdb answers `["=","name"]` with `400` on `/nodes`
own credentials, not the client's), a `404` (which records a backend holds is but `500` on `/facts`, and `/metrics/v2` with a flat `403`, so a rule drawn at
exactly what backends disagree about), `408`, `429` and every `5xx` still return `500` would replay one and swallow the other. `502` is kept for what it actually
`502`. A refused query is not counted as a partial round on `/healthz`, and describes: backends **disagreeing** on the status, or a backend that answered
nothing about it is cached. nothing at all. The rule is the same on `/pdb/query`, `/pdb/meta` and `/metrics`,
so no route answers a failure differently from any other.
Unanimity is asked for only where the backends are peers answering the same
question. The per-certname routes are not: one backend **owns** the node and the
others are fallbacks, so the owner's reply is the answer whatever they said — a
backend that does not hold the certname answers `404` to say the node is not its
own, not to disagree about the request. When the owner answers nothing at all the
first fallback that did answer stands in, so an unreachable owner does not take
the node with it, and `502` is left for a request no backend answered.
A unanimous `4xx` blames the request, so it is not counted as a partial round on
`/healthz` and nothing about it is cached. A unanimous `5xx` is the backends
reporting their own fault, so it is replayed just as faithfully but still counts
as a failed round and still falls back to a stale cache entry where there is one.
Responses carry PuppetDB's `X-Records` when the query asked for a total, and on Responses carry PuppetDB's `X-Records` when the query asked for a total, and on
the merged paths `X-Backends` (see [Backend health](#backend-health)). Cached the merged paths `X-Backends` (see [Backend health](#backend-health)). Successful
paths add two more headers `pdbmux` sets itself, `X-Cache` and `Age` — see query responses carry PuppetDB's own `Content-Type: application/json;charset=utf-8`.
[Caching](#caching). Nothing tells a client whether a response came from the cache: PuppetDB sets no
`X-Cache` or `Age`, so neither does `pdbmux` — see [Caching](#caching).
## Merge semantics ## Merge semantics
@@ -188,8 +205,10 @@ response itself, so nothing has to query each backend to find out:
- **`/facts`** gains one extra fact record per `certname`, alongside the node's - **`/facts`** gains one extra fact record per `certname`, alongside the node's
real facts, in the shape of a real fact record — `certname`, `name`, `value`, real facts, in the shape of a real fact record — `certname`, `name`, `value`,
`environment` — with `value` set to the **backend name** from `backends` / `environment` — with `value` set to the **hostname** of the backend's URL from
`PDBMUX_BACKENDS`. `environment` is copied from that node's own facts (all `backends` / `PDBMUX_BACKENDS` (no scheme or port), e.g.
`puppetdb.puppet.svc.cluster.local` for `new=http://puppetdb.puppet.svc.cluster.local:8080`.
Two backends may therefore not share a hostname; config validation rejects it. `environment` is copied from that node's own facts (all
four keys are always present, since clients index them directly). four keys are always present, since clients index them directly).
- **`/nodes`** gains a `pdbmux_source` **key** on each merged node record. A - **`/nodes`** gains a `pdbmux_source` **key** on each merged node record. A
node record carries no facts, so this is a synthetic field, not a fact — the node record carries no facts, so this is a synthetic field, not a fact — the
@@ -230,7 +249,13 @@ matters more.
friends ask for specific facts, and the synthetic record is not one of them. friends ask for specific facts, and the synthetic record is not one of them.
Only the outer query is inspected: a `name` filter inside an `in`/`select_facts` Only the outer query is inspected: a `name` filter inside an `in`/`select_facts`
subquery narrows which *nodes* match, not which facts come back, so injection subquery narrows which *nodes* match, not which facts come back, so injection
still happens; still happens. The exception is a query whose `name` comparison (`=`, `~` or
`in` an `array`) selects `pdbmux_source` itself, built only from
`and`/`or`/`not` over `=`, `~` and `in`-`array` on `certname`, `environment`,
`name` and `value`: it is still fanned out for the real facts it names, and
the synthetic records are added after being filtered by the query locally.
Their fetch carries only the query's top-level `and` conjuncts on
`certname`, so a single-host lookup stays single-host;
- the path is `/facts/<name>` for any other fact. The path segment is the same - the path is `/facts/<name>` for any other fact. The path segment is the same
outer `name` constraint, so only the fact's own path carries the record; outer `name` constraint, so only the fact's own path carries the record;
- injection is turned off (see `source_fact_enabled`). - injection is turned off (see `source_fact_enabled`).
@@ -242,13 +267,13 @@ record of that name, so the route does not serve its own fan-out: it takes the
records from the `/facts` merge that produces them, which makes the `certname` records from the `/facts` merge that produces them, which makes the `certname`
set, the owner and the `environment` identical to the ones an unfiltered `/facts` set, the owner and the `environment` identical to the ones an unfiltered `/facts`
response reports, and lets the request's own `query` narrow the result upstream. response reports, and lets the request's own `query` narrow the result upstream.
That costs one `/facts` fan-out per cache miss — the widest fan-out `pdbmux` That costs one `/facts` fan-out per request — the widest fan-out `pdbmux`
makes — on a rare, user-initiated path. makes — on a rare, user-initiated path.
`/facts/pdbmux_source/<value>` pins the backend name, so it answers with the `/facts/pdbmux_source/<value>` pins the backend host, so it answers with the
nodes that backend owns. The `<value>` segment never reaches that fan-out: the nodes that backend owns. The `<value>` segment never reaches that fan-out: the
synthetic record's value is always a backend name, so a value naming none is synthetic record's value is always a backend host, so any other value — a
answered `[]` from the configured names alone, with no fan-out at all, and a backend's name included — is answered `[]` from the configured hosts alone, with no fan-out at all, and a
value naming one filters a record set fetched under a key the value is not part value naming one filters a record set fetched under a key the value is not part
of. The record set is a property of the estate rather than of the filter, so of. The record set is a property of the estate rather than of the filter, so
every value of it — and the unfiltered path — share one entry and one fetch. every value of it — and the unfiltered path — share one entry and one fetch.
@@ -264,11 +289,12 @@ such a query correctly for every operator (`not`, `or`, subqueries) and does not
pretend to for some. Read the fact from an unfiltered (or `certname`-filtered) pretend to for some. Read the fact from an unfiltered (or `certname`-filtered)
`/facts` response, from the path route above, and filter client-side. `/facts` response, from the path route above, and filter client-side.
**Not covered:** `/factsets` and `/inventory`. Both carry facts, but `pdbmux` **Not covered:** `/factsets`, `/inventory` and the per-certname routes. The
does not merge either today — they take the unmerged pass-through path, where first two carry facts but are not merged today — they take the unmerged
the answer comes from whichever backend replied first rather than from a merge pass-through path, where the answer comes from whichever backend replied first
winner, so there is no owner to attribute. Injecting there would state a rather than from a merge winner, so there is no owner to attribute. The
provenance that isn't true. per-certname routes do resolve to an owner, but their bodies are passed through
verbatim rather than rebuilt, so nothing is added to them either.
### Metadata and metrics ### Metadata and metrics
@@ -402,8 +428,8 @@ not answering.
asked, never what a merged answer means: a response built from a subset is asked, never what a merged answer means: a response built from a subset is
still served, as before. Every merged response carries `X-Backends: still served, as before. Every merged response carries `X-Backends:
<contributed>/<configured>` naming how many backends' records went into it, so <contributed>/<configured>` naming how many backends' records went into it, so
a client can tell a full answer from a partial one. On a cache hit the header a client can tell a full answer from a partial one. On a `/nodes` cache hit the
describes the stored body, not the current backend count. header describes the stored body, not the current backend count.
- **`/healthz`** gives each backend a `state` (`healthy`, `unhealthy`, - **`/healthz`** gives each backend a `state` (`healthy`, `unhealthy`,
`probe_unsupported`, `unprobed`, or `unmonitored` when probing is off), `probe_unsupported`, `unprobed`, or `unmonitored` when probing is off),
`consecutive_failures`, `consecutive_failures`,
@@ -423,21 +449,27 @@ not answering.
## Caching ## Caching
`pdbmux` caches merged `/nodes`, `/facts`, `/facts/<name>[/<value>]` and `pdbmux` caches the merged `/nodes` record set **in memory** so a busy
`/fact-names` record sets **in memory** so a busy Puppetboard does not re-fan-out Puppetboard does not re-fan-out the node list every few seconds. Its report
the same query every few seconds — its facts overview and fact drilldown are two columns only move when a node finishes a run, so a `30s` answer is a `30s`-old
of the pages that hit hardest. Everything else runs uncached — including report timestamp and nothing else.
`extract` aggregates on those paths, and
the `/pdb/meta/v1/*` and `/metrics/*` endpoints, which are served live on every **No fact-serving path is cached.** `/facts`, `/facts/<name>[/<value>]`,
request. The cache is an interface, and `/reports` gets its own (S3-backed) `/fact-names`, `/factsets*`, `/nodes/<certname>/facts`, `/fact-contents`,
backend later without further handler changes. `/fact-paths` and `/inventory` all fan out on every request, so a fact answer is
exactly as current as the backend's own. Cache fact data client-side if you want
it cached. Everything else runs uncached too — including `extract` aggregates on
`/nodes`, and the `/pdb/meta/v1/*` and `/metrics/*` endpoints. The cache is an
interface, and `/reports` gets its own (S3-backed) backend later without further
handler changes.
- **Key** — `<path>?<params>`, where the params are the ones that actually - **Key** — `<path>?<params>`, where the params are the ones that actually
determine the response, URL-encoded with keys sorted ascending and a repeated determine the response, URL-encoded with keys sorted ascending and a repeated
param's values sorted ascending. Param order in the request is therefore param's values sorted ascending. Param order in the request is therefore
irrelevant: one canonical key per distinct request. A request with no params irrelevant: one canonical key per distinct request. A request with no params
keys on the bare path. keys on the bare path.
- **TTL** — `facts_ttl`, default `30s`, **hard cap `30s`**. A larger configured - **TTL** — `facts_ttl` (the key predates the cache narrowing to `/nodes`),
default `30s`, **hard cap `30s`**. A larger configured
value is **clamped** down to the cap, not rejected, so a stray env var cannot value is **clamped** down to the cap, not rejected, so a stray env var cannot
crash-loop a container; `pdbmux config show` prints crash-loop a container; `pdbmux config show` prints
`facts_ttl : 30s (clamped from 600s, cap 30s)` when that happens. `facts_ttl: 0` `facts_ttl : 30s (clamped from 600s, cap 30s)` when that happens. `facts_ttl: 0`
@@ -446,7 +478,8 @@ backend later without further handler changes.
has passed `pdbmux` always re-queries the backends; the expired copy is served has passed `pdbmux` always re-queries the backends; the expired copy is served
**only** if every backend fails, which turns a `502` into slightly-old data. A **only** if every backend fails, which turns a `502` into slightly-old data. A
healthy backend is never shadowed by a stale entry, and a query every backend healthy backend is never shadowed by a stale entry, and a query every backend
refuses is answered with the refusal rather than the stale copy. *refuses* with a `4xx` is answered with the refusal rather than the stale copy —
a unanimous `5xx` is an outage like any other and still takes the stale copy.
- **Bounded** — `facts_cache_bytes` (default 64 MiB) is a byte budget, evicted - **Bounded** — `facts_cache_bytes` (default 64 MiB) is a byte budget, evicted
least-recently-used; reads count as use, so a stale entry that is still being least-recently-used; reads count as use, so a stale entry that is still being
asked for survives. A single response larger than the whole budget is not asked for survives. A single response larger than the whole budget is not
@@ -460,29 +493,20 @@ backend later without further handler changes.
client goes away leaves the flight running for the others. The flight is client goes away leaves the flight running for the others. The flight is
cancelled once its last participant leaves, so a lone client disconnecting cancelled once its last participant leaves, so a lone client disconnecting
releases the upstream connections straight away. releases the upstream connections straight away.
- **Response headers** — every response on a cached path carries `X-Cache` - **Invisible to clients** — a cached response is byte-for-byte a live one, with
(`hit` served from a fresh entry, `miss` built by this request, `stale` the the same headers. PuppetDB emits no `X-Cache` and no `Age`, so neither does
expired-entry fallback) and `Age` in whole seconds since the served copy was `pdbmux`; read the cache state off `/healthz` instead.
stored (`0` on a `miss`). Uncached paths carry neither.
- **Visibility** — `/healthz` carries a `cache` object: `backend` - **Visibility** — `/healthz` carries a `cache` object: `backend`
(`memory`/`none`), `ttl`, `entries`, `stale_entries`, `bytes`, `serving_stale`, (`memory`/`none`), `ttl`, `entries`, `stale_entries`, `bytes`, `serving_stale`,
`stale_served` and `last_stale_served`. `serving_stale` is `true` from the `stale_served` and `last_stale_served`. `serving_stale` is `true` from the
moment a stale fallback is served until the next response comes from a live moment a stale fallback is served until the next response comes from a live
fan-out or a fresh entry. fan-out or a fresh entry.
- **Provenance is stored, not re-applied** — what a cache entry holds is the - **Merged bodies are stored, not re-merged** — an entry holds the fully merged
fully merged body, `pdbmux_source` already injected and upstream records of `/nodes` body, `pdbmux_source` already stamped on, and the
that name already dropped. Attribution names the backend that supplied the `X-Records`/`X-Backends` it was built with, so a hit reproduces all three.
data, which is a property of that fetch, so it stays correct for as long as the Attribution is a property of that fetch, so it stays correct for as long as the
body does and ages out with it — `X-Cache` and `Age` say how old both are. Two body does and ages out with it. Two requests share an entry only when they
requests can only share an entry when they share a key, and the key is path share a key, and the key is path plus query.
plus query, which is exactly what decides whether injection applies; a
name-filtered `/facts` query and a plain one therefore cache separately and
neither is ever served the other's shape. The one path that keys on less than
it is asked is the `pdbmux_source` drilldown, whose `<value>` is dropped from
the key and applied to the shared entry instead. `source_fact` and
`source_fact_enabled` are read once at startup, and the cache lives for the
same process, so changing either cannot leave differently-shaped entries
behind.
## Config ## Config
@@ -508,7 +532,7 @@ backends: # order is a tie-break only, not a ranking
merge: freshness # freshness | static merge: freshness # freshness | static
timeout: 10s # per-upstream request timeout timeout: 10s # per-upstream request timeout
freshness_ttl: 30s # freshness-map cache TTL (freshness merge only) freshness_ttl: 30s # freshness-map cache TTL (freshness merge only)
facts_ttl: 30s # /facts + /nodes response cache TTL; 0 disables, capped at 30s facts_ttl: 30s # merged /nodes response cache TTL; 0 disables, capped at 30s
facts_cache_bytes: 67108864 # byte budget for that cache (64 MiB), LRU-evicted facts_cache_bytes: 67108864 # byte budget for that cache (64 MiB), LRU-evicted
source_fact: pdbmux_source # name of the synthetic provenance fact source_fact: pdbmux_source # name of the synthetic provenance fact
source_fact_enabled: true # false serves backends' records untouched source_fact_enabled: true # false serves backends' records untouched
+180 -209
View File
@@ -61,6 +61,7 @@ type countingBackend struct {
hits map[string]int hits map[string]int
bodies map[string]string bodies map[string]string
fail bool fail bool
dead bool
block chan struct{} block chan struct{}
} }
@@ -70,12 +71,15 @@ func newCountingBackend(t *testing.T, bodies map[string]string) *countingBackend
cb.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { cb.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
cb.mu.Lock() cb.mu.Lock()
cb.hits[r.URL.Path]++ cb.hits[r.URL.Path]++
fail, block, body := cb.fail, cb.block, cb.bodies[r.URL.Path] fail, dead, block, body := cb.fail, cb.dead, cb.block, cb.bodies[r.URL.Path]
cb.mu.Unlock() cb.mu.Unlock()
if block != nil { if block != nil {
<-block <-block
} }
if dead {
panic(http.ErrAbortHandler)
}
if fail { if fail {
http.Error(w, "boom", http.StatusInternalServerError) http.Error(w, "boom", http.StatusInternalServerError)
return return
@@ -108,6 +112,12 @@ func (cb *countingBackend) totalHits() int {
return n return n
} }
func (cb *countingBackend) setDead(v bool) {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.dead = v
}
func (cb *countingBackend) setFail(v bool) { func (cb *countingBackend) setFail(v bool) {
cb.mu.Lock() cb.mu.Lock()
defer cb.mu.Unlock() defer cb.mu.Unlock()
@@ -126,13 +136,13 @@ func (cb *countingBackend) setBlock(ch chan struct{}) {
cb.block = ch cb.block = ch
} }
// newCachedServer builds a server whose facts cache runs on a fake clock. // newCachedServer builds a server whose /nodes cache runs on a fake clock.
func newCachedServer(t *testing.T, cfg Config) (*Server, *fakeClock) { func newCachedServer(t *testing.T, cfg Config) (*Server, *fakeClock) {
t.Helper() t.Helper()
srv := newTestServer(cfg) srv := newTestServer(cfg)
mc, ok := srv.factsCache.(*memoryCache) mc, ok := srv.nodeCache.(*memoryCache)
if !ok { if !ok {
t.Fatalf("expected a memory cache, got %T", srv.factsCache) t.Fatalf("expected a memory cache, got %T", srv.nodeCache)
} }
clk := newFakeClock() clk := newFakeClock()
mc.now = clk.now mc.now = clk.now
@@ -267,14 +277,14 @@ func TestNoopCache_AlwaysMisses(t *testing.T) {
} }
} }
func TestHandler_FactsCacheFreshHit(t *testing.T) { func TestHandler_NodesCacheFreshHit(t *testing.T) {
body := `[` + fact("h1", "role", "web", "") + `]` body := `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`
a := newCountingBackend(t, map[string]string{factsPath: body}) a := newCountingBackend(t, map[string]string{nodesPath: body})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
first := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`) first := doGet(t, srv.Handler(), nodesPath, `["=","certname","h1"]`)
second := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`) second := doGet(t, srv.Handler(), nodesPath, `["=","certname","h1"]`)
if first.Code != http.StatusOK || second.Code != http.StatusOK { if first.Code != http.StatusOK || second.Code != http.StatusOK {
t.Fatalf("statuses %d/%d", first.Code, second.Code) t.Fatalf("statuses %d/%d", first.Code, second.Code)
@@ -282,40 +292,40 @@ func TestHandler_FactsCacheFreshHit(t *testing.T) {
if first.Body.String() != second.Body.String() { if first.Body.String() != second.Body.String() {
t.Errorf("cache hit changed the body:\n %s\n %s", first.Body.String(), second.Body.String()) t.Errorf("cache hit changed the body:\n %s\n %s", first.Body.String(), second.Body.String())
} }
if got := a.hitCount(factsPath); got != 1 { if got := a.hitCount(nodesPath); got != 1 {
t.Errorf("backend a saw %d requests, want 1 (second served from cache)", got) t.Errorf("backend a saw %d requests, want 1 (second served from cache)", got)
} }
if got := b.hitCount(factsPath); got != 1 { if got := b.hitCount(nodesPath); got != 1 {
t.Errorf("backend b saw %d requests, want 1", got) t.Errorf("backend b saw %d requests, want 1", got)
} }
// A different query is a different key and must go upstream. // A different query is a different key and must go upstream.
doGet(t, srv.Handler(), factsPath, `["=","name","osfamily"]`) doGet(t, srv.Handler(), nodesPath, `["=","certname","h2"]`)
if got := a.hitCount(factsPath); got != 2 { if got := a.hitCount(nodesPath); got != 2 {
t.Errorf("a different query should refetch: %d requests, want 2", got) t.Errorf("a different query should refetch: %d requests, want 2", got)
} }
} }
func TestHandler_FactsCacheExpires(t *testing.T) { func TestHandler_NodesCacheExpires(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
doGet(t, srv.Handler(), factsPath, "") doGet(t, srv.Handler(), nodesPath, "")
clk.advance(31 * time.Second) clk.advance(31 * time.Second)
doGet(t, srv.Handler(), factsPath, "") doGet(t, srv.Handler(), nodesPath, "")
if got := a.hitCount(factsPath); got != 2 { if got := a.hitCount(nodesPath); got != 2 {
t.Errorf("an expired entry should refetch: %d requests, want 2", got) t.Errorf("an expired entry should refetch: %d requests, want 2", got)
} }
} }
func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) { func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "old", "") + `]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("old", "2026-01-01T00:00:00.000Z") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
warm := doGet(t, srv.Handler(), factsPath, "") warm := doGet(t, srv.Handler(), nodesPath, "")
if warm.Code != http.StatusOK { if warm.Code != http.StatusOK {
t.Fatalf("warm-up status %d", warm.Code) t.Fatalf("warm-up status %d", warm.Code)
} }
@@ -325,8 +335,8 @@ func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
// Backends healthy but the entry expired: fresh data wins, never the stale copy. // Backends healthy but the entry expired: fresh data wins, never the stale copy.
clk.advance(31 * time.Second) clk.advance(31 * time.Second)
a.setBody(factsPath, `[`+fact("h1", "role", "new", "")+`]`) a.setBody(nodesPath, `[`+node("new", "2026-01-01T00:00:00.000Z")+`]`)
refetch := doGet(t, srv.Handler(), factsPath, "") refetch := doGet(t, srv.Handler(), nodesPath, "")
if !strings.Contains(refetch.Body.String(), `"new"`) { if !strings.Contains(refetch.Body.String(), `"new"`) {
t.Errorf("a healthy backend must not be shadowed by the stale entry: %s", refetch.Body.String()) t.Errorf("a healthy backend must not be shadowed by the stale entry: %s", refetch.Body.String())
} }
@@ -338,7 +348,7 @@ func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
clk.advance(31 * time.Second) clk.advance(31 * time.Second)
a.setFail(true) a.setFail(true)
b.setFail(true) b.setFail(true)
stale := doGet(t, srv.Handler(), factsPath, "") stale := doGet(t, srv.Handler(), nodesPath, "")
if stale.Code != http.StatusOK { if stale.Code != http.StatusOK {
t.Fatalf("stale fallback status %d: %s", stale.Code, stale.Body.String()) t.Fatalf("stale fallback status %d: %s", stale.Code, stale.Body.String())
} }
@@ -352,51 +362,14 @@ func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
} }
} }
// The stored entry is the whole estate's record set and the pinned value narrows
// it per request, so the fallback has to keep narrowing: a client asking for one
// backend's records must not be handed every backend's because the entry expired.
func TestHandler_StaleSourceFactDrilldownStaysFilteredByOwner(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "osfamily", "RedHat", "") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h2", "osfamily", "Debian", "") + `]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
drilldown := sourceFactURL + "/a"
want := map[string]string{"h1": "a"}
warm := doGet(t, srv.Handler(), drilldown, "")
if warm.Code != http.StatusOK {
t.Fatalf("warm-up status %d: %s", warm.Code, warm.Body.String())
}
if got, n := sourceValues(t, warm.Body.Bytes(), defaultSourceFact); n != len(want) || !reflect.DeepEqual(got, want) {
t.Fatalf("warm-up %s = %v (%d records), want %v", drilldown, got, n, want)
}
// Every backend down and the entry expired: the stale copy is served.
clk.advance(31 * time.Second)
a.setFail(true)
b.setFail(true)
rec := doGet(t, srv.Handler(), drilldown, "")
if rec.Code != http.StatusOK {
t.Fatalf("stale fallback status %d: %s", rec.Code, rec.Body.String())
}
if got := rec.Header().Get(cacheStatusHeader); got != "stale" {
t.Fatalf("%s = %q, want stale: the request did not take the fallback path", cacheStatusHeader, got)
}
got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact)
if n != len(want) || !reflect.DeepEqual(got, want) {
t.Errorf("stale %s = %v (%d records), want only backend a's %v", drilldown, got, n, want)
}
}
func TestHandler_NoCacheEntryMeansBackendFailureIs502(t *testing.T) { func TestHandler_NoCacheEntryMeansBackendFailureIs502(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
a.setFail(true) a.setDead(true)
b.setFail(true) b.setDead(true)
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
rec := doGet(t, srv.Handler(), factsPath, "") rec := doGet(t, srv.Handler(), nodesPath, "")
if rec.Code != http.StatusBadGateway { if rec.Code != http.StatusBadGateway {
t.Fatalf("status %d, want 502", rec.Code) t.Fatalf("status %d, want 502", rec.Code)
} }
@@ -406,8 +379,8 @@ func TestHandler_NoCacheEntryMeansBackendFailureIs502(t *testing.T) {
} }
func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) { func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler() h := srv.Handler()
@@ -422,13 +395,13 @@ func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
wg.Add(1) wg.Add(1)
go func(i int) { go func(i int) {
defer wg.Done() defer wg.Done()
codes[i] = doGet(t, h, factsPath, "").Code codes[i] = doGet(t, h, nodesPath, "").Code
}(i) }(i)
} }
// The leader is parked inside the blocked backend, so every caller that // The leader is parked inside the blocked backend, so every caller that
// reaches the handler before the release joins its flight. // reaches the handler before the release joins its flight.
waitFor(t, func() bool { return a.hitCount(factsPath) >= 1 }) waitFor(t, func() bool { return a.hitCount(nodesPath) >= 1 })
time.Sleep(250 * time.Millisecond) time.Sleep(250 * time.Millisecond)
close(release) close(release)
wg.Wait() wg.Wait()
@@ -438,10 +411,10 @@ func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
t.Fatalf("caller %d got %d", i, code) t.Fatalf("caller %d got %d", i, code)
} }
} }
if got := a.hitCount(factsPath); got != 1 { if got := a.hitCount(nodesPath); got != 1 {
t.Errorf("backend a saw %d requests, want 1 for %d concurrent callers", got, callers) t.Errorf("backend a saw %d requests, want 1 for %d concurrent callers", got, callers)
} }
if got := b.hitCount(factsPath); got != 1 { if got := b.hitCount(nodesPath); got != 1 {
t.Errorf("backend b saw %d requests, want 1 for %d concurrent callers", got, callers) t.Errorf("backend b saw %d requests, want 1 for %d concurrent callers", got, callers)
} }
} }
@@ -557,15 +530,15 @@ func TestFlightGroup_LeaderPanicFailsLeaderAndWaiters(t *testing.T) {
} }
func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) { func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
panicBuild := func(context.Context) (cachedResponse, error) { panic("build exploded") } panicBuild := func(context.Context) (cachedResponse, error) { panic("build exploded") }
serve := func(build func(context.Context) (cachedResponse, error)) *httptest.ResponseRecorder { serve := func(build func(context.Context) (cachedResponse, error)) *httptest.ResponseRecorder {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, factsPath, nil) req := httptest.NewRequest(http.MethodGet, nodesPath, nil)
srv.serveCached(rec, req, factsPath, nil, build) srv.serveCached(rec, req, nodesPath, nil, build)
return rec return rec
} }
@@ -575,7 +548,7 @@ func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
t.Fatalf("status %d (%s), want 502", rec.Code, rec.Body.String()) t.Fatalf("status %d (%s), want 502", rec.Code, rec.Body.String())
} }
stored := `[` + fact("h1", "role", "web", "") + `]` stored := `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`
rec = serve(func(context.Context) (cachedResponse, error) { rec = serve(func(context.Context) (cachedResponse, error) {
return cachedResponse{Body: json.RawMessage(stored), Records: 1}, nil return cachedResponse{Body: json.RawMessage(stored), Records: 1}, nil
}) })
@@ -600,16 +573,16 @@ func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
} }
func TestServeCached_BuildPanicFailsConcurrentWaiters(t *testing.T) { func TestServeCached_BuildPanicFailsConcurrentWaiters(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
entered := make(chan struct{}) entered := make(chan struct{})
release := make(chan struct{}) release := make(chan struct{})
serve := func(build func(context.Context) (cachedResponse, error)) *httptest.ResponseRecorder { serve := func(build func(context.Context) (cachedResponse, error)) *httptest.ResponseRecorder {
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, factsPath, nil) req := httptest.NewRequest(http.MethodGet, nodesPath, nil)
srv.serveCached(rec, req, factsPath, nil, build) srv.serveCached(rec, req, nodesPath, nil, build)
return rec return rec
} }
@@ -659,7 +632,7 @@ func TestHandler_LeaderDisconnectDoesNotFailFollowers(t *testing.T) {
body := `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]` body := `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`
a := newCountingBackend(t, map[string]string{nodesPath: body}) a := newCountingBackend(t, map[string]string{nodesPath: body})
b := newCountingBackend(t, map[string]string{nodesPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(withHost(a.srv.URL, hostA), withHost(b.srv.URL, hostB)))
h := srv.Handler() h := srv.Handler()
release := make(chan struct{}) release := make(chan struct{})
@@ -700,7 +673,7 @@ func TestHandler_LeaderDisconnectDoesNotFailFollowers(t *testing.T) {
} }
// The leader built this body, so its provenance names the backend that // The leader built this body, so its provenance names the backend that
// answered the leader's fan-out. // answered the leader's fan-out.
want := `[` + stamped(t, node("h1", "2026-01-01T00:00:00.000Z"), defaultSourceFact, "a") + `]` want := `[` + stamped(t, node("h1", "2026-01-01T00:00:00.000Z"), defaultSourceFact, hostA) + `]`
if got := strings.TrimSpace(follower.Body.String()); !sameJSON(t, got, want) { if got := strings.TrimSpace(follower.Body.String()); !sameJSON(t, got, want) {
t.Errorf("follower body = %s, want %s", got, want) t.Errorf("follower body = %s, want %s", got, want)
} }
@@ -710,8 +683,8 @@ func TestHandler_LeaderDisconnectDoesNotFailFollowers(t *testing.T) {
// The flight the leader started still populated the cache. // The flight the leader started still populated the cache.
warmed := doGet(t, h, nodesPath, "") warmed := doGet(t, h, nodesPath, "")
if got := warmed.Header().Get(cacheStatusHeader); got != "hit" { if got := strings.TrimSpace(warmed.Body.String()); !sameJSON(t, got, want) {
t.Errorf("%s = %q, want hit: the abandoned leader's flight must still warm the cache", cacheStatusHeader, got) t.Errorf("warmed body = %s, want %s: the abandoned leader's flight must still warm the cache", got, want)
} }
if got := a.hitCount(nodesPath); got != 1 { if got := a.hitCount(nodesPath); got != 1 {
t.Errorf("backend a saw %d requests after the cached read, want 1", got) t.Errorf("backend a saw %d requests after the cached read, want 1", got)
@@ -750,7 +723,7 @@ func TestHandler_SoloDisconnectAbortsFanOutPromptly(t *testing.T) {
}() }()
start := time.Now() start := time.Now()
req := httptest.NewRequest(http.MethodGet, factsPath, nil).WithContext(ctx) req := httptest.NewRequest(http.MethodGet, nodesPath, nil).WithContext(ctx)
srv.Handler().ServeHTTP(httptest.NewRecorder(), req) srv.Handler().ServeHTTP(httptest.NewRecorder(), req)
if elapsed := time.Since(start); elapsed >= cfg.Timeout/2 { if elapsed := time.Since(start); elapsed >= cfg.Timeout/2 {
t.Errorf("handler returned after %s, want well under cfg.Timeout %s", elapsed, cfg.Timeout) t.Errorf("handler returned after %s, want well under cfg.Timeout %s", elapsed, cfg.Timeout)
@@ -783,7 +756,7 @@ func TestHandler_DisconnectedRequestsDoNotPinBackends(t *testing.T) {
time.Sleep(disconnectAfter) time.Sleep(disconnectAfter)
cancel() cancel()
}() }()
target := factsPath + "?query=" + url.QueryEscape(strconv.Itoa(i)) target := nodesPath + "?query=" + url.QueryEscape(strconv.Itoa(i))
req := httptest.NewRequest(http.MethodGet, target, nil).WithContext(ctx) req := httptest.NewRequest(http.MethodGet, target, nil).WithContext(ctx)
h.ServeHTTP(httptest.NewRecorder(), req) h.ServeHTTP(httptest.NewRecorder(), req)
}(i) }(i)
@@ -1116,47 +1089,39 @@ func assertGoroutinesSettle(t *testing.T, baseline int, frames ...string) {
} }
} }
func TestServeCached_CacheStatusHeaders(t *testing.T) { // PuppetDB emits no X-Cache and no Age, so a client cannot tell a cached
stored := `[` + fact("h1", "role", "web", "") + `]` // response from a live one. Covers the fresh hit, the miss and the stale
a := newCountingBackend(t, map[string]string{factsPath: stored}) // fallback, which are the three ways a response can leave the cached path.
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) func TestHandler_CachedResponsesCarryNoCacheHeaders(t *testing.T) {
a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler() h := srv.Handler()
rec := doGet(t, h, factsPath, "") assertNoCacheHeaders(t, doGet(t, h, nodesPath, ""), "miss")
if got := rec.Header().Get(cacheStatusHeader); got != "miss" {
t.Errorf("first request %s = %q, want miss", cacheStatusHeader, got)
}
if got := rec.Header().Get(ageHeader); got != "0" {
t.Errorf("first request %s = %q, want 0", ageHeader, got)
}
clk.advance(7 * time.Second) clk.advance(7 * time.Second)
rec = doGet(t, h, factsPath, "") assertNoCacheHeaders(t, doGet(t, h, nodesPath, ""), "fresh hit")
if got := rec.Header().Get(cacheStatusHeader); got != "hit" {
t.Errorf("cached request %s = %q, want hit", cacheStatusHeader, got)
}
if got := rec.Header().Get(ageHeader); got != "7" {
t.Errorf("cached request %s = %q, want 7", ageHeader, got)
}
// Past the TTL with every backend down, the stale fallback must say so.
clk.advance(24 * time.Second) clk.advance(24 * time.Second)
a.setFail(true) a.setFail(true)
b.setFail(true) b.setFail(true)
rec = doGet(t, h, factsPath, "") rec := doGet(t, h, nodesPath, "")
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Fatalf("stale fallback status %d (%s)", rec.Code, rec.Body.String()) t.Fatalf("stale fallback status %d (%s)", rec.Code, rec.Body.String())
} }
if got := rec.Header().Get(cacheStatusHeader); got != "stale" { if serving, _, _ := srv.stale.snapshot(); !serving {
t.Errorf("stale fallback %s = %q, want stale", cacheStatusHeader, got) t.Fatal("the third request did not take the stale fallback path")
} }
if got := rec.Header().Get(ageHeader); got != "31" { assertNoCacheHeaders(t, rec, "stale fallback")
t.Errorf("stale fallback %s = %q, want 31 seconds since the entry was stored", ageHeader, got) }
func assertNoCacheHeaders(t *testing.T, rec *httptest.ResponseRecorder, what string) {
t.Helper()
for _, h := range []string{"X-Cache", "Age"} {
if got := rec.Header().Get(h); got != "" {
t.Errorf("%s response carries %s: %q, want no such header", what, h, got)
} }
want := `[` + fact("h1", "role", "web", "") + `,` + factEnv("h1", defaultSourceFact, "a", "") + `]`
if got := strings.TrimSpace(rec.Body.String()); !sameJSON(t, got, want) {
t.Errorf("stale body = %s, want %s", got, want)
} }
} }
@@ -1193,18 +1158,18 @@ func (c *recordingCache) Stats() CacheStats { return CacheStats{Backend: "record
func TestServeCached_PutRunsOnDetachedContext(t *testing.T) { func TestServeCached_PutRunsOnDetachedContext(t *testing.T) {
srv := newTestServer(cacheTestConfig("http://backend.invalid", "http://backend.invalid")) srv := newTestServer(cacheTestConfig("http://backend.invalid", "http://backend.invalid"))
cache := newRecordingCache() cache := newRecordingCache()
srv.factsCache = cache srv.nodeCache = cache
entered := make(chan struct{}) entered := make(chan struct{})
release := make(chan struct{}) release := make(chan struct{})
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
defer cancel() defer cancel()
req := httptest.NewRequest(http.MethodGet, factsPath, nil).WithContext(ctx) req := httptest.NewRequest(http.MethodGet, nodesPath, nil).WithContext(ctx)
served := make(chan struct{}) served := make(chan struct{})
go func() { go func() {
defer close(served) defer close(served)
srv.serveCached(httptest.NewRecorder(), req, factsPath, nil, func(context.Context) (cachedResponse, error) { srv.serveCached(httptest.NewRecorder(), req, nodesPath, nil, func(context.Context) (cachedResponse, error) {
close(entered) close(entered)
<-release <-release
return cachedResponse{Body: json.RawMessage(`[]`), Records: -1}, nil return cachedResponse{Body: json.RawMessage(`[]`), Records: -1}, nil
@@ -1236,8 +1201,8 @@ func TestServeCached_PutRunsOnDetachedContext(t *testing.T) {
} }
func TestHandler_HealthzReportsCacheState(t *testing.T) { func TestHandler_HealthzReportsCacheState(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
hr := health(t, srv) hr := health(t, srv)
@@ -1248,7 +1213,7 @@ func TestHandler_HealthzReportsCacheState(t *testing.T) {
t.Fatalf("a cold cache should be empty and not stale: %+v", hr.Cache) t.Fatalf("a cold cache should be empty and not stale: %+v", hr.Cache)
} }
doGet(t, srv.Handler(), factsPath, "") doGet(t, srv.Handler(), nodesPath, "")
hr = health(t, srv) hr = health(t, srv)
if hr.Cache.Entries != 1 || hr.Cache.StaleEntries != 0 || hr.Cache.Bytes == 0 { if hr.Cache.Entries != 1 || hr.Cache.StaleEntries != 0 || hr.Cache.Bytes == 0 {
t.Fatalf("after one request: %+v", hr.Cache) t.Fatalf("after one request: %+v", hr.Cache)
@@ -1264,7 +1229,7 @@ func TestHandler_HealthzReportsCacheState(t *testing.T) {
a.setFail(true) a.setFail(true)
b.setFail(true) b.setFail(true)
doGet(t, srv.Handler(), factsPath, "") doGet(t, srv.Handler(), nodesPath, "")
hr = health(t, srv) hr = health(t, srv)
if !hr.Cache.ServingStale || hr.Cache.StaleServed != 1 || hr.Cache.LastStale == "" { if !hr.Cache.ServingStale || hr.Cache.StaleServed != 1 || hr.Cache.LastStale == "" {
t.Errorf("staleness not surfaced in /healthz: %+v", hr.Cache) t.Errorf("staleness not surfaced in /healthz: %+v", hr.Cache)
@@ -1444,9 +1409,9 @@ func TestNewServer_ClampsFactsTTL(t *testing.T) {
cfg.CacheBytes = 1 << 20 cfg.CacheBytes = 1 << 20
srv := newTestServer(cfg) srv := newTestServer(cfg)
mc, ok := srv.factsCache.(*memoryCache) mc, ok := srv.nodeCache.(*memoryCache)
if !ok { if !ok {
t.Fatalf("expected a memory cache, got %T", srv.factsCache) t.Fatalf("expected a memory cache, got %T", srv.nodeCache)
} }
if mc.ttl != maxFactsTTL { if mc.ttl != maxFactsTTL {
t.Errorf("cache ttl = %s, want %s", mc.ttl, maxFactsTTL) t.Errorf("cache ttl = %s, want %s", mc.ttl, maxFactsTTL)
@@ -1520,51 +1485,19 @@ func stamped(t *testing.T, raw, field, value string) string {
// The injector is per-request but a cache entry is shared, so a second caller is // The injector is per-request but a cache entry is shared, so a second caller is
// served a body built for the first. Provenance names the backend that supplied // served a body built for the first. Provenance names the backend that supplied
// the data, which is a property of that fetch, so the shared body stays correct. // the data, which is a property of that fetch, so the shared body stays correct.
func TestHandler_CachedFactsKeepSourceAttribution(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler()
first := doGet(t, h, factsPath, "")
if got := first.Header().Get(cacheStatusHeader); got != "miss" {
t.Fatalf("first %s = %q, want miss", cacheStatusHeader, got)
}
srcs, n := sourceValues(t, first.Body.Bytes(), defaultSourceFact)
if n != 1 || srcs["h1"] != "a" {
t.Fatalf("first request sources = %v (%d records), want h1 -> a", srcs, n)
}
second := doGet(t, h, factsPath, "")
if got := second.Header().Get(cacheStatusHeader); got != "hit" {
t.Fatalf("second %s = %q, want hit", cacheStatusHeader, got)
}
srcs, n = sourceValues(t, second.Body.Bytes(), defaultSourceFact)
if n != 1 || srcs["h1"] != "a" {
t.Errorf("cached request sources = %v (%d records), want exactly one h1 -> a", srcs, n)
}
if got := a.hitCount(factsPath); got != 1 {
t.Errorf("backend a saw %d requests, want 1: the second read must come from the cache", got)
}
}
// The same, for the /nodes stamp rather than the synthetic /facts record.
func TestHandler_CachedNodesKeepSourceStamp(t *testing.T) { func TestHandler_CachedNodesKeepSourceStamp(t *testing.T) {
a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-02T00:00:00.000Z") + `]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-02T00:00:00.000Z") + `]`})
b := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(withHost(a.srv.URL, hostA), withHost(b.srv.URL, hostB)))
h := srv.Handler() h := srv.Handler()
first := doGet(t, h, nodesPath, "") first := doGet(t, h, nodesPath, "")
if got := nodeSources(t, first.Body.Bytes(), defaultSourceFact); got["h1"] != "a" { if got := nodeSources(t, first.Body.Bytes(), defaultSourceFact); got["h1"] != hostA {
t.Fatalf("first request stamp = %v, want h1 -> a", got) t.Fatalf("first request stamp = %v, want h1 -> a", got)
} }
second := doGet(t, h, nodesPath, "") second := doGet(t, h, nodesPath, "")
if got := second.Header().Get(cacheStatusHeader); got != "hit" { want := `[` + stamped(t, node("h1", "2026-01-02T00:00:00.000Z"), defaultSourceFact, hostA) + `]`
t.Fatalf("second %s = %q, want hit", cacheStatusHeader, got)
}
want := `[` + stamped(t, node("h1", "2026-01-02T00:00:00.000Z"), defaultSourceFact, "a") + `]`
if got := strings.TrimSpace(second.Body.String()); !sameJSON(t, got, want) { if got := strings.TrimSpace(second.Body.String()); !sameJSON(t, got, want) {
t.Errorf("cached body = %s, want %s", got, want) t.Errorf("cached body = %s, want %s", got, want)
} }
@@ -1577,77 +1510,115 @@ func TestHandler_CachedNodesKeepSourceStamp(t *testing.T) {
// while the entry is served the attribution is the one that fetch had, and the // while the entry is served the attribution is the one that fetch had, and the
// rebuild after the TTL picks up the move. // rebuild after the TTL picks up the move.
func TestHandler_CachedSourceAgesWithItsData(t *testing.T) { func TestHandler_CachedSourceAgesWithItsData(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`}) a := newCountingBackend(t, map[string]string{nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(withHost(a.srv.URL, hostA), withHost(b.srv.URL, hostB)))
h := srv.Handler() h := srv.Handler()
doGet(t, h, factsPath, "") doGet(t, h, nodesPath, "")
// The node moves to b while the entry is still fresh. // The node moves to b while the entry is still fresh.
a.setBody(factsPath, `[]`) a.setBody(nodesPath, `[]`)
b.setBody(factsPath, `[`+fact("h1", "role", "web", "")+`]`) b.setBody(nodesPath, `[`+node("h1", "2026-01-01T00:00:00.000Z")+`]`)
cached := doGet(t, h, factsPath, "") cached := doGet(t, h, nodesPath, "")
if got := cached.Header().Get(cacheStatusHeader); got != "hit" { if got := nodeSources(t, cached.Body.Bytes(), defaultSourceFact); got["h1"] != hostA {
t.Fatalf("%s = %q, want hit", cacheStatusHeader, got) t.Errorf("cached stamp = %v, want h1 -> a: the body and its attribution come from the same fetch", got)
} }
if srcs, _ := sourceValues(t, cached.Body.Bytes(), defaultSourceFact); srcs["h1"] != "a" { if got := a.hitCount(nodesPath); got != 1 {
t.Errorf("cached sources = %v, want h1 -> a: the body and its attribution come from the same fetch", srcs) t.Errorf("backend a saw %d requests, want 1 while the entry is fresh", got)
} }
clk.advance(31 * time.Second) clk.advance(31 * time.Second)
rebuilt := doGet(t, h, factsPath, "") rebuilt := doGet(t, h, nodesPath, "")
if srcs, _ := sourceValues(t, rebuilt.Body.Bytes(), defaultSourceFact); srcs["h1"] != "b" { if got := nodeSources(t, rebuilt.Body.Bytes(), defaultSourceFact); got["h1"] != hostB {
t.Errorf("rebuilt sources = %v, want h1 -> b once the entry expired", srcs) t.Errorf("rebuilt stamp = %v, want h1 -> b once the entry expired", got)
} }
} }
// The injection gate is a pure function of path and query, both of which are in // Fact answers must be as current as a backend's own, so every fact-serving
// the cache key, so a gated request can never be served an injected body cached // route refetches: the second request sees data that changed between the two.
// for an ungated one. func TestHandler_FactPathsAreNeverCached(t *testing.T) {
func TestHandler_SourceGateIsPerCacheKey(t *testing.T) { for _, path := range []string{
factsPath,
factsPath + "/role",
factsPath + "/role/web",
factNamesPath,
factsetsPath,
nodesPath + "/h1/facts",
"/pdb/query/v4/fact-contents",
"/pdb/query/v4/fact-paths",
"/pdb/query/v4/inventory",
} {
t.Run(path, func(t *testing.T) {
before := `[{"certname":"h1","name":"role","value":"before"}]`
after := `[{"certname":"h1","name":"role","value":"after"}]`
a := newCountingBackend(t, map[string]string{path: before})
b := newCountingBackend(t, map[string]string{path: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler()
if first := doGet(t, h, path, ""); !strings.Contains(first.Body.String(), "before") {
t.Fatalf("first %s = %s, want the backend's own records", path, first.Body.String())
}
a.setBody(path, after)
second := doGet(t, h, path, "")
if strings.Contains(second.Body.String(), "before") {
t.Errorf("second %s = %s, want the changed data rather than a cached copy", path, second.Body.String())
}
if !strings.Contains(second.Body.String(), "after") {
t.Errorf("second %s = %s, want the changed data", path, second.Body.String())
}
if got := a.hitCount(path); got != 2 {
t.Errorf("backend a saw %d requests for %s, want one per request", got, path)
}
})
}
}
// No fact response may carry the cache headers PuppetDB never sends, whatever
// else pdbmux is doing with the merged body.
func TestHandler_FactResponsesCarryNoCacheHeaders(t *testing.T) {
body := `[` + fact("h1", "role", "web", "") + `]` body := `[` + fact("h1", "role", "web", "") + `]`
a := newCountingBackend(t, map[string]string{factsPath: body}) for _, path := range []string{factsPath, factsPath + "/role", factNamesPath, sourceFactURL} {
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) t.Run(path, func(t *testing.T) {
a := newCountingBackend(t, map[string]string{path: body, factsPath: body})
b := newCountingBackend(t, map[string]string{path: `[]`, factsPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler() h := srv.Handler()
const nameFiltered = `["=","name","role"]` assertNoCacheHeaders(t, doGet(t, h, path, ""), path)
for _, pass := range []string{"first", "cached"} { assertNoCacheHeaders(t, doGet(t, h, path, ""), path+" repeated")
open := doGet(t, h, factsPath, "") })
if srcs, n := sourceValues(t, open.Body.Bytes(), defaultSourceFact); n != 1 || srcs["h1"] != "a" {
t.Errorf("%s unfiltered request sources = %v (%d records), want h1 -> a", pass, srcs, n)
}
gated := doGet(t, h, factsPath, nameFiltered)
if _, n := sourceValues(t, gated.Body.Bytes(), defaultSourceFact); n != 0 {
t.Errorf("%s name-filtered request carried %d synthetic record(s), want none", pass, n)
}
if got := strings.TrimSpace(gated.Body.String()); !sameJSON(t, got, body) {
t.Errorf("%s name-filtered body = %s, want %s", pass, got, body)
}
}
if got := a.hitCount(factsPath); got != 2 {
t.Errorf("backend a saw %d requests, want 2: one per distinct cache key", got)
} }
} }
// Suppression of an upstream fact of the configured name is part of the merged // openvoxdb answers a successful query with application/json;charset=utf-8
// body, so it survives into the cache rather than being re-applied per request. // (src/puppetlabs/puppetdb/query_eng.clj:555 through http.clj:80), so a merged
func TestHandler_SuppressionSurvivesCacheHit(t *testing.T) { // response has to as well or a client can tell the proxy from the real thing.
upstream := `[` + fact("h1", defaultSourceFact, "somewhere-else", "") + `,` + fact("h1", "role", "web", "") + `]` func TestHandler_SuccessContentTypeMatchesPuppetDB(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: upstream}) bodies := map[string]string{
b := newCountingBackend(t, map[string]string{factsPath: `[]`}) nodesPath: `[` + node("h1", "2026-01-01T00:00:00.000Z") + `]`,
factsPath: `[` + fact("h1", "role", "web", "") + `]`,
factNamesPath: `["role"]`,
reportsPath: `[` + report("h1", "abc", "2026-01-01T00:00:00.000Z") + `]`,
}
a := newCountingBackend(t, bodies)
b := newCountingBackend(t, map[string]string{nodesPath: `[]`, factsPath: `[]`, factNamesPath: `[]`, reportsPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler() h := srv.Handler()
doGet(t, h, factsPath, "") for path := range bodies {
cached := doGet(t, h, factsPath, "") // Twice, so a cache hit is held to the same content type as a live build.
if got := cached.Header().Get(cacheStatusHeader); got != "hit" { for _, pass := range []string{"live", "repeated"} {
t.Fatalf("%s = %q, want hit", cacheStatusHeader, got) rec := doGet(t, h, path, "")
if rec.Code != http.StatusOK {
t.Fatalf("%s %s status %d (%s)", pass, path, rec.Code, rec.Body.String())
}
if got := rec.Header().Get("Content-Type"); got != "application/json;charset=utf-8" {
t.Errorf("%s %s Content-Type = %q, want application/json;charset=utf-8", pass, path, got)
}
} }
srcs, n := sourceValues(t, cached.Body.Bytes(), defaultSourceFact)
if n != 1 || srcs["h1"] != "a" {
t.Errorf("cached sources = %v (%d records), want exactly one h1 -> a, the upstream value dropped", srcs, n)
} }
} }
+85
View File
@@ -0,0 +1,85 @@
package main
import (
"context"
"net/http"
"strings"
)
const (
factsetsPath = "/pdb/query/v4/factsets"
catalogsPath = "/pdb/query/v4/catalogs"
certnameRouteName = "/pdb/query/v4/{nodes,factsets,catalogs}/<certname>"
)
// certnamePrefixes are the endpoints whose next path segment is a certname.
// openvoxdb resolves that segment to a parent row before it serves anything
// under it — status-not-found-response for the singleton
// (src/puppetlabs/puppetdb/http/handlers.clj:98-120,252-253,342-343,372 and
// http.clj:238-242) and parent-check for every child path
// (middleware.clj:381-398, mounted at handlers.clj:345-347,255-262,373-380) —
// so each of these paths describes exactly one node's data and 404s when the
// backend holds none of it.
var certnamePrefixes = []string{nodesPath + "/", factsetsPath + "/", catalogsPath + "/"}
// certnameFor returns the certname a path is keyed on, empty when the path is
// not one of these routes.
func certnameFor(path string) string {
for _, prefix := range certnamePrefixes {
rest, ok := strings.CutPrefix(path, prefix)
if !ok {
continue
}
certname, _, _ := strings.Cut(rest, "/")
return certname
}
return ""
}
func isCertnameRoute(path string) bool { return certnameFor(path) != "" }
// serveByOwner answers a path keyed on one certname from the backend that owns
// that certname, so a node resolves here exactly as it resolves in the merged
// collections. Without it the answer is whichever backend is configured first,
// and a node both backends hold reports one backend's facts through /facts and
// the other's through /nodes/<certname>/facts.
//
// The owner's reply is passed through whole rather than merged: openvoxdb
// answers these paths with a bare object or a parent-check 404, neither of
// which is the record array the merges are built on. The remaining backends are
// tried after it, so a node only one backend holds is still served and a
// certname no backend holds still answers with openvoxdb's own 404 body.
//
// askOwnerFirst is what makes the owner's reply the answer even when it is an
// error. A backend that does not hold the certname 404s to say so, so it has no
// opinion to weigh against the owner's.
func (s *Server) serveByOwner(w http.ResponseWriter, r *http.Request) {
s.proxyOrdered(w, r, s.ownerFirst(r.Context(), certnameFor(r.URL.Path)), askOwnerFirst)
}
// ownerFirst puts the backend holding a certname's newest report ahead of the
// rest, from the same freshness map the /facts merge attributes records with.
// Static merge has no per-certname owner, and neither has a certname absent
// from the map, so both keep configured order — the tie-break the merges use.
func (s *Server) ownerFirst(ctx context.Context, certname string) []Backend {
if s.cfg.Merge == mergeStatic || certname == "" {
return s.cfg.Backends
}
owner := s.freshnessMap(ctx, nil)[certname]
if owner == "" {
return s.cfg.Backends
}
ordered := make([]Backend, 0, len(s.cfg.Backends))
for _, b := range s.cfg.Backends {
if b.Name == owner {
ordered = append(ordered, b)
}
}
for _, b := range s.cfg.Backends {
if b.Name != owner {
ordered = append(ordered, b)
}
}
return ordered
}
+435
View File
@@ -0,0 +1,435 @@
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
)
// The bodies below are the shapes openvoxdb 8.9.0 actually serves, captured
// from a live backend: a bare object for /factsets/<certname>
// (src/puppetlabs/puppetdb/http/handlers.clj:98-108,342-343), a fact array for
// /nodes/<certname>/facts (handlers.clj:373-376) and a pretty-printed
// {"error": ...} body for a certname the backend has no row for
// (http.clj:238-242 for the singleton, middleware.clj:381-398 for the child
// path, which names the parent — "node" — rather than the child).
func factsetBody(certname, timestamp, role string) string {
return fmt.Sprintf(`{
"timestamp" : %q,
"facts" : {
"data" : [ {
"name" : "role",
"value" : %q
} ]
},
"certname" : %q,
"hash" : "d775f92fb438b96276e721bb09524e6361156b06",
"producer_timestamp" : %q,
"producer" : "compiler.example.net",
"environment" : "develop"
}`, timestamp, role, certname, timestamp)
}
func nodeFactsBody(certname, role string) string {
return fmt.Sprintf(`[{"certname":%q,"environment":"develop","name":"role","value":%q}]`, certname, role)
}
func notFoundBody(kind, id string) string {
return fmt.Sprintf("{\n \"error\" : \"No information is known about %s %s\"\n}", kind, id)
}
// voxBackend answers the per-certname routes as openvoxdb does: the certnames
// it holds get a bare object or a fact array, and every other certname gets the
// parent-check 404. Each backend labels its facts with its own name, so a
// response says which backend answered it.
type voxBackend struct {
srv *httptest.Server
name string
// nodes maps a certname it holds to that node's report_timestamp.
nodes map[string]string
fail bool
// perNodeStatus and perNodeBody answer the per-certname routes with a fixed
// reply, leaving /nodes alone so the freshness map still resolves an owner.
perNodeStatus int
perNodeBody string
// perNodeDead aborts the per-certname routes without a status, standing in
// for a backend that is unreachable only for that request.
perNodeDead bool
mu sync.Mutex
gotParams map[string]url.Values
}
func newVoxBackend(t *testing.T, name string, nodes map[string]string) *voxBackend {
t.Helper()
vb := &voxBackend{name: name, nodes: nodes, gotParams: map[string]url.Values{}}
vb.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
vb.mu.Lock()
vb.gotParams[r.URL.Path] = r.URL.Query()
vb.mu.Unlock()
if vb.fail {
http.Error(w, "boom", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json;charset=utf-8")
if isCertnameRoute(r.URL.Path) {
if vb.perNodeDead {
panic(http.ErrAbortHandler)
}
if vb.perNodeStatus != 0 {
w.WriteHeader(vb.perNodeStatus)
_, _ = io.WriteString(w, vb.perNodeBody)
return
}
}
if certname, ok := strings.CutPrefix(r.URL.Path, factsetsPath+"/"); ok {
ts, held := vb.nodes[certname]
if !held {
vb.notFound(w, "factset", certname)
return
}
_, _ = io.WriteString(w, factsetBody(certname, ts, vb.name))
return
}
if rest, ok := strings.CutPrefix(r.URL.Path, nodesPath+"/"); ok {
certname, _, _ := strings.Cut(rest, "/")
if _, held := vb.nodes[certname]; !held {
vb.notFound(w, "node", certname)
return
}
_, _ = io.WriteString(w, nodeFactsBody(certname, vb.name))
return
}
switch r.URL.Path {
case nodesPath:
_, _ = io.WriteString(w, "["+strings.Join(vb.nodeRecords(), ",")+"]")
case factsPath:
_, _ = io.WriteString(w, "["+strings.Join(vb.factRecords(), ",")+"]")
default:
_, _ = io.WriteString(w, "[]")
}
}))
t.Cleanup(vb.srv.Close)
return vb
}
func (vb *voxBackend) notFound(w http.ResponseWriter, kind, id string) {
w.WriteHeader(http.StatusNotFound)
_, _ = io.WriteString(w, notFoundBody(kind, id))
}
func (vb *voxBackend) nodeRecords() []string {
out := make([]string, 0, len(vb.nodes))
for certname, ts := range vb.nodes {
out = append(out, node(certname, ts))
}
return out
}
func (vb *voxBackend) factRecords() []string {
out := make([]string, 0, len(vb.nodes))
for certname := range vb.nodes {
out = append(out, fact(certname, "role", vb.name, ""))
}
return out
}
func (vb *voxBackend) params(path string) url.Values {
vb.mu.Lock()
defer vb.mu.Unlock()
return vb.gotParams[path]
}
// perNodePaths are the routes a certname resolves on, with the body field that
// names the backend that answered.
var perNodePaths = []struct {
name string
path func(certname string) string
}{
{"factsets/<certname>", func(cn string) string { return factsetsPath + "/" + cn }},
{"nodes/<certname>/facts", func(cn string) string { return nodesPath + "/" + cn + "/facts" }},
}
func TestCertnameRoutes_ServeTheBackendHoldingTheNode(t *testing.T) {
for _, tc := range []struct {
name string
aNodes map[string]string
bNodes map[string]string
certname string
want string
}{
{
name: "held only by a",
aNodes: map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"},
bNodes: map[string]string{},
certname: "h1.example.net",
want: "a",
},
{
name: "held only by b",
aNodes: map[string]string{},
bNodes: map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"},
certname: "h1.example.net",
want: "b",
},
{
// b reported later, so b owns the node even though a is configured
// first and would answer a plain pass-through.
name: "held by both, newer report on b",
aNodes: map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"},
bNodes: map[string]string{"h1.example.net": "2026-07-20T00:00:00Z"},
certname: "h1.example.net",
want: "b",
},
{
name: "held by both, newer report on a",
aNodes: map[string]string{"h1.example.net": "2026-07-20T00:00:00Z"},
bNodes: map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"},
certname: "h1.example.net",
want: "a",
},
} {
t.Run(tc.name, func(t *testing.T) {
a := newVoxBackend(t, "a", tc.aNodes)
b := newVoxBackend(t, "b", tc.bNodes)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
for _, rt := range perNodePaths {
rec := doGet(t, srv.Handler(), rt.path(tc.certname), "")
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d: %s", rt.name, rec.Code, rec.Body.String())
}
if got := rec.Body.String(); !strings.Contains(got, `"`+tc.want+`"`) {
t.Errorf("%s answered by the wrong backend, want %q: %s", rt.name, tc.want, got)
}
}
// The collection merge has to agree: one node cannot report one
// backend's facts here and another's through /facts.
rec := doGet(t, srv.Handler(), factsPath, `["=","certname","`+tc.certname+`"]`)
if got := factRoleValue(t, rec.Body.Bytes(), tc.certname); got != tc.want {
t.Errorf("/facts resolved %s to %q, want %q; the routes disagree", tc.certname, got, tc.want)
}
})
}
}
// factRoleValue reads the certname's "role" fact out of a merged /facts body.
func factRoleValue(t *testing.T, body []byte, certname string) string {
t.Helper()
var recs []struct {
Certname string `json:"certname"`
Name string `json:"name"`
Value string `json:"value"`
}
if err := json.Unmarshal(body, &recs); err != nil {
t.Fatalf("unmarshal %s: %v", body, err)
}
for _, rec := range recs {
if rec.Certname == certname && rec.Name == "role" {
return rec.Value
}
}
t.Fatalf("no role fact for %s in %s", certname, body)
return ""
}
// A certname neither backend holds must still look like openvoxdb: its own 404
// status and error body, not an empty success.
func TestCertnameRoutes_UnknownCertnameKeepsTheUpstream404(t *testing.T) {
a := newVoxBackend(t, "a", map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"})
b := newVoxBackend(t, "b", map[string]string{"h2.example.net": "2026-07-01T00:00:00Z"})
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
for _, tc := range []struct {
path string
want string
}{
{factsetsPath + "/gone.example.net", notFoundBody("factset", "gone.example.net")},
{nodesPath + "/gone.example.net/facts", notFoundBody("node", "gone.example.net")},
} {
rec := doGet(t, srv.Handler(), tc.path, "")
if rec.Code != http.StatusNotFound {
t.Errorf("%s: status %d, want 404: %s", tc.path, rec.Code, rec.Body.String())
}
if got := rec.Body.String(); got != tc.want {
t.Errorf("%s: body %q, want %q", tc.path, got, tc.want)
}
if got := rec.Header().Get("Content-Type"); got != "application/json;charset=utf-8" {
t.Errorf("%s: content type %q", tc.path, got)
}
}
}
// A backend that is down is indistinguishable from one that has never heard of
// the node, so the surviving backend still answers.
func TestCertnameRoutes_OneBackendDown(t *testing.T) {
a := newVoxBackend(t, "a", map[string]string{"h1.example.net": "2026-07-20T00:00:00Z"})
b := newVoxBackend(t, "b", map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"})
// a owns h1 by freshness, so the fallback is exercised only if a is skipped.
a.fail = true
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
for _, rt := range perNodePaths {
rec := doGet(t, srv.Handler(), rt.path("h1.example.net"), "")
if rec.Code != http.StatusOK {
t.Fatalf("%s: status %d, want the survivor's 200: %s", rt.name, rec.Code, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), `"b"`) {
t.Errorf("%s: want b's record, got %s", rt.name, rec.Body.String())
}
}
}
// Static merge attributes no certname to a backend, so these routes keep the
// configured order the /facts merge falls back to.
func TestCertnameRoutes_StaticMergeKeepsConfiguredOrder(t *testing.T) {
a := newVoxBackend(t, "a", map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"})
b := newVoxBackend(t, "b", map[string]string{"h1.example.net": "2026-07-20T00:00:00Z"})
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), factsetsPath+"/h1.example.net", "")
if !strings.Contains(rec.Body.String(), `"a"`) {
t.Errorf("static merge should keep the first configured backend, got %s", rec.Body.String())
}
}
// openvoxdb ANDs the request's own query onto the certname restriction
// (src/puppetlabs/puppetdb/http/query.clj:136-143,158-164), so the owner has to
// receive it unchanged.
func TestCertnameRoutes_ForwardQueryToTheOwner(t *testing.T) {
a := newVoxBackend(t, "a", map[string]string{"h1.example.net": "2026-07-20T00:00:00Z"})
b := newVoxBackend(t, "b", map[string]string{"h1.example.net": "2026-07-01T00:00:00Z"})
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
const q = `["=","name","role"]`
path := nodesPath + "/h1.example.net/facts"
if rec := doGet(t, srv.Handler(), path, q); rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
if got := a.params(path).Get("query"); got != q {
t.Errorf("owner got query %q, want %q", got, q)
}
if got := b.params(path).Get("query"); got != "" {
t.Errorf("non-owner was queried with %q; only the owner should be asked", got)
}
}
func TestCertnameFor(t *testing.T) {
for _, tc := range []struct{ path, want string }{
{factsetsPath + "/h1.example.net", "h1.example.net"},
{nodesPath + "/h1.example.net", "h1.example.net"},
{nodesPath + "/h1.example.net/facts", "h1.example.net"},
{nodesPath + "/h1.example.net/facts/role/web", "h1.example.net"},
{nodesPath + "/h1.example.net/resources", "h1.example.net"},
{catalogsPath + "/h1.example.net/edges", "h1.example.net"},
// Collections and fact-name drilldowns are merged elsewhere and must
// not be mistaken for a certname.
{nodesPath, ""},
{nodesPath + "/", ""},
{factsPath, ""},
{factsPath + "/role", ""},
{factsetsPath, ""},
{reportsPath + "/abc123/events", ""},
} {
if got := certnameFor(tc.path); got != tc.want {
t.Errorf("certnameFor(%q) = %q, want %q", tc.path, got, tc.want)
}
}
}
func TestCertnameRoutes_ClaimedByTheRouteTable(t *testing.T) {
for _, path := range []string{
factsetsPath + "/h1.example.net",
nodesPath + "/h1.example.net",
nodesPath + "/h1.example.net/facts",
catalogsPath + "/h1.example.net",
} {
if got := routeFor(path).name; got != certnameRouteName {
t.Errorf("routeFor(%q) = %q, want %q", path, got, certnameRouteName)
}
}
}
// The owner holds the node, so its answer is the answer: a non-owner's 404 only
// says the node is not its own, and demanding it agree turns the owner's real
// error into a 502 that describes neither backend.
func TestCertnameRoutes_OwnerAnswerWins(t *testing.T) {
const certname = "h1.example.net"
for _, tc := range []struct {
name string
// setUp configures the owner (a, which reported later) and the backend
// that does not hold the node (b).
setUp func(a, b *voxBackend)
wantCode int
wantBody string
}{
{
name: "owner errors while the non-owner 404s",
setUp: func(a, b *voxBackend) { a.perNodeStatus, a.perNodeBody = http.StatusInternalServerError, "boom" },
wantCode: http.StatusInternalServerError,
wantBody: "boom",
},
{
name: "owner and non-owner error differently",
setUp: func(a, b *voxBackend) {
a.perNodeStatus, a.perNodeBody = http.StatusInternalServerError, "boom"
b.perNodeStatus, b.perNodeBody = http.StatusServiceUnavailable, "busy"
},
wantCode: http.StatusInternalServerError,
wantBody: "boom",
},
{
// The fallback chain is there so an unreachable owner does not take the
// node with it; a backend that did answer explains more than a 502.
name: "owner unreachable, fallback answers",
setUp: func(a, b *voxBackend) {
a.perNodeDead = true
b.perNodeStatus, b.perNodeBody = http.StatusServiceUnavailable, "busy"
},
wantCode: http.StatusServiceUnavailable,
wantBody: "busy",
},
{
name: "owner succeeds while the non-owner errors",
setUp: func(a, b *voxBackend) {
b.perNodeStatus, b.perNodeBody = http.StatusInternalServerError, "boom"
},
wantCode: http.StatusOK,
wantBody: `"a"`,
},
{
// Nothing answered at all, so there is no upstream reply to replay.
name: "every backend unreachable",
setUp: func(a, b *voxBackend) { a.perNodeDead, b.perNodeDead = true, true },
wantCode: http.StatusBadGateway,
},
} {
t.Run(tc.name, func(t *testing.T) {
a := newVoxBackend(t, "a", map[string]string{certname: "2026-07-20T00:00:00Z"})
b := newVoxBackend(t, "b", map[string]string{})
tc.setUp(a, b)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
for _, rt := range perNodePaths {
rec := doGet(t, srv.Handler(), rt.path(certname), "")
if rec.Code != tc.wantCode {
t.Fatalf("%s: status = %d, want %d: %s", rt.name, rec.Code, tc.wantCode, rec.Body.String())
}
if tc.wantBody != "" && !strings.Contains(rec.Body.String(), tc.wantBody) {
t.Errorf("%s: body = %q, want it to contain %q", rt.name, rec.Body.String(), tc.wantBody)
}
}
})
}
}
+27 -5
View File
@@ -2,6 +2,7 @@ package main
import ( import (
"fmt" "fmt"
"net/url"
"os" "os"
"path/filepath" "path/filepath"
"strconv" "strconv"
@@ -59,7 +60,7 @@ type Config struct {
Merge string `yaml:"merge"` Merge string `yaml:"merge"`
Timeout time.Duration `yaml:"timeout"` Timeout time.Duration `yaml:"timeout"`
FreshnessTTL time.Duration `yaml:"freshness_ttl"` FreshnessTTL time.Duration `yaml:"freshness_ttl"`
FactsTTL time.Duration `yaml:"facts_ttl"` // 0 disables the /facts+/nodes cache FactsTTL time.Duration `yaml:"facts_ttl"` // 0 disables the merged /nodes cache
CacheBytes int64 `yaml:"facts_cache_bytes"` // byte budget for that cache CacheBytes int64 `yaml:"facts_cache_bytes"` // byte budget for that cache
SourceFact string `yaml:"source_fact"` SourceFact string `yaml:"source_fact"`
@@ -289,12 +290,22 @@ func (c Config) configHint() string {
return ConfigPath() return ConfigPath()
} }
// backendHost is the upstream's hostname, the value of the source fact.
func backendHost(raw string) string {
u, err := url.Parse(raw)
if err != nil {
return ""
}
return strings.ToLower(u.Hostname())
}
func (c Config) Validate() error { func (c Config) Validate() error {
if len(c.Backends) == 0 { if len(c.Backends) == 0 {
return fmt.Errorf("no backends configured: set %sBACKENDS to \"name=url,name=url\" or add a backends list to %s", return fmt.Errorf("no backends configured: set %sBACKENDS to \"name=url,name=url\" or add a backends list to %s",
envPrefix, c.configHint()) envPrefix, c.configHint())
} }
seen := map[string]bool{} seen := map[string]bool{}
hosts := map[string]string{}
for _, b := range c.Backends { for _, b := range c.Backends {
if b.Name == "" || b.URL == "" { if b.Name == "" || b.URL == "" {
return fmt.Errorf("backend requires both name and url: %+v", b) return fmt.Errorf("backend requires both name and url: %+v", b)
@@ -303,6 +314,17 @@ func (c Config) Validate() error {
return fmt.Errorf("duplicate backend name %q", b.Name) return fmt.Errorf("duplicate backend name %q", b.Name)
} }
seen[b.Name] = true seen[b.Name] = true
if !c.SourceFactEnabled {
continue
}
host := backendHost(b.URL)
if host == "" {
return fmt.Errorf("backend %q url %q has no host", b.Name, b.URL)
}
if other, dup := hosts[host]; dup {
return fmt.Errorf("backends %q and %q share host %q; %s values would collide", other, b.Name, host, c.SourceFact)
}
hosts[host] = b.Name
} }
switch c.Merge { switch c.Merge {
case mergeFreshness, mergeStatic: case mergeFreshness, mergeStatic:
@@ -344,8 +366,8 @@ func (c Config) Validate() error {
return nil return nil
} }
// cacheEnabled reports whether a facts/nodes cache should be built: both a TTL // cacheEnabled reports whether the merged /nodes cache should be built: both a
// and a byte budget are required. // TTL and a byte budget are required.
func (c Config) cacheEnabled() bool { return c.FactsTTL > 0 && c.CacheBytes > 0 } func (c Config) cacheEnabled() bool { return c.FactsTTL > 0 && c.CacheBytes > 0 }
func writeDefaultConfig(path string) error { func writeDefaultConfig(path string) error {
@@ -366,8 +388,8 @@ func writeDefaultConfig(path string) error {
"# PDBMUX_HEALTH_PROBE_ENABLED, PDBMUX_HEALTH_PROBE_PATH,\n" + "# PDBMUX_HEALTH_PROBE_ENABLED, PDBMUX_HEALTH_PROBE_PATH,\n" +
"# PDBMUX_HEALTH_PROBE_INTERVAL, PDBMUX_HEALTH_PROBE_TIMEOUT,\n" + "# PDBMUX_HEALTH_PROBE_INTERVAL, PDBMUX_HEALTH_PROBE_TIMEOUT,\n" +
"# PDBMUX_HEALTH_PROBE_FAILURES, PDBMUX_HEALTH_PROBE_SUCCESSES.\n" + "# PDBMUX_HEALTH_PROBE_FAILURES, PDBMUX_HEALTH_PROBE_SUCCESSES.\n" +
"# facts_ttl caches merged /facts and /nodes in memory; it is capped at 30s\n" + "# facts_ttl caches merged /nodes in memory; it is capped at 30s (a larger\n" +
"# (a larger value is clamped) and 0 disables the cache.\n" + "# value is clamped) and 0 disables the cache. Fact answers are never cached.\n" +
"# health_probe_* polls each backend's status endpoint so queries skip a\n" + "# health_probe_* polls each backend's status endpoint so queries skip a\n" +
"# backend that is down; when every backend is down all are queried anyway.\n\n") "# backend that is down; when every backend is down all are queried anyway.\n\n")
if err := os.WriteFile(path, append(header, data...), 0o644); err != nil { if err := os.WriteFile(path, append(header, data...), 0o644); err != nil {
+27 -3
View File
@@ -15,7 +15,7 @@ func testConfigValid() Config {
cfg := DefaultConfig() cfg := DefaultConfig()
cfg.Backends = []Backend{ cfg.Backends = []Backend{
{Name: "a", URL: "http://localhost:18080"}, {Name: "a", URL: "http://localhost:18080"},
{Name: "b", URL: "http://localhost:18081"}, {Name: "b", URL: "http://127.0.0.1:18081"},
} }
return cfg return cfg
} }
@@ -57,7 +57,7 @@ func TestLoad_NoBackendsLoadsButFailsValidation(t *testing.T) {
func TestLoad_BackendsKeepConfiguredOrder(t *testing.T) { func TestLoad_BackendsKeepConfiguredOrder(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir()) t.Setenv("XDG_CONFIG_HOME", t.TempDir())
clearEnv(t) clearEnv(t)
t.Setenv(envPrefix+"BACKENDS", "a=http://localhost:18080,b=http://localhost:18081") t.Setenv(envPrefix+"BACKENDS", "a=http://localhost:18080,b=http://127.0.0.1:18081")
cfg, err := Load("") cfg, err := Load("")
if err != nil { if err != nil {
@@ -229,8 +229,20 @@ func TestValidate(t *testing.T) {
}{ }{
{"ok", func(*Config) {}, false}, {"ok", func(*Config) {}, false},
{"no backends", func(c *Config) { c.Backends = nil }, true}, {"no backends", func(c *Config) { c.Backends = nil }, true},
{"dup name", func(c *Config) { c.Backends = append(c.Backends, Backend{Name: "a", URL: "x"}) }, true}, {"dup name", func(c *Config) { c.Backends = append(c.Backends, Backend{Name: "a", URL: "http://c"}) }, true},
{"missing url", func(c *Config) { c.Backends[0].URL = "" }, true}, {"missing url", func(c *Config) { c.Backends[0].URL = "" }, true},
{"url without host", func(c *Config) { c.Backends[0].URL = "localhost:18080" }, true},
{"shared host on another port", func(c *Config) { c.Backends[1].URL = "http://localhost:18081" }, true},
{"same host differing by scheme", func(c *Config) { c.Backends[1].URL = "https://localhost" }, true},
{"same host differing by case", func(c *Config) { c.Backends[1].URL = "http://LocalHost:18081" }, true},
{"shared host while source fact disabled", func(c *Config) {
c.SourceFactEnabled = false
c.Backends[1].URL = "http://localhost:18081"
}, false},
{"url without host while source fact disabled", func(c *Config) {
c.SourceFactEnabled = false
c.Backends[0].URL = "localhost:18080"
}, false},
{"bad merge", func(c *Config) { c.Merge = "wrong" }, true}, {"bad merge", func(c *Config) { c.Merge = "wrong" }, true},
{"zero timeout", func(c *Config) { c.Timeout = 0 }, true}, {"zero timeout", func(c *Config) { c.Timeout = 0 }, true},
{"empty source fact while enabled", func(c *Config) { c.SourceFact = "" }, true}, {"empty source fact while enabled", func(c *Config) { c.SourceFact = "" }, true},
@@ -512,3 +524,15 @@ func clearEnv(t *testing.T) {
t.Setenv(envPrefix+k, "") t.Setenv(envPrefix+k, "")
} }
} }
func TestBackendHost(t *testing.T) {
for raw, want := range map[string]string{
"https://PuppetDB.Example:8081/x": "puppetdb.example",
"http://[::1]:8080": "::1",
"localhost:8080": "",
} {
if got := backendHost(raw); got != want {
t.Errorf("backendHost(%q) = %q, want %q", raw, got, want)
}
}
}
+1 -1
View File
@@ -154,7 +154,7 @@ func startBackend(ctx context.Context, t fatalf, name, netName string) *backend
fail("starting openvoxdb for %s: %v", name, err) fail("starting openvoxdb for %s: %v", name, err)
} }
return &backend{name: name, url: fmt.Sprintf("http://127.0.0.1:%d", port), pg: pg, pdb: pdb} return &backend{name: name, url: fmt.Sprintf("http://%s.localhost:%d", name, port), pg: pg, pdb: pdb}
} }
// waitForPuppetDBRunning gates on the trapperkeeper status service reporting // waitForPuppetDBRunning gates on the trapperkeeper status service reporting
+3
View File
@@ -42,6 +42,9 @@ func fixtureTime(offset time.Duration) string {
const ( const (
backendAName = "pdb-a" backendAName = "pdb-a"
backendBName = "pdb-b" backendBName = "pdb-b"
// The source fact's value is the backend host, so each backend gets its own.
backendAHost = backendAName + ".localhost"
backendBHost = backendBName + ".localhost"
) )
type nodeFixture struct { type nodeFixture struct {
+2 -2
View File
@@ -77,8 +77,8 @@ func TestBackendDeathAndRecovery(t *testing.T) {
if got, _ := factValue(facts, nodeShared, "owner"); got != backendAName { if got, _ := factValue(facts, nodeShared, "owner"); got != backendAName {
t.Errorf("owner of %s with %s down = %v, want %q", nodeShared, h.b.name, got, backendAName) t.Errorf("owner of %s with %s down = %v, want %q", nodeShared, h.b.name, got, backendAName)
} }
if got, _ := factValue(facts, nodeShared, defaultSourceFact); got != backendAName { if got, _ := factValue(facts, nodeShared, defaultSourceFact); got != backendAHost {
t.Errorf("%s for %s with %s down = %v, want %q", defaultSourceFact, nodeShared, h.b.name, got, backendAName) t.Errorf("%s for %s with %s down = %v, want %q", defaultSourceFact, nodeShared, h.b.name, got, backendAHost)
} }
}) })
+2 -2
View File
@@ -84,8 +84,8 @@ func TestNodeLookupShowsProvenance(t *testing.T) {
if !strings.Contains(line, defaultSourceFact) { if !strings.Contains(line, defaultSourceFact) {
continue continue
} }
if !strings.Contains(line, backendBName) { if !strings.Contains(line, backendBHost) {
t.Fatalf("node-lookup reports %q for %s, want the owning backend %q", strings.TrimSpace(line), nodeShared, backendBName) t.Fatalf("node-lookup reports %q for %s, want the owning backend %q", strings.TrimSpace(line), nodeShared, backendBHost)
} }
return return
} }
+12 -12
View File
@@ -286,8 +286,8 @@ func TestSharedNodeResolvesToTheFresherBackend(t *testing.T) {
if got := node["report_timestamp"]; got != tsSharedOnB { if got := node["report_timestamp"]; got != tsSharedOnB {
t.Errorf("merged /nodes report_timestamp for %s = %v, want the fresher %q", nodeShared, got, tsSharedOnB) t.Errorf("merged /nodes report_timestamp for %s = %v, want the fresher %q", nodeShared, got, tsSharedOnB)
} }
if got := node[defaultSourceFact]; got != backendBName { if got := node[defaultSourceFact]; got != backendBHost {
t.Errorf("merged /nodes %s for %s = %v, want %q", defaultSourceFact, nodeShared, got, backendBName) t.Errorf("merged /nodes %s for %s = %v, want %q", defaultSourceFact, nodeShared, got, backendBHost)
} }
rows := get(t, factsPath, query(`["=","certname","`+nodeShared+`"]`)).rows(t) rows := get(t, factsPath, query(`["=","certname","`+nodeShared+`"]`)).rows(t)
@@ -769,10 +769,10 @@ func TestSourceFactInjectionAndGating(t *testing.T) {
t.Run("facts carry one source record per node", func(t *testing.T) { t.Run("facts carry one source record per node", func(t *testing.T) {
rows := get(t, factsPath, nil).rows(t) rows := get(t, factsPath, nil).rows(t)
want := map[string]string{ want := map[string]string{
nodeAlpha: backendAName, nodeAlpha: backendAHost,
nodeBeta: backendBName, nodeBeta: backendBHost,
nodeGamma: backendBName, nodeGamma: backendBHost,
nodeShared: backendBName, // won on freshness, not on configured order nodeShared: backendBHost, // won on freshness, not on configured order
} }
got := map[string]int{} got := map[string]int{}
for _, row := range rows { for _, row := range rows {
@@ -839,10 +839,10 @@ func TestSourceFactInjectionAndGating(t *testing.T) {
// records /facts carries rather than the empty set the backends hold. // records /facts carries rather than the empty set the backends hold.
func TestSourceFactDrilldown(t *testing.T) { func TestSourceFactDrilldown(t *testing.T) {
want := map[string]string{ want := map[string]string{
nodeAlpha: backendAName, nodeAlpha: backendAHost,
nodeBeta: backendBName, nodeBeta: backendBHost,
nodeGamma: backendBName, nodeGamma: backendBHost,
nodeShared: backendBName, // won on freshness, not on configured order nodeShared: backendBHost, // won on freshness, not on configured order
} }
path := factsPath + "/" + defaultSourceFact path := factsPath + "/" + defaultSourceFact
@@ -882,8 +882,8 @@ func TestSourceFactDrilldown(t *testing.T) {
value string value string
want []string want []string
}{ }{
{backendAName, []string{nodeAlpha}}, {backendAHost, []string{nodeAlpha}},
{backendBName, []string{nodeBeta, nodeGamma, nodeShared}}, {backendBHost, []string{nodeBeta, nodeGamma, nodeShared}},
{"nosuchbackend", nil}, {"nosuchbackend", nil},
} { } {
rows := get(t, path+"/"+tc.value, nil).rows(t) rows := get(t, path+"/"+tc.value, nil).rows(t)
+67 -1
View File
@@ -4,6 +4,7 @@ package main
import ( import (
"context" "context"
"net/url"
"strings" "strings"
"testing" "testing"
) )
@@ -22,7 +23,7 @@ func TestRejectedQuerySurfacesUpstreamStatus(t *testing.T) {
t.Run(tc.name, func(t *testing.T) { t.Run(tc.name, func(t *testing.T) {
params := query(tc.q) params := query(tc.q)
status, upstream := h.a.queryRaw(ctx, t, tc.path, params) status, upstream := h.a.queryRaw(ctx, t, tc.path, params)
if !clientShaped(status) { if !replayableStatus(status) || status >= 500 {
t.Fatalf("backend %s answered HTTP %d for %s, which is not the client error this test needs: %s", t.Fatalf("backend %s answered HTTP %d for %s, which is not the client error this test needs: %s",
h.a.name, status, tc.q, upstream) h.a.name, status, tc.q, upstream)
} }
@@ -56,3 +57,68 @@ func TestRejectedQuerySurfacesUpstreamStatus(t *testing.T) {
}) })
} }
} }
// openvoxdb answers the same malformed clause with 400 on /nodes and 500 on
// /facts — engine.clj's rewrite-fact-query runs an unguarded nth only for the
// facts entity. Both are the backends' real answer, so both have to reach the
// client rather than one of them becoming a 502.
func TestUnanimousServerErrorSurfacesUpstreamStatus(t *testing.T) {
ctx := context.Background()
params := query(`["=","name"]`)
status, upstream := h.a.queryRaw(ctx, t, factsPath, params)
if status < 500 {
t.Skipf("backend %s answered HTTP %d for the arity bug this test needs", h.a.name, status)
}
if other, _ := h.b.queryRaw(ctx, t, factsPath, params); other != status {
t.Fatalf("backends answered %d and %d, so the failure is not unanimous", status, other)
}
resp := rawGet(t, factsPath, params)
if resp.status != status {
t.Fatalf("pdbmux answered HTTP %d, want the upstream %d: %s", resp.status, status, resp.body)
}
if got, want := strings.TrimSpace(string(resp.body)), strings.TrimSpace(string(upstream)); want != "" && got != want {
t.Errorf("pdbmux body = %q, want openvoxdb's own explanation %q", got, want)
}
for _, b := range []*backend{h.a, h.b} {
if strings.Contains(string(resp.body), b.url) {
t.Errorf("replayed body names backend %s: %q", b.name, resp.body)
}
}
}
// /metrics and /pdb/meta had their own copy of the gateway error, so a status
// both backends agreed on never reached the client on those routes.
func TestUnanimousMetricsStatusIsReplayed(t *testing.T) {
ctx := context.Background()
for _, path := range []string{"/metrics/v1/mbeans", "/metrics/v2/list"} {
t.Run(path, func(t *testing.T) {
status, _ := h.a.queryRaw(ctx, t, path, nil)
if status < 400 {
t.Skipf("%s answers HTTP %d on this estate, so there is nothing to replay", path, status)
}
if other, _ := h.b.queryRaw(ctx, t, path, nil); other != status {
t.Fatalf("backends answered %d and %d, so the status is not unanimous", status, other)
}
if resp := rawGet(t, path, nil); resp.status != status {
t.Fatalf("pdbmux answered HTTP %d, want the upstream %d: %s", resp.status, status, resp.body)
}
})
}
}
// A client cannot tell pdbmux from a PuppetDB by the content type either.
func TestSuccessContentTypeMatchesUpstream(t *testing.T) {
ctx := context.Background()
params := url.Values{"limit": {"1"}}
_, _ = h.a.queryRaw(ctx, t, nodesPath, params)
resp := rawGet(t, nodesPath, params)
if resp.status != 200 {
t.Fatalf("status %d: %s", resp.status, resp.body)
}
if got := resp.header.Get("Content-Type"); got != "application/json;charset=utf-8" {
t.Errorf("Content-Type = %q, want openvoxdb's own %q", got, "application/json;charset=utf-8")
}
}
+167 -31
View File
@@ -6,6 +6,8 @@ import (
"net/url" "net/url"
"reflect" "reflect"
"slices" "slices"
"sync"
"sync/atomic"
"testing" "testing"
) )
@@ -257,15 +259,15 @@ func TestHandler_FactsBySourceNamePathSynthesisesOneRecordPerNode(t *testing.T)
for _, merge := range []string{mergeStatic, mergeFreshness} { for _, merge := range []string{mergeStatic, mergeFreshness} {
t.Run(merge, func(t *testing.T) { t.Run(merge, func(t *testing.T) {
a, b := sourceFactBackends(t) a, b := sourceFactBackends(t)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, merge)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, merge))
// h3 lives in both: static keeps the first configured backend, freshness // h3 lives in both: static keeps the first configured backend, freshness
// the one holding its newer report. // the one holding its newer report.
wantH3 := "a" wantH3 := hostA
if merge == mergeFreshness { if merge == mergeFreshness {
wantH3 = "b" wantH3 = hostB
} }
want := map[string]string{"h1": "a", "h2": "b", "h3": wantH3} want := map[string]string{"h1": hostA, "h2": hostB, "h3": wantH3}
rec := doGet(t, srv.Handler(), sourceFactURL, "") rec := doGet(t, srv.Handler(), sourceFactURL, "")
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
@@ -313,20 +315,21 @@ func TestHandler_FactsBySourceNamePathCarriesTheOwnersEnvironment(t *testing.T)
} }
} }
// The pinned value names a backend, so the sub-route is the drilldown filtered // The pinned value is a backend host, so the sub-route is the drilldown filtered
// by owner; a value naming no backend matches nothing. // by owner; any other value, a backend's name included, matches nothing.
func TestHandler_FactsBySourceNameAndValueFiltersByOwner(t *testing.T) { func TestHandler_FactsBySourceNameAndValueFiltersByOwner(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
value string value string
want map[string]string want map[string]string
}{ }{
{"a", map[string]string{"h1": "a"}}, {hostA, map[string]string{"h1": hostA}},
{"b", map[string]string{"h2": "b", "h3": "b"}}, {hostB, map[string]string{"h2": hostB, "h3": hostB}},
{"a", map[string]string{}},
{"nosuchbackend", map[string]string{}}, {"nosuchbackend", map[string]string{}},
} { } {
t.Run(tc.value, func(t *testing.T) { t.Run(tc.value, func(t *testing.T) {
a, b := sourceFactBackends(t) a, b := sourceFactBackends(t)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeFreshness))
rec := doGet(t, srv.Handler(), sourceFactURL+"/"+tc.value, "") rec := doGet(t, srv.Handler(), sourceFactURL+"/"+tc.value, "")
got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact) got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact)
@@ -370,20 +373,20 @@ func TestHandler_FactsBySourceNameUnknownValueSkipsFanOut(t *testing.T) {
} }
} }
// The record set is the estate's, not the pinned value's, so every value shares // The record set is the estate's, not the pinned value's: the <value> segment is
// one entry: two valid values must not cost two whole-estate fan-outs. // applied to the merged records rather than fanned out as a path of its own.
func TestHandler_FactsBySourceNameValuesShareOneFetch(t *testing.T) { func TestHandler_FactsBySourceNameValuesFilterOneFetch(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "osfamily", "RedHat", "") + `]`}) a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "osfamily", "RedHat", "") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h2", "osfamily", "Debian", "") + `]`}) b := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h2", "osfamily", "Debian", "") + `]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(withHost(a.srv.URL, hostA), withHost(b.srv.URL, hostB)))
for _, tc := range []struct { for _, tc := range []struct {
path string path string
want map[string]string want map[string]string
}{ }{
{sourceFactURL + "/a", map[string]string{"h1": "a"}}, {sourceFactURL + "/" + hostA, map[string]string{"h1": hostA}},
{sourceFactURL + "/b", map[string]string{"h2": "b"}}, {sourceFactURL + "/" + hostB, map[string]string{"h2": hostB}},
{sourceFactURL, map[string]string{"h1": "a", "h2": "b"}}, {sourceFactURL, map[string]string{"h1": hostA, "h2": hostB}},
} { } {
rec := doGet(t, srv.Handler(), tc.path, "") rec := doGet(t, srv.Handler(), tc.path, "")
got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact) got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact)
@@ -392,8 +395,8 @@ func TestHandler_FactsBySourceNameValuesShareOneFetch(t *testing.T) {
} }
} }
for _, cb := range []*countingBackend{a, b} { for _, cb := range []*countingBackend{a, b} {
if got := cb.hitCount(factsPath); got != 1 { if got := cb.hitCount(factsPath); got != 3 {
t.Errorf("%s fetched %d times, want 1 shared fetch", factsPath, got) t.Errorf("%s fetched %d times, want one uncached fetch per request", factsPath, got)
} }
if got := cb.hitCount(sourceFactURL); got != 0 { if got := cb.hitCount(sourceFactURL); got != 0 {
t.Errorf("%s was fanned out %d times, want 0", sourceFactURL, got) t.Errorf("%s was fanned out %d times, want 0", sourceFactURL, got)
@@ -432,12 +435,12 @@ func TestHandler_FactsBySourceNamePathRespectsQuery(t *testing.T) {
a, b := sourceFactBackends(t) a, b := sourceFactBackends(t)
a.bodies[factsPath] = `[` + factEnv("h1", "osfamily", "RedHat", "prod") + `]` a.bodies[factsPath] = `[` + factEnv("h1", "osfamily", "RedHat", "prod") + `]`
b.bodies[factsPath] = `[]` b.bodies[factsPath] = `[]`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeFreshness))
const q = `["=","certname","h1"]` const q = `["=","certname","h1"]`
rec := doGet(t, srv.Handler(), sourceFactURL, q) rec := doGet(t, srv.Handler(), sourceFactURL, q)
got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact) got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact)
if n != 1 || got["h1"] != "a" { if n != 1 || got["h1"] != hostA {
t.Fatalf("%s?query=%s = %v (%d records), want only h1: %s", sourceFactURL, q, got, n, rec.Body.String()) t.Fatalf("%s?query=%s = %v (%d records), want only h1: %s", sourceFactURL, q, got, n, rec.Body.String())
} }
// The query has to reach the backends for them to narrow anything. // The query has to reach the backends for them to narrow anything.
@@ -455,11 +458,11 @@ func TestHandler_FactsBySourceNamePathSuppressesUpstream(t *testing.T) {
a.bodies[factsPath] = `[` + factEnv("h1", "osfamily", "RedHat", "prod") + `,` + a.bodies[factsPath] = `[` + factEnv("h1", "osfamily", "RedHat", "prod") + `,` +
fact("h1", defaultSourceFact, "stale", "") + `]` fact("h1", defaultSourceFact, "stale", "") + `]`
b.bodies[factsPath] = `[]` b.bodies[factsPath] = `[]`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeFreshness))
rec := doGet(t, srv.Handler(), sourceFactURL, "") rec := doGet(t, srv.Handler(), sourceFactURL, "")
got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact) got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact)
if n != 1 || got["h1"] != "a" { if n != 1 || got["h1"] != hostA {
t.Fatalf("%s = %v (%d records), want exactly the synthetic one: %s", sourceFactURL, got, n, rec.Body.String()) t.Fatalf("%s = %v (%d records), want exactly the synthetic one: %s", sourceFactURL, got, n, rec.Body.String())
} }
} }
@@ -678,22 +681,155 @@ func TestHandler_FactNamesSurvivesOneBackend(t *testing.T) {
} }
} }
// Both routes are merged record sets, so they use the same cache /facts does. // Every route that serves fact records fans out per request, even with the
func TestHandler_FactRoutesAreCached(t *testing.T) { // cache configured: a fact answer is only ever as old as the backend's own.
func TestHandler_FactRoutesAreNotCached(t *testing.T) {
for _, path := range []string{roleFactPath, roleFactPath + "/web", sourceFactURL, factNamesPath} { for _, path := range []string{roleFactPath, roleFactPath + "/web", sourceFactURL, factNamesPath} {
t.Run(path, func(t *testing.T) { t.Run(path, func(t *testing.T) {
// The source-fact drilldown is answered from the /facts merge, so
// that is the path its fan-out lands on.
fanOut := path
if path == sourceFactURL {
fanOut = factsPath
}
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
a.bodies[path] = `[]` a.bodies[fanOut] = `[]`
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
b.bodies[path] = `[]` b.bodies[fanOut] = `[]`
srv := newTestServer(cacheTestConfig(a.srv.URL, b.srv.URL)) srv := newTestServer(cacheTestConfig(a.srv.URL, b.srv.URL))
if got := doGet(t, srv.Handler(), path, "").Header().Get(cacheStatusHeader); got != "miss" { doGet(t, srv.Handler(), path, "")
t.Errorf("first %s = %q, want miss", cacheStatusHeader, got) doGet(t, srv.Handler(), path, "")
} if got := a.hits(fanOut); got != 2 {
if got := doGet(t, srv.Handler(), path, "").Header().Get(cacheStatusHeader); got != "hit" { t.Errorf("backend a saw %d requests for %s, want one per request", got, fanOut)
t.Errorf("second %s = %q, want hit", cacheStatusHeader, got)
} }
}) })
} }
} }
// node-lookup and puppetboard filter /facts by name with an AST rather than the
// drilldown path; those queries select the synthetic records the path serves.
func TestHandler_FactsQuerySelectsSourceFact(t *testing.T) {
src := defaultSourceFact
tests := []struct {
query string
want map[string]string
}{
{`["=","name","` + src + `"]`, map[string]string{"h1": hostA, "h2": hostB, "h3": hostB}},
{`["and",["=","name","` + src + `"]]`, map[string]string{"h1": hostA, "h2": hostB, "h3": hostB}},
{`["and",["=","name","` + src + `"],["=","value","` + hostB + `"]]`, map[string]string{"h2": hostB, "h3": hostB}},
{`["and",["=","name","` + src + `"],["=","value","b"]]`, map[string]string{}},
{`["and",["=","name","` + src + `"],["not",["~","value","^b\\.localhost$"]]]`, map[string]string{"h1": hostA}},
{`["and",["=","certname","h3"],["=","name","` + src + `"]]`, map[string]string{"h3": hostB}},
{`["and",["or",["=","name","osfamily"],["=","name","` + src + `"]]]`, map[string]string{"h1": hostA, "h2": hostB, "h3": hostB}},
{`["in","name",["array",["osfamily","` + src + `"]]]`, map[string]string{"h1": hostA, "h2": hostB, "h3": hostB}},
{`["and",["or",["=","name","` + src + `"]],["=","environment","prod"]]`, map[string]string{"h1": hostA}},
}
for _, tc := range tests {
t.Run(tc.query, func(t *testing.T) {
a, b := sourceFactBackends(t)
a.bodies[factsPath] = `[` + factEnv("h1", "osfamily", "RedHat", "prod") + `,` + fact("h1", src, "stale", "") + `]`
srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeFreshness))
rec := doGet(t, srv.Handler(), factsPath, tc.query)
if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
got, n := sourceValues(t, rec.Body.Bytes(), src)
if n != len(tc.want) || !reflect.DeepEqual(got, tc.want) {
t.Fatalf("got %v (%d records), want %v: %s", got, n, tc.want, rec.Body.String())
}
})
}
}
// The query still reaches the backends, so the real facts it names come back
// beside the synthetic ones.
func TestHandler_FactsQueryMixedNamesKeepsRealFacts(t *testing.T) {
a, b := sourceFactBackends(t)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
rec := doGet(t, srv.Handler(), factsPath, `["in","name",["array",["osfamily","`+defaultSourceFact+`"]]]`)
if os, _ := sourceValues(t, rec.Body.Bytes(), "osfamily"); len(os) != 3 {
t.Errorf("osfamily = %v, want all three nodes: %s", os, rec.Body.String())
}
if h := rec.Header().Get(backendsHeader); h != "2/2" {
t.Errorf("%s = %q, want 2/2", backendsHeader, h)
}
}
// With the feature off the name is an ordinary fact, so the query is fanned out
// untouched and nothing is synthesised.
func TestHandler_FactsQuerySourceFactDisabled(t *testing.T) {
a, b := sourceFactBackends(t)
cfg := testConfig(a.srv.URL, b.srv.URL, mergeFreshness)
cfg.SourceFactEnabled = false
srv := newTestServer(cfg)
rec := doGet(t, srv.Handler(), factsPath, `["=","name","`+defaultSourceFact+`"]`)
if _, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact); n != 0 {
t.Errorf("disabled feature synthesised %d records: %s", n, rec.Body.String())
}
}
// The synthetic set is fetched with the query's top-level certname conjuncts
// only, so a single-host lookup never pulls the whole estate.
func TestHandler_FactsQuerySourceFactNarrowsUpstream(t *testing.T) {
a, b := sourceFactBackends(t)
var mu sync.Mutex
var got []string
h3Only := map[*fakeBackend]string{
a: `[` + factEnv("h3", "osfamily", "RedHat", "prod") + `]`,
b: `[` + factEnv("h3", "osfamily", "Debian", "dev") + `]`,
}
for fb, body := range h3Only {
fb.handle = func(w http.ResponseWriter, r *http.Request) bool {
if r.URL.Path != factsPath {
return false
}
q := r.URL.Query().Get("query")
mu.Lock()
got = append(got, q)
mu.Unlock()
if q == `["=","certname","h3"]` {
_, _ = w.Write([]byte(body))
return true
}
return false
}
}
srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeFreshness))
query := `["and",["=","certname","h3"],["=","environment","dev"],["=","name","` + defaultSourceFact + `"]]`
rec := doGet(t, srv.Handler(), factsPath, query)
if vals, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact); n != 1 || vals["h3"] != hostB {
t.Fatalf("got %v (%d records), want h3=%s: %s", vals, n, hostB, rec.Body.String())
}
for _, q := range got {
if q == "" {
t.Fatalf("an upstream /facts fetch was unfiltered: %q", got)
}
}
if !slices.Contains(got, `["=","certname","h3"]`) {
t.Fatalf("synthetic fetch was not narrowed to the certname: %q", got)
}
}
// A backend lost between the two fetches is reported as missing.
func TestHandler_FactsQuerySourceFactCountsSecondFetch(t *testing.T) {
a, b := sourceFactBackends(t)
var calls atomic.Int32
b.handle = func(w http.ResponseWriter, r *http.Request) bool {
if r.URL.Path == factsPath && calls.Add(1) > 1 {
http.Error(w, "boom", http.StatusInternalServerError)
return true
}
return false
}
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness))
rec := doGet(t, srv.Handler(), factsPath, `["=","name","`+defaultSourceFact+`"]`)
if h := rec.Header().Get(backendsHeader); h != "1/2" {
t.Errorf("%s = %q, want 1/2", backendsHeader, h)
}
}
+18 -3
View File
@@ -72,8 +72,10 @@ func (s *Server) rawBackend(ctx context.Context, b Backend, path, rawQuery strin
} }
} }
// aliveRaw drops backends that errored or answered non-2xx, writing a 502 and // aliveRaw drops backends that errored or answered non-2xx, returning ok=false
// returning ok=false only when none is left. // when none is left. The reply it writes then is the same one the merged query
// handlers write, so /pdb/meta and /metrics answer a failure exactly as
// /pdb/query does: a unanimous upstream error replayed, anything else a 502.
func (s *Server) aliveRaw(w http.ResponseWriter, results []rawResult, path string) ([]rawResult, bool) { func (s *Server) aliveRaw(w http.ResponseWriter, results []rawResult, path string) ([]rawResult, bool) {
var alive []rawResult var alive []rawResult
for _, res := range results { for _, res := range results {
@@ -84,12 +86,25 @@ func (s *Server) aliveRaw(w http.ResponseWriter, results []rawResult, path strin
alive = append(alive, res) alive = append(alive, res)
} }
if len(alive) == 0 { if len(alive) == 0 {
http.Error(w, "all backends failed", http.StatusBadGateway) s.writeUpstreamError(w, peerOutcome(rawUpstreamErrors(results)))
return nil, false return nil, false
} }
return alive, true return alive, true
} }
// rawUpstreamErrors reduces a raw fan-out to one entry per backend, nil where
// the backend answered 2xx or never answered at all.
func rawUpstreamErrors(results []rawResult) []*upstreamError {
errs := make([]*upstreamError, len(results))
for i, res := range results {
if res.err != nil || res.ok() {
continue
}
errs[i] = newUpstreamError(res.status, res.contentType, res.body)
}
return errs
}
func (r rawResult) reason() string { func (r rawResult) reason() string {
if r.err != nil { if r.err != nil {
return r.err.Error() return r.err.Error()
+57 -2
View File
@@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"strings"
"testing" "testing"
) )
@@ -85,9 +86,9 @@ func TestMetaVersion_OneBackendDown(t *testing.T) {
func TestMetaVersion_AllBackendsDown(t *testing.T) { func TestMetaVersion_AllBackendsDown(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
a.fail = true a.dead = true
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
b.fail = true b.dead = true
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := metaGet(t, srv.Handler(), metaVersionPath); rec.Code != http.StatusBadGateway { if rec := metaGet(t, srv.Handler(), metaVersionPath); rec.Code != http.StatusBadGateway {
@@ -95,6 +96,60 @@ func TestMetaVersion_AllBackendsDown(t *testing.T) {
} }
} }
// /pdb/meta had its own copy of the gateway error, so a status both backends
// agreed on never reached the client. It answers failures the way the query
// routes do now.
func TestMeta_ReplaysUnanimousUpstreamStatus(t *testing.T) {
for _, path := range []string{metaVersionPath, metaServerTimePath} {
for _, status := range []int{http.StatusNotFound, http.StatusForbidden, http.StatusInternalServerError} {
t.Run(path+"/"+http.StatusText(status), func(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = status, "upstream said no"
b.reject, b.rejectBody = status, "upstream said no"
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), path)
if rec.Code != status {
t.Fatalf("status %d, want the upstream %d: %s", rec.Code, status, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "upstream said no") {
t.Errorf("body = %q, want the upstream explanation", rec.Body.String())
}
})
}
}
}
func TestMeta_DisagreeingStatusesStay502(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusNotFound, "gone"
b.reject, b.rejectBody = http.StatusInternalServerError, "boom"
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := metaGet(t, srv.Handler(), metaVersionPath); rec.Code != http.StatusBadGateway {
t.Errorf("status %d, want 502 when backends disagree", rec.Code)
}
}
// A replayed body must not name a backend, on /pdb/meta as anywhere else.
func TestMeta_ReplayedBodyIsRedacted(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusInternalServerError, "upstream "+a.srv.URL+" blew up"
b.reject, b.rejectBody = http.StatusInternalServerError, "upstream "+b.srv.URL+" blew up"
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), metaVersionPath)
if rec.Code != http.StatusInternalServerError {
t.Fatalf("status %d, want the upstream 500", rec.Code)
}
if strings.Contains(rec.Body.String(), hostOf(t, a.srv.URL)) {
t.Errorf("replayed body names a backend: %q", rec.Body.String())
}
}
func TestMetaServerTime_FirstReachableBackend(t *testing.T) { func TestMetaServerTime_FirstReachableBackend(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
a.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T01:00:00.000Z"}` a.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T01:00:00.000Z"}`
+7 -1
View File
@@ -65,7 +65,13 @@ func (s *Server) handleMetrics(w http.ResponseWriter, r *http.Request) {
writeRaw(w, alive[0]) writeRaw(w, alive[0])
return return
} }
w.Header().Set("Content-Type", "application/json") // The merged body is the backends' own payload reshaped, so it keeps their
// content type rather than announcing a different one.
contentType := alive[0].contentType
if contentType == "" {
contentType = jsonContentType
}
w.Header().Set("Content-Type", contentType)
_, _ = w.Write(body) _, _ = w.Write(body)
} }
+54 -2
View File
@@ -6,6 +6,7 @@ import (
"net/http/httptest" "net/http/httptest"
"reflect" "reflect"
"strconv" "strconv"
"strings"
"testing" "testing"
) )
@@ -229,9 +230,9 @@ func TestMetrics_OneBackendDown(t *testing.T) {
func TestMetrics_AllBackendsDown(t *testing.T) { func TestMetrics_AllBackendsDown(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
a.fail = true a.dead = true
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
b.fail = true b.dead = true
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := metaGet(t, srv.Handler(), numNodesEscaped); rec.Code != http.StatusBadGateway { if rec := metaGet(t, srv.Handler(), numNodesEscaped); rec.Code != http.StatusBadGateway {
@@ -239,6 +240,57 @@ func TestMetrics_AllBackendsDown(t *testing.T) {
} }
} }
// /metrics/v1/mbeans is a unanimous 404 on this estate and /metrics/v2 a
// unanimous 403; both used to come back as 502.
func TestMetrics_ReplaysUnanimousUpstreamStatus(t *testing.T) {
for _, status := range []int{http.StatusNotFound, http.StatusForbidden, http.StatusInternalServerError} {
t.Run(http.StatusText(status), func(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = status, "Forbidden request: /metrics/v2/list (method :get)."
b.reject, b.rejectBody = status, "Forbidden request: /metrics/v2/list (method :get)."
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), numNodesEscaped)
if rec.Code != status {
t.Fatalf("status %d, want the upstream %d: %s", rec.Code, status, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "Forbidden request") {
t.Errorf("body = %q, want the upstream explanation", rec.Body.String())
}
})
}
}
func TestMetrics_DisagreeingStatusesStay502(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusNotFound, "gone"
b.reject, b.rejectBody = http.StatusForbidden, "denied"
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := metaGet(t, srv.Handler(), numNodesEscaped); rec.Code != http.StatusBadGateway {
t.Errorf("status %d, want 502 when backends disagree", rec.Code)
}
}
// One backend refusing is not the estate's answer, so the other still serves.
func TestMetrics_OneRefusalStillServesTheOther(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusForbidden, "denied"
b.bodies[numNodesPath] = `{"request":{},"value":{"Value":7},"status":200}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), numNodesEscaped)
if rec.Code != http.StatusOK {
t.Fatalf("status %d, want the surviving backend's 200: %s", rec.Code, rec.Body.String())
}
if got := metricValue(t, rec.Body.Bytes())["Value"]; got != float64(7) {
t.Errorf("Value = %v, want 7", got)
}
}
func TestMetrics_RejectsNonGET(t *testing.T) { func TestMetrics_RejectsNonGET(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
+2
View File
@@ -21,6 +21,7 @@ var aggregateProbes = map[string]string{
eventCountsPath: eventCountsPath, eventCountsPath: eventCountsPath,
aggregateEventCountsPath: aggregateEventCountsPath, aggregateEventCountsPath: aggregateEventCountsPath,
reportsPath + "/<hash>/<sub>": reportsPath + "/abc123/events", reportsPath + "/<hash>/<sub>": reportsPath + "/abc123/events",
certnameRouteName: nodesPath + "/h1/facts",
} }
func TestQueryRoutes_EveryRouteIsProbed(t *testing.T) { func TestQueryRoutes_EveryRouteIsProbed(t *testing.T) {
@@ -107,6 +108,7 @@ func TestQueryRoutes_UnsummedRoutesAreTheKnownOnes(t *testing.T) {
eventCountsPath, eventCountsPath,
factNamesPath, factNamesPath,
reportsPath + "/<hash>/<sub>", reportsPath + "/<hash>/<sub>",
certnameRouteName,
} }
var got []string var got []string
for _, rt := range queryRoutes { for _, rt := range queryRoutes {
+142 -101
View File
@@ -9,6 +9,7 @@ import (
"log" "log"
"net/http" "net/http"
"net/url" "net/url"
"slices"
"strconv" "strconv"
"strings" "strings"
"sync" "sync"
@@ -32,11 +33,6 @@ const (
// PuppetDB only sends this when the request carries include_total=true. // PuppetDB only sends this when the request carries include_total=true.
recordsHeader = "X-Records" recordsHeader = "X-Records"
// Set by pdbmux, not by PuppetDB: how a cache-backed response was answered
// and how old the served copy is.
cacheStatusHeader = "X-Cache"
ageHeader = "Age"
// Set by pdbmux: "<contributed>/<configured>" backends behind a merged response. // Set by pdbmux: "<contributed>/<configured>" backends behind a merged response.
backendsHeader = "X-Backends" backendsHeader = "X-Backends"
) )
@@ -50,13 +46,21 @@ type backendResult struct {
var errAllBackendsFailed = errors.New("all backends failed") var errAllBackendsFailed = errors.New("all backends failed")
// jsonContentType is the spelling openvoxdb puts on the wire for a query
// response, charset and all. Its own reader treats the space after the
// semicolon as insignificant (http.clj:55-66, simple-utf8-ctype?), but a client
// comparing the raw header must see no difference between pdbmux and a PuppetDB.
const jsonContentType = "application/json;charset=utf-8"
type Server struct { type Server struct {
cfg Config cfg Config
// hosts maps backend name to the hostname the source fact reports.
hosts map[string]string
client *http.Client client *http.Client
log *log.Logger log *log.Logger
// factsCache is nil when caching is disabled; cacheFor hands out a noop then. // nodeCache is nil when caching is disabled; cacheFor hands out a noop then.
factsCache Cache nodeCache Cache
flights flightGroup flights flightGroup
stale staleTracker stale staleTracker
@@ -64,7 +68,8 @@ type Server struct {
health *prober health *prober
partial partialTracker partial partialTracker
// now is shared with the cache's clock so Age matches the stored timestamp. // now is shared with the cache's clock so a stale fallback is reported
// against the same timestamps the cache stamps entries with.
now func() time.Time now func() time.Time
// freshness cache (freshness merge only). // freshness cache (freshness merge only).
@@ -80,9 +85,13 @@ func NewServer(cfg Config, logger *log.Logger) *Server {
client: &http.Client{Timeout: cfg.Timeout}, client: &http.Client{Timeout: cfg.Timeout},
log: logger, log: logger,
now: time.Now, now: time.Now,
hosts: map[string]string{},
}
for _, b := range cfg.Backends {
s.hosts[b.Name] = backendHost(b.URL)
} }
if cfg.cacheEnabled() { if cfg.cacheEnabled() {
s.factsCache = newMemoryCache(cfg.FactsTTL, cfg.CacheBytes) s.nodeCache = newMemoryCache(cfg.FactsTTL, cfg.CacheBytes)
} }
s.health = newProber(cfg, logger) s.health = newProber(cfg, logger)
return s return s
@@ -95,20 +104,21 @@ func (s *Server) StartProbes(ctx context.Context) { s.health.Start(ctx) }
// StopProbes stops the probing goroutines and waits for them to exit. // StopProbes stops the probing goroutines and waits for them to exit.
func (s *Server) StopProbes() { s.health.Stop() } func (s *Server) StopProbes() { s.health.Stop() }
// cacheFor picks the cache backing a request. The merged /nodes and fact record // cacheFor picks the cache backing a request. Only the merged /nodes record set
// sets — /facts, /facts/<name>[/<value>] and /fact-names — share the in-memory // is cached: its report columns move once per node run, while fact answers must
// cache; every other path is uncached until the reports cache lands, and a new // be as current as a backend's own, so no fact-serving path is cached here.
// backend is a case here rather than a change to any handler. // Every other path is uncached until the reports cache lands, and a new backend
// is a case here rather than a change to any handler.
func (s *Server) cacheFor(path string, params url.Values) (Cache, bool) { func (s *Server) cacheFor(path string, params url.Values) (Cache, bool) {
switch { switch path {
case path == factsPath, path == nodesPath, path == factNamesPath, isFactsSubPath(path): case nodesPath:
// An aggregate row is a combined count, not the merged record set the // An aggregate row is a combined count, not the merged record set the
// cache was built for, so it stays on the live path. // cache was built for, so it stays on the live path.
if spec, err := parseAggregate(params.Get("query")); spec != nil || err != nil { if spec, err := parseAggregate(params.Get("query")); spec != nil || err != nil {
return noopCache{}, false return noopCache{}, false
} }
if s.factsCache != nil { if s.nodeCache != nil {
return s.factsCache, true return s.nodeCache, true
} }
} }
return noopCache{}, false return noopCache{}, false
@@ -177,6 +187,12 @@ var queryRoutes = []route{
serve: (*Server).serveFirstHolder, serve: (*Server).serveFirstHolder,
unsummed: "one backend holds the report, so nothing is merged across backends", unsummed: "one backend holds the report, so nothing is merged across backends",
}, },
{
name: certnameRouteName,
matches: isCertnameRoute,
serve: (*Server).serveByOwner,
unsummed: "one certname's own data, from the backend that owns it; nothing is merged across backends",
},
} }
// unmergedRoute answers every path no merged route claims. // unmergedRoute answers every path no merged route claims.
@@ -378,7 +394,65 @@ func (s *Server) serveUnion(w http.ResponseWriter, r *http.Request, path string,
} }
func (s *Server) serveFacts(w http.ResponseWriter, r *http.Request) { func (s *Server) serveFacts(w http.ResponseWriter, r *http.Request) {
s.serveMerged(w, r, factsPath, s.mergeFactsResponse(r)) query := r.URL.Query().Get("query")
inject := s.newSourceInjector(query, true)
if ast := inject.sourceQuery(query); ast != nil {
s.serveSourceFactQuery(w, r, inject, ast)
return
}
s.serveMerged(w, r, factsPath, s.mergeFactsWith(inject))
}
// serveSourceFactQuery answers a /facts query that selects the owned fact by
// name. The query still goes upstream for the real facts it selects; the
// synthetic records come from the same merge serveSourceFact uses, fetched for
// the query's certname scope, and are filtered by the query locally, since no
// backend holds them.
func (s *Server) serveSourceFactQuery(w http.ResponseWriter, r *http.Request, inject *sourceInjector, ast []json.RawMessage) {
params := queryParams(r.URL.Query().Get("query"))
s.serveCached(w, r, factsPath, params, func(ctx context.Context) (cachedResponse, error) {
alive, err := s.aliveResults(ctx, factsPath, params)
if err != nil {
return cachedResponse{}, err
}
recs := s.mergeFactsWith(inject)(alive)
synth, scoped, err := s.sourceFacts(ctx, s.newSourceInjector("", true), queryParams(certnameScope(ast)))
if err != nil {
return cachedResponse{}, err
}
for _, raw := range synth {
var f factFields
if json.Unmarshal(raw, &f) != nil {
continue
}
if match, _ := matchFact(ast, f); match {
recs = append(recs, raw)
}
}
resp := cachedResponse{Body: encodeRecords(recs), Records: -1}
s.countBackends(&resp, aliveInBoth(alive, scoped))
return resp, nil
})
}
func aliveInBoth(a, b []backendResult) []backendResult {
var out []backendResult
for _, x := range a {
if slices.ContainsFunc(b, func(y backendResult) bool { return y.name == x.name }) {
out = append(out, x)
}
}
return out
}
// sourceFacts is the owned fact's synthetic record set for the nodes params
// selects, taken from the /facts merge so owner and environment match it.
func (s *Server) sourceFacts(ctx context.Context, inject *sourceInjector, params url.Values) ([]json.RawMessage, []backendResult, error) {
alive, err := s.aliveResults(ctx, factsPath, params)
if err != nil {
return nil, nil, err
}
return sourceFactRecords(s.mergeFactsWith(inject)(alive), inject.name, "", false), alive, nil
} }
// Both path forms are the facts entity with a name (and value) constraint ANDed // Both path forms are the facts entity with a name (and value) constraint ANDed
@@ -408,17 +482,16 @@ func (s *Server) serveFactsByName(w http.ResponseWriter, r *http.Request) {
// value would be a fresh cache key, a fresh flight and a fresh whole-estate // value would be a fresh cache key, a fresh flight and a fresh whole-estate
// fan-out. // fan-out.
func (s *Server) serveSourceFact(w http.ResponseWriter, r *http.Request, inject *sourceInjector, value string, valued bool) { func (s *Server) serveSourceFact(w http.ResponseWriter, r *http.Request, inject *sourceInjector, value string, valued bool) {
// The synthetic record's value is always a backend name, so any other value // The synthetic record's value is always a backend host, so any other value
// matches zero records. The response is complete rather than degraded, so it // matches zero records. The response is complete rather than degraded, so it
// reports every configured backend. // reports every configured backend.
if valued && !s.hasBackend(value) { if valued && !s.hasBackendHost(value) {
n := len(s.cfg.Backends) n := len(s.cfg.Backends)
writeCached(w, cachedResponse{Records: -1, Backends: n, Configured: n}) writeCached(w, cachedResponse{Records: -1, Backends: n, Configured: n})
return return
} }
params := queryParams(r.URL.Query().Get("query")) params := queryParams(r.URL.Query().Get("query"))
merge := s.mergeFactsWith(inject)
var filter recordFilter var filter recordFilter
if valued { if valued {
filter = func(recs []json.RawMessage) []json.RawMessage { filter = func(recs []json.RawMessage) []json.RawMessage {
@@ -428,20 +501,19 @@ func (s *Server) serveSourceFact(w http.ResponseWriter, r *http.Request, inject
// The stored set is the whole owned-fact record set, so every value of it // The stored set is the whole owned-fact record set, so every value of it
// keys, and waits on, the same fetch. // keys, and waits on, the same fetch.
s.serveFiltered(w, r, factsPath+"/"+inject.name, params, filter, func(ctx context.Context) (cachedResponse, error) { s.serveFiltered(w, r, factsPath+"/"+inject.name, params, filter, func(ctx context.Context) (cachedResponse, error) {
alive, err := s.aliveResults(ctx, factsPath, params) recs, alive, err := s.sourceFacts(ctx, inject, params)
if err != nil { if err != nil {
return cachedResponse{}, err return cachedResponse{}, err
} }
recs := sourceFactRecords(merge(alive), inject.name, "", false)
resp := cachedResponse{Body: encodeRecords(recs), Records: -1} resp := cachedResponse{Body: encodeRecords(recs), Records: -1}
s.countBackends(&resp, alive) s.countBackends(&resp, alive)
return resp, nil return resp, nil
}) })
} }
func (s *Server) hasBackend(name string) bool { func (s *Server) hasBackendHost(host string) bool {
for _, b := range s.cfg.Backends { for _, h := range s.hosts {
if b.Name == name { if h == host {
return true return true
} }
} }
@@ -521,11 +593,9 @@ func (s *Server) serveFirstHolder(w http.ResponseWriter, r *http.Request) {
alive = append(alive, res) alive = append(alive, res)
} }
if len(alive) == 0 { if len(alive) == 0 {
if ue := unanimousClientError(results); ue != nil { // Unanimity is the whole answer here too: every backend saying 404 means
s.writeUpstreamError(w, ue) // nobody holds the report, while one silent backend leaves that unknown.
return s.writeUpstreamError(w, peerOutcome(backendUpstreamErrors(results)))
}
http.Error(w, "no backend holds this report", http.StatusNotFound)
return return
} }
for _, res := range alive { for _, res := range alive {
@@ -537,8 +607,8 @@ func (s *Server) serveFirstHolder(w http.ResponseWriter, r *http.Request) {
writeJSON(w, nil) writeJSON(w, nil)
} }
// Returns an *upstreamError when every backend refused the query the same way, // Returns an *upstreamError when every backend answered the same way, and
// and errAllBackendsFailed when every backend failed for any other reason. // errAllBackendsFailed when every backend failed for any other reason.
func (s *Server) aliveResults(ctx context.Context, path string, params url.Values) ([]backendResult, error) { func (s *Server) aliveResults(ctx context.Context, path string, params url.Values) ([]backendResult, error) {
results := s.fanOut(ctx, path, params) results := s.fanOut(ctx, path, params)
@@ -548,13 +618,17 @@ func (s *Server) aliveResults(ctx context.Context, path string, params url.Value
alive = append(alive, res) alive = append(alive, res)
} }
} }
var unanimous *upstreamError
if len(alive) == 0 { if len(alive) == 0 {
// A query every backend refuses identically is the client's mistake, not an unanimous = unanimousUpstreamError(backendUpstreamErrors(results))
// outage, so it is neither logged as one nor counted as degraded service.
if ue := unanimousClientError(results); ue != nil {
s.log.Printf("info: every backend refused %s: %v", path, ue)
return nil, ue
} }
// A query every backend refuses identically is the client's mistake, not an
// outage, so it is neither logged as one nor counted as degraded service. A
// unanimous 5xx is replayed just as faithfully, but it is the backends
// reporting their own fault, so it still counts against service health.
if clientRefusal(unanimous) {
s.log.Printf("info: every backend refused %s: %v", path, unanimous)
return nil, unanimous
} }
for _, res := range results { for _, res := range results {
if res.err != nil { if res.err != nil {
@@ -563,6 +637,9 @@ func (s *Server) aliveResults(ctx context.Context, path string, params url.Value
} }
s.partial.record(len(alive), len(s.cfg.Backends), s.now()) s.partial.record(len(alive), len(s.cfg.Backends), s.now())
if len(alive) == 0 { if len(alive) == 0 {
if unanimous != nil {
return nil, unanimous
}
return nil, errAllBackendsFailed return nil, errAllBackendsFailed
} }
return alive, nil return alive, nil
@@ -578,10 +655,9 @@ type cachedResponse struct {
Configured int `json:"configured"` // backends configured at build time Configured int `json:"configured"` // backends configured at build time
} }
// recordFilter narrows a response's records after it has been built or read back // recordFilter narrows a response's records after it has been built, so a route
// from the cache, so requests differing only in the filter share one stored entry // that answers from another path's merge can pin part of it per request. It
// and one fan-out. It leaves Records alone, so it only suits responses that set // leaves Records alone, so it only suits responses that set no X-Records.
// no X-Records.
type recordFilter func([]json.RawMessage) []json.RawMessage type recordFilter func([]json.RawMessage) []json.RawMessage
func (f recordFilter) apply(resp cachedResponse) cachedResponse { func (f recordFilter) apply(resp cachedResponse) cachedResponse {
@@ -627,7 +703,7 @@ func (s *Server) serveFiltered(w http.ResponseWriter, r *http.Request, path stri
s.log.Printf("warning: cache lookup for %s failed: %v", key, err) s.log.Printf("warning: cache lookup for %s failed: %v", key, err)
case status == CacheFresh: case status == CacheFresh:
s.stale.markFresh() s.stale.markFresh()
s.writeStored(w, ent, CacheFresh, filter) s.writeStored(w, ent, filter)
return return
case status == CacheStale: case status == CacheStale:
stale = &ent stale = &ent
@@ -666,18 +742,17 @@ func (s *Server) serveFiltered(w http.ResponseWriter, r *http.Request, path stri
} }
// A refused query is answered, not degraded, so stale records are no reply // A refused query is answered, not degraded, so stale records are no reply
// to it: the client has to see why the query was rejected. // to it: the client has to see why the query was rejected.
if stale != nil && !errors.As(err, new(*upstreamError)) { if stale != nil && !clientRefusal(err) {
s.stale.markStale(s.now()) s.stale.markStale(s.now())
s.log.Printf("warning: serving stale %s from cache (stored %s): %v", s.log.Printf("warning: serving stale %s from cache (stored %s): %v",
path, stale.StoredAt.UTC().Format(time.RFC3339), err) path, stale.StoredAt.UTC().Format(time.RFC3339), err)
s.writeStored(w, *stale, CacheStale, filter) s.writeStored(w, *stale, filter)
return return
} }
s.writeUpstreamError(w, err) s.writeUpstreamError(w, err)
return return
} }
s.stale.markFresh() s.stale.markFresh()
s.setCacheHeaders(w, CacheMiss, time.Time{})
writeCached(w, filter.apply(resp)) writeCached(w, filter.apply(resp))
} }
@@ -690,41 +765,21 @@ func (s *Server) flightTimeout() time.Duration {
return defaultTimeout return defaultTimeout
} }
func (s *Server) writeStored(w http.ResponseWriter, ent CacheEntry, status CacheStatus, filter recordFilter) { // writeStored answers from a cache entry. Whether a response came from the cache
// is deliberately invisible to the client: PuppetDB sets no X-Cache or Age, so
// neither does pdbmux, and /healthz carries the cache state instead.
func (s *Server) writeStored(w http.ResponseWriter, ent CacheEntry, filter recordFilter) {
var resp cachedResponse var resp cachedResponse
if err := json.Unmarshal(ent.Body, &resp); err != nil { if err := json.Unmarshal(ent.Body, &resp); err != nil {
s.log.Printf("warning: unreadable cache entry: %v", err) s.log.Printf("warning: unreadable cache entry: %v", err)
http.Error(w, "unreadable cache entry", http.StatusBadGateway) http.Error(w, "unreadable cache entry", http.StatusBadGateway)
return return
} }
s.setCacheHeaders(w, status, ent.StoredAt)
writeCached(w, filter.apply(resp)) writeCached(w, filter.apply(resp))
} }
// setCacheHeaders labels a response from a cache-backed path: X-Cache is
// hit/stale/miss and Age is whole seconds since the served copy was stored (0
// for a response built by this request). It reads the same clock the cache
// stamps entries with, so the two never disagree.
func (s *Server) setCacheHeaders(w http.ResponseWriter, status CacheStatus, storedAt time.Time) {
label := "miss"
switch status {
case CacheFresh:
label = "hit"
case CacheStale:
label = "stale"
}
age := 0
if !storedAt.IsZero() {
if secs := int(s.now().Sub(storedAt).Seconds()); secs > 0 {
age = secs
}
}
w.Header().Set(cacheStatusHeader, label)
w.Header().Set(ageHeader, strconv.Itoa(age))
}
func writeCached(w http.ResponseWriter, resp cachedResponse) { func writeCached(w http.ResponseWriter, resp cachedResponse) {
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", jsonContentType)
if resp.Records >= 0 { if resp.Records >= 0 {
w.Header().Set(recordsHeader, strconv.Itoa(resp.Records)) w.Header().Set(recordsHeader, strconv.Itoa(resp.Records))
} }
@@ -774,10 +829,6 @@ func (s *Server) mergeNodesResponse(r *http.Request) func([]backendResult) []jso
} }
} }
func (s *Server) mergeFactsResponse(r *http.Request) func([]backendResult) []json.RawMessage {
return s.mergeFactsWith(s.newSourceInjector(r.URL.Query().Get("query"), true))
}
// The path segment of /facts/<name> is the same outer `name` constraint the // The path segment of /facts/<name> is the same outer `name` constraint the
// query gate already rules injection out on, so nothing is synthesised here: // query gate already rules injection out on, so nothing is synthesised here:
// the source fact's own path is diverted to serveSourceFact before this. // the source fact's own path is diverted to serveSourceFact before this.
@@ -905,13 +956,21 @@ func (s *Server) queryBackend(ctx context.Context, b Backend, path string, param
return recs, total, err return recs, total, err
} }
// The record shape is unknown, so a union would be guesswork: the first 2xx wins and the first error response is replayed when none succeeds. // The record shape is unknown, so a union would be guesswork: the first 2xx wins, and every backend was asked the same question, so only a reply they all gave is replayed.
func (s *Server) proxyUnmerged(w http.ResponseWriter, r *http.Request) { func (s *Server) proxyUnmerged(w http.ResponseWriter, r *http.Request) {
var fallback *bufferedResponse s.proxyOrdered(w, r, s.cfg.Backends, askPeers)
for _, b := range s.cfg.Backends { }
// proxyOrdered asks backends in the given order, which is what decides the
// answer when more than one of them holds the path. order says why they are in
// that order, which is what decides whose reply is served when none answers 2xx.
func (s *Server) proxyOrdered(w http.ResponseWriter, r *http.Request, backends []Backend, order askOrder) {
replies := make([]*upstreamError, 0, len(backends))
for _, b := range backends {
resp, err := s.passThrough(r, b) resp, err := s.passThrough(r, b)
if err != nil { if err != nil {
s.log.Printf("warning: backend %q pass-through failed for %s: %v", b.Name, r.URL.Path, err) s.log.Printf("warning: backend %q pass-through failed for %s: %v", b.Name, r.URL.Path, err)
replies = append(replies, nil)
continue continue
} }
if resp.StatusCode >= 200 && resp.StatusCode < 300 { if resp.StatusCode >= 200 && resp.StatusCode < 300 {
@@ -923,27 +982,9 @@ func (s *Server) proxyUnmerged(w http.ResponseWriter, r *http.Request) {
} }
body, _ := io.ReadAll(resp.Body) body, _ := io.ReadAll(resp.Body)
_ = resp.Body.Close() _ = resp.Body.Close()
if fallback == nil { replies = append(replies, newUpstreamError(resp.StatusCode, resp.Header.Get("Content-Type"), body))
fallback = &bufferedResponse{
status: resp.StatusCode,
contentType: resp.Header.Get("Content-Type"),
body: body,
} }
} s.writeUpstreamError(w, order.outcome(replies))
}
if fallback == nil {
http.Error(w, "all backends failed", http.StatusBadGateway)
return
}
setContentType(w, fallback.contentType)
w.WriteHeader(fallback.status)
_, _ = w.Write(s.redactBackends(fallback.body))
}
type bufferedResponse struct {
status int
contentType string
body []byte
} }
func (s *Server) passThrough(r *http.Request, b Backend) (*http.Response, error) { func (s *Server) passThrough(r *http.Request, b Backend) (*http.Response, error) {
@@ -1026,8 +1067,8 @@ type cacheHealth struct {
func (s *Server) cacheHealth() cacheHealth { func (s *Server) cacheHealth() cacheHealth {
stats := CacheStats{Backend: "none"} stats := CacheStats{Backend: "none"}
ttl := time.Duration(0) ttl := time.Duration(0)
if s.factsCache != nil { if s.nodeCache != nil {
stats = s.factsCache.Stats() stats = s.nodeCache.Stats()
ttl = s.cfg.FactsTTL ttl = s.cfg.FactsTTL
} }
serving, served, last := s.stale.snapshot() serving, served, last := s.stale.snapshot()
@@ -1095,7 +1136,7 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
} }
func writeJSON(w http.ResponseWriter, recs []json.RawMessage) { func writeJSON(w http.ResponseWriter, recs []json.RawMessage) {
w.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", jsonContentType)
if recs == nil { if recs == nil {
recs = []json.RawMessage{} recs = []json.RawMessage{}
} }
+35 -2
View File
@@ -4,6 +4,7 @@ import (
"encoding/json" "encoding/json"
"io" "io"
"log" "log"
"net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url" "net/url"
@@ -29,7 +30,10 @@ type fakeBackend struct {
// totals is the X-Records count advertised per path when the request asks // totals is the X-Records count advertised per path when the request asks
// for include_total. // for include_total.
totals map[string]int totals map[string]int
fail bool // return 500 for everything fail bool // answer 500 for everything, as a backend admitting a fault
// dead aborts the connection instead of answering, standing in for a backend
// that is unreachable rather than one that replies badly.
dead bool
delay time.Duration // artificial latency delay time.Duration // artificial latency
// reject answers every request with this status and rejectBody, standing in // reject answers every request with this status and rejectBody, standing in
// for a PuppetDB refusing a query it cannot answer. // for a PuppetDB refusing a query it cannot answer.
@@ -41,6 +45,8 @@ type fakeBackend struct {
// rawPaths records the still-escaped request paths, so tests can assert an // rawPaths records the still-escaped request paths, so tests can assert an
// MBean name's percent-encoding survived the proxy. // MBean name's percent-encoding survived the proxy.
rawPaths []string rawPaths []string
// handle, when set, runs first and answers the request itself by returning true.
handle func(w http.ResponseWriter, r *http.Request) bool
} }
func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend { func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend {
@@ -60,6 +66,12 @@ func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend {
fb.gotParams[r.URL.Path] = r.URL.Query() fb.gotParams[r.URL.Path] = r.URL.Query()
fb.rawPaths = append(fb.rawPaths, r.URL.EscapedPath()) fb.rawPaths = append(fb.rawPaths, r.URL.EscapedPath())
fb.mu.Unlock() fb.mu.Unlock()
if fb.handle != nil && fb.handle(w, r) {
return
}
if fb.dead {
panic(http.ErrAbortHandler)
}
if fb.fail { if fb.fail {
http.Error(w, "boom", http.StatusInternalServerError) http.Error(w, "boom", http.StatusInternalServerError)
return return
@@ -152,6 +164,27 @@ func truncate(t *testing.T, body, limit string) string {
return string(out) return string(out)
} }
// Test backends are reached as hostA/hostB, since the source fact's value is the
// backend host and both httptest servers listen on 127.0.0.1.
const (
hostA = "a.localhost"
hostB = "b.localhost"
)
func withHost(rawURL, host string) string {
u, err := url.Parse(rawURL)
if err != nil || u.Port() == "" {
return rawURL
}
u.Host = net.JoinHostPort(host, u.Port())
return u.String()
}
// hostedConfig is testConfig with the backends reached as hostA and hostB.
func hostedConfig(aURL, bURL, merge string) Config {
return testConfig(withHost(aURL, hostA), withHost(bURL, hostB), merge)
}
func testConfig(aURL, bURL, merge string) Config { func testConfig(aURL, bURL, merge string) Config {
return Config{ return Config{
Listen: ":0", Listen: ":0",
@@ -285,7 +318,7 @@ func TestHandler_OneBackendDown(t *testing.T) {
func TestHandler_BothBackendsDown(t *testing.T) { func TestHandler_BothBackendsDown(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
a.fail, b.fail = true, true a.dead, b.dead = true, true
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), nodesPath, "") rec := doGet(t, srv.Handler(), nodesPath, "")
+178 -3
View File
@@ -3,6 +3,8 @@ package main
import ( import (
"encoding/json" "encoding/json"
"log" "log"
"regexp"
"slices"
"strings" "strings"
) )
@@ -11,6 +13,8 @@ import (
// callers need no branch. // callers need no branch.
type sourceInjector struct { type sourceInjector struct {
name string name string
// hosts maps backend name to the value emitted.
hosts map[string]string
// inject is false when the query shape rules synthesis out. Suppression of an // inject is false when the query shape rules synthesis out. Suppression of an
// upstream fact of the same name does not depend on it. // upstream fact of the same name does not depend on it.
inject bool inject bool
@@ -23,7 +27,7 @@ func (s *Server) newSourceInjector(query string, factEntity bool) *sourceInjecto
if !s.cfg.SourceFactEnabled || s.cfg.SourceFact == "" { if !s.cfg.SourceFactEnabled || s.cfg.SourceFact == "" {
return nil return nil
} }
return &sourceInjector{name: s.cfg.SourceFact, inject: injectable(query, factEntity)} return &sourceInjector{name: s.cfg.SourceFact, hosts: s.hosts, inject: injectable(query, factEntity)}
} }
// claims reports whether an upstream record carries the name pdbmux owns, and is // claims reports whether an upstream record carries the name pdbmux owns, and is
@@ -68,7 +72,7 @@ func (si *sourceInjector) factRecord(certname, backend, environment string) json
Environment string `json:"environment"` Environment string `json:"environment"`
Name string `json:"name"` Name string `json:"name"`
Value string `json:"value"` Value string `json:"value"`
}{Certname: certname, Environment: environment, Name: si.name, Value: backend}) }{Certname: certname, Environment: environment, Name: si.name, Value: si.hosts[backend]})
if err != nil { if err != nil {
return nil return nil
} }
@@ -85,7 +89,7 @@ func (si *sourceInjector) stamp(raw json.RawMessage, backend string) json.RawMes
if json.Unmarshal(raw, &obj) != nil || obj == nil { if json.Unmarshal(raw, &obj) != nil || obj == nil {
return raw return raw
} }
value, err := json.Marshal(backend) value, err := json.Marshal(si.hosts[backend])
if err != nil { if err != nil {
return raw return raw
} }
@@ -204,3 +208,174 @@ func constrainsField(parts []json.RawMessage, field string) bool {
var name string var name string
return json.Unmarshal(parts[1], &name) == nil && name == field return json.Unmarshal(parts[1], &name) == nil && name == field
} }
// sourceQuery returns the parsed /facts query when one of its name comparisons
// selects the owned fact and the whole query can be evaluated locally against a
// fact record; nil otherwise, which leaves the query on the gated path.
// ponytail: only a name match that is true for the owned fact selects it, so
// ["not",["=","name","osfamily"]] stays on the gated path and synthesises
// nothing; widen namesFact if a client needs negated selections.
func (si *sourceInjector) sourceQuery(query string) []json.RawMessage {
if si == nil {
return nil
}
var ast []json.RawMessage
if json.Unmarshal([]byte(query), &ast) != nil {
return nil
}
probe := factFields{Name: si.name}
if _, ok := matchFact(ast, probe); !ok || !namesFact(ast, probe) {
return nil
}
return ast
}
// factFields are the queryable columns of the facts entity (openvoxdb
// engine.clj facts-query). value is a string here: the synthetic value always is.
type factFields struct {
Certname string `json:"certname"`
Environment string `json:"environment"`
Name string `json:"name"`
Value string `json:"value"`
}
func (f factFields) column(name string) (string, bool) {
switch name {
case "certname":
return f.Certname, true
case "environment":
return f.Environment, true
case "name":
return f.Name, true
case "value":
return f.Value, true
}
return "", false
}
// namesFact reports whether a name comparison in the boolean skeleton matches f.
func namesFact(parts []json.RawMessage, f factFields) bool {
var op, field string
if len(parts) < 2 || json.Unmarshal(parts[0], &op) != nil {
return false
}
switch op {
case "and", "or", "not":
for _, p := range parts[1:] {
var sub []json.RawMessage
if json.Unmarshal(p, &sub) == nil && namesFact(sub, f) {
return true
}
}
return false
}
if json.Unmarshal(parts[1], &field) != nil || field != "name" {
return false
}
match, _ := matchFact(parts, f)
return match
}
// matchFact evaluates a facts query against one record with openvoxdb's
// semantics for and/or/not, =, ~ and in-array. ok is false for anything else,
// including subqueries, so a caller can refuse rather than guess.
func matchFact(parts []json.RawMessage, f factFields) (match, ok bool) {
var op string
if len(parts) == 0 || json.Unmarshal(parts[0], &op) != nil {
return false, false
}
switch op {
case "and", "or":
match = op == "and"
for _, p := range parts[1:] {
var sub []json.RawMessage
if json.Unmarshal(p, &sub) != nil {
return false, false
}
m, ok := matchFact(sub, f)
if !ok {
return false, false
}
if op == "and" {
match = match && m
} else {
match = match || m
}
}
return match, true
case "not":
var sub []json.RawMessage
if len(parts) != 2 || json.Unmarshal(parts[1], &sub) != nil {
return false, false
}
m, ok := matchFact(sub, f)
return !m, ok
}
var field string
if len(parts) != 3 || json.Unmarshal(parts[1], &field) != nil {
return false, false
}
got, known := f.column(field)
if !known {
return false, false
}
switch op {
case "=":
var want any
if json.Unmarshal(parts[2], &want) != nil {
return false, false
}
return want == any(got), true
case "~":
var pattern string
if json.Unmarshal(parts[2], &pattern) != nil {
return false, false
}
re, err := regexp.Compile(pattern)
if err != nil {
return false, false
}
return re.MatchString(got), true
case "in":
var arr []json.RawMessage
var tag string
var values []any
if json.Unmarshal(parts[2], &arr) != nil || len(arr) != 2 ||
json.Unmarshal(arr[0], &tag) != nil || tag != "array" ||
json.Unmarshal(arr[1], &values) != nil {
return false, false
}
return slices.Contains(values, any(got)), true
}
return false, false
}
// certnameScope returns the top-level and's certname comparisons as a query,
// or "" when there are none. Pushing them upstream keeps every record of a
// selected node on every backend, so the merge's ownership is unchanged; an
// environment comparison is not pushed because it can hide the owner's records
// when backends disagree on a node's environment.
func certnameScope(ast []json.RawMessage) string {
var op string
if len(ast) < 2 || json.Unmarshal(ast[0], &op) != nil || op != "and" {
return ""
}
var keep []string
for _, p := range ast[1:] {
var sub []json.RawMessage
var field string
if json.Unmarshal(p, &sub) != nil || len(sub) != 3 || json.Unmarshal(sub[1], &field) != nil || field != "certname" {
continue
}
if _, ok := matchFact(sub, factFields{}); ok {
keep = append(keep, string(p))
}
}
switch len(keep) {
case 0:
return ""
case 1:
return keep[0]
}
return `["and",` + strings.Join(keep, ",") + `]`
}
+126 -15
View File
@@ -77,7 +77,7 @@ func TestHandler_FactsSourceFollowsMergeOwner(t *testing.T) {
b := newFakeBackend(t, b := newFakeBackend(t,
`[`+node("h1", "2026-07-01T00:00:00Z")+`,`+node("h2", "2026-07-20T00:00:00Z")+`]`, `[`+node("h1", "2026-07-01T00:00:00Z")+`,`+node("h2", "2026-07-20T00:00:00Z")+`]`,
`[`+factEnv("h1", "role", "web-b", "staging")+`,`+factEnv("h2", "role", "db-b", "staging")+`]`) `[`+factEnv("h1", "role", "web-b", "staging")+`,`+factEnv("h2", "role", "db-b", "staging")+`]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeFreshness)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeFreshness))
rec := doGet(t, srv.Handler(), factsPath, "") rec := doGet(t, srv.Handler(), factsPath, "")
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
@@ -87,7 +87,7 @@ func TestHandler_FactsSourceFollowsMergeOwner(t *testing.T) {
if n != 2 { if n != 2 {
t.Fatalf("expected one %s record per certname, got %d: %s", defaultSourceFact, n, rec.Body.String()) t.Fatalf("expected one %s record per certname, got %d: %s", defaultSourceFact, n, rec.Body.String())
} }
if got["h1"] != "a" || got["h2"] != "b" { if got["h1"] != hostA || got["h2"] != hostB {
t.Errorf("provenance must name the backend that won the merge, got %v", got) t.Errorf("provenance must name the backend that won the merge, got %v", got)
} }
} }
@@ -127,14 +127,14 @@ func TestHandler_NodesSourceStamped(t *testing.T) {
a := newFakeBackend(t, a := newFakeBackend(t,
`[`+node("h1", "2026-07-01T00:00:00Z")+`,`+node("h2", "2026-07-20T00:00:00Z")+`]`, `[]`) `[`+node("h1", "2026-07-01T00:00:00Z")+`,`+node("h2", "2026-07-20T00:00:00Z")+`]`, `[]`)
b := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`) b := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), nodesPath, "") rec := doGet(t, srv.Handler(), nodesPath, "")
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String()) t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
} }
got := nodeSources(t, rec.Body.Bytes(), defaultSourceFact) got := nodeSources(t, rec.Body.Bytes(), defaultSourceFact)
if got["h1"] != "b" || got["h2"] != "a" { if got["h1"] != hostB || got["h2"] != hostA {
t.Errorf("node provenance = %v, want h1=b h2=a", got) t.Errorf("node provenance = %v, want h1=b h2=a", got)
} }
} }
@@ -186,13 +186,13 @@ func TestHandler_SourceFactNameOverride(t *testing.T) {
a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`,
`[`+fact("h1", "role", "web", "")+`]`) `[`+fact("h1", "role", "web", "")+`]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
cfg := testConfig(a.srv.URL, b.srv.URL, mergeStatic) cfg := hostedConfig(a.srv.URL, b.srv.URL, mergeStatic)
cfg.SourceFact = "origin_pdb" cfg.SourceFact = "origin_pdb"
srv := newTestServer(cfg) srv := newTestServer(cfg)
facts := doGet(t, srv.Handler(), factsPath, "") facts := doGet(t, srv.Handler(), factsPath, "")
got, n := sourceValues(t, facts.Body.Bytes(), "origin_pdb") got, n := sourceValues(t, facts.Body.Bytes(), "origin_pdb")
if n != 1 || got["h1"] != "a" { if n != 1 || got["h1"] != hostA {
t.Errorf("override name not honoured: %s", facts.Body.String()) t.Errorf("override name not honoured: %s", facts.Body.String())
} }
if _, n := sourceValues(t, facts.Body.Bytes(), defaultSourceFact); n != 0 { if _, n := sourceValues(t, facts.Body.Bytes(), defaultSourceFact); n != 0 {
@@ -200,7 +200,7 @@ func TestHandler_SourceFactNameOverride(t *testing.T) {
} }
nodes := doGet(t, srv.Handler(), nodesPath, "") nodes := doGet(t, srv.Handler(), nodesPath, "")
if got := nodeSources(t, nodes.Body.Bytes(), "origin_pdb"); got["h1"] != "a" { if got := nodeSources(t, nodes.Body.Bytes(), "origin_pdb"); got["h1"] != hostA {
t.Errorf("override name not honoured on /nodes: %v", got) t.Errorf("override name not honoured on /nodes: %v", got)
} }
} }
@@ -211,14 +211,14 @@ func TestHandler_UpstreamSourceFactOverridden(t *testing.T) {
a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`,
`[`+fact("h1", "role", "web", "")+`,`+fact("h1", defaultSourceFact, "stale-value", "")+`]`) `[`+fact("h1", "role", "web", "")+`,`+fact("h1", defaultSourceFact, "stale-value", "")+`]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), factsPath, "") rec := doGet(t, srv.Handler(), factsPath, "")
got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact) got, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact)
if n != 1 { if n != 1 {
t.Fatalf("expected exactly 1 %s record, got %d: %s", defaultSourceFact, n, rec.Body.String()) t.Fatalf("expected exactly 1 %s record, got %d: %s", defaultSourceFact, n, rec.Body.String())
} }
if got["h1"] != "a" { if got["h1"] != hostA {
t.Errorf("upstream value survived: %v", got) t.Errorf("upstream value survived: %v", got)
} }
} }
@@ -232,9 +232,10 @@ func TestHandler_UpstreamSourceFactSuppressedOnEveryGateState(t *testing.T) {
query string query string
want string // synthetic value, or "" when the gate blocks injection want string // synthetic value, or "" when the gate blocks injection
}{ }{
{"injection on", "", "a"}, {"injection on", "", hostA},
{"gated by extract", `["extract",["certname","name","value"],["=","certname","h1"]]`, ""}, {"gated by extract", `["extract",["certname","name","value"],["=","certname","h1"]]`, ""},
{"gated by name filter", `["=","name","` + defaultSourceFact + `"]`, ""}, {"gated by name filter", `["or",["=","name","role"],["=","name","os"]]`, ""},
{"selected by name filter", `["or",["=","name","role"],["=","name","` + defaultSourceFact + `"]]`, hostA},
{"gated by nested extract", `["and",["=","certname","h1"],["extract",["certname"]]]`, ""}, {"gated by nested extract", `["and",["=","certname","h1"],["extract",["certname"]]]`, ""},
} }
for _, tc := range tests { for _, tc := range tests {
@@ -242,7 +243,7 @@ func TestHandler_UpstreamSourceFactSuppressedOnEveryGateState(t *testing.T) {
a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`,
`[`+fact("h1", "role", "web", "")+`,`+fact("h1", defaultSourceFact, upstream, "")+`]`) `[`+fact("h1", "role", "web", "")+`,`+fact("h1", defaultSourceFact, upstream, "")+`]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(hostedConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), factsPath, tc.query) rec := doGet(t, srv.Handler(), factsPath, tc.query)
if rec.Code != http.StatusOK { if rec.Code != http.StatusOK {
@@ -391,7 +392,6 @@ func TestHandler_SourceNotInjectedWhenNameFiltered(t *testing.T) {
for _, q := range []string{ for _, q := range []string{
`["=","name","role"]`, `["=","name","role"]`,
`["and",["=","certname","h1"],["=","name","role"]]`, `["and",["=","certname","h1"],["=","name","role"]]`,
`["=","name","` + defaultSourceFact + `"]`,
} { } {
rec := doGet(t, srv.Handler(), factsPath, q) rec := doGet(t, srv.Handler(), factsPath, q)
if _, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact); n != 0 { if _, n := sourceValues(t, rec.Body.Bytes(), defaultSourceFact); n != 0 {
@@ -560,11 +560,122 @@ func TestMergeFacts_GatedSuppressesUpstream(t *testing.T) {
func TestMergeFacts_SourceOrderedAfterOwnersFacts(t *testing.T) { func TestMergeFacts_SourceOrderedAfterOwnersFacts(t *testing.T) {
a := recs(t, "a", fact("h1", "role", "web-a", ""), fact("h1", "kernel", "Linux", "")) a := recs(t, "a", fact("h1", "role", "web-a", ""), fact("h1", "kernel", "Linux", ""))
b := recs(t, "b", fact("h1", "role", "web-b", "")) b := recs(t, "b", fact("h1", "role", "web-b", ""))
merged := mergeFacts([]backendResult{a, b}, nil, &sourceInjector{name: defaultSourceFact, inject: true}) si := &sourceInjector{name: defaultSourceFact, hosts: map[string]string{"a": "a.example", "b": "b.example"}, inject: true}
merged := mergeFacts([]backendResult{a, b}, nil, si)
got := factValues(t, merged) got := factValues(t, merged)
want := []string{"h1:role=web-a", "h1:kernel=Linux", "h1:" + defaultSourceFact + "=a"} want := []string{"h1:role=web-a", "h1:kernel=Linux", "h1:" + defaultSourceFact + "=a.example"}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Errorf("merged = %v, want %v", got, want) t.Errorf("merged = %v, want %v", got, want)
} }
} }
func TestMatchFact(t *testing.T) {
f := factFields{Certname: "h1", Environment: "prod", Name: defaultSourceFact, Value: "a"}
tests := []struct {
query string
match, ok bool
}{
{`["=","name","` + defaultSourceFact + `"]`, true, true},
{`["=","name","osfamily"]`, false, true},
{`["=","value","a"]`, true, true},
{`["=","value",1]`, false, true},
{`["=","value",{"a":1}]`, false, true},
{`["~","value","^a$"]`, true, true},
{`["~","certname","^h"]`, true, true},
{`["not",["=","value","a"]]`, false, true},
{`["in","name",["array",["osfamily","` + defaultSourceFact + `"]]]`, true, true},
{`["in","name",["array",["osfamily"]]]`, false, true},
{`["and",["=","certname","h1"],["=","environment","prod"]]`, true, true},
{`["and",["=","certname","h1"],["=","environment","dev"]]`, false, true},
{`["or",["=","certname","h2"],["=","environment","prod"]]`, true, true},
{`["and"]`, true, true},
{`["or"]`, false, true},
{`["and",["=","name","` + defaultSourceFact + `"],["null?","value",false]]`, false, false},
{`["in","certname",["extract",["certname"],["select_nodes",["=","certname","h1"]]]]`, false, false},
{`["=","fact_contents","x"]`, false, false},
{`["~","value","("]`, false, false},
{`["<","value","b"]`, false, false},
{`["extract",["certname"],["=","name","x"]]`, false, false},
}
for _, tc := range tests {
var ast []json.RawMessage
if err := json.Unmarshal([]byte(tc.query), &ast); err != nil {
t.Fatal(err)
}
if m, ok := matchFact(ast, f); m != tc.match || ok != tc.ok {
t.Errorf("matchFact(%s) = %v, %v; want %v, %v", tc.query, m, ok, tc.match, tc.ok)
}
}
}
func TestSourceInjector_SourceQuery(t *testing.T) {
si := &sourceInjector{name: defaultSourceFact}
tests := []struct {
query string
want bool
}{
{`["=","name","` + defaultSourceFact + `"]`, true},
{`["and",["=","name","` + defaultSourceFact + `"]]`, true},
{`["and",["or",["=","name","ipaddress"],["=","name","` + defaultSourceFact + `"]]]`, true},
{`["and",["=","name","` + defaultSourceFact + `"],["not",["~","value","new"]]]`, true},
{`["in","name",["array",["ipaddress","` + defaultSourceFact + `"]]]`, true},
{`["~","name","^pdbmux"]`, true},
{`["=","name","ipaddress"]`, false},
{`["=","certname","h1"]`, false},
{`["=","value","` + defaultSourceFact + `"]`, false},
{`["and",["=","name","` + defaultSourceFact + `"],["in","certname",["extract",["certname"],["select_nodes",["=","certname","h1"]]]]]`, false},
{``, false},
{`not json`, false},
}
for _, tc := range tests {
if got := si.sourceQuery(tc.query) != nil; got != tc.want {
t.Errorf("sourceQuery(%s) = %v, want %v", tc.query, got, tc.want)
}
}
var off *sourceInjector
if off.sourceQuery(`["=","name","`+defaultSourceFact+`"]`) != nil {
t.Error("disabled injector selected a query")
}
}
// The value is the backend URL's hostname alone; the backend name is not used.
func TestSourceInjector_ValueIsBackendHostname(t *testing.T) {
cfg := testConfig("http://puppetdb.puppet.svc.cluster.local:8080", "https://puppetdbapi.service.consul/pdb", mergeStatic)
srv := newTestServer(cfg)
si := srv.newSourceInjector("", true)
for backend, want := range map[string]string{"a": "puppetdb.puppet.svc.cluster.local", "b": "puppetdbapi.service.consul"} {
var rec factFields
if err := json.Unmarshal(si.factRecord("h1", backend, ""), &rec); err != nil {
t.Fatal(err)
}
if rec.Value != want {
t.Errorf("fact value for %s = %q, want %q", backend, rec.Value, want)
}
if got := nodeSources(t, []byte(`[`+string(si.stamp([]byte(`{"certname":"h1"}`), backend))+`]`), defaultSourceFact); got["h1"] != want {
t.Errorf("node stamp for %s = %v, want %q", backend, got, want)
}
}
}
func TestCertnameScope(t *testing.T) {
tests := map[string]string{
`["=","name","x"]`: ``,
`["and",["=","name","x"],["=","environment","prod"]]`: ``,
`["or",["=","certname","h1"],["=","name","x"]]`: ``,
`["and",["not",["=","certname","h1"]],["=","name","x"]]`: ``,
`["and",["=","certname","h1"],["=","name","x"]]`: `["=","certname","h1"]`,
`["and",["~","certname","["],["=","name","x"]]`: ``,
`["and",["in","certname",["extract","certname",["=","a","b"]]],["=","name","x"]]`: ``,
`["and",["in","certname",["array",["h1","h2"]]],["~","certname","^h"],["=","name","x"]]`: `["and",["in","certname",["array",["h1","h2"]]],["~","certname","^h"]]`,
}
for in, want := range tests {
var ast []json.RawMessage
if err := json.Unmarshal([]byte(in), &ast); err != nil {
t.Fatal(err)
}
if got := certnameScope(ast); got != want {
t.Errorf("certnameScope(%s) = %q, want %q", in, got, want)
}
}
}
+123 -36
View File
@@ -35,45 +35,42 @@ func (e *upstreamError) Error() string {
return fmt.Sprintf("HTTP %d: %s", e.status, strings.TrimSpace(string(e.body))) return fmt.Sprintf("HTTP %d: %s", e.status, strings.TrimSpace(string(e.body)))
} }
// clientShaped reports whether a status blames the request rather than the // replayableStatus reports whether a status is an error a backend explained and
// backend or pdbmux itself. openvoxdb answers every bad query with 400 — // so can stand as the answer when every backend gave it. Everything from 400 up
// src/puppetlabs/puppetdb/middleware.clj:98-116 and http.clj:115-129 — and its // qualifies, because openvoxdb does not reserve 5xx for its own faults: a
// other 4xx say something a client of pdbmux neither caused nor can fix: 403 is // malformed query reaches the client as a 400 only when something on the way
// pdbmux's own certificate being refused (middleware.clj:44-58); 404 is either // down happens to throw a class the query engine catches (query_eng.clj:571-583
// an absent object or an unknown path (http.clj:238-242, // catches IllegalArgumentException and the :puppetlabs.puppetdb.query/invalid
// middleware.clj:381-398), and which objects a backend holds is the one thing // ExceptionInfo), and as a 500 otherwise. ["=","name"] is a 400 on /nodes and a
// backends are meant to disagree about; 408 and 429 report a backend's timing // 500 with an empty body on /facts, because only the facts entity runs
// and capacity. A backend that times out mid-query answers 200 with a truncated // rewrite-fact-query, whose name-constraint does an unguarded (nth clause 2)
// body rather than any 4xx (query_eng.clj:463-483), so no timeout reaches here. // (engine.clj:2988-3003); a bare string or number for query is a 500 reading
// "Output of convert-query-params does not match schema" from the schema check
// in front of the parser. Drawing the line at 500 would replay one of those and
// swallow the other, which is the route-dependence clients notice.
// //
// Every 4xx left over blames the request and so is safe to replay, though none // Below 400 nothing is replayed. A 3xx carries its meaning in Location, which
// is reachable while the fan-out forwards no client header: 406 needs an Accept // the fan-out does not keep, and a non-200 2xx is not an error at all.
// the query app refuses (http/server.clj:72; must-accept-type in func replayableStatus(status int) bool {
// http.clj:136-145 is unwired), and 415 a Content-Encoding on a POST to return status >= 400
// /commands (middleware.clj:165-178), which pdbmux never proxies.
func clientShaped(status int) bool {
switch status {
case http.StatusForbidden, http.StatusNotFound,
http.StatusRequestTimeout, http.StatusTooManyRequests:
return false
}
return status >= 400 && status < 500
} }
// unanimousClientError returns the rejection to replay when every backend in a // unanimousUpstreamError returns the reply to replay when every backend answered
// fan-out refused the same query with the same client-shaped status, and nil // the same request the same way, and nil otherwise. Every backend is asked the
// otherwise. Every backend is asked the same question, so unanimity is what // same question, so unanimity is what separates the estate's real answer from a
// distinguishes a bad query from a sick estate: a transport failure, a 5xx, or // sick estate: a transport failure, or two backends disagreeing on the status,
// two backends disagreeing on the status all leave at least one backend whose // leaves at least one backend whose answer is evidence about the backend rather
// answer is evidence about the backend rather than about the request. // than about the request, and that is what 502 describes.
//
// A nil entry stands for a backend that did not answer with an HTTP status at
// all, which defeats unanimity.
// //
// The reply returned is the first in configured order, so a client retrying a // The reply returned is the first in configured order, so a client retrying a
// rejected query is told the same thing every time. // rejected query is told the same thing every time.
func unanimousClientError(results []backendResult) *upstreamError { func unanimousUpstreamError(errs []*upstreamError) *upstreamError {
var first *upstreamError var first *upstreamError
for _, res := range results { for _, ue := range errs {
var ue *upstreamError if ue == nil || !replayableStatus(ue.status) {
if !errors.As(res.err, &ue) || !clientShaped(ue.status) {
return nil return nil
} }
if first == nil { if first == nil {
@@ -85,9 +82,99 @@ func unanimousClientError(results []backendResult) *upstreamError {
return first return first
} }
// writeUpstreamError answers a fan-out that produced no records. A unanimous // askOrder says why a set of backends was asked, which is what decides whose
// client-shaped rejection is replayed with the backend's own status and // reply becomes the client's answer when none of them answered 2xx. The two
// explanation; anything else is reported as a gateway failure. // cases are not settings on one rule, they are different questions, and reading
// a reply under the wrong one is how a real answer turns into a 502.
type askOrder int
const (
// askPeers: every backend was asked the same question and any of them could
// have answered it, so each reply is an opinion about that question. Only a
// reply they all gave is the estate's answer; anything else leaves a backend
// whose reply is evidence about the backend, which is what 502 reports.
askPeers askOrder = iota
// askOwnerFirst: the path names one entity, so the backend holding it is
// asked first — or, where no owner is known, they are asked in configured
// order — and the rest only as fallbacks. A backend that does not hold the
// entity replies 404 to say exactly that, which is not a dissent from the
// holder's reply, so requiring the two to agree asks a question nobody was
// posed. The first backend that answered is the one that knows.
askOwnerFirst
)
// outcome turns a round's replies — one entry per backend in the order they were
// asked, nil where a backend produced no HTTP status at all — into the error to
// answer with.
func (o askOrder) outcome(replies []*upstreamError) error {
if o == askOwnerFirst {
return ownerOutcome(replies)
}
return peerOutcome(replies)
}
// peerOutcome is the error to answer a fan-out of peers with when it produced
// nothing: their unanimous reply where there is one, and pdbmux's gateway error
// otherwise.
func peerOutcome(errs []*upstreamError) error {
if ue := unanimousUpstreamError(errs); ue != nil {
return ue
}
return errAllBackendsFailed
}
// ownerOutcome is the error to answer an owner-routed path with: the reply of
// the first backend that answered. The owner is asked first, so that is the
// owner's own reply — it holds the entity, so its 500 is the truth about this
// request whatever a backend that does not hold the entity said.
//
// An owner that produced no status at all leaves a nil entry and the next
// backend's reply stands instead. The fallback chain exists so a node whose
// owner is unreachable is still served, and a backend that did answer explains
// more than a 502 that describes neither. Only a round where nothing answered is
// pdbmux's own gateway error.
func ownerOutcome(replies []*upstreamError) error {
for _, ue := range replies {
if ue != nil && replayableStatus(ue.status) {
return ue
}
}
return errAllBackendsFailed
}
// backendUpstreamErrors reduces a merged fan-out's results to one entry per
// backend, nil where the backend answered or failed without a status.
func backendUpstreamErrors(results []backendResult) []*upstreamError {
errs := make([]*upstreamError, len(results))
for i, res := range results {
var ue *upstreamError
if errors.As(res.err, &ue) {
errs[i] = ue
}
}
return errs
}
// clientRefusal reports whether err is a unanimous upstream rejection blaming
// the request. Only those mean the estate is healthy and the query was wrong,
// which is why they are neither counted as degraded service nor answered from
// the cache; a unanimous 5xx is replayed just the same but is the backends
// reporting their own fault.
//
// Every 4xx openvoxdb answers a fan-out with is about the request, including the
// 403 on /metrics/v2/list, which is that backend's own policy rather than an
// authentication failure. That holds only while pdbmux presents no client
// certificate: authenticate to backends and a rejected certificate becomes a
// 403 on every route at once, which this would read as a healthy estate refusing
// a bad query and hide a total outage behind a replayed 403.
func clientRefusal(err error) bool {
var ue *upstreamError
return errors.As(err, &ue) && ue != nil && ue.status < 500
}
// writeUpstreamError answers a round that produced no records. An upstream
// error the askOrder resolved to is replayed with the backend's own status,
// content type and explanation; anything else is reported as a gateway failure.
func (s *Server) writeUpstreamError(w http.ResponseWriter, err error) { func (s *Server) writeUpstreamError(w http.ResponseWriter, err error) {
var ue *upstreamError var ue *upstreamError
if !errors.As(err, &ue) { if !errors.As(err, &ue) {
@@ -98,7 +185,7 @@ func (s *Server) writeUpstreamError(w http.ResponseWriter, err error) {
if len(strings.TrimSpace(string(body))) == 0 { if len(strings.TrimSpace(string(body))) == 0 {
setContentType(w, "text/plain; charset=utf-8") setContentType(w, "text/plain; charset=utf-8")
w.WriteHeader(ue.status) w.WriteHeader(ue.status)
_, _ = fmt.Fprintf(w, "upstream rejected the query: %s\n", http.StatusText(ue.status)) _, _ = fmt.Fprintf(w, "upstream answered %d %s\n", ue.status, http.StatusText(ue.status))
return return
} }
setContentType(w, ue.contentType) setContentType(w, ue.contentType)
+210 -38
View File
@@ -14,11 +14,16 @@ import (
const badOrderBy = `Unrecognized column 'bogus' specified in :order_by` const badOrderBy = `Unrecognized column 'bogus' specified in :order_by`
// upstreamJSONContentType is spelled out rather than taken from jsonContentType,
// so the assertion is against what openvoxdb answers and not against whatever
// pdbmux happens to be configured with.
const upstreamJSONContentType = "application/json;charset=utf-8"
func upErr(status int, body string) error { func upErr(status int, body string) error {
return newUpstreamError(status, "text/plain; charset=utf-8", []byte(body)) return newUpstreamError(status, "text/plain; charset=utf-8", []byte(body))
} }
func TestUnanimousClientError_AgreementRule(t *testing.T) { func TestUnanimousUpstreamError_AgreementRule(t *testing.T) {
refused := errors.New("dial tcp: connection refused") refused := errors.New("dial tcp: connection refused")
for _, tc := range []struct { for _, tc := range []struct {
name string name string
@@ -29,18 +34,25 @@ func TestUnanimousClientError_AgreementRule(t *testing.T) {
{"all agree on 415", []error{upErr(415, "bad media"), upErr(415, "bad media")}, 415}, {"all agree on 415", []error{upErr(415, "bad media"), upErr(415, "bad media")}, 415},
{"differing bodies still agree", []error{upErr(400, "one"), upErr(400, "two")}, 400}, {"differing bodies still agree", []error{upErr(400, "one"), upErr(400, "two")}, 400},
{"disagreeing 4xx", []error{upErr(400, "bad"), upErr(404, "gone")}, 0}, {"disagreeing 4xx", []error{upErr(400, "bad"), upErr(404, "gone")}, 0},
// Both client-shaped, so only the same-status rule refuses these.
{"agree on shape, disagree on code", []error{upErr(400, "x"), upErr(415, "y")}, 0}, {"agree on shape, disagree on code", []error{upErr(400, "x"), upErr(415, "y")}, 0},
{"a majority agrees", []error{upErr(400, "x"), upErr(400, "y"), upErr(422, "z")}, 0}, {"a majority agrees", []error{upErr(400, "x"), upErr(400, "y"), upErr(422, "z")}, 0},
{"the first differs", []error{upErr(422, "z"), upErr(400, "x"), upErr(400, "y")}, 0}, {"the first differs", []error{upErr(422, "z"), upErr(400, "x"), upErr(400, "y")}, 0},
{"4xx with a 5xx", []error{upErr(400, "bad"), upErr(500, "boom")}, 0}, {"4xx with a 5xx", []error{upErr(400, "bad"), upErr(500, "boom")}, 0},
{"4xx with a transport failure", []error{upErr(400, "bad"), refused}, 0}, {"4xx with a transport failure", []error{upErr(400, "bad"), refused}, 0},
{"all 403", []error{upErr(403, "denied"), upErr(403, "denied")}, 0}, {"5xx with a transport failure", []error{upErr(500, "boom"), refused}, 0},
{"all 404", []error{upErr(404, "gone"), upErr(404, "gone")}, 0}, // Statuses the old 4xx-only rule swallowed. Every one of them is what
{"all 429", []error{upErr(429, "slow down"), upErr(429, "slow down")}, 0}, // both backends actually said, so every one of them is the answer.
{"all 408", []error{upErr(408, "too slow"), upErr(408, "too slow")}, 0}, {"all 403", []error{upErr(403, "denied"), upErr(403, "denied")}, 403},
{"all 500", []error{upErr(500, "boom"), upErr(500, "boom")}, 0}, {"all 404", []error{upErr(404, "gone"), upErr(404, "gone")}, 404},
{"all 503", []error{upErr(503, "unavailable"), upErr(503, "unavailable")}, 0}, {"all 429", []error{upErr(429, "slow down"), upErr(429, "slow down")}, 429},
{"all 408", []error{upErr(408, "too slow"), upErr(408, "too slow")}, 408},
{"all 500", []error{upErr(500, "boom"), upErr(500, "boom")}, 500},
{"all 503", []error{upErr(503, "unavailable"), upErr(503, "unavailable")}, 503},
{"disagreeing 5xx", []error{upErr(500, "boom"), upErr(503, "later")}, 0},
// Nothing below 400 is an error a backend explained, and a 3xx's meaning
// lives in a Location header the fan-out never kept.
{"all 302", []error{upErr(302, ""), upErr(302, "")}, 0},
{"all 204", []error{upErr(204, ""), upErr(204, "")}, 0},
{"no backends", nil, 0}, {"no backends", nil, 0},
{"a backend succeeded", []error{upErr(400, "bad"), nil}, 0}, {"a backend succeeded", []error{upErr(400, "bad"), nil}, 0},
} { } {
@@ -49,7 +61,7 @@ func TestUnanimousClientError_AgreementRule(t *testing.T) {
for i, err := range tc.errs { for i, err := range tc.errs {
results[i] = backendResult{name: fmt.Sprintf("b%d", i), err: err} results[i] = backendResult{name: fmt.Sprintf("b%d", i), err: err}
} }
got := unanimousClientError(results) got := unanimousUpstreamError(backendUpstreamErrors(results))
switch { switch {
case tc.want == 0 && got != nil: case tc.want == 0 && got != nil:
t.Fatalf("want no replay, got HTTP %d", got.status) t.Fatalf("want no replay, got HTTP %d", got.status)
@@ -64,17 +76,33 @@ func TestUnanimousClientError_AgreementRule(t *testing.T) {
// Two backends can explain the same rejection differently; the reply is the // Two backends can explain the same rejection differently; the reply is the
// first in configured order so a retried query is answered the same way twice. // first in configured order so a retried query is answered the same way twice.
func TestUnanimousClientError_PicksFirstInConfiguredOrder(t *testing.T) { func TestUnanimousUpstreamError_PicksFirstInConfiguredOrder(t *testing.T) {
results := []backendResult{ results := []backendResult{
{name: "a", err: upErr(400, "from a")}, {name: "a", err: upErr(400, "from a")},
{name: "b", err: upErr(400, "from b")}, {name: "b", err: upErr(400, "from b")},
} }
got := unanimousClientError(results) got := unanimousUpstreamError(backendUpstreamErrors(results))
if got == nil || !strings.Contains(string(got.body), "from a") { if got == nil || !strings.Contains(string(got.body), "from a") {
t.Fatalf("body = %q, want the first backend's", got) t.Fatalf("body = %q, want the first backend's", got)
} }
} }
// Only a 4xx says the estate is well and the query was wrong. A 5xx is replayed
// too, but it must keep counting as a backend fault.
func TestClientRefusal_OnlyFourXX(t *testing.T) {
for status, want := range map[int]bool{400: true, 404: true, 429: true, 499: true, 500: false, 503: false} {
if got := clientRefusal(upErr(status, "x")); got != want {
t.Errorf("clientRefusal(%d) = %v, want %v", status, got, want)
}
}
if clientRefusal(errAllBackendsFailed) {
t.Error("a transport failure is not a client refusal")
}
if clientRefusal(unanimousUpstreamError(nil)) {
t.Error("an absent unanimous error is not a client refusal")
}
}
func TestUpstreamError_BodyIsCapped(t *testing.T) { func TestUpstreamError_BodyIsCapped(t *testing.T) {
ue := newUpstreamError(400, "text/plain", []byte(strings.Repeat("x", upstreamBodyLimit*2))) ue := newUpstreamError(400, "text/plain", []byte(strings.Repeat("x", upstreamBodyLimit*2)))
if len(ue.body) != upstreamBodyLimit { if len(ue.body) != upstreamBodyLimit {
@@ -112,31 +140,78 @@ func TestHandler_MergedReplaysNonBadRequestStatus(t *testing.T) {
} }
} }
// A unanimous 403 is pdbmux's own credentials being refused, not the client's // A status every backend agreed on is the estate's own answer whatever it is,
// query, so it must not be handed back as the client's fault. // so the statuses the 4xx-only rule used to swallow now reach the client.
func TestHandler_MergedForbiddenStays502(t *testing.T) { func TestHandler_MergedReplaysEveryUnanimousStatus(t *testing.T) {
for _, tc := range []struct {
status int
body string
}{
{http.StatusForbidden, "certificate not allowed"},
{http.StatusNotFound, "Not Found"},
{http.StatusTooManyRequests, "slow down"},
{http.StatusInternalServerError, "Output of convert-query-params does not match schema"},
{http.StatusServiceUnavailable, "starting up"},
} {
t.Run(http.StatusText(tc.status), func(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusForbidden, "certificate not allowed" a.reject, a.rejectBody = tc.status, tc.body
b.reject, b.rejectBody = http.StatusForbidden, "certificate not allowed" b.reject, b.rejectBody = tc.status, tc.body
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway { rec := doGet(t, srv.Handler(), nodesPath, "")
t.Fatalf("status = %d, want 502", rec.Code) if rec.Code != tc.status {
t.Fatalf("status = %d, want the upstream %d: %s", rec.Code, tc.status, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), tc.body) {
t.Errorf("body = %q, want the upstream explanation", rec.Body.String())
}
})
} }
} }
// Every backend answering 404 is ambiguous between a bad path and a record // A backend that answered nothing at all leaves no unanimity to replay, whatever
// nobody holds, so it stays a gateway error on a merged route. // the others said.
func TestHandler_MergedNotFoundStays502(t *testing.T) { func TestHandler_MergedServerErrorWithUnreachableStays502(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
dead := newFakeBackend(t, `[]`, `[]`)
deadURL := dead.srv.URL
dead.srv.Close()
srv := newTestServer(testConfig(a.srv.URL, deadURL, mergeStatic))
if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway {
t.Fatalf("status = %d, want 502 when a backend never answered", rec.Code)
}
}
func TestHandler_MergedDisagreeingServerErrorsStay502(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusNotFound, "Not Found" a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
b.reject, b.rejectBody = http.StatusNotFound, "Not Found" b.reject, b.rejectBody = http.StatusServiceUnavailable, "later"
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway { if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway {
t.Fatalf("status = %d, want 502", rec.Code) t.Fatalf("status = %d, want 502 when backends disagree", rec.Code)
}
}
// A unanimous 5xx is replayed, but it is the backends admitting a fault, so it
// still has to read as degraded service.
func TestHandler_UnanimousServerErrorIsStillADegradedRound(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
b.reject, b.rejectBody = http.StatusInternalServerError, "boom"
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want the upstream 500", rec.Code)
}
if hr := health(t, srv); hr.Query.PartialRounds == 0 {
t.Errorf("query health = %+v, want the 5xx counted as a failed round", hr.Query)
} }
} }
@@ -255,7 +330,7 @@ func TestHandler_ReportSubResourceRejectionReplayedButNotItsAbsence(t *testing.T
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
if rec := doGet(t, newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)).Handler(), if rec := doGet(t, newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)).Handler(),
reportsPath+"/nope/events", ""); rec.Code != http.StatusNotFound { reportsPath+"/nope/events", ""); rec.Code != http.StatusNotFound {
t.Fatalf("status = %d, want pdbmux's own 404 when nobody holds the report", rec.Code) t.Fatalf("status = %d, want 404 when nobody holds the report", rec.Code)
} }
a.reject, a.rejectBody = http.StatusBadRequest, badOrderBy a.reject, a.rejectBody = http.StatusBadRequest, badOrderBy
@@ -296,18 +371,18 @@ func TestHandler_RejectedQueryIsNotCached(t *testing.T) {
b.reject, b.rejectBody = http.StatusBadRequest, badOrderBy b.reject, b.rejectBody = http.StatusBadRequest, badOrderBy
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
rec := doGet(t, srv.Handler(), factsPath, "") rec := doGet(t, srv.Handler(), nodesPath, "")
if rec.Code != http.StatusBadRequest { if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want the upstream 400 replayed: %s", rec.Code, rec.Body.String()) t.Fatalf("status = %d, want the upstream 400 replayed: %s", rec.Code, rec.Body.String())
} }
if n := srv.factsCache.Stats().Entries; n != 0 { if n := srv.nodeCache.Stats().Entries; n != 0 {
t.Fatalf("cache holds %d entries after a rejected query", n) t.Fatalf("cache holds %d entries after a rejected query", n)
} }
// The rejection left nothing behind, so the next good query is served fresh. // The rejection left nothing behind, so the next good query is served fresh.
a.reject, b.reject = 0, 0 a.reject, b.reject = 0, 0
a.factsBody = `[` + fact("h1", "role", "web", "") + `]` a.nodesBody = `[` + node("web", "2026-01-01T00:00:00.000Z") + `]`
ok := doGet(t, srv.Handler(), factsPath, "") ok := doGet(t, srv.Handler(), nodesPath, "")
if ok.Code != http.StatusOK || !strings.Contains(ok.Body.String(), "web") { if ok.Code != http.StatusOK || !strings.Contains(ok.Body.String(), "web") {
t.Fatalf("follow-up = %d %s", ok.Code, ok.Body.String()) t.Fatalf("follow-up = %d %s", ok.Code, ok.Body.String())
} }
@@ -316,13 +391,13 @@ func TestHandler_RejectedQueryIsNotCached(t *testing.T) {
func TestHandler_AllBackendsFailedIsNotCached(t *testing.T) { func TestHandler_AllBackendsFailedIsNotCached(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`) a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
a.fail, b.fail = true, true a.dead, b.dead = true, true
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
if rec := doGet(t, srv.Handler(), factsPath, ""); rec.Code != http.StatusBadGateway { if rec := doGet(t, srv.Handler(), nodesPath, ""); rec.Code != http.StatusBadGateway {
t.Fatalf("status = %d, want 502", rec.Code) t.Fatalf("status = %d, want 502", rec.Code)
} }
if n := srv.factsCache.Stats().Entries; n != 0 { if n := srv.nodeCache.Stats().Entries; n != 0 {
t.Fatalf("cache holds %d entries after a failed fan-out", n) t.Fatalf("cache holds %d entries after a failed fan-out", n)
} }
} }
@@ -330,18 +405,18 @@ func TestHandler_AllBackendsFailedIsNotCached(t *testing.T) {
// Stale records answer an outage. They do not answer a query the estate refused: // Stale records answer an outage. They do not answer a query the estate refused:
// the client has to be told why, not handed data for a question it did not ask. // the client has to be told why, not handed data for a question it did not ask.
func TestHandler_RejectedQueryIsNotAnsweredFromStale(t *testing.T) { func TestHandler_RejectedQueryIsNotAnsweredFromStale(t *testing.T) {
a := newFakeBackend(t, `[]`, `[`+fact("h1", "role", "old", "")+`]`) a := newFakeBackend(t, `[`+node("old", "2026-01-01T00:00:00.000Z")+`]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
if warm := doGet(t, srv.Handler(), factsPath, ""); warm.Code != http.StatusOK { if warm := doGet(t, srv.Handler(), nodesPath, ""); warm.Code != http.StatusOK {
t.Fatalf("warm-up status %d", warm.Code) t.Fatalf("warm-up status %d", warm.Code)
} }
clk.advance(31 * time.Second) clk.advance(31 * time.Second)
a.reject, a.rejectBody = http.StatusBadRequest, badOrderBy a.reject, a.rejectBody = http.StatusBadRequest, badOrderBy
b.reject, b.rejectBody = http.StatusBadRequest, badOrderBy b.reject, b.rejectBody = http.StatusBadRequest, badOrderBy
rec := doGet(t, srv.Handler(), factsPath, "") rec := doGet(t, srv.Handler(), nodesPath, "")
if rec.Code != http.StatusBadRequest { if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want the rejection rather than the stale entry: %s", rec.Code, rec.Body.String()) t.Fatalf("status = %d, want the rejection rather than the stale entry: %s", rec.Code, rec.Body.String())
} }
@@ -355,16 +430,16 @@ func TestHandler_RejectedQueryIsNotAnsweredFromStale(t *testing.T) {
// An outage still falls back to the stale copy, unchanged by the replay path. // An outage still falls back to the stale copy, unchanged by the replay path.
func TestHandler_OutageStillFallsBackToStale(t *testing.T) { func TestHandler_OutageStillFallsBackToStale(t *testing.T) {
a := newFakeBackend(t, `[]`, `[`+fact("h1", "role", "old", "")+`]`) a := newFakeBackend(t, `[`+node("old", "2026-01-01T00:00:00.000Z")+`]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`) b := newFakeBackend(t, `[]`, `[]`)
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL)) srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
if warm := doGet(t, srv.Handler(), factsPath, ""); warm.Code != http.StatusOK { if warm := doGet(t, srv.Handler(), nodesPath, ""); warm.Code != http.StatusOK {
t.Fatalf("warm-up status %d", warm.Code) t.Fatalf("warm-up status %d", warm.Code)
} }
clk.advance(31 * time.Second) clk.advance(31 * time.Second)
a.fail, b.fail = true, true a.fail, b.fail = true, true
rec := doGet(t, srv.Handler(), factsPath, "") rec := doGet(t, srv.Handler(), nodesPath, "")
if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), "old") { if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), "old") {
t.Fatalf("stale fallback = %d %s", rec.Code, rec.Body.String()) t.Fatalf("stale fallback = %d %s", rec.Code, rec.Body.String())
} }
@@ -493,3 +568,100 @@ func hostOf(t *testing.T, raw string) string {
} }
return u.Host return u.Host
} }
// Every backend answering 500 is the estate's answer on the first-holder route
// as much as on a merged one.
func TestHandler_ReportSubResourceReplaysUnanimousServerError(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = http.StatusInternalServerError, "boom"
b.reject, b.rejectBody = http.StatusInternalServerError, "boom"
rec := doGet(t, newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)).Handler(),
reportsPath+"/nope/events", "")
if rec.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want the upstream 500: %s", rec.Code, rec.Body.String())
}
}
// One backend saying 404 and one saying nothing leaves it unknown whether the
// report exists, which is the gateway error's job to say.
func TestHandler_ReportSubResourceUnreachableBackendIs502(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
b.dead = true
rec := doGet(t, newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)).Handler(),
reportsPath+"/nope/events", "")
if rec.Code != http.StatusBadGateway {
t.Fatalf("status = %d, want 502 when a backend never answered", rec.Code)
}
}
// The pass-through route used to replay whichever backend answered first, even
// when the others disagreed or never answered. It follows the same rule now.
func TestProxyUnmerged_ReplayRule(t *testing.T) {
for _, tc := range []struct {
name string
aStatus int
bStatus int
bDead bool
want int
}{
{"unanimous 500 is replayed", 500, 500, false, 500},
{"unanimous 404 is replayed", 404, 404, false, 404},
{"unanimous 403 is replayed", 403, 403, false, 403},
{"disagreement is a gateway error", 400, 404, false, http.StatusBadGateway},
{"a silent backend is a gateway error", 404, 0, true, http.StatusBadGateway},
} {
t.Run(tc.name, func(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
a.reject, a.rejectBody = tc.aStatus, "upstream said no"
b.reject, b.rejectBody = tc.bStatus, "upstream said no"
b.dead = tc.bDead
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), resourcesPath, `["=","type","File"]`)
if rec.Code != tc.want {
t.Fatalf("status = %d, want %d: %s", rec.Code, tc.want, rec.Body.String())
}
})
}
}
// pdbmux has to be indistinguishable from a PuppetDB, and openvoxdb answers a
// query with a charset on the content type.
func TestHandler_SuccessContentTypeMatchesUpstream(t *testing.T) {
a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
h := srv.Handler()
for _, path := range []string{nodesPath, factsPath, factNamesPath, reportsPath} {
t.Run(path, func(t *testing.T) {
rec := doGet(t, h, path, "")
if rec.Code != http.StatusOK {
t.Fatalf("status = %d: %s", rec.Code, rec.Body.String())
}
if got := rec.Header().Get("Content-Type"); got != upstreamJSONContentType {
t.Errorf("Content-Type = %q, want %q", got, upstreamJSONContentType)
}
})
}
}
func TestHandler_CachedSuccessContentTypeMatchesUpstream(t *testing.T) {
a := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler()
// The miss builds the entry and the hit replays it; both are the client's view.
for _, label := range []string{"miss", "hit"} {
rec := doGet(t, h, nodesPath, "")
if got := rec.Header().Get("Content-Type"); got != upstreamJSONContentType {
t.Errorf("%s Content-Type = %q, want %q", label, got, upstreamJSONContentType)
}
}
}