4 Commits

Author SHA1 Message Date
unkin-agent c0aab83c20 Convert a single-flight panic into an error
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
- flightGroup.Do recovers a panicking fn so the leader and every waiter get a non-nil error instead of a zero-value success served as 200 []
- Note in the README that facts_cache_bytes budgets body bytes only
2026-09-05 21:03:23 +10:00
unkin-agent 1e2acc9c14 feat: cache merged /facts and /nodes in memory, stale on backend failure
A busy Puppetboard re-fans-out the same /facts query every few seconds, and a
502 is worse than 30-second-old facts when every PuppetDB is unreachable.

- Add a `Cache` interface (get reports fresh/stale/miss, put, stats) keyed on
  `<path>?<params>` with keys and repeated values sorted, plus a no-op default
  so uncached paths behave exactly as before.
- Route serveMerged/serveUnion/serveSummed through `serveCached`, so the
  reports cache drops in at `cacheFor` without touching a handler.
- Back /facts and /nodes with a byte-bounded LRU: `facts_ttl` (default 30s,
  clamped to a 30s cap) and `facts_cache_bytes` (default 64 MiB); expired
  entries are kept and served only when every backend fails.
- Single-flight identical keys so N concurrent requests cause one fan-out.
- Surface `cache` state and `serving_stale` in /healthz and the cache settings
  in `config show`.
2026-09-05 21:03:23 +10:00
benvin 394f7df3a7 Merge pull request 'Serve PuppetDB meta and metrics endpoints, sum node and resource counts' (#11) from benvin/meta-metrics-and-counts into main
Reviewed-on: #11
2026-09-05 20:56:06 +10:00
unkin-agent b6d59af7ef Serve PuppetDB meta and metrics endpoints, sum node and resource counts
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
## Why
Puppetboard 7.0.1 cannot run against pdbmux: it exits at import when
/pdb/meta/v1/version 404s, and its landing page, metrics and radiator views
404 on the Jolokia surface.

## How
- Serve /pdb/meta/v1/version, reporting the lowest version any backend runs,
  and /pdb/meta/v1/server-time from the first reachable backend.
- Merge the Jolokia surface (/metrics/v2/read, /metrics/v2/list,
  /metrics/v1/mbeans): objects union, numeric attributes sum by default, and
  Min/Max/Uptime/StartTime plus the distribution stats take a bound or a mean.
- Route /nodes extract-count queries to the summing path ahead of the certname
  merge, and give /resources aggregates the same path.
- Document the endpoints and merge semantics in the README.
- Cover version disagreement, metric rules, escaped MBean names, count summing
  and the non-aggregate /nodes merge with httptest backends.
2026-09-05 20:41:35 +10:00
12 changed files with 2541 additions and 50 deletions
+102 -13
View File
@@ -24,15 +24,21 @@ not PQL) is forwarded verbatim.
| Path | Behaviour |
|---|---|
| `GET /pdb/query/v4/nodes` | Fan out to all backends, dedupe by `certname`, keep the record with the newer `report_timestamp`. |
| `GET /pdb/query/v4/nodes` | Fan out to all backends, dedupe by `certname`, keep the record with the newer `report_timestamp`. An `extract`/`count` query is **summed** instead. |
| `GET /pdb/query/v4/facts` | Fan out to all, and per `certname` keep **all** facts from the backend that owns that node (see merge semantics). |
| `GET /pdb/query/v4/resources` | An `extract`/`count` query is fanned out and **summed**; any other query is an unmerged pass-through. |
| `GET /pdb/query/v4/reports` | Fan out to all and serve the **union**, deduped by report `hash`, re-ordered and re-paged across backends. |
| `GET /pdb/query/v4/events` | Fan out to all and serve the **union**, deduped by record identity, re-ordered and re-paged. |
| `GET /pdb/query/v4/event-counts` | Fan out to all and **sum** each subject's counts into one row per subject. |
| `GET /pdb/query/v4/aggregate-event-counts` | Fan out to all and **sum** the summary object's counts. |
| `GET /pdb/query/v4/reports/<hash>/{events,logs,metrics}` | Ask every backend; serve the answer from whichever backend actually holds that report. `404` when none does. |
| `GET /pdb/query/v4/*` (any other) | No merge rule, so backends are tried in configured order and the first success is streamed back verbatim; if all reject it, the first upstream error response is replayed. |
| `GET /healthz` | Per-backend reachability. `200 {"status":"ok"}` if all reachable, `200 degraded` if some fail, `503 down` if all fail. |
| `GET /pdb/meta/v1/version` | Fan out to all and report the **lowest** version any backend runs. |
| `GET /pdb/meta/v1/server-time` | Fan out to all and serve the first reachable backend's clock. |
| `GET /metrics/v2/read/<mbean>` | Fan out to all and merge the Jolokia response; numeric attributes are **summed** by default (see merge semantics). |
| `GET /metrics/v2/list` | Fan out to all and serve the **union** of the backends' MBean trees. |
| `GET /metrics/v1/mbeans[/<mbean>]` | Same merge, applied to the legacy envelope-less body. |
| `GET /healthz` | Per-backend reachability plus cache state. `200 {"status":"ok"}` if all reachable, `200 degraded` if some fail, `503 down` if all fail. |
Fan-out is concurrent. If one backend errors or times out, `pdbmux` serves the
surviving backends' results and logs a warning; a merged endpoint only returns `502` when
@@ -62,12 +68,20 @@ unknown fields survive untouched.
reporting to more than one backend stores identical records in each).
- **Aggregates** — `extract`/`group_by` rows are counts, not records, so each
backend returns a partial answer that has to be **added**, not deduped. This
covers `/event-counts`, `/aggregate-event-counts`, and a `/reports` query whose
`extract` carries a `["function", ...]` column.
- The grouping key is the row's non-aggregate fields: for `/reports` they come
from the query — the plain `extract` fields plus any `group_by` clause — and
for the event-count endpoints from the row itself (`subject_type`/`subject`,
or `summarize_by`), whose remaining fields are all counts.
covers `/event-counts`, `/aggregate-event-counts`, and any `/reports`,
`/nodes` or `/resources` query whose `extract` carries a `["function", ...]`
column.
- The grouping key is the row's non-aggregate fields: for `/reports`,
`/nodes` and `/resources` they come from the query — the plain `extract`
fields plus any `group_by` clause — and for the event-count endpoints from
the row itself (`subject_type`/`subject`, or `summarize_by`), whose
remaining fields are all counts.
- On `/nodes` this takes precedence over the `certname` merge: a count row has
no `certname`, so deduping would collapse every backend's count into one
backend's number. A `/nodes` query with no `function` column — including a
plain `extract` projection — still merges by `certname`.
- `/resources` has no cross-backend record identity to dedupe on, so only its
aggregate queries merge; everything else stays an unmerged pass-through.
- Rows sharing a key collapse into one with their numeric columns summed. A key
only one backend reported is passed through byte-for-byte. An aggregate column
that is absent or non-numeric in a row is skipped, never zeroed, so the
@@ -77,6 +91,39 @@ unknown fields survive untouched.
- `include_total=true` on a summed endpoint reports the **merged** row count,
not the sum of the backends' `X-Records`, since shared keys collapse.
### Metadata and metrics
- **`/pdb/meta/v1/version`** — when the backends agree, that version is served.
When they differ, `pdbmux` reports the **lowest**: a client reads this as the
feature level it may rely on, and the estate can only be relied on for what its
oldest PuppetDB implements. Versions compare segment by segment, numerically
where both segments are numbers (`7.9.0` < `7.12.0`), lexically otherwise.
A backend whose body is unparseable is skipped rather than treated as lowest.
- **`/pdb/meta/v1/server-time`** — the clock of whichever PuppetDB answered is
not estate state and has no meaningful merge, so the first **reachable**
backend in configured order supplies it, the same tie-break used elsewhere.
- **`/metrics/...`** — the Jolokia envelope's `value` is merged and the rest of
the envelope comes from the first backend (with the newest `timestamp`).
Values merge recursively:
- Objects merge over the **union** of their keys, so an MBean attribute only
one backend exposes still survives.
- Numbers combine by the attribute's own name. The default is a **sum** —
almost everything here is a population count (`num-nodes`, `num-resources`,
queue depth, command totals) whose estate-wide value is the total, and rates
are additive throughput. The exceptions describe a distribution or a bound,
where adding two servers' numbers yields a figure that was never true of
either: `Min` takes the minimum; `Max`, `Uptime` and `StartTime` take the
maximum; `Mean`, `Median`, `StdDev` and `*Percentile` take the unweighted
arithmetic mean (`pdbmux` has no per-backend sample counts to weight by).
Matching is case-insensitive.
- Strings, booleans, arrays, nulls and mixed kinds keep the first backend's
value — there is no sound way to add them.
- Jolokia signals a bad MBean as a non-2xx `status` **inside** an HTTP 200.
Such a backend is skipped; if every backend does so, the first one's error
envelope is replayed verbatim so the client sees the real reason.
- MBean names arrive percent-encoded over Jolokia's own `!`-escaping; the raw
path is forwarded so neither layer is lost.
### Paging and ordering on the merged endpoints
Each backend applies `order_by`/`limit`/`offset` to its own slice only, so
@@ -93,6 +140,43 @@ Each backend applies `order_by`/`limit`/`offset` to its own slice only, so
- A malformed `limit`, `offset` or `order_by` gets a `400` rather than being
forwarded.
## Caching
`pdbmux` caches merged `/nodes` and `/facts` record sets **in memory** so a busy
Puppetboard does not re-fan-out the same query every few seconds. Everything else
runs uncached — including `extract`/`count` aggregates on those two paths, and
the `/pdb/meta/v1/*` and `/metrics/*` endpoints, which are served live on every
request. The cache is an interface, and `/reports` gets its own (S3-backed)
backend later without further handler changes.
- **Key** — `<path>?<params>`, where the params are the ones that actually
determine the response, URL-encoded with keys sorted ascending and a repeated
param's values sorted ascending. Param order in the request is therefore
irrelevant: one canonical key per distinct request. A request with no params
keys on the bare path.
- **TTL** — `facts_ttl`, default `30s`, **hard cap `30s`**. A larger configured
value is **clamped** down to the cap, not rejected, so a stray env var cannot
crash-loop a container; `pdbmux config show` prints
`facts_ttl : 30s (clamped from 600s, cap 30s)` when that happens. `facts_ttl: 0`
disables the cache entirely and the merged endpoints behave exactly as before.
- **Stale on failure only** — an expired entry is kept, not dropped. When the TTL
has passed `pdbmux` always re-queries the backends; the expired copy is served
**only** if every backend fails, which turns a `502` into slightly-old data. A
healthy backend is never shadowed by a stale entry.
- **Bounded** — `facts_cache_bytes` (default 64 MiB) is a byte budget, evicted
least-recently-used; reads count as use, so a stale entry that is still being
asked for survives. A single response larger than the whole budget is not
cached at all. The budget counts stored response bodies only — cache keys and
the list/map bookkeeping are not accounted for, so it is a target for body
bytes rather than a hard cap on process memory.
- **Single-flight** — concurrent requests for the same key collapse into one
upstream fan-out; the rest wait for it and share the result.
- **Visibility** — `/healthz` carries a `cache` object: `backend`
(`memory`/`none`), `ttl`, `entries`, `stale_entries`, `bytes`, `serving_stale`,
`stale_served` and `last_stale_served`. `serving_stale` is `true` from the
moment a stale fallback is served until the next response comes from a live
fan-out or a fresh entry.
## Config
Precedence (lowest → highest): **defaults < config file < env vars (`PDBMUX_*`) < flags**.
@@ -114,9 +198,11 @@ backends: # order is a tie-break only, not a ranking
url: http://puppetdb1.example.com:8080
- name: pdb-b
url: https://puppetdb2.example.com
merge: freshness # freshness | static
timeout: 10s # per-upstream request timeout
freshness_ttl: 30s # freshness-map cache TTL (freshness merge only)
merge: freshness # freshness | static
timeout: 10s # per-upstream request timeout
freshness_ttl: 30s # freshness-map cache TTL (freshness merge only)
facts_ttl: 30s # /facts + /nodes response cache TTL; 0 disables, capped at 30s
facts_cache_bytes: 67108864 # byte budget for that cache (64 MiB), LRU-evicted
```
`backends[*].url` is a **base** URL (`scheme://host[:port]`); `pdbmux` appends
@@ -129,6 +215,8 @@ the `/pdb/query/v4/...` path per request.
| `PDBMUX_MERGE` | `merge` |
| `PDBMUX_TIMEOUT` | `timeout` (Go duration, e.g. `10s`) |
| `PDBMUX_FRESHNESS_TTL` | `freshness_ttl` |
| `PDBMUX_FACTS_TTL` | `facts_ttl` (clamped to 30s) |
| `PDBMUX_FACTS_CACHE_BYTES` | `facts_cache_bytes` (plain integer bytes) |
| `PDBMUX_BACKENDS` | whole backend list, as `name=url,name=url` |
Flags: `--config`, `--listen`, `--merge`.
@@ -162,5 +250,6 @@ A static (`CGO_ENABLED=0`) binary on a distroless base. Configure it with
`PDBMUX_*` env vars (at minimum `PDBMUX_BACKENDS`), or mount a config file — a
configmap at `/etc/pdbmux/config.yaml` is picked up with no env var at all, and
any other mount path works via `PDBMUX_CONFIG`. Env vars still override file
values, so the two mix. Stateless, so run as many replicas as you like; use
`/healthz` for liveness/readiness probes.
values, so the two mix. Run as many replicas as you like — the only state is the
in-memory cache, which is per-replica and bounded by `facts_cache_bytes`, so size
the memory limit above it. Use `/healthz` for liveness/readiness probes.
+267
View File
@@ -0,0 +1,267 @@
package main
import (
"container/list"
"context"
"fmt"
"net/url"
"runtime/debug"
"sort"
"sync"
"time"
)
// CacheStatus distinguishes the three outcomes of a lookup: nothing stored, a
// stored entry within its TTL, and a stored entry past it.
type CacheStatus int
const (
CacheMiss CacheStatus = iota
CacheFresh
CacheStale
)
func (s CacheStatus) String() string {
switch s {
case CacheFresh:
return "fresh"
case CacheStale:
return "stale"
default:
return "miss"
}
}
// CacheEntry is a stored response body and the time it was stored.
type CacheEntry struct {
Body []byte
StoredAt time.Time
}
// CacheStats is the cache state reported by /healthz.
type CacheStats struct {
Backend string `json:"backend"`
Entries int `json:"entries"`
StaleEntries int `json:"stale_entries"`
Bytes int64 `json:"bytes"`
}
// Cache stores merged responses keyed by cacheKey. Get reports freshness rather
// than hiding expired entries so a caller can fall back to a stale body when the
// upstream fetch fails. The context and error exist for out-of-process backends
// (the reports cache lands on S3); an in-process backend ignores both.
//
// A Body handed back by Get aliases the cache's copy and must not be mutated.
type Cache interface {
Get(ctx context.Context, key string) (CacheEntry, CacheStatus, error)
Put(ctx context.Context, key string, body []byte) error
Stats() CacheStats
}
// cacheKey is the request path when there are no params, else "<path>?<params>"
// where params is url.Values.Encode() over a copy whose repeated values have
// been sorted. Encode() already sorts keys ascending, so both the order params
// arrive in and the order of a repeated param's values are irrelevant to the
// key: one canonical string per distinct request.
func cacheKey(path string, params url.Values) string {
if len(params) == 0 {
return path
}
norm := make(url.Values, len(params))
for k, vs := range params {
sorted := append([]string(nil), vs...)
sort.Strings(sorted)
norm[k] = sorted
}
encoded := norm.Encode()
if encoded == "" {
return path
}
return path + "?" + encoded
}
// noopCache is the default for every path with no cache configured, so wiring a
// handler through the cache leaves its behaviour unchanged.
type noopCache struct{}
func (noopCache) Get(context.Context, string) (CacheEntry, CacheStatus, error) {
return CacheEntry{}, CacheMiss, nil
}
func (noopCache) Put(context.Context, string, []byte) error { return nil }
func (noopCache) Stats() CacheStats { return CacheStats{Backend: "none"} }
type memoryEntry struct {
key string
body []byte
storedAt time.Time
}
// memoryCache is a byte-bounded LRU. Expired entries are kept, not dropped, so
// they remain available as a stale fallback; only the byte budget evicts.
type memoryCache struct {
ttl time.Duration
maxBytes int64
now func() time.Time
mu sync.Mutex
ll *list.List // front = most recently used
items map[string]*list.Element
bytes int64
}
func newMemoryCache(ttl time.Duration, maxBytes int64) *memoryCache {
return &memoryCache{
ttl: ttl,
maxBytes: maxBytes,
now: time.Now,
ll: list.New(),
items: make(map[string]*list.Element),
}
}
func (c *memoryCache) Get(_ context.Context, key string) (CacheEntry, CacheStatus, error) {
c.mu.Lock()
defer c.mu.Unlock()
el, ok := c.items[key]
if !ok {
return CacheEntry{}, CacheMiss, nil
}
c.ll.MoveToFront(el)
e := el.Value.(*memoryEntry)
status := CacheFresh
if c.now().Sub(e.storedAt) >= c.ttl {
status = CacheStale
}
return CacheEntry{Body: e.body, StoredAt: e.storedAt}, status, nil
}
func (c *memoryCache) Put(_ context.Context, key string, body []byte) error {
// A response larger than the whole budget would evict everything else.
if int64(len(body)) > c.maxBytes {
return nil
}
stored := append([]byte(nil), body...)
c.mu.Lock()
defer c.mu.Unlock()
if el, ok := c.items[key]; ok {
e := el.Value.(*memoryEntry)
c.bytes += int64(len(stored)) - int64(len(e.body))
e.body, e.storedAt = stored, c.now()
c.ll.MoveToFront(el)
} else {
c.items[key] = c.ll.PushFront(&memoryEntry{key: key, body: stored, storedAt: c.now()})
c.bytes += int64(len(stored))
}
for c.bytes > c.maxBytes {
back := c.ll.Back()
if back == nil {
break
}
e := c.ll.Remove(back).(*memoryEntry)
delete(c.items, e.key)
c.bytes -= int64(len(e.body))
}
return nil
}
func (c *memoryCache) Stats() CacheStats {
c.mu.Lock()
defer c.mu.Unlock()
st := CacheStats{Backend: "memory", Entries: len(c.items), Bytes: c.bytes}
now := c.now()
for el := c.ll.Front(); el != nil; el = el.Next() {
if now.Sub(el.Value.(*memoryEntry).storedAt) >= c.ttl {
st.StaleEntries++
}
}
return st
}
// flightGroup collapses concurrent identical builds so N simultaneous requests
// for one key cause one upstream fan-out.
type flightGroup struct {
mu sync.Mutex
calls map[string]*flightCall
}
type flightCall struct {
wg sync.WaitGroup
resp cachedResponse
err error
}
// flightPanic is a panic from a flight's fn, reported to the leader and to every
// waiter as an error so callers keep their error handling (stale fallback, 502)
// instead of seeing a zero-value success.
type flightPanic struct {
value any
stack []byte
}
func (p *flightPanic) Error() string {
return fmt.Sprintf("panic building response: %v\n%s", p.value, p.stack)
}
// Do returns fn's result and whether this caller shared another's in-flight run.
func (g *flightGroup) Do(key string, fn func() (cachedResponse, error)) (resp cachedResponse, err error, shared bool) {
g.mu.Lock()
if g.calls == nil {
g.calls = make(map[string]*flightCall)
}
if c, ok := g.calls[key]; ok {
g.mu.Unlock()
c.wg.Wait()
return c.resp, c.err, true
}
c := &flightCall{}
c.wg.Add(1)
g.calls[key] = c
g.mu.Unlock()
defer func() {
if r := recover(); r != nil {
c.resp, c.err = cachedResponse{}, &flightPanic{value: r, stack: debug.Stack()}
resp, err = c.resp, c.err
}
// Done only after the results are stored, so waiters read them.
c.wg.Done()
g.mu.Lock()
delete(g.calls, key)
g.mu.Unlock()
}()
c.resp, c.err = fn()
return c.resp, c.err, false
}
// staleTracker records stale fallbacks for /healthz. serving flips back to false
// as soon as a response is served from a live fan-out or a fresh entry.
type staleTracker struct {
mu sync.Mutex
serving bool
served uint64
last time.Time
}
func (t *staleTracker) markStale(now time.Time) {
t.mu.Lock()
defer t.mu.Unlock()
t.serving = true
t.served++
t.last = now
}
func (t *staleTracker) markFresh() {
t.mu.Lock()
defer t.mu.Unlock()
t.serving = false
}
func (t *staleTracker) snapshot() (serving bool, served uint64, last time.Time) {
t.mu.Lock()
defer t.mu.Unlock()
return t.serving, t.served, t.last
}
+821
View File
@@ -0,0 +1,821 @@
package main
import (
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
// fakeClock drives the cache's TTL without sleeping.
type fakeClock struct {
mu sync.Mutex
t time.Time
}
func newFakeClock() *fakeClock { return &fakeClock{t: time.Unix(1_800_000_000, 0)} }
func (c *fakeClock) now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.t
}
func (c *fakeClock) advance(d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.t = c.t.Add(d)
}
// countingBackend is a PuppetDB stand-in that counts requests per path and can
// be made to fail or block, so cache hits and single-flight are observable.
type countingBackend struct {
srv *httptest.Server
mu sync.Mutex
hits map[string]int
bodies map[string]string
fail bool
block chan struct{}
}
func newCountingBackend(t *testing.T, bodies map[string]string) *countingBackend {
t.Helper()
cb := &countingBackend{hits: map[string]int{}, bodies: bodies}
cb.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
cb.mu.Lock()
cb.hits[r.URL.Path]++
fail, block, body := cb.fail, cb.block, cb.bodies[r.URL.Path]
cb.mu.Unlock()
if block != nil {
<-block
}
if fail {
http.Error(w, "boom", http.StatusInternalServerError)
return
}
if body == "" {
body = "[]"
}
w.Header().Set("Content-Type", "application/json")
_, _ = io.WriteString(w, body)
}))
t.Cleanup(cb.srv.Close)
return cb
}
func (cb *countingBackend) hitCount(path string) int {
cb.mu.Lock()
defer cb.mu.Unlock()
return cb.hits[path]
}
func (cb *countingBackend) setFail(v bool) {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.fail = v
}
func (cb *countingBackend) setBody(path, body string) {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.bodies[path] = body
}
func (cb *countingBackend) setBlock(ch chan struct{}) {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.block = ch
}
// newCachedServer builds a server whose facts cache runs on a fake clock.
func newCachedServer(t *testing.T, cfg Config) (*Server, *fakeClock) {
t.Helper()
srv := newTestServer(cfg)
mc, ok := srv.factsCache.(*memoryCache)
if !ok {
t.Fatalf("expected a memory cache, got %T", srv.factsCache)
}
clk := newFakeClock()
mc.now = clk.now
return srv, clk
}
func cacheTestConfig(aURL, bURL string) Config {
cfg := testConfig(aURL, bURL, mergeStatic)
cfg.FactsTTL = 30 * time.Second
cfg.CacheBytes = defaultCacheSize
return cfg
}
func TestCacheKey_CanonicalOrdering(t *testing.T) {
a := cacheKey(factsPath, url.Values{
"query": {`["=","name","role"]`},
"limit": {"10"},
"expand": {"z", "a"},
})
b := cacheKey(factsPath, url.Values{
"expand": {"a", "z"},
"limit": {"10"},
"query": {`["=","name","role"]`},
})
if a != b {
t.Errorf("param order must not change the key:\n %s\n %s", a, b)
}
if !strings.HasPrefix(a, factsPath+"?") {
t.Errorf("key must start with the path, got %q", a)
}
if got := cacheKey(nodesPath, nil); got != nodesPath {
t.Errorf("no params should give the bare path, got %q", got)
}
if cacheKey(factsPath, url.Values{"query": {"x"}}) == cacheKey(factsPath, url.Values{"query": {"y"}}) {
t.Error("different queries must not share a key")
}
if cacheKey(factsPath, url.Values{"query": {"x"}}) == cacheKey(nodesPath, url.Values{"query": {"x"}}) {
t.Error("different paths must not share a key")
}
}
func TestMemoryCache_FreshThenStale(t *testing.T) {
c := newMemoryCache(30*time.Second, 1<<20)
clk := newFakeClock()
c.now = clk.now
if _, status, _ := c.Get(context.Background(), "k"); status != CacheMiss {
t.Fatalf("empty cache should miss, got %s", status)
}
if err := c.Put(context.Background(), "k", []byte(`{"body":[]}`)); err != nil {
t.Fatal(err)
}
ent, status, err := c.Get(context.Background(), "k")
if err != nil || status != CacheFresh || string(ent.Body) != `{"body":[]}` {
t.Fatalf("want a fresh hit, got %s %q (%v)", status, ent.Body, err)
}
clk.advance(29 * time.Second)
if _, status, _ := c.Get(context.Background(), "k"); status != CacheFresh {
t.Fatalf("inside the TTL should still be fresh, got %s", status)
}
clk.advance(2 * time.Second)
ent, status, _ = c.Get(context.Background(), "k")
if status != CacheStale {
t.Fatalf("past the TTL should be stale, got %s", status)
}
if string(ent.Body) != `{"body":[]}` {
t.Errorf("a stale entry must still carry its body, got %q", ent.Body)
}
if st := c.Stats(); st.Entries != 1 || st.StaleEntries != 1 {
t.Errorf("stats should report 1 entry, 1 stale, got %+v", st)
}
}
func TestMemoryCache_EvictsLeastRecentlyUsed(t *testing.T) {
body := []byte("0123456789") // 10 bytes
c := newMemoryCache(time.Minute, 25)
ctx := context.Background()
for _, k := range []string{"a", "b"} {
if err := c.Put(ctx, k, body); err != nil {
t.Fatal(err)
}
}
// Reading "a" makes "b" the eviction candidate.
if _, status, _ := c.Get(ctx, "a"); status != CacheFresh {
t.Fatalf("a should be cached, got %s", status)
}
if err := c.Put(ctx, "c", body); err != nil {
t.Fatal(err)
}
if _, status, _ := c.Get(ctx, "b"); status != CacheMiss {
t.Errorf("b was least recently used and should have been evicted, got %s", status)
}
for _, k := range []string{"a", "c"} {
if _, status, _ := c.Get(ctx, k); status != CacheFresh {
t.Errorf("%s should have survived eviction, got %s", k, status)
}
}
if st := c.Stats(); st.Entries != 2 || st.Bytes != 20 {
t.Errorf("stats after eviction = %+v, want 2 entries / 20 bytes", st)
}
}
func TestMemoryCache_SkipsOversizedEntry(t *testing.T) {
c := newMemoryCache(time.Minute, 8)
if err := c.Put(context.Background(), "big", []byte("123456789")); err != nil {
t.Fatal(err)
}
if _, status, _ := c.Get(context.Background(), "big"); status != CacheMiss {
t.Error("an entry larger than the whole budget must not be stored")
}
if st := c.Stats(); st.Bytes != 0 {
t.Errorf("bytes = %d, want 0", st.Bytes)
}
}
func TestNoopCache_AlwaysMisses(t *testing.T) {
var c Cache = noopCache{}
if err := c.Put(context.Background(), "k", []byte("x")); err != nil {
t.Fatal(err)
}
if _, status, _ := c.Get(context.Background(), "k"); status != CacheMiss {
t.Errorf("noop cache must always miss, got %s", status)
}
if st := c.Stats(); st.Backend != "none" {
t.Errorf("backend = %q, want none", st.Backend)
}
}
func TestHandler_FactsCacheFreshHit(t *testing.T) {
body := `[` + fact("h1", "role", "web", "") + `]`
a := newCountingBackend(t, map[string]string{factsPath: body})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
first := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`)
second := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`)
if first.Code != http.StatusOK || second.Code != http.StatusOK {
t.Fatalf("statuses %d/%d", first.Code, second.Code)
}
if first.Body.String() != second.Body.String() {
t.Errorf("cache hit changed the body:\n %s\n %s", first.Body.String(), second.Body.String())
}
if got := a.hitCount(factsPath); got != 1 {
t.Errorf("backend a saw %d requests, want 1 (second served from cache)", got)
}
if got := b.hitCount(factsPath); got != 1 {
t.Errorf("backend b saw %d requests, want 1", got)
}
// A different query is a different key and must go upstream.
doGet(t, srv.Handler(), factsPath, `["=","name","osfamily"]`)
if got := a.hitCount(factsPath); got != 2 {
t.Errorf("a different query should refetch: %d requests, want 2", got)
}
}
func TestHandler_FactsCacheExpires(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
doGet(t, srv.Handler(), factsPath, "")
clk.advance(31 * time.Second)
doGet(t, srv.Handler(), factsPath, "")
if got := a.hitCount(factsPath); got != 2 {
t.Errorf("an expired entry should refetch: %d requests, want 2", got)
}
}
func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "old", "") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
warm := doGet(t, srv.Handler(), factsPath, "")
if warm.Code != http.StatusOK {
t.Fatalf("warm-up status %d", warm.Code)
}
if !strings.Contains(warm.Body.String(), `"old"`) {
t.Fatalf("warm-up body = %s", warm.Body.String())
}
// Backends healthy but the entry expired: fresh data wins, never the stale copy.
clk.advance(31 * time.Second)
a.setBody(factsPath, `[`+fact("h1", "role", "new", "")+`]`)
refetch := doGet(t, srv.Handler(), factsPath, "")
if !strings.Contains(refetch.Body.String(), `"new"`) {
t.Errorf("a healthy backend must not be shadowed by the stale entry: %s", refetch.Body.String())
}
if serving, _, _ := srv.stale.snapshot(); serving {
t.Error("serving_stale must stay false while backends are healthy")
}
// Every backend down and the entry expired: the stale copy is served.
clk.advance(31 * time.Second)
a.setFail(true)
b.setFail(true)
stale := doGet(t, srv.Handler(), factsPath, "")
if stale.Code != http.StatusOK {
t.Fatalf("stale fallback status %d: %s", stale.Code, stale.Body.String())
}
if !strings.Contains(stale.Body.String(), `"new"`) {
t.Errorf("stale body = %s, want the last cached copy", stale.Body.String())
}
serving, served, last := srv.stale.snapshot()
if !serving || served != 1 || last.IsZero() {
t.Errorf("stale tracker = %v/%d/%v, want serving=true served=1", serving, served, last)
}
}
func TestHandler_NoCacheEntryMeansBackendFailureIs502(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
a.setFail(true)
b.setFail(true)
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
rec := doGet(t, srv.Handler(), factsPath, "")
if rec.Code != http.StatusBadGateway {
t.Fatalf("status %d, want 502", rec.Code)
}
if !strings.Contains(rec.Body.String(), "all backends failed") {
t.Errorf("body = %q", rec.Body.String())
}
}
func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
h := srv.Handler()
release := make(chan struct{})
a.setBlock(release)
b.setBlock(release)
const callers = 16
var wg sync.WaitGroup
codes := make([]int, callers)
for i := range callers {
wg.Add(1)
go func(i int) {
defer wg.Done()
codes[i] = doGet(t, h, factsPath, "").Code
}(i)
}
// The leader is parked inside the blocked backend, so every caller that
// reaches the handler before the release joins its flight.
waitFor(t, func() bool { return a.hitCount(factsPath) >= 1 })
time.Sleep(250 * time.Millisecond)
close(release)
wg.Wait()
for i, code := range codes {
if code != http.StatusOK {
t.Fatalf("caller %d got %d", i, code)
}
}
if got := a.hitCount(factsPath); got != 1 {
t.Errorf("backend a saw %d requests, want 1 for %d concurrent callers", got, callers)
}
if got := b.hitCount(factsPath); got != 1 {
t.Errorf("backend b saw %d requests, want 1 for %d concurrent callers", got, callers)
}
}
func TestFlightGroup_LeaderRunsOnce(t *testing.T) {
var g flightGroup
var calls, shared atomic.Int64
entered := make(chan struct{})
release := make(chan struct{})
run := func() (cachedResponse, error) {
if calls.Add(1) == 1 {
close(entered)
}
<-release
return cachedResponse{Body: json.RawMessage(`[]`), Records: -1}, nil
}
var wg sync.WaitGroup
for range 16 {
wg.Add(1)
go func() {
defer wg.Done()
if _, _, s := g.Do("k", run); s {
shared.Add(1)
}
}()
}
<-entered
time.Sleep(250 * time.Millisecond)
close(release)
wg.Wait()
if calls.Load() != 1 {
t.Errorf("fn ran %d times, want 1", calls.Load())
}
if shared.Load() != 15 {
t.Errorf("%d callers shared the flight, want 15", shared.Load())
}
// The key is released once the flight finishes.
if _, _, s := g.Do("k", func() (cachedResponse, error) { return cachedResponse{Records: -1}, nil }); s {
t.Error("a later call must start its own flight")
}
}
func TestFlightGroup_LeaderPanicFailsLeaderAndWaiters(t *testing.T) {
var g flightGroup
entered := make(chan struct{})
release := make(chan struct{})
type result struct {
resp cachedResponse
err error
shared bool
}
waiters := make([]result, 8)
var wg sync.WaitGroup
var leader result
wg.Add(1)
go func() {
defer wg.Done()
leader.resp, leader.err, leader.shared = g.Do("k", func() (cachedResponse, error) {
close(entered)
<-release
panic("build exploded")
})
}()
<-entered
for i := range waiters {
wg.Add(1)
go func(i int) {
defer wg.Done()
waiters[i].resp, waiters[i].err, waiters[i].shared = g.Do("k", func() (cachedResponse, error) {
t.Error("a waiter must not run its own fn")
return cachedResponse{}, nil
})
}(i)
}
// Park the waiters in Wait() before the leader panics.
time.Sleep(250 * time.Millisecond)
close(release)
wg.Wait()
if leader.err == nil {
t.Errorf("leader err = nil, want a panic error (resp %+v)", leader.resp)
}
for i, w := range waiters {
if !w.shared {
t.Errorf("waiter %d did not share the flight", i)
}
if w.err == nil {
t.Fatalf("waiter %d unblocked with err = nil and resp %+v, want an error", i, w.resp)
}
if !strings.Contains(w.err.Error(), "build exploded") {
t.Errorf("waiter %d err = %v, want the panic value", i, w.err)
}
}
// The key is released on the panic path, so a later call leads its own flight.
resp, err, shared := g.Do("k", func() (cachedResponse, error) {
return cachedResponse{Body: json.RawMessage(`[]`), Records: 0}, nil
})
if shared {
t.Error("a call after a panicking flight must start its own flight")
}
if err != nil || resp.Records != 0 {
t.Errorf("later call = (%+v, %v), want a clean success", resp, err)
}
}
func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
panicBuild := func() (cachedResponse, error) { panic("build exploded") }
serve := func(build func() (cachedResponse, error)) *httptest.ResponseRecorder {
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, factsPath, nil)
srv.serveCached(rec, req, factsPath, nil, build)
return rec
}
// No entry to fall back to: a panicking build must not look like a success.
rec := serve(panicBuild)
if rec.Code != http.StatusBadGateway {
t.Fatalf("status %d (%s), want 502", rec.Code, rec.Body.String())
}
stored := `[` + fact("h1", "role", "web", "") + `]`
rec = serve(func() (cachedResponse, error) {
return cachedResponse{Body: json.RawMessage(stored), Records: 1}, nil
})
if rec.Code != http.StatusOK {
t.Fatalf("priming status %d (%s)", rec.Code, rec.Body.String())
}
clk.advance(31 * time.Second)
rec = serve(panicBuild)
if rec.Code != http.StatusOK {
t.Fatalf("stale fallback status %d (%s), want 200", rec.Code, rec.Body.String())
}
if got := strings.TrimSpace(rec.Body.String()); got != stored {
t.Errorf("stale body = %s, want %s", got, stored)
}
if got := rec.Header().Get(recordsHeader); got != "1" {
t.Errorf("%s = %q, want 1", recordsHeader, got)
}
if hr := health(t, srv); !hr.Cache.ServingStale || hr.Cache.StaleServed != 1 {
t.Errorf("panic fallback not counted as stale: %+v", hr.Cache)
}
}
func TestServeCached_BuildPanicFailsConcurrentWaiters(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
entered := make(chan struct{})
release := make(chan struct{})
serve := func(build func() (cachedResponse, error)) *httptest.ResponseRecorder {
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, factsPath, nil)
srv.serveCached(rec, req, factsPath, nil, build)
return rec
}
var wg sync.WaitGroup
wg.Add(1)
var leaderRec *httptest.ResponseRecorder
go func() {
defer wg.Done()
leaderRec = serve(func() (cachedResponse, error) {
close(entered)
<-release
panic("build exploded")
})
}()
<-entered
const waiters = 8
recs := make([]*httptest.ResponseRecorder, waiters)
for i := range waiters {
wg.Add(1)
go func(i int) {
defer wg.Done()
recs[i] = serve(func() (cachedResponse, error) {
t.Error("a waiter must not run its own build")
return cachedResponse{}, nil
})
}(i)
}
time.Sleep(250 * time.Millisecond)
close(release)
wg.Wait()
if leaderRec.Code != http.StatusBadGateway {
t.Errorf("leader status %d, want 502", leaderRec.Code)
}
for i, rec := range recs {
if rec.Code != http.StatusBadGateway {
t.Fatalf("waiter %d got %d (%s), want 502 rather than an empty success",
i, rec.Code, rec.Body.String())
}
}
}
func TestHandler_HealthzReportsCacheState(t *testing.T) {
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
hr := health(t, srv)
if hr.Cache.Backend != "memory" || hr.Cache.TTL != "30s" {
t.Fatalf("cache health = %+v, want memory/30s", hr.Cache)
}
if hr.Cache.Entries != 0 || hr.Cache.ServingStale {
t.Fatalf("a cold cache should be empty and not stale: %+v", hr.Cache)
}
doGet(t, srv.Handler(), factsPath, "")
hr = health(t, srv)
if hr.Cache.Entries != 1 || hr.Cache.StaleEntries != 0 || hr.Cache.Bytes == 0 {
t.Fatalf("after one request: %+v", hr.Cache)
}
clk.advance(31 * time.Second)
if hr = health(t, srv); hr.Cache.StaleEntries != 1 {
t.Fatalf("expired entry should count as stale: %+v", hr.Cache)
}
if hr.Cache.ServingStale {
t.Error("an expired entry alone is not serving_stale")
}
a.setFail(true)
b.setFail(true)
doGet(t, srv.Handler(), factsPath, "")
hr = health(t, srv)
if !hr.Cache.ServingStale || hr.Cache.StaleServed != 1 || hr.Cache.LastStale == "" {
t.Errorf("staleness not surfaced in /healthz: %+v", hr.Cache)
}
if hr.Status != "down" {
t.Errorf("status = %q, want down", hr.Status)
}
}
func TestHandler_HealthzReportsDisabledCache(t *testing.T) {
a := newCountingBackend(t, nil)
b := newCountingBackend(t, nil)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) // FactsTTL zero
hr := health(t, srv)
if hr.Cache.Backend != "none" || hr.Cache.TTL != "0" {
t.Errorf("disabled cache health = %+v, want none/0", hr.Cache)
}
}
func TestHandler_UncachedPathsStillRefetch(t *testing.T) {
a := newCountingBackend(t, map[string]string{reportsPath: `[{"hash":"h"}]`})
b := newCountingBackend(t, map[string]string{reportsPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
doGet(t, srv.Handler(), reportsPath, "")
doGet(t, srv.Handler(), reportsPath, "")
if got := a.hitCount(reportsPath); got != 2 {
t.Errorf("/reports is uncached: %d requests, want 2", got)
}
}
func TestHandler_NodesRecordSetCached(t *testing.T) {
a := newCountingBackend(t, map[string]string{nodesPath: `[{"certname":"h1"}]`})
b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
doGet(t, srv.Handler(), nodesPath, "")
doGet(t, srv.Handler(), nodesPath, "")
if got := a.hitCount(nodesPath); got != 1 {
t.Errorf("merged /nodes is cached: %d requests, want 1", got)
}
}
func TestHandler_NodesAggregateNotCached(t *testing.T) {
a := newCountingBackend(t, map[string]string{nodesPath: `[{"count":90}]`})
b := newCountingBackend(t, map[string]string{nodesPath: `[{"count":53}]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
first := doGet(t, srv.Handler(), nodesPath, nodeCountQuery)
second := doGet(t, srv.Handler(), nodesPath, nodeCountQuery)
if first.Code != http.StatusOK || second.Code != http.StatusOK {
t.Fatalf("statuses %d/%d", first.Code, second.Code)
}
if got := a.hitCount(nodesPath); got != 2 {
t.Errorf("/nodes aggregates are uncached: %d requests, want 2", got)
}
if got := counts(t, second.Body.Bytes(), "count"); !slices.Equal(got, []float64{143}) {
t.Errorf("count = %v, want [143]", got)
}
}
func TestHandler_ResourcesAggregateNotCached(t *testing.T) {
a := newCountingBackend(t, map[string]string{resourcesPath: `[{"count":7}]`})
b := newCountingBackend(t, map[string]string{resourcesPath: `[{"count":5}]`})
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
doGet(t, srv.Handler(), resourcesPath, nodeCountQuery)
doGet(t, srv.Handler(), resourcesPath, nodeCountQuery)
if got := a.hitCount(resourcesPath); got != 2 {
t.Errorf("/resources aggregates are uncached: %d requests, want 2", got)
}
}
func TestHandler_MetaAndMetricsNotCached(t *testing.T) {
bodies := map[string]string{
metaVersionPath: `{"version":"8.4.0"}`,
metaServerTimePath: `{"server_time":"2026-01-01T00:00:00.000Z"}`,
"/metrics/v2/read/x": `{"request":{},"value":{"Value":1},"status":200}`,
}
a := newCountingBackend(t, bodies)
b := newCountingBackend(t, bodies)
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
for path := range bodies {
doGet(t, srv.Handler(), path, "")
doGet(t, srv.Handler(), path, "")
if got := a.hitCount(path); got != 2 {
t.Errorf("%s is uncached: %d requests, want 2", path, got)
}
}
}
func TestConfig_FactsTTLClampedAtCap(t *testing.T) {
dir := t.TempDir()
t.Setenv("XDG_CONFIG_HOME", dir)
clearEnv(t)
path := filepath.Join(dir, appName, configFileName)
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte("facts_ttl: 5m\n"), 0o644); err != nil {
t.Fatal(err)
}
cfg, err := Load("")
if err != nil {
t.Fatal(err)
}
if cfg.FactsTTL != maxFactsTTL {
t.Errorf("file facts_ttl = %s, want it clamped to %s", cfg.FactsTTL, maxFactsTTL)
}
t.Setenv(envPrefix+"FACTS_TTL", "10m")
cfg, err = Load("")
if err != nil {
t.Fatal(err)
}
if cfg.FactsTTL != maxFactsTTL {
t.Errorf("env facts_ttl = %s, want it clamped to %s", cfg.FactsTTL, maxFactsTTL)
}
if cfg.factsTTLClamped != 10*time.Minute {
t.Errorf("pre-clamp value = %s, want 10m0s", cfg.factsTTLClamped)
}
out := captureStdout(t, func() { printConfig(cfg) })
if !strings.Contains(out, "facts_ttl : 30s (clamped from 600s, cap 30s)") {
t.Errorf("config show must report the clamp, got:\n%s", out)
}
t.Setenv(envPrefix+"FACTS_TTL", "5s")
cfg, err = Load("")
if err != nil {
t.Fatal(err)
}
if cfg.FactsTTL != 5*time.Second || cfg.factsTTLClamped != 0 {
t.Errorf("a value under the cap must pass through: %s / %s", cfg.FactsTTL, cfg.factsTTLClamped)
}
t.Setenv(envPrefix+"FACTS_TTL", "0s")
cfg, err = Load("")
if err != nil {
t.Fatal(err)
}
if cfg.cacheEnabled() {
t.Error("facts_ttl 0 must disable the cache")
}
out = captureStdout(t, func() { printConfig(cfg) })
if !strings.Contains(out, "(cache disabled)") {
t.Errorf("config show must say the cache is off, got:\n%s", out)
}
}
func TestNewServer_ClampsFactsTTL(t *testing.T) {
cfg := testConfig("http://a.invalid", "http://b.invalid", mergeStatic)
cfg.FactsTTL = time.Hour
cfg.CacheBytes = 1 << 20
srv := newTestServer(cfg)
mc, ok := srv.factsCache.(*memoryCache)
if !ok {
t.Fatalf("expected a memory cache, got %T", srv.factsCache)
}
if mc.ttl != maxFactsTTL {
t.Errorf("cache ttl = %s, want %s", mc.ttl, maxFactsTTL)
}
}
func TestConfig_ValidateRejectsNegativeCacheSettings(t *testing.T) {
base := testConfig("http://a.invalid", "http://b.invalid", mergeStatic)
for name, mutate := range map[string]func(*Config){
"negative facts_ttl": func(c *Config) { c.FactsTTL = -time.Second },
"negative facts_cache_bytes": func(c *Config) { c.CacheBytes = -1 },
} {
cfg := base
mutate(&cfg)
if err := cfg.Validate(); err == nil {
t.Errorf("%s should not validate", name)
}
}
}
func health(t *testing.T, srv *Server) healthReport {
t.Helper()
rec := doGet(t, srv.Handler(), "/healthz", "")
var hr healthReport
if err := json.Unmarshal(rec.Body.Bytes(), &hr); err != nil {
t.Fatalf("decode healthz: %v (%s)", err, rec.Body.String())
}
return hr
}
func waitFor(t *testing.T, cond func() bool) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for !cond() {
if time.Now().After(deadline) {
t.Fatal("condition not met within 5s")
}
time.Sleep(time.Millisecond)
}
}
+47 -2
View File
@@ -25,6 +25,13 @@ const (
defaultTimeout = 10 * time.Second
defaultFreshnessTTL = 30 * time.Second
// maxFactsTTL is a hard cap, not a default: a larger configured value is
// clamped down to it rather than rejected, so a stray env var cannot make a
// container crash-loop.
maxFactsTTL = 30 * time.Second
defaultFactsTTL = 30 * time.Second
defaultCacheSize = int64(64 << 20)
)
var exampleBackends = []Backend{
@@ -43,8 +50,11 @@ type Config struct {
Merge string `yaml:"merge"`
Timeout time.Duration `yaml:"timeout"`
FreshnessTTL time.Duration `yaml:"freshness_ttl"`
FactsTTL time.Duration `yaml:"facts_ttl"` // 0 disables the /facts+/nodes cache
CacheBytes int64 `yaml:"facts_cache_bytes"` // byte budget for that cache
sourcePath string // file this config was read from, empty if none was found
sourcePath string // file this config was read from, empty if none was found
factsTTLClamped time.Duration // pre-clamp facts_ttl, zero when nothing was clamped
}
// SourcePath returns the config file Load read, or "" when none was loaded.
@@ -61,6 +71,8 @@ func DefaultConfig() Config {
Merge: mergeFreshness,
Timeout: defaultTimeout,
FreshnessTTL: defaultFreshnessTTL,
FactsTTL: defaultFactsTTL,
CacheBytes: defaultCacheSize,
}
}
@@ -139,9 +151,19 @@ func Load(flagPath string) (Config, error) {
}
applyEnv(&cfg, os.Getenv)
cfg.clampFactsTTL()
return cfg, nil
}
// clampFactsTTL pins facts_ttl to maxFactsTTL, remembering the configured value
// so `config show` can say the cap was applied.
func (c *Config) clampFactsTTL() {
if c.FactsTTL > maxFactsTTL {
c.factsTTLClamped = c.FactsTTL
c.FactsTTL = maxFactsTTL
}
}
func applyEnv(cfg *Config, getenv func(string) string) {
if v := getenv(envPrefix + "LISTEN"); v != "" {
cfg.Listen = v
@@ -159,6 +181,16 @@ func applyEnv(cfg *Config, getenv func(string) string) {
cfg.FreshnessTTL = d
}
}
if v := getenv(envPrefix + "FACTS_TTL"); v != "" {
if d, err := time.ParseDuration(v); err == nil {
cfg.FactsTTL = d
}
}
if v := getenv(envPrefix + "FACTS_CACHE_BYTES"); v != "" {
if n, err := strconv.ParseInt(v, 10, 64); err == nil {
cfg.CacheBytes = n
}
}
if v := getenv(envPrefix + "BACKENDS"); v != "" {
if bs := parseBackends(v); len(bs) > 0 {
cfg.Backends = bs
@@ -216,9 +248,19 @@ func (c Config) Validate() error {
if c.Timeout <= 0 {
return fmt.Errorf("timeout must be positive")
}
if c.FactsTTL < 0 {
return fmt.Errorf("facts_ttl must not be negative (0 disables the cache)")
}
if c.CacheBytes < 0 {
return fmt.Errorf("facts_cache_bytes must not be negative")
}
return nil
}
// cacheEnabled reports whether a facts/nodes cache should be built: both a TTL
// and a byte budget are required.
func (c Config) cacheEnabled() bool { return c.FactsTTL > 0 && c.CacheBytes > 0 }
func writeDefaultConfig(path string) error {
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
return fmt.Errorf("creating config dir: %w", err)
@@ -231,7 +273,10 @@ func writeDefaultConfig(path string) error {
"# A merging proxy presenting one PuppetDB v4 query surface over several\n" +
"# PuppetDB backends. The backend URLs below are placeholders — edit them.\n" +
"# Env overrides: PDBMUX_LISTEN, PDBMUX_MERGE, PDBMUX_TIMEOUT,\n" +
"# PDBMUX_FRESHNESS_TTL, PDBMUX_BACKENDS (name=url,name=url).\n\n")
"# PDBMUX_FRESHNESS_TTL, PDBMUX_FACTS_TTL, PDBMUX_FACTS_CACHE_BYTES,\n" +
"# PDBMUX_BACKENDS (name=url,name=url).\n" +
"# facts_ttl caches merged /facts and /nodes in memory; it is capped at 30s\n" +
"# (a larger value is clamped) and 0 disables the cache.\n\n")
if err := os.WriteFile(path, append(header, data...), 0o644); err != nil {
return fmt.Errorf("writing config: %w", err)
}
+1 -1
View File
@@ -391,7 +391,7 @@ func captureStdout(t *testing.T, f func()) string {
func clearEnv(t *testing.T) {
t.Helper()
for _, k := range []string{"CONFIG", "LISTEN", "MERGE", "TIMEOUT", "FRESHNESS_TTL", "BACKENDS"} {
for _, k := range []string{"CONFIG", "LISTEN", "MERGE", "TIMEOUT", "FRESHNESS_TTL", "FACTS_TTL", "FACTS_CACHE_BYTES", "BACKENDS"} {
t.Setenv(envPrefix+k, "")
}
}
+14
View File
@@ -151,6 +151,18 @@ func runServer(cfg Config) error {
}
}
func factsTTLString(cfg Config) string {
s := durationString(cfg.FactsTTL)
switch {
case cfg.factsTTLClamped > 0:
return fmt.Sprintf("%s (clamped from %s, cap %s)",
s, durationString(cfg.factsTTLClamped), durationString(maxFactsTTL))
case !cfg.cacheEnabled():
return s + " (cache disabled)"
}
return s
}
func printConfig(cfg Config) {
if p := cfg.SourcePath(); p != "" {
fmt.Printf("config file : %s (loaded)\n", p)
@@ -161,6 +173,8 @@ func printConfig(cfg Config) {
fmt.Printf("merge : %s\n", cfg.Merge)
fmt.Printf("timeout : %s\n", durationString(cfg.Timeout))
fmt.Printf("freshness_ttl: %s\n", durationString(cfg.FreshnessTTL))
fmt.Printf("facts_ttl : %s\n", factsTTLString(cfg))
fmt.Printf("facts_cache : %d bytes\n", cfg.CacheBytes)
fmt.Println("backends:")
for _, b := range cfg.Backends {
fmt.Printf(" - %-8s %s\n", b.Name, b.URL)
+195
View File
@@ -0,0 +1,195 @@
package main
import (
"context"
"encoding/json"
"io"
"net/http"
"strconv"
"strings"
"sync"
)
const (
metaVersionPath = "/pdb/meta/v1/version"
metaServerTimePath = "/pdb/meta/v1/server-time"
)
// rawResult is one backend's verbatim response, for endpoints whose payload is
// not a PuppetDB record array.
type rawResult struct {
name string
status int
contentType string
body []byte
err error
}
// ok reports whether the backend answered 2xx.
func (r rawResult) ok() bool {
return r.err == nil && r.status >= 200 && r.status < 300
}
// fanOutRaw asks every backend for path concurrently and returns one result per
// backend, in configured order, without interpreting the bodies.
func (s *Server) fanOutRaw(ctx context.Context, path, rawQuery string) []rawResult {
results := make([]rawResult, len(s.cfg.Backends))
var wg sync.WaitGroup
for i, b := range s.cfg.Backends {
wg.Add(1)
go func(i int, b Backend) {
defer wg.Done()
results[i] = s.rawBackend(ctx, b, path, rawQuery)
}(i, b)
}
wg.Wait()
return results
}
func (s *Server) rawBackend(ctx context.Context, b Backend, path, rawQuery string) rawResult {
target := strings.TrimRight(b.URL, "/") + path
if rawQuery != "" {
target += "?" + rawQuery
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
return rawResult{name: b.Name, err: err}
}
resp, err := s.client.Do(req)
if err != nil {
return rawResult{name: b.Name, err: err}
}
defer func() { _ = resp.Body.Close() }()
body, err := io.ReadAll(resp.Body)
if err != nil {
return rawResult{name: b.Name, err: err}
}
return rawResult{
name: b.Name,
status: resp.StatusCode,
contentType: resp.Header.Get("Content-Type"),
body: body,
}
}
// aliveRaw drops backends that errored or answered non-2xx, writing a 502 and
// returning ok=false only when none is left.
func (s *Server) aliveRaw(w http.ResponseWriter, results []rawResult, path string) ([]rawResult, bool) {
var alive []rawResult
for _, res := range results {
if !res.ok() {
s.log.Printf("warning: backend %q failed for %s: %s", res.name, path, res.reason())
continue
}
alive = append(alive, res)
}
if len(alive) == 0 {
http.Error(w, "all backends failed", http.StatusBadGateway)
return nil, false
}
return alive, true
}
func (r rawResult) reason() string {
if r.err != nil {
return r.err.Error()
}
return "HTTP " + strconv.Itoa(r.status) + ": " + strings.TrimSpace(string(r.body))
}
// handleMetaVersion serves /pdb/meta/v1/version. Clients (pypuppetdb, and so
// Puppetboard's startup check) treat the answer as the feature level they may
// rely on, so the merged answer is the *lowest* version any backend reports:
// the estate can only be counted on for what its oldest member implements.
func (s *Server) handleMetaVersion(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only GET is supported", http.StatusMethodNotAllowed)
return
}
alive, ok := s.aliveRaw(w, s.fanOutRaw(r.Context(), metaVersionPath, r.URL.RawQuery), metaVersionPath)
if !ok {
return
}
lowest := alive[0]
lowestVer, hasVer := metaField(lowest.body, "version")
for _, res := range alive[1:] {
v, ok := metaField(res.body, "version")
if !ok {
continue
}
if !hasVer || compareVersions(v, lowestVer) < 0 {
lowest, lowestVer, hasVer = res, v, true
}
}
writeRaw(w, lowest)
}
// handleMetaServerTime serves /pdb/meta/v1/server-time. The clock of whichever
// PuppetDB answered is not estate state and does not merge, so the first
// reachable backend in configured order supplies it — the same tie-break rule
// used everywhere else.
func (s *Server) handleMetaServerTime(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only GET is supported", http.StatusMethodNotAllowed)
return
}
alive, ok := s.aliveRaw(w, s.fanOutRaw(r.Context(), metaServerTimePath, r.URL.RawQuery), metaServerTimePath)
if !ok {
return
}
writeRaw(w, alive[0])
}
// metaField pulls a string field out of a `{"version": "..."}`-shaped body.
func metaField(body []byte, field string) (string, bool) {
var obj map[string]json.RawMessage
if json.Unmarshal(body, &obj) != nil {
return "", false
}
var s string
if json.Unmarshal(obj[field], &s) != nil || s == "" {
return "", false
}
return s, true
}
// compareVersions orders dotted version strings segment by segment, comparing
// numerically where both segments are numbers and lexically otherwise, so
// "7.12.1" sorts below "8.4.0" and below "7.12.2". A prefix is lower than a
// longer string sharing it ("7.12" < "7.12.1"), and a pre-release suffix is
// compared as text within its segment ("8.0.0" < "8.0.0-SNAPSHOT").
func compareVersions(a, b string) int {
as, bs := strings.Split(a, "."), strings.Split(b, ".")
for i := 0; i < len(as) && i < len(bs); i++ {
an, aok := strconv.Atoi(as[i])
bn, bok := strconv.Atoi(bs[i])
if aok == nil && bok == nil {
if an != bn {
return sign(an - bn)
}
continue
}
if c := strings.Compare(as[i], bs[i]); c != 0 {
return c
}
}
return sign(len(as) - len(bs))
}
func sign(n int) int {
switch {
case n < 0:
return -1
case n > 0:
return 1
default:
return 0
}
}
func writeRaw(w http.ResponseWriter, res rawResult) {
setContentType(w, res.contentType)
w.WriteHeader(res.status)
_, _ = w.Write(res.body)
}
+170
View File
@@ -0,0 +1,170 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
)
func metaGet(t *testing.T, h http.Handler, path string) *httptest.ResponseRecorder {
t.Helper()
rec := httptest.NewRecorder()
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, path, nil))
return rec
}
func metaString(t *testing.T, body []byte, field string) string {
t.Helper()
var obj map[string]string
if err := json.Unmarshal(body, &obj); err != nil {
t.Fatalf("unmarshal %s: %v", body, err)
}
return obj[field]
}
func TestMetaVersion_BackendsAgree(t *testing.T) {
// Puppetboard's check_db_version() calls this at import and exits 2 on any
// non-200, so a 404 here is the difference between a running dashboard and
// CrashLoopBackOff.
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[metaVersionPath] = `{"version":"7.12.1"}`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[metaVersionPath] = `{"version":"7.12.1"}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), metaVersionPath)
if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
if got := metaString(t, rec.Body.Bytes(), "version"); got != "7.12.1" {
t.Errorf("version = %q, want 7.12.1", got)
}
}
func TestMetaVersion_DisagreementReportsLowest(t *testing.T) {
// The estate can only be relied on for what its oldest PuppetDB implements.
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[metaVersionPath] = `{"version":"8.4.0"}`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[metaVersionPath] = `{"version":"7.12.1"}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if got := metaString(t, metaGet(t, srv.Handler(), metaVersionPath).Body.Bytes(), "version"); got != "7.12.1" {
t.Errorf("version = %q, want the lower 7.12.1", got)
}
}
func TestMetaVersion_LowestIsIndependentOfBackendOrder(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[metaVersionPath] = `{"version":"7.12.1"}`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[metaVersionPath] = `{"version":"8.4.0"}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if got := metaString(t, metaGet(t, srv.Handler(), metaVersionPath).Body.Bytes(), "version"); got != "7.12.1" {
t.Errorf("version = %q, want the lower 7.12.1", got)
}
}
func TestMetaVersion_OneBackendDown(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.fail = true
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[metaVersionPath] = `{"version":"8.4.0"}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), metaVersionPath)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200 serving the survivor, got %d", rec.Code)
}
if got := metaString(t, rec.Body.Bytes(), "version"); got != "8.4.0" {
t.Errorf("version = %q, want 8.4.0", got)
}
}
func TestMetaVersion_AllBackendsDown(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.fail = true
b := newFakeBackend(t, `[]`, `[]`)
b.fail = true
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := metaGet(t, srv.Handler(), metaVersionPath); rec.Code != http.StatusBadGateway {
t.Errorf("status %d, want 502", rec.Code)
}
}
func TestMetaServerTime_FirstReachableBackend(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T01:00:00.000Z"}`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T02:00:00.000Z"}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
got := metaString(t, metaGet(t, srv.Handler(), metaServerTimePath).Body.Bytes(), "server_time")
if got != "2026-08-29T01:00:00.000Z" {
t.Errorf("server_time = %q, want the first backend's", got)
}
}
func TestMetaServerTime_SkipsDeadBackend(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.fail = true
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T02:00:00.000Z"}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), metaServerTimePath)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200 serving the survivor, got %d", rec.Code)
}
if got := metaString(t, rec.Body.Bytes(), "server_time"); got != "2026-08-29T02:00:00.000Z" {
t.Errorf("server_time = %q, want the survivor's", got)
}
}
func TestMetaVersion_RejectsNonGET(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := httptest.NewRecorder()
srv.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, metaVersionPath, nil))
if rec.Code != http.StatusMethodNotAllowed {
t.Errorf("status %d, want 405", rec.Code)
}
}
func TestCompareVersions(t *testing.T) {
cases := []struct {
a, b string
want int
}{
{"7.12.1", "7.12.1", 0},
{"7.12.1", "8.4.0", -1},
{"8.4.0", "7.12.1", 1},
{"7.9.0", "7.12.0", -1}, // numeric, not lexical: 9 < 12
{"7.12", "7.12.1", -1},
{"8.0.0", "8.0.0-SNAPSHOT", -1},
{"8.0.0-SNAPSHOT", "8.0.0", 1},
}
for _, c := range cases {
if got := compareVersions(c.a, c.b); got != c.want {
t.Errorf("compareVersions(%q, %q) = %d, want %d", c.a, c.b, got, c.want)
}
}
}
func TestMetaField_MalformedBodyIgnored(t *testing.T) {
// A backend serving junk must not become the "lowest" version.
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[metaVersionPath] = `not json`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[metaVersionPath] = `{"version":"8.4.0"}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if got := metaString(t, metaGet(t, srv.Handler(), metaVersionPath).Body.Bytes(), "version"); got != "8.4.0" {
t.Errorf("version = %q, want 8.4.0 from the only parseable backend", got)
}
}
+272
View File
@@ -0,0 +1,272 @@
package main
import (
"bytes"
"encoding/json"
"net/http"
"strconv"
"strings"
)
// metricsPrefix covers PuppetDB's Jolokia surface, which sits at the server root
// rather than under /pdb: pypuppetdb's metric() reads /metrics/v2/read/<mbean>,
// lists via /metrics/v2/list, and falls back to /metrics/v1/mbeans/<mbean>.
const metricsPrefix = "/metrics/"
type mergeRule int
const (
ruleSum mergeRule = iota
ruleMin
ruleMax
ruleMean
)
// mergeRuleFor picks how one numeric MBean attribute combines across backends.
// The default is a sum: the metrics Puppetboard renders are population counts
// (num-nodes, num-resources, queue depth, command totals) whose estate-wide
// value is the total. The exceptions are attributes describing a distribution or
// a bound rather than a quantity, where adding two servers' numbers yields a
// figure that was never true of either.
func mergeRuleFor(attr string) mergeRule {
a := strings.ToLower(attr)
switch a {
case "min":
return ruleMin
case "max", "uptime", "starttime":
return ruleMax
case "mean", "median", "stddev":
return ruleMean
}
if strings.HasSuffix(a, "percentile") {
return ruleMean
}
return ruleSum
}
func (s *Server) handleMetrics(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "only GET is supported", http.StatusMethodNotAllowed)
return
}
// MBean names carry Jolokia's !-escapes under percent-encoding; the raw path
// is forwarded so neither layer is lost.
path := r.URL.EscapedPath()
alive, ok := s.aliveRaw(w, s.fanOutRaw(r.Context(), path, r.URL.RawQuery), path)
if !ok {
return
}
body, ok := mergeMetrics(alive, metricAttribute(r.URL.Path))
if !ok {
// Every backend answered 2xx but none carried a mergeable payload — a
// Jolokia error envelope, or a body that is not a JSON object. Replaying
// the first keeps the upstream error text the client expects.
writeRaw(w, alive[0])
return
}
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write(body)
}
// metricAttribute names the single attribute a read asked for, when the URL
// carries one (/metrics/v2/read/<mbean>/<attribute>), so a scalar response body
// still gets the right numeric rule. Empty means the response is an object whose
// own keys name its attributes.
func metricAttribute(path string) string {
rest, ok := strings.CutPrefix(path, metricsPrefix)
if !ok {
return ""
}
parts := strings.Split(rest, "/") // v2/read/<mbean>[/<attribute>]
if len(parts) < 4 {
return ""
}
return parts[len(parts)-1]
}
// mergeMetrics folds the backends' Jolokia responses into one. A response is
// either a Jolokia envelope ({"request":…,"value":…,"status":200}), where only
// "value" merges and the rest comes from the first backend, or a bare attribute
// object (metrics/v1), which merges whole. ok=false means nothing was mergeable.
func mergeMetrics(alive []rawResult, attr string) ([]byte, bool) {
var objs []map[string]json.RawMessage
for _, res := range alive {
var obj map[string]json.RawMessage
if decodeJSON(res.body, &obj) != nil || obj == nil {
continue
}
// Jolokia reports a per-request failure inside an HTTP 200.
if n, ok := numberOf(obj["status"]); ok && (n < 200 || n >= 300) {
continue
}
objs = append(objs, obj)
}
if len(objs) == 0 {
return nil, false
}
_, hasValue := objs[0]["value"]
_, hasStatus := objs[0]["status"]
if !hasValue || !hasStatus {
vals := make([]any, 0, len(objs))
for _, obj := range objs {
vals = append(vals, decodedObject(obj))
}
return marshal(mergeMetricValue(vals, attr))
}
vals := make([]any, 0, len(objs))
for _, obj := range objs {
var v any
if decodeJSON(obj["value"], &v) == nil {
vals = append(vals, v)
}
}
if len(vals) == 0 {
return nil, false
}
merged, ok := marshal(mergeMetricValue(vals, attr))
if !ok {
return nil, false
}
out := make(map[string]json.RawMessage, len(objs[0]))
for k, v := range objs[0] {
out[k] = v
}
out["value"] = merged
// The envelope timestamp says when the answer was produced; the newest one
// describes the merged answer.
if ts, ok := maxField(objs, "timestamp"); ok {
out["timestamp"] = ts
}
return marshal(out)
}
// maxField returns the largest numeric value of a field across the responses.
func maxField(objs []map[string]json.RawMessage, field string) (json.RawMessage, bool) {
var best json.RawMessage
var bestN float64
for _, obj := range objs {
n, ok := numberOf(obj[field])
if !ok {
continue
}
if best == nil || n > bestN {
best, bestN = obj[field], n
}
}
return best, best != nil
}
// mergeMetricValue folds one attribute's value from every backend into one.
// Objects merge key by key over the union of keys, so a backend missing an
// attribute still contributes the rest. Numbers combine by the attribute's rule.
// Anything else — strings, booleans, arrays, nulls, or a mix of kinds — keeps
// the first backend's value, there being no sound way to add them.
func mergeMetricValue(vals []any, attr string) any {
if len(vals) == 0 {
return nil
}
if len(vals) == 1 {
return vals[0]
}
objs := make([]map[string]any, 0, len(vals))
for _, v := range vals {
if m, ok := v.(map[string]any); ok {
objs = append(objs, m)
}
}
if len(objs) == len(vals) {
out := map[string]any{}
for _, m := range objs {
for k := range m {
if _, done := out[k]; done {
continue
}
sub := make([]any, 0, len(objs))
for _, o := range objs {
if v, ok := o[k]; ok {
sub = append(sub, v)
}
}
out[k] = mergeMetricValue(sub, k)
}
}
return out
}
nums := make([]float64, 0, len(vals))
for _, v := range vals {
n, ok := v.(json.Number)
if !ok {
break
}
f, err := n.Float64()
if err != nil {
break
}
nums = append(nums, f)
}
if len(nums) != len(vals) {
return vals[0]
}
return combineNumbers(nums, attr)
}
func combineNumbers(nums []float64, attr string) json.RawMessage {
acc := nums[0]
switch mergeRuleFor(attr) {
case ruleMin:
for _, n := range nums[1:] {
if n < acc {
acc = n
}
}
case ruleMax:
for _, n := range nums[1:] {
if n > acc {
acc = n
}
}
case ruleMean:
for _, n := range nums[1:] {
acc += n
}
acc /= float64(len(nums))
default:
for _, n := range nums[1:] {
acc += n
}
}
return json.RawMessage(strconv.FormatFloat(acc, 'f', -1, 64))
}
// decodedObject re-reads an object's fields as generic values so the whole thing
// can go through mergeMetricValue.
func decodedObject(obj map[string]json.RawMessage) any {
out := make(map[string]any, len(obj))
for k, raw := range obj {
var v any
if decodeJSON(raw, &v) == nil {
out[k] = v
}
}
return out
}
// decodeJSON keeps integers exact by decoding numbers as json.Number.
func decodeJSON(data []byte, v any) error {
dec := json.NewDecoder(bytes.NewReader(data))
dec.UseNumber()
return dec.Decode(v)
}
// marshal reports ok=false rather than an error: an unmarshalable merge result
// has only one recovery, replaying a backend's body verbatim.
func marshal(v any) ([]byte, bool) {
b, err := json.Marshal(v)
return b, err == nil
}
+285
View File
@@ -0,0 +1,285 @@
package main
import (
"encoding/json"
"net/http"
"net/http/httptest"
"reflect"
"strconv"
"testing"
)
const (
numNodesMBean = "puppetlabs.puppetdb.population:name=num-nodes"
numNodesPath = metricsPrefix + "v2/read/" + numNodesMBean
// What pypuppetdb actually sends: quote() percent-encodes ':' and '='.
numNodesEscaped = metricsPrefix + "v2/read/puppetlabs.puppetdb.population%3Aname%3Dnum-nodes"
)
// jolokiaRead wraps an MBean value in the envelope PuppetDB's Jolokia returns.
func jolokiaRead(mbean, value string, timestamp int) string {
return `{"request":{"mbean":"` + mbean + `","type":"read"},` +
`"value":` + value + `,"timestamp":` + strconv.Itoa(timestamp) + `,"status":200}`
}
func metricValue(t *testing.T, body []byte) map[string]any {
t.Helper()
var env map[string]json.RawMessage
if err := json.Unmarshal(body, &env); err != nil {
t.Fatalf("unmarshal envelope %s: %v", body, err)
}
var val map[string]any
if err := json.Unmarshal(env["value"], &val); err != nil {
t.Fatalf("unmarshal value %s: %v", env["value"], err)
}
return val
}
func TestMetrics_ReadSumsPopulationCounts(t *testing.T) {
// Puppetboard's landing page and radiator read num-nodes when
// DEFAULT_ENVIRONMENT is '*'; each backend only knows its own nodes.
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":90}`, 1000)
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":53}`, 2000)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), numNodesEscaped)
if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
if got := metricValue(t, rec.Body.Bytes())["Value"]; got != float64(143) {
t.Errorf("Value = %v, want 143", got)
}
}
func TestMetrics_EscapedMBeanNameSurvives(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 1)
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 1)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
metaGet(t, srv.Handler(), numNodesEscaped)
if !a.sawRawPath(numNodesEscaped) {
t.Errorf("backend saw %v, want the percent-encoded path %q", a.rawPaths, numNodesEscaped)
}
}
func TestMetrics_EnvelopeKeepsNewestTimestamp(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 1000)
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 2000)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
var env map[string]any
if err := json.Unmarshal(metaGet(t, srv.Handler(), numNodesEscaped).Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if env["timestamp"] != float64(2000) {
t.Errorf("timestamp = %v, want 2000", env["timestamp"])
}
if env["status"] != float64(200) {
t.Errorf("status = %v, want 200", env["status"])
}
}
func TestMetrics_PerAttributeRules(t *testing.T) {
const mbean = "puppetlabs.puppetdb.mq:name=global.processing-time"
path := metricsPrefix + "v2/read/" + mbean
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[path] = jolokiaRead(mbean,
`{"Count":10,"Min":2,"Max":9,"Mean":4,"StdDev":1,"50thPercentile":3,"MeanRate":1.5}`, 1)
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[path] = jolokiaRead(mbean,
`{"Count":6,"Min":1,"Max":20,"Mean":6,"StdDev":3,"50thPercentile":5,"MeanRate":0.5}`, 1)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
got := metricValue(t, metaGet(t, srv.Handler(), path).Body.Bytes())
want := map[string]any{
"Count": float64(16), // counts add
"Min": float64(1), // a bound stays a bound
"Max": float64(20),
"Mean": float64(5), // distribution stats average
"StdDev": float64(2),
"50thPercentile": float64(4),
"MeanRate": float64(2), // throughput adds
}
if !reflect.DeepEqual(got, want) {
t.Errorf("merged value = %v, want %v", got, want)
}
}
func TestMetrics_ScalarReadUsesURLAttributeName(t *testing.T) {
// /metrics/v2/read/<mbean>/<attribute> answers with a bare number, so the
// rule has to come from the URL rather than an object key.
const mbean = "puppetlabs.puppetdb.population:name=num-resources"
sumPath := metricsPrefix + "v2/read/" + mbean + "/Value"
maxPath := metricsPrefix + "v2/read/" + mbean + "/Max"
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[sumPath] = jolokiaRead(mbean, `1000`, 1)
a.bodies[maxPath] = jolokiaRead(mbean, `1000`, 1)
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[sumPath] = jolokiaRead(mbean, `234`, 1)
b.bodies[maxPath] = jolokiaRead(mbean, `234`, 1)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
for _, c := range []struct {
path string
want float64
}{{sumPath, 1234}, {maxPath, 1000}} {
var env map[string]any
if err := json.Unmarshal(metaGet(t, srv.Handler(), c.path).Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if env["value"] != c.want {
t.Errorf("%s value = %v, want %v", c.path, env["value"], c.want)
}
}
}
func TestMetrics_ListUnionsDomains(t *testing.T) {
// Puppetboard's /metrics page calls metric() with no name, which is a
// Jolokia list; a backend-local MBean must not vanish from the browse tree.
const path = metricsPrefix + "v2/list"
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[path] = `{"value":{"java.lang":{"type=Memory":{"attr":{"HeapMemoryUsage":{"rw":false}}}}},"status":200,"timestamp":1}`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[path] = `{"value":{"puppetlabs.puppetdb.population":{"name=num-nodes":{"attr":{"Value":{"rw":false}}}}},"status":200,"timestamp":1}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
got := metricValue(t, metaGet(t, srv.Handler(), path).Body.Bytes())
if _, ok := got["java.lang"]; !ok {
t.Errorf("java.lang missing from merged list: %v", got)
}
if _, ok := got["puppetlabs.puppetdb.population"]; !ok {
t.Errorf("puppetlabs.puppetdb.population missing from merged list: %v", got)
}
}
func TestMetrics_V1BareObjectMerged(t *testing.T) {
// metrics/v1/mbeans has no Jolokia envelope; the whole body is the value.
const path = metricsPrefix + "v1/mbeans/" + numNodesMBean
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[path] = `{"Value":90}`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[path] = `{"Value":53}`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
var got map[string]any
if err := json.Unmarshal(metaGet(t, srv.Handler(), path).Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
if got["Value"] != float64(143) {
t.Errorf("Value = %v, want 143", got["Value"])
}
}
func TestMetrics_MissingMBeanKeepsUpstreamError(t *testing.T) {
// Jolokia reports a bad MBean as a 200 with an error envelope, which
// pypuppetdb turns into DoesNotComputeError; the client must still see it.
const path = metricsPrefix + "v2/read/nope:name=nothing"
errEnv := `{"request":{"mbean":"nope:name=nothing"},"error_type":"javax.management.InstanceNotFoundException","error":"nope:name=nothing is not registered","status":404}`
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[path] = errEnv
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[path] = errEnv
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), path)
var env map[string]any
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
t.Fatal(err)
}
if env["error"] == nil {
t.Errorf("expected the upstream Jolokia error to be replayed, got %s", rec.Body.String())
}
}
func TestMetrics_ErroringBackendIgnoredWhenAnotherAnswers(t *testing.T) {
const path = metricsPrefix + "v2/read/" + numNodesMBean
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[path] = `{"request":{},"error":"boom","status":500}`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[path] = jolokiaRead(numNodesMBean, `{"Value":53}`, 1)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if got := metricValue(t, metaGet(t, srv.Handler(), path).Body.Bytes())["Value"]; got != float64(53) {
t.Errorf("Value = %v, want 53 from the backend that answered", got)
}
}
func TestMetrics_OneBackendDown(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.fail = true
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":53}`, 1)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := metaGet(t, srv.Handler(), numNodesEscaped)
if rec.Code != http.StatusOK {
t.Fatalf("expected 200 serving the survivor, got %d", rec.Code)
}
if got := metricValue(t, rec.Body.Bytes())["Value"]; got != float64(53) {
t.Errorf("Value = %v, want 53", got)
}
}
func TestMetrics_AllBackendsDown(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.fail = true
b := newFakeBackend(t, `[]`, `[]`)
b.fail = true
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
if rec := metaGet(t, srv.Handler(), numNodesEscaped); rec.Code != http.StatusBadGateway {
t.Errorf("status %d, want 502", rec.Code)
}
}
func TestMetrics_RejectsNonGET(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
b := newFakeBackend(t, `[]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := httptest.NewRecorder()
srv.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, numNodesEscaped, nil))
if rec.Code != http.StatusMethodNotAllowed {
t.Errorf("status %d, want 405", rec.Code)
}
}
func TestMergeMetricValue_NonNumericKeepsFirst(t *testing.T) {
a := map[string]any{"Name": "pdb-a", "Enabled": true}
b := map[string]any{"Name": "pdb-b", "Enabled": false}
got, ok := mergeMetricValue([]any{a, b}, "").(map[string]any)
if !ok {
t.Fatalf("expected an object, got %T", got)
}
if got["Name"] != "pdb-a" || got["Enabled"] != true {
t.Errorf("merged = %v, want the first backend's strings and booleans", got)
}
}
func TestMergeRuleFor(t *testing.T) {
cases := map[string]mergeRule{
"Count": ruleSum,
"Value": ruleSum,
"MeanRate": ruleSum,
"queue-depth": ruleSum,
"min": ruleMin,
"Max": ruleMax,
"Uptime": ruleMax,
"StartTime": ruleMax,
"Mean": ruleMean,
"StdDev": ruleMean,
"99thPercentile": ruleMean,
}
for attr, want := range cases {
if got := mergeRuleFor(attr); got != want {
t.Errorf("mergeRuleFor(%q) = %v, want %v", attr, got, want)
}
}
}
+238 -34
View File
@@ -3,6 +3,7 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
@@ -17,6 +18,7 @@ import (
const (
factsPath = "/pdb/query/v4/facts"
nodesPath = "/pdb/query/v4/nodes"
resourcesPath = "/pdb/query/v4/resources"
reportsPath = "/pdb/query/v4/reports"
eventsPath = "/pdb/query/v4/events"
eventCountsPath = "/pdb/query/v4/event-counts"
@@ -34,11 +36,18 @@ type backendResult struct {
err error
}
var errAllBackendsFailed = errors.New("all backends failed")
type Server struct {
cfg Config
client *http.Client
log *log.Logger
// factsCache is nil when caching is disabled; cacheFor hands out a noop then.
factsCache Cache
flights flightGroup
stale staleTracker
// freshness cache (freshness merge only).
mu sync.Mutex
freshData freshness
@@ -46,17 +55,44 @@ type Server struct {
}
func NewServer(cfg Config, logger *log.Logger) *Server {
return &Server{
cfg.clampFactsTTL()
s := &Server{
cfg: cfg,
client: &http.Client{Timeout: cfg.Timeout},
log: logger,
}
if cfg.cacheEnabled() {
s.factsCache = newMemoryCache(cfg.FactsTTL, cfg.CacheBytes)
}
return s
}
// cacheFor picks the cache backing a request. Merged /facts and /nodes record
// sets share the in-memory cache; every other path is uncached until the reports
// cache lands, and a new backend is a case here rather than a change to any
// handler.
func (s *Server) cacheFor(path string, params url.Values) (Cache, bool) {
switch path {
case factsPath, nodesPath:
// An aggregate row is a summed count, not the merged record set the
// cache was built for, so it stays on the live path.
if parseAggregate(params.Get("query")) != nil {
return noopCache{}, false
}
if s.factsCache != nil {
return s.factsCache, true
}
}
return noopCache{}, false
}
func (s *Server) Handler() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("/healthz", s.handleHealth)
mux.HandleFunc("/pdb/query/v4/", s.handleQuery)
mux.HandleFunc(metaVersionPath, s.handleMetaVersion)
mux.HandleFunc(metaServerTimePath, s.handleMetaServerTime)
mux.HandleFunc(metricsPrefix, s.handleMetrics)
return mux
}
@@ -67,7 +103,9 @@ func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request) {
}
switch r.URL.Path {
case nodesPath:
s.serveMerged(w, r, nodesPath, s.mergeNodesResponse)
s.serveNodes(w, r)
case resourcesPath:
s.serveResources(w, r)
case factsPath:
s.serveMerged(w, r, factsPath, s.mergeFactsResponse)
case reportsPath:
@@ -103,11 +141,14 @@ func isReportSubResource(path string) bool {
}
func (s *Server) serveMerged(w http.ResponseWriter, r *http.Request, path string, merge func([]backendResult) []json.RawMessage) {
alive, ok := s.aliveResults(w, r, path, queryParams(r.URL.Query().Get("query")))
if !ok {
return
}
writeJSON(w, merge(alive))
params := queryParams(r.URL.Query().Get("query"))
s.serveCached(w, r, path, params, func() (cachedResponse, error) {
alive, err := s.aliveResults(r.Context(), path, params)
if err != nil {
return cachedResponse{}, err
}
return cachedResponse{Body: encodeRecords(merge(alive)), Records: -1}, nil
})
}
// Reports and events are immutable history, so both backends' records belong in the merged view.
@@ -119,19 +160,41 @@ func (s *Server) serveUnion(w http.ResponseWriter, r *http.Request, path string,
return
}
alive, ok := s.aliveResults(w, r, path, page.upstreamParams(in))
if !ok {
// Keyed on the request's own params, not the upstream ones: upstreamParams
// folds offset into limit, so different windows would collide.
s.serveCached(w, r, path, in, func() (cachedResponse, error) {
alive, err := s.aliveResults(r.Context(), path, page.upstreamParams(in))
if err != nil {
return cachedResponse{}, err
}
merged := mergeUnion(alive, key)
sortRecords(merged, page.order)
resp := cachedResponse{Body: encodeRecords(page.apply(merged)), Records: -1}
if page.wantTotal {
if total := sumTotals(alive); total >= 0 {
resp.Records = total
}
}
return resp, nil
})
}
// A count row carries no certname, so the certname-keyed merge would collapse every backend's count into one backend's; aggregates take the summing path instead.
func (s *Server) serveNodes(w http.ResponseWriter, r *http.Request) {
if spec := parseAggregate(r.URL.Query().Get("query")); spec != nil {
s.serveSummed(w, r, nodesPath, spec.columns)
return
}
s.serveMerged(w, r, nodesPath, s.mergeNodesResponse)
}
merged := mergeUnion(alive, key)
sortRecords(merged, page.order)
if page.wantTotal {
if total := sumTotals(alive); total >= 0 {
w.Header().Set(recordsHeader, strconv.Itoa(total))
}
// Only aggregates merge: a resource record has no cross-backend identity to dedupe on, so a plain query stays on the pass-through path.
func (s *Server) serveResources(w http.ResponseWriter, r *http.Request) {
if spec := parseAggregate(r.URL.Query().Get("query")); spec != nil {
s.serveSummed(w, r, resourcesPath, spec.columns)
return
}
writeJSON(w, page.apply(merged))
s.proxyUnmerged(w, r)
}
// An `extract` query with a `function` column returns synthetic aggregate rows that carry no identity, so they are summed rather than unioned.
@@ -152,17 +215,19 @@ func (s *Server) serveSummed(w http.ResponseWriter, r *http.Request, path string
return
}
alive, ok := s.aliveResults(w, r, path, page.upstreamParams(in))
if !ok {
return
}
merged := sumRows(alive, columns)
sortRecords(merged, page.order)
if page.wantTotal {
w.Header().Set(recordsHeader, strconv.Itoa(len(merged)))
}
writeJSON(w, page.apply(merged))
s.serveCached(w, r, path, in, func() (cachedResponse, error) {
alive, err := s.aliveResults(r.Context(), path, page.upstreamParams(in))
if err != nil {
return cachedResponse{}, err
}
merged := sumRows(alive, columns)
sortRecords(merged, page.order)
resp := cachedResponse{Body: encodeRecords(page.apply(merged)), Records: -1}
if page.wantTotal {
resp.Records = len(merged)
}
return resp, nil
})
}
// A backend without the report answers 404, indistinguishable from a failure, so every backend is consulted before serving empty.
@@ -190,9 +255,9 @@ func (s *Server) serveFirstHolder(w http.ResponseWriter, r *http.Request) {
writeJSON(w, nil)
}
// Writes a 502 and returns ok=false only when every backend failed.
func (s *Server) aliveResults(w http.ResponseWriter, r *http.Request, path string, params url.Values) ([]backendResult, bool) {
results := s.fanOut(r.Context(), path, params)
// Returns errAllBackendsFailed only when every backend failed.
func (s *Server) aliveResults(ctx context.Context, path string, params url.Values) ([]backendResult, error) {
results := s.fanOut(ctx, path, params)
var alive []backendResult
for _, res := range results {
@@ -203,10 +268,114 @@ func (s *Server) aliveResults(w http.ResponseWriter, r *http.Request, path strin
alive = append(alive, res)
}
if len(alive) == 0 {
http.Error(w, "all backends failed", http.StatusBadGateway)
return nil, false
return nil, errAllBackendsFailed
}
return alive, true
return alive, nil
}
// cachedResponse is the stored form of a merged response: the JSON body plus the
// X-Records value it carried, so a cache hit reproduces both.
type cachedResponse struct {
Body json.RawMessage `json:"body"`
Records int `json:"records"` // -1 when the response sets no X-Records
}
// serveCached answers from the cache when the entry is fresh, otherwise runs
// build — single-flighted, so N concurrent identical requests cause one upstream
// fan-out — and stores the result. A build failure falls back to a stale entry
// when one exists; that is the only path on which stale data is served. Paths
// with no cache configured run build directly, unchanged.
func (s *Server) serveCached(w http.ResponseWriter, r *http.Request, path string, params url.Values, build func() (cachedResponse, error)) {
cache, enabled := s.cacheFor(path, params)
if !enabled {
resp, err := build()
if err != nil {
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
writeCached(w, resp)
return
}
key := cacheKey(path, params)
var stale *CacheEntry
ent, status, err := cache.Get(r.Context(), key)
switch {
case err != nil:
s.log.Printf("warning: cache lookup for %s failed: %v", key, err)
case status == CacheFresh:
s.stale.markFresh()
s.writeStored(w, ent.Body)
return
case status == CacheStale:
stale = &ent
}
resp, err, _ := s.flights.Do(key, func() (cachedResponse, error) {
built, buildErr := build()
if buildErr != nil {
return cachedResponse{}, buildErr
}
body, marshalErr := json.Marshal(built)
if marshalErr != nil {
s.log.Printf("warning: encoding cache entry for %s failed: %v", key, marshalErr)
return built, nil
}
// The leader's request may be cancelled while followers still wait.
if putErr := cache.Put(context.WithoutCancel(r.Context()), key, body); putErr != nil {
s.log.Printf("warning: cache store for %s failed: %v", key, putErr)
}
return built, nil
})
if err != nil {
if stale != nil {
s.stale.markStale(time.Now())
s.log.Printf("warning: serving stale %s from cache (stored %s): %v",
path, stale.StoredAt.UTC().Format(time.RFC3339), err)
s.writeStored(w, stale.Body)
return
}
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
s.stale.markFresh()
writeCached(w, resp)
}
func (s *Server) writeStored(w http.ResponseWriter, body []byte) {
var resp cachedResponse
if err := json.Unmarshal(body, &resp); err != nil {
s.log.Printf("warning: unreadable cache entry: %v", err)
http.Error(w, "unreadable cache entry", http.StatusBadGateway)
return
}
writeCached(w, resp)
}
func writeCached(w http.ResponseWriter, resp cachedResponse) {
w.Header().Set("Content-Type", "application/json")
if resp.Records >= 0 {
w.Header().Set(recordsHeader, strconv.Itoa(resp.Records))
}
// resp.Body is shared with the cache and with every caller of a single
// flight, so it is written, never appended to.
body := []byte(resp.Body)
if len(body) == 0 {
body = []byte("[]")
}
_, _ = w.Write(body)
_, _ = w.Write([]byte("\n"))
}
func encodeRecords(recs []json.RawMessage) json.RawMessage {
if recs == nil {
recs = []json.RawMessage{}
}
b, err := json.Marshal(recs)
if err != nil {
return json.RawMessage("[]")
}
return b
}
func queryParams(query string) url.Values {
@@ -372,13 +541,48 @@ func setContentType(w http.ResponseWriter, contentType string) {
type healthReport struct {
Status string `json:"status"`
Backends map[string]string `json:"backends"` // name -> "ok" | error text
Cache cacheHealth `json:"cache"`
}
type cacheHealth struct {
Backend string `json:"backend"` // "memory" | "none"
TTL string `json:"ttl"`
Entries int `json:"entries"`
StaleEntries int `json:"stale_entries"` // cached entries past their TTL
Bytes int64 `json:"bytes"`
ServingStale bool `json:"serving_stale"` // last cached response came from a stale entry
StaleServed uint64 `json:"stale_served"`
LastStale string `json:"last_stale_served,omitempty"`
}
func (s *Server) cacheHealth() cacheHealth {
stats := CacheStats{Backend: "none"}
ttl := time.Duration(0)
if s.factsCache != nil {
stats = s.factsCache.Stats()
ttl = s.cfg.FactsTTL
}
serving, served, last := s.stale.snapshot()
h := cacheHealth{
Backend: stats.Backend,
TTL: durationString(ttl),
Entries: stats.Entries,
StaleEntries: stats.StaleEntries,
Bytes: stats.Bytes,
ServingStale: serving,
StaleServed: served,
}
if !last.IsZero() {
h.LastStale = last.UTC().Format(time.RFC3339)
}
return h
}
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
probe := `["=","certname","pdbmux-healthz-probe"]`
results := s.fanOut(r.Context(), nodesPath, queryParams(probe))
report := healthReport{Backends: map[string]string{}}
report := healthReport{Backends: map[string]string{}, Cache: s.cacheHealth()}
healthy := 0
for _, res := range results {
if res.err != nil {
+129
View File
@@ -34,6 +34,9 @@ type fakeBackend struct {
mu sync.Mutex
gotParams map[string]url.Values
// rawPaths records the still-escaped request paths, so tests can assert an
// MBean name's percent-encoding survived the proxy.
rawPaths []string
}
func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend {
@@ -51,6 +54,7 @@ func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend {
}
fb.mu.Lock()
fb.gotParams[r.URL.Path] = r.URL.Query()
fb.rawPaths = append(fb.rawPaths, r.URL.EscapedPath())
fb.mu.Unlock()
if fb.fail {
http.Error(w, "boom", http.StatusInternalServerError)
@@ -91,6 +95,14 @@ func (fb *fakeBackend) params(path string) (url.Values, bool) {
return v, ok
}
// sawRawPath reports whether the backend was asked for a path with exactly that
// escaping.
func (fb *fakeBackend) sawRawPath(p string) bool {
fb.mu.Lock()
defer fb.mu.Unlock()
return slices.Contains(fb.rawPaths, p)
}
// gotQuery returns the PuppetDB query param the backend saw for a path.
func (fb *fakeBackend) gotQuery(path string) string {
v, _ := fb.params(path)
@@ -858,3 +870,120 @@ func TestHandler_EventCountsBadPagingParam(t *testing.T) {
t.Errorf("expected 400 for a malformed limit, got %d", rec.Code)
}
}
// What Puppetboard's landing page sends when DEFAULT_ENVIRONMENT names a real
// environment: an extract/count with no group_by, so every backend returns one
// anonymous row.
const nodeCountQuery = `["extract",[["function","count"]],["and",["=","catalog_environment","production"]]]`
func TestHandler_NodesAggregateSummed(t *testing.T) {
// A count row has no certname, so the certname-keyed merge would have
// collapsed both backends' counts into one backend's number.
a := newFakeBackend(t, `[{"count":90}]`, `[]`)
b := newFakeBackend(t, `[{"count":53}]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), nodesPath, nodeCountQuery)
if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
if got := counts(t, rec.Body.Bytes(), "count"); !slices.Equal(got, []float64{143}) {
t.Errorf("count = %v, want [143]", got)
}
}
func TestHandler_NodesAggregateGroupedSummed(t *testing.T) {
a := newFakeBackend(t, `[{"count":4,"catalog_environment":"production"},{"count":1,"catalog_environment":"dev"}]`, `[]`)
b := newFakeBackend(t, `[{"count":3,"catalog_environment":"production"}]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), nodesPath,
`["extract",[["function","count"],"catalog_environment"],["~","certname",".*"],["group_by","catalog_environment"]]`)
var got []map[string]any
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
byEnv := map[string]float64{}
for _, row := range got {
e, _ := row["catalog_environment"].(string)
n, _ := row["count"].(float64)
byEnv[e] = n
}
want := map[string]float64{"production": 7, "dev": 1}
if !reflect.DeepEqual(byEnv, want) {
t.Errorf("counts = %v, want %v", byEnv, want)
}
}
func TestHandler_NodesNonAggregateStillMergedByCertname(t *testing.T) {
// Regression: routing aggregates to the summing path must not divert plain
// queries, including an extract projection that carries no function column.
a := newFakeBackend(t,
`[`+node("h1", "2026-07-01T00:00:00Z")+`,`+node("h2", "2026-07-10T00:00:00Z")+`]`, `[]`)
b := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`)
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
for _, q := range []string{
`["=","certname","h1"]`,
`["extract",["certname","report_timestamp"],["~","certname",".*"]]`,
} {
rec := doGet(t, srv.Handler(), nodesPath, q)
var got []recordMeta
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatalf("query %s: %v", q, err)
}
if len(got) != 2 {
t.Fatalf("query %s: expected 2 deduped nodes, got %d: %s", q, len(got), rec.Body.String())
}
for _, m := range got {
if m.Certname == "h1" && m.ReportTimestamp != "2026-07-20T00:00:00Z" {
t.Errorf("query %s: h1 should be the newer record, got %s", q, m.ReportTimestamp)
}
}
}
}
func TestHandler_ResourcesAggregateSummed(t *testing.T) {
// /resources is otherwise an unmerged pass-through, so before this the
// landing page's resource total was whichever backend answered first.
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[resourcesPath] = `[{"count":1000}]`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[resourcesPath] = `[{"count":234}]`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), resourcesPath,
`["extract",[["function","count"]],["=","environment","production"]]`)
if rec.Code != http.StatusOK {
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
}
if got := counts(t, rec.Body.Bytes(), "count"); !slices.Equal(got, []float64{1234}) {
t.Errorf("count = %v, want [1234]", got)
}
}
func TestHandler_ResourcesNonAggregateStillPassesThrough(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[resourcesPath] = `[{"certname":"h1","type":"File"}]`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[resourcesPath] = `[{"certname":"h2","type":"File"}]`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
rec := doGet(t, srv.Handler(), resourcesPath, `["=","type","File"]`)
if got := rec.Body.String(); !strings.Contains(got, `"h1"`) || strings.Contains(got, `"h2"`) {
t.Errorf("body = %s, want the first backend's response verbatim", got)
}
}
func TestHandler_ResourcesAggregateAsksEveryBackend(t *testing.T) {
a := newFakeBackend(t, `[]`, `[]`)
a.bodies[resourcesPath] = `[{"count":1}]`
b := newFakeBackend(t, `[]`, `[]`)
b.bodies[resourcesPath] = `[{"count":1}]`
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
doGet(t, srv.Handler(), resourcesPath, `["extract",[["function","count"]],["=","environment","production"]]`)
if _, ok := b.params(resourcesPath); !ok {
t.Error("second backend was never asked for the resource count")
}
}