Compare commits
4 Commits
v0.2.0
...
c0aab83c20
| Author | SHA1 | Date | |
|---|---|---|---|
| c0aab83c20 | |||
| 1e2acc9c14 | |||
| 394f7df3a7 | |||
| b6d59af7ef |
@@ -24,15 +24,21 @@ not PQL) is forwarded verbatim.
|
||||
|
||||
| Path | Behaviour |
|
||||
|---|---|
|
||||
| `GET /pdb/query/v4/nodes` | Fan out to all backends, dedupe by `certname`, keep the record with the newer `report_timestamp`. |
|
||||
| `GET /pdb/query/v4/nodes` | Fan out to all backends, dedupe by `certname`, keep the record with the newer `report_timestamp`. An `extract`/`count` query is **summed** instead. |
|
||||
| `GET /pdb/query/v4/facts` | Fan out to all, and per `certname` keep **all** facts from the backend that owns that node (see merge semantics). |
|
||||
| `GET /pdb/query/v4/resources` | An `extract`/`count` query is fanned out and **summed**; any other query is an unmerged pass-through. |
|
||||
| `GET /pdb/query/v4/reports` | Fan out to all and serve the **union**, deduped by report `hash`, re-ordered and re-paged across backends. |
|
||||
| `GET /pdb/query/v4/events` | Fan out to all and serve the **union**, deduped by record identity, re-ordered and re-paged. |
|
||||
| `GET /pdb/query/v4/event-counts` | Fan out to all and **sum** each subject's counts into one row per subject. |
|
||||
| `GET /pdb/query/v4/aggregate-event-counts` | Fan out to all and **sum** the summary object's counts. |
|
||||
| `GET /pdb/query/v4/reports/<hash>/{events,logs,metrics}` | Ask every backend; serve the answer from whichever backend actually holds that report. `404` when none does. |
|
||||
| `GET /pdb/query/v4/*` (any other) | No merge rule, so backends are tried in configured order and the first success is streamed back verbatim; if all reject it, the first upstream error response is replayed. |
|
||||
| `GET /healthz` | Per-backend reachability. `200 {"status":"ok"}` if all reachable, `200 degraded` if some fail, `503 down` if all fail. |
|
||||
| `GET /pdb/meta/v1/version` | Fan out to all and report the **lowest** version any backend runs. |
|
||||
| `GET /pdb/meta/v1/server-time` | Fan out to all and serve the first reachable backend's clock. |
|
||||
| `GET /metrics/v2/read/<mbean>` | Fan out to all and merge the Jolokia response; numeric attributes are **summed** by default (see merge semantics). |
|
||||
| `GET /metrics/v2/list` | Fan out to all and serve the **union** of the backends' MBean trees. |
|
||||
| `GET /metrics/v1/mbeans[/<mbean>]` | Same merge, applied to the legacy envelope-less body. |
|
||||
| `GET /healthz` | Per-backend reachability plus cache state. `200 {"status":"ok"}` if all reachable, `200 degraded` if some fail, `503 down` if all fail. |
|
||||
|
||||
Fan-out is concurrent. If one backend errors or times out, `pdbmux` serves the
|
||||
surviving backends' results and logs a warning; a merged endpoint only returns `502` when
|
||||
@@ -62,12 +68,20 @@ unknown fields survive untouched.
|
||||
reporting to more than one backend stores identical records in each).
|
||||
- **Aggregates** — `extract`/`group_by` rows are counts, not records, so each
|
||||
backend returns a partial answer that has to be **added**, not deduped. This
|
||||
covers `/event-counts`, `/aggregate-event-counts`, and a `/reports` query whose
|
||||
`extract` carries a `["function", ...]` column.
|
||||
- The grouping key is the row's non-aggregate fields: for `/reports` they come
|
||||
from the query — the plain `extract` fields plus any `group_by` clause — and
|
||||
for the event-count endpoints from the row itself (`subject_type`/`subject`,
|
||||
or `summarize_by`), whose remaining fields are all counts.
|
||||
covers `/event-counts`, `/aggregate-event-counts`, and any `/reports`,
|
||||
`/nodes` or `/resources` query whose `extract` carries a `["function", ...]`
|
||||
column.
|
||||
- The grouping key is the row's non-aggregate fields: for `/reports`,
|
||||
`/nodes` and `/resources` they come from the query — the plain `extract`
|
||||
fields plus any `group_by` clause — and for the event-count endpoints from
|
||||
the row itself (`subject_type`/`subject`, or `summarize_by`), whose
|
||||
remaining fields are all counts.
|
||||
- On `/nodes` this takes precedence over the `certname` merge: a count row has
|
||||
no `certname`, so deduping would collapse every backend's count into one
|
||||
backend's number. A `/nodes` query with no `function` column — including a
|
||||
plain `extract` projection — still merges by `certname`.
|
||||
- `/resources` has no cross-backend record identity to dedupe on, so only its
|
||||
aggregate queries merge; everything else stays an unmerged pass-through.
|
||||
- Rows sharing a key collapse into one with their numeric columns summed. A key
|
||||
only one backend reported is passed through byte-for-byte. An aggregate column
|
||||
that is absent or non-numeric in a row is skipped, never zeroed, so the
|
||||
@@ -77,6 +91,39 @@ unknown fields survive untouched.
|
||||
- `include_total=true` on a summed endpoint reports the **merged** row count,
|
||||
not the sum of the backends' `X-Records`, since shared keys collapse.
|
||||
|
||||
### Metadata and metrics
|
||||
|
||||
- **`/pdb/meta/v1/version`** — when the backends agree, that version is served.
|
||||
When they differ, `pdbmux` reports the **lowest**: a client reads this as the
|
||||
feature level it may rely on, and the estate can only be relied on for what its
|
||||
oldest PuppetDB implements. Versions compare segment by segment, numerically
|
||||
where both segments are numbers (`7.9.0` < `7.12.0`), lexically otherwise.
|
||||
A backend whose body is unparseable is skipped rather than treated as lowest.
|
||||
- **`/pdb/meta/v1/server-time`** — the clock of whichever PuppetDB answered is
|
||||
not estate state and has no meaningful merge, so the first **reachable**
|
||||
backend in configured order supplies it, the same tie-break used elsewhere.
|
||||
- **`/metrics/...`** — the Jolokia envelope's `value` is merged and the rest of
|
||||
the envelope comes from the first backend (with the newest `timestamp`).
|
||||
Values merge recursively:
|
||||
- Objects merge over the **union** of their keys, so an MBean attribute only
|
||||
one backend exposes still survives.
|
||||
- Numbers combine by the attribute's own name. The default is a **sum** —
|
||||
almost everything here is a population count (`num-nodes`, `num-resources`,
|
||||
queue depth, command totals) whose estate-wide value is the total, and rates
|
||||
are additive throughput. The exceptions describe a distribution or a bound,
|
||||
where adding two servers' numbers yields a figure that was never true of
|
||||
either: `Min` takes the minimum; `Max`, `Uptime` and `StartTime` take the
|
||||
maximum; `Mean`, `Median`, `StdDev` and `*Percentile` take the unweighted
|
||||
arithmetic mean (`pdbmux` has no per-backend sample counts to weight by).
|
||||
Matching is case-insensitive.
|
||||
- Strings, booleans, arrays, nulls and mixed kinds keep the first backend's
|
||||
value — there is no sound way to add them.
|
||||
- Jolokia signals a bad MBean as a non-2xx `status` **inside** an HTTP 200.
|
||||
Such a backend is skipped; if every backend does so, the first one's error
|
||||
envelope is replayed verbatim so the client sees the real reason.
|
||||
- MBean names arrive percent-encoded over Jolokia's own `!`-escaping; the raw
|
||||
path is forwarded so neither layer is lost.
|
||||
|
||||
### Paging and ordering on the merged endpoints
|
||||
|
||||
Each backend applies `order_by`/`limit`/`offset` to its own slice only, so
|
||||
@@ -93,6 +140,43 @@ Each backend applies `order_by`/`limit`/`offset` to its own slice only, so
|
||||
- A malformed `limit`, `offset` or `order_by` gets a `400` rather than being
|
||||
forwarded.
|
||||
|
||||
## Caching
|
||||
|
||||
`pdbmux` caches merged `/nodes` and `/facts` record sets **in memory** so a busy
|
||||
Puppetboard does not re-fan-out the same query every few seconds. Everything else
|
||||
runs uncached — including `extract`/`count` aggregates on those two paths, and
|
||||
the `/pdb/meta/v1/*` and `/metrics/*` endpoints, which are served live on every
|
||||
request. The cache is an interface, and `/reports` gets its own (S3-backed)
|
||||
backend later without further handler changes.
|
||||
|
||||
- **Key** — `<path>?<params>`, where the params are the ones that actually
|
||||
determine the response, URL-encoded with keys sorted ascending and a repeated
|
||||
param's values sorted ascending. Param order in the request is therefore
|
||||
irrelevant: one canonical key per distinct request. A request with no params
|
||||
keys on the bare path.
|
||||
- **TTL** — `facts_ttl`, default `30s`, **hard cap `30s`**. A larger configured
|
||||
value is **clamped** down to the cap, not rejected, so a stray env var cannot
|
||||
crash-loop a container; `pdbmux config show` prints
|
||||
`facts_ttl : 30s (clamped from 600s, cap 30s)` when that happens. `facts_ttl: 0`
|
||||
disables the cache entirely and the merged endpoints behave exactly as before.
|
||||
- **Stale on failure only** — an expired entry is kept, not dropped. When the TTL
|
||||
has passed `pdbmux` always re-queries the backends; the expired copy is served
|
||||
**only** if every backend fails, which turns a `502` into slightly-old data. A
|
||||
healthy backend is never shadowed by a stale entry.
|
||||
- **Bounded** — `facts_cache_bytes` (default 64 MiB) is a byte budget, evicted
|
||||
least-recently-used; reads count as use, so a stale entry that is still being
|
||||
asked for survives. A single response larger than the whole budget is not
|
||||
cached at all. The budget counts stored response bodies only — cache keys and
|
||||
the list/map bookkeeping are not accounted for, so it is a target for body
|
||||
bytes rather than a hard cap on process memory.
|
||||
- **Single-flight** — concurrent requests for the same key collapse into one
|
||||
upstream fan-out; the rest wait for it and share the result.
|
||||
- **Visibility** — `/healthz` carries a `cache` object: `backend`
|
||||
(`memory`/`none`), `ttl`, `entries`, `stale_entries`, `bytes`, `serving_stale`,
|
||||
`stale_served` and `last_stale_served`. `serving_stale` is `true` from the
|
||||
moment a stale fallback is served until the next response comes from a live
|
||||
fan-out or a fresh entry.
|
||||
|
||||
## Config
|
||||
|
||||
Precedence (lowest → highest): **defaults < config file < env vars (`PDBMUX_*`) < flags**.
|
||||
@@ -114,9 +198,11 @@ backends: # order is a tie-break only, not a ranking
|
||||
url: http://puppetdb1.example.com:8080
|
||||
- name: pdb-b
|
||||
url: https://puppetdb2.example.com
|
||||
merge: freshness # freshness | static
|
||||
timeout: 10s # per-upstream request timeout
|
||||
freshness_ttl: 30s # freshness-map cache TTL (freshness merge only)
|
||||
merge: freshness # freshness | static
|
||||
timeout: 10s # per-upstream request timeout
|
||||
freshness_ttl: 30s # freshness-map cache TTL (freshness merge only)
|
||||
facts_ttl: 30s # /facts + /nodes response cache TTL; 0 disables, capped at 30s
|
||||
facts_cache_bytes: 67108864 # byte budget for that cache (64 MiB), LRU-evicted
|
||||
```
|
||||
|
||||
`backends[*].url` is a **base** URL (`scheme://host[:port]`); `pdbmux` appends
|
||||
@@ -129,6 +215,8 @@ the `/pdb/query/v4/...` path per request.
|
||||
| `PDBMUX_MERGE` | `merge` |
|
||||
| `PDBMUX_TIMEOUT` | `timeout` (Go duration, e.g. `10s`) |
|
||||
| `PDBMUX_FRESHNESS_TTL` | `freshness_ttl` |
|
||||
| `PDBMUX_FACTS_TTL` | `facts_ttl` (clamped to 30s) |
|
||||
| `PDBMUX_FACTS_CACHE_BYTES` | `facts_cache_bytes` (plain integer bytes) |
|
||||
| `PDBMUX_BACKENDS` | whole backend list, as `name=url,name=url` |
|
||||
|
||||
Flags: `--config`, `--listen`, `--merge`.
|
||||
@@ -162,5 +250,6 @@ A static (`CGO_ENABLED=0`) binary on a distroless base. Configure it with
|
||||
`PDBMUX_*` env vars (at minimum `PDBMUX_BACKENDS`), or mount a config file — a
|
||||
configmap at `/etc/pdbmux/config.yaml` is picked up with no env var at all, and
|
||||
any other mount path works via `PDBMUX_CONFIG`. Env vars still override file
|
||||
values, so the two mix. Stateless, so run as many replicas as you like; use
|
||||
`/healthz` for liveness/readiness probes.
|
||||
values, so the two mix. Run as many replicas as you like — the only state is the
|
||||
in-memory cache, which is per-replica and bounded by `facts_cache_bytes`, so size
|
||||
the memory limit above it. Use `/healthz` for liveness/readiness probes.
|
||||
|
||||
@@ -0,0 +1,267 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"container/list"
|
||||
"context"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"runtime/debug"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CacheStatus distinguishes the three outcomes of a lookup: nothing stored, a
|
||||
// stored entry within its TTL, and a stored entry past it.
|
||||
type CacheStatus int
|
||||
|
||||
const (
|
||||
CacheMiss CacheStatus = iota
|
||||
CacheFresh
|
||||
CacheStale
|
||||
)
|
||||
|
||||
func (s CacheStatus) String() string {
|
||||
switch s {
|
||||
case CacheFresh:
|
||||
return "fresh"
|
||||
case CacheStale:
|
||||
return "stale"
|
||||
default:
|
||||
return "miss"
|
||||
}
|
||||
}
|
||||
|
||||
// CacheEntry is a stored response body and the time it was stored.
|
||||
type CacheEntry struct {
|
||||
Body []byte
|
||||
StoredAt time.Time
|
||||
}
|
||||
|
||||
// CacheStats is the cache state reported by /healthz.
|
||||
type CacheStats struct {
|
||||
Backend string `json:"backend"`
|
||||
Entries int `json:"entries"`
|
||||
StaleEntries int `json:"stale_entries"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
|
||||
// Cache stores merged responses keyed by cacheKey. Get reports freshness rather
|
||||
// than hiding expired entries so a caller can fall back to a stale body when the
|
||||
// upstream fetch fails. The context and error exist for out-of-process backends
|
||||
// (the reports cache lands on S3); an in-process backend ignores both.
|
||||
//
|
||||
// A Body handed back by Get aliases the cache's copy and must not be mutated.
|
||||
type Cache interface {
|
||||
Get(ctx context.Context, key string) (CacheEntry, CacheStatus, error)
|
||||
Put(ctx context.Context, key string, body []byte) error
|
||||
Stats() CacheStats
|
||||
}
|
||||
|
||||
// cacheKey is the request path when there are no params, else "<path>?<params>"
|
||||
// where params is url.Values.Encode() over a copy whose repeated values have
|
||||
// been sorted. Encode() already sorts keys ascending, so both the order params
|
||||
// arrive in and the order of a repeated param's values are irrelevant to the
|
||||
// key: one canonical string per distinct request.
|
||||
func cacheKey(path string, params url.Values) string {
|
||||
if len(params) == 0 {
|
||||
return path
|
||||
}
|
||||
norm := make(url.Values, len(params))
|
||||
for k, vs := range params {
|
||||
sorted := append([]string(nil), vs...)
|
||||
sort.Strings(sorted)
|
||||
norm[k] = sorted
|
||||
}
|
||||
encoded := norm.Encode()
|
||||
if encoded == "" {
|
||||
return path
|
||||
}
|
||||
return path + "?" + encoded
|
||||
}
|
||||
|
||||
// noopCache is the default for every path with no cache configured, so wiring a
|
||||
// handler through the cache leaves its behaviour unchanged.
|
||||
type noopCache struct{}
|
||||
|
||||
func (noopCache) Get(context.Context, string) (CacheEntry, CacheStatus, error) {
|
||||
return CacheEntry{}, CacheMiss, nil
|
||||
}
|
||||
|
||||
func (noopCache) Put(context.Context, string, []byte) error { return nil }
|
||||
|
||||
func (noopCache) Stats() CacheStats { return CacheStats{Backend: "none"} }
|
||||
|
||||
type memoryEntry struct {
|
||||
key string
|
||||
body []byte
|
||||
storedAt time.Time
|
||||
}
|
||||
|
||||
// memoryCache is a byte-bounded LRU. Expired entries are kept, not dropped, so
|
||||
// they remain available as a stale fallback; only the byte budget evicts.
|
||||
type memoryCache struct {
|
||||
ttl time.Duration
|
||||
maxBytes int64
|
||||
now func() time.Time
|
||||
|
||||
mu sync.Mutex
|
||||
ll *list.List // front = most recently used
|
||||
items map[string]*list.Element
|
||||
bytes int64
|
||||
}
|
||||
|
||||
func newMemoryCache(ttl time.Duration, maxBytes int64) *memoryCache {
|
||||
return &memoryCache{
|
||||
ttl: ttl,
|
||||
maxBytes: maxBytes,
|
||||
now: time.Now,
|
||||
ll: list.New(),
|
||||
items: make(map[string]*list.Element),
|
||||
}
|
||||
}
|
||||
|
||||
func (c *memoryCache) Get(_ context.Context, key string) (CacheEntry, CacheStatus, error) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
el, ok := c.items[key]
|
||||
if !ok {
|
||||
return CacheEntry{}, CacheMiss, nil
|
||||
}
|
||||
c.ll.MoveToFront(el)
|
||||
e := el.Value.(*memoryEntry)
|
||||
status := CacheFresh
|
||||
if c.now().Sub(e.storedAt) >= c.ttl {
|
||||
status = CacheStale
|
||||
}
|
||||
return CacheEntry{Body: e.body, StoredAt: e.storedAt}, status, nil
|
||||
}
|
||||
|
||||
func (c *memoryCache) Put(_ context.Context, key string, body []byte) error {
|
||||
// A response larger than the whole budget would evict everything else.
|
||||
if int64(len(body)) > c.maxBytes {
|
||||
return nil
|
||||
}
|
||||
stored := append([]byte(nil), body...)
|
||||
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if el, ok := c.items[key]; ok {
|
||||
e := el.Value.(*memoryEntry)
|
||||
c.bytes += int64(len(stored)) - int64(len(e.body))
|
||||
e.body, e.storedAt = stored, c.now()
|
||||
c.ll.MoveToFront(el)
|
||||
} else {
|
||||
c.items[key] = c.ll.PushFront(&memoryEntry{key: key, body: stored, storedAt: c.now()})
|
||||
c.bytes += int64(len(stored))
|
||||
}
|
||||
for c.bytes > c.maxBytes {
|
||||
back := c.ll.Back()
|
||||
if back == nil {
|
||||
break
|
||||
}
|
||||
e := c.ll.Remove(back).(*memoryEntry)
|
||||
delete(c.items, e.key)
|
||||
c.bytes -= int64(len(e.body))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *memoryCache) Stats() CacheStats {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
st := CacheStats{Backend: "memory", Entries: len(c.items), Bytes: c.bytes}
|
||||
now := c.now()
|
||||
for el := c.ll.Front(); el != nil; el = el.Next() {
|
||||
if now.Sub(el.Value.(*memoryEntry).storedAt) >= c.ttl {
|
||||
st.StaleEntries++
|
||||
}
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// flightGroup collapses concurrent identical builds so N simultaneous requests
|
||||
// for one key cause one upstream fan-out.
|
||||
type flightGroup struct {
|
||||
mu sync.Mutex
|
||||
calls map[string]*flightCall
|
||||
}
|
||||
|
||||
type flightCall struct {
|
||||
wg sync.WaitGroup
|
||||
resp cachedResponse
|
||||
err error
|
||||
}
|
||||
|
||||
// flightPanic is a panic from a flight's fn, reported to the leader and to every
|
||||
// waiter as an error so callers keep their error handling (stale fallback, 502)
|
||||
// instead of seeing a zero-value success.
|
||||
type flightPanic struct {
|
||||
value any
|
||||
stack []byte
|
||||
}
|
||||
|
||||
func (p *flightPanic) Error() string {
|
||||
return fmt.Sprintf("panic building response: %v\n%s", p.value, p.stack)
|
||||
}
|
||||
|
||||
// Do returns fn's result and whether this caller shared another's in-flight run.
|
||||
func (g *flightGroup) Do(key string, fn func() (cachedResponse, error)) (resp cachedResponse, err error, shared bool) {
|
||||
g.mu.Lock()
|
||||
if g.calls == nil {
|
||||
g.calls = make(map[string]*flightCall)
|
||||
}
|
||||
if c, ok := g.calls[key]; ok {
|
||||
g.mu.Unlock()
|
||||
c.wg.Wait()
|
||||
return c.resp, c.err, true
|
||||
}
|
||||
c := &flightCall{}
|
||||
c.wg.Add(1)
|
||||
g.calls[key] = c
|
||||
g.mu.Unlock()
|
||||
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
c.resp, c.err = cachedResponse{}, &flightPanic{value: r, stack: debug.Stack()}
|
||||
resp, err = c.resp, c.err
|
||||
}
|
||||
// Done only after the results are stored, so waiters read them.
|
||||
c.wg.Done()
|
||||
g.mu.Lock()
|
||||
delete(g.calls, key)
|
||||
g.mu.Unlock()
|
||||
}()
|
||||
|
||||
c.resp, c.err = fn()
|
||||
return c.resp, c.err, false
|
||||
}
|
||||
|
||||
// staleTracker records stale fallbacks for /healthz. serving flips back to false
|
||||
// as soon as a response is served from a live fan-out or a fresh entry.
|
||||
type staleTracker struct {
|
||||
mu sync.Mutex
|
||||
serving bool
|
||||
served uint64
|
||||
last time.Time
|
||||
}
|
||||
|
||||
func (t *staleTracker) markStale(now time.Time) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.serving = true
|
||||
t.served++
|
||||
t.last = now
|
||||
}
|
||||
|
||||
func (t *staleTracker) markFresh() {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.serving = false
|
||||
}
|
||||
|
||||
func (t *staleTracker) snapshot() (serving bool, served uint64, last time.Time) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
return t.serving, t.served, t.last
|
||||
}
|
||||
+821
@@ -0,0 +1,821 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// fakeClock drives the cache's TTL without sleeping.
|
||||
type fakeClock struct {
|
||||
mu sync.Mutex
|
||||
t time.Time
|
||||
}
|
||||
|
||||
func newFakeClock() *fakeClock { return &fakeClock{t: time.Unix(1_800_000_000, 0)} }
|
||||
|
||||
func (c *fakeClock) now() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.t
|
||||
}
|
||||
|
||||
func (c *fakeClock) advance(d time.Duration) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.t = c.t.Add(d)
|
||||
}
|
||||
|
||||
// countingBackend is a PuppetDB stand-in that counts requests per path and can
|
||||
// be made to fail or block, so cache hits and single-flight are observable.
|
||||
type countingBackend struct {
|
||||
srv *httptest.Server
|
||||
|
||||
mu sync.Mutex
|
||||
hits map[string]int
|
||||
bodies map[string]string
|
||||
fail bool
|
||||
block chan struct{}
|
||||
}
|
||||
|
||||
func newCountingBackend(t *testing.T, bodies map[string]string) *countingBackend {
|
||||
t.Helper()
|
||||
cb := &countingBackend{hits: map[string]int{}, bodies: bodies}
|
||||
cb.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
cb.mu.Lock()
|
||||
cb.hits[r.URL.Path]++
|
||||
fail, block, body := cb.fail, cb.block, cb.bodies[r.URL.Path]
|
||||
cb.mu.Unlock()
|
||||
|
||||
if block != nil {
|
||||
<-block
|
||||
}
|
||||
if fail {
|
||||
http.Error(w, "boom", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if body == "" {
|
||||
body = "[]"
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = io.WriteString(w, body)
|
||||
}))
|
||||
t.Cleanup(cb.srv.Close)
|
||||
return cb
|
||||
}
|
||||
|
||||
func (cb *countingBackend) hitCount(path string) int {
|
||||
cb.mu.Lock()
|
||||
defer cb.mu.Unlock()
|
||||
return cb.hits[path]
|
||||
}
|
||||
|
||||
func (cb *countingBackend) setFail(v bool) {
|
||||
cb.mu.Lock()
|
||||
defer cb.mu.Unlock()
|
||||
cb.fail = v
|
||||
}
|
||||
|
||||
func (cb *countingBackend) setBody(path, body string) {
|
||||
cb.mu.Lock()
|
||||
defer cb.mu.Unlock()
|
||||
cb.bodies[path] = body
|
||||
}
|
||||
|
||||
func (cb *countingBackend) setBlock(ch chan struct{}) {
|
||||
cb.mu.Lock()
|
||||
defer cb.mu.Unlock()
|
||||
cb.block = ch
|
||||
}
|
||||
|
||||
// newCachedServer builds a server whose facts cache runs on a fake clock.
|
||||
func newCachedServer(t *testing.T, cfg Config) (*Server, *fakeClock) {
|
||||
t.Helper()
|
||||
srv := newTestServer(cfg)
|
||||
mc, ok := srv.factsCache.(*memoryCache)
|
||||
if !ok {
|
||||
t.Fatalf("expected a memory cache, got %T", srv.factsCache)
|
||||
}
|
||||
clk := newFakeClock()
|
||||
mc.now = clk.now
|
||||
return srv, clk
|
||||
}
|
||||
|
||||
func cacheTestConfig(aURL, bURL string) Config {
|
||||
cfg := testConfig(aURL, bURL, mergeStatic)
|
||||
cfg.FactsTTL = 30 * time.Second
|
||||
cfg.CacheBytes = defaultCacheSize
|
||||
return cfg
|
||||
}
|
||||
|
||||
func TestCacheKey_CanonicalOrdering(t *testing.T) {
|
||||
a := cacheKey(factsPath, url.Values{
|
||||
"query": {`["=","name","role"]`},
|
||||
"limit": {"10"},
|
||||
"expand": {"z", "a"},
|
||||
})
|
||||
b := cacheKey(factsPath, url.Values{
|
||||
"expand": {"a", "z"},
|
||||
"limit": {"10"},
|
||||
"query": {`["=","name","role"]`},
|
||||
})
|
||||
if a != b {
|
||||
t.Errorf("param order must not change the key:\n %s\n %s", a, b)
|
||||
}
|
||||
if !strings.HasPrefix(a, factsPath+"?") {
|
||||
t.Errorf("key must start with the path, got %q", a)
|
||||
}
|
||||
if got := cacheKey(nodesPath, nil); got != nodesPath {
|
||||
t.Errorf("no params should give the bare path, got %q", got)
|
||||
}
|
||||
if cacheKey(factsPath, url.Values{"query": {"x"}}) == cacheKey(factsPath, url.Values{"query": {"y"}}) {
|
||||
t.Error("different queries must not share a key")
|
||||
}
|
||||
if cacheKey(factsPath, url.Values{"query": {"x"}}) == cacheKey(nodesPath, url.Values{"query": {"x"}}) {
|
||||
t.Error("different paths must not share a key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMemoryCache_FreshThenStale(t *testing.T) {
|
||||
c := newMemoryCache(30*time.Second, 1<<20)
|
||||
clk := newFakeClock()
|
||||
c.now = clk.now
|
||||
|
||||
if _, status, _ := c.Get(context.Background(), "k"); status != CacheMiss {
|
||||
t.Fatalf("empty cache should miss, got %s", status)
|
||||
}
|
||||
if err := c.Put(context.Background(), "k", []byte(`{"body":[]}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ent, status, err := c.Get(context.Background(), "k")
|
||||
if err != nil || status != CacheFresh || string(ent.Body) != `{"body":[]}` {
|
||||
t.Fatalf("want a fresh hit, got %s %q (%v)", status, ent.Body, err)
|
||||
}
|
||||
|
||||
clk.advance(29 * time.Second)
|
||||
if _, status, _ := c.Get(context.Background(), "k"); status != CacheFresh {
|
||||
t.Fatalf("inside the TTL should still be fresh, got %s", status)
|
||||
}
|
||||
|
||||
clk.advance(2 * time.Second)
|
||||
ent, status, _ = c.Get(context.Background(), "k")
|
||||
if status != CacheStale {
|
||||
t.Fatalf("past the TTL should be stale, got %s", status)
|
||||
}
|
||||
if string(ent.Body) != `{"body":[]}` {
|
||||
t.Errorf("a stale entry must still carry its body, got %q", ent.Body)
|
||||
}
|
||||
if st := c.Stats(); st.Entries != 1 || st.StaleEntries != 1 {
|
||||
t.Errorf("stats should report 1 entry, 1 stale, got %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMemoryCache_EvictsLeastRecentlyUsed(t *testing.T) {
|
||||
body := []byte("0123456789") // 10 bytes
|
||||
c := newMemoryCache(time.Minute, 25)
|
||||
ctx := context.Background()
|
||||
|
||||
for _, k := range []string{"a", "b"} {
|
||||
if err := c.Put(ctx, k, body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
// Reading "a" makes "b" the eviction candidate.
|
||||
if _, status, _ := c.Get(ctx, "a"); status != CacheFresh {
|
||||
t.Fatalf("a should be cached, got %s", status)
|
||||
}
|
||||
if err := c.Put(ctx, "c", body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if _, status, _ := c.Get(ctx, "b"); status != CacheMiss {
|
||||
t.Errorf("b was least recently used and should have been evicted, got %s", status)
|
||||
}
|
||||
for _, k := range []string{"a", "c"} {
|
||||
if _, status, _ := c.Get(ctx, k); status != CacheFresh {
|
||||
t.Errorf("%s should have survived eviction, got %s", k, status)
|
||||
}
|
||||
}
|
||||
if st := c.Stats(); st.Entries != 2 || st.Bytes != 20 {
|
||||
t.Errorf("stats after eviction = %+v, want 2 entries / 20 bytes", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMemoryCache_SkipsOversizedEntry(t *testing.T) {
|
||||
c := newMemoryCache(time.Minute, 8)
|
||||
if err := c.Put(context.Background(), "big", []byte("123456789")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, status, _ := c.Get(context.Background(), "big"); status != CacheMiss {
|
||||
t.Error("an entry larger than the whole budget must not be stored")
|
||||
}
|
||||
if st := c.Stats(); st.Bytes != 0 {
|
||||
t.Errorf("bytes = %d, want 0", st.Bytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoopCache_AlwaysMisses(t *testing.T) {
|
||||
var c Cache = noopCache{}
|
||||
if err := c.Put(context.Background(), "k", []byte("x")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, status, _ := c.Get(context.Background(), "k"); status != CacheMiss {
|
||||
t.Errorf("noop cache must always miss, got %s", status)
|
||||
}
|
||||
if st := c.Stats(); st.Backend != "none" {
|
||||
t.Errorf("backend = %q, want none", st.Backend)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_FactsCacheFreshHit(t *testing.T) {
|
||||
body := `[` + fact("h1", "role", "web", "") + `]`
|
||||
a := newCountingBackend(t, map[string]string{factsPath: body})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
first := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`)
|
||||
second := doGet(t, srv.Handler(), factsPath, `["=","name","role"]`)
|
||||
|
||||
if first.Code != http.StatusOK || second.Code != http.StatusOK {
|
||||
t.Fatalf("statuses %d/%d", first.Code, second.Code)
|
||||
}
|
||||
if first.Body.String() != second.Body.String() {
|
||||
t.Errorf("cache hit changed the body:\n %s\n %s", first.Body.String(), second.Body.String())
|
||||
}
|
||||
if got := a.hitCount(factsPath); got != 1 {
|
||||
t.Errorf("backend a saw %d requests, want 1 (second served from cache)", got)
|
||||
}
|
||||
if got := b.hitCount(factsPath); got != 1 {
|
||||
t.Errorf("backend b saw %d requests, want 1", got)
|
||||
}
|
||||
|
||||
// A different query is a different key and must go upstream.
|
||||
doGet(t, srv.Handler(), factsPath, `["=","name","osfamily"]`)
|
||||
if got := a.hitCount(factsPath); got != 2 {
|
||||
t.Errorf("a different query should refetch: %d requests, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_FactsCacheExpires(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
doGet(t, srv.Handler(), factsPath, "")
|
||||
clk.advance(31 * time.Second)
|
||||
doGet(t, srv.Handler(), factsPath, "")
|
||||
|
||||
if got := a.hitCount(factsPath); got != 2 {
|
||||
t.Errorf("an expired entry should refetch: %d requests, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_ServesStaleOnlyWhenBackendsFail(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "old", "") + `]`})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
warm := doGet(t, srv.Handler(), factsPath, "")
|
||||
if warm.Code != http.StatusOK {
|
||||
t.Fatalf("warm-up status %d", warm.Code)
|
||||
}
|
||||
if !strings.Contains(warm.Body.String(), `"old"`) {
|
||||
t.Fatalf("warm-up body = %s", warm.Body.String())
|
||||
}
|
||||
|
||||
// Backends healthy but the entry expired: fresh data wins, never the stale copy.
|
||||
clk.advance(31 * time.Second)
|
||||
a.setBody(factsPath, `[`+fact("h1", "role", "new", "")+`]`)
|
||||
refetch := doGet(t, srv.Handler(), factsPath, "")
|
||||
if !strings.Contains(refetch.Body.String(), `"new"`) {
|
||||
t.Errorf("a healthy backend must not be shadowed by the stale entry: %s", refetch.Body.String())
|
||||
}
|
||||
if serving, _, _ := srv.stale.snapshot(); serving {
|
||||
t.Error("serving_stale must stay false while backends are healthy")
|
||||
}
|
||||
|
||||
// Every backend down and the entry expired: the stale copy is served.
|
||||
clk.advance(31 * time.Second)
|
||||
a.setFail(true)
|
||||
b.setFail(true)
|
||||
stale := doGet(t, srv.Handler(), factsPath, "")
|
||||
if stale.Code != http.StatusOK {
|
||||
t.Fatalf("stale fallback status %d: %s", stale.Code, stale.Body.String())
|
||||
}
|
||||
if !strings.Contains(stale.Body.String(), `"new"`) {
|
||||
t.Errorf("stale body = %s, want the last cached copy", stale.Body.String())
|
||||
}
|
||||
|
||||
serving, served, last := srv.stale.snapshot()
|
||||
if !serving || served != 1 || last.IsZero() {
|
||||
t.Errorf("stale tracker = %v/%d/%v, want serving=true served=1", serving, served, last)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_NoCacheEntryMeansBackendFailureIs502(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
a.setFail(true)
|
||||
b.setFail(true)
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
rec := doGet(t, srv.Handler(), factsPath, "")
|
||||
if rec.Code != http.StatusBadGateway {
|
||||
t.Fatalf("status %d, want 502", rec.Code)
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), "all backends failed") {
|
||||
t.Errorf("body = %q", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_SingleFlightCollapsesConcurrentRequests(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
h := srv.Handler()
|
||||
|
||||
release := make(chan struct{})
|
||||
a.setBlock(release)
|
||||
b.setBlock(release)
|
||||
|
||||
const callers = 16
|
||||
var wg sync.WaitGroup
|
||||
codes := make([]int, callers)
|
||||
for i := range callers {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
codes[i] = doGet(t, h, factsPath, "").Code
|
||||
}(i)
|
||||
}
|
||||
|
||||
// The leader is parked inside the blocked backend, so every caller that
|
||||
// reaches the handler before the release joins its flight.
|
||||
waitFor(t, func() bool { return a.hitCount(factsPath) >= 1 })
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
close(release)
|
||||
wg.Wait()
|
||||
|
||||
for i, code := range codes {
|
||||
if code != http.StatusOK {
|
||||
t.Fatalf("caller %d got %d", i, code)
|
||||
}
|
||||
}
|
||||
if got := a.hitCount(factsPath); got != 1 {
|
||||
t.Errorf("backend a saw %d requests, want 1 for %d concurrent callers", got, callers)
|
||||
}
|
||||
if got := b.hitCount(factsPath); got != 1 {
|
||||
t.Errorf("backend b saw %d requests, want 1 for %d concurrent callers", got, callers)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFlightGroup_LeaderRunsOnce(t *testing.T) {
|
||||
var g flightGroup
|
||||
var calls, shared atomic.Int64
|
||||
entered := make(chan struct{})
|
||||
release := make(chan struct{})
|
||||
|
||||
run := func() (cachedResponse, error) {
|
||||
if calls.Add(1) == 1 {
|
||||
close(entered)
|
||||
}
|
||||
<-release
|
||||
return cachedResponse{Body: json.RawMessage(`[]`), Records: -1}, nil
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for range 16 {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if _, _, s := g.Do("k", run); s {
|
||||
shared.Add(1)
|
||||
}
|
||||
}()
|
||||
}
|
||||
<-entered
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
close(release)
|
||||
wg.Wait()
|
||||
|
||||
if calls.Load() != 1 {
|
||||
t.Errorf("fn ran %d times, want 1", calls.Load())
|
||||
}
|
||||
if shared.Load() != 15 {
|
||||
t.Errorf("%d callers shared the flight, want 15", shared.Load())
|
||||
}
|
||||
// The key is released once the flight finishes.
|
||||
if _, _, s := g.Do("k", func() (cachedResponse, error) { return cachedResponse{Records: -1}, nil }); s {
|
||||
t.Error("a later call must start its own flight")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFlightGroup_LeaderPanicFailsLeaderAndWaiters(t *testing.T) {
|
||||
var g flightGroup
|
||||
entered := make(chan struct{})
|
||||
release := make(chan struct{})
|
||||
|
||||
type result struct {
|
||||
resp cachedResponse
|
||||
err error
|
||||
shared bool
|
||||
}
|
||||
waiters := make([]result, 8)
|
||||
var wg sync.WaitGroup
|
||||
|
||||
var leader result
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
leader.resp, leader.err, leader.shared = g.Do("k", func() (cachedResponse, error) {
|
||||
close(entered)
|
||||
<-release
|
||||
panic("build exploded")
|
||||
})
|
||||
}()
|
||||
|
||||
<-entered
|
||||
for i := range waiters {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
waiters[i].resp, waiters[i].err, waiters[i].shared = g.Do("k", func() (cachedResponse, error) {
|
||||
t.Error("a waiter must not run its own fn")
|
||||
return cachedResponse{}, nil
|
||||
})
|
||||
}(i)
|
||||
}
|
||||
// Park the waiters in Wait() before the leader panics.
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
close(release)
|
||||
wg.Wait()
|
||||
|
||||
if leader.err == nil {
|
||||
t.Errorf("leader err = nil, want a panic error (resp %+v)", leader.resp)
|
||||
}
|
||||
for i, w := range waiters {
|
||||
if !w.shared {
|
||||
t.Errorf("waiter %d did not share the flight", i)
|
||||
}
|
||||
if w.err == nil {
|
||||
t.Fatalf("waiter %d unblocked with err = nil and resp %+v, want an error", i, w.resp)
|
||||
}
|
||||
if !strings.Contains(w.err.Error(), "build exploded") {
|
||||
t.Errorf("waiter %d err = %v, want the panic value", i, w.err)
|
||||
}
|
||||
}
|
||||
|
||||
// The key is released on the panic path, so a later call leads its own flight.
|
||||
resp, err, shared := g.Do("k", func() (cachedResponse, error) {
|
||||
return cachedResponse{Body: json.RawMessage(`[]`), Records: 0}, nil
|
||||
})
|
||||
if shared {
|
||||
t.Error("a call after a panicking flight must start its own flight")
|
||||
}
|
||||
if err != nil || resp.Records != 0 {
|
||||
t.Errorf("later call = (%+v, %v), want a clean success", resp, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeCached_BuildPanicIs502ThenStale(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
panicBuild := func() (cachedResponse, error) { panic("build exploded") }
|
||||
serve := func(build func() (cachedResponse, error)) *httptest.ResponseRecorder {
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, factsPath, nil)
|
||||
srv.serveCached(rec, req, factsPath, nil, build)
|
||||
return rec
|
||||
}
|
||||
|
||||
// No entry to fall back to: a panicking build must not look like a success.
|
||||
rec := serve(panicBuild)
|
||||
if rec.Code != http.StatusBadGateway {
|
||||
t.Fatalf("status %d (%s), want 502", rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
stored := `[` + fact("h1", "role", "web", "") + `]`
|
||||
rec = serve(func() (cachedResponse, error) {
|
||||
return cachedResponse{Body: json.RawMessage(stored), Records: 1}, nil
|
||||
})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("priming status %d (%s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
clk.advance(31 * time.Second)
|
||||
rec = serve(panicBuild)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("stale fallback status %d (%s), want 200", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := strings.TrimSpace(rec.Body.String()); got != stored {
|
||||
t.Errorf("stale body = %s, want %s", got, stored)
|
||||
}
|
||||
if got := rec.Header().Get(recordsHeader); got != "1" {
|
||||
t.Errorf("%s = %q, want 1", recordsHeader, got)
|
||||
}
|
||||
if hr := health(t, srv); !hr.Cache.ServingStale || hr.Cache.StaleServed != 1 {
|
||||
t.Errorf("panic fallback not counted as stale: %+v", hr.Cache)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeCached_BuildPanicFailsConcurrentWaiters(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
entered := make(chan struct{})
|
||||
release := make(chan struct{})
|
||||
serve := func(build func() (cachedResponse, error)) *httptest.ResponseRecorder {
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, factsPath, nil)
|
||||
srv.serveCached(rec, req, factsPath, nil, build)
|
||||
return rec
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(1)
|
||||
var leaderRec *httptest.ResponseRecorder
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
leaderRec = serve(func() (cachedResponse, error) {
|
||||
close(entered)
|
||||
<-release
|
||||
panic("build exploded")
|
||||
})
|
||||
}()
|
||||
|
||||
<-entered
|
||||
const waiters = 8
|
||||
recs := make([]*httptest.ResponseRecorder, waiters)
|
||||
for i := range waiters {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
recs[i] = serve(func() (cachedResponse, error) {
|
||||
t.Error("a waiter must not run its own build")
|
||||
return cachedResponse{}, nil
|
||||
})
|
||||
}(i)
|
||||
}
|
||||
time.Sleep(250 * time.Millisecond)
|
||||
close(release)
|
||||
wg.Wait()
|
||||
|
||||
if leaderRec.Code != http.StatusBadGateway {
|
||||
t.Errorf("leader status %d, want 502", leaderRec.Code)
|
||||
}
|
||||
for i, rec := range recs {
|
||||
if rec.Code != http.StatusBadGateway {
|
||||
t.Fatalf("waiter %d got %d (%s), want 502 rather than an empty success",
|
||||
i, rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_HealthzReportsCacheState(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{factsPath: `[` + fact("h1", "role", "web", "") + `]`})
|
||||
b := newCountingBackend(t, map[string]string{factsPath: `[]`})
|
||||
srv, clk := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
hr := health(t, srv)
|
||||
if hr.Cache.Backend != "memory" || hr.Cache.TTL != "30s" {
|
||||
t.Fatalf("cache health = %+v, want memory/30s", hr.Cache)
|
||||
}
|
||||
if hr.Cache.Entries != 0 || hr.Cache.ServingStale {
|
||||
t.Fatalf("a cold cache should be empty and not stale: %+v", hr.Cache)
|
||||
}
|
||||
|
||||
doGet(t, srv.Handler(), factsPath, "")
|
||||
hr = health(t, srv)
|
||||
if hr.Cache.Entries != 1 || hr.Cache.StaleEntries != 0 || hr.Cache.Bytes == 0 {
|
||||
t.Fatalf("after one request: %+v", hr.Cache)
|
||||
}
|
||||
|
||||
clk.advance(31 * time.Second)
|
||||
if hr = health(t, srv); hr.Cache.StaleEntries != 1 {
|
||||
t.Fatalf("expired entry should count as stale: %+v", hr.Cache)
|
||||
}
|
||||
if hr.Cache.ServingStale {
|
||||
t.Error("an expired entry alone is not serving_stale")
|
||||
}
|
||||
|
||||
a.setFail(true)
|
||||
b.setFail(true)
|
||||
doGet(t, srv.Handler(), factsPath, "")
|
||||
hr = health(t, srv)
|
||||
if !hr.Cache.ServingStale || hr.Cache.StaleServed != 1 || hr.Cache.LastStale == "" {
|
||||
t.Errorf("staleness not surfaced in /healthz: %+v", hr.Cache)
|
||||
}
|
||||
if hr.Status != "down" {
|
||||
t.Errorf("status = %q, want down", hr.Status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_HealthzReportsDisabledCache(t *testing.T) {
|
||||
a := newCountingBackend(t, nil)
|
||||
b := newCountingBackend(t, nil)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic)) // FactsTTL zero
|
||||
|
||||
hr := health(t, srv)
|
||||
if hr.Cache.Backend != "none" || hr.Cache.TTL != "0" {
|
||||
t.Errorf("disabled cache health = %+v, want none/0", hr.Cache)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_UncachedPathsStillRefetch(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{reportsPath: `[{"hash":"h"}]`})
|
||||
b := newCountingBackend(t, map[string]string{reportsPath: `[]`})
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
doGet(t, srv.Handler(), reportsPath, "")
|
||||
doGet(t, srv.Handler(), reportsPath, "")
|
||||
if got := a.hitCount(reportsPath); got != 2 {
|
||||
t.Errorf("/reports is uncached: %d requests, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_NodesRecordSetCached(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{nodesPath: `[{"certname":"h1"}]`})
|
||||
b := newCountingBackend(t, map[string]string{nodesPath: `[]`})
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
doGet(t, srv.Handler(), nodesPath, "")
|
||||
doGet(t, srv.Handler(), nodesPath, "")
|
||||
if got := a.hitCount(nodesPath); got != 1 {
|
||||
t.Errorf("merged /nodes is cached: %d requests, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_NodesAggregateNotCached(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{nodesPath: `[{"count":90}]`})
|
||||
b := newCountingBackend(t, map[string]string{nodesPath: `[{"count":53}]`})
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
first := doGet(t, srv.Handler(), nodesPath, nodeCountQuery)
|
||||
second := doGet(t, srv.Handler(), nodesPath, nodeCountQuery)
|
||||
if first.Code != http.StatusOK || second.Code != http.StatusOK {
|
||||
t.Fatalf("statuses %d/%d", first.Code, second.Code)
|
||||
}
|
||||
if got := a.hitCount(nodesPath); got != 2 {
|
||||
t.Errorf("/nodes aggregates are uncached: %d requests, want 2", got)
|
||||
}
|
||||
if got := counts(t, second.Body.Bytes(), "count"); !slices.Equal(got, []float64{143}) {
|
||||
t.Errorf("count = %v, want [143]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_ResourcesAggregateNotCached(t *testing.T) {
|
||||
a := newCountingBackend(t, map[string]string{resourcesPath: `[{"count":7}]`})
|
||||
b := newCountingBackend(t, map[string]string{resourcesPath: `[{"count":5}]`})
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
doGet(t, srv.Handler(), resourcesPath, nodeCountQuery)
|
||||
doGet(t, srv.Handler(), resourcesPath, nodeCountQuery)
|
||||
if got := a.hitCount(resourcesPath); got != 2 {
|
||||
t.Errorf("/resources aggregates are uncached: %d requests, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_MetaAndMetricsNotCached(t *testing.T) {
|
||||
bodies := map[string]string{
|
||||
metaVersionPath: `{"version":"8.4.0"}`,
|
||||
metaServerTimePath: `{"server_time":"2026-01-01T00:00:00.000Z"}`,
|
||||
"/metrics/v2/read/x": `{"request":{},"value":{"Value":1},"status":200}`,
|
||||
}
|
||||
a := newCountingBackend(t, bodies)
|
||||
b := newCountingBackend(t, bodies)
|
||||
srv, _ := newCachedServer(t, cacheTestConfig(a.srv.URL, b.srv.URL))
|
||||
|
||||
for path := range bodies {
|
||||
doGet(t, srv.Handler(), path, "")
|
||||
doGet(t, srv.Handler(), path, "")
|
||||
if got := a.hitCount(path); got != 2 {
|
||||
t.Errorf("%s is uncached: %d requests, want 2", path, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_FactsTTLClampedAtCap(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Setenv("XDG_CONFIG_HOME", dir)
|
||||
clearEnv(t)
|
||||
|
||||
path := filepath.Join(dir, appName, configFileName)
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte("facts_ttl: 5m\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
cfg, err := Load("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.FactsTTL != maxFactsTTL {
|
||||
t.Errorf("file facts_ttl = %s, want it clamped to %s", cfg.FactsTTL, maxFactsTTL)
|
||||
}
|
||||
|
||||
t.Setenv(envPrefix+"FACTS_TTL", "10m")
|
||||
cfg, err = Load("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.FactsTTL != maxFactsTTL {
|
||||
t.Errorf("env facts_ttl = %s, want it clamped to %s", cfg.FactsTTL, maxFactsTTL)
|
||||
}
|
||||
if cfg.factsTTLClamped != 10*time.Minute {
|
||||
t.Errorf("pre-clamp value = %s, want 10m0s", cfg.factsTTLClamped)
|
||||
}
|
||||
|
||||
out := captureStdout(t, func() { printConfig(cfg) })
|
||||
if !strings.Contains(out, "facts_ttl : 30s (clamped from 600s, cap 30s)") {
|
||||
t.Errorf("config show must report the clamp, got:\n%s", out)
|
||||
}
|
||||
|
||||
t.Setenv(envPrefix+"FACTS_TTL", "5s")
|
||||
cfg, err = Load("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.FactsTTL != 5*time.Second || cfg.factsTTLClamped != 0 {
|
||||
t.Errorf("a value under the cap must pass through: %s / %s", cfg.FactsTTL, cfg.factsTTLClamped)
|
||||
}
|
||||
|
||||
t.Setenv(envPrefix+"FACTS_TTL", "0s")
|
||||
cfg, err = Load("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.cacheEnabled() {
|
||||
t.Error("facts_ttl 0 must disable the cache")
|
||||
}
|
||||
out = captureStdout(t, func() { printConfig(cfg) })
|
||||
if !strings.Contains(out, "(cache disabled)") {
|
||||
t.Errorf("config show must say the cache is off, got:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewServer_ClampsFactsTTL(t *testing.T) {
|
||||
cfg := testConfig("http://a.invalid", "http://b.invalid", mergeStatic)
|
||||
cfg.FactsTTL = time.Hour
|
||||
cfg.CacheBytes = 1 << 20
|
||||
srv := newTestServer(cfg)
|
||||
|
||||
mc, ok := srv.factsCache.(*memoryCache)
|
||||
if !ok {
|
||||
t.Fatalf("expected a memory cache, got %T", srv.factsCache)
|
||||
}
|
||||
if mc.ttl != maxFactsTTL {
|
||||
t.Errorf("cache ttl = %s, want %s", mc.ttl, maxFactsTTL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_ValidateRejectsNegativeCacheSettings(t *testing.T) {
|
||||
base := testConfig("http://a.invalid", "http://b.invalid", mergeStatic)
|
||||
for name, mutate := range map[string]func(*Config){
|
||||
"negative facts_ttl": func(c *Config) { c.FactsTTL = -time.Second },
|
||||
"negative facts_cache_bytes": func(c *Config) { c.CacheBytes = -1 },
|
||||
} {
|
||||
cfg := base
|
||||
mutate(&cfg)
|
||||
if err := cfg.Validate(); err == nil {
|
||||
t.Errorf("%s should not validate", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func health(t *testing.T, srv *Server) healthReport {
|
||||
t.Helper()
|
||||
rec := doGet(t, srv.Handler(), "/healthz", "")
|
||||
var hr healthReport
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &hr); err != nil {
|
||||
t.Fatalf("decode healthz: %v (%s)", err, rec.Body.String())
|
||||
}
|
||||
return hr
|
||||
}
|
||||
|
||||
func waitFor(t *testing.T, cond func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for !cond() {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("condition not met within 5s")
|
||||
}
|
||||
time.Sleep(time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -25,6 +25,13 @@ const (
|
||||
|
||||
defaultTimeout = 10 * time.Second
|
||||
defaultFreshnessTTL = 30 * time.Second
|
||||
|
||||
// maxFactsTTL is a hard cap, not a default: a larger configured value is
|
||||
// clamped down to it rather than rejected, so a stray env var cannot make a
|
||||
// container crash-loop.
|
||||
maxFactsTTL = 30 * time.Second
|
||||
defaultFactsTTL = 30 * time.Second
|
||||
defaultCacheSize = int64(64 << 20)
|
||||
)
|
||||
|
||||
var exampleBackends = []Backend{
|
||||
@@ -43,8 +50,11 @@ type Config struct {
|
||||
Merge string `yaml:"merge"`
|
||||
Timeout time.Duration `yaml:"timeout"`
|
||||
FreshnessTTL time.Duration `yaml:"freshness_ttl"`
|
||||
FactsTTL time.Duration `yaml:"facts_ttl"` // 0 disables the /facts+/nodes cache
|
||||
CacheBytes int64 `yaml:"facts_cache_bytes"` // byte budget for that cache
|
||||
|
||||
sourcePath string // file this config was read from, empty if none was found
|
||||
sourcePath string // file this config was read from, empty if none was found
|
||||
factsTTLClamped time.Duration // pre-clamp facts_ttl, zero when nothing was clamped
|
||||
}
|
||||
|
||||
// SourcePath returns the config file Load read, or "" when none was loaded.
|
||||
@@ -61,6 +71,8 @@ func DefaultConfig() Config {
|
||||
Merge: mergeFreshness,
|
||||
Timeout: defaultTimeout,
|
||||
FreshnessTTL: defaultFreshnessTTL,
|
||||
FactsTTL: defaultFactsTTL,
|
||||
CacheBytes: defaultCacheSize,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -139,9 +151,19 @@ func Load(flagPath string) (Config, error) {
|
||||
}
|
||||
|
||||
applyEnv(&cfg, os.Getenv)
|
||||
cfg.clampFactsTTL()
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// clampFactsTTL pins facts_ttl to maxFactsTTL, remembering the configured value
|
||||
// so `config show` can say the cap was applied.
|
||||
func (c *Config) clampFactsTTL() {
|
||||
if c.FactsTTL > maxFactsTTL {
|
||||
c.factsTTLClamped = c.FactsTTL
|
||||
c.FactsTTL = maxFactsTTL
|
||||
}
|
||||
}
|
||||
|
||||
func applyEnv(cfg *Config, getenv func(string) string) {
|
||||
if v := getenv(envPrefix + "LISTEN"); v != "" {
|
||||
cfg.Listen = v
|
||||
@@ -159,6 +181,16 @@ func applyEnv(cfg *Config, getenv func(string) string) {
|
||||
cfg.FreshnessTTL = d
|
||||
}
|
||||
}
|
||||
if v := getenv(envPrefix + "FACTS_TTL"); v != "" {
|
||||
if d, err := time.ParseDuration(v); err == nil {
|
||||
cfg.FactsTTL = d
|
||||
}
|
||||
}
|
||||
if v := getenv(envPrefix + "FACTS_CACHE_BYTES"); v != "" {
|
||||
if n, err := strconv.ParseInt(v, 10, 64); err == nil {
|
||||
cfg.CacheBytes = n
|
||||
}
|
||||
}
|
||||
if v := getenv(envPrefix + "BACKENDS"); v != "" {
|
||||
if bs := parseBackends(v); len(bs) > 0 {
|
||||
cfg.Backends = bs
|
||||
@@ -216,9 +248,19 @@ func (c Config) Validate() error {
|
||||
if c.Timeout <= 0 {
|
||||
return fmt.Errorf("timeout must be positive")
|
||||
}
|
||||
if c.FactsTTL < 0 {
|
||||
return fmt.Errorf("facts_ttl must not be negative (0 disables the cache)")
|
||||
}
|
||||
if c.CacheBytes < 0 {
|
||||
return fmt.Errorf("facts_cache_bytes must not be negative")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// cacheEnabled reports whether a facts/nodes cache should be built: both a TTL
|
||||
// and a byte budget are required.
|
||||
func (c Config) cacheEnabled() bool { return c.FactsTTL > 0 && c.CacheBytes > 0 }
|
||||
|
||||
func writeDefaultConfig(path string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return fmt.Errorf("creating config dir: %w", err)
|
||||
@@ -231,7 +273,10 @@ func writeDefaultConfig(path string) error {
|
||||
"# A merging proxy presenting one PuppetDB v4 query surface over several\n" +
|
||||
"# PuppetDB backends. The backend URLs below are placeholders — edit them.\n" +
|
||||
"# Env overrides: PDBMUX_LISTEN, PDBMUX_MERGE, PDBMUX_TIMEOUT,\n" +
|
||||
"# PDBMUX_FRESHNESS_TTL, PDBMUX_BACKENDS (name=url,name=url).\n\n")
|
||||
"# PDBMUX_FRESHNESS_TTL, PDBMUX_FACTS_TTL, PDBMUX_FACTS_CACHE_BYTES,\n" +
|
||||
"# PDBMUX_BACKENDS (name=url,name=url).\n" +
|
||||
"# facts_ttl caches merged /facts and /nodes in memory; it is capped at 30s\n" +
|
||||
"# (a larger value is clamped) and 0 disables the cache.\n\n")
|
||||
if err := os.WriteFile(path, append(header, data...), 0o644); err != nil {
|
||||
return fmt.Errorf("writing config: %w", err)
|
||||
}
|
||||
|
||||
+1
-1
@@ -391,7 +391,7 @@ func captureStdout(t *testing.T, f func()) string {
|
||||
|
||||
func clearEnv(t *testing.T) {
|
||||
t.Helper()
|
||||
for _, k := range []string{"CONFIG", "LISTEN", "MERGE", "TIMEOUT", "FRESHNESS_TTL", "BACKENDS"} {
|
||||
for _, k := range []string{"CONFIG", "LISTEN", "MERGE", "TIMEOUT", "FRESHNESS_TTL", "FACTS_TTL", "FACTS_CACHE_BYTES", "BACKENDS"} {
|
||||
t.Setenv(envPrefix+k, "")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -151,6 +151,18 @@ func runServer(cfg Config) error {
|
||||
}
|
||||
}
|
||||
|
||||
func factsTTLString(cfg Config) string {
|
||||
s := durationString(cfg.FactsTTL)
|
||||
switch {
|
||||
case cfg.factsTTLClamped > 0:
|
||||
return fmt.Sprintf("%s (clamped from %s, cap %s)",
|
||||
s, durationString(cfg.factsTTLClamped), durationString(maxFactsTTL))
|
||||
case !cfg.cacheEnabled():
|
||||
return s + " (cache disabled)"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func printConfig(cfg Config) {
|
||||
if p := cfg.SourcePath(); p != "" {
|
||||
fmt.Printf("config file : %s (loaded)\n", p)
|
||||
@@ -161,6 +173,8 @@ func printConfig(cfg Config) {
|
||||
fmt.Printf("merge : %s\n", cfg.Merge)
|
||||
fmt.Printf("timeout : %s\n", durationString(cfg.Timeout))
|
||||
fmt.Printf("freshness_ttl: %s\n", durationString(cfg.FreshnessTTL))
|
||||
fmt.Printf("facts_ttl : %s\n", factsTTLString(cfg))
|
||||
fmt.Printf("facts_cache : %d bytes\n", cfg.CacheBytes)
|
||||
fmt.Println("backends:")
|
||||
for _, b := range cfg.Backends {
|
||||
fmt.Printf(" - %-8s %s\n", b.Name, b.URL)
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const (
|
||||
metaVersionPath = "/pdb/meta/v1/version"
|
||||
metaServerTimePath = "/pdb/meta/v1/server-time"
|
||||
)
|
||||
|
||||
// rawResult is one backend's verbatim response, for endpoints whose payload is
|
||||
// not a PuppetDB record array.
|
||||
type rawResult struct {
|
||||
name string
|
||||
status int
|
||||
contentType string
|
||||
body []byte
|
||||
err error
|
||||
}
|
||||
|
||||
// ok reports whether the backend answered 2xx.
|
||||
func (r rawResult) ok() bool {
|
||||
return r.err == nil && r.status >= 200 && r.status < 300
|
||||
}
|
||||
|
||||
// fanOutRaw asks every backend for path concurrently and returns one result per
|
||||
// backend, in configured order, without interpreting the bodies.
|
||||
func (s *Server) fanOutRaw(ctx context.Context, path, rawQuery string) []rawResult {
|
||||
results := make([]rawResult, len(s.cfg.Backends))
|
||||
var wg sync.WaitGroup
|
||||
for i, b := range s.cfg.Backends {
|
||||
wg.Add(1)
|
||||
go func(i int, b Backend) {
|
||||
defer wg.Done()
|
||||
results[i] = s.rawBackend(ctx, b, path, rawQuery)
|
||||
}(i, b)
|
||||
}
|
||||
wg.Wait()
|
||||
return results
|
||||
}
|
||||
|
||||
func (s *Server) rawBackend(ctx context.Context, b Backend, path, rawQuery string) rawResult {
|
||||
target := strings.TrimRight(b.URL, "/") + path
|
||||
if rawQuery != "" {
|
||||
target += "?" + rawQuery
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
|
||||
if err != nil {
|
||||
return rawResult{name: b.Name, err: err}
|
||||
}
|
||||
resp, err := s.client.Do(req)
|
||||
if err != nil {
|
||||
return rawResult{name: b.Name, err: err}
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return rawResult{name: b.Name, err: err}
|
||||
}
|
||||
return rawResult{
|
||||
name: b.Name,
|
||||
status: resp.StatusCode,
|
||||
contentType: resp.Header.Get("Content-Type"),
|
||||
body: body,
|
||||
}
|
||||
}
|
||||
|
||||
// aliveRaw drops backends that errored or answered non-2xx, writing a 502 and
|
||||
// returning ok=false only when none is left.
|
||||
func (s *Server) aliveRaw(w http.ResponseWriter, results []rawResult, path string) ([]rawResult, bool) {
|
||||
var alive []rawResult
|
||||
for _, res := range results {
|
||||
if !res.ok() {
|
||||
s.log.Printf("warning: backend %q failed for %s: %s", res.name, path, res.reason())
|
||||
continue
|
||||
}
|
||||
alive = append(alive, res)
|
||||
}
|
||||
if len(alive) == 0 {
|
||||
http.Error(w, "all backends failed", http.StatusBadGateway)
|
||||
return nil, false
|
||||
}
|
||||
return alive, true
|
||||
}
|
||||
|
||||
func (r rawResult) reason() string {
|
||||
if r.err != nil {
|
||||
return r.err.Error()
|
||||
}
|
||||
return "HTTP " + strconv.Itoa(r.status) + ": " + strings.TrimSpace(string(r.body))
|
||||
}
|
||||
|
||||
// handleMetaVersion serves /pdb/meta/v1/version. Clients (pypuppetdb, and so
|
||||
// Puppetboard's startup check) treat the answer as the feature level they may
|
||||
// rely on, so the merged answer is the *lowest* version any backend reports:
|
||||
// the estate can only be counted on for what its oldest member implements.
|
||||
func (s *Server) handleMetaVersion(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "only GET is supported", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
alive, ok := s.aliveRaw(w, s.fanOutRaw(r.Context(), metaVersionPath, r.URL.RawQuery), metaVersionPath)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
lowest := alive[0]
|
||||
lowestVer, hasVer := metaField(lowest.body, "version")
|
||||
for _, res := range alive[1:] {
|
||||
v, ok := metaField(res.body, "version")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if !hasVer || compareVersions(v, lowestVer) < 0 {
|
||||
lowest, lowestVer, hasVer = res, v, true
|
||||
}
|
||||
}
|
||||
writeRaw(w, lowest)
|
||||
}
|
||||
|
||||
// handleMetaServerTime serves /pdb/meta/v1/server-time. The clock of whichever
|
||||
// PuppetDB answered is not estate state and does not merge, so the first
|
||||
// reachable backend in configured order supplies it — the same tie-break rule
|
||||
// used everywhere else.
|
||||
func (s *Server) handleMetaServerTime(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "only GET is supported", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
alive, ok := s.aliveRaw(w, s.fanOutRaw(r.Context(), metaServerTimePath, r.URL.RawQuery), metaServerTimePath)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
writeRaw(w, alive[0])
|
||||
}
|
||||
|
||||
// metaField pulls a string field out of a `{"version": "..."}`-shaped body.
|
||||
func metaField(body []byte, field string) (string, bool) {
|
||||
var obj map[string]json.RawMessage
|
||||
if json.Unmarshal(body, &obj) != nil {
|
||||
return "", false
|
||||
}
|
||||
var s string
|
||||
if json.Unmarshal(obj[field], &s) != nil || s == "" {
|
||||
return "", false
|
||||
}
|
||||
return s, true
|
||||
}
|
||||
|
||||
// compareVersions orders dotted version strings segment by segment, comparing
|
||||
// numerically where both segments are numbers and lexically otherwise, so
|
||||
// "7.12.1" sorts below "8.4.0" and below "7.12.2". A prefix is lower than a
|
||||
// longer string sharing it ("7.12" < "7.12.1"), and a pre-release suffix is
|
||||
// compared as text within its segment ("8.0.0" < "8.0.0-SNAPSHOT").
|
||||
func compareVersions(a, b string) int {
|
||||
as, bs := strings.Split(a, "."), strings.Split(b, ".")
|
||||
for i := 0; i < len(as) && i < len(bs); i++ {
|
||||
an, aok := strconv.Atoi(as[i])
|
||||
bn, bok := strconv.Atoi(bs[i])
|
||||
if aok == nil && bok == nil {
|
||||
if an != bn {
|
||||
return sign(an - bn)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if c := strings.Compare(as[i], bs[i]); c != 0 {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return sign(len(as) - len(bs))
|
||||
}
|
||||
|
||||
func sign(n int) int {
|
||||
switch {
|
||||
case n < 0:
|
||||
return -1
|
||||
case n > 0:
|
||||
return 1
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
func writeRaw(w http.ResponseWriter, res rawResult) {
|
||||
setContentType(w, res.contentType)
|
||||
w.WriteHeader(res.status)
|
||||
_, _ = w.Write(res.body)
|
||||
}
|
||||
+170
@@ -0,0 +1,170 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func metaGet(t *testing.T, h http.Handler, path string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, path, nil))
|
||||
return rec
|
||||
}
|
||||
|
||||
func metaString(t *testing.T, body []byte, field string) string {
|
||||
t.Helper()
|
||||
var obj map[string]string
|
||||
if err := json.Unmarshal(body, &obj); err != nil {
|
||||
t.Fatalf("unmarshal %s: %v", body, err)
|
||||
}
|
||||
return obj[field]
|
||||
}
|
||||
|
||||
func TestMetaVersion_BackendsAgree(t *testing.T) {
|
||||
// Puppetboard's check_db_version() calls this at import and exits 2 on any
|
||||
// non-200, so a 404 here is the difference between a running dashboard and
|
||||
// CrashLoopBackOff.
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[metaVersionPath] = `{"version":"7.12.1"}`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[metaVersionPath] = `{"version":"7.12.1"}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := metaGet(t, srv.Handler(), metaVersionPath)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := metaString(t, rec.Body.Bytes(), "version"); got != "7.12.1" {
|
||||
t.Errorf("version = %q, want 7.12.1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaVersion_DisagreementReportsLowest(t *testing.T) {
|
||||
// The estate can only be relied on for what its oldest PuppetDB implements.
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[metaVersionPath] = `{"version":"8.4.0"}`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[metaVersionPath] = `{"version":"7.12.1"}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
if got := metaString(t, metaGet(t, srv.Handler(), metaVersionPath).Body.Bytes(), "version"); got != "7.12.1" {
|
||||
t.Errorf("version = %q, want the lower 7.12.1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaVersion_LowestIsIndependentOfBackendOrder(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[metaVersionPath] = `{"version":"7.12.1"}`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[metaVersionPath] = `{"version":"8.4.0"}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
if got := metaString(t, metaGet(t, srv.Handler(), metaVersionPath).Body.Bytes(), "version"); got != "7.12.1" {
|
||||
t.Errorf("version = %q, want the lower 7.12.1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaVersion_OneBackendDown(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.fail = true
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[metaVersionPath] = `{"version":"8.4.0"}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := metaGet(t, srv.Handler(), metaVersionPath)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200 serving the survivor, got %d", rec.Code)
|
||||
}
|
||||
if got := metaString(t, rec.Body.Bytes(), "version"); got != "8.4.0" {
|
||||
t.Errorf("version = %q, want 8.4.0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaVersion_AllBackendsDown(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.fail = true
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.fail = true
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
if rec := metaGet(t, srv.Handler(), metaVersionPath); rec.Code != http.StatusBadGateway {
|
||||
t.Errorf("status %d, want 502", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaServerTime_FirstReachableBackend(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T01:00:00.000Z"}`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T02:00:00.000Z"}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
got := metaString(t, metaGet(t, srv.Handler(), metaServerTimePath).Body.Bytes(), "server_time")
|
||||
if got != "2026-08-29T01:00:00.000Z" {
|
||||
t.Errorf("server_time = %q, want the first backend's", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaServerTime_SkipsDeadBackend(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.fail = true
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[metaServerTimePath] = `{"server_time":"2026-08-29T02:00:00.000Z"}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := metaGet(t, srv.Handler(), metaServerTimePath)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200 serving the survivor, got %d", rec.Code)
|
||||
}
|
||||
if got := metaString(t, rec.Body.Bytes(), "server_time"); got != "2026-08-29T02:00:00.000Z" {
|
||||
t.Errorf("server_time = %q, want the survivor's", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaVersion_RejectsNonGET(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
srv.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, metaVersionPath, nil))
|
||||
if rec.Code != http.StatusMethodNotAllowed {
|
||||
t.Errorf("status %d, want 405", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompareVersions(t *testing.T) {
|
||||
cases := []struct {
|
||||
a, b string
|
||||
want int
|
||||
}{
|
||||
{"7.12.1", "7.12.1", 0},
|
||||
{"7.12.1", "8.4.0", -1},
|
||||
{"8.4.0", "7.12.1", 1},
|
||||
{"7.9.0", "7.12.0", -1}, // numeric, not lexical: 9 < 12
|
||||
{"7.12", "7.12.1", -1},
|
||||
{"8.0.0", "8.0.0-SNAPSHOT", -1},
|
||||
{"8.0.0-SNAPSHOT", "8.0.0", 1},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := compareVersions(c.a, c.b); got != c.want {
|
||||
t.Errorf("compareVersions(%q, %q) = %d, want %d", c.a, c.b, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetaField_MalformedBodyIgnored(t *testing.T) {
|
||||
// A backend serving junk must not become the "lowest" version.
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[metaVersionPath] = `not json`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[metaVersionPath] = `{"version":"8.4.0"}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
if got := metaString(t, metaGet(t, srv.Handler(), metaVersionPath).Body.Bytes(), "version"); got != "8.4.0" {
|
||||
t.Errorf("version = %q, want 8.4.0 from the only parseable backend", got)
|
||||
}
|
||||
}
|
||||
+272
@@ -0,0 +1,272 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// metricsPrefix covers PuppetDB's Jolokia surface, which sits at the server root
|
||||
// rather than under /pdb: pypuppetdb's metric() reads /metrics/v2/read/<mbean>,
|
||||
// lists via /metrics/v2/list, and falls back to /metrics/v1/mbeans/<mbean>.
|
||||
const metricsPrefix = "/metrics/"
|
||||
|
||||
type mergeRule int
|
||||
|
||||
const (
|
||||
ruleSum mergeRule = iota
|
||||
ruleMin
|
||||
ruleMax
|
||||
ruleMean
|
||||
)
|
||||
|
||||
// mergeRuleFor picks how one numeric MBean attribute combines across backends.
|
||||
// The default is a sum: the metrics Puppetboard renders are population counts
|
||||
// (num-nodes, num-resources, queue depth, command totals) whose estate-wide
|
||||
// value is the total. The exceptions are attributes describing a distribution or
|
||||
// a bound rather than a quantity, where adding two servers' numbers yields a
|
||||
// figure that was never true of either.
|
||||
func mergeRuleFor(attr string) mergeRule {
|
||||
a := strings.ToLower(attr)
|
||||
switch a {
|
||||
case "min":
|
||||
return ruleMin
|
||||
case "max", "uptime", "starttime":
|
||||
return ruleMax
|
||||
case "mean", "median", "stddev":
|
||||
return ruleMean
|
||||
}
|
||||
if strings.HasSuffix(a, "percentile") {
|
||||
return ruleMean
|
||||
}
|
||||
return ruleSum
|
||||
}
|
||||
|
||||
func (s *Server) handleMetrics(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
http.Error(w, "only GET is supported", http.StatusMethodNotAllowed)
|
||||
return
|
||||
}
|
||||
// MBean names carry Jolokia's !-escapes under percent-encoding; the raw path
|
||||
// is forwarded so neither layer is lost.
|
||||
path := r.URL.EscapedPath()
|
||||
alive, ok := s.aliveRaw(w, s.fanOutRaw(r.Context(), path, r.URL.RawQuery), path)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
body, ok := mergeMetrics(alive, metricAttribute(r.URL.Path))
|
||||
if !ok {
|
||||
// Every backend answered 2xx but none carried a mergeable payload — a
|
||||
// Jolokia error envelope, or a body that is not a JSON object. Replaying
|
||||
// the first keeps the upstream error text the client expects.
|
||||
writeRaw(w, alive[0])
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write(body)
|
||||
}
|
||||
|
||||
// metricAttribute names the single attribute a read asked for, when the URL
|
||||
// carries one (/metrics/v2/read/<mbean>/<attribute>), so a scalar response body
|
||||
// still gets the right numeric rule. Empty means the response is an object whose
|
||||
// own keys name its attributes.
|
||||
func metricAttribute(path string) string {
|
||||
rest, ok := strings.CutPrefix(path, metricsPrefix)
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
parts := strings.Split(rest, "/") // v2/read/<mbean>[/<attribute>]
|
||||
if len(parts) < 4 {
|
||||
return ""
|
||||
}
|
||||
return parts[len(parts)-1]
|
||||
}
|
||||
|
||||
// mergeMetrics folds the backends' Jolokia responses into one. A response is
|
||||
// either a Jolokia envelope ({"request":…,"value":…,"status":200}), where only
|
||||
// "value" merges and the rest comes from the first backend, or a bare attribute
|
||||
// object (metrics/v1), which merges whole. ok=false means nothing was mergeable.
|
||||
func mergeMetrics(alive []rawResult, attr string) ([]byte, bool) {
|
||||
var objs []map[string]json.RawMessage
|
||||
for _, res := range alive {
|
||||
var obj map[string]json.RawMessage
|
||||
if decodeJSON(res.body, &obj) != nil || obj == nil {
|
||||
continue
|
||||
}
|
||||
// Jolokia reports a per-request failure inside an HTTP 200.
|
||||
if n, ok := numberOf(obj["status"]); ok && (n < 200 || n >= 300) {
|
||||
continue
|
||||
}
|
||||
objs = append(objs, obj)
|
||||
}
|
||||
if len(objs) == 0 {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
_, hasValue := objs[0]["value"]
|
||||
_, hasStatus := objs[0]["status"]
|
||||
if !hasValue || !hasStatus {
|
||||
vals := make([]any, 0, len(objs))
|
||||
for _, obj := range objs {
|
||||
vals = append(vals, decodedObject(obj))
|
||||
}
|
||||
return marshal(mergeMetricValue(vals, attr))
|
||||
}
|
||||
|
||||
vals := make([]any, 0, len(objs))
|
||||
for _, obj := range objs {
|
||||
var v any
|
||||
if decodeJSON(obj["value"], &v) == nil {
|
||||
vals = append(vals, v)
|
||||
}
|
||||
}
|
||||
if len(vals) == 0 {
|
||||
return nil, false
|
||||
}
|
||||
merged, ok := marshal(mergeMetricValue(vals, attr))
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
out := make(map[string]json.RawMessage, len(objs[0]))
|
||||
for k, v := range objs[0] {
|
||||
out[k] = v
|
||||
}
|
||||
out["value"] = merged
|
||||
// The envelope timestamp says when the answer was produced; the newest one
|
||||
// describes the merged answer.
|
||||
if ts, ok := maxField(objs, "timestamp"); ok {
|
||||
out["timestamp"] = ts
|
||||
}
|
||||
return marshal(out)
|
||||
}
|
||||
|
||||
// maxField returns the largest numeric value of a field across the responses.
|
||||
func maxField(objs []map[string]json.RawMessage, field string) (json.RawMessage, bool) {
|
||||
var best json.RawMessage
|
||||
var bestN float64
|
||||
for _, obj := range objs {
|
||||
n, ok := numberOf(obj[field])
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if best == nil || n > bestN {
|
||||
best, bestN = obj[field], n
|
||||
}
|
||||
}
|
||||
return best, best != nil
|
||||
}
|
||||
|
||||
// mergeMetricValue folds one attribute's value from every backend into one.
|
||||
// Objects merge key by key over the union of keys, so a backend missing an
|
||||
// attribute still contributes the rest. Numbers combine by the attribute's rule.
|
||||
// Anything else — strings, booleans, arrays, nulls, or a mix of kinds — keeps
|
||||
// the first backend's value, there being no sound way to add them.
|
||||
func mergeMetricValue(vals []any, attr string) any {
|
||||
if len(vals) == 0 {
|
||||
return nil
|
||||
}
|
||||
if len(vals) == 1 {
|
||||
return vals[0]
|
||||
}
|
||||
|
||||
objs := make([]map[string]any, 0, len(vals))
|
||||
for _, v := range vals {
|
||||
if m, ok := v.(map[string]any); ok {
|
||||
objs = append(objs, m)
|
||||
}
|
||||
}
|
||||
if len(objs) == len(vals) {
|
||||
out := map[string]any{}
|
||||
for _, m := range objs {
|
||||
for k := range m {
|
||||
if _, done := out[k]; done {
|
||||
continue
|
||||
}
|
||||
sub := make([]any, 0, len(objs))
|
||||
for _, o := range objs {
|
||||
if v, ok := o[k]; ok {
|
||||
sub = append(sub, v)
|
||||
}
|
||||
}
|
||||
out[k] = mergeMetricValue(sub, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
nums := make([]float64, 0, len(vals))
|
||||
for _, v := range vals {
|
||||
n, ok := v.(json.Number)
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
f, err := n.Float64()
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
nums = append(nums, f)
|
||||
}
|
||||
if len(nums) != len(vals) {
|
||||
return vals[0]
|
||||
}
|
||||
return combineNumbers(nums, attr)
|
||||
}
|
||||
|
||||
func combineNumbers(nums []float64, attr string) json.RawMessage {
|
||||
acc := nums[0]
|
||||
switch mergeRuleFor(attr) {
|
||||
case ruleMin:
|
||||
for _, n := range nums[1:] {
|
||||
if n < acc {
|
||||
acc = n
|
||||
}
|
||||
}
|
||||
case ruleMax:
|
||||
for _, n := range nums[1:] {
|
||||
if n > acc {
|
||||
acc = n
|
||||
}
|
||||
}
|
||||
case ruleMean:
|
||||
for _, n := range nums[1:] {
|
||||
acc += n
|
||||
}
|
||||
acc /= float64(len(nums))
|
||||
default:
|
||||
for _, n := range nums[1:] {
|
||||
acc += n
|
||||
}
|
||||
}
|
||||
return json.RawMessage(strconv.FormatFloat(acc, 'f', -1, 64))
|
||||
}
|
||||
|
||||
// decodedObject re-reads an object's fields as generic values so the whole thing
|
||||
// can go through mergeMetricValue.
|
||||
func decodedObject(obj map[string]json.RawMessage) any {
|
||||
out := make(map[string]any, len(obj))
|
||||
for k, raw := range obj {
|
||||
var v any
|
||||
if decodeJSON(raw, &v) == nil {
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// decodeJSON keeps integers exact by decoding numbers as json.Number.
|
||||
func decodeJSON(data []byte, v any) error {
|
||||
dec := json.NewDecoder(bytes.NewReader(data))
|
||||
dec.UseNumber()
|
||||
return dec.Decode(v)
|
||||
}
|
||||
|
||||
// marshal reports ok=false rather than an error: an unmarshalable merge result
|
||||
// has only one recovery, replaying a backend's body verbatim.
|
||||
func marshal(v any) ([]byte, bool) {
|
||||
b, err := json.Marshal(v)
|
||||
return b, err == nil
|
||||
}
|
||||
+285
@@ -0,0 +1,285 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const (
|
||||
numNodesMBean = "puppetlabs.puppetdb.population:name=num-nodes"
|
||||
numNodesPath = metricsPrefix + "v2/read/" + numNodesMBean
|
||||
// What pypuppetdb actually sends: quote() percent-encodes ':' and '='.
|
||||
numNodesEscaped = metricsPrefix + "v2/read/puppetlabs.puppetdb.population%3Aname%3Dnum-nodes"
|
||||
)
|
||||
|
||||
// jolokiaRead wraps an MBean value in the envelope PuppetDB's Jolokia returns.
|
||||
func jolokiaRead(mbean, value string, timestamp int) string {
|
||||
return `{"request":{"mbean":"` + mbean + `","type":"read"},` +
|
||||
`"value":` + value + `,"timestamp":` + strconv.Itoa(timestamp) + `,"status":200}`
|
||||
}
|
||||
|
||||
func metricValue(t *testing.T, body []byte) map[string]any {
|
||||
t.Helper()
|
||||
var env map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &env); err != nil {
|
||||
t.Fatalf("unmarshal envelope %s: %v", body, err)
|
||||
}
|
||||
var val map[string]any
|
||||
if err := json.Unmarshal(env["value"], &val); err != nil {
|
||||
t.Fatalf("unmarshal value %s: %v", env["value"], err)
|
||||
}
|
||||
return val
|
||||
}
|
||||
|
||||
func TestMetrics_ReadSumsPopulationCounts(t *testing.T) {
|
||||
// Puppetboard's landing page and radiator read num-nodes when
|
||||
// DEFAULT_ENVIRONMENT is '*'; each backend only knows its own nodes.
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":90}`, 1000)
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":53}`, 2000)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := metaGet(t, srv.Handler(), numNodesEscaped)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := metricValue(t, rec.Body.Bytes())["Value"]; got != float64(143) {
|
||||
t.Errorf("Value = %v, want 143", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_EscapedMBeanNameSurvives(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 1)
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 1)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
metaGet(t, srv.Handler(), numNodesEscaped)
|
||||
if !a.sawRawPath(numNodesEscaped) {
|
||||
t.Errorf("backend saw %v, want the percent-encoded path %q", a.rawPaths, numNodesEscaped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_EnvelopeKeepsNewestTimestamp(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 1000)
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":1}`, 2000)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
var env map[string]any
|
||||
if err := json.Unmarshal(metaGet(t, srv.Handler(), numNodesEscaped).Body.Bytes(), &env); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if env["timestamp"] != float64(2000) {
|
||||
t.Errorf("timestamp = %v, want 2000", env["timestamp"])
|
||||
}
|
||||
if env["status"] != float64(200) {
|
||||
t.Errorf("status = %v, want 200", env["status"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_PerAttributeRules(t *testing.T) {
|
||||
const mbean = "puppetlabs.puppetdb.mq:name=global.processing-time"
|
||||
path := metricsPrefix + "v2/read/" + mbean
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[path] = jolokiaRead(mbean,
|
||||
`{"Count":10,"Min":2,"Max":9,"Mean":4,"StdDev":1,"50thPercentile":3,"MeanRate":1.5}`, 1)
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[path] = jolokiaRead(mbean,
|
||||
`{"Count":6,"Min":1,"Max":20,"Mean":6,"StdDev":3,"50thPercentile":5,"MeanRate":0.5}`, 1)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
got := metricValue(t, metaGet(t, srv.Handler(), path).Body.Bytes())
|
||||
want := map[string]any{
|
||||
"Count": float64(16), // counts add
|
||||
"Min": float64(1), // a bound stays a bound
|
||||
"Max": float64(20),
|
||||
"Mean": float64(5), // distribution stats average
|
||||
"StdDev": float64(2),
|
||||
"50thPercentile": float64(4),
|
||||
"MeanRate": float64(2), // throughput adds
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("merged value = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_ScalarReadUsesURLAttributeName(t *testing.T) {
|
||||
// /metrics/v2/read/<mbean>/<attribute> answers with a bare number, so the
|
||||
// rule has to come from the URL rather than an object key.
|
||||
const mbean = "puppetlabs.puppetdb.population:name=num-resources"
|
||||
sumPath := metricsPrefix + "v2/read/" + mbean + "/Value"
|
||||
maxPath := metricsPrefix + "v2/read/" + mbean + "/Max"
|
||||
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[sumPath] = jolokiaRead(mbean, `1000`, 1)
|
||||
a.bodies[maxPath] = jolokiaRead(mbean, `1000`, 1)
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[sumPath] = jolokiaRead(mbean, `234`, 1)
|
||||
b.bodies[maxPath] = jolokiaRead(mbean, `234`, 1)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
for _, c := range []struct {
|
||||
path string
|
||||
want float64
|
||||
}{{sumPath, 1234}, {maxPath, 1000}} {
|
||||
var env map[string]any
|
||||
if err := json.Unmarshal(metaGet(t, srv.Handler(), c.path).Body.Bytes(), &env); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if env["value"] != c.want {
|
||||
t.Errorf("%s value = %v, want %v", c.path, env["value"], c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_ListUnionsDomains(t *testing.T) {
|
||||
// Puppetboard's /metrics page calls metric() with no name, which is a
|
||||
// Jolokia list; a backend-local MBean must not vanish from the browse tree.
|
||||
const path = metricsPrefix + "v2/list"
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[path] = `{"value":{"java.lang":{"type=Memory":{"attr":{"HeapMemoryUsage":{"rw":false}}}}},"status":200,"timestamp":1}`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[path] = `{"value":{"puppetlabs.puppetdb.population":{"name=num-nodes":{"attr":{"Value":{"rw":false}}}}},"status":200,"timestamp":1}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
got := metricValue(t, metaGet(t, srv.Handler(), path).Body.Bytes())
|
||||
if _, ok := got["java.lang"]; !ok {
|
||||
t.Errorf("java.lang missing from merged list: %v", got)
|
||||
}
|
||||
if _, ok := got["puppetlabs.puppetdb.population"]; !ok {
|
||||
t.Errorf("puppetlabs.puppetdb.population missing from merged list: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_V1BareObjectMerged(t *testing.T) {
|
||||
// metrics/v1/mbeans has no Jolokia envelope; the whole body is the value.
|
||||
const path = metricsPrefix + "v1/mbeans/" + numNodesMBean
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[path] = `{"Value":90}`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[path] = `{"Value":53}`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
var got map[string]any
|
||||
if err := json.Unmarshal(metaGet(t, srv.Handler(), path).Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got["Value"] != float64(143) {
|
||||
t.Errorf("Value = %v, want 143", got["Value"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_MissingMBeanKeepsUpstreamError(t *testing.T) {
|
||||
// Jolokia reports a bad MBean as a 200 with an error envelope, which
|
||||
// pypuppetdb turns into DoesNotComputeError; the client must still see it.
|
||||
const path = metricsPrefix + "v2/read/nope:name=nothing"
|
||||
errEnv := `{"request":{"mbean":"nope:name=nothing"},"error_type":"javax.management.InstanceNotFoundException","error":"nope:name=nothing is not registered","status":404}`
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[path] = errEnv
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[path] = errEnv
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := metaGet(t, srv.Handler(), path)
|
||||
var env map[string]any
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if env["error"] == nil {
|
||||
t.Errorf("expected the upstream Jolokia error to be replayed, got %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_ErroringBackendIgnoredWhenAnotherAnswers(t *testing.T) {
|
||||
const path = metricsPrefix + "v2/read/" + numNodesMBean
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[path] = `{"request":{},"error":"boom","status":500}`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[path] = jolokiaRead(numNodesMBean, `{"Value":53}`, 1)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
if got := metricValue(t, metaGet(t, srv.Handler(), path).Body.Bytes())["Value"]; got != float64(53) {
|
||||
t.Errorf("Value = %v, want 53 from the backend that answered", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_OneBackendDown(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.fail = true
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[numNodesPath] = jolokiaRead(numNodesMBean, `{"Value":53}`, 1)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := metaGet(t, srv.Handler(), numNodesEscaped)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200 serving the survivor, got %d", rec.Code)
|
||||
}
|
||||
if got := metricValue(t, rec.Body.Bytes())["Value"]; got != float64(53) {
|
||||
t.Errorf("Value = %v, want 53", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_AllBackendsDown(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.fail = true
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.fail = true
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
if rec := metaGet(t, srv.Handler(), numNodesEscaped); rec.Code != http.StatusBadGateway {
|
||||
t.Errorf("status %d, want 502", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetrics_RejectsNonGET(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
srv.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, numNodesEscaped, nil))
|
||||
if rec.Code != http.StatusMethodNotAllowed {
|
||||
t.Errorf("status %d, want 405", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeMetricValue_NonNumericKeepsFirst(t *testing.T) {
|
||||
a := map[string]any{"Name": "pdb-a", "Enabled": true}
|
||||
b := map[string]any{"Name": "pdb-b", "Enabled": false}
|
||||
got, ok := mergeMetricValue([]any{a, b}, "").(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("expected an object, got %T", got)
|
||||
}
|
||||
if got["Name"] != "pdb-a" || got["Enabled"] != true {
|
||||
t.Errorf("merged = %v, want the first backend's strings and booleans", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeRuleFor(t *testing.T) {
|
||||
cases := map[string]mergeRule{
|
||||
"Count": ruleSum,
|
||||
"Value": ruleSum,
|
||||
"MeanRate": ruleSum,
|
||||
"queue-depth": ruleSum,
|
||||
"min": ruleMin,
|
||||
"Max": ruleMax,
|
||||
"Uptime": ruleMax,
|
||||
"StartTime": ruleMax,
|
||||
"Mean": ruleMean,
|
||||
"StdDev": ruleMean,
|
||||
"99thPercentile": ruleMean,
|
||||
}
|
||||
for attr, want := range cases {
|
||||
if got := mergeRuleFor(attr); got != want {
|
||||
t.Errorf("mergeRuleFor(%q) = %v, want %v", attr, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package main
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
@@ -17,6 +18,7 @@ import (
|
||||
const (
|
||||
factsPath = "/pdb/query/v4/facts"
|
||||
nodesPath = "/pdb/query/v4/nodes"
|
||||
resourcesPath = "/pdb/query/v4/resources"
|
||||
reportsPath = "/pdb/query/v4/reports"
|
||||
eventsPath = "/pdb/query/v4/events"
|
||||
eventCountsPath = "/pdb/query/v4/event-counts"
|
||||
@@ -34,11 +36,18 @@ type backendResult struct {
|
||||
err error
|
||||
}
|
||||
|
||||
var errAllBackendsFailed = errors.New("all backends failed")
|
||||
|
||||
type Server struct {
|
||||
cfg Config
|
||||
client *http.Client
|
||||
log *log.Logger
|
||||
|
||||
// factsCache is nil when caching is disabled; cacheFor hands out a noop then.
|
||||
factsCache Cache
|
||||
flights flightGroup
|
||||
stale staleTracker
|
||||
|
||||
// freshness cache (freshness merge only).
|
||||
mu sync.Mutex
|
||||
freshData freshness
|
||||
@@ -46,17 +55,44 @@ type Server struct {
|
||||
}
|
||||
|
||||
func NewServer(cfg Config, logger *log.Logger) *Server {
|
||||
return &Server{
|
||||
cfg.clampFactsTTL()
|
||||
s := &Server{
|
||||
cfg: cfg,
|
||||
client: &http.Client{Timeout: cfg.Timeout},
|
||||
log: logger,
|
||||
}
|
||||
if cfg.cacheEnabled() {
|
||||
s.factsCache = newMemoryCache(cfg.FactsTTL, cfg.CacheBytes)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// cacheFor picks the cache backing a request. Merged /facts and /nodes record
|
||||
// sets share the in-memory cache; every other path is uncached until the reports
|
||||
// cache lands, and a new backend is a case here rather than a change to any
|
||||
// handler.
|
||||
func (s *Server) cacheFor(path string, params url.Values) (Cache, bool) {
|
||||
switch path {
|
||||
case factsPath, nodesPath:
|
||||
// An aggregate row is a summed count, not the merged record set the
|
||||
// cache was built for, so it stays on the live path.
|
||||
if parseAggregate(params.Get("query")) != nil {
|
||||
return noopCache{}, false
|
||||
}
|
||||
if s.factsCache != nil {
|
||||
return s.factsCache, true
|
||||
}
|
||||
}
|
||||
return noopCache{}, false
|
||||
}
|
||||
|
||||
func (s *Server) Handler() http.Handler {
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("/healthz", s.handleHealth)
|
||||
mux.HandleFunc("/pdb/query/v4/", s.handleQuery)
|
||||
mux.HandleFunc(metaVersionPath, s.handleMetaVersion)
|
||||
mux.HandleFunc(metaServerTimePath, s.handleMetaServerTime)
|
||||
mux.HandleFunc(metricsPrefix, s.handleMetrics)
|
||||
return mux
|
||||
}
|
||||
|
||||
@@ -67,7 +103,9 @@ func (s *Server) handleQuery(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
switch r.URL.Path {
|
||||
case nodesPath:
|
||||
s.serveMerged(w, r, nodesPath, s.mergeNodesResponse)
|
||||
s.serveNodes(w, r)
|
||||
case resourcesPath:
|
||||
s.serveResources(w, r)
|
||||
case factsPath:
|
||||
s.serveMerged(w, r, factsPath, s.mergeFactsResponse)
|
||||
case reportsPath:
|
||||
@@ -103,11 +141,14 @@ func isReportSubResource(path string) bool {
|
||||
}
|
||||
|
||||
func (s *Server) serveMerged(w http.ResponseWriter, r *http.Request, path string, merge func([]backendResult) []json.RawMessage) {
|
||||
alive, ok := s.aliveResults(w, r, path, queryParams(r.URL.Query().Get("query")))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
writeJSON(w, merge(alive))
|
||||
params := queryParams(r.URL.Query().Get("query"))
|
||||
s.serveCached(w, r, path, params, func() (cachedResponse, error) {
|
||||
alive, err := s.aliveResults(r.Context(), path, params)
|
||||
if err != nil {
|
||||
return cachedResponse{}, err
|
||||
}
|
||||
return cachedResponse{Body: encodeRecords(merge(alive)), Records: -1}, nil
|
||||
})
|
||||
}
|
||||
|
||||
// Reports and events are immutable history, so both backends' records belong in the merged view.
|
||||
@@ -119,19 +160,41 @@ func (s *Server) serveUnion(w http.ResponseWriter, r *http.Request, path string,
|
||||
return
|
||||
}
|
||||
|
||||
alive, ok := s.aliveResults(w, r, path, page.upstreamParams(in))
|
||||
if !ok {
|
||||
// Keyed on the request's own params, not the upstream ones: upstreamParams
|
||||
// folds offset into limit, so different windows would collide.
|
||||
s.serveCached(w, r, path, in, func() (cachedResponse, error) {
|
||||
alive, err := s.aliveResults(r.Context(), path, page.upstreamParams(in))
|
||||
if err != nil {
|
||||
return cachedResponse{}, err
|
||||
}
|
||||
merged := mergeUnion(alive, key)
|
||||
sortRecords(merged, page.order)
|
||||
resp := cachedResponse{Body: encodeRecords(page.apply(merged)), Records: -1}
|
||||
if page.wantTotal {
|
||||
if total := sumTotals(alive); total >= 0 {
|
||||
resp.Records = total
|
||||
}
|
||||
}
|
||||
return resp, nil
|
||||
})
|
||||
}
|
||||
|
||||
// A count row carries no certname, so the certname-keyed merge would collapse every backend's count into one backend's; aggregates take the summing path instead.
|
||||
func (s *Server) serveNodes(w http.ResponseWriter, r *http.Request) {
|
||||
if spec := parseAggregate(r.URL.Query().Get("query")); spec != nil {
|
||||
s.serveSummed(w, r, nodesPath, spec.columns)
|
||||
return
|
||||
}
|
||||
s.serveMerged(w, r, nodesPath, s.mergeNodesResponse)
|
||||
}
|
||||
|
||||
merged := mergeUnion(alive, key)
|
||||
sortRecords(merged, page.order)
|
||||
if page.wantTotal {
|
||||
if total := sumTotals(alive); total >= 0 {
|
||||
w.Header().Set(recordsHeader, strconv.Itoa(total))
|
||||
}
|
||||
// Only aggregates merge: a resource record has no cross-backend identity to dedupe on, so a plain query stays on the pass-through path.
|
||||
func (s *Server) serveResources(w http.ResponseWriter, r *http.Request) {
|
||||
if spec := parseAggregate(r.URL.Query().Get("query")); spec != nil {
|
||||
s.serveSummed(w, r, resourcesPath, spec.columns)
|
||||
return
|
||||
}
|
||||
writeJSON(w, page.apply(merged))
|
||||
s.proxyUnmerged(w, r)
|
||||
}
|
||||
|
||||
// An `extract` query with a `function` column returns synthetic aggregate rows that carry no identity, so they are summed rather than unioned.
|
||||
@@ -152,17 +215,19 @@ func (s *Server) serveSummed(w http.ResponseWriter, r *http.Request, path string
|
||||
return
|
||||
}
|
||||
|
||||
alive, ok := s.aliveResults(w, r, path, page.upstreamParams(in))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
merged := sumRows(alive, columns)
|
||||
sortRecords(merged, page.order)
|
||||
if page.wantTotal {
|
||||
w.Header().Set(recordsHeader, strconv.Itoa(len(merged)))
|
||||
}
|
||||
writeJSON(w, page.apply(merged))
|
||||
s.serveCached(w, r, path, in, func() (cachedResponse, error) {
|
||||
alive, err := s.aliveResults(r.Context(), path, page.upstreamParams(in))
|
||||
if err != nil {
|
||||
return cachedResponse{}, err
|
||||
}
|
||||
merged := sumRows(alive, columns)
|
||||
sortRecords(merged, page.order)
|
||||
resp := cachedResponse{Body: encodeRecords(page.apply(merged)), Records: -1}
|
||||
if page.wantTotal {
|
||||
resp.Records = len(merged)
|
||||
}
|
||||
return resp, nil
|
||||
})
|
||||
}
|
||||
|
||||
// A backend without the report answers 404, indistinguishable from a failure, so every backend is consulted before serving empty.
|
||||
@@ -190,9 +255,9 @@ func (s *Server) serveFirstHolder(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, nil)
|
||||
}
|
||||
|
||||
// Writes a 502 and returns ok=false only when every backend failed.
|
||||
func (s *Server) aliveResults(w http.ResponseWriter, r *http.Request, path string, params url.Values) ([]backendResult, bool) {
|
||||
results := s.fanOut(r.Context(), path, params)
|
||||
// Returns errAllBackendsFailed only when every backend failed.
|
||||
func (s *Server) aliveResults(ctx context.Context, path string, params url.Values) ([]backendResult, error) {
|
||||
results := s.fanOut(ctx, path, params)
|
||||
|
||||
var alive []backendResult
|
||||
for _, res := range results {
|
||||
@@ -203,10 +268,114 @@ func (s *Server) aliveResults(w http.ResponseWriter, r *http.Request, path strin
|
||||
alive = append(alive, res)
|
||||
}
|
||||
if len(alive) == 0 {
|
||||
http.Error(w, "all backends failed", http.StatusBadGateway)
|
||||
return nil, false
|
||||
return nil, errAllBackendsFailed
|
||||
}
|
||||
return alive, true
|
||||
return alive, nil
|
||||
}
|
||||
|
||||
// cachedResponse is the stored form of a merged response: the JSON body plus the
|
||||
// X-Records value it carried, so a cache hit reproduces both.
|
||||
type cachedResponse struct {
|
||||
Body json.RawMessage `json:"body"`
|
||||
Records int `json:"records"` // -1 when the response sets no X-Records
|
||||
}
|
||||
|
||||
// serveCached answers from the cache when the entry is fresh, otherwise runs
|
||||
// build — single-flighted, so N concurrent identical requests cause one upstream
|
||||
// fan-out — and stores the result. A build failure falls back to a stale entry
|
||||
// when one exists; that is the only path on which stale data is served. Paths
|
||||
// with no cache configured run build directly, unchanged.
|
||||
func (s *Server) serveCached(w http.ResponseWriter, r *http.Request, path string, params url.Values, build func() (cachedResponse, error)) {
|
||||
cache, enabled := s.cacheFor(path, params)
|
||||
if !enabled {
|
||||
resp, err := build()
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
writeCached(w, resp)
|
||||
return
|
||||
}
|
||||
|
||||
key := cacheKey(path, params)
|
||||
var stale *CacheEntry
|
||||
ent, status, err := cache.Get(r.Context(), key)
|
||||
switch {
|
||||
case err != nil:
|
||||
s.log.Printf("warning: cache lookup for %s failed: %v", key, err)
|
||||
case status == CacheFresh:
|
||||
s.stale.markFresh()
|
||||
s.writeStored(w, ent.Body)
|
||||
return
|
||||
case status == CacheStale:
|
||||
stale = &ent
|
||||
}
|
||||
|
||||
resp, err, _ := s.flights.Do(key, func() (cachedResponse, error) {
|
||||
built, buildErr := build()
|
||||
if buildErr != nil {
|
||||
return cachedResponse{}, buildErr
|
||||
}
|
||||
body, marshalErr := json.Marshal(built)
|
||||
if marshalErr != nil {
|
||||
s.log.Printf("warning: encoding cache entry for %s failed: %v", key, marshalErr)
|
||||
return built, nil
|
||||
}
|
||||
// The leader's request may be cancelled while followers still wait.
|
||||
if putErr := cache.Put(context.WithoutCancel(r.Context()), key, body); putErr != nil {
|
||||
s.log.Printf("warning: cache store for %s failed: %v", key, putErr)
|
||||
}
|
||||
return built, nil
|
||||
})
|
||||
if err != nil {
|
||||
if stale != nil {
|
||||
s.stale.markStale(time.Now())
|
||||
s.log.Printf("warning: serving stale %s from cache (stored %s): %v",
|
||||
path, stale.StoredAt.UTC().Format(time.RFC3339), err)
|
||||
s.writeStored(w, stale.Body)
|
||||
return
|
||||
}
|
||||
http.Error(w, err.Error(), http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
s.stale.markFresh()
|
||||
writeCached(w, resp)
|
||||
}
|
||||
|
||||
func (s *Server) writeStored(w http.ResponseWriter, body []byte) {
|
||||
var resp cachedResponse
|
||||
if err := json.Unmarshal(body, &resp); err != nil {
|
||||
s.log.Printf("warning: unreadable cache entry: %v", err)
|
||||
http.Error(w, "unreadable cache entry", http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
writeCached(w, resp)
|
||||
}
|
||||
|
||||
func writeCached(w http.ResponseWriter, resp cachedResponse) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if resp.Records >= 0 {
|
||||
w.Header().Set(recordsHeader, strconv.Itoa(resp.Records))
|
||||
}
|
||||
// resp.Body is shared with the cache and with every caller of a single
|
||||
// flight, so it is written, never appended to.
|
||||
body := []byte(resp.Body)
|
||||
if len(body) == 0 {
|
||||
body = []byte("[]")
|
||||
}
|
||||
_, _ = w.Write(body)
|
||||
_, _ = w.Write([]byte("\n"))
|
||||
}
|
||||
|
||||
func encodeRecords(recs []json.RawMessage) json.RawMessage {
|
||||
if recs == nil {
|
||||
recs = []json.RawMessage{}
|
||||
}
|
||||
b, err := json.Marshal(recs)
|
||||
if err != nil {
|
||||
return json.RawMessage("[]")
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func queryParams(query string) url.Values {
|
||||
@@ -372,13 +541,48 @@ func setContentType(w http.ResponseWriter, contentType string) {
|
||||
type healthReport struct {
|
||||
Status string `json:"status"`
|
||||
Backends map[string]string `json:"backends"` // name -> "ok" | error text
|
||||
Cache cacheHealth `json:"cache"`
|
||||
}
|
||||
|
||||
type cacheHealth struct {
|
||||
Backend string `json:"backend"` // "memory" | "none"
|
||||
TTL string `json:"ttl"`
|
||||
Entries int `json:"entries"`
|
||||
StaleEntries int `json:"stale_entries"` // cached entries past their TTL
|
||||
Bytes int64 `json:"bytes"`
|
||||
ServingStale bool `json:"serving_stale"` // last cached response came from a stale entry
|
||||
StaleServed uint64 `json:"stale_served"`
|
||||
LastStale string `json:"last_stale_served,omitempty"`
|
||||
}
|
||||
|
||||
func (s *Server) cacheHealth() cacheHealth {
|
||||
stats := CacheStats{Backend: "none"}
|
||||
ttl := time.Duration(0)
|
||||
if s.factsCache != nil {
|
||||
stats = s.factsCache.Stats()
|
||||
ttl = s.cfg.FactsTTL
|
||||
}
|
||||
serving, served, last := s.stale.snapshot()
|
||||
h := cacheHealth{
|
||||
Backend: stats.Backend,
|
||||
TTL: durationString(ttl),
|
||||
Entries: stats.Entries,
|
||||
StaleEntries: stats.StaleEntries,
|
||||
Bytes: stats.Bytes,
|
||||
ServingStale: serving,
|
||||
StaleServed: served,
|
||||
}
|
||||
if !last.IsZero() {
|
||||
h.LastStale = last.UTC().Format(time.RFC3339)
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
|
||||
probe := `["=","certname","pdbmux-healthz-probe"]`
|
||||
results := s.fanOut(r.Context(), nodesPath, queryParams(probe))
|
||||
|
||||
report := healthReport{Backends: map[string]string{}}
|
||||
report := healthReport{Backends: map[string]string{}, Cache: s.cacheHealth()}
|
||||
healthy := 0
|
||||
for _, res := range results {
|
||||
if res.err != nil {
|
||||
|
||||
+129
@@ -34,6 +34,9 @@ type fakeBackend struct {
|
||||
|
||||
mu sync.Mutex
|
||||
gotParams map[string]url.Values
|
||||
// rawPaths records the still-escaped request paths, so tests can assert an
|
||||
// MBean name's percent-encoding survived the proxy.
|
||||
rawPaths []string
|
||||
}
|
||||
|
||||
func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend {
|
||||
@@ -51,6 +54,7 @@ func newFakeBackend(t *testing.T, nodesBody, factsBody string) *fakeBackend {
|
||||
}
|
||||
fb.mu.Lock()
|
||||
fb.gotParams[r.URL.Path] = r.URL.Query()
|
||||
fb.rawPaths = append(fb.rawPaths, r.URL.EscapedPath())
|
||||
fb.mu.Unlock()
|
||||
if fb.fail {
|
||||
http.Error(w, "boom", http.StatusInternalServerError)
|
||||
@@ -91,6 +95,14 @@ func (fb *fakeBackend) params(path string) (url.Values, bool) {
|
||||
return v, ok
|
||||
}
|
||||
|
||||
// sawRawPath reports whether the backend was asked for a path with exactly that
|
||||
// escaping.
|
||||
func (fb *fakeBackend) sawRawPath(p string) bool {
|
||||
fb.mu.Lock()
|
||||
defer fb.mu.Unlock()
|
||||
return slices.Contains(fb.rawPaths, p)
|
||||
}
|
||||
|
||||
// gotQuery returns the PuppetDB query param the backend saw for a path.
|
||||
func (fb *fakeBackend) gotQuery(path string) string {
|
||||
v, _ := fb.params(path)
|
||||
@@ -858,3 +870,120 @@ func TestHandler_EventCountsBadPagingParam(t *testing.T) {
|
||||
t.Errorf("expected 400 for a malformed limit, got %d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// What Puppetboard's landing page sends when DEFAULT_ENVIRONMENT names a real
|
||||
// environment: an extract/count with no group_by, so every backend returns one
|
||||
// anonymous row.
|
||||
const nodeCountQuery = `["extract",[["function","count"]],["and",["=","catalog_environment","production"]]]`
|
||||
|
||||
func TestHandler_NodesAggregateSummed(t *testing.T) {
|
||||
// A count row has no certname, so the certname-keyed merge would have
|
||||
// collapsed both backends' counts into one backend's number.
|
||||
a := newFakeBackend(t, `[{"count":90}]`, `[]`)
|
||||
b := newFakeBackend(t, `[{"count":53}]`, `[]`)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := doGet(t, srv.Handler(), nodesPath, nodeCountQuery)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := counts(t, rec.Body.Bytes(), "count"); !slices.Equal(got, []float64{143}) {
|
||||
t.Errorf("count = %v, want [143]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_NodesAggregateGroupedSummed(t *testing.T) {
|
||||
a := newFakeBackend(t, `[{"count":4,"catalog_environment":"production"},{"count":1,"catalog_environment":"dev"}]`, `[]`)
|
||||
b := newFakeBackend(t, `[{"count":3,"catalog_environment":"production"}]`, `[]`)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := doGet(t, srv.Handler(), nodesPath,
|
||||
`["extract",[["function","count"],"catalog_environment"],["~","certname",".*"],["group_by","catalog_environment"]]`)
|
||||
var got []map[string]any
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
byEnv := map[string]float64{}
|
||||
for _, row := range got {
|
||||
e, _ := row["catalog_environment"].(string)
|
||||
n, _ := row["count"].(float64)
|
||||
byEnv[e] = n
|
||||
}
|
||||
want := map[string]float64{"production": 7, "dev": 1}
|
||||
if !reflect.DeepEqual(byEnv, want) {
|
||||
t.Errorf("counts = %v, want %v", byEnv, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_NodesNonAggregateStillMergedByCertname(t *testing.T) {
|
||||
// Regression: routing aggregates to the summing path must not divert plain
|
||||
// queries, including an extract projection that carries no function column.
|
||||
a := newFakeBackend(t,
|
||||
`[`+node("h1", "2026-07-01T00:00:00Z")+`,`+node("h2", "2026-07-10T00:00:00Z")+`]`, `[]`)
|
||||
b := newFakeBackend(t, `[`+node("h1", "2026-07-20T00:00:00Z")+`]`, `[]`)
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
for _, q := range []string{
|
||||
`["=","certname","h1"]`,
|
||||
`["extract",["certname","report_timestamp"],["~","certname",".*"]]`,
|
||||
} {
|
||||
rec := doGet(t, srv.Handler(), nodesPath, q)
|
||||
var got []recordMeta
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
|
||||
t.Fatalf("query %s: %v", q, err)
|
||||
}
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("query %s: expected 2 deduped nodes, got %d: %s", q, len(got), rec.Body.String())
|
||||
}
|
||||
for _, m := range got {
|
||||
if m.Certname == "h1" && m.ReportTimestamp != "2026-07-20T00:00:00Z" {
|
||||
t.Errorf("query %s: h1 should be the newer record, got %s", q, m.ReportTimestamp)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_ResourcesAggregateSummed(t *testing.T) {
|
||||
// /resources is otherwise an unmerged pass-through, so before this the
|
||||
// landing page's resource total was whichever backend answered first.
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[resourcesPath] = `[{"count":1000}]`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[resourcesPath] = `[{"count":234}]`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := doGet(t, srv.Handler(), resourcesPath,
|
||||
`["extract",[["function","count"]],["=","environment","production"]]`)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := counts(t, rec.Body.Bytes(), "count"); !slices.Equal(got, []float64{1234}) {
|
||||
t.Errorf("count = %v, want [1234]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_ResourcesNonAggregateStillPassesThrough(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[resourcesPath] = `[{"certname":"h1","type":"File"}]`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[resourcesPath] = `[{"certname":"h2","type":"File"}]`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
rec := doGet(t, srv.Handler(), resourcesPath, `["=","type","File"]`)
|
||||
if got := rec.Body.String(); !strings.Contains(got, `"h1"`) || strings.Contains(got, `"h2"`) {
|
||||
t.Errorf("body = %s, want the first backend's response verbatim", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandler_ResourcesAggregateAsksEveryBackend(t *testing.T) {
|
||||
a := newFakeBackend(t, `[]`, `[]`)
|
||||
a.bodies[resourcesPath] = `[{"count":1}]`
|
||||
b := newFakeBackend(t, `[]`, `[]`)
|
||||
b.bodies[resourcesPath] = `[{"count":1}]`
|
||||
srv := newTestServer(testConfig(a.srv.URL, b.srv.URL, mergeStatic))
|
||||
|
||||
doGet(t, srv.Handler(), resourcesPath, `["extract",[["function","count"]],["=","environment","production"]]`)
|
||||
if _, ok := b.params(resourcesPath); !ok {
|
||||
t.Error("second backend was never asked for the resource count")
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user