Resetting the run on every change of outcome kind meant only consecutive
same-kind probes accumulated, so a backend failing every probe but
alternating kinds — a fronting proxy flipping 404 and 503 — never reached
the failure threshold and kept taking every query.
- count one run of consecutive not-OK probes for the down decision, and use
the kinds only to pick which non-healthy state the run enters
- send a run containing any real failure to unhealthy; only a run of nothing
but refusals enters probe_unsupported and stays in service
- classify 429 as a real failure: it is the backend reporting its own
capacity, so an overloaded backend gets backed off
- document that probe_unsupported means "not verified" and that reachable is
the /healthz field carrying actual reachability
A backend that 404s on the health probe path but serves queries fine was
marked down and excluded from every fan-out for good, since the fail-open
only triggers when no backend is left healthy.
Classify a probe reply that refuses the request itself - any 4xx, plus 501 -
as evidence about the probe, not the backend. Such a backend keeps serving
queries and reports the distinct probe_unsupported state on /healthz.
Transport failures and 5xx, 503 included, still mark a backend down.
Log the misconfiguration once per transition with the backend, probe path
and status. Track failure runs per outcome kind so a 404 run and a 503 run
never add up to one threshold.
A down backend costs a full timeout stall on every request, since fan-out
has no way to know before it asks, and the client is never told the answer
came from fewer backends than are configured.
- poll each backend's status endpoint in the background, one goroutine per
backend, with failure/success thresholds so a blip cannot flap it
- skip backends the prober has down, and fall open to querying all of them
when none is left healthy
- treat a not-yet-probed backend as healthy so a restart drops no traffic
- log only up/down transitions
- stamp merged responses with X-Backends: <contributed>/<configured>
- report per-backend probe state and the last round's partiality on /healthz
- add health_probe_enabled, health_probe_path, health_probe_interval,
health_probe_timeout, health_probe_failures and health_probe_successes,
with matching PDBMUX_* env vars and a --health-probe flag
unkin-agent