The decaying failure window let a repeating cycle of one failure and a run
of rejections readmit a dead backend once per cycle, forever: any rule that
readmits on "no real failure lately" flaps under a periodic pattern.
Replace it with a per-backend latch: has this probe endpoint ever answered
with a verdict we can read? Until it has, there is no health signal, so the
backend is never gated and stays in service as probe_unsupported. Once it
has, the path works and every unsuccessful probe counts, rejections
included. The latch never clears, so no pattern can argue a backend back in.
A backend whose health_probe_path is wrong rejects every probe, so no
success can ever arrive to clear the sticky run-failed flag. One transient
failure in the middle of those rejections excluded such a backend from the
pool for the life of the process, reintroducing the stranding bug.
Decide unhealthy vs probe_unsupported on whether a real failure landed
within the last health_probe_failures probes, floored at 3 so the window
always spans a reject/reject/failure cycle. Sustained alternation keeps a
failure in every window and stays unhealthy; an aged-out failure leaves a
pure-rejection run on probe_unsupported and back in service.
Resetting the run on every change of outcome kind meant only consecutive
same-kind probes accumulated, so a backend failing every probe but
alternating kinds — a fronting proxy flipping 404 and 503 — never reached
the failure threshold and kept taking every query.
- count one run of consecutive not-OK probes for the down decision, and use
the kinds only to pick which non-healthy state the run enters
- send a run containing any real failure to unhealthy; only a run of nothing
but refusals enters probe_unsupported and stays in service
- classify 429 as a real failure: it is the backend reporting its own
capacity, so an overloaded backend gets backed off
- document that probe_unsupported means "not verified" and that reachable is
the /healthz field carrying actual reachability
A backend that 404s on the health probe path but serves queries fine was
marked down and excluded from every fan-out for good, since the fail-open
only triggers when no backend is left healthy.
Classify a probe reply that refuses the request itself - any 4xx, plus 501 -
as evidence about the probe, not the backend. Such a backend keeps serving
queries and reports the distinct probe_unsupported state on /healthz.
Transport failures and 5xx, 503 included, still mark a backend down.
Log the misconfiguration once per transition with the backend, probe path
and status. Track failure runs per outcome kind so a 404 run and a 503 run
never add up to one threshold.
A down backend costs a full timeout stall on every request, since fan-out
has no way to know before it asks, and the client is never told the answer
came from fewer backends than are configured.
- poll each backend's status endpoint in the background, one goroutine per
backend, with failure/success thresholds so a blip cannot flap it
- skip backends the prober has down, and fall open to querying all of them
when none is left healthy
- treat a not-yet-probed backend as healthy so a restart drops no traffic
- log only up/down transitions
- stamp merged responses with X-Backends: <contributed>/<configured>
- report per-backend probe state and the last round's partiality on /healthz
- add health_probe_enabled, health_probe_path, health_probe_interval,
health_probe_timeout, health_probe_failures and health_probe_successes,
with matching PDBMUX_* env vars and a --health-probe flag
unkin-agent