Health-check backends and skip the ones that are down
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful

A down backend costs a full timeout stall on every request, since fan-out
has no way to know before it asks, and the client is never told the answer
came from fewer backends than are configured.

- poll each backend's status endpoint in the background, one goroutine per
  backend, with failure/success thresholds so a blip cannot flap it
- skip backends the prober has down, and fall open to querying all of them
  when none is left healthy
- treat a not-yet-probed backend as healthy so a restart drops no traffic
- log only up/down transitions
- stamp merged responses with X-Backends: <contributed>/<configured>
- report per-backend probe state and the last round's partiality on /healthz
- add health_probe_enabled, health_probe_path, health_probe_interval,
  health_probe_timeout, health_probe_failures and health_probe_successes,
  with matching PDBMUX_* env vars and a --health-probe flag
This commit is contained in:
2026-09-05 23:30:28 +10:00
parent 83c89ad426
commit 514377c7cb
8 changed files with 1130 additions and 32 deletions
+1 -1
View File
@@ -354,7 +354,7 @@ func TestHandler_Health(t *testing.T) {
if err := json.Unmarshal(rec.Body.Bytes(), &hr); err != nil {
t.Fatal(err)
}
if hr.Status != "ok" || hr.Backends["a"] != "ok" || hr.Backends["b"] != "ok" {
if hr.Status != "ok" || hr.Backends["a"].Reachable != "ok" || hr.Backends["b"].Reachable != "ok" {
t.Fatalf("unexpected health: %+v", hr)
}
}