Health-check backends and skip the ones that are down #14

Merged

benvin merged 3 commits from benvin/backend-health into main 2026-09-06 00:18:36 +10:00

Conversation 0 Commits 3 Files Changed 8

+1620 -32

unkin-agent commented 2026-09-05 23:30:50 +10:00

Member

Copy Link

Copy Source

A down backend costs a full timeout stall on every request, since fan-out has no way to know before it asks, and the client is never told the answer came from fewer backends than are configured.

Summary

• poll each backend's status endpoint (/status/v1/services, configurable) in the background, with failure/success thresholds so a blip cannot flap a backend in or out
• skip backends the prober has down; fall open to querying all of them when none is left healthy, and treat a not-yet-probed backend as healthy so a restart drops no traffic
• stamp merged responses with X-Backends: <contributed>/<configured> and report per-backend probe state on /healthz; log only up/down transitions
• add health_probe_enabled, health_probe_path, health_probe_interval, health_probe_timeout, health_probe_failures, health_probe_successes, matching PDBMUX_* env vars and a --health-probe flag

Partial results keep their current meaning; disabling the probe restores current behaviour exactly.

A down backend costs a full `timeout` stall on every request, since fan-out has no way to know before it asks, and the client is never told the answer came from fewer backends than are configured. ## Summary - poll each backend's status endpoint (`/status/v1/services`, configurable) in the background, with failure/success thresholds so a blip cannot flap a backend in or out - skip backends the prober has down; fall open to querying all of them when none is left healthy, and treat a not-yet-probed backend as healthy so a restart drops no traffic - stamp merged responses with `X-Backends: <contributed>/<configured>` and report per-backend probe state on `/healthz`; log only up/down transitions - add `health_probe_enabled`, `health_probe_path`, `health_probe_interval`, `health_probe_timeout`, `health_probe_failures`, `health_probe_successes`, matching `PDBMUX_*` env vars and a `--health-probe` flag Partial results keep their current meaning; disabling the probe restores current behaviour exactly.

unkin-agent added 1 commit 2026-09-05 23:30:50 +10:00

Health-check backends and skip the ones that are down ... 514377c7cb

A down backend costs a full timeout stall on every request, since fan-out
has no way to know before it asks, and the client is never told the answer
came from fewer backends than are configured.

- poll each backend's status endpoint in the background, one goroutine per
  backend, with failure/success thresholds so a blip cannot flap it
- skip backends the prober has down, and fall open to querying all of them
  when none is left healthy
- treat a not-yet-probed backend as healthy so a restart drops no traffic
- log only up/down transitions
- stamp merged responses with X-Backends: <contributed>/<configured>
- report per-backend probe state and the last round's partiality on /healthz
- add health_probe_enabled, health_probe_path, health_probe_interval,
  health_probe_timeout, health_probe_failures and health_probe_successes,
  with matching PDBMUX_* env vars and a --health-probe flag

unkin-agent added 1 commit 2026-09-05 23:55:14 +10:00

Keep a backend whose probe path is wrong in service ... 42fdc36737

A backend that 404s on the health probe path but serves queries fine was
marked down and excluded from every fan-out for good, since the fail-open
only triggers when no backend is left healthy.

Classify a probe reply that refuses the request itself - any 4xx, plus 501 -
as evidence about the probe, not the backend. Such a backend keeps serving
queries and reports the distinct probe_unsupported state on /healthz.
Transport failures and 5xx, 503 included, still mark a backend down.
Log the misconfiguration once per transition with the backend, probe path
and status. Track failure runs per outcome kind so a 404 run and a 503 run
never add up to one threshold.

unkin-agent added 1 commit 2026-09-06 00:15:02 +10:00

Count one run of failed probes, whatever kind they are ... 8f9e4125da

Resetting the run on every change of outcome kind meant only consecutive
same-kind probes accumulated, so a backend failing every probe but
alternating kinds — a fronting proxy flipping 404 and 503 — never reached
the failure threshold and kept taking every query.

- count one run of consecutive not-OK probes for the down decision, and use
  the kinds only to pick which non-healthy state the run enters
- send a run containing any real failure to unhealthy; only a run of nothing
  but refusals enters probe_unsupported and stays in service
- classify 429 as a real failure: it is the backend reporting its own
  capacity, so an overloaded backend gets backed off
- document that probe_unsupported means "not verified" and that reachable is
  the /healthz field carrying actual reachability

benvin merged commit d34782b028 into main 2026-09-06 00:18:36 +10:00

benvin deleted branch benvin/backend-health 2026-09-06 00:18:36 +10:00

unkin-agent referenced this pull request 2026-09-06 00:28:27 +10:00

Gate a backend only on a probe that has answered for it #15

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/pdbmux#14