Gate a backend only on a probe that has answered for it #15

Merged

benvin merged 5 commits from benvin/health-failure-window into main 2026-09-06 10:28:50 +10:00

Conversation 0 Commits 5 Files Changed 4

+407 -113

unkin-agent commented 2026-09-06 00:28:27 +10:00

Member

Copy Link

Copy Source

Gate a backend on its health probe only once that probe has answered for it.

• Latch per backend whether the probe endpoint has ever replied readably. Rejections and transport failures do not latch.
• A backend whose probe has never answered stays in service as probe_unsupported.
• Once latched, ordinary thresholds apply and a rejection run counts as failure.
• Remove failWindow and the window sizing. The old rule readmitted a dead backend on a repeating F,R×window cycle.

Trade-off: a probe path that worked and is later removed keeps the latch set, so that backend is excluded despite serving queries.

Gate a backend on its health probe only once that probe has answered for it. - Latch per backend whether the probe endpoint has ever replied readably. Rejections and transport failures do not latch. - A backend whose probe has never answered stays in service as `probe_unsupported`. - Once latched, ordinary thresholds apply and a rejection run counts as failure. - Remove `failWindow` and the window sizing. The old rule readmitted a dead backend on a repeating `F,R×window` cycle. Trade-off: a probe path that worked and is later removed keeps the latch set, so that backend is excluded despite serving queries.

unkin-agent added 1 commit 2026-09-06 00:28:27 +10:00

Bound a probe failure's weight to a window of probes ... 7fd5f72de1

A backend whose health_probe_path is wrong rejects every probe, so no
success can ever arrive to clear the sticky run-failed flag. One transient
failure in the middle of those rejections excluded such a backend from the
pool for the life of the process, reintroducing the stranding bug.

Decide unhealthy vs probe_unsupported on whether a real failure landed
within the last health_probe_failures probes, floored at 3 so the window
always spans a reject/reject/failure cycle. Sustained alternation keeps a
failure in every window and stays unhealthy; an aged-out failure leaves a
pure-rejection run on probe_unsupported and back in service.

unkin-agent added 1 commit 2026-09-06 00:53:38 +10:00

Gate a backend only on a probe that has answered for it ... 31283969c3

The decaying failure window let a repeating cycle of one failure and a run
of rejections readmit a dead backend once per cycle, forever: any rule that
readmits on "no real failure lately" flaps under a periodic pattern.

Replace it with a per-backend latch: has this probe endpoint ever answered
with a verdict we can read? Until it has, there is no health signal, so the
backend is never gated and stays in service as probe_unsupported. Once it
has, the path works and every unsuccessful probe counts, rejections
included. The latch never clears, so no pattern can argue a backend back in.

unkin-agent added 1 commit 2026-09-06 00:54:41 +10:00

Describe probe_unsupported by the latch in the healthz report 6ebe0b4a44

unkin-agent added 1 commit 2026-09-06 00:55:28 +10:00

Drop the recovery test the per-state table now covers e16ca9b701

unkin-agent added 1 commit 2026-09-06 00:56:03 +10:00

Split the probe-reply kinds cleanly in the README ac0a2c32ae

unkin-agent changed title from Bound a probe failure's weight to a window of probes to Gate a backend only on a probe that has answered for it 2026-09-06 00:57:15 +10:00

benvin merged commit 8f84da94ff into main 2026-09-06 10:28:50 +10:00

benvin deleted branch benvin/health-failure-window 2026-09-06 10:28:51 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

bug

Something is not working

duplicate

This issue or pull request already exists

enhancement

New feature

help wanted

Need some help

invalid

Something is wrong

question

More information is needed

wontfix

This won't be fixed

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/pdbmux#15