Gate a backend only on a probe that has answered for it #15

Merged
benvin merged 5 commits from benvin/health-failure-window into main 2026-09-06 10:28:50 +10:00

5 Commits

Author SHA1 Message Date
unkin-agent ac0a2c32ae Split the probe-reply kinds cleanly in the README
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
2026-09-06 00:56:01 +10:00
unkin-agent e16ca9b701 Drop the recovery test the per-state table now covers
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was canceled
ci/woodpecker/pr/test Pipeline was canceled
2026-09-06 00:55:25 +10:00
unkin-agent 6ebe0b4a44 Describe probe_unsupported by the latch in the healthz report
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was canceled
ci/woodpecker/pr/test Pipeline was canceled
2026-09-06 00:54:38 +10:00
unkin-agent 31283969c3 Gate a backend only on a probe that has answered for it
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was canceled
ci/woodpecker/pr/test Pipeline was canceled
The decaying failure window let a repeating cycle of one failure and a run
of rejections readmit a dead backend once per cycle, forever: any rule that
readmits on "no real failure lately" flaps under a periodic pattern.

Replace it with a per-backend latch: has this probe endpoint ever answered
with a verdict we can read? Until it has, there is no health signal, so the
backend is never gated and stays in service as probe_unsupported. Once it
has, the path works and every unsuccessful probe counts, rejections
included. The latch never clears, so no pattern can argue a backend back in.
2026-09-06 00:53:32 +10:00
unkin-agent 7fd5f72de1 Bound a probe failure's weight to a window of probes
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
A backend whose health_probe_path is wrong rejects every probe, so no
success can ever arrive to clear the sticky run-failed flag. One transient
failure in the middle of those rejections excluded such a backend from the
pool for the life of the process, reintroducing the stranding bug.

Decide unhealthy vs probe_unsupported on whether a real failure landed
within the last health_probe_failures probes, floored at 3 so the window
always spans a reject/reject/failure cycle. Sustained alternation keeps a
failure in every window and stays unhealthy; an aged-out failure leaves a
pure-rejection run on probe_unsupported and back in service.
2026-09-06 00:28:01 +10:00