fix(ha): stop startup when the quick connect store is unreachable #37

2026-09-26T19:09:49+10:00

unkin-agent commented

2026-09-26 19:09:49 +10:00

Makes quick connect's fail-closed guarantee real, and visible to the orchestrator.

InitializeServices builds and reads the store before anything is served, both inside the retry: abortConnect=false fails on the read, its default on the build.
Retried for 30s so a rollout survives valkey restarting alongside it; past that it logs Critical with the real elapsed and stops startup.
A failed start exits non-zero, so k8s shows Error, not Completed.
The 10-minute setup-server hold is skipped when DOTNET_RUNNING_IN_CONTAINER is set.
docs/FORK-DIFF.md records the precedence between the components sharing the connection.

Tests (RequiresDocker): down past the deadline stops startup on either connect shape; back inside it starts; the process exits 1.

Makes quick connect's fail-closed guarantee real, and visible to the orchestrator. - `InitializeServices` builds and reads the store before anything is served, both inside the retry: `abortConnect=false` fails on the read, its default on the build. - Retried for 30s so a rollout survives valkey restarting alongside it; past that it logs `Critical` with the real elapsed and stops startup. - A failed start exits non-zero, so k8s shows `Error`, not `Completed`. - The 10-minute setup-server hold is skipped when `DOTNET_RUNNING_IN_CONTAINER` is set. - `docs/FORK-DIFF.md` records the precedence between the components sharing the connection. Tests (`RequiresDocker`): down past the deadline stops startup on either connect shape; back inside it starts; the process exits 1.

unkin-agent added 1 commit 2026-09-26 19:09:50 +10:00

fail startup when the quick connect store is unreachable

ci/woodpecker/push/ci Pipeline failed

Details

ci/woodpecker/pr/ci Pipeline failed

Details

a99ca458bf

Read the configured quick connect store during InitializeServices and abort
startup, with a log line naming valkey, instead of coming up and failing on
the first request that needs the store.

unkin-agent added 1 commit 2026-09-26 19:30:45 +10:00

install libfontconfig1 for the docker test step

ci/woodpecker/push/ci Pipeline was successful

Details

ci/woodpecker/pr/ci Pipeline was successful

Details

624d528d28

The startup tests build a real app host, which probes the Skia encoder, and
loading libSkiaSharp needs fontconfig.

unkin-agent commented

2026-09-26 20:43:06 +10:00

Emby.Server.Implementations/ApplicationHost.cs:653 — one-shot read: a single momentary valkey blip at exactly this instant kills the pod, where an already-running pod shrugs the same blip off at the cost of quick connect only. During a rollout that turns a sub-second hiccup into every replica failing to come up -> retry the read a few times over a bounded window before giving up.
Jellyfin.Server/Program.cs:255-268 — the failure path this PR now routes into logs critical, sleeps 10 minutes serving 503, then returns without setting Environment.ExitCode, so the container exits 0 and k8s reports Completed rather than a crash. "The outage is visible where the server is started" does not hold for the orchestrator -> set a non-zero exit code and skip the 10-minute wait for this cause.
Emby.Server.Implementations/MediaEncoding/TranscodeStoreConnectivityProbe.cs:37,47 — "connecting must not be able to abort startup" and the swallow-and-continue branch are now unreachable: it is an IHostedService, so it first runs at Program.cs:228, after ApplicationHost.cs:653 has already aborted on the same multiplexer and the same connection string. RedisScanLeaderLease.cs:75-78 fails open on the same dependency too -> reconcile the three policies or drop the claims that are no longer true.
nit: tests/Jellyfin.Server.Tests/QuickConnect/QuickConnectStartupTests.cs:188-200 — when InitializeServices throws, the host built at :188 is never returned and never disposed, leaving a live ConnectionMultiplexer reconnecting for the rest of the run; that is the path UnreachableStore_StopsStartup takes -> dispose it on the throwing path.
nit: tests/Jellyfin.Server.Tests/QuickConnect/QuickConnectStartupTests.cs:150 — JELLYFIN_FFMPEG__NOVALIDATION is set process-wide and never cleared, so it leaks into every later test in the assembly -> clear it in DisposeAsync alongside the connection string.

- `Emby.Server.Implementations/ApplicationHost.cs:653` — one-shot read: a single momentary valkey blip at exactly this instant kills the pod, where an already-running pod shrugs the same blip off at the cost of quick connect only. During a rollout that turns a sub-second hiccup into every replica failing to come up -> retry the read a few times over a bounded window before giving up. - `Jellyfin.Server/Program.cs:255-268` — the failure path this PR now routes into logs critical, sleeps 10 minutes serving 503, then returns without setting `Environment.ExitCode`, so the container exits 0 and k8s reports `Completed` rather than a crash. "The outage is visible where the server is started" does not hold for the orchestrator -> set a non-zero exit code and skip the 10-minute wait for this cause. - `Emby.Server.Implementations/MediaEncoding/TranscodeStoreConnectivityProbe.cs:37,47` — "connecting must not be able to abort startup" and the swallow-and-continue branch are now unreachable: it is an `IHostedService`, so it first runs at `Program.cs:228`, after `ApplicationHost.cs:653` has already aborted on the same multiplexer and the same connection string. `RedisScanLeaderLease.cs:75-78` fails open on the same dependency too -> reconcile the three policies or drop the claims that are no longer true. - nit: `tests/Jellyfin.Server.Tests/QuickConnect/QuickConnectStartupTests.cs:188-200` — when `InitializeServices` throws, the host built at :188 is never returned and never disposed, leaving a live `ConnectionMultiplexer` reconnecting for the rest of the run; that is the path `UnreachableStore_StopsStartup` takes -> dispose it on the throwing path. - nit: `tests/Jellyfin.Server.Tests/QuickConnect/QuickConnectStartupTests.cs:150` — `JELLYFIN_FFMPEG__NOVALIDATION` is set process-wide and never cleared, so it leaks into every later test in the assembly -> clear it in `DisposeAsync` alongside the connection string.

unkin-agent added 1 commit 2026-09-26 22:14:22 +10:00

retry the quick connect startup probe, and exit non-zero when a start fails

ci/woodpecker/push/ci Pipeline was successful

Details

ci/woodpecker/pr/ci Pipeline was successful

Details

6c7f76fd26

unkin-agent commented

2026-09-26 22:35:23 +10:00

Emby.Server.Implementations/ApplicationHost.cs:681 — Resolve<IQuickConnectStore>() is outside the retried region and resolving it eagerly runs ConnectionMultiplexer.Connect, so a connection string that omits abortConnect=false throws here: no retry, and the Critical message that names the key and the remedy never fires. deploy/helm/jellyfin-ha/values.yaml:39-40 documents exactly that shape (redis:6379, redis:6379,password=secret), and UnreachableRedisAtStartup_FailsClosed pins the eager throw -> move the resolve inside the loop’s try so both the retry and the Critical message cover it.
nit: Emby.Server.Implementations/ApplicationHost.cs:702 — {Seconds} is the configured deadline constant, so the message always reads "after 30s" no matter how long the loop ran; the final attempt’s connect/sync timeout sits on top of it -> log Stopwatch.GetElapsedTime(startTimestamp).
nit: Jellyfin.Server/Program.cs:270 — options.StartupMode is null or Configuration.StartupMode.MediaServer && !IsRunningInContainer() reads as though && were part of the pattern -> parenthesise the is expression.
nit: Emby.Server.Implementations/ApplicationHost.cs:695 — each attempt also logs an Error from RedisQuickConnectStore.CallAsync, so a dead store emits ~30 Error plus ~29 Warning lines before the one line that matters -> drop the per-attempt warning or move it to Debug.

- `Emby.Server.Implementations/ApplicationHost.cs:681` — `Resolve<IQuickConnectStore>()` is outside the retried region and resolving it eagerly runs `ConnectionMultiplexer.Connect`, so a connection string that omits `abortConnect=false` throws here: no retry, and the Critical message that names the key and the remedy never fires. `deploy/helm/jellyfin-ha/values.yaml:39-40` documents exactly that shape (`redis:6379`, `redis:6379,password=secret`), and `UnreachableRedisAtStartup_FailsClosed` pins the eager throw -> move the resolve inside the loop’s `try` so both the retry and the Critical message cover it. - nit: `Emby.Server.Implementations/ApplicationHost.cs:702` — `{Seconds}` is the configured deadline constant, so the message always reads "after 30s" no matter how long the loop ran; the final attempt’s connect/sync timeout sits on top of it -> log `Stopwatch.GetElapsedTime(startTimestamp)`. - nit: `Jellyfin.Server/Program.cs:270` — `options.StartupMode is null or Configuration.StartupMode.MediaServer && !IsRunningInContainer()` reads as though `&&` were part of the pattern -> parenthesise the `is` expression. - nit: `Emby.Server.Implementations/ApplicationHost.cs:695` — each attempt also logs an Error from `RedisQuickConnectStore.CallAsync`, so a dead store emits ~30 Error plus ~29 Warning lines before the one line that matters -> drop the per-attempt warning or move it to Debug.

unkin-agent added 1 commit 2026-09-26 22:56:31 +10:00

retry building the quick connect store in the startup probe, not just reading it

ci/woodpecker/pr/ci Pipeline was successful

Details

ci/woodpecker/push/ci Pipeline was successful

Details

33a5fbce9b

benvin merged commit e3c1d9cb6b into main

2026-09-27 11:24:20 +10:00

benvin deleted branch benvin/quickconnect-startup-fail-closed

2026-09-27 11:24:20 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/jellyfin-ha-src#37