On-demand metadata refresh can duplicate the leader's work #25

New Issue

2026-09-13T13:33:17+10:00

unkin-agent commented

2026-09-13 13:33:17 +10:00

Context: multi-replica stateless deployment (two 2-replica apps, cheeztv and fafflix, each one PostgreSQL plus valkey, no sticky sessions). Not a 12.0 regression — live in the deployed build. Tracking: #12.

Problem

MediaBrowser.Providers/Manager/ProviderManager.cs:65,67 keeps _refreshQueue and _activeRefreshes process-local, and on-demand refresh is not gated on the scan leader.

Two pods asked to refresh the same item — or one pod's on-demand refresh racing the scan leader's scheduled refresh — each make the provider calls and each write the result. Duplicated work and provider quota, plus racing writes to the same item rows.

Lowest priority in #12: the outcome is wasted work rather than wrong data, and the overlap window is short.

Proposal

Take a short-lived shared lock per item id (valkey SET NX with a TTL) around the refresh, so a second pod skips or waits. Smaller change, keeps refresh latency local.
Or route on-demand refresh requests to the scan leader. Cleaner ownership model, but adds a cross-pod hop to a user-facing action and a failure mode when no leader is elected.

Acceptance

Two independently constructed instances against real PostgreSQL: both request refresh of the same item concurrently, and the provider calls happen once.
Whichever approach is chosen, a lost lock or absent leader degrades to the current behaviour rather than dropping the refresh silently.
Tests use PostgreSqlTestServer in tests/Jellyfin.Server.Tests and run in the postgres-migration-chain Woodpecker step (JELLYFIN_TEST_POSTGRES). In-memory providers have hidden different semantics and are how three bugs reached production.

Context: multi-replica stateless deployment (two 2-replica apps, `cheeztv` and `fafflix`, each one PostgreSQL plus valkey, no sticky sessions). Not a 12.0 regression — live in the deployed build. Tracking: #12. ## Problem `MediaBrowser.Providers/Manager/ProviderManager.cs:65,67` keeps `_refreshQueue` and `_activeRefreshes` process-local, and on-demand refresh is not gated on the scan leader. Two pods asked to refresh the same item — or one pod's on-demand refresh racing the scan leader's scheduled refresh — each make the provider calls and each write the result. Duplicated work and provider quota, plus racing writes to the same item rows. Lowest priority in #12: the outcome is wasted work rather than wrong data, and the overlap window is short. ## Proposal - Take a short-lived shared lock per item id (valkey `SET NX` with a TTL) around the refresh, so a second pod skips or waits. Smaller change, keeps refresh latency local. - Or route on-demand refresh requests to the scan leader. Cleaner ownership model, but adds a cross-pod hop to a user-facing action and a failure mode when no leader is elected. ## Acceptance - Two independently constructed instances against real PostgreSQL: both request refresh of the same item concurrently, and the provider calls happen once. - Whichever approach is chosen, a lost lock or absent leader degrades to the current behaviour rather than dropping the refresh silently. - Tests use `PostgreSqlTestServer` in `tests/Jellyfin.Server.Tests` and run in the `postgres-migration-chain` Woodpecker step (`JELLYFIN_TEST_POSTGRES`). In-memory providers have hidden different semantics and are how three bugs reached production.

unkin-agent referenced this issue

2026-09-13 13:33:42 +10:00

HA: in-process single-instance assumptions under multi-replica deployment (tracking) #12

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/jellyfin-ha-src#25