Gate periodic library-mutating tasks behind a scan-leader lease #1

2026-08-10T23:52:55+10:00

unkinben commented

2026-08-10 23:52:55 +10:00

Why

In a multi-pod deployment every pod runs the scheduled-task timers, so periodic
library-mutating tasks (library refresh, people and chapter refresh, audio
normalization, media-segment and keyframe extraction, collection and user-data
cleanup, database optimization) fire concurrently against the shared database
and library. That duplicates work and races the pods against each other.

How

Add an IScanLeaderLease abstraction that elects a single scan leader through
a Redis TTL lease keyed on the pod identity, mirroring the existing transcode
lease machinery.
RedisScanLeaderLease acquires or renews the lease with an atomic Lua script
and fails safe: when Redis is unreachable it treats the pod as leader, so
scans never stall (every pod scanning is preferable to none).
NullScanLeaderLease preserves single-instance behavior when election is
disabled or no Redis connection is configured.
Gate only the timer-driven path in ScheduledTaskWorker: when election is
enabled and a task key is in the gated set, a non-leader re-arms its trigger
and skips enqueueing. Manual and API-triggered runs bypass this path and still
run on any pod.
Wire the lease and options through TaskManager into each worker. The new
worker constructor parameters are optional, so behavior is unchanged when
election is off.
Configuration binds from Jellyfin:ScanLeader and reuses the existing
Jellyfin:TranscodeStore:RedisConnectionString.

Covered by unit tests for lease acquire/renew/takeover semantics, the Redis
fail-safe path, and worker gating (a non-leader skips a gated periodic task
while manual execution and non-gated tasks still run).

## Why In a multi-pod deployment every pod runs the scheduled-task timers, so periodic library-mutating tasks (library refresh, people and chapter refresh, audio normalization, media-segment and keyframe extraction, collection and user-data cleanup, database optimization) fire concurrently against the shared database and library. That duplicates work and races the pods against each other. ## How - Add an `IScanLeaderLease` abstraction that elects a single scan leader through a Redis TTL lease keyed on the pod identity, mirroring the existing transcode lease machinery. - `RedisScanLeaderLease` acquires or renews the lease with an atomic Lua script and fails safe: when Redis is unreachable it treats the pod as leader, so scans never stall (every pod scanning is preferable to none). - `NullScanLeaderLease` preserves single-instance behavior when election is disabled or no Redis connection is configured. - Gate only the timer-driven path in `ScheduledTaskWorker`: when election is enabled and a task key is in the gated set, a non-leader re-arms its trigger and skips enqueueing. Manual and API-triggered runs bypass this path and still run on any pod. - Wire the lease and options through `TaskManager` into each worker. The new worker constructor parameters are optional, so behavior is unchanged when election is off. - Configuration binds from `Jellyfin:ScanLeader` and reuses the existing `Jellyfin:TranscodeStore:RedisConnectionString`. Covered by unit tests for lease acquire/renew/takeover semantics, the Redis fail-safe path, and worker gating (a non-leader skips a gated periodic task while manual execution and non-gated tasks still run).

unkinben added 1 commit 2026-08-10 23:52:55 +10:00

Gate periodic library-mutating tasks behind a scan-leader lease

ABI Compatibility / ABI - HEAD (pull_request) Has been cancelled

ABI Compatibility / ABI - BASE (pull_request) Has been cancelled

OpenAPI / OpenAPI - HEAD (pull_request) Has been cancelled

OpenAPI / OpenAPI - BASE (pull_request) Has been cancelled

Tests / run-phase5-tests (pull_request) Has been cancelled

Tests / run-tests (pull_request) Has been cancelled

Project Automation / Project board (pull_request) Has been cancelled

Merge Conflict Labeler / Labeling (pull_request) Has been cancelled

ABI Compatibility / ABI - Difference (pull_request) Has been cancelled

OpenAPI / OpenAPI - Difference (pull_request) Has been cancelled

OpenAPI / OpenAPI - Publish Unstable Spec (pull_request) Has been cancelled

OpenAPI / OpenAPI - Publish Stable Spec (pull_request) Has been cancelled

0008bde28e

In a multi-pod deployment every pod runs the scheduled-task timers, so
periodic library-mutating tasks (library refresh, people/chapter refresh,
audio normalization, media-segment and keyframe extraction, collection and
user-data cleanup, database optimization) fire concurrently against the shared
database and library, duplicating work and racing each other.

Add an IScanLeaderLease abstraction that elects a single scan leader via a
Redis TTL lease keyed on the pod identity, mirroring the existing transcode
lease machinery. RedisScanLeaderLease acquires or renews the lease with an
atomic Lua script and fails safe by treating the pod as leader whenever Redis
is unreachable, so scans never stall. NullScanLeaderLease preserves the
single-instance behavior when election is disabled or no Redis connection is
configured.

Gate only the timer-driven path in ScheduledTaskWorker: when election is
enabled and a task key is in the gated set, a non-leader re-arms its trigger
and skips enqueueing. Manual and API-triggered runs bypass this path and still
run on any pod. Wiring is additive and the new worker constructor parameters
are optional, so existing behavior is unchanged when election is off.

Signed-off-by: Ben Vincent <ben@unkin.net>

unkinben referenced this pull request

2026-08-11 07:24:16 +10:00

ci: add Woodpecker build+test pipeline, drop GitHub Actions #2

benvin merged commit 4920aa871a into main

2026-08-11 21:11:58 +10:00

benvin deleted branch benvin/scan-leader-election

2026-08-11 21:11:59 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/jellyfin-ha-src#1