Gate periodic library-mutating tasks behind a scan-leader lease
ABI Compatibility / ABI - HEAD (pull_request) Has been cancelled
ABI Compatibility / ABI - BASE (pull_request) Has been cancelled
OpenAPI / OpenAPI - HEAD (pull_request) Has been cancelled
OpenAPI / OpenAPI - BASE (pull_request) Has been cancelled
Tests / run-phase5-tests (pull_request) Has been cancelled
Tests / run-tests (pull_request) Has been cancelled
Project Automation / Project board (pull_request) Has been cancelled
Merge Conflict Labeler / Labeling (pull_request) Has been cancelled
ABI Compatibility / ABI - Difference (pull_request) Has been cancelled
OpenAPI / OpenAPI - Difference (pull_request) Has been cancelled
OpenAPI / OpenAPI - Publish Unstable Spec (pull_request) Has been cancelled
OpenAPI / OpenAPI - Publish Stable Spec (pull_request) Has been cancelled

In a multi-pod deployment every pod runs the scheduled-task timers, so
periodic library-mutating tasks (library refresh, people/chapter refresh,
audio normalization, media-segment and keyframe extraction, collection and
user-data cleanup, database optimization) fire concurrently against the shared
database and library, duplicating work and racing each other.

Add an IScanLeaderLease abstraction that elects a single scan leader via a
Redis TTL lease keyed on the pod identity, mirroring the existing transcode
lease machinery. RedisScanLeaderLease acquires or renews the lease with an
atomic Lua script and fails safe by treating the pod as leader whenever Redis
is unreachable, so scans never stall. NullScanLeaderLease preserves the
single-instance behavior when election is disabled or no Redis connection is
configured.

Gate only the timer-driven path in ScheduledTaskWorker: when election is
enabled and a task key is in the gated set, a non-leader re-arms its trigger
and skips enqueueing. Manual and API-triggered runs bypass this path and still
run on any pod. Wiring is additive and the new worker constructor parameters
are optional, so existing behavior is unchanged when election is off.

Signed-off-by: Ben Vincent <ben@unkin.net>
This commit is contained in:
2026-08-10 23:51:18 +10:00
parent d4f9c12c22
commit 0008bde28e
9 changed files with 656 additions and 3 deletions
@@ -0,0 +1,21 @@
using System.Threading;
using System.Threading.Tasks;
namespace MediaBrowser.Controller.ScheduledTasks;
/// <summary>
/// Provides a distributed leader lease that gates periodic, library-mutating scheduled tasks
/// to a single instance across a multi-pod deployment.
/// </summary>
public interface IScanLeaderLease
{
/// <summary>
/// Attempts to acquire the scan-leader lease, or renews it when this instance already holds it.
/// </summary>
/// <param name="cancellationToken">A cancellation token.</param>
/// <returns>
/// <c>true</c> if this instance holds the leader lease and gated periodic tasks may run here;
/// otherwise <c>false</c>.
/// </returns>
Task<bool> TryAcquireOrRenewAsync(CancellationToken cancellationToken = default);
}
@@ -0,0 +1,16 @@
using System.Threading;
using System.Threading.Tasks;
namespace MediaBrowser.Controller.ScheduledTasks;
/// <summary>
/// A no-op <see cref="IScanLeaderLease"/> used when scan-leader election is disabled or no Redis
/// connection is configured. Every instance is treated as the leader, preserving the default
/// single-instance behavior where all periodic tasks run locally.
/// </summary>
public sealed class NullScanLeaderLease : IScanLeaderLease
{
/// <inheritdoc />
public Task<bool> TryAcquireOrRenewAsync(CancellationToken cancellationToken = default)
=> Task.FromResult(true);
}
@@ -0,0 +1,38 @@
namespace MediaBrowser.Controller.ScheduledTasks;
/// <summary>
/// Configuration options for scan-leader election, which gates periodic library-mutating
/// scheduled tasks to a single leader instance in a multi-pod deployment.
/// </summary>
public sealed class ScanLeaderOptions
{
/// <summary>
/// Gets or sets a value indicating whether scan-leader election is enabled. When disabled,
/// every instance runs its periodic tasks as before.
/// </summary>
public bool Enabled { get; set; }
/// <summary>
/// Gets or sets the duration in seconds for which the scan-leader lease is held before it must
/// be renewed. A leader that stops renewing loses the lease after this duration.
/// </summary>
public int LeaseDurationSeconds { get; set; } = 60;
/// <summary>
/// Gets or sets the set of scheduled task keys whose periodic (timer-driven) execution is gated
/// to the scan leader. Tasks not listed here run on every instance, and manual or API-triggered
/// runs are never gated.
/// </summary>
public string[] GatedTaskKeys { get; set; } =
{
"RefreshLibrary",
"RefreshPeople",
"RefreshChapterImages",
"AudioNormalization",
"TaskExtractMediaSegments",
"KeyframeExtraction",
"CleanCollectionsAndPlaylists",
"CleanupUserDataTask",
"OptimizeDatabaseTask"
};
}