unkin/jellyfin-ha-src
4
0
Fork 0
Code Issues 12 Pull Requests Packages Projects Releases Wiki Activity

SSO login fails with "Invalid or expired state" across replicas: OAuth state held in-process by the SSO plugin #27

New Issue
Closed
opened 2026-09-19 16:37:17 +10:00 by unkin-agent · 1 comment
unkin-agent commented 2026-09-19 16:37:17 +10:00
Member
Copy Link
Copy Source

Problem

SSO login fails intermittently with "Invalid or expired state" on the 2-replica cheeztv and fafflix deployments. LDAP login is unaffected; this is browser SSO only.

The SSO-Auth plugin (9p4/jellyfin-plugin-sso, 4.0.0.4, baked into the image) generates the OAuth state and holds it in-process — it exposes a /SSO/OID/States endpoint consistent with an in-memory dictionary, with no shared backing store. /sso/OID/start lands on whichever pod the load balancer picks; Authentik redirects the browser back to the callback URL, which can land on the other pod. That pod has never seen the state and rejects the login.

Reproduced in production on fafflix, both replicas serving legs of the same login:

05:31:29  fafflix-1  SSOController initialized
05:31:30  fafflix-0  SSOController initialized
05:31:39  fafflix-1
05:31:40  fafflix-0

sessionAffinity: ClientIP is set on the Services but does not help and should not be read as evidence that stickiness is handled here: traffic arrives via the gateway, so every request carries the gateway's source IP rather than the end user's. ClientIP affinity can't distinguish users behind the same gateway, and with multiple gateway pods it doesn't even pin consistently pod-to-pod.

Same class of bug as the in-process single-instance assumptions tracked in #12, but the offending state lives in a third-party plugin rather than in Jellyfin itself, so it doesn't fit as a child of that tracking issue.

Proposal

Fork 9p4/jellyfin-plugin-sso and back the OAuth state with valkey — the same store the fork already uses for the transcode session store and the scan-leader lease (Jellyfin__TranscodeStore__RedisConnectionString). Any pod can then validate a state generated by any other pod.

The fork also has to close a second, unrelated gap: SSO-Auth 4.0.0.4 targets the 10.11 plugin ABI and has no build targeted at 12.0. It currently loads on 12.0 only because Jellyfin's version gate checks 10.11.0.0 < 12.0.0.0. It loads, registers its 27 routes, and serves its config page — but the Jellyfin-facing calls (user provisioning, AuthenticateWithProvider, session-token minting) have never actually been exercised on 12.0, because every login so far has failed earlier in the flow on the state check. Upstream has been unchanged since 2026-05-12, with no PR or issue open for Jellyfin 12 support. The fork needs its own 12.0-targeted build, not just a version-gate pass.

Rejected alternatives:

  • Scale to 1 replica — loses HA, defeats the point of the deployment.
  • Cookie/session affinity at the gateway — standing direction is to avoid relying on sticky sessions and keep the deployment stateless as much as possible; this would also just paper over the same underlying assumption.

Downstream work once the fork exists:

  • New repository, with its own build/release pipeline.
  • unkin/jellyfin-ha's Dockerfile.runtime currently fetches upstream 9p4/jellyfin-plugin-sso, sha256-pinned, via the artifactapi github-proxy remote, and docker-entrypoint.sh syncs it into the shared RWX /config/plugins. Both need repinning to the fork's release artifact.

Acceptance

  • SSO login succeeds repeatedly against a 2-replica deployment with the /sso/OID/start and callback legs deliberately forced onto different pods.
  • No reliance on session/cookie affinity for correctness.
  • Plugin loads and completes initialisation against Jellyfin 12.0 under its own ABI target, not by slipping past the version gate.
  • A real end-to-end OIDC login exercises the previously-untested Jellyfin-facing calls: user provisioning, AuthenticateWithProvider, and session-token minting.

Same class as #12; tracked separately because the fix lives in a forked third-party plugin, not in Jellyfin.

## Problem SSO login fails intermittently with "Invalid or expired state" on the 2-replica `cheeztv` and `fafflix` deployments. LDAP login is unaffected; this is browser SSO only. The `SSO-Auth` plugin (`9p4/jellyfin-plugin-sso`, 4.0.0.4, baked into the image) generates the OAuth `state` and holds it in-process — it exposes a `/SSO/OID/States` endpoint consistent with an in-memory dictionary, with no shared backing store. `/sso/OID/start` lands on whichever pod the load balancer picks; Authentik redirects the browser back to the callback URL, which can land on the other pod. That pod has never seen the state and rejects the login. Reproduced in production on `fafflix`, both replicas serving legs of the same login: ``` 05:31:29 fafflix-1 SSOController initialized 05:31:30 fafflix-0 SSOController initialized 05:31:39 fafflix-1 05:31:40 fafflix-0 ``` `sessionAffinity: ClientIP` is set on the Services but does not help and should not be read as evidence that stickiness is handled here: traffic arrives via the gateway, so every request carries the gateway's source IP rather than the end user's. `ClientIP` affinity can't distinguish users behind the same gateway, and with multiple gateway pods it doesn't even pin consistently pod-to-pod. Same class of bug as the in-process single-instance assumptions tracked in #12, but the offending state lives in a third-party plugin rather than in Jellyfin itself, so it doesn't fit as a child of that tracking issue. ## Proposal Fork `9p4/jellyfin-plugin-sso` and back the OAuth state with valkey — the same store the fork already uses for the transcode session store and the scan-leader lease (`Jellyfin__TranscodeStore__RedisConnectionString`). Any pod can then validate a state generated by any other pod. The fork also has to close a second, unrelated gap: SSO-Auth 4.0.0.4 targets the 10.11 plugin ABI and has no build targeted at 12.0. It currently loads on 12.0 only because Jellyfin's version gate checks `10.11.0.0 < 12.0.0.0`. It loads, registers its 27 routes, and serves its config page — but the Jellyfin-facing calls (user provisioning, `AuthenticateWithProvider`, session-token minting) have never actually been exercised on 12.0, because every login so far has failed earlier in the flow on the state check. Upstream has been unchanged since 2026-05-12, with no PR or issue open for Jellyfin 12 support. The fork needs its own 12.0-targeted build, not just a version-gate pass. Rejected alternatives: - Scale to 1 replica — loses HA, defeats the point of the deployment. - Cookie/session affinity at the gateway — standing direction is to avoid relying on sticky sessions and keep the deployment stateless as much as possible; this would also just paper over the same underlying assumption. Downstream work once the fork exists: - New repository, with its own build/release pipeline. - `unkin/jellyfin-ha`'s `Dockerfile.runtime` currently fetches upstream `9p4/jellyfin-plugin-sso`, sha256-pinned, via the artifactapi github-proxy remote, and `docker-entrypoint.sh` syncs it into the shared RWX `/config/plugins`. Both need repinning to the fork's release artifact. ## Acceptance - SSO login succeeds repeatedly against a 2-replica deployment with the `/sso/OID/start` and callback legs deliberately forced onto different pods. - No reliance on session/cookie affinity for correctness. - Plugin loads and completes initialisation against Jellyfin 12.0 under its own ABI target, not by slipping past the version gate. - A real end-to-end OIDC login exercises the previously-untested Jellyfin-facing calls: user provisioning, `AuthenticateWithProvider`, and session-token minting. Same class as #12; tracked separately because the fix lives in a forked third-party plugin, not in Jellyfin.
unkin-agent commented 2026-09-20 22:58:30 +10:00
Author
Member
Copy Link
Copy Source

Resolved. Fork unkin/jellyfin-plugin-sso 5.0.0.0, shipped in jellyfin-ha v0.3.3.

Acceptance, verified against fafflix pod logs:

  • Cross-pod login — 4 consecutive logins, every one served by both pods, all reaching Auth request created.... Strongest case: state created on fafflix-0 at 22:51:32 (its first ever SSO activity), redirect leg served by fafflix-1 at 22:51:42 after an 8-hour gap with no SSO traffic, auth leg back on fafflix-0. The state cannot have been local to fafflix-1. Attempt 3 is the mirror (1 → 0 → 1). Old code would have failed twice in that sequence.
  • No affinity — sessionAffinity: ClientIP removed from both Services; spec.sessionAffinity: None live.
  • 12.0 ABI — Loaded plugin: SSO-Auth 5.0.0.0 from SSO Authentication_5.0.0.0, targetAbi 12.0.0.0. No InvalidCastException/MissingMethodException.
  • Jellyfin-facing calls — SSO user benvin doesn't exist, creating... → SSO user link doesn't exist, creating... → Auth request created... → Creating new access token. Provisioning and token minting executed on 12.0.
  • Invalid or expired state appears nowhere in any pod log; InMemorySsoStateStore never selected.

Two corrections to this issue's premises, for the record:

  • AuthenticateWithProvider does not exist in the plugin or on ISessionManager in 12.0. The call is _sessionManager.AuthenticateDirect.
  • The state is shared by three legs, not two — POST /sso/OID/Auth/{provider} reads it as well, and the shared item includes the PKCE CodeVerifier.

Not covered: cheeztv received no SSO traffic during testing, so its store selection is untested rather than failed. Same image and config as fafflix.

Resolved. Fork `unkin/jellyfin-plugin-sso` 5.0.0.0, shipped in `jellyfin-ha` v0.3.3. Acceptance, verified against `fafflix` pod logs: - **Cross-pod login** — 4 consecutive logins, every one served by both pods, all reaching `Auth request created...`. Strongest case: state created on `fafflix-0` at 22:51:32 (its first ever SSO activity), redirect leg served by `fafflix-1` at 22:51:42 after an 8-hour gap with no SSO traffic, auth leg back on `fafflix-0`. The state cannot have been local to `fafflix-1`. Attempt 3 is the mirror (1 → 0 → 1). Old code would have failed twice in that sequence. - **No affinity** — `sessionAffinity: ClientIP` removed from both Services; `spec.sessionAffinity: None` live. - **12.0 ABI** — `Loaded plugin: SSO-Auth 5.0.0.0` from `SSO Authentication_5.0.0.0`, `targetAbi 12.0.0.0`. No `InvalidCastException`/`MissingMethodException`. - **Jellyfin-facing calls** — `SSO user benvin doesn't exist, creating...` → `SSO user link doesn't exist, creating...` → `Auth request created...` → `Creating new access token`. Provisioning and token minting executed on 12.0. - `Invalid or expired state` appears nowhere in any pod log; `InMemorySsoStateStore` never selected. Two corrections to this issue's premises, for the record: - `AuthenticateWithProvider` does not exist in the plugin or on `ISessionManager` in 12.0. The call is `_sessionManager.AuthenticateDirect`. - The state is shared by **three** legs, not two — `POST /sso/OID/Auth/{provider}` reads it as well, and the shared item includes the PKCE `CodeVerifier`. Not covered: `cheeztv` received no SSO traffic during testing, so its store selection is untested rather than failed. Same image and config as `fafflix`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/jellyfin-ha-src#27
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?