Deploy jellyfin-ha as a true-HA StatefulSet under a new media project #237

Merged

benvin merged 10 commits from benvin/jellyfin-ha into main 2026-08-15 13:39:00 +10:00

Conversation 1 Commits 10 Files Changed 28

+1047

unkinben commented 2026-07-05 22:32:43 +10:00

Owner

Copy Link

Copy Source

Why

Deploy the jellyfin-ha fork as a genuine high-availability service rather than a
single replica, so its two headline capabilities can actually be exercised: the
Redis-coordinated distributed transcoding (a surviving pod resumes the HLS
segments of a pod that dies mid-stream) and the experimental PostgreSQL main
database (which is what lets more than one replica share the same library). It
lands in its own jellyfin namespace under a new media ArgoCD project.

How

Workload — StatefulSet, 2 replicas. The Deployment becomes a StatefulSet so
each pod has a stable name. The fork's Redis transcode lease keys ownership on
JELLYFIN_INSTANCE_ID, which is set from the downward-API pod name, giving each
replica a unique, stable lease identity. Soft podAntiAffinity spreads the two
pods across nodes and a PodDisruptionBudget keeps minAvailable: 1 through
drains and rollouts.

Main database — CloudNativePG. A CNPG trio in-namespace mirrors the litellm
pattern: a 3-instance Cluster, a PgBouncer Pooler, and Ceph RGW (barman) S3
backups to a dedicated cnpg-jellyfin bucket owned by a cephrgw
ObjectStoreUser. An init container writes /config/config/database.xml
selecting the fork's Jellyfin-PostgreSQL provider, and the connection string is
composed from the CNPG-generated jellyfin-postgres-app secret (username /
password / dbname) pointed at the pooler service — the password is never
rendered into the manifest. Library-item metadata therefore moves off SQLite;
metadata images, plugins, subtitles and config XML stay on /config.

Storage.

• /config is now a shared ReadWriteMany cephfs PVC (raid5, retain) so every
  replica reads/writes the same metadata and config.
• /config/transcodes — the fork's real transcode temp path — is a shared RWX
  PVC (raid5, delete). This is the load-bearing fix: takeover reads the dead
  pod's in-flight .ts/.m3u8 segments off shared storage, so per-pod scratch
  would silently break it.
• /cache is per-pod via a volumeClaimTemplate (RWO).
• The media library stays a fresh, empty RWX PVC mounted read-only; populating
  it is out of scope.

Hardware transcoding. The container requests the gpu.intel.com/i915 Intel
device-plugin resource (which pins the pod to a GPU-labelled node and injects the
DRI render node — no /dev/dri hostPath or privileged container) plus the
render/video supplemental groups. VA-API hardware acceleration is now on by
default: the inject-config init container seeds /config/config/encoding.xml
with HardwareAccelerationType vaapi, EnableHardwareEncoding, the injected
render node (/dev/dri/renderD128) and h264/hevc hardware decode, so transcodes
use the iGPU on first boot with no manual admin-UI step. Both seed files
(database.xml, encoding.xml) are written only when absent, so later admin
changes persisted to the shared RWX /config are never clobbered on restart.

Networking. The Gateway/HTTPRoute (traefik-internal,
jellyfin.k8s.syd1.au.unkin.net) is unchanged; the Service gains
sessionAffinity: ClientIP to keep a client pinned to one replica and reduce
transcode-session churn.

Redis. The in-namespace single-replica Redis stays as the transcode lease
store.

Follow-up

UDP auto-discovery is disabled, but scheduled library scans still run on every
replica (redundant scans). Single-scanner leader election is a planned follow-up
pending a fork feature, tracked separately.

## Why Deploy the jellyfin-ha fork as a genuine high-availability service rather than a single replica, so its two headline capabilities can actually be exercised: the Redis-coordinated distributed transcoding (a surviving pod resumes the HLS segments of a pod that dies mid-stream) and the experimental PostgreSQL main database (which is what lets more than one replica share the same library). It lands in its own `jellyfin` namespace under a new `media` ArgoCD project. ## How **Workload — StatefulSet, 2 replicas.** The Deployment becomes a StatefulSet so each pod has a stable name. The fork's Redis transcode lease keys ownership on `JELLYFIN_INSTANCE_ID`, which is set from the downward-API pod name, giving each replica a unique, stable lease identity. Soft `podAntiAffinity` spreads the two pods across nodes and a `PodDisruptionBudget` keeps `minAvailable: 1` through drains and rollouts. **Main database — CloudNativePG.** A CNPG trio in-namespace mirrors the litellm pattern: a 3-instance `Cluster`, a PgBouncer `Pooler`, and Ceph RGW (barman) S3 backups to a dedicated `cnpg-jellyfin` bucket owned by a `cephrgw` `ObjectStoreUser`. An init container writes `/config/config/database.xml` selecting the fork's `Jellyfin-PostgreSQL` provider, and the connection string is composed from the CNPG-generated `jellyfin-postgres-app` secret (username / password / dbname) pointed at the pooler service — the password is never rendered into the manifest. Library-item metadata therefore moves off SQLite; metadata images, plugins, subtitles and config XML stay on `/config`. **Storage.** - `/config` is now a shared `ReadWriteMany` cephfs PVC (raid5, retain) so every replica reads/writes the same metadata and config. - `/config/transcodes` — the fork's real transcode temp path — is a shared RWX PVC (raid5, delete). This is the load-bearing fix: takeover reads the dead pod's in-flight `.ts`/`.m3u8` segments off shared storage, so per-pod scratch would silently break it. - `/cache` is per-pod via a `volumeClaimTemplate` (RWO). - The media library stays a fresh, empty RWX PVC mounted read-only; populating it is out of scope. **Hardware transcoding.** The container requests the `gpu.intel.com/i915` Intel device-plugin resource (which pins the pod to a GPU-labelled node and injects the DRI render node — no `/dev/dri` hostPath or privileged container) plus the render/video supplemental groups. VA-API hardware acceleration is now on by default: the `inject-config` init container seeds `/config/config/encoding.xml` with `HardwareAccelerationType` `vaapi`, `EnableHardwareEncoding`, the injected render node (`/dev/dri/renderD128`) and h264/hevc hardware decode, so transcodes use the iGPU on first boot with no manual admin-UI step. Both seed files (`database.xml`, `encoding.xml`) are written only when absent, so later admin changes persisted to the shared RWX `/config` are never clobbered on restart. **Networking.** The Gateway/HTTPRoute (traefik-internal, `jellyfin.k8s.syd1.au.unkin.net`) is unchanged; the Service gains `sessionAffinity: ClientIP` to keep a client pinned to one replica and reduce transcode-session churn. **Redis.** The in-namespace single-replica Redis stays as the transcode lease store. ## Follow-up UDP auto-discovery is disabled, but scheduled library scans still run on every replica (redundant scans). Single-scanner leader election is a planned follow-up pending a fork feature, tracked separately.

unkinben referenced this pull request 2026-08-09 13:40:55 +10:00

Deploy jellyfin-ha to ArgoCD (jellyfin namespace, mediafs library) #348

unkinben force-pushed benvin/jellyfin-ha from c9c76f250b to ffcf646d87 2026-08-10 23:27:37 +10:00 Compare

unkinben changed title from Deploy jellyfin-ha under a new media project to Deploy jellyfin-ha as a true-HA StatefulSet under a new media project 2026-08-10 23:29:00 +10:00

unkinben added 2 commits 2026-08-10 23:37:37 +10:00

Add jellyfin (HA fork) app under a new media project ... ed75a6d1e1

Deploys the jellyfin-ha fork (git.unkin.net/unkin/jellyfin-ha) to au-syd1
via ArgoCD, under a dedicated media AppProject/ApplicationSet rather than
extending platform.

- New media AppProject + media-apps ApplicationSet (watches
  apps/overlays/*/jellyfin); registered in the argocd kustomizations
- apps/base/jellyfin: namespace, deployment (single replica to start),
  service, in-namespace Redis (transcode session store), gateway + httproute
  at jellyfin.k8s.syd1.au.unkin.net
- Storage: RWO config (cephrbd), RWX transcode scratch and RWX media
  library (cephfs) per the HA fork's pod-takeover requirement
- au-syd1 overlay

Rework jellyfin-ha into a true-HA StatefulSet deployment ... f89e1c8260

Turn the single-replica jellyfin-ha app into a proper high-availability
deployment so the fork's Redis-coordinated distributed transcoding and
PostgreSQL main database can actually be exercised.

- Replace the Deployment with a 2-replica StatefulSet for stable pod
  identity; set JELLYFIN_INSTANCE_ID from metadata.name (the fork's Redis
  transcode-lease owner id), add soft podAntiAffinity and a PDB
  minAvailable 1.
- Move the main Jellyfin DB to PostgreSQL via a CloudNativePG trio
  (3-instance Cluster, PgBouncer Pooler, Ceph RGW barman backups) mirroring
  the litellm pattern; an init container writes database.xml selecting the
  fork's Jellyfin-PostgreSQL provider and the DSN is composed from the
  CNPG-generated app secret pointed at the pooler.
- Share /config on an RWX cephfs PVC across replicas; keep /cache per-pod
  via a volumeClaimTemplate.
- Fix the transcode mount to the fork's real path /config/transcodes on the
  RWX PVC (raid5) so a surviving pod can resume the segments of the pod it
  takes over.
- Add Intel iGPU hardware transcoding via the gpu.intel.com/i915 device
  plugin resource plus render/video supplemental groups.
- Switch the Service to sessionAffinity ClientIP to reduce transcode churn.
- Disable UDP auto-discovery. Library scans still run on every replica; a
  single-scanner leader election is a planned follow-up.

unkinben force-pushed benvin/jellyfin-ha from ffcf646d87 to f89e1c8260 2026-08-10 23:37:37 +10:00 Compare

unkinben added 1 commit 2026-08-11 07:23:48 +10:00

Enable Intel iGPU hardware transcode by default for jellyfin-ha ... ceb2114467

The pod already requests gpu.intel.com/i915 and joins the render/video
groups, but Jellyfin transcodes in software until hardware acceleration
is turned on in its encoding config, which the fork does not template.

Seed /config/config/encoding.xml from the init container so VAAPI on the
injected Intel render node (/dev/dri/renderD128) is active on first boot:
HardwareAccelerationType vaapi, EnableHardwareEncoding, h264/hevc decode,
tonemapping left off. Rename the init container to inject-config and write
each seed file only when absent, so admin changes persisted to the shared
RWX /config are never clobbered on restart.

unkinben added 1 commit 2026-08-11 07:36:25 +10:00

jellyfin-ha: pull image from artifactapi docker-internal registry ... 648ff16940

The build repo's release pipeline pushes the runtime image to the
artifactapi local docker registry, not Gitea. Point the StatefulSet at
artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.1.0 so
the deploy pulls the image the build actually produces. Pull is anonymous
in-cluster (no imagePullSecret), matching every other docker-internal
workload in the estate (encapi, pdbmux, bind-operator, cephrgw, age-api).

unkin-agent added 1 commit 2026-08-15 11:55:58 +10:00

jellyfin: split media PVC into tv/movies on cephfs-raid5-delete ... 6769b5291e

Separate tv and movies onto their own volumes so sonarr/radarr can each
mount and manage their library individually later, and move media storage
off cephfs-raid6-retain onto cephfs-raid5-delete.

- Remove single jellyfin-media PVC (RWX, cephfs-raid6-retain)
- Add jellyfin-media-tv and jellyfin-media-movies PVCs (RWX, cephfs-raid5-delete, 500Gi each, expandable)
- Mount tv read-only at /media/tv and movies read-only at /media/movies in the statefulset
- Update kustomization resources to reference the two new PVCs

unkin-agent added 3 commits 2026-08-15 12:09:02 +10:00

Merge remote-tracking branch 'origin/main' into benvin/jellyfin-ha 1f63cc370d

jellyfin: add cephrgw config-backup bucket for k8up ... 6b30958a24

Give the jellyfin backup user a second bucket so k8up can store restic
backups of the jellyfin-config PVC alongside the existing CNPG barman
bucket (one user, two buckets).

- Add a jellyfin-config-backup Bucket owned by the existing
  cnpg-jellyfin-backup ObjectStoreUser
- Add a read-write BucketAccess emitting jellyfin-config-backup-s3 with
  the S3 creds k8up consumes
- Wire the new file into the jellyfin kustomization

jellyfin: back up the config PVC with a k8up Schedule ... 57ac95bdec

Protect the jellyfin-config PVC (library metadata, plugins, config XML)
with daily restic backups to the new Ceph RGW config-backup bucket.

- Add a k8up.io Schedule: daily backup (02:00), weekly prune (Sun 03:00,
  keep 14 daily/8 weekly/12 monthly) and weekly check (Sun 04:00)
- Source S3 creds from the cephrgw BucketAccess Secret and the restic
  repo password from Vault via a VaultStaticSecret (kv path
  kubernetes/namespace/jellyfin/default/k8up-restic)
- Add the namespace VaultAuth (default role/SA) VSO needs to sync it
- Mount the reflected vault-ca-cert into the restic pods so restic trusts
  the internal unkin.net CA on s3.ceph.unkin.net
- Wire the new files into the jellyfin kustomization

unkin-agent commented 2026-08-15 12:18:39 +10:00

Member

Copy Link

Copy Source

Review — kustomize build --enable-helm apps/overlays/au-syd1/jellyfin renders cleanly (5 PVCs, 2 Buckets + 1 ObjectStoreUser + 1 BucketAccess, valid k8up Schedule). vault-ca-cert is auto-reflected into every namespace (certificates/vault-ca-cert.yaml has reflection-auto-enabled: true, empty allowed-namespaces), so both the CNPG endpointCA and the Schedule mount resolve in the jellyfin ns. k8up Schedule TLS/volume wiring, S3 cred keys, bucket ref, restic secret ref and crons all check out. Media PVC split, RO /media mounts, and no dangling jellyfin-media ref confirmed.

One substantive issue:

apps/base/jellyfin/schedule.yaml — k8up will back up the whole namespace, not just jellyfin-config. A k8up Schedule/Backup has no per-PVC selector; it scans the namespace and (with the vendored chart default skipWithoutAnnotation: false, k8up-4.10.0/k8up/values.yaml:57) backs up every PVC unless it carries k8up.io/backup: "false". None of these PVCs have that annotation, so the nightly restic job will also pull jellyfin-media-tv (500Gi), jellyfin-media-movies (500Gi), jellyfin-transcode (100Gi scratch), jellyfin-redis-data, and the per-pod cache PVCs into the config-backup repo — contrary to the stated "backups of the jellyfin-config PVC" intent. The RWO ones (cache/redis) will additionally fail to mount into the backup pod when in use, causing failed jobs.

Fix: annotate jellyfin-transcode, jellyfin-media-tv, jellyfin-media-movies, jellyfin-redis-data (and the StatefulSet cache volumeClaimTemplate) with k8up.io/backup: "false" so only jellyfin-config is captured.

Merge-order note (expected, not a defect): the k8up runner image (ghcr.io/k8up-io/k8up) needs the terraform-artifactapi allowlist PR merged+applied before backup jobs can pull.

Review — `kustomize build --enable-helm apps/overlays/au-syd1/jellyfin` renders cleanly (5 PVCs, 2 Buckets + 1 ObjectStoreUser + 1 BucketAccess, valid k8up Schedule). vault-ca-cert is auto-reflected into every namespace (certificates/vault-ca-cert.yaml has `reflection-auto-enabled: true`, empty allowed-namespaces), so both the CNPG `endpointCA` and the Schedule mount resolve in the jellyfin ns. k8up Schedule TLS/volume wiring, S3 cred keys, bucket ref, restic secret ref and crons all check out. Media PVC split, RO /media mounts, and no dangling `jellyfin-media` ref confirmed. One substantive issue: **`apps/base/jellyfin/schedule.yaml` — k8up will back up the whole namespace, not just jellyfin-config.** A k8up Schedule/Backup has no per-PVC selector; it scans the namespace and (with the vendored chart default `skipWithoutAnnotation: false`, k8up-4.10.0/k8up/values.yaml:57) backs up **every** PVC unless it carries `k8up.io/backup: "false"`. None of these PVCs have that annotation, so the nightly restic job will also pull `jellyfin-media-tv` (500Gi), `jellyfin-media-movies` (500Gi), `jellyfin-transcode` (100Gi scratch), `jellyfin-redis-data`, and the per-pod `cache` PVCs into the config-backup repo — contrary to the stated "backups of the jellyfin-config PVC" intent. The RWO ones (cache/redis) will additionally fail to mount into the backup pod when in use, causing failed jobs. Fix: annotate `jellyfin-transcode`, `jellyfin-media-tv`, `jellyfin-media-movies`, `jellyfin-redis-data` (and the StatefulSet `cache` volumeClaimTemplate) with `k8up.io/backup: "false"` so only `jellyfin-config` is captured. Merge-order note (expected, not a defect): the k8up runner image (ghcr.io/k8up-io/k8up) needs the terraform-artifactapi allowlist PR merged+applied before backup jobs can pull.

unkin-agent added 1 commit 2026-08-15 12:34:27 +10:00

jellyfin: scope k8up backup to config PVC only ... 066dc41b55

The jellyfin-config Schedule had no PVC selector and the k8up chart runs
skipWithoutAnnotation=false, so restic swept in every PVC in the namespace
(media, transcode scratch, redis, per-pod cache, CNPG data) — RWO volumes
also fail to mount while in use. k8up 4.10.0's Schedule CRD has no positive
PVC selector, so exclude every non-config PVC by annotation.

- Annotate media-tv, media-movies, transcode and redis-data PVCs with
  k8up.io/backup: "false".
- Annotate the per-pod cache volumeClaimTemplate the same way.
- Propagate the annotation to the CNPG data PVCs via inheritedMetadata
  (postgres has its own barmanObjectStore backup).
- Leave jellyfin-config unannotated so it remains the only backup target.

unkin-agent added 1 commit 2026-08-15 13:03:51 +10:00

jellyfin: back media PVCs with shared static CephFS PVs f5a721cd0b

unkin-agent referenced this pull request 2026-08-15 13:13:17 +10:00

arrstack: sonarr/radarr/prowlarr/nzbget on shared CephFS media #366

benvin merged commit 6c8c0dd9e4 into main 2026-08-15 13:39:00 +10:00

benvin deleted branch benvin/jellyfin-ha 2026-08-15 13:39:00 +10:00

benvin referenced this issue from a commit 2026-08-15 16:22:07 +10:00

arrstack: sonarr/radarr/prowlarr/nzbget on shared CephFS media (#366)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/argocd-apps#237