arrstack: run sonarr/radarr/prowlarr active-active on shared Postgres (#383)

## Why

The arrstack `sonarr`/`radarr`/`prowlarr` apps currently run as single-replica upstream LinuxServer images on SQLite (RWO config PVC, `Recreate` strategy) — no HA, no horizontal scale, and a hard one-pod ceiling because SQLite can't be shared. Our `-unkin2` forks move all app state into PostgreSQL and use Postgres advisory locks for cross-replica coordination (migrations, scheduled tasks, command execution), which makes an **active-active multi-replica** deployment safe.

This is a fresh Postgres backend — **no SQLite data migration**. The current single-replica pods are throwaway baselines; the new pods start on an empty per-app database and the fork's provider migrates the schema on first start. `arrproxy` (its own `arrproxy-db`) and `nzbget` are untouched.

## What

New `apps/base/arrstack/postgres/`:
- `cnpg_cluster.yaml` — CNPG `Cluster` `arrstack-postgres` (3 instances, `cephrbd-fast-delete`), throwaway `app` initdb owner, and 3 `managed.roles` (`sonarr`/`radarr`/`prowlarr`, login non-superuser) each with `passwordSecret: <app>-db`.
- `database-{sonarr,radarr,prowlarr}.yaml` — CNPG `Database` `<app>-main` owned by `<app>`, `databaseReclaimPolicy: retain`.
- `cnpg_backup.yaml` — cephrgw `ObjectStoreUser` + `Bucket cnpg-arrstack` + nightly `ScheduledBackup` (mirrors jellyfin).
- `vaultstaticsecret.yaml` — 3 VSO `VaultStaticSecret`s syncing `<app>-db` (keys `username`/`password`) from `kv/kubernetes/namespace/arrstack/default/<app>-db` (already seeded); these are both the CNPG role `passwordSecret` and the Deployment's Postgres creds.
- sync-waves: `<app>-db` Secrets (0) → Cluster (1) → Databases (2).
- added `postgres` to `apps/base/arrstack/kustomization.yaml`.

Per app (`sonarr`/`radarr`/`prowlarr`):
- `deployment.yaml`: image → `docker-internal/<app>:v<base>-unkin2` (sonarr `v5.0.0`, radarr `v6.4.2`, prowlarr `v2.6.2`); `replicas: 3`; `RollingUpdate`. Removed the `apikey-init` initContainer and `PUID`/`PGID`/`TZ`. Runs the binary directly (`command: [/app/<Bin>]`, `args: [-nobrowser, -data=/config, -nosingleinstancecheck]`). Env from the new ConfigMap + `secretKeyRef`: `<App>__Postgres__User`/`__Password` (Secret `<app>-db`) and `<App>__Auth__ApiKey` (existing `<app>-apikey`, key `apitoken`). `/<app>/ping` probes kept. `securityContext` runAsUser/runAsGroup/fsGroup 1000 for CephFS group-write.
- new `configmap.yaml`: non-secret `<App>__…` env (Postgres Host/Port/MainDb, `Log__DbEnabled=false`, `Auth__Method=External`, `Auth__Required=DisabledForLocalAddresses`, `App__InstanceName`, `Server__Port`, `Server__UrlBase=/<app>`, `Update__Mechanism=External`).
- `pvc-config.yaml` → RWX `cephfs-raid5-retain` (shared `/config` for `config.xml`/MediaCover).
- unchanged: `service.yaml`, `gateway.yaml`, `httproute.yaml`, apikey `vaultstaticsecret.yaml` — arrproxy still addresses the Service VIP, so N replicas are transparent.

## Validation

`kustomize build apps/overlays/au-syd1/arrstack` clean; `kubeconform` **64/64 valid**; `yamllint` and the repo pre-commit (incl. no-plain-secrets) pass. Runtime validation (pods Ready, single migrate under advisory lock, scheduled-task/command single-owner, UI via `https://arrstack.unkin.net/<app>/`, 3→1→3 rescale) is post-merge once ArgoCD syncs.

## Decisions / unknowns

- **Fork image UID/fsGroup**: the `-unkin2` images have no `USER` (run as root); pinned `runAsUser`/`runAsGroup`/`fsGroup: 1000` with `fsGroupChangePolicy: OnRootMismatch`. Verify group-write on the shared CephFS `/config` on first deploy.
- **Config storage**: chose RWX `cephfs-raid5-retain` shared `/config` (per spec preference) over per-pod `emptyDir`.
- **Postgres storage/version/resources**: mirrored jellyfin/arrproxy CNPG conventions — `cephrbd-fast-delete` 10Gi, PG17, 3 instances.

Reviewed-on: #383
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>

apps/base/arrstack/kustomization.yaml

@@ -9,6 +9,7 @@ resources:
   - pv-media-movies.yaml
   - pvc-media-tv.yaml
   - pvc-media-movies.yaml
+  - postgres
   - sonarr
   - radarr
   - prowlarr

apps/base/arrstack/postgres/cnpg_backup.yaml

@@ -0,0 +1,45 @@
+---
+# Ceph RGW (S3) backup target for the shared arrstack CNPG cluster, provisioned
+# by the in-estate cephrgw-operator: one dedicated bucket + owner user. CNPG
+# reads the S3 credential Secret (cnpg-arrstack-backup-s3) from this namespace.
+apiVersion: ceph.unkin.net/v1alpha1
+kind: ObjectStoreUser
+metadata:
+  name: cnpg-arrstack-backup
+  namespace: arrstack
+spec:
+  displayName: "CNPG backup owner (arrstack)"
+  uid: cnpg-arrstack-backup
+  maxBuckets: 5
+  secretName: cnpg-arrstack-backup-s3
+  retainOnDelete: true
+---
+apiVersion: ceph.unkin.net/v1alpha1
+kind: Bucket
+metadata:
+  name: cnpg-arrstack
+  namespace: arrstack
+spec:
+  placementTarget: ec
+  bucketName: cnpg-arrstack
+  ownerRef: cnpg-arrstack-backup
+  versioning: false
+  tags:
+    app: arrstack
+    purpose: cnpg-backup
+  retainOnDelete: true
+---
+# Nightly base backup on top of always-on WAL archiving. Staggered from the
+# other CNPG clusters (6-field cron, seconds first).
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: cnpg-arrstack-nightly
+  namespace: arrstack
+spec:
+  schedule: "0 45 3 * * *"
+  immediate: false
+  backupOwnerReference: self
+  method: barmanObjectStore
+  cluster:
+    name: arrstack-postgres

apps/base/arrstack/postgres/cnpg_cluster.yaml

@@ -0,0 +1,166 @@
+---
+# Shared PostgreSQL backend for the -unkin2 fork sonarr/radarr/prowlarr, whose
+# Npgsql/EF Core provider moves each *arr off SQLite into Postgres and makes the
+# shared-nothing, active-active multi-replica deployment possible. One cluster,
+# one throwaway initdb owner ("app"), and three managed login roles — one per
+# app — each with its own per-app database (see database-*.yaml). Role passwords
+# come from the VSO-synced <app>-db Secrets (vaultstaticsecret.yaml), so no
+# credential is rendered into git.
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: arrstack-postgres
+  namespace: arrstack
+  annotations:
+    # Wave 1: the per-app <app>-db Secrets (wave 0) must exist first — CNPG reads
+    # them as the managed roles' passwordSecret. ArgoCD gates dependents on the
+    # Cluster's health status.
+    argocd.argoproj.io/sync-wave: "1"
+spec:
+  inheritedMetadata:
+    annotations:
+      k8up.io/backup: "false"
+  affinity:
+    podAntiAffinityType: preferred
+  backup:
+    retentionPolicy: 30d
+    barmanObjectStore:
+      destinationPath: s3://cnpg-arrstack
+      endpointURL: https://s3.ceph.unkin.net
+      endpointCA:
+        name: vault-ca-cert
+        key: ca.crt
+      s3Credentials:
+        accessKeyId:
+          name: cnpg-arrstack-backup-s3
+          key: AWS_ACCESS_KEY_ID
+        secretAccessKey:
+          name: cnpg-arrstack-backup-s3
+          key: AWS_SECRET_ACCESS_KEY
+      serverName: arrstack
+      data:
+        compression: bzip2
+        jobs: 2
+      wal:
+        compression: zstd
+        maxParallel: 2
+  bootstrap:
+    initdb:
+      # Throwaway owner + database: the real per-app databases are provisioned by
+      # the CNPG Database CRDs, owned by the managed roles below.
+      database: app
+      encoding: UTF8
+      localeCType: C
+      localeCollate: C
+      owner: app
+  managed:
+    roles:
+      - name: sonarr
+        ensure: present
+        comment: Sonarr application role (owns sonarr-main)
+        login: true
+        superuser: false
+        createdb: false
+        createrole: false
+        inherit: true
+        replication: false
+        connectionLimit: -1
+        passwordSecret:
+          name: sonarr-db
+      - name: radarr
+        ensure: present
+        comment: Radarr application role (owns radarr-main)
+        login: true
+        superuser: false
+        createdb: false
+        createrole: false
+        inherit: true
+        replication: false
+        connectionLimit: -1
+        passwordSecret:
+          name: radarr-db
+      - name: prowlarr
+        ensure: present
+        comment: Prowlarr application role (owns prowlarr-main)
+        login: true
+        superuser: false
+        createdb: false
+        createrole: false
+        inherit: true
+        replication: false
+        connectionLimit: -1
+        passwordSecret:
+          name: prowlarr-db
+  enablePDB: true
+  enableSuperuserAccess: false
+  failoverDelay: 0
+  imageName: ghcr.io/cloudnative-pg/postgresql:17-system-trixie
+  instances: 3
+  logLevel: info
+  maxSyncReplicas: 0
+  minSyncReplicas: 0
+  monitoring:
+    customQueriesConfigMap:
+      - key: queries
+        name: cnpg-default-monitoring
+    disableDefaultQueries: false
+    enablePodMonitor: false
+  postgresql:
+    parameters:
+      archive_mode: "on"
+      archive_timeout: 5min
+      dynamic_shared_memory_type: posix
+      effective_cache_size: 256MB
+      full_page_writes: "on"
+      log_destination: csvlog
+      log_directory: /controller/log
+      log_filename: postgres
+      log_rotation_age: "0"
+      log_rotation_size: "0"
+      log_truncate_on_rotation: "false"
+      logging_collector: "on"
+      max_connections: "200"
+      max_parallel_workers: "16"
+      max_replication_slots: "16"
+      max_worker_processes: "16"
+      shared_buffers: 128MB
+      shared_memory_type: mmap
+      ssl_max_protocol_version: TLSv1.3
+      ssl_min_protocol_version: TLSv1.3
+      wal_keep_size: 256MB
+      wal_level: logical
+      wal_log_hints: "on"
+      wal_receiver_timeout: 5s
+      wal_sender_timeout: 5s
+    syncReplicaElectionConstraint:
+      enabled: false
+  primaryUpdateMethod: restart
+  primaryUpdateStrategy: unsupervised
+  probes:
+    liveness:
+      isolationCheck:
+        connectionTimeout: 1000
+        enabled: true
+        requestTimeout: 1000
+  replicationSlots:
+    highAvailability:
+      enabled: true
+      slotPrefix: _cnpg_
+    synchronizeReplicas:
+      enabled: true
+    updateInterval: 30
+  resources:
+    limits:
+      cpu: "1"
+      memory: 1Gi
+    requests:
+      cpu: 50m
+      memory: 512Mi
+  smartShutdownTimeout: 180
+  startDelay: 3600
+  stopDelay: 1800
+  storage:
+    resizeInUseVolumes: true
+    size: 10Gi
+    storageClass: cephrbd-fast-delete
+  switchoverDelay: 3600

apps/base/arrstack/postgres/database-prowlarr.yaml

@@ -0,0 +1,17 @@
+---
+# Per-app database owned by the prowlarr managed role. The fork's provider runs
+# its own schema migrations on first start (advisory-locked, so only one replica
+# migrates). retain: the database survives a Database CRD delete.
+apiVersion: postgresql.cnpg.io/v1
+kind: Database
+metadata:
+  name: prowlarr-main
+  namespace: arrstack
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+spec:
+  cluster:
+    name: arrstack-postgres
+  name: prowlarr-main
+  owner: prowlarr
+  databaseReclaimPolicy: retain

apps/base/arrstack/postgres/database-radarr.yaml

@@ -0,0 +1,17 @@
+---
+# Per-app database owned by the radarr managed role. The fork's provider runs its
+# own schema migrations on first start (advisory-locked, so only one replica
+# migrates). retain: the database survives a Database CRD delete.
+apiVersion: postgresql.cnpg.io/v1
+kind: Database
+metadata:
+  name: radarr-main
+  namespace: arrstack
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+spec:
+  cluster:
+    name: arrstack-postgres
+  name: radarr-main
+  owner: radarr
+  databaseReclaimPolicy: retain

apps/base/arrstack/postgres/database-sonarr.yaml

@@ -0,0 +1,17 @@
+---
+# Per-app database owned by the sonarr managed role. The fork's provider runs its
+# own schema migrations on first start (advisory-locked, so only one replica
+# migrates). retain: the database survives a Database CRD delete.
+apiVersion: postgresql.cnpg.io/v1
+kind: Database
+metadata:
+  name: sonarr-main
+  namespace: arrstack
+  annotations:
+    argocd.argoproj.io/sync-wave: "2"
+spec:
+  cluster:
+    name: arrstack-postgres
+  name: sonarr-main
+  owner: sonarr
+  databaseReclaimPolicy: retain

apps/base/arrstack/postgres/kustomization.yaml

@@ -0,0 +1,11 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - vaultstaticsecret.yaml
+  - cnpg_cluster.yaml
+  - cnpg_backup.yaml
+  - database-sonarr.yaml
+  - database-radarr.yaml
+  - database-prowlarr.yaml

apps/base/arrstack/postgres/vaultstaticsecret.yaml

@@ -0,0 +1,66 @@
+---
+# Per-app Postgres role credentials. Each is seeded out-of-band at
+# kv/kubernetes/namespace/arrstack/default/<app>-db (keys: username, password);
+# the default k8s role's templated policy already grants read on
+# kv/data/kubernetes/namespace/{{sa_namespace}}/{{sa_name}}/* for the
+# arrstack/default ServiceAccount, so no terraform-vault change is needed. VSO
+# syncs each into the <app>-db Secret, which is both the CNPG managed role's
+# passwordSecret (cnpg_cluster.yaml) and the source of the app Deployment's
+# <App>__Postgres__User/__Password env. Wave 0: must exist before the Cluster
+# (wave 1) reconciles the roles.
+apiVersion: secrets.hashicorp.com/v1beta1
+kind: VaultStaticSecret
+metadata:
+  name: sonarr-db
+  namespace: arrstack
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  destination:
+    create: true
+    name: sonarr-db
+    overwrite: true
+  hmacSecretData: true
+  mount: kv
+  path: kubernetes/namespace/arrstack/default/sonarr-db
+  refreshAfter: 5m
+  type: kv-v2
+  vaultAuthRef: default
+---
+apiVersion: secrets.hashicorp.com/v1beta1
+kind: VaultStaticSecret
+metadata:
+  name: radarr-db
+  namespace: arrstack
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  destination:
+    create: true
+    name: radarr-db
+    overwrite: true
+  hmacSecretData: true
+  mount: kv
+  path: kubernetes/namespace/arrstack/default/radarr-db
+  refreshAfter: 5m
+  type: kv-v2
+  vaultAuthRef: default
+---
+apiVersion: secrets.hashicorp.com/v1beta1
+kind: VaultStaticSecret
+metadata:
+  name: prowlarr-db
+  namespace: arrstack
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  destination:
+    create: true
+    name: prowlarr-db
+    overwrite: true
+  hmacSecretData: true
+  mount: kv
+  path: kubernetes/namespace/arrstack/default/prowlarr-db
+  refreshAfter: 5m
+  type: kv-v2
+  vaultAuthRef: default

apps/base/arrstack/prowlarr/configmap.yaml

@@ -0,0 +1,24 @@
+---
+# Non-secret env for the -unkin2 fork. The fork reads Servarr config from
+# Prowlarr__<Section>__<Key> env (no config.xml edits, no s6/PUID). Postgres
+# wiring points every replica at the same shared DB (arrstack-postgres-rw /
+# prowlarr-main); Auth__Method=External defers UI auth to arrproxy/oauth2-proxy;
+# Server__UrlBase keeps the /prowlarr prefix so arrproxy path-routing works;
+# App__InstanceName is identical across replicas (shared session-cookie name).
+# User/Password/ApiKey come from Secrets (see deployment.yaml), not here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: prowlarr-env
+  namespace: arrstack
+data:
+  Prowlarr__Postgres__Host: arrstack-postgres-rw.arrstack.svc.cluster.local
+  Prowlarr__Postgres__Port: "5432"
+  Prowlarr__Postgres__MainDb: prowlarr-main
+  Prowlarr__Log__DbEnabled: "false"
+  Prowlarr__Auth__Method: External
+  Prowlarr__Auth__Required: DisabledForLocalAddresses
+  Prowlarr__App__InstanceName: Prowlarr
+  Prowlarr__Server__Port: "9696"
+  Prowlarr__Server__UrlBase: /prowlarr
+  Prowlarr__Update__Mechanism: External

apps/base/arrstack/prowlarr/deployment.yaml

@@ -5,10 +5,13 @@ metadata:
   name: prowlarr
   namespace: arrstack
 spec:
-  replicas: 1
+  # Active-active: the -unkin2 fork keeps all state in the shared Postgres
+  # (arrstack-postgres) and coordinates via Postgres advisory locks, so N
+  # replicas run concurrently behind the prowlarr Service. RollingUpdate is safe
+  # — no SQLite, no RWO lock.
+  replicas: 3
   strategy:
-    # RWO config PVC + single stateful SQLite DB: never run two pods at once.
+    type: RollingUpdate
-    type: Recreate
   selector:
     matchLabels:
       app: prowlarr
@@ -18,96 +21,49 @@ spec:
         app: prowlarr
     spec:
       securityContext:
+        # Fork image has no USER (runs as root by default); pin it to a non-root
+        # UID and group-write the shared RWX CephFS /config. OnRootMismatch
+        # avoids a recursive chown of the whole volume.
+        runAsUser: 1000
+        runAsGroup: 1000
         fsGroup: 1000
         fsGroupChangePolicy: OnRootMismatch
-      initContainers:
-        # Enforce the Vault-sourced API key and the reverse-proxy URL base in
-        # /config/config.xml before the app starts. Vault is source of truth
-        # (override bootstrap): the key is minted in Vault, synced by VSO into the
-        # prowlarr-apikey Secret, and written here. UrlBase=/prowlarr lets arrproxy
-        # forward arrstack.unkin.net/prowlarr/... with the prefix preserved (no 307).
-        # Runs as root to fix ownership; touches only <ApiKey>, <UrlBase>, and
-        # <AuthenticationMethod>=External + <AuthenticationRequired>. External makes
-        # the *arr defer UI login to arrproxy/oauth2-proxy (same no-auth request
-        # handler as None, but permits remote access without prompting).
-        - name: apikey-init
-          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/library/busybox:1.37.0
-          imagePullPolicy: IfNotPresent
-          securityContext:
-            runAsUser: 0
-          env:
-            - name: API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: prowlarr-apikey
-                  key: apitoken
-          command:
-            - sh
-            - -c
-            - |
-              set -eu
-              case "$API_KEY" in
-                "" | *[!0-9a-fA-F]*)
-                  echo "config-init: API_KEY missing or not hex; refusing" >&2
-                  exit 1
-                  ;;
-              esac
-              URL_BASE=/prowlarr
-              AUTH_METHOD=External
-              AUTH_REQUIRED=Enabled
-              CFG=/config/config.xml
-              if [ ! -f "$CFG" ]; then
-                printf '<Config>\n  <ApiKey>%s</ApiKey>\n  <UrlBase>%s</UrlBase>\n  <AuthenticationMethod>%s</AuthenticationMethod>\n  <AuthenticationRequired>%s</AuthenticationRequired>\n</Config>\n' "$API_KEY" "$URL_BASE" "$AUTH_METHOD" "$AUTH_REQUIRED" > "$CFG"
-              else
-                if grep -q '<ApiKey>' "$CFG"; then
-                  sed -i "s|<ApiKey>[^<]*</ApiKey>|<ApiKey>${API_KEY}</ApiKey>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <ApiKey>${API_KEY}</ApiKey>|" "$CFG"
-                fi
-                if grep -q '<UrlBase>' "$CFG"; then
-                  sed -i "s|<UrlBase>[^<]*</UrlBase>|<UrlBase>${URL_BASE}</UrlBase>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <UrlBase>${URL_BASE}</UrlBase>|" "$CFG"
-                fi
-                if grep -q '<AuthenticationMethod>' "$CFG"; then
-                  sed -i "s|<AuthenticationMethod>[^<]*</AuthenticationMethod>|<AuthenticationMethod>${AUTH_METHOD}</AuthenticationMethod>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <AuthenticationMethod>${AUTH_METHOD}</AuthenticationMethod>|" "$CFG"
-                fi
-                if grep -q '<AuthenticationRequired>' "$CFG"; then
-                  sed -i "s|<AuthenticationRequired>[^<]*</AuthenticationRequired>|<AuthenticationRequired>${AUTH_REQUIRED}</AuthenticationRequired>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <AuthenticationRequired>${AUTH_REQUIRED}</AuthenticationRequired>|" "$CFG"
-                fi
-              fi
-              chown 1000:1000 "$CFG"
-              chmod 600 "$CFG"
-              echo "config-init: <ApiKey>, <UrlBase>=${URL_BASE}, <AuthenticationMethod>=${AUTH_METHOD}, <AuthenticationRequired>=${AUTH_REQUIRED} enforced from Vault"
-          resources:
-            requests:
-              cpu: 50m
-              memory: 32Mi
-            limits:
-              cpu: 200m
-              memory: 64Mi
-          volumeMounts:
-            - name: config
-              mountPath: /config
       containers:
         - name: prowlarr
-          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/linuxserver/prowlarr:2.5.2
+          image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/prowlarr:v2.6.2-unkin2
           imagePullPolicy: IfNotPresent
+          command:
+            - /app/Prowlarr
+          args:
+            - -nobrowser
+            - -data=/config
+            # Required: bypass the single-instance guard so multiple replicas
+            # can share one /config. Cross-replica safety is the Postgres layer,
+            # not a local lock file.
+            - -nosingleinstancecheck
           ports:
             - name: http
               containerPort: 9696
               protocol: TCP
+          envFrom:
+            - configMapRef:
+                name: prowlarr-env
           env:
-            - name: PUID
+            - name: Prowlarr__Postgres__User
-              value: "1000"
+              valueFrom:
-            - name: PGID
+                secretKeyRef:
-              value: "1000"
+                  name: prowlarr-db
-            - name: TZ
+                  key: username
-              value: Australia/Sydney
+            - name: Prowlarr__Postgres__Password
+              valueFrom:
+                secretKeyRef:
+                  name: prowlarr-db
+                  key: password
+            - name: Prowlarr__Auth__ApiKey
+              valueFrom:
+                secretKeyRef:
+                  name: prowlarr-apikey
+                  key: apitoken
           livenessProbe:
             httpGet:
               path: /prowlarr/ping
@@ -130,7 +86,7 @@ spec:
               memory: 256Mi
             limits:
               cpu: "1"
-              memory: 512Mi
+              memory: 1Gi
           volumeMounts:
             - name: config
               mountPath: /config

apps/base/arrstack/prowlarr/kustomization.yaml

@@ -5,6 +5,7 @@ kind: Kustomization
 resources:
   - pvc-config.yaml
   - vaultstaticsecret.yaml
+  - configmap.yaml
   - deployment.yaml
   - service.yaml
   - gateway.yaml

apps/base/arrstack/prowlarr/pvc-config.yaml

@@ -1,6 +1,7 @@
 ---
-# Prowlarr config + SQLite DB. RWO on cephrbd (block) — the arr apps' SQLite
+# Prowlarr /config. RWX on CephFS so all replicas share it (the -unkin2 fork
-# does not tolerate CephFS locking. Retain: this is state.
+# keeps the database in Postgres; /config now holds only config.xml + assets,
+# which tolerate — and want — shared access). Retain: this is state.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -8,9 +9,9 @@ metadata:
   namespace: arrstack
 spec:
   accessModes:
-    - ReadWriteOnce
+    - ReadWriteMany
   resources:
     requests:
       storage: 5Gi
-  storageClassName: cephrbd-fast-retain
+  storageClassName: cephfs-raid5-retain
   volumeMode: Filesystem

apps/base/arrstack/radarr/configmap.yaml

@@ -0,0 +1,24 @@
+---
+# Non-secret env for the -unkin2 fork. The fork reads Servarr config from
+# Radarr__<Section>__<Key> env (no config.xml edits, no s6/PUID). Postgres wiring
+# points every replica at the same shared DB (arrstack-postgres-rw / radarr-main);
+# Auth__Method=External defers UI auth to arrproxy/oauth2-proxy; Server__UrlBase
+# keeps the /radarr prefix so arrproxy path-routing works; App__InstanceName is
+# identical across replicas (shared session-cookie name). User/Password/ApiKey
+# come from Secrets (see deployment.yaml), not here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: radarr-env
+  namespace: arrstack
+data:
+  Radarr__Postgres__Host: arrstack-postgres-rw.arrstack.svc.cluster.local
+  Radarr__Postgres__Port: "5432"
+  Radarr__Postgres__MainDb: radarr-main
+  Radarr__Log__DbEnabled: "false"
+  Radarr__Auth__Method: External
+  Radarr__Auth__Required: DisabledForLocalAddresses
+  Radarr__App__InstanceName: Radarr
+  Radarr__Server__Port: "7878"
+  Radarr__Server__UrlBase: /radarr
+  Radarr__Update__Mechanism: External

apps/base/arrstack/radarr/deployment.yaml

@@ -5,10 +5,13 @@ metadata:
   name: radarr
   namespace: arrstack
 spec:
-  replicas: 1
+  # Active-active: the -unkin2 fork keeps all state in the shared Postgres
+  # (arrstack-postgres) and coordinates via Postgres advisory locks, so N
+  # replicas run concurrently behind the radarr Service. RollingUpdate is safe —
+  # no SQLite, no RWO lock.
+  replicas: 3
   strategy:
-    # RWO config PVC + single stateful SQLite DB: never run two pods at once.
+    type: RollingUpdate
-    type: Recreate
   selector:
     matchLabels:
       app: radarr
@@ -18,96 +21,49 @@ spec:
         app: radarr
     spec:
       securityContext:
+        # Fork image has no USER (runs as root by default); pin it to a non-root
+        # UID and group-write the shared RWX CephFS /config (MediaCover etc.).
+        # OnRootMismatch avoids a recursive chown of the whole media tree.
+        runAsUser: 1000
+        runAsGroup: 1000
         fsGroup: 1000
         fsGroupChangePolicy: OnRootMismatch
-      initContainers:
-        # Enforce the Vault-sourced API key and the reverse-proxy URL base in
-        # /config/config.xml before the app starts. Vault is source of truth
-        # (override bootstrap): the key is minted in Vault, synced by VSO into the
-        # radarr-apikey Secret, and written here. UrlBase=/radarr lets arrproxy
-        # forward arrstack.unkin.net/radarr/... with the prefix preserved (no 307).
-        # Runs as root to fix ownership; touches only <ApiKey>, <UrlBase>, and
-        # <AuthenticationMethod>=External + <AuthenticationRequired>. External makes
-        # the *arr defer UI login to arrproxy/oauth2-proxy (same no-auth request
-        # handler as None, but permits remote access without prompting).
-        - name: apikey-init
-          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/library/busybox:1.37.0
-          imagePullPolicy: IfNotPresent
-          securityContext:
-            runAsUser: 0
-          env:
-            - name: API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: radarr-apikey
-                  key: apitoken
-          command:
-            - sh
-            - -c
-            - |
-              set -eu
-              case "$API_KEY" in
-                "" | *[!0-9a-fA-F]*)
-                  echo "config-init: API_KEY missing or not hex; refusing" >&2
-                  exit 1
-                  ;;
-              esac
-              URL_BASE=/radarr
-              AUTH_METHOD=External
-              AUTH_REQUIRED=Enabled
-              CFG=/config/config.xml
-              if [ ! -f "$CFG" ]; then
-                printf '<Config>\n  <ApiKey>%s</ApiKey>\n  <UrlBase>%s</UrlBase>\n  <AuthenticationMethod>%s</AuthenticationMethod>\n  <AuthenticationRequired>%s</AuthenticationRequired>\n</Config>\n' "$API_KEY" "$URL_BASE" "$AUTH_METHOD" "$AUTH_REQUIRED" > "$CFG"
-              else
-                if grep -q '<ApiKey>' "$CFG"; then
-                  sed -i "s|<ApiKey>[^<]*</ApiKey>|<ApiKey>${API_KEY}</ApiKey>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <ApiKey>${API_KEY}</ApiKey>|" "$CFG"
-                fi
-                if grep -q '<UrlBase>' "$CFG"; then
-                  sed -i "s|<UrlBase>[^<]*</UrlBase>|<UrlBase>${URL_BASE}</UrlBase>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <UrlBase>${URL_BASE}</UrlBase>|" "$CFG"
-                fi
-                if grep -q '<AuthenticationMethod>' "$CFG"; then
-                  sed -i "s|<AuthenticationMethod>[^<]*</AuthenticationMethod>|<AuthenticationMethod>${AUTH_METHOD}</AuthenticationMethod>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <AuthenticationMethod>${AUTH_METHOD}</AuthenticationMethod>|" "$CFG"
-                fi
-                if grep -q '<AuthenticationRequired>' "$CFG"; then
-                  sed -i "s|<AuthenticationRequired>[^<]*</AuthenticationRequired>|<AuthenticationRequired>${AUTH_REQUIRED}</AuthenticationRequired>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <AuthenticationRequired>${AUTH_REQUIRED}</AuthenticationRequired>|" "$CFG"
-                fi
-              fi
-              chown 1000:1000 "$CFG"
-              chmod 600 "$CFG"
-              echo "config-init: <ApiKey>, <UrlBase>=${URL_BASE}, <AuthenticationMethod>=${AUTH_METHOD}, <AuthenticationRequired>=${AUTH_REQUIRED} enforced from Vault"
-          resources:
-            requests:
-              cpu: 50m
-              memory: 32Mi
-            limits:
-              cpu: 200m
-              memory: 64Mi
-          volumeMounts:
-            - name: config
-              mountPath: /config
       containers:
         - name: radarr
-          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/linuxserver/radarr:6.3.0
+          image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/radarr:v6.4.2-unkin2
           imagePullPolicy: IfNotPresent
+          command:
+            - /app/Radarr
+          args:
+            - -nobrowser
+            - -data=/config
+            # Required: bypass the single-instance guard so multiple replicas
+            # can share one /config. Cross-replica safety is the Postgres layer,
+            # not a local lock file.
+            - -nosingleinstancecheck
           ports:
             - name: http
               containerPort: 7878
               protocol: TCP
+          envFrom:
+            - configMapRef:
+                name: radarr-env
           env:
-            - name: PUID
+            - name: Radarr__Postgres__User
-              value: "1000"
+              valueFrom:
-            - name: PGID
+                secretKeyRef:
-              value: "1000"
+                  name: radarr-db
-            - name: TZ
+                  key: username
-              value: Australia/Sydney
+            - name: Radarr__Postgres__Password
+              valueFrom:
+                secretKeyRef:
+                  name: radarr-db
+                  key: password
+            - name: Radarr__Auth__ApiKey
+              valueFrom:
+                secretKeyRef:
+                  name: radarr-apikey
+                  key: apitoken
           livenessProbe:
             httpGet:
               path: /radarr/ping

apps/base/arrstack/radarr/kustomization.yaml

@@ -5,6 +5,7 @@ kind: Kustomization
 resources:
   - pvc-config.yaml
   - vaultstaticsecret.yaml
+  - configmap.yaml
   - deployment.yaml
   - service.yaml
   - gateway.yaml

apps/base/arrstack/radarr/pvc-config.yaml

@@ -1,6 +1,7 @@
 ---
-# Radarr config + SQLite DB. RWO on cephrbd (block) — the arr apps' SQLite
+# Radarr /config. RWX on CephFS so all replicas share it (the -unkin2 fork keeps
-# does not tolerate CephFS locking. Retain: this is state.
+# the database in Postgres; /config now holds only config.xml + MediaCover, which
+# tolerate — and want — shared access). Retain: this is state.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -8,9 +9,9 @@ metadata:
   namespace: arrstack
 spec:
   accessModes:
-    - ReadWriteOnce
+    - ReadWriteMany
   resources:
     requests:
       storage: 5Gi
-  storageClassName: cephrbd-fast-retain
+  storageClassName: cephfs-raid5-retain
   volumeMode: Filesystem

apps/base/arrstack/sonarr/configmap.yaml

@@ -0,0 +1,24 @@
+---
+# Non-secret env for the -unkin2 fork. The fork reads Servarr config from
+# Sonarr__<Section>__<Key> env (no config.xml edits, no s6/PUID). Postgres wiring
+# points every replica at the same shared DB (arrstack-postgres-rw / sonarr-main);
+# Auth__Method=External defers UI auth to arrproxy/oauth2-proxy; Server__UrlBase
+# keeps the /sonarr prefix so arrproxy path-routing works; App__InstanceName is
+# identical across replicas (shared session-cookie name). User/Password/ApiKey
+# come from Secrets (see deployment.yaml), not here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: sonarr-env
+  namespace: arrstack
+data:
+  Sonarr__Postgres__Host: arrstack-postgres-rw.arrstack.svc.cluster.local
+  Sonarr__Postgres__Port: "5432"
+  Sonarr__Postgres__MainDb: sonarr-main
+  Sonarr__Log__DbEnabled: "false"
+  Sonarr__Auth__Method: External
+  Sonarr__Auth__Required: DisabledForLocalAddresses
+  Sonarr__App__InstanceName: Sonarr
+  Sonarr__Server__Port: "8989"
+  Sonarr__Server__UrlBase: /sonarr
+  Sonarr__Update__Mechanism: External

apps/base/arrstack/sonarr/deployment.yaml

@@ -5,10 +5,13 @@ metadata:
   name: sonarr
   namespace: arrstack
 spec:
-  replicas: 1
+  # Active-active: the -unkin2 fork keeps all state in the shared Postgres
+  # (arrstack-postgres) and coordinates via Postgres advisory locks, so N
+  # replicas run concurrently behind the sonarr Service. RollingUpdate is safe —
+  # no SQLite, no RWO lock.
+  replicas: 3
   strategy:
-    # RWO config PVC + single stateful SQLite DB: never run two pods at once.
+    type: RollingUpdate
-    type: Recreate
   selector:
     matchLabels:
       app: sonarr
@@ -18,99 +21,49 @@ spec:
         app: sonarr
     spec:
       securityContext:
-        # LinuxServer images init as root via s6 then step down to PUID/PGID.
+        # Fork image has no USER (runs as root by default); pin it to a non-root
-        # fsGroup makes the shared CephFS group-writable for that user;
+        # UID and group-write the shared RWX CephFS /config (MediaCover etc.).
         # OnRootMismatch avoids a recursive chown of the whole media tree.
+        runAsUser: 1000
+        runAsGroup: 1000
         fsGroup: 1000
         fsGroupChangePolicy: OnRootMismatch
-      initContainers:
-        # Enforce the Vault-sourced API key and the reverse-proxy URL base in
-        # /config/config.xml before the app starts. Vault is source of truth
-        # (override bootstrap): the key is minted in Vault, synced by VSO into the
-        # sonarr-apikey Secret, and written here. UrlBase=/sonarr lets arrproxy
-        # forward arrstack.unkin.net/sonarr/... with the prefix preserved (no 307).
-        # Runs as root to fix ownership; touches only <ApiKey>, <UrlBase>, and
-        # <AuthenticationMethod>=External + <AuthenticationRequired>. External makes
-        # the *arr defer UI login to arrproxy/oauth2-proxy (same no-auth request
-        # handler as None, but permits remote access without prompting).
-        - name: apikey-init
-          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/library/busybox:1.37.0
-          imagePullPolicy: IfNotPresent
-          securityContext:
-            runAsUser: 0
-          env:
-            - name: API_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: sonarr-apikey
-                  key: apitoken
-          command:
-            - sh
-            - -c
-            - |
-              set -eu
-              case "$API_KEY" in
-                "" | *[!0-9a-fA-F]*)
-                  echo "config-init: API_KEY missing or not hex; refusing" >&2
-                  exit 1
-                  ;;
-              esac
-              URL_BASE=/sonarr
-              AUTH_METHOD=External
-              AUTH_REQUIRED=Enabled
-              CFG=/config/config.xml
-              if [ ! -f "$CFG" ]; then
-                printf '<Config>\n  <ApiKey>%s</ApiKey>\n  <UrlBase>%s</UrlBase>\n  <AuthenticationMethod>%s</AuthenticationMethod>\n  <AuthenticationRequired>%s</AuthenticationRequired>\n</Config>\n' "$API_KEY" "$URL_BASE" "$AUTH_METHOD" "$AUTH_REQUIRED" > "$CFG"
-              else
-                if grep -q '<ApiKey>' "$CFG"; then
-                  sed -i "s|<ApiKey>[^<]*</ApiKey>|<ApiKey>${API_KEY}</ApiKey>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <ApiKey>${API_KEY}</ApiKey>|" "$CFG"
-                fi
-                if grep -q '<UrlBase>' "$CFG"; then
-                  sed -i "s|<UrlBase>[^<]*</UrlBase>|<UrlBase>${URL_BASE}</UrlBase>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <UrlBase>${URL_BASE}</UrlBase>|" "$CFG"
-                fi
-                if grep -q '<AuthenticationMethod>' "$CFG"; then
-                  sed -i "s|<AuthenticationMethod>[^<]*</AuthenticationMethod>|<AuthenticationMethod>${AUTH_METHOD}</AuthenticationMethod>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <AuthenticationMethod>${AUTH_METHOD}</AuthenticationMethod>|" "$CFG"
-                fi
-                if grep -q '<AuthenticationRequired>' "$CFG"; then
-                  sed -i "s|<AuthenticationRequired>[^<]*</AuthenticationRequired>|<AuthenticationRequired>${AUTH_REQUIRED}</AuthenticationRequired>|" "$CFG"
-                else
-                  sed -i "s|<Config>|<Config>\n  <AuthenticationRequired>${AUTH_REQUIRED}</AuthenticationRequired>|" "$CFG"
-                fi
-              fi
-              chown 1000:1000 "$CFG"
-              chmod 600 "$CFG"
-              echo "config-init: <ApiKey>, <UrlBase>=${URL_BASE}, <AuthenticationMethod>=${AUTH_METHOD}, <AuthenticationRequired>=${AUTH_REQUIRED} enforced from Vault"
-          resources:
-            requests:
-              cpu: 50m
-              memory: 32Mi
-            limits:
-              cpu: 200m
-              memory: 64Mi
-          volumeMounts:
-            - name: config
-              mountPath: /config
       containers:
         - name: sonarr
-          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/linuxserver/sonarr:4.0.19
+          image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/sonarr:v5.0.0-unkin2
           imagePullPolicy: IfNotPresent
+          command:
+            - /app/Sonarr
+          args:
+            - -nobrowser
+            - -data=/config
+            # Required: bypass the single-instance guard so multiple replicas
+            # can share one /config. Cross-replica safety is the Postgres layer,
+            # not a local lock file.
+            - -nosingleinstancecheck
           ports:
             - name: http
               containerPort: 8989
               protocol: TCP
+          envFrom:
+            - configMapRef:
+                name: sonarr-env
           env:
-            - name: PUID
+            - name: Sonarr__Postgres__User
-              value: "1000"
+              valueFrom:
-            - name: PGID
+                secretKeyRef:
-              value: "1000"
+                  name: sonarr-db
-            - name: TZ
+                  key: username
-              value: Australia/Sydney
+            - name: Sonarr__Postgres__Password
+              valueFrom:
+                secretKeyRef:
+                  name: sonarr-db
+                  key: password
+            - name: Sonarr__Auth__ApiKey
+              valueFrom:
+                secretKeyRef:
+                  name: sonarr-apikey
+                  key: apitoken
           livenessProbe:
             httpGet:
               path: /sonarr/ping

apps/base/arrstack/sonarr/kustomization.yaml

@@ -5,6 +5,7 @@ kind: Kustomization
 resources:
   - pvc-config.yaml
   - vaultstaticsecret.yaml
+  - configmap.yaml
   - deployment.yaml
   - service.yaml
   - gateway.yaml

apps/base/arrstack/sonarr/pvc-config.yaml

@@ -1,6 +1,7 @@
 ---
-# Sonarr config + SQLite DB. RWO on cephrbd (block) — the arr apps' SQLite
+# Sonarr /config. RWX on CephFS so all replicas share it (the -unkin2 fork keeps
-# does not tolerate CephFS locking. Retain: this is state.
+# the database in Postgres; /config now holds only config.xml + MediaCover, which
+# tolerate — and want — shared access). Retain: this is state.
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
@@ -8,9 +9,9 @@ metadata:
   namespace: arrstack
 spec:
   accessModes:
-    - ReadWriteOnce
+    - ReadWriteMany
   resources:
     requests:
       storage: 5Gi
-  storageClassName: cephrbd-fast-retain
+  storageClassName: cephfs-raid5-retain
   volumeMode: Filesystem