unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

arrstack: run sonarr/radarr/prowlarr active-active on shared Postgres #383

Merged
benvin merged 1 commits from benvin/arrstack-active-active into main 2026-08-18 21:29:59 +10:00
Conversation 1 Commits 1 Files Changed 20
+547 -264
unkin-agent commented 2026-08-18 20:46:19 +10:00
Member
Copy Link
Copy Source

Why

The arrstack sonarr/radarr/prowlarr apps currently run as single-replica upstream LinuxServer images on SQLite (RWO config PVC, Recreate strategy) — no HA, no horizontal scale, and a hard one-pod ceiling because SQLite can't be shared. Our -unkin2 forks move all app state into PostgreSQL and use Postgres advisory locks for cross-replica coordination (migrations, scheduled tasks, command execution), which makes an active-active multi-replica deployment safe.

This is a fresh Postgres backend — no SQLite data migration. The current single-replica pods are throwaway baselines; the new pods start on an empty per-app database and the fork's provider migrates the schema on first start. arrproxy (its own arrproxy-db) and nzbget are untouched.

What

New apps/base/arrstack/postgres/:

  • cnpg_cluster.yaml — CNPG Cluster arrstack-postgres (3 instances, cephrbd-fast-delete), throwaway app initdb owner, and 3 managed.roles (sonarr/radarr/prowlarr, login non-superuser) each with passwordSecret: <app>-db.
  • database-{sonarr,radarr,prowlarr}.yaml — CNPG Database <app>-main owned by <app>, databaseReclaimPolicy: retain.
  • cnpg_backup.yaml — cephrgw ObjectStoreUser + Bucket cnpg-arrstack + nightly ScheduledBackup (mirrors jellyfin).
  • vaultstaticsecret.yaml — 3 VSO VaultStaticSecrets syncing <app>-db (keys username/password) from kv/kubernetes/namespace/arrstack/default/<app>-db (already seeded); these are both the CNPG role passwordSecret and the Deployment's Postgres creds.
  • sync-waves: <app>-db Secrets (0) → Cluster (1) → Databases (2).
  • added postgres to apps/base/arrstack/kustomization.yaml.

Per app (sonarr/radarr/prowlarr):

  • deployment.yaml: image → docker-internal/<app>:v<base>-unkin2 (sonarr v5.0.0, radarr v6.4.2, prowlarr v2.6.2); replicas: 3; RollingUpdate. Removed the apikey-init initContainer and PUID/PGID/TZ. Runs the binary directly (command: [/app/<Bin>], args: [-nobrowser, -data=/config, -nosingleinstancecheck]). Env from the new ConfigMap + secretKeyRef: <App>__Postgres__User/__Password (Secret <app>-db) and <App>__Auth__ApiKey (existing <app>-apikey, key apitoken). /<app>/ping probes kept. securityContext runAsUser/runAsGroup/fsGroup 1000 for CephFS group-write.
  • new configmap.yaml: non-secret <App>__… env (Postgres Host/Port/MainDb, Log__DbEnabled=false, Auth__Method=External, Auth__Required=DisabledForLocalAddresses, App__InstanceName, Server__Port, Server__UrlBase=/<app>, Update__Mechanism=External).
  • pvc-config.yaml → RWX cephfs-raid5-retain (shared /config for config.xml/MediaCover).
  • unchanged: service.yaml, gateway.yaml, httproute.yaml, apikey vaultstaticsecret.yaml — arrproxy still addresses the Service VIP, so N replicas are transparent.

Validation

kustomize build apps/overlays/au-syd1/arrstack clean; kubeconform 64/64 valid; yamllint and the repo pre-commit (incl. no-plain-secrets) pass. Runtime validation (pods Ready, single migrate under advisory lock, scheduled-task/command single-owner, UI via https://arrstack.unkin.net/<app>/, 3→1→3 rescale) is post-merge once ArgoCD syncs.

Decisions / unknowns

  • Fork image UID/fsGroup: the -unkin2 images have no USER (run as root); pinned runAsUser/runAsGroup/fsGroup: 1000 with fsGroupChangePolicy: OnRootMismatch. Verify group-write on the shared CephFS /config on first deploy.
  • Config storage: chose RWX cephfs-raid5-retain shared /config (per spec preference) over per-pod emptyDir.
  • Postgres storage/version/resources: mirrored jellyfin/arrproxy CNPG conventions — cephrbd-fast-delete 10Gi, PG17, 3 instances.
## Why The arrstack `sonarr`/`radarr`/`prowlarr` apps currently run as single-replica upstream LinuxServer images on SQLite (RWO config PVC, `Recreate` strategy) — no HA, no horizontal scale, and a hard one-pod ceiling because SQLite can't be shared. Our `-unkin2` forks move all app state into PostgreSQL and use Postgres advisory locks for cross-replica coordination (migrations, scheduled tasks, command execution), which makes an **active-active multi-replica** deployment safe. This is a fresh Postgres backend — **no SQLite data migration**. The current single-replica pods are throwaway baselines; the new pods start on an empty per-app database and the fork's provider migrates the schema on first start. `arrproxy` (its own `arrproxy-db`) and `nzbget` are untouched. ## What New `apps/base/arrstack/postgres/`: - `cnpg_cluster.yaml` — CNPG `Cluster` `arrstack-postgres` (3 instances, `cephrbd-fast-delete`), throwaway `app` initdb owner, and 3 `managed.roles` (`sonarr`/`radarr`/`prowlarr`, login non-superuser) each with `passwordSecret: <app>-db`. - `database-{sonarr,radarr,prowlarr}.yaml` — CNPG `Database` `<app>-main` owned by `<app>`, `databaseReclaimPolicy: retain`. - `cnpg_backup.yaml` — cephrgw `ObjectStoreUser` + `Bucket cnpg-arrstack` + nightly `ScheduledBackup` (mirrors jellyfin). - `vaultstaticsecret.yaml` — 3 VSO `VaultStaticSecret`s syncing `<app>-db` (keys `username`/`password`) from `kv/kubernetes/namespace/arrstack/default/<app>-db` (already seeded); these are both the CNPG role `passwordSecret` and the Deployment's Postgres creds. - sync-waves: `<app>-db` Secrets (0) → Cluster (1) → Databases (2). - added `postgres` to `apps/base/arrstack/kustomization.yaml`. Per app (`sonarr`/`radarr`/`prowlarr`): - `deployment.yaml`: image → `docker-internal/<app>:v<base>-unkin2` (sonarr `v5.0.0`, radarr `v6.4.2`, prowlarr `v2.6.2`); `replicas: 3`; `RollingUpdate`. Removed the `apikey-init` initContainer and `PUID`/`PGID`/`TZ`. Runs the binary directly (`command: [/app/<Bin>]`, `args: [-nobrowser, -data=/config, -nosingleinstancecheck]`). Env from the new ConfigMap + `secretKeyRef`: `<App>__Postgres__User`/`__Password` (Secret `<app>-db`) and `<App>__Auth__ApiKey` (existing `<app>-apikey`, key `apitoken`). `/<app>/ping` probes kept. `securityContext` runAsUser/runAsGroup/fsGroup 1000 for CephFS group-write. - new `configmap.yaml`: non-secret `<App>__…` env (Postgres Host/Port/MainDb, `Log__DbEnabled=false`, `Auth__Method=External`, `Auth__Required=DisabledForLocalAddresses`, `App__InstanceName`, `Server__Port`, `Server__UrlBase=/<app>`, `Update__Mechanism=External`). - `pvc-config.yaml` → RWX `cephfs-raid5-retain` (shared `/config` for `config.xml`/MediaCover). - unchanged: `service.yaml`, `gateway.yaml`, `httproute.yaml`, apikey `vaultstaticsecret.yaml` — arrproxy still addresses the Service VIP, so N replicas are transparent. ## Validation `kustomize build apps/overlays/au-syd1/arrstack` clean; `kubeconform` **64/64 valid**; `yamllint` and the repo pre-commit (incl. no-plain-secrets) pass. Runtime validation (pods Ready, single migrate under advisory lock, scheduled-task/command single-owner, UI via `https://arrstack.unkin.net/<app>/`, 3→1→3 rescale) is post-merge once ArgoCD syncs. ## Decisions / unknowns - **Fork image UID/fsGroup**: the `-unkin2` images have no `USER` (run as root); pinned `runAsUser`/`runAsGroup`/`fsGroup: 1000` with `fsGroupChangePolicy: OnRootMismatch`. Verify group-write on the shared CephFS `/config` on first deploy. - **Config storage**: chose RWX `cephfs-raid5-retain` shared `/config` (per spec preference) over per-pod `emptyDir`. - **Postgres storage/version/resources**: mirrored jellyfin/arrproxy CNPG conventions — `cephrbd-fast-delete` 10Gi, PG17, 3 instances.
unkin-agent added 1 commit 2026-08-18 20:46:20 +10:00
arrstack: run sonarr/radarr/prowlarr active-active on shared Postgres (#arrstack)
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
f15d768227 
Convert the three *arr apps from single-replica upstream LinuxServer/SQLite to
our -unkin2 forks running active-active (replicas 3) on a shared CNPG Postgres
cluster.

- new apps/base/arrstack/postgres/: CNPG Cluster arrstack-postgres (3 instances,
  3 managed login roles sonarr/radarr/prowlarr, throwaway initdb owner), per-app
  Database CRDs (<app>-main), cephrgw backup bucket + nightly ScheduledBackup,
  and 3 VaultStaticSecrets syncing the <app>-db role credentials.
- per app: image -> docker-internal/<app>:v<base>-unkin2; replicas 3 +
  RollingUpdate; drop the apikey-init initContainer + PUID/PGID/TZ; run the
  binary directly with -nosingleinstancecheck; env from a new ConfigMap plus
  Postgres user/password and the API key from Secrets; /config PVC -> RWX
  cephfs-raid5-retain (shared).
unkin-agent commented 2026-08-18 20:55:12 +10:00
Author
Member
Copy Link
Copy Source

Correctness review — verdict: essentially CLEAN. Verified the env contract against the fork source at v5.0.0-unkin2 (unkin/sonarr): all keys bind to real Options classes (PostgresOptions/AuthOptions/ServerOptions/AppOptions/UpdateOptions/LogOptions), and Sonarr__Log__DbEnabled=false correctly routes to AddDummyLogDatabase() (Bootstrap.cs:100/241) so the absent Postgres__LogDb (only <app>-main provisioned) is fine. kustomize build + kubeconform clean (64 resources, 0 invalid). Config PVCs are RWX cephfs-raid5-retain (multi-attach safe), Services/ports/UrlBase unchanged, arrproxy reads the same apitoken key the apps now consume — no routing break. CNPG managed-role→VSS-Opaque-secret pattern mirrors the proven apps/base/puppet cluster. Sync-waves (secret 0 → cluster 1 → db 2) are ordered right.

Two pre-deploy items (not manifest bugs, but boot-blocking if wrong):

  1. Seeded username must equal the role name. Each app connects as <app>-db.username (ConnectionStringFactory → Npgsql ... Username) and its DB <app>-main is owned by role <app>. On a fresh DB the connecting role must BE the owner to create the schema, and CNPG also uses this same secret as the role passwordSecret. So kv/kubernetes/namespace/arrstack/default/<app>-db must have username=sonarr|radarr|prowlarr respectively (not a shared/other name) or first-boot migration fails with a permissions error. Worth a quick confirm before sync.

  2. Nit (optional): add a startupProbe. liveness is initialDelay 30 / period 30 / failureThreshold 3 (~120s budget). First boot has one replica migrating under the advisory lock while the other two wait; empty-schema migration is fast so this is low-risk, but a startupProbe would harden the RollingUpdate against lock-wait if migrations ever grow.

No blocking issues; safe to merge once (1) is confirmed.

Correctness review — **verdict: essentially CLEAN**. Verified the env contract against the fork source at `v5.0.0-unkin2` (unkin/sonarr): all keys bind to real Options classes (`PostgresOptions`/`AuthOptions`/`ServerOptions`/`AppOptions`/`UpdateOptions`/`LogOptions`), and `Sonarr__Log__DbEnabled=false` correctly routes to `AddDummyLogDatabase()` (Bootstrap.cs:100/241) so the absent `Postgres__LogDb` (only `<app>-main` provisioned) is fine. `kustomize build` + `kubeconform` clean (64 resources, 0 invalid). Config PVCs are RWX `cephfs-raid5-retain` (multi-attach safe), Services/ports/UrlBase unchanged, arrproxy reads the same `apitoken` key the apps now consume — no routing break. CNPG managed-role→VSS-Opaque-secret pattern mirrors the proven `apps/base/puppet` cluster. Sync-waves (secret 0 → cluster 1 → db 2) are ordered right. Two pre-deploy items (not manifest bugs, but boot-blocking if wrong): 1. **Seeded `username` must equal the role name.** Each app connects as `<app>-db`.`username` (`ConnectionStringFactory` → `Npgsql ... Username`) and its DB `<app>-main` is owned by role `<app>`. On a fresh DB the connecting role must BE the owner to create the schema, and CNPG also uses this same secret as the role `passwordSecret`. So `kv/kubernetes/namespace/arrstack/default/<app>-db` must have `username=sonarr|radarr|prowlarr` respectively (not a shared/other name) or first-boot migration fails with a permissions error. Worth a quick confirm before sync. 2. **Nit (optional): add a `startupProbe`.** liveness is `initialDelay 30 / period 30 / failureThreshold 3` (~120s budget). First boot has one replica migrating under the advisory lock while the other two wait; empty-schema migration is fast so this is low-risk, but a `startupProbe` would harden the RollingUpdate against lock-wait if migrations ever grow. No blocking issues; safe to merge once (1) is confirmed.
benvin merged commit 187fabb72b into main 2026-08-18 21:29:59 +10:00
benvin deleted branch benvin/arrstack-active-active 2026-08-18 21:30:00 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#383
Delete Branch "benvin/arrstack-active-active"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?