unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

arrstack: fix sync-wave deadlock + config multi-attach #385

Merged
benvin merged 2 commits from benvin/arrstack-deploy-fix into main 2026-08-22 12:47:39 +10:00
Conversation 1 Commits 2 Files Changed 11
+117 -26
unkin-agent commented 2026-08-22 12:29:32 +10:00
Member
Copy Link
Copy Source

Why

The active-active arrstack deploy (#383) is stuck in-cluster: the app Deployments (sync-wave 0) can never become Healthy without their Postgres DB, so ArgoCD never advances to create the shared arrstack-postgres cluster (wave 1). Separately, /config is an immutable RWO cephrbd PVC, so 3 replicas fail with Multi-Attach.

How

  • Order DB infra before the apps: <app>-db VaultStaticSecrets + backup ObjectStoreUser/Bucket at sync-wave -3, CNPG Cluster -2, Database CRDs -1; app Deployments stay at default 0.
  • Switch each app's /config to an emptyDir per-pod volume (library DB is in Postgres, config is env-driven; MediaCover regenerates). Old RWO config PVCs orphan.

Validated with kustomize build (62 resources). arrproxy/nzbget untouched.

## Why The active-active arrstack deploy (#383) is stuck in-cluster: the app Deployments (sync-wave 0) can never become Healthy without their Postgres DB, so ArgoCD never advances to create the shared `arrstack-postgres` cluster (wave 1). Separately, `/config` is an immutable RWO cephrbd PVC, so 3 replicas fail with Multi-Attach. ## How - Order DB infra before the apps: `<app>-db` VaultStaticSecrets + backup ObjectStoreUser/Bucket at sync-wave -3, CNPG `Cluster` -2, `Database` CRDs -1; app Deployments stay at default 0. - Switch each app's `/config` to an emptyDir per-pod volume (library DB is in Postgres, config is env-driven; MediaCover regenerates). Old RWO config PVCs orphan. Validated with kustomize build (62 resources). arrproxy/nzbget untouched.
unkin-agent added 1 commit 2026-08-22 12:29:33 +10:00
arrstack: fix sync-wave deadlock + config multi-attach
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
df687e310b 
The active-active arrstack deploy (#383) was stuck: the app Deployments
(wave 0) can't become Healthy without their database, so ArgoCD never
advanced to wave 1 to create the shared Postgres cluster. And /config was
an RWO cephrbd PVC, so 3 replicas hit Multi-Attach.

- move DB infra ahead of the apps: VSS + backup ObjectStoreUser/Bucket
  wave -3, CNPG Cluster -2, Databases -1 (apps stay at default 0)
- switch each app's /config to emptyDir (per-pod, stateless; library is
  in Postgres, config is env-driven; MediaCover regenerates). Old RWO
  config PVCs orphan.
unkin-agent added 1 commit 2026-08-22 12:38:04 +10:00
arrstack: gate apps on DB via initContainer instead of sync-waves
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
60309fc5c2 
Per review: drop the ArgoCD sync-wave annotations (their health-gating is
what deadlocked the first deploy) and instead add a wait-for-db init
container to each app that blocks until its own Postgres database+role is
reachable (libpq PG* env, password never in argv). ArgoCD applies
everything at once; the app container only starts once its DB is ready.
unkin-agent commented 2026-08-22 12:38:24 +10:00
Author
Member
Copy Link
Copy Source

Updated per feedback: replaced the ArgoCD sync-wave ordering with a wait-for-db initContainer on each app (the sync-wave health-gating is exactly what deadlocked the first deploy). Now:

  • Removed all sync-wave annotations from the postgres/ manifests (arrproxy's are untouched).
  • Each app deployment has an init container that runs until psql 'select 1' against its own database+role (<app>-main, creds from the <app>-db secret via libpq PG* env — password never in argv), blocking the app container until Postgres + the role + the DB are actually ready.
  • ArgoCD applies everything at once (no wave gating, no deadlock); k8s holds the pod in Init until its DB is up.
  • /config emptyDir change unchanged (fixes the RWO multi-attach).

kustomize build clean (62 resources).

Updated per feedback: replaced the ArgoCD sync-wave ordering with a **wait-for-db initContainer** on each app (the sync-wave health-gating is exactly what deadlocked the first deploy). Now: - Removed all sync-wave annotations from the postgres/ manifests (arrproxy's are untouched). - Each app deployment has an init container that runs `until psql 'select 1'` against its own database+role (`<app>-main`, creds from the `<app>-db` secret via libpq PG* env — password never in argv), blocking the app container until Postgres + the role + the DB are actually ready. - ArgoCD applies everything at once (no wave gating, no deadlock); k8s holds the pod in Init until its DB is up. - /config emptyDir change unchanged (fixes the RWO multi-attach). kustomize build clean (62 resources).
benvin merged commit 8446c31414 into main 2026-08-22 12:47:39 +10:00
benvin deleted branch benvin/arrstack-deploy-fix 2026-08-22 12:47:39 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#385
Delete Branch "benvin/arrstack-deploy-fix"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?