fix: make plugin sync safe for concurrent replica starts #12

Merged
benvin merged 1 commits from benvin/fix-plugin-sync-race into main 2026-09-13 18:57:11 +10:00
Member

Replicas share /config over RWX CephFS. The plugin sync unconditionally rm -rf'd then cp -a'd each plugin dir on every start, so two replicas starting together raced: one hit a permission error deleting a dir the other was writing, or crashed later with UnauthorizedAccessException loading a half-copied plugin. Reproduced 3/3 with two simultaneous starts.

  • skip the sync once the correct version is already installed
  • stage a new/changed version privately per replica, then atomic-rename into place
  • treat a lost rename race as success, not an error
  • best-effort remove stale differently-versioned copies so they don't shadow the current one
  • no locking: a stuck lock from a dead replica would be worse than the race
Replicas share /config over RWX CephFS. The plugin sync unconditionally rm -rf'd then cp -a'd each plugin dir on every start, so two replicas starting together raced: one hit a permission error deleting a dir the other was writing, or crashed later with UnauthorizedAccessException loading a half-copied plugin. Reproduced 3/3 with two simultaneous starts. - skip the sync once the correct version is already installed - stage a new/changed version privately per replica, then atomic-rename into place - treat a lost rename race as success, not an error - best-effort remove stale differently-versioned copies so they don't shadow the current one - no locking: a stuck lock from a dead replica would be worse than the race
unkin-agent added 1 commit 2026-09-13 18:50:11 +10:00
fix: make plugin sync safe for concurrent replica starts
ci/woodpecker/pr/build Pipeline was successful
e0a195179b
Two replicas starting together on the shared /config volume race the
unconditional rm+cp, so one dies with a permission error or a plugin
UnauthorizedAccessException. Skip the sync once the correct version
is already present, stage new/changed versions and move them into
place with an atomic rename, and treat a lost rename race or an
already-removed stale version as success rather than failure.
benvin merged commit e103b57e97 into main 2026-09-13 18:57:11 +10:00
benvin deleted branch benvin/fix-plugin-sync-race 2026-09-13 18:57:11 +10:00
Sign in to join this conversation.