puppetserver: auto-restart JVM when its binaries are replaced by an upgrade #499

Merged

benvin merged 1 commits from benvin/puppetserver-jvm-restart-on-upgrade into develop 2026-07-30 20:50:31 +10:00

Conversation 0 Commits 1 Files Changed 1

+60

unkinben commented 2026-07-30 00:46:53 +10:00

Owner

Copy Link

Copy Source

Why

An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME while the puppetserver JVMs kept running against the now-deleted files. The running JVM re-execs jspawnhelper from its original (deleted) JAVA_HOME on every posix_spawn, so ProcessBuilder fails with error=2, No such file or directory. That broke the exec ENC (/opt/cobbler-enc/cobbler-enc) and 500'd every catalog compile across all 6 masters, failing 136/143 nodes. The masters could not self-heal because nothing restarts the JVM after a java upgrade. Incident was resolved by manually restarting puppetserver on all 6 masters.

Change

• Add /usr/local/bin/puppetserver_jvm_guard.sh: restarts puppetserver if a running JVM (puppet-server-release.jar) is executing from deleted binaries.
• Add a puppetserver-jvm-guard systemd timer (every 5 min, mirrors the existing generate-types timer pattern) that runs the guard, so any future JVM/library upgrade recovers automatically.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME while the puppetserver JVMs kept running against the now-deleted files. The running JVM re-execs `jspawnhelper` from its original (deleted) JAVA_HOME on every posix_spawn, so `ProcessBuilder` fails with `error=2, No such file or directory`. That broke the exec ENC (`/opt/cobbler-enc/cobbler-enc`) and 500'd every catalog compile across all 6 masters, failing 136/143 nodes. The masters could not self-heal because nothing restarts the JVM after a java upgrade. Incident was resolved by manually restarting puppetserver on all 6 masters. ## Change - Add `/usr/local/bin/puppetserver_jvm_guard.sh`: restarts puppetserver if a running JVM (`puppet-server-release.jar`) is executing from deleted binaries. - Add a `puppetserver-jvm-guard` systemd timer (every 5 min, mirrors the existing generate-types timer pattern) that runs the guard, so any future JVM/library upgrade recovers automatically. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

unkinben added 1 commit 2026-07-30 00:46:54 +10:00

profiles::puppet::server: restart puppetserver when its JVM binary is replaced ... 85a5afda6b

An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered
by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME
while the puppetserver JVMs kept running against the now-deleted files. The
running JVM re-execs jspawnhelper from its original (deleted) JAVA_HOME on every
posix_spawn, so ProcessBuilder fails with "error=2, No such file or directory".
That broke the exec ENC (/opt/cobbler-enc/cobbler-enc) and 500'd every catalog
compile across all 6 masters, failing 136/143 nodes. The masters could not
self-heal because nothing restarts the JVM on a java upgrade.

Add a puppetserver-jvm-guard systemd timer (every 5 min) that detects a
puppetserver JVM executing from deleted binaries and restarts the service,
so any future JVM/library upgrade recovers automatically.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

benvin merged commit dd38651a9f into develop 2026-07-30 20:50:31 +10:00

benvin deleted branch benvin/puppetserver-jvm-restart-on-upgrade 2026-07-30 20:50:31 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/puppet-prod#499