unkinben dd38651a9f puppetserver: auto-restart JVM when its binaries are replaced by an upgrade (#499)
## Why

An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME while the puppetserver JVMs kept running against the now-deleted files. The running JVM re-execs `jspawnhelper` from its original (deleted) JAVA_HOME on every posix_spawn, so `ProcessBuilder` fails with `error=2, No such file or directory`. That broke the exec ENC (`/opt/cobbler-enc/cobbler-enc`) and 500'd every catalog compile across all 6 masters, failing 136/143 nodes. The masters could not self-heal because nothing restarts the JVM after a java upgrade. Incident was resolved by manually restarting puppetserver on all 6 masters.

## Change

- Add `/usr/local/bin/puppetserver_jvm_guard.sh`: restarts puppetserver if a running JVM (`puppet-server-release.jar`) is executing from deleted binaries.
- Add a `puppetserver-jvm-guard` systemd timer (every 5 min, mirrors the existing generate-types timer pattern) that runs the guard, so any future JVM/library upgrade recovers automatically.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #499
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-30 20:50:30 +10:00
2026-03-17 17:38:22 +11:00
2026-02-03 19:56:14 +11:00
2023-07-02 14:21:09 +10:00
2023-06-21 22:03:43 +10:00
2025-07-08 20:19:36 +10:00
2024-02-17 22:57:36 +11:00
S
Description
production puppet-control repository
3.7 MiB
Languages
Puppet 65.8%
HTML 28.5%
Ruby 5.7%