## Why
An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME while the puppetserver JVMs kept running against the now-deleted files. The running JVM re-execs `jspawnhelper` from its original (deleted) JAVA_HOME on every posix_spawn, so `ProcessBuilder` fails with `error=2, No such file or directory`. That broke the exec ENC (`/opt/cobbler-enc/cobbler-enc`) and 500'd every catalog compile across all 6 masters, failing 136/143 nodes. The masters could not self-heal because nothing restarts the JVM after a java upgrade. Incident was resolved by manually restarting puppetserver on all 6 masters.
## Change
- Add `/usr/local/bin/puppetserver_jvm_guard.sh`: restarts puppetserver if a running JVM (`puppet-server-release.jar`) is executing from deleted binaries.
- Add a `puppetserver-jvm-guard` systemd timer (every 5 min, mirrors the existing generate-types timer pattern) that runs the guard, so any future JVM/library upgrade recovers automatically.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #499
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- this command will always fail, remove the systemd dropin
- create script that will run and exit with 0
- create systemd service/timer to run script daily
- manage puppetserver package
- set order for puppetserver classes
- for profiles::puppet::server class:
- set param types using stdlib where possible
- set default values for all params
- move configuration data to hieradata
- wait for enc_role fact to match role
- exclude puppet::client from puppermaster nodes
- add puppetdb_api and puppetdb_sql role
- add puppetdb_api and puppetdb_sql profile
- add prodinf01n05 to /etc/hosts file
- set listen_address for all services to be hosts ip
- set storeconfigs and storeconfigs_backend to be managed by puppetmaster profile
- set through puppet.conf
- created symbolic link from develop -> production in code/environments
- changed puppet-g10k script to be generated from a template
- parameterised g10k into hieradata
unkinben