## Why
An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME while the puppetserver JVMs kept running against the now-deleted files. The running JVM re-execs `jspawnhelper` from its original (deleted) JAVA_HOME on every posix_spawn, so `ProcessBuilder` fails with `error=2, No such file or directory`. That broke the exec ENC (`/opt/cobbler-enc/cobbler-enc`) and 500'd every catalog compile across all 6 masters, failing 136/143 nodes. The masters could not self-heal because nothing restarts the JVM after a java upgrade. Incident was resolved by manually restarting puppetserver on all 6 masters.
## Change
- Add `/usr/local/bin/puppetserver_jvm_guard.sh`: restarts puppetserver if a running JVM (`puppet-server-release.jar`) is executing from deleted binaries.
- Add a `puppetserver-jvm-guard` systemd timer (every 5 min, mirrors the existing generate-types timer pattern) that runs the guard, so any future JVM/library upgrade recovers automatically.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #499
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
We want to manage Fedora 44 hosts with this Puppet codebase, starting with the `base` role only. Fedora reuses the RedHat osfamily code paths (dnf/yum, crypto-policies, firewalld, openvox), so this adds the Fedora-specific hieradata, repositories, and gates needed for `base` to compile and apply, mirroring how AlmaLinux is wired and consuming the artifactapi `rpm-internal-f<major>` / `rpm-vendor-f<major>` local repos. Everything is keyed off `facts.os.release.major` so a future Fedora release only needs its artifactapi local repos created, not new hieradata.
## How
- Add `hieradata/os/Fedora/all_releases.yaml`:
- Define dnf repos via `profiles::yum::global::repos`: `fedora` and `updates` proxied through artifactapi's `fedora` remote, plus the artifactapi locals `rpm-internal-f%{major}` and `rpm-vendor-f%{major}`; GPG via the on-disk `fedora-gpg-keys`; metalink/mirrorlist cleared so only the artifactapi baseurl is used.
- Set `crypto_policies::policy: DEFAULT`, `profiles::puppet::agent::openvox_enable: true`, and `lm-sensors:📦 lm_sensors`.
- Fix up the base package set for Fedora: exclude `p7zip`/`dstat`/`iotop` (absent on Fedora) and include `7zip` and `iotop-c`, plus the same `crypto-policies-scripts`/`lzo`/`policycoreutils`/`unar`/`xz` additions AlmaLinux carries.
- Wire `profiles::fedora::base` via `hiera_include`.
- Add `profiles::fedora::base` (ensures NetworkManager enabled) as the Fedora analogue of `profiles::almalinux::base`; deliberately minimal so it can grow into workstation/laptop use later.
- Make `profiles::puppet::agent` select the OpenVox distribution path (`fedora/<major>` on Fedora, `el/<major>` elsewhere); AlmaLinux/Debian behaviour is unchanged.
## Validation
`puppet-lint`, puppet manifest validate, and `yamllint` all pass via the repo's pre-commit hooks on the changed files.
## Note for reviewer
OpenVox does not yet publish a Fedora 44 build (`openvox7/fedora/` currently has only 36/40/41), so `openvox-agent` will 404 until upstream publishes f44 or a build is placed in `rpm-internal-f44`. The Puppet code produces the correct path for when that exists; installing the agent is a prerequisite for a Fedora 44 host to actually run.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #497
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Migrate the VM puppet agents off the legacy VM puppetmasters onto the new puppet-on-kubernetes servers, one wave at a time and reversible without re-enrolment.
Changing `server`/`ca_server` alone is insufficient: each agent's ssldir holds a client certificate signed by the **old** CA, which the new k8s CA neither trusts nor recognises. Migrated nodes switch to a **fresh ssldir**, so the agent generates a new key + CSR on its next run (autosigned immediately by the k8s CA, which runs `autosign = true`), pulls its catalog from the new compilers, and leaves the old CA-signed creds intact on disk for rollback.
The reconfiguration catalog is compiled by the **old** server (the node still points there when it applies the change). `profiles::puppet::client` rewrites `puppet.conf` with the new endpoints + ssldir and notifies `Service['puppet']`, so the very next agent run lands on k8s.
**Single-boolean design (reworked):** cutover is now driven by one hiera key. `profiles::puppet::migrate` ships wired into `profiles::base` with `enabled => false`, so every node evaluates it but nothing changes until the boolean is flipped. `profiles::puppet::client` includes migrate and folds the migrate endpoints into the effective values it renders. Flip `profiles::puppet::migrate::enabled: true` at any hiera layer (node/role/common) to migrate a target -- no other keys required.
## Changes
- `profiles::base` (`site/profiles/manifests/base.pp`): `include profiles::puppet::migrate` alongside the other puppet profiles (ships disabled).
- `profiles::puppet::migrate` (`site/profiles/manifests/puppet/migrate.pp`): unchanged params (`enabled=false`, `new_server`, `new_ca_server`, `new_ssldir`); when enabled and the node is not a puppetmaster it owns the fresh ssldir directory. Class-header runbook rewritten for the single-boolean flow. No longer opt-in via `hiera_include`.
- `profiles::puppet::client` (`site/profiles/manifests/puppet/client.pp`): `include profiles::puppet::migrate` and compute `$effective_server` / `$effective_ca_server` / `$effective_report_server` / `$effective_ssldir`, feeding those to the template. Keeps the existing optional `$ssldir` / `$report_server` params as manual overrides.
- `puppet.conf.erb`: render from the `@effective_*` vars.
## Precedence
- **ssldir / report_server** (`Optional`, default `undef`): explicit `profiles::puppet::client::ssldir` / `::report_server` in hiera wins; else the migrate-derived value when `enabled`; else the legacy default (ssldir omitted -> puppet default; report_server -> effective server).
- **server / ca_server** (`String`, no undef sentinel): the migrate endpoint wins while `enabled => true`; otherwise the client param (legacy default unless set explicitly). To pin a bespoke server, leave `enabled => false` for that target and set `profiles::puppet::client::server` directly.
- No include cycle: migrate.pp never references client.pp; its guard uses `$facts['enc_role']`.
## Runbook
- **Canary (one node)** -> `hieradata/nodes/<certname>.yaml`: `profiles::puppet::migrate::enabled: true`
- **Wave (a role)** -> `hieradata/roles/<t1>/<t2>[/<t3>].yaml`: `profiles::puppet::migrate::enabled: true`
- **Estate flip** -> `hieradata/common.yaml`: `profiles::puppet::migrate::enabled: true`
- **Rollback** -> remove the key (or set `false`) at that layer; next run rewrites puppet.conf back to the legacy server + legacy ssldir (original CA-signed cert intact).
- The k8s endpoints (`new_server` / `new_ca_server` / `new_ssldir`) are `profiles::puppet::migrate` param defaults; override them in hiera if the k8s names change.
## Verification
- Render comparison: unmigrated node renders a **byte-identical** `puppet.conf` to `develop` (diff empty). Migrated render (only `enabled: true`) emits `server`/`ca_server`/`report_server` = k8s endpoints and `ssldir = /etc/puppetlabs/puppet/ssl-k8s`.
- `puppet-lint` (repo args `--no-80chars-check --no-documentation-check --no-puppet_url_without_modules-check --fail-on-warnings`): clean on all 3 manifests.
- `puppet parser validate`: OK. ERB syntax: OK. Repo pre-commit hooks (erb-validate, puppet-lint, puppet-validate): all Passed.
Reviewed-on: #493
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- enable openvox repo
- ensure puppetdb-termini and puppetserver are purged
- set openvox-server as the package to install
- set termini package to openvoxdb-termini
Reviewed-on: #412
- change from puppet-agent to openvox-agent
- upgrade version from 7.34 to 7.36
- ensure workflow of: Yumrepo -> dnf-makecache -> Package
Reviewed-on: #408
- change puppetdb::sql to using the patroni profile
- change puppetdb::api to use new patroni cluster
- remove references to puppetlabs-puppetdb managed database
- update consul rules to enable sessions
Reviewed-on: https://git.query.consul/unkinben/puppet-prod/pulls/318
- this command will always fail, remove the systemd dropin
- create script that will run and exit with 0
- create systemd service/timer to run script daily
- ensure a database, user and credential is created for each grafana node
- ensure all databases for a region are included in a mariadb cluster
- refine params with stdlib types
- manage python script/venv to sign ssh host certificates
- add approle_id to puppetmaster eyaml files
- add class to sign ssh-rsa host keys
- add facts to check if the current principals match the desired principals
- add consul support
- enable local script checks in consul agents
- add a test DB/User for consult to verify the psql instance is running
- manage the postgresql repo and gpg key
- change profiles::puppet::agent to require Yumrepo['puppet']
- remove managed repos hieradata
- remove profiles::yum::* classes that are not required
- remove missed rebase comment
- deep merge postgresql_config_entries in common.yaml
- add postgresql_config_entries into a new hieradata/roles/infra/puppetdb/sql.yaml
- set puppetdb role to import the options
- wait for the enc_role fact to be updated and match
- move puppetdb db/api host values to common.yaml
- add vault cert altnames for consul query/service addresses
- add consul services/rules/checks
- add puppet service
- add puppetca service
- add ability to write to puppet/puppetca service in consul
- add puppet.(query,service).consul to dns_alt_names of all masters
- add puppetca.(query,service).consul to dns_alt_names of puppetca
- manage puppetserver package
- set order for puppetserver classes
- for profiles::puppet::server class:
- set param types using stdlib where possible
- set default values for all params
- move configuration data to hieradata
- wait for enc_role fact to match role
- exclude puppet::client from puppermaster nodes
- set yum::versionlock to be only for redhat family
- set puppet-agent require statement to use apt or yum
- remove requirement of downloading puppet7-release-$dist.deb
- create all paths in $base_path for vault certificate
- set correct $PATH for update-ca-certificates
- dynamically set debian release name
- split packages to install from common.yaml to os-specific
- create groups profile to manage local groups
- change sysadmin to be a member of admins group
- setup admins sudo rules
unkinben