## Why
A package versionlock is only effective if it is applied **before** the package resource is evaluated. When the package is upgraded/installed first and locked afterwards, dnf can pull an unpinned version and the lock then pins whatever landed. This is the exact drift that let new RKE2 nodes boot 1.33.11 instead of the pinned version, and it also matters because a global `dnf update -y` autoupdate timer (`profiles::yum::autoupdater`) relies on the versionlock plugin already having the lock written.
## Changes
- `rke2::install` orders the versionlock ahead of the package: adds `before => Package["rke2-${node_type}"]` so the lock exists before install/upgrade.
- `profiles::puppet::agent` (RedHat branch) adds `before => Package[$use_package]` on the versionlock for the same reason.
- Leaves `profiles::gitea::runner` and `profiles::vault::server` unchanged: both already order the versionlock ahead of the package (`before => Package` / `before => Class['vault']`).
- Changes ordering only; locked versions (including the RKE2 pin) are untouched. No dependency cycle is introduced (packages hold no back-reference to the versionlock).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #514
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME while the puppetserver JVMs kept running against the now-deleted files. The running JVM re-execs `jspawnhelper` from its original (deleted) JAVA_HOME on every posix_spawn, so `ProcessBuilder` fails with `error=2, No such file or directory`. That broke the exec ENC (`/opt/cobbler-enc/cobbler-enc`) and 500'd every catalog compile across all 6 masters, failing 136/143 nodes. The masters could not self-heal because nothing restarts the JVM after a java upgrade. Incident was resolved by manually restarting puppetserver on all 6 masters.
## Change
- Add `/usr/local/bin/puppetserver_jvm_guard.sh`: restarts puppetserver if a running JVM (`puppet-server-release.jar`) is executing from deleted binaries.
- Add a `puppetserver-jvm-guard` systemd timer (every 5 min, mirrors the existing generate-types timer pattern) that runs the guard, so any future JVM/library upgrade recovers automatically.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #499
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
We want to manage Fedora 44 hosts with this Puppet codebase, starting with the `base` role only. Fedora reuses the RedHat osfamily code paths (dnf/yum, crypto-policies, firewalld, openvox), so this adds the Fedora-specific hieradata, repositories, and gates needed for `base` to compile and apply, mirroring how AlmaLinux is wired and consuming the artifactapi `rpm-internal-f<major>` / `rpm-vendor-f<major>` local repos. Everything is keyed off `facts.os.release.major` so a future Fedora release only needs its artifactapi local repos created, not new hieradata.
## How
- Add `hieradata/os/Fedora/all_releases.yaml`:
- Define dnf repos via `profiles::yum::global::repos`: `fedora` and `updates` proxied through artifactapi's `fedora` remote, plus the artifactapi locals `rpm-internal-f%{major}` and `rpm-vendor-f%{major}`; GPG via the on-disk `fedora-gpg-keys`; metalink/mirrorlist cleared so only the artifactapi baseurl is used.
- Set `crypto_policies::policy: DEFAULT`, `profiles::puppet::agent::openvox_enable: true`, and `lm-sensors:📦 lm_sensors`.
- Fix up the base package set for Fedora: exclude `p7zip`/`dstat`/`iotop` (absent on Fedora) and include `7zip` and `iotop-c`, plus the same `crypto-policies-scripts`/`lzo`/`policycoreutils`/`unar`/`xz` additions AlmaLinux carries.
- Wire `profiles::fedora::base` via `hiera_include`.
- Add `profiles::fedora::base` (ensures NetworkManager enabled) as the Fedora analogue of `profiles::almalinux::base`; deliberately minimal so it can grow into workstation/laptop use later.
- Make `profiles::puppet::agent` select the OpenVox distribution path (`fedora/<major>` on Fedora, `el/<major>` elsewhere); AlmaLinux/Debian behaviour is unchanged.
## Validation
`puppet-lint`, puppet manifest validate, and `yamllint` all pass via the repo's pre-commit hooks on the changed files.
## Note for reviewer
OpenVox does not yet publish a Fedora 44 build (`openvox7/fedora/` currently has only 36/40/41), so `openvox-agent` will 404 until upstream publishes f44 or a build is placed in `rpm-internal-f44`. The Puppet code produces the correct path for when that exists; installing the agent is a prerequisite for a Fedora 44 host to actually run.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #497
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Migrate the VM puppet agents off the legacy VM puppetmasters onto the new puppet-on-kubernetes servers, one wave at a time and reversible without re-enrolment.
Changing `server`/`ca_server` alone is insufficient: each agent's ssldir holds a client certificate signed by the **old** CA, which the new k8s CA neither trusts nor recognises. Migrated nodes switch to a **fresh ssldir**, so the agent generates a new key + CSR on its next run (autosigned immediately by the k8s CA, which runs `autosign = true`), pulls its catalog from the new compilers, and leaves the old CA-signed creds intact on disk for rollback.
The reconfiguration catalog is compiled by the **old** server (the node still points there when it applies the change). `profiles::puppet::client` rewrites `puppet.conf` with the new endpoints + ssldir and notifies `Service['puppet']`, so the very next agent run lands on k8s.
**Single-boolean design (reworked):** cutover is now driven by one hiera key. `profiles::puppet::migrate` ships wired into `profiles::base` with `enabled => false`, so every node evaluates it but nothing changes until the boolean is flipped. `profiles::puppet::client` includes migrate and folds the migrate endpoints into the effective values it renders. Flip `profiles::puppet::migrate::enabled: true` at any hiera layer (node/role/common) to migrate a target -- no other keys required.
## Changes
- `profiles::base` (`site/profiles/manifests/base.pp`): `include profiles::puppet::migrate` alongside the other puppet profiles (ships disabled).
- `profiles::puppet::migrate` (`site/profiles/manifests/puppet/migrate.pp`): unchanged params (`enabled=false`, `new_server`, `new_ca_server`, `new_ssldir`); when enabled and the node is not a puppetmaster it owns the fresh ssldir directory. Class-header runbook rewritten for the single-boolean flow. No longer opt-in via `hiera_include`.
- `profiles::puppet::client` (`site/profiles/manifests/puppet/client.pp`): `include profiles::puppet::migrate` and compute `$effective_server` / `$effective_ca_server` / `$effective_report_server` / `$effective_ssldir`, feeding those to the template. Keeps the existing optional `$ssldir` / `$report_server` params as manual overrides.
- `puppet.conf.erb`: render from the `@effective_*` vars.
## Precedence
- **ssldir / report_server** (`Optional`, default `undef`): explicit `profiles::puppet::client::ssldir` / `::report_server` in hiera wins; else the migrate-derived value when `enabled`; else the legacy default (ssldir omitted -> puppet default; report_server -> effective server).
- **server / ca_server** (`String`, no undef sentinel): the migrate endpoint wins while `enabled => true`; otherwise the client param (legacy default unless set explicitly). To pin a bespoke server, leave `enabled => false` for that target and set `profiles::puppet::client::server` directly.
- No include cycle: migrate.pp never references client.pp; its guard uses `$facts['enc_role']`.
## Runbook
- **Canary (one node)** -> `hieradata/nodes/<certname>.yaml`: `profiles::puppet::migrate::enabled: true`
- **Wave (a role)** -> `hieradata/roles/<t1>/<t2>[/<t3>].yaml`: `profiles::puppet::migrate::enabled: true`
- **Estate flip** -> `hieradata/common.yaml`: `profiles::puppet::migrate::enabled: true`
- **Rollback** -> remove the key (or set `false`) at that layer; next run rewrites puppet.conf back to the legacy server + legacy ssldir (original CA-signed cert intact).
- The k8s endpoints (`new_server` / `new_ca_server` / `new_ssldir`) are `profiles::puppet::migrate` param defaults; override them in hiera if the k8s names change.
## Verification
- Render comparison: unmigrated node renders a **byte-identical** `puppet.conf` to `develop` (diff empty). Migrated render (only `enabled: true`) emits `server`/`ca_server`/`report_server` = k8s endpoints and `ssldir = /etc/puppetlabs/puppet/ssl-k8s`.
- `puppet-lint` (repo args `--no-80chars-check --no-documentation-check --no-puppet_url_without_modules-check --fail-on-warnings`): clean on all 3 manifests.
- `puppet parser validate`: OK. ERB syntax: OK. Repo pre-commit hooks (erb-validate, puppet-lint, puppet-validate): all Passed.
Reviewed-on: #493
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- enable openvox repo
- ensure puppetdb-termini and puppetserver are purged
- set openvox-server as the package to install
- set termini package to openvoxdb-termini
Reviewed-on: #412
- change from puppet-agent to openvox-agent
- upgrade version from 7.34 to 7.36
- ensure workflow of: Yumrepo -> dnf-makecache -> Package
Reviewed-on: #408
- change puppetdb::sql to using the patroni profile
- change puppetdb::api to use new patroni cluster
- remove references to puppetlabs-puppetdb managed database
- update consul rules to enable sessions
Reviewed-on: https://git.query.consul/unkinben/puppet-prod/pulls/318
- this command will always fail, remove the systemd dropin
- create script that will run and exit with 0
- create systemd service/timer to run script daily
- ensure a database, user and credential is created for each grafana node
- ensure all databases for a region are included in a mariadb cluster
- refine params with stdlib types
- manage python script/venv to sign ssh host certificates
- add approle_id to puppetmaster eyaml files
- add class to sign ssh-rsa host keys
- add facts to check if the current principals match the desired principals
- add consul support
- enable local script checks in consul agents
- add a test DB/User for consult to verify the psql instance is running
- manage the postgresql repo and gpg key
- change profiles::puppet::agent to require Yumrepo['puppet']
- remove managed repos hieradata
- remove profiles::yum::* classes that are not required
- remove missed rebase comment
- deep merge postgresql_config_entries in common.yaml
- add postgresql_config_entries into a new hieradata/roles/infra/puppetdb/sql.yaml
- set puppetdb role to import the options
- wait for the enc_role fact to be updated and match
- move puppetdb db/api host values to common.yaml
- add vault cert altnames for consul query/service addresses
- add consul services/rules/checks
- add puppet service
- add puppetca service
- add ability to write to puppet/puppetca service in consul
- add puppet.(query,service).consul to dns_alt_names of all masters
- add puppetca.(query,service).consul to dns_alt_names of puppetca
- manage puppetserver package
- set order for puppetserver classes
- for profiles::puppet::server class:
- set param types using stdlib where possible
- set default values for all params
- move configuration data to hieradata
- wait for enc_role fact to match role
- exclude puppet::client from puppermaster nodes
- set yum::versionlock to be only for redhat family
- set puppet-agent require statement to use apt or yum
- remove requirement of downloading puppet7-release-$dist.deb
- create all paths in $base_path for vault certificate
- set correct $PATH for update-ca-certificates
- dynamically set debian release name
- split packages to install from common.yaml to os-specific
- create groups profile to manage local groups
- change sysadmin to be a member of admins group
- setup admins sudo rules
unkinben