unkin/puppet-prod
3
0
Fork 0
Code Issues 10 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add profiles::puppet::migrate: repoint VM agents onto puppet-on-k8s #493

Merged
benvin merged 2 commits from benvin/puppet-k8s-migrate into develop 2026-07-25 00:04:11 +10:00
Conversation 0 Commits 2 Files Changed 4
+152 -14
unkinben commented 2026-07-24 22:44:14 +10:00
Owner
Copy Link
Copy Source

Why

Migrate the VM puppet agents off the legacy VM puppetmasters onto the new puppet-on-kubernetes servers, one wave at a time and reversible without re-enrolment.

Changing server/ca_server alone is insufficient: each agent's ssldir holds a client certificate signed by the old CA, which the new k8s CA neither trusts nor recognises. Migrated nodes switch to a fresh ssldir, so the agent generates a new key + CSR on its next run (autosigned immediately by the k8s CA, which runs autosign = true), pulls its catalog from the new compilers, and leaves the old CA-signed creds intact on disk for rollback.

The reconfiguration catalog is compiled by the old server (the node still points there when it applies the change). profiles::puppet::client rewrites puppet.conf with the new endpoints + ssldir and notifies Service['puppet'], so the very next agent run lands on k8s.

Single-boolean design (reworked): cutover is now driven by one hiera key. profiles::puppet::migrate ships wired into profiles::base with enabled => false, so every node evaluates it but nothing changes until the boolean is flipped. profiles::puppet::client includes migrate and folds the migrate endpoints into the effective values it renders. Flip profiles::puppet::migrate::enabled: true at any hiera layer (node/role/common) to migrate a target -- no other keys required.

Changes

  • profiles::base (site/profiles/manifests/base.pp): include profiles::puppet::migrate alongside the other puppet profiles (ships disabled).
  • profiles::puppet::migrate (site/profiles/manifests/puppet/migrate.pp): unchanged params (enabled=false, new_server, new_ca_server, new_ssldir); when enabled and the node is not a puppetmaster it owns the fresh ssldir directory. Class-header runbook rewritten for the single-boolean flow. No longer opt-in via hiera_include.
  • profiles::puppet::client (site/profiles/manifests/puppet/client.pp): include profiles::puppet::migrate and compute $effective_server / $effective_ca_server / $effective_report_server / $effective_ssldir, feeding those to the template. Keeps the existing optional $ssldir / $report_server params as manual overrides.
  • puppet.conf.erb: render from the @effective_* vars.

Precedence

  • ssldir / report_server (Optional, default undef): explicit profiles::puppet::client::ssldir / ::report_server in hiera wins; else the migrate-derived value when enabled; else the legacy default (ssldir omitted -> puppet default; report_server -> effective server).
  • server / ca_server (String, no undef sentinel): the migrate endpoint wins while enabled => true; otherwise the client param (legacy default unless set explicitly). To pin a bespoke server, leave enabled => false for that target and set profiles::puppet::client::server directly.
  • No include cycle: migrate.pp never references client.pp; its guard uses $facts['enc_role'].

Runbook

  • Canary (one node) -> hieradata/nodes/<certname>.yaml: profiles::puppet::migrate::enabled: true
  • Wave (a role) -> hieradata/roles/<t1>/<t2>[/<t3>].yaml: profiles::puppet::migrate::enabled: true
  • Estate flip -> hieradata/common.yaml: profiles::puppet::migrate::enabled: true
  • Rollback -> remove the key (or set false) at that layer; next run rewrites puppet.conf back to the legacy server + legacy ssldir (original CA-signed cert intact).
  • The k8s endpoints (new_server / new_ca_server / new_ssldir) are profiles::puppet::migrate param defaults; override them in hiera if the k8s names change.

Verification

  • Render comparison: unmigrated node renders a byte-identical puppet.conf to develop (diff empty). Migrated render (only enabled: true) emits server/ca_server/report_server = k8s endpoints and ssldir = /etc/puppetlabs/puppet/ssl-k8s.
  • puppet-lint (repo args --no-80chars-check --no-documentation-check --no-puppet_url_without_modules-check --fail-on-warnings): clean on all 3 manifests.
  • puppet parser validate: OK. ERB syntax: OK. Repo pre-commit hooks (erb-validate, puppet-lint, puppet-validate): all Passed.
## Why Migrate the VM puppet agents off the legacy VM puppetmasters onto the new puppet-on-kubernetes servers, one wave at a time and reversible without re-enrolment. Changing `server`/`ca_server` alone is insufficient: each agent's ssldir holds a client certificate signed by the **old** CA, which the new k8s CA neither trusts nor recognises. Migrated nodes switch to a **fresh ssldir**, so the agent generates a new key + CSR on its next run (autosigned immediately by the k8s CA, which runs `autosign = true`), pulls its catalog from the new compilers, and leaves the old CA-signed creds intact on disk for rollback. The reconfiguration catalog is compiled by the **old** server (the node still points there when it applies the change). `profiles::puppet::client` rewrites `puppet.conf` with the new endpoints + ssldir and notifies `Service['puppet']`, so the very next agent run lands on k8s. **Single-boolean design (reworked):** cutover is now driven by one hiera key. `profiles::puppet::migrate` ships wired into `profiles::base` with `enabled => false`, so every node evaluates it but nothing changes until the boolean is flipped. `profiles::puppet::client` includes migrate and folds the migrate endpoints into the effective values it renders. Flip `profiles::puppet::migrate::enabled: true` at any hiera layer (node/role/common) to migrate a target -- no other keys required. ## Changes - `profiles::base` (`site/profiles/manifests/base.pp`): `include profiles::puppet::migrate` alongside the other puppet profiles (ships disabled). - `profiles::puppet::migrate` (`site/profiles/manifests/puppet/migrate.pp`): unchanged params (`enabled=false`, `new_server`, `new_ca_server`, `new_ssldir`); when enabled and the node is not a puppetmaster it owns the fresh ssldir directory. Class-header runbook rewritten for the single-boolean flow. No longer opt-in via `hiera_include`. - `profiles::puppet::client` (`site/profiles/manifests/puppet/client.pp`): `include profiles::puppet::migrate` and compute `$effective_server` / `$effective_ca_server` / `$effective_report_server` / `$effective_ssldir`, feeding those to the template. Keeps the existing optional `$ssldir` / `$report_server` params as manual overrides. - `puppet.conf.erb`: render from the `@effective_*` vars. ## Precedence - **ssldir / report_server** (`Optional`, default `undef`): explicit `profiles::puppet::client::ssldir` / `::report_server` in hiera wins; else the migrate-derived value when `enabled`; else the legacy default (ssldir omitted -> puppet default; report_server -> effective server). - **server / ca_server** (`String`, no undef sentinel): the migrate endpoint wins while `enabled => true`; otherwise the client param (legacy default unless set explicitly). To pin a bespoke server, leave `enabled => false` for that target and set `profiles::puppet::client::server` directly. - No include cycle: migrate.pp never references client.pp; its guard uses `$facts['enc_role']`. ## Runbook - **Canary (one node)** -> `hieradata/nodes/<certname>.yaml`: `profiles::puppet::migrate::enabled: true` - **Wave (a role)** -> `hieradata/roles/<t1>/<t2>[/<t3>].yaml`: `profiles::puppet::migrate::enabled: true` - **Estate flip** -> `hieradata/common.yaml`: `profiles::puppet::migrate::enabled: true` - **Rollback** -> remove the key (or set `false`) at that layer; next run rewrites puppet.conf back to the legacy server + legacy ssldir (original CA-signed cert intact). - The k8s endpoints (`new_server` / `new_ca_server` / `new_ssldir`) are `profiles::puppet::migrate` param defaults; override them in hiera if the k8s names change. ## Verification - Render comparison: unmigrated node renders a **byte-identical** `puppet.conf` to `develop` (diff empty). Migrated render (only `enabled: true`) emits `server`/`ca_server`/`report_server` = k8s endpoints and `ssldir = /etc/puppetlabs/puppet/ssl-k8s`. - `puppet-lint` (repo args `--no-80chars-check --no-documentation-check --no-puppet_url_without_modules-check --fail-on-warnings`): clean on all 3 manifests. - `puppet parser validate`: OK. ERB syntax: OK. Repo pre-commit hooks (erb-validate, puppet-lint, puppet-validate): all Passed.
unkinben added 1 commit 2026-07-24 22:44:15 +10:00
Add profiles::puppet::migrate to repoint VM agents onto puppet-on-k8s
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
22faf00628 
Migrate VM puppet agents off the legacy VM puppetmasters and onto the new
puppet-on-kubernetes servers, per-wave via hiera and reversible without
re-enrolment. Changing server/ca_server alone is insufficient: the agent's
ssldir holds a cert signed by the OLD CA that the new CA neither trusts nor
recognises. This switches migrated nodes to a FRESH ssldir so the agent
generates a new CSR (autosigned on the k8s CA) while the old creds stay on
disk for rollback.

- Add profiles::puppet::migrate: opt-in (hiera_include) toggle that owns the
  fresh ssldir directory and documents per-node/per-role/common wiring plus
  rollback in its class header.
- Extend profiles::puppet::client with optional $ssldir and $report_server
  params (default undef); the ERB template omits both lines when unset, so
  unmigrated nodes render a byte-identical puppet.conf.
- puppet.conf stays owned solely by client.pp's template; migrate.pp adds no
  competing File resource.
unkinben added 1 commit 2026-07-24 23:49:10 +10:00
puppet::migrate: drive k8s cutover from a single enabled boolean
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
7d75dcd188 
Wire profiles::puppet::migrate into profiles::base (shipping enabled=>false)
so every node evaluates it. client.pp now includes migrate and folds
new_server/new_ca_server/new_ssldir into the effective server/ca_server/
report_server/ssldir it renders, so flipping the one
profiles::puppet::migrate::enabled key at any hiera layer (node/role/common)
repoints a target -- no per-node client param overrides.

Explicit client::ssldir/::report_server still override the migrate-derived
value; unmigrated nodes render a byte-identical puppet.conf.
benvin merged commit aebe80e73f into develop 2026-07-25 00:04:11 +10:00
benvin deleted branch benvin/puppet-k8s-migrate 2026-07-25 00:04:12 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/puppet-prod#493
Delete Branch "benvin/puppet-k8s-migrate"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?