Add profiles::puppet::migrate: repoint VM agents onto puppet-on-k8s (#493)

## Why

Migrate the VM puppet agents off the legacy VM puppetmasters onto the new puppet-on-kubernetes servers, one wave at a time and reversible without re-enrolment.

Changing `server`/`ca_server` alone is insufficient: each agent's ssldir holds a client certificate signed by the **old** CA, which the new k8s CA neither trusts nor recognises. Migrated nodes switch to a **fresh ssldir**, so the agent generates a new key + CSR on its next run (autosigned immediately by the k8s CA, which runs `autosign = true`), pulls its catalog from the new compilers, and leaves the old CA-signed creds intact on disk for rollback.

The reconfiguration catalog is compiled by the **old** server (the node still points there when it applies the change). `profiles::puppet::client` rewrites `puppet.conf` with the new endpoints + ssldir and notifies `Service['puppet']`, so the very next agent run lands on k8s.

**Single-boolean design (reworked):** cutover is now driven by one hiera key. `profiles::puppet::migrate` ships wired into `profiles::base` with `enabled => false`, so every node evaluates it but nothing changes until the boolean is flipped. `profiles::puppet::client` includes migrate and folds the migrate endpoints into the effective values it renders. Flip `profiles::puppet::migrate::enabled: true` at any hiera layer (node/role/common) to migrate a target -- no other keys required.

## Changes

- `profiles::base` (`site/profiles/manifests/base.pp`): `include profiles::puppet::migrate` alongside the other puppet profiles (ships disabled).
- `profiles::puppet::migrate` (`site/profiles/manifests/puppet/migrate.pp`): unchanged params (`enabled=false`, `new_server`, `new_ca_server`, `new_ssldir`); when enabled and the node is not a puppetmaster it owns the fresh ssldir directory. Class-header runbook rewritten for the single-boolean flow. No longer opt-in via `hiera_include`.
- `profiles::puppet::client` (`site/profiles/manifests/puppet/client.pp`): `include profiles::puppet::migrate` and compute `$effective_server` / `$effective_ca_server` / `$effective_report_server` / `$effective_ssldir`, feeding those to the template. Keeps the existing optional `$ssldir` / `$report_server` params as manual overrides.
- `puppet.conf.erb`: render from the `@effective_*` vars.

## Precedence

- **ssldir / report_server** (`Optional`, default `undef`): explicit `profiles::puppet::client::ssldir` / `::report_server` in hiera wins; else the migrate-derived value when `enabled`; else the legacy default (ssldir omitted -> puppet default; report_server -> effective server).
- **server / ca_server** (`String`, no undef sentinel): the migrate endpoint wins while `enabled => true`; otherwise the client param (legacy default unless set explicitly). To pin a bespoke server, leave `enabled => false` for that target and set `profiles::puppet::client::server` directly.
- No include cycle: migrate.pp never references client.pp; its guard uses `$facts['enc_role']`.

## Runbook

- **Canary (one node)** -> `hieradata/nodes/<certname>.yaml`: `profiles::puppet::migrate::enabled: true`
- **Wave (a role)** -> `hieradata/roles/<t1>/<t2>[/<t3>].yaml`: `profiles::puppet::migrate::enabled: true`
- **Estate flip** -> `hieradata/common.yaml`: `profiles::puppet::migrate::enabled: true`
- **Rollback** -> remove the key (or set `false`) at that layer; next run rewrites puppet.conf back to the legacy server + legacy ssldir (original CA-signed cert intact).
- The k8s endpoints (`new_server` / `new_ca_server` / `new_ssldir`) are `profiles::puppet::migrate` param defaults; override them in hiera if the k8s names change.

## Verification

- Render comparison: unmigrated node renders a **byte-identical** `puppet.conf` to `develop` (diff empty). Migrated render (only `enabled: true`) emits `server`/`ca_server`/`report_server` = k8s endpoints and `ssldir = /etc/puppetlabs/puppet/ssl-k8s`.
- `puppet-lint` (repo args `--no-80chars-check --no-documentation-check --no-puppet_url_without_modules-check --fail-on-warnings`): clean on all 3 manifests.
- `puppet parser validate`: OK. ERB syntax: OK. Repo pre-commit hooks (erb-validate, puppet-lint, puppet-validate): all Passed.

Reviewed-on: #493
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>

site/profiles/manifests/base.pp

@@ -12,6 +12,9 @@ class profiles::base () {
     # manage the puppet agent
     include profiles::puppet::agent
     include profiles::puppet::client
+    # k8s migration switch; ships disabled, flip its ::enabled boolean in
+    # hiera (node/role/common) to repoint a target onto puppet-on-k8s.
+    include profiles::puppet::migrate
 
     # include the base profiles
     include profiles::base::repos

site/profiles/manifests/puppet/client.pp

@@ -4,19 +4,64 @@
 #
 # site/profile/manifests/puppet/client.pp
 class profiles::puppet::client (
-  Array  $dns_alt_names       = [$trusted['certname']],
+  Array  $dns_alt_names                  = [$trusted['certname']],
-  String $server              = 'puppetmaster',
+  String $server                         = 'puppetmaster',
-  String $ca_server           = 'puppetca',
+  String $ca_server                      = 'puppetca',
-  String $environment         = 'develop',
+  Optional[String] $report_server        = undef,
-  Integer $runinterval        = 1800,
+  String $environment                    = 'develop',
-  Integer $runtimeout         = 3600,
+  Integer $runinterval                   = 1800,
-  Boolean $show_diff          = true,
+  Integer $runtimeout                    = 3600,
-  Boolean $usecacheonfailure  = false,
+  Boolean $show_diff                     = true,
-  Integer $facts_soft_limit   = 4096,
+  Boolean $usecacheonfailure             = false,
-  Boolean $splay              = true,
+  Integer $facts_soft_limit              = 4096,
-  Integer $splaylimit         = 600,
+  Boolean $splay                         = true,
+  Integer $splaylimit                    = 600,
+  Optional[Stdlib::Absolutepath] $ssldir = undef,
 ) {
 
+  # Pull in the k8s-migration switch. When
+  # profiles::puppet::migrate::enabled is true at ANY hiera layer
+  # (node/role/common) the effective endpoints below flip to the k8s
+  # servers and a fresh ssldir -- no other hiera keys required. This class
+  # never includes client.pp, so there is no include cycle.
+  include profiles::puppet::migrate
+
+  # Effective values fed to the template. Precedence differs by param type:
+  #
+  #   ssldir / report_server (Optional, default undef): an explicit client
+  #     param wins; else the migrate-derived value when enabled; else the
+  #     legacy default (undef -> template omits ssldir; report_server -> server).
+  #
+  #   server / ca_server (String, no undef sentinel): the migrate value wins
+  #     when migrate::enabled, otherwise the client param (which is the legacy
+  #     default unless an operator set it explicitly in hiera). To pin a
+  #     bespoke server while enabled, leave enabled => false for that target
+  #     and set profiles::puppet::client::server directly.
+  $migrate_on = $profiles::puppet::migrate::enabled
+
+  $effective_server = $migrate_on ? {
+    true    => $profiles::puppet::migrate::new_server,
+    default => $server,
+  }
+  $effective_ca_server = $migrate_on ? {
+    true    => $profiles::puppet::migrate::new_ca_server,
+    default => $ca_server,
+  }
+  # report_server: explicit param wins; else follow the effective server.
+  $effective_report_server = $report_server ? {
+    undef   => $effective_server,
+    default => $report_server,
+  }
+  # ssldir: explicit param wins; else the fresh k8s ssldir when migrating;
+  # else undef (template omits the key, i.e. puppet's built-in default).
+  $effective_ssldir = $ssldir ? {
+    undef   => $migrate_on ? {
+      true    => $profiles::puppet::migrate::new_ssldir,
+      default => undef,
+    },
+    default => $ssldir,
+  }
+
   # dont manage puppet.conf if this is a puppetmaster
   if $facts['enc_role'] != 'roles::infra::puppet::master' {
 

site/profiles/manifests/puppet/migrate.pp

@@ -0,0 +1,87 @@
+# Class: profiles::puppet::migrate
+#
+# Repoints a VM puppet agent from the legacy VM puppetmasters onto the new
+# puppet-on-kubernetes servers.
+#
+# The agent's existing ssldir holds a client certificate signed by the OLD CA;
+# the new k8s CA will neither trust that cert nor recognise the agent. Rather
+# than revoke/clean the old creds (which would burn the rollback path), this
+# class switches the agent to a FRESH ssldir. On the next run the agent finds
+# an empty ssldir, generates a new key + CSR, submits it to the new CA
+# (autosign = true on the k8s side signs it immediately), and then pulls its
+# catalog from the new compilers. The OLD ssldir is left untouched on disk, so
+# reverting the hiera below restores the node to the legacy masters with no
+# re-enrolment.
+#
+# The catalog that performs the switch is compiled by the OLD server (the node
+# still points at it when it applies this change). profiles::puppet::client
+# rewrites puppet.conf with the new server / ca_server / report_server / ssldir
+# and notifies Service['puppet'], so the very next agent run lands on k8s.
+#
+# NOTE: this is deliberately hiera-driven per node/role so waves can be rolled
+# out and rolled back one target at a time. The class only manages the fresh
+# ssldir; the actual puppet.conf changes ride on profiles::puppet::client's
+# existing template. client.pp `include`s this class and, when enabled, folds
+# new_server / new_ca_server / new_ssldir into the effective server /
+# ca_server / report_server / ssldir it renders. Do NOT add a second File
+# resource for puppet.conf here -- client.pp already owns it.
+#
+# == Wiring (single boolean)
+#
+# This class ships wired into profiles::base with enabled => false, so every
+# node evaluates it but nothing changes until the boolean is flipped. Because
+# client.pp includes it and reads its params, setting the ONE key below at any
+# hiera layer repoints a target -- no client.pp param overrides needed.
+#
+# The endpoints (new_server / new_ca_server / new_ssldir) are param defaults
+# on this class; override them in hiera too if the k8s names ever change.
+#
+# --- Canary: one node -> hieradata/nodes/<certname>.yaml -------------------
+#   profiles::puppet::migrate::enabled: true
+# ---------------------------------------------------------------------------
+#
+# --- Wave: a whole role -> hieradata/roles/<t1>/<t2>[/<t3>].yaml -----------
+#   profiles::puppet::migrate::enabled: true
+# ---------------------------------------------------------------------------
+#
+# --- Estate flip -> hieradata/common.yaml (retires the legacy masters) -----
+#   profiles::puppet::migrate::enabled: true
+# ---------------------------------------------------------------------------
+#
+# Manual override: profiles::puppet::client::ssldir and ::report_server are
+# Optional and, when set explicitly in hiera, win over the migrate-derived
+# value. server / ca_server have no undef sentinel, so while enabled => true
+# the migrate endpoints win; to point a target at a bespoke server leave
+# enabled => false for it and set profiles::puppet::client::server directly.
+#
+# == Rollback (per target)
+#
+# Remove the profiles::puppet::migrate::enabled key (or set it back to false)
+# at the node/role/common layer. The next run rewrites puppet.conf back to the
+# legacy server + legacy ssldir, which still contains the original CA-signed
+# cert, and the node is back on the VM masters.
+#
+# site/profiles/manifests/puppet/migrate.pp
+class profiles::puppet::migrate (
+  Boolean $enabled                          = false,
+  String $new_server                        = 'puppet.k8s.syd1.au.unkin.net',
+  String $new_ca_server                     = 'puppetca.k8s.syd1.au.unkin.net',
+  Stdlib::Absolutepath $new_ssldir          = '/etc/puppetlabs/puppet/ssl-k8s',
+) {
+
+  # never touch the puppet masters themselves (same guard as client.pp)
+  if $enabled and $facts['enc_role'] != 'roles::infra::puppet::master' {
+
+    # Ensure the fresh ssldir exists with agent-correct ownership/perms. The
+    # puppet agent will create it on demand, but pre-creating it keeps the very
+    # first post-switch run from racing directory creation and makes the
+    # migration state auditable. Contents (keys, certs, crl) are populated by
+    # the agent against the new CA -- we only own the directory itself.
+    file { $new_ssldir:
+      ensure => 'directory',
+      owner  => 'root',
+      group  => 'root',
+      mode   => '0771',
+    }
+  }
+}

site/profiles/templates/puppet/client/puppet.conf.erb

@@ -1,12 +1,15 @@
 [main]
 dns_alt_names = <%= @dns_alt_names_string %>
+<% unless @effective_ssldir.nil? -%>
+ssldir = <%= @effective_ssldir %>
+<% end -%>
 
 [agent]
-server = <%= @server %>
+server = <%= @effective_server %>
-ca_server = <%= @ca_server %>
+ca_server = <%= @effective_ca_server %>
 environment = <%= @environment %>
 report = true
-report_server = <%= @server %>
+report_server = <%= @effective_report_server.nil? ? @effective_server : @effective_report_server %>
 runinterval = <%= @runinterval %>
 runtimeout = <%= @runtimeout %>
 show_diff = <%= @show_diff %>