ceph: manage /etc/ceph/ceph.conf on osd and mon/mgr/mds hosts (#515)

## Why

`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.

## How hosts are identified (osd-only vs mon/mgr/mds)

The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):

- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.

The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.

## Changes

- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.

## Live-cluster safety review (please read before merge)

This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.

- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
  - osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
  - mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
  - prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.

Rendered samples:

- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.

Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>

hieradata/common.yaml

@@ -409,6 +409,45 @@ profiles::ceph::client::mons:
   - 10.18.15.2
   - 10.18.15.3
 
+# de96a98f cluster topology (prodnxsr0001-0019) - single source of truth for
+# /etc/ceph/ceph.conf rendered by profiles::ceph::client on the k8s (osd) and
+# incus (mon/mgr/mds) roles. fsid/mons are overridden per-role in the role hiera.
+# public_network is the /32 of every ceph host's ceph-public loopback.
+profiles::ceph::client::cluster_public_ips:
+  - 198.18.23.1
+  - 198.18.23.2
+  - 198.18.23.3
+  - 198.18.23.4
+  - 198.18.23.5
+  - 198.18.23.6
+  - 198.18.23.7
+  - 198.18.23.8
+  - 198.18.23.9
+  - 198.18.23.10
+  - 198.18.23.11
+  - 198.18.23.12
+  - 198.18.23.13
+  - 198.18.23.14
+  - 198.18.23.15
+  - 198.18.23.16
+  - 198.18.23.17
+  - 198.18.23.18
+  - 198.18.23.19
+profiles::ceph::client::mon_initial_members:
+  - prodnxsr0009
+  - prodnxsr0010
+  - prodnxsr0011
+  - prodnxsr0012
+  - prodnxsr0013
+# two mds daemon instances per mon/mgr/mds host (rendered only where
+# render_mds_config is true, i.e. the incus node role).
+profiles::ceph::client::mds_instances:
+  prodnxsr0009: 2
+  prodnxsr0010: 2
+  prodnxsr0011: 2
+  prodnxsr0012: 2
+  prodnxsr0013: 2
+
 profiles::ceph::conf::config:
   global:
     auth_client_required: 'cephx'

hieradata/roles/infra/incus/node.yaml

@@ -203,7 +203,9 @@ profiles::accounts::sysadmin::extra_groups:
   - incus-admin
 
 # manage cephfs mounts
-profiles::ceph::client::manage_ceph_conf: false
+profiles::ceph::client::manage_ceph_conf: true
+# mon/mgr/mds host: render the [mds] + [mds.*] sections
+profiles::ceph::client::render_mds_config: true
 profiles::ceph::client::manage_ceph_package: false
 profiles::ceph::client::manage_ceph_paths: false
 profiles::ceph::client::fsid: 'de96a98f-3d23-465a-a899-86d3d67edab8'

hieradata/roles/infra/k8s.yaml

@@ -92,7 +92,7 @@ profiles::packages::include:
 
 profiles::selinux::setenforce::mode: disabled
 
-profiles::ceph::client::manage_ceph_conf: false
+profiles::ceph::client::manage_ceph_conf: true
 profiles::ceph::client::manage_ceph_package: false
 profiles::ceph::client::manage_ceph_paths: false
 profiles::ceph::client::fsid: 'de96a98f-3d23-465a-a899-86d3d67edab8'

hieradata/roles/infra/k8s/node.yaml

@@ -33,7 +33,7 @@ profiles::packages::include:
 
 profiles::selinux::setenforce::mode: disabled
 
-profiles::ceph::client::manage_ceph_conf: false
+profiles::ceph::client::manage_ceph_conf: true
 profiles::ceph::client::manage_ceph_package: false
 profiles::ceph::client::manage_ceph_paths: false
 profiles::ceph::client::fsid: 'de96a98f-3d23-465a-a899-86d3d67edab8'

site/profiles/manifests/ceph/client.pp

@@ -2,6 +2,9 @@
 class profiles::ceph::client (
   String $fsid,
   Array[Stdlib::Host] $mons,
+  # cluster topology (single source of truth: hieradata/common.yaml)
+  Array[Stdlib::Host] $cluster_public_ips,
+  Array[Stdlib::Host] $mon_initial_members,
   Stdlib::Absolutepath $config_file = '/etc/ceph/ceph.conf',
   Boolean $manage_ceph_conf = true,
   Boolean $manage_ceph_package = true,
@@ -10,6 +13,24 @@ class profiles::ceph::client (
   String $group = 'ceph',
   Stdlib::Filemode $mode = '0644',
   Hash $keyrings = {},
+  # [global] tunables (defaults match the live hand-maintained ceph.conf)
+  String $auth_client_required = 'cephx',
+  String $auth_cluster_required = 'cephx',
+  String $auth_service_required = 'cephx',
+  Boolean $mon_allow_pool_delete = true,
+  Boolean $ms_bind_ipv4 = true,
+  Boolean $ms_bind_ipv6 = false,
+  Integer $osd_crush_chooseleaf_type = 1,
+  Integer $osd_pool_default_min_size = 2,
+  Integer $osd_pool_default_size = 3,
+  Integer $osd_pool_default_pg_num = 128,
+  # mds config sections; only rendered on mon/mgr/mds hosts (render_mds_config)
+  Boolean $render_mds_config = false,
+  Hash[String, Integer] $mds_instances = {},
+  Hash $mds_common = {
+    'keyring'            => '/var/lib/ceph/mds/ceph-$id/keyring',
+    'mds_standby_replay' => true,
+  },
 ) {
 
   # dont run this on proxmox nodes
@@ -22,6 +43,13 @@ class profiles::ceph::client (
       }
     }
 
+    # only depend on the package when this class manages it; on the ceph
+    # hosts the package is delivered by cephadm / profiles::packages instead.
+    $config_require = $manage_ceph_package ? {
+      true    => Package['ceph-common'],
+      default => undef,
+    }
+
     # manage the ceph directory
     if $manage_ceph_paths {
       file { '/etc/ceph':
@@ -29,11 +57,11 @@ class profiles::ceph::client (
         owner   => $owner,
         group   => $group,
         mode    => $mode,
-        require => Package['ceph-common'],
+        require => $config_require,
       }
     }
 
-    # create a basic client config
+    # render /etc/ceph/ceph.conf from cluster topology in hiera
     if $manage_ceph_conf {
       file { $config_file:
         ensure  => file,
@@ -41,7 +69,7 @@ class profiles::ceph::client (
         group   => $group,
         mode    => $mode,
         content => template('profiles/ceph/client.conf.erb'),
-        require => Package['ceph-common'],
+        require => $config_require,
       }
     }
 

site/profiles/templates/ceph/client.conf.erb

@@ -1,3 +1,28 @@
 [global]
-	fsid = <%= @fsid %>
-	mon_host = <%= @mons.join(' ') %>
+  auth_client_required = <%= @auth_client_required %>
+  auth_cluster_required = <%= @auth_cluster_required %>
+  auth_service_required = <%= @auth_service_required %>
+  fsid = <%= @fsid %>
+  mon_allow_pool_delete = <%= @mon_allow_pool_delete %>
+  mon_initial_members = <%= @mon_initial_members.join(',') %>
+  mon_host = <%= @mons.join(',') %>
+  ms_bind_ipv4 = <%= @ms_bind_ipv4 %>
+  ms_bind_ipv6 = <%= @ms_bind_ipv6 %>
+  osd_crush_chooseleaf_type = <%= @osd_crush_chooseleaf_type %>
+  osd_pool_default_min_size = <%= @osd_pool_default_min_size %>
+  osd_pool_default_size = <%= @osd_pool_default_size %>
+  osd_pool_default_pg_num = <%= @osd_pool_default_pg_num %>
+  public_network = <%= @cluster_public_ips.map { |ip| "#{ip}/32" }.join(',') %>
+<% if @render_mds_config -%>
+
+[mds]
+  keyring = <%= @mds_common['keyring'] %>
+  mds_standby_replay = <%= @mds_common['mds_standby_replay'] %>
+<% @mds_instances.sort.each do |host, count| -%>
+<% (1..count).each do |instance| -%>
+
+[mds.<%= host %>-<%= instance %>]
+  host = <%= host %>
+<% end -%>
+<% end -%>
+<% end -%>