## Why
`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.
## How hosts are identified (osd-only vs mon/mgr/mds)
The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):
- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.
The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.
## Changes
- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.
## Live-cluster safety review (please read before merge)
This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.
- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
- osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
- mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
- prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.
Rendered samples:
- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.
Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- only install a base config
- wait for 3 masters before deploying helm charts
- remove cluster-domain
- manage nginx ingres via rke2 helmconfig
Reviewed-on: #403
- cattle-system namespace is created earlier than helm
- leave namespaces.yaml to manage cattle-system namespace (required
before installing helm/rancher)
Reviewed-on: #399
This change will install rancher, purelb and cert-manager, then
configure a dmz and common ip pool to be used by loadbalancers. The
nginx ingres controller is configured to use 198.18.200.0 (common) and
announce the ip from all nodes so that it becomes an anycast ip in ospf.
- manage the install of rancher, purelb and cert-manager
- add rancher ingress routes
- add nginx externalip/loadBalancer
Reviewed-on: #395
- enable changing the source address for learned ospf routes
- this enables the loopback0 interface to be used as a default src address
- ensure k8s nodes use loopback0 as default src
- ensure incus nodes use loopback0 as default src
Reviewed-on: #388
- enable enough ceph/frr to join to cephfs
- notify sshd when restarting the network
- update ssh principals to include all ssh interfaces
Reviewed-on: #362
unkinben