unkinben
0fd01ac1a7
## Why `/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`. ## How hosts are identified (osd-only vs mon/mgr/mds) The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera): - **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only. - **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`. The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here. ## Changes - Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host). - Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections. - Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`. - Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile). - Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections. ## Live-cluster safety review (please read before merge) This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below. - **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts: - osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`. - mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`. - prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant. - **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match. - RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`. Rendered samples: - osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else. - mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated. Reviewed-on: #515 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
81 lines
2.4 KiB
Puppet
81 lines
2.4 KiB
Puppet
# profiles::ceph::client
|
|
class profiles::ceph::client (
|
|
String $fsid,
|
|
Array[Stdlib::Host] $mons,
|
|
# cluster topology (single source of truth: hieradata/common.yaml)
|
|
Array[Stdlib::Host] $cluster_public_ips,
|
|
Array[Stdlib::Host] $mon_initial_members,
|
|
Stdlib::Absolutepath $config_file = '/etc/ceph/ceph.conf',
|
|
Boolean $manage_ceph_conf = true,
|
|
Boolean $manage_ceph_package = true,
|
|
Boolean $manage_ceph_paths = true,
|
|
String $owner = 'ceph',
|
|
String $group = 'ceph',
|
|
Stdlib::Filemode $mode = '0644',
|
|
Hash $keyrings = {},
|
|
# [global] tunables (defaults match the live hand-maintained ceph.conf)
|
|
String $auth_client_required = 'cephx',
|
|
String $auth_cluster_required = 'cephx',
|
|
String $auth_service_required = 'cephx',
|
|
Boolean $mon_allow_pool_delete = true,
|
|
Boolean $ms_bind_ipv4 = true,
|
|
Boolean $ms_bind_ipv6 = false,
|
|
Integer $osd_crush_chooseleaf_type = 1,
|
|
Integer $osd_pool_default_min_size = 2,
|
|
Integer $osd_pool_default_size = 3,
|
|
Integer $osd_pool_default_pg_num = 128,
|
|
# mds config sections; only rendered on mon/mgr/mds hosts (render_mds_config)
|
|
Boolean $render_mds_config = false,
|
|
Hash[String, Integer] $mds_instances = {},
|
|
Hash $mds_common = {
|
|
'keyring' => '/var/lib/ceph/mds/ceph-$id/keyring',
|
|
'mds_standby_replay' => true,
|
|
},
|
|
) {
|
|
|
|
# dont run this on proxmox nodes
|
|
if $facts['enc_role'] != 'roles::infra::proxmox::node' {
|
|
|
|
# install the ceph client package
|
|
if $manage_ceph_package {
|
|
package { 'ceph-common':
|
|
ensure => installed,
|
|
}
|
|
}
|
|
|
|
# only depend on the package when this class manages it; on the ceph
|
|
# hosts the package is delivered by cephadm / profiles::packages instead.
|
|
$config_require = $manage_ceph_package ? {
|
|
true => Package['ceph-common'],
|
|
default => undef,
|
|
}
|
|
|
|
# manage the ceph directory
|
|
if $manage_ceph_paths {
|
|
file { '/etc/ceph':
|
|
ensure => directory,
|
|
owner => $owner,
|
|
group => $group,
|
|
mode => $mode,
|
|
require => $config_require,
|
|
}
|
|
}
|
|
|
|
# render /etc/ceph/ceph.conf from cluster topology in hiera
|
|
if $manage_ceph_conf {
|
|
file { $config_file:
|
|
ensure => file,
|
|
owner => $owner,
|
|
group => $group,
|
|
mode => $mode,
|
|
content => template('profiles/ceph/client.conf.erb'),
|
|
require => $config_require,
|
|
}
|
|
}
|
|
|
|
# manage ceph keyrings
|
|
create_resources('profiles::ceph::keyring', $keyrings)
|
|
}
|
|
|
|
}
|