unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
0fd01ac1a73b24e86163b8e7f158bccb9565e305
puppet-prod/hieradata/roles/infra/incus/node.yaml
T
unkinben 0fd01ac1a7 ceph: manage /etc/ceph/ceph.conf on osd and mon/mgr/mds hosts (#515) 
## Why

`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.

## How hosts are identified (osd-only vs mon/mgr/mds)

The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):

- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.

The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.

## Changes

- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.

## Live-cluster safety review (please read before merge)

This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.

- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
  - osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
  - mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
  - prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.

Rendered samples:

- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.

Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-09 00:00:20 +10:00

285 lines
6.8 KiB
YAML
Raw Blame History

---
hiera_include:
- profiles::selinux::frr
- frrouting
- incus
- zfs
- profiles::ceph::node
- profiles::ceph::mon
- profiles::ceph::mgr
- profiles::ceph::mds
- profiles::ceph::osd
- profiles::ceph::client
- profiles::ceph::dashboard
- profiles::storage::cephfsvols
- exporters::frr_exporter
# FIXME: puppet-python wants to try manage python-dev, which is required by the ceph package
python::manage_dev_package: false
profiles::packages::include:
bridge-utils: {}
cephadm: {}
ceph-common: {}
profiles::pki::vault::alt_names:
- incus.service.consul
- incus.query.consul
- "incus.service.%{facts.country}-%{facts.region}.consul"
profiles::pki::vault::ip_sans:
- "%{hiera('networking_loopback0_ip')}"
- "%{hiera('networking_loopback1_ip')}"
- "%{hiera('networking_loopback2_ip')}"
profiles::ssh::sign::principals:
- incus.service.consul
- incus.query.consul
- "incus.service.%{facts.country}-%{facts.region}.consul"
- "%{hiera('networking_loopback0_ip')}"
- "%{facts.networking.interfaces.enp2s0.ip}"
- "%{facts.networking.interfaces.enp3s0.ip}"
# configure consul service
profiles::consul::client::host_addr: "%{hiera('networking_loopback0_ip')}"
consul::services:
incus:
service_name: 'incus'
tags:
- 'incus'
- 'container'
- 'lxd'
address: "%{hiera('networking_loopback0_ip')}"
port: 8443
checks:
- id: 'incus_https_check'
name: 'incus HTTPS Check'
http: "https://%{hiera('networking_loopback0_ip')}:8443"
method: 'GET'
tls_skip_verify: true
interval: '10s'
timeout: '1s'
cephmgr:
service_name: 'cephmgr'
tags:
- 'metrics'
- 'metrics_scheme=http'
- 'metrics_job=ceph'
address: "%{hiera('networking_loopback2_ip')}"
port: 9283
checks:
- id: 'cephmgr_metrics_http_check'
name: 'cephmgr metrics HTTP Check'
http: "http://%{hiera('networking_loopback2_ip')}:9283"
method: 'GET'
tls_skip_verify: true
interval: '10s'
timeout: '1s'
profiles::consul::client::node_rules:
- resource: service
segment: incus
disposition: write
- resource: service
segment: cephmgr
disposition: write
- resource: service
segment: frr_exporter
disposition: write
# additional repos
profiles::yum::global::repos:
ceph:
ensure: present
ceph-noarch:
ensure: present
frr-extras:
ensure: present
frr-stable:
ensure: present
zfs-kmod:
ensure: present
# dns
profiles::dns::base::primary_interface: loopback0
# dashboard/haproxy
profiles::ceph::dashboard::ipaddress: "%{hiera('networking_loopback2_ip')}"
# networking
systemd::manage_networkd: true
systemd::manage_all_network_files: true
networking::interfaces:
enp2s0:
type: physical
txqueuelen: 10000
forwarding: true
enp3s0:
type: physical
mtu: 1500
txqueuelen: 10000
forwarding: true
loopback0:
type: dummy
ipaddress: "%{hiera('networking_loopback0_ip')}"
netmask: 255.255.255.255
mtu: 1500
loopback1:
type: dummy
ipaddress: "%{hiera('networking_loopback1_ip')}"
netmask: 255.255.255.255
mtu: 1500
loopback2:
type: dummy
ipaddress: "%{hiera('networking_loopback2_ip')}"
netmask: 255.255.255.255
mtu: 1500
# frrouting
exporters::frr_exporter::enable: true
frrouting::ospfd_router_id: "%{hiera('networking_loopback0_ip')}"
frrouting::ospf_preferred_source_enable: true
frrouting::ospf_preferred_source: "%{hiera('networking_loopback0_ip')}"
frrouting::ospfd_redistribute:
- connected
frrouting::ospfd_interfaces:
enp2s0:
area: 0.0.0.0
enp3s0:
area: 0.0.0.0
loopback0:
area: 0.0.0.0
loopback1:
area: 0.0.0.0
loopback2:
area: 0.0.0.0
brcom1:
area: 0.0.0.0
brdmz1:
area: 0.0.0.0
brwan1:
area: 0.0.0.0
frrouting::daemons:
ospfd: true
# add loopback interfaces to ssh list
ssh::server::options:
ListenAddress:
- "%{hiera('networking_loopback0_ip')}"
- "%{facts.networking.interfaces.enp2s0.ip}"
- "%{facts.networking.interfaces.enp3s0.ip}"
# zfs settings
zfs::manage_repo: false
zfs::zfs_arc_min: ~
zfs::zfs_arc_max: 4294967296 # 4GB
zfs::zpools:
fastpool:
ensure: present
disk: /dev/nvme1n1
ashift: 12
zfs::datasets:
fastpool:
canmount: 'off'
acltype: posix
atime: 'off'
relatime: 'off'
compression: 'zstd'
xattr: 'sa'
fastpool/data:
canmount: 'on'
mountpoint: '/data'
fastpool/data/incus:
canmount: 'on'
mountpoint: '/data/incus'
# manage incus
incus::init: true
incus::bridge: br10
incus::server_port: 8443
incus::server_addr: "%{hiera('networking_loopback0_ip')}"
# add sysadmin to incus-admin group
profiles::accounts::sysadmin::extra_groups:
- incus-admin
# manage cephfs mounts
profiles::ceph::client::manage_ceph_conf: true
# mon/mgr/mds host: render the [mds] + [mds.*] sections
profiles::ceph::client::render_mds_config: true
profiles::ceph::client::manage_ceph_package: false
profiles::ceph::client::manage_ceph_paths: false
profiles::ceph::client::fsid: 'de96a98f-3d23-465a-a899-86d3d67edab8'
profiles::ceph::client::mons:
- 198.18.23.9
- 198.18.23.10
- 198.18.23.11
- 198.18.23.12
- 198.18.23.13
profiles::ceph::client::keyrings:
media:
key: "%{hiera('ceph::key::media')}"
apps:
key: "%{hiera('ceph::key::apps')}"
profiles::storage::cephfsvols::volumes:
cephfsvol_media:
mount: "/shared/media"
keyring: "/etc/ceph/ceph.client.media.keyring"
cephfs_name: "media"
cephfs_fs: "mediafs"
cephfs_mon: "%{alias('profiles::ceph::client::mons')}"
require: "Profiles::Ceph::Keyring[media]"
cephfsvol_apps:
mount: "/shared/apps"
keyring: "/etc/ceph/ceph.client.apps.keyring"
cephfs_name: "apps"
cephfs_fs: "appfs"
cephfs_mon: "%{alias('profiles::ceph::client::mons')}"
require: "Profiles::Ceph::Keyring[apps]"
# sysctl recommendations
sysctl::base::values:
fs.aio-max-nr:
value: '524288'
fs.inotify.max_queued_events:
value: '1048576'
fs.inotify.max_user_instances:
value: '1048576'
fs.inotify.max_user_watches:
value: '1048576'
kernel.dmesg_restrict:
value: '1'
kernel.keys.maxbytes:
value: '2000000'
kernel.keys.maxkeys:
value: '2000'
net.core.bpf_jit_limit:
value: '1000000000'
net.ipv4.neigh.default.gc_thresh3:
value: '8192'
net.ipv6.neigh.default.gc_thresh3:
value: '8192'
vm.max_map_count:
value: '262144'
net.ipv4.conf.all.forwarding:
value: '1'
net.ipv6.conf.all.forwarding:
value: '1'
net.ipv4.tcp_l3mdev_accept:
value: '0'
net.ipv4.conf.default.rp_filter:
value: '0'
net.ipv4.conf.all.rp_filter:
value: '0'
# limits.d recommendations
limits::entries:
'*/nofile':
both: 1048576
'root/nofile':
both: 1048576
'*/memlock':
both: unlimited
'root/memlock':
both: unlimited
Reference in New Issue View Git Blame Copy Permalink