unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
979c188c3437099863e0b53c57942ef7ed6523a3
puppet-prod/hieradata/roles/infra/k8s.yaml
T
unkinben 0fd01ac1a7 ceph: manage /etc/ceph/ceph.conf on osd and mon/mgr/mds hosts (#515) 
## Why

`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.

## How hosts are identified (osd-only vs mon/mgr/mds)

The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):

- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.

The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.

## Changes

- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.

## Live-cluster safety review (please read before merge)

This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.

- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
  - osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
  - mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
  - prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.

Rendered samples:

- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.

Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-09 00:00:20 +10:00

209 lines
5.6 KiB
YAML
Raw Blame History

---
hiera_include:
- profiles::selinux::setenforce
- profiles::ceph::node
- profiles::ceph::osd
- profiles::ceph::client
- exporters::frr_exporter
- frrouting
- rke2
# manage rke2
rke2::bootstrap_node: prodnxsr0001.main.unkin.net
rke2::join_url: https://join-k8s.service.consul:9345
# pull the airgap image bundle via artifactapi (host-reachable pre-CNI), not github directly
rke2::container_archive_source: 'https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/rancher/rke2/releases/download'
rke2::manage_registries: true
rke2::registries:
docker.io:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "dockerhub/$1"
disable-default-registry-endpoint: true
ghcr.io:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "ghcr/$1"
disable-default-registry-endpoint: true
quay.io:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "quay/$1"
disable-default-registry-endpoint: true
registry.k8s.io:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "k8s-registry/$1"
disable-default-registry-endpoint: true
registry.gitlab.com:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "gitlab/$1"
disable-default-registry-endpoint: true
docker.elastic.co:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "elastic/$1"
disable-default-registry-endpoint: true
gcr.io:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "gcr/$1"
disable-default-registry-endpoint: true
docker.litellm.ai:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "litellm/$1"
disable-default-registry-endpoint: true
public.ecr.aws:
endpoint:
- "https://artifactapi.k8s.syd1.au.unkin.net"
rewrite:
"^(.*)$": "ecr-public/$1"
disable-default-registry-endpoint: true
rke2::config_hash:
bind-address: "%{hiera('networking_loopback0_ip')}"
node-ip: "%{hiera('networking_loopback0_ip')}"
node-external-ip: "%{hiera('networking_loopback0_ip')}"
write-kubeconfig-mode: 644
kubelet-arg:
- '--node-status-update-frequency=4s'
- '--max-pods=100'
node-label:
- "region=%{facts.region}"
- "country=%{facts.country}"
- "asset=%{facts.dmi.product.serial_number}"
- "zone=%{zone}"
# FIXME: puppet-python wants to try manage python-dev, which is required by the ceph package
python::manage_dev_package: false
profiles::packages::include:
bridge-utils: {}
cephadm: {}
profiles::selinux::setenforce::mode: disabled
profiles::ceph::client::manage_ceph_conf: true
profiles::ceph::client::manage_ceph_package: false
profiles::ceph::client::manage_ceph_paths: false
profiles::ceph::client::fsid: 'de96a98f-3d23-465a-a899-86d3d67edab8'
profiles::ceph::client::mons:
- 198.18.23.9
- 198.18.23.10
- 198.18.23.11
- 198.18.23.12
- 198.18.23.13
# additional repos
profiles::yum::global::repos:
ceph:
ensure: present
ceph-noarch:
ensure: present
frr-extras:
ensure: present
frr-stable:
ensure: present
rancher-rke2-common-latest:
ensure: present
rancher-rke2-1-33-latest:
ensure: present
# dns
profiles::dns::base::primary_interface: loopback0
# networking
systemd::manage_networkd: true
systemd::manage_all_network_files: true
networking::interfaces:
"%{hiera('networking_1000_iface')}":
type: physical
ipaddress: "%{hiera('networking_1000_ip')}"
gateway: 198.18.15.254
txqueuelen: 10000
forwarding: true
"%{hiera('networking_2500_iface')}":
type: physical
ipaddress: "%{hiera('networking_2500_ip')}"
mtu: 1500
txqueuelen: 10000
forwarding: true
loopback0:
type: dummy
ipaddress: "%{hiera('networking_loopback0_ip')}"
netmask: 255.255.255.255
mtu: 1500
loopback1:
type: dummy
ipaddress: "%{hiera('networking_loopback1_ip')}"
netmask: 255.255.255.255
mtu: 1500
loopback2:
type: dummy
ipaddress: "%{hiera('networking_loopback2_ip')}"
netmask: 255.255.255.255
mtu: 1500
# configure consul service
profiles::consul::client::host_addr: "%{hiera('networking_loopback0_ip')}"
profiles::consul::client::node_rules:
- resource: service
segment: frr_exporter
disposition: write
# frrouting
exporters::frr_exporter::enable: true
frrouting::ospfd_router_id: "%{hiera('networking_loopback0_ip')}"
frrouting::ospf_preferred_source_enable: true
frrouting::ospf_preferred_source: "%{hiera('networking_loopback0_ip')}"
frrouting::ospfd_redistribute:
- connected
frrouting::ospfd_interfaces:
"%{hiera('networking_1000_iface')}":
area: 0.0.0.0
"%{hiera('networking_2500_iface')}":
area: 0.0.0.0
loopback0:
area: 0.0.0.0
loopback1:
area: 0.0.0.0
loopback2:
area: 0.0.0.0
frrouting::daemons:
ospfd: true
frrouting::ospf_exclude_k8s_enable: true
frrouting::k8s_cluster_cidr: '10.42.0.0/16' # RKE2 cluster-cidr (pods)
frrouting::k8s_service_cidr: '10.43.0.0/16' # RKE2 service-cidr
# sysctl recommendations
sysctl::base::values:
net.ipv4.conf.default.rp_filter:
value: '0'
net.ipv4.conf.all.rp_filter:
value: '0'
fs.inotify.max_user_watches:
value: '524288'
fs.inotify.max_user_instances:
value: '512'
# add loopback interfaces to ssh list
ssh::server::options:
ListenAddress:
- "%{hiera('networking_loopback0_ip')}"
- "%{hiera('networking_1000_ip')}"
- "%{hiera('networking_2500_ip')}"
profiles::ssh::sign::principals:
- "%{hiera('networking_loopback0_ip')}"
- "%{hiera('networking_1000_ip')}"
- "%{hiera('networking_2500_ip')}"
Reference in New Issue View Git Blame Copy Permalink