unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
7e1f2c336ca4ad06c256ef5018fdaa541f8fe9cd
puppet-prod/hieradata/common.yaml
T
unkinben 0fd01ac1a7 ceph: manage /etc/ceph/ceph.conf on osd and mon/mgr/mds hosts (#515) 
## Why

`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.

## How hosts are identified (osd-only vs mon/mgr/mds)

The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):

- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.

The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.

## Changes

- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.

## Live-cluster safety review (please read before merge)

This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.

- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
  - osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
  - mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
  - prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.

Rendered samples:

- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.

Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-09 00:00:20 +10:00

531 lines
15 KiB
YAML
Raw Blame History

---
lookup_options:
hiera_classes:
merge:
strategy: deep
profiles::packages::include:
merge:
strategy: deep
profiles::packages::exclude:
merge:
strategy: deep
profiles::pki::vault::alt_names:
merge:
strategy: deep
profiles::pki::vault::ip_sans:
merge:
strategy: deep
profiles::yum::global::managed_repos:
merge:
strategy: deep
profiles::haproxy::server::defaults:
merge:
strategy: deep
profiles::haproxy::server::globals:
merge:
strategy: deep
profiles::haproxy::server::frontends:
merge:
strategy: deep
profiles::haproxy::server::backends:
merge:
strategy: deep
profiles::haproxy::server::mappings:
merge:
strategy: deep
profiles::haproxy::server::listeners:
merge:
strategy: deep
profiles::accounts::root::sshkeys:
merge:
strategy: deep
profiles::accounts::sysadmin::sshkeys:
merge:
strategy: deep
haproxy::backend:
merge:
strategy: deep
sudo::configs:
merge:
strategy: deep
profiles::base::groups::local:
merge:
strategy: deep
profiles::dns::resolver::zones:
merge:
strategy: deep
profiles::dns::resolver::acls:
merge:
strategy: deep
profiles::dns::resolver::views:
merge:
strategy: deep
profiles::dns::resolver::keys:
merge:
strategy: deep
profiles::dns::master::zones:
merge:
strategy: deep
profiles::dns::master::acls:
merge:
strategy: deep
profiles::dns::master::views:
merge:
strategy: deep
profiles::dns::master::keys:
merge:
strategy: deep
consul::services:
merge:
strategy: deep
consul::watch:
merge:
strategy: deep
consul::check:
merge:
strategy: deep
profiles::consul::client::node_rules:
merge:
strategy: deep
profiles::consul::prepared_query::rules:
merge:
strategy: deep
profiles::puppet::server::dns_alt_names:
merge:
strategy: deep
profiles::puppet::client::dns_alt_names:
merge:
strategy: deep
profiles::base::hosts::additional_hosts:
merge:
strategy: deep
postgresql_config_entries:
merge:
strategy: deep
profiles::yum::global::repos:
merge:
strategy: deep
profiles::nginx::simpleproxy::nginx_aliases:
merge:
strategy: deep
networking::interfaces:
merge:
strategy: deep
networking::interface_defaults:
merge:
strategy: deep
networking::routes:
merge:
strategy: deep
networking::route_defaults:
merge:
strategy: deep
ssh::server::options:
merge:
strategy: deep
mysql::db:
merge:
strategy: deep
profiles::ceph::client::keyrings:
merge:
strategy: deep
profiles::ceph::conf::config:
merge:
strategy: deep
profiles::nginx::simpleproxy::locations:
merge:
strategy: deep
certbot::client::domains:
merge:
strategy: deep
keepalived::vrrp_script:
merge:
strategy: deep
keepalived::vrrp_instance:
merge:
strategy: deep
profiles::etcd::node::initial_cluster_token:
convert_to: Sensitive
profiles::dns::updater::key_secret:
convert_to: Sensitive
sysctl::base::values:
merge:
strategy: deep
limits::entries:
merge:
strategy: deep
zfs::zpools:
merge:
strategy: deep
zfs::datasets:
merge:
strategy: deep
rke2::config_hash:
merge:
strategy: deep
postfix::configs:
merge:
strategy: deep
postfix::maps:
merge:
strategy: deep
postfix::virtuals:
merge:
strategy: deep
stalwart::postgresql_password:
convert_to: Sensitive
stalwart::s3_secret_key:
convert_to: Sensitive
stalwart::fallback_admin_password:
convert_to: Sensitive
facts_path: '/opt/puppetlabs/facter/facts.d'
hiera_include:
- timezone
- networking
- ssh::server
- profiles::accounts::rundeck
- limits
- sysctl::base
- exporters::node_exporter
profiles::ntp::client::peers:
- 0.au.pool.ntp.org
- 1.au.pool.ntp.org
- 2.au.pool.ntp.org
- 3.au.pool.ntp.org
consul::install_method: 'package'
consul::manage_repo: false
consul::bin_dir: /usr/bin
vault::install_method: 'repo'
vault::manage_repo: false
vault::bin_dir: /usr/bin
vault::manage_service_file: true
vault::manage_config_dir: true
vault::disable_mlock: false
profiles::dns::base::nameservers:
- 198.18.200.7
profiles::dns::master::basedir: '/var/named/sources'
# dns record publishing. During the k8s cutover both methods run; set
# manage_export false once k8s is authoritative.
# - export: legacy exported-resources -> puppet DNS master
# - nsupdate: RFC2136 to the k8s bind-authoritative write endpoint (.9),
# inert until the TSIG key is set in eyaml:
# profiles::dns::updater::key_secret: ENC[...]
# (must match the key the bind-authoritative zones allow-update
# with; algorithm hmac-sha256)
# k8s is now authoritative: clients resolve via the k8s bind-resolvers
# (profiles::dns::base::nameservers 198.18.200.7, since #490), which forward the
# unkin.net/main.unkin.net/*.18.198.in-addr.arpa zones to the k8s
# bind-authoritative cluster fed by the nsupdate path below. The legacy
# exported-resources -> VM-master /var/named/sources files are no longer in any
# client's resolution path, so stop exporting them. Rollback: set back to true.
profiles::dns::updater::manage_export: false
profiles::dns::updater::manage_nsupdate: true
profiles::dns::updater::server: '198.18.200.9'
profiles::dns::updater::key_name: 'client-update'
profiles::dns::updater::key_algorithm: 'hmac-sha256'
#profiles::dns::base::ns_role: 'roles::infra::dns::resolver'
#profiles::dns::base::use_ns: 'region'
profiles::consul::server::members_role: roles::infra::storage::consul
profiles::consul::token::node_editor::accessor_id: '024e27bd-c5bb-41e7-a578-b766509e11bc'
profiles::consul::client::members_lookup: true
profiles::consul::client::members_role: roles::infra::storage::consul
profiles::consul::client::node_rules:
- resource: node
segment: "%{facts.networking.hostname}"
disposition: write
- resource: node
segment: "%{facts.networking.fqdn}"
disposition: write
- resource: node
segment: ''
disposition: read
- resource: service
segment: node_exporter
disposition: write
profiles::packages::include:
bash-completion: {}
bzip2: {}
ccze: {}
curl: {}
dstat: {}
expect: {}
gzip: {}
git: {}
htop: {}
inotify-tools: {}
iotop: {}
jq: {}
lz4: {}
mtr: {}
ncdu: {}
neovim: {}
p7zip: {}
pbzip2: {}
pigz: {}
pv: {}
python3.11: {}
rsync: {}
screen: {}
socat: {}
strace: {}
sysstat: {}
tar: {}
tmux: {}
traceroute: {}
unzip: {}
vim: {}
vnstat: {}
wget: {}
zsh: {}
zstd: {}
iwl100-firmware:
ensure: absent
iwl1000-firmware:
ensure: absent
iwl105-firmware:
ensure: absent
iwl135-firmware:
ensure: absent
iwl2000-firmware:
ensure: absent
iwl2030-firmware:
ensure: absent
iwl3160-firmware:
ensure: absent
iwl5000-firmware:
ensure: absent
iwl5150-firmware:
ensure: absent
iwl6000-firmware:
ensure: absent
iwl6000g2a-firmware:
ensure: absent
iwl6050-firmware:
ensure: absent
iwl7260-firmware:
ensure: absent
puppet7-release:
ensure: absent
profiles::base::scripts::scripts:
puppet: puppetwrapper.py
profiles::puppet::client::server: 'puppet.query.consul'
profiles::puppet::client::ca_server: 'puppetca.query.consul'
profiles::puppet::client::environment: 'develop'
profiles::puppet::client::runinterval: 1800
profiles::puppet::client::runtimeout: 3600
profiles::puppet::client::show_diff: true
profiles::puppet::client::usecacheonfailure: false
profiles::puppet::client::dns_alt_names:
- "%{trusted.certname}"
# puppetdb
puppetdbapi: puppetdbapi.query.consul
puppetdbsql: puppetdbsql.service.au-syd1.consul
exporters::node_exporter::enable: true
exporters::node_exporter::cleanup_old_node_exporter: true
prometheus::systemd_exporter::export_scrape_job: true
ssh::server::storeconfigs_enabled: false
ssh::server::options:
Protocol: '2'
ListenAddress:
- '127.0.0.1'
- '%{facts.networking.ip}'
SyslogFacility: 'AUTHPRIV'
HostKey:
- /etc/ssh/ssh_host_rsa_key
- /etc/ssh/ssh_host_ecdsa_key
- /etc/ssh/ssh_host_ed25519_key
HostCertificate: /etc/ssh/ssh_host_rsa_key-cert.pem
AuthorizedKeysFile: .ssh/authorized_keys
PermitRootLogin: no
PasswordAuthentication: no
ChallengeResponseAuthentication: no
PubkeyAuthentication: yes
GSSAPIAuthentication: yes
GSSAPICleanupCredentials: yes
UsePAM: yes
X11Forwarding: no
PrintMotd: no
AcceptEnv:
- LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
- LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
- LC_IDENTIFICATION LC_ALL LANGUAGE
- XMODIFIERS
Subsystem: sftp /usr/libexec/openssh/sftp-server
profiles::ssh::knownhosts::lines:
- '@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1HD97vYxLTniE4qNpGuftUlvmkEXIuX8+7nbENv/IzsGUghEDRtyThjQ7ojNKIsQ7f8wXr0gMcI+fAPfrbcOMHCAoYMomikwL0b3h95SZI40q3CyM+0DMnwiVVDX6C1QxkO2Rv9cszSkCa85NotJhXiUuTBI9BFcRPy+mAhbpAru+bfypYofI0wW97XNTl8Jgwmni5MgutBIQAokFIn5ux8iWxndCH3AqDtmkwC5DfQeQ+wZx7rkwqJEpJffQzrjb1gIM6P9hDCVBBVPh/3o80IJ69rFWrJAZUb+JpG4cXJH0NcSW+wqc3JCT/x3q8VlHwOTXSlNNKtOJCRx73mB8e1XTTy2a9FgpKDDg5XQXWHAViJDz1RTRL9gRefMylRgKz4bXoTuY9kJWM8hPTyUejtukbJThlBJc3OmDxBZBF7F0iqB11pHexok43OCEiANodVa36eWu9/5X032Vm48fZ1/akDPY/NSy3wAn7kwut+A0/JAHFHASrq+1mt9YurkJegI+YHXO6eEWpBIpmI7ORHJbGL4MhkHrxYzVamuP8CkU7tXzsv138+wpOcRHNp9yJY4PT40BZkRf/O3O+jt3pj9Dj8rvgywF2W6hFzywh3Y78upOprRkQlQtHfsI8EyrYI8/hUw2u3H+3yPXh3YjWfqvWVG1BRLRHBV7m90uaw=='
profiles::base::groups::local:
admins:
ensure: present
gid: 10000
allowdupe: false
forcelocal: true
sudo::configs:
admins:
priority: 10
content: |
%admins ALL=(ALL) NOPASSWD: ALL
profiles::accounts::sysadmin::sshkeys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ8SRLlPiDylBpdWR9LpvPg4fDVD+DZst4yRPFwMMhta4mnB1H9XuvZkptDhXywWQ7QIcqa2WbhCen0OQJCtwn3s7EYtacmF5MxmwBYocPoK2AArGuh6NA9rwTdLrPdzhZ+gwe88PAzRLNzjm0ZBR+mA9saMbPJdqpKp0AWeAM8QofRQAWuCzQg9i0Pn1KDMvVDRHCZof4pVlHSTyHNektq4ifovn0zhKC8jD/cYu95mc5ftBbORexpGiQWwQ3HZw1IBe0ZETB1qPIPwsoJpt3suvMrL6T2//fcIIUE3TcyJKb/yhztja4TZs5jT8370G/vhlT70He0YPxqHub8ZfBv0khlkY93VBWYpNGJwM1fVqlw7XbfBNdOuJivJac8eW317ZdiDnKkBTxapThpPG3et9ib1HoPGKRsd/fICzNz16h2R3tddSdihTFL+bmTCa6Lo+5t5uRuFjQvhSLSgO2/gRAprc3scYOB4pY/lxOFfq3pU2VvSJtRgLNEYMUYKk= ben@unkin.net
profiles::accounts::rundeck::sshkeys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD4F7VcorbGpyZzBFexz7c/o1JBscrl7hZU0UkWV7fq6YLizW0r6fOzD99hMwu1kdYCjPxbvuUSDEHfyBIp2EgLWU6wFVoufQqlMyOV85+ivQZUc1VNV+X9T+U4v3u/01hkAmlpXtbkwhMSR4Wi+tdABd04+D3CuMDM37mvnFmBBmi41X4Mr1rJhOQumn1XHQ7EYbsdw2mxfEVVeWpZIHz5BjNKSGzEIAYZbFt6s0Y7X3J5RT+Gjqmu043Tc8nNIUFlR9E10qd3Euf9RiBYxBx3z+yfOzJPBzWNBSHv1+PIbO5Mq+z5JaAfoFZO41L7nw+FjV6JJUCVLr6Vq+bCxyA7LW4Oq9ZahSrt/vrT0kTa0tA5U9bqK6e7pB//dm7PzoROtTq0XksV8RseA/fvIje20uaN1z9dynx+UcbszXu9pQ5GIg1o7b5DEi3OZHJwpgdudiCyEeR4+00G0z4PjpEMnTSMHAJ53WxtjzrPAOBnAmPE7hPu4coU+XrCWEXAvRMloJmca68e+zFX7VvFK82KVDuQ99vQ6w4X73IESKoLzyAVxpelwHaDG4fN+zqYfqubVQU1L5cUeYKxqm5r3Us6VvMaYs1ZMUmDGXHOq4FNhGUJYxSjkLvunM6qyAAJQCd6Pw/2TV3UQVerbouGOZaeBLvRguHWSbDrO99Zu+t87w== rundeck_runner
networking::interface_defaults:
ensure: present
family: inet
method: static
netmask: 255.255.255.0
onboot: true
networking::route_defaults:
ensure: present
interface: eth0
netmask: 0.0.0.0
network: default
# logging:
victorialogs::client::journald::enable: true
victorialogs::client::journald::inserturl: https://vlinsert.service.consul:9428/insert/journald
# FIXME these are for the proxmox ceph cluster
profiles::ceph::client::fsid: 7f7f00cb-95de-498c-8dcc-14b54e4e9ca8
profiles::ceph::client::mons:
- 10.18.15.1
- 10.18.15.2
- 10.18.15.3
# de96a98f cluster topology (prodnxsr0001-0019) - single source of truth for
# /etc/ceph/ceph.conf rendered by profiles::ceph::client on the k8s (osd) and
# incus (mon/mgr/mds) roles. fsid/mons are overridden per-role in the role hiera.
# public_network is the /32 of every ceph host's ceph-public loopback.
profiles::ceph::client::cluster_public_ips:
- 198.18.23.1
- 198.18.23.2
- 198.18.23.3
- 198.18.23.4
- 198.18.23.5
- 198.18.23.6
- 198.18.23.7
- 198.18.23.8
- 198.18.23.9
- 198.18.23.10
- 198.18.23.11
- 198.18.23.12
- 198.18.23.13
- 198.18.23.14
- 198.18.23.15
- 198.18.23.16
- 198.18.23.17
- 198.18.23.18
- 198.18.23.19
profiles::ceph::client::mon_initial_members:
- prodnxsr0009
- prodnxsr0010
- prodnxsr0011
- prodnxsr0012
- prodnxsr0013
# two mds daemon instances per mon/mgr/mds host (rendered only where
# render_mds_config is true, i.e. the incus node role).
profiles::ceph::client::mds_instances:
prodnxsr0009: 2
prodnxsr0010: 2
prodnxsr0011: 2
prodnxsr0012: 2
prodnxsr0013: 2
profiles::ceph::conf::config:
global:
auth_client_required: 'cephx'
auth_cluster_required: 'cephx'
auth_service_required: 'cephx'
fsid: 'de96a98f-3d23-465a-a899-86d3d67edab8'
mon_allow_pool_delete: true
mon_initial_members: 'prodnxsr0009,prodnxsr0010,prodnxsr0011,prodnxsr0012,prodnxsr0013'
mon_host: '198.18.23.9,198.18.23.10,198.18.23.11,198.18.23.12,198.18.23.13'
ms_bind_ipv4: true
ms_bind_ipv6: false
osd_crush_chooseleaf_type: 1
osd_pool_default_min_size: 2
osd_pool_default_size: 3
osd_pool_default_pg_num: 128
public_network: >
198.18.23.1/32,198.18.23.2/32,198.18.23.3/32,198.18.23.4/32,
198.18.23.5/32,198.18.23.6/32,198.18.23.7/32,198.18.23.8/32,
198.18.23.9/32,198.18.23.10/32,198.18.23.11/32,198.18.23.12/32,
198.18.23.13/32
client.rgw.ausyd1nxvm2115:
rgw_realm: unkin
rgw_zonegroup: au
rgw_zone: syd1
client.rgw.ausyd1nxvm2116:
rgw_realm: unkin
rgw_zonegroup: au
rgw_zone: syd1
client.rgw.ausyd1nxvm2117:
rgw_realm: unkin
rgw_zonegroup: au
rgw_zone: syd1
client.rgw.ausyd1nxvm2118:
rgw_realm: unkin
rgw_zonegroup: au
rgw_zone: syd1
client.rgw.ausyd1nxvm2119:
rgw_realm: unkin
rgw_zonegroup: au
rgw_zone: syd1
mds:
keyring: /var/lib/ceph/mds/ceph-$id/keyring
mds_standby_replay: true
mds.prodnxsr0009-1:
host: prodnxsr0009
mds.prodnxsr0009-2:
host: prodnxsr0009
mds.prodnxsr0010-1:
host: prodnxsr0010
mds.prodnxsr0010-2:
host: prodnxsr0010
mds.prodnxsr0011-1:
host: prodnxsr0011
mds.prodnxsr0011-2:
host: prodnxsr0011
mds.prodnxsr0012-1:
host: prodnxsr0012
mds.prodnxsr0012-2:
host: prodnxsr0012
mds.prodnxsr0013-1:
host: prodnxsr0013
mds.prodnxsr0013-2:
host: prodnxsr0013
#profiles::base::hosts::additional_hosts:
# - ip: 198.18.17.9
# hostname: prodinf01n09.main.unkin.net
# aliases:
# - prodinf01n09
# - ntp01.main.unkin.net
# - ip: 198.18.17.10
# hostname: prodinf01n10.main.unkin.net
# aliases:
# - prodinf01n10
# - ntp02.main.unkin.net
# - ip: 198.18.17.22
# hostname: prodinf01n22.main.unkin.net
# aliases:
# - prodinf01n22
# - repos.main.unkin.net
Reference in New Issue View Git Blame Copy Permalink