ceph: manage /etc/ceph/ceph.conf on osd and mon/mgr/mds hosts #515

Merged

benvin merged 3 commits from benvin/manage-ceph-conf into develop 2026-08-09 00:00:21 +10:00

Conversation 0 Commits 3 Files Changed 6

+102 -8

unkinben commented 2026-08-08 19:57:16 +10:00

Owner

Copy Link

Copy Source

Why

/etc/ceph/ceph.conf on the prodnxsr ceph cluster (fsid de96a98f) is currently unmanaged and hand-maintained via the runbook (profiles::ceph::client::manage_ceph_conf is false on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes public_network to every ceph host's /32.

How hosts are identified (osd-only vs mon/mgr/mds)

The authoritative, declarative signal is the role's hiera_include list (ENC role -> role hiera):

• osd-only hosts = roles::infra::k8s::* (prodnxsr0001-0008 and prodnxsr0014-0019): include profiles::ceph::osd + ::client, NOT ::mds -> [global] only.
• mon/mgr/mds hosts = roles::infra::incus::node (prodnxsr0009-0013): also include profiles::ceph::mds -> [global] + [mds] + [mds.*].

The template keys on profiles::ceph::client::render_mds_config, set true only in the incus node role hiera (the single role that includes profiles::ceph::mds). The is_ceph_mds fact is a runtime fallback, not used here.

Changes

• Add cluster topology as a single source of truth in hieradata/common.yaml: cluster_public_ips (all 19 ceph host /32s, 198.18.23.1-19), mon_initial_members (the 5 mons), and the mds_instances map (two mds daemons per mon/mgr/mds host).
• Rework profiles/ceph/client.conf.erb to render [global] always and the [mds]/[mds.*] sections only when render_mds_config is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
• Extend profiles::ceph::client with topology + tunable params (sample values as defaults) and gate [mds] on render_mds_config.
• Drop the hard Package[ceph-common] dependency when the class does not manage the package (cephadm / profiles::packages deliver it on the k8s and incus hosts; k8s hosts do not declare ceph-common, so an unconditional require would fail to compile).
• Enable manage_ceph_conf: true on roles/infra/k8s.yaml, roles/infra/k8s/node.yaml, and roles/infra/incus/node.yaml; add render_mds_config: true to the incus node role. prodnxsr0014-0019 run roles::infra::k8s::compute (confirmed live via PuppetDB; is_ceph_osd true, is_ceph_mds false), so they are already covered by k8s.yaml and get the [global]-only variant with no mds sections.

Live-cluster safety review (please read before merge)

This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file except for the one deliberate public_network normalization below.

• Intended change: public_network is normalized to all 19 ceph host /32s on every host (198.18.23.1-19). Adoption is now a single public_network-line change on every cluster host, no longer a no-op on the osd hosts:
  • osd hosts prodnxsr0001-0008 currently list .1-.13 live -> first convergence adds .14-.19.
  • mon/mgr/mds hosts prodnxsr0009-0013 currently list only .9-.13 live -> first convergence adds .1-.8 and .14-.19.
  • prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full [global] variant.
• Verified byte diff against the live files (/etc/ceph/ceph.conf pulled from prodnxsr0008 and prodnxsr0009): each differs by exactly one line (the expanded public_network); on prodnxsr0009 the [mds] + [mds.*] sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live .9-.13 value reproduces the live file byte-for-byte, confirming the format match.
• RGW hosts (roles::infra::ceph::rgw, ausyd1nxvm2115-2119) include only profiles::ceph::rgw -> profiles::ceph::conf; they do not include profiles::ceph::client, so their variant is untouched and there is no duplicate File[/etc/ceph/ceph.conf].

Rendered samples:

• osd-only (prodnxsr0008): [global] with all 19 /32s in public_network, nothing else.
• mon/mgr/mds (prodnxsr0009): the same [global] plus [mds] and [mds.<host>-1|2] for prodnxsr0009-0013, blank-line separated.

## Why `/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`. ## How hosts are identified (osd-only vs mon/mgr/mds) The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera): - **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only. - **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`. The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here. ## Changes - Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host). - Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections. - Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`. - Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile). - Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections. ## Live-cluster safety review (please read before merge) This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below. - **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts: - osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`. - mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`. - prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant. - **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match. - RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`. Rendered samples: - osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else. - mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.

unkinben added 1 commit 2026-08-08 19:57:16 +10:00

ceph: manage /etc/ceph/ceph.conf on osd and mon/mgr/mds hosts ... 2d125487c3

Bring the hand-maintained /etc/ceph/ceph.conf under Puppet on the
prodnxsr ceph cluster (fsid de96a98f). The file is identified per host
by role: k8s roles include profiles::ceph::osd only and get a [global]
section; the incus node role also includes profiles::ceph::mds and
additionally gets the [mds] + [mds.*] sections.

- Add cluster topology as a single source of truth in common.yaml:
  cluster_public_ips (all 13 ceph host /32s), mon_initial_members, and
  the mds_instances map (two mds daemons per mon/mgr/mds host).
- Render /etc/ceph/ceph.conf from that topology in the reworked
  client.conf.erb, preserving the live two-space indent and key order.
- Gate the [mds] sections on render_mds_config, set true only in the
  incus node role hiera (the role that includes profiles::ceph::mds).
- Drop the hard Package[ceph-common] dependency when the class does not
  manage the package (cephadm/profiles::packages deliver it on the k8s
  and incus hosts).
- Enable manage_ceph_conf on the k8s and incus node roles.

public_network is normalized to all 13 ceph host /32s on every host;
this rewrites it on the mon/mgr/mds hosts (adding .1-.8) as the one
intended content change. RGW hosts keep their profiles::ceph::conf
variant and are untouched.

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben added 1 commit 2026-08-08 20:04:06 +10:00

ceph: match live blank-line separators between mds sections ... 5b04aa341d

Diffing the render against the live /etc/ceph/ceph.conf on prodnxsr0009
showed the hand-maintained file separates the mds sections with blank
lines: one before [mds] and one before each [mds.X-i] block (no trailing
blank line after the last). Emit those separators so a mon/mgr/mds host
renders byte-identical to live except for the intended public_network
normalization. Verified: osd host stays byte-identical, mon host differs
only on the public_network line.

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben added 1 commit 2026-08-08 22:25:55 +10:00

ceph: include prodnxsr0014-0019 in public_network ... fa3b97058c

Why: prodnxsr0014-0019 were racked as roles::base but have since joined
the de96a98f ceph cluster as OSD hosts (live enc_role is now
roles::infra::k8s::compute, is_ceph_osd true). Their ceph-public /32s
must appear in public_network on every cluster member, and they must
receive the osd-only ceph.conf.

How:
- Expand profiles::ceph::client::cluster_public_ips from .1-.13 to
  .1-.19 (add ceph-public loopbacks 198.18.23.14-19).
- No role/hiera change needed for 0014-0019: they run
  roles::infra::k8s::compute, already covered by roles/infra/k8s.yaml
  (manage_ceph_conf true, render_mds_config unset/false), so they get
  the [global]-only variant with no mds sections.

Consequence: first convergence now rewrites public_network on the
existing osd hosts (0001-0008) and mon hosts (0009-0013) too, adding
.14-.19; adoption is a single public_network-line change on every
cluster host rather than a no-op on osd hosts. Verified against the
live files: osd (0008) and mon (0009) each differ by exactly the
public_network line, mds formatting on 0009 unchanged.

benvin merged commit 0fd01ac1a7 into develop 2026-08-09 00:00:21 +10:00

benvin deleted branch benvin/manage-ceph-conf 2026-08-09 00:00:21 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/puppet-prod#515