Move ceph mon/mgr/mds/rgw into k8s (ceph-system) #582
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Ceph mon/mgr/mds and RGW run as systemd daemons and Incus containers on prodnxsr0009–0013. Those nodes are being rebuilt as k8s nodes, and the daemons need GitOps ownership. OSDs stay on bare metal.
Proposal
Run mon, mgr, mds and RGW in a hand-rolled
ceph-systemnamespace (no Rook, no hostNetwork). OSDs reach the pods via flannel and kube-proxy. Mons get static PureLB LB IPs withexternalTrafficPolicy: Local, routed by OSPF, so hosts that aren't k8s nodes yet can still reach them during the rebuild. The bootstrap tier must not depend on Ceph, Vault, cluster DNS or artifactapi pulls.radosgw.service.consul/s3.ceph.unkin.net, then retire ausyd1nxvm2115–2119.cephfsvolsmounts, so every CephFS client is in k8s.ceph-systembase: ceph.conf ConfigMap, toolbox pod, one PureLB Service per mon (Local).ceph mon removethe node's monk8s::compute, thenceph-volume lvm activate --allmon_host(puppet + CSI), unset noout, wait for HEALTH_OKmds.*/client.rgw.*conf,mon_initial_members.Acceptance
ceph-systemunder ArgoCD.