Move ceph mon/mgr/mds/rgw into k8s (ceph-system) #582

Open
opened 2026-11-15 19:47:31 +11:00 by unkin-agent · 0 comments
Member

Problem

Ceph mon/mgr/mds and RGW run as systemd daemons and Incus containers on prodnxsr0009–0013. Those nodes are being rebuilt as k8s nodes, and the daemons need GitOps ownership. OSDs stay on bare metal.

Proposal

Run mon, mgr, mds and RGW in a hand-rolled ceph-system namespace (no Rook, no hostNetwork). OSDs reach the pods via flannel and kube-proxy. Mons get static PureLB LB IPs with externalTrafficPolicy: Local, routed by OSPF, so hosts that aren't k8s nodes yet can still reach them during the rebuild. The bootstrap tier must not depend on Ceph, Vault, cluster DNS or artifactapi pulls.

  1. Move RGW to k8s: Deployment and Service (#583, needs terraform-artifactapi#78), then cert and HTTPRoute, then repoint radosgw.service.consul/s3.ceph.unkin.net, then retire ausyd1nxvm2115–2119.
  2. Move the Incus workloads off prodnxsr0009–0013 and drop the host cephfsvols mounts, so every CephFS client is in k8s.
  3. Add the ceph-system base: ceph.conf ConfigMap, toolbox pod, one PureLB Service per mon (Local).
  4. Move MDS to a StatefulSet on the pod network (one cephx entity per ordinal). Keep 3 filesystems × (active + standby-replay) plus spare standbys, then stop the systemd MDS.
  5. Rebuild prodnxsr0009–0012 one node at a time (OS disk only):
    • set noout, then ceph mon remove the node's mon
    • reinstall as k8s::compute, then ceph-volume lvm activate --all
    • set up openebs local, then add a k8s mon on its LB IP
    • update mon_host (puppet + CSI), unset noout, wait for HEALTH_OK
  6. Before rebuilding prodnxsr0013, move mgr to a StatefulSet on the pod network and stop the systemd mgr.
  7. Rebuild prodnxsr0013 the same way as step 5.
  8. Clean up puppet: incus role ceph profiles, mds.*/client.rgw.* conf, mon_initial_members.
  9. Move the bind resolvers off cephrbd onto openebs.
  10. Rotate the RGW zone system key (independent).

Acceptance

  • No ceph mon/mgr/mds/rgw systemd units or RGW containers remain.
  • 5 mons are in quorum on static LB IPs, and mgr/mds/rgw run in ceph-system under ArgoCD.
  • A cold start with Vault, artifactapi and cluster DNS down still brings Ceph up.
## Problem Ceph mon/mgr/mds and RGW run as systemd daemons and Incus containers on prodnxsr0009–0013. Those nodes are being rebuilt as k8s nodes, and the daemons need GitOps ownership. OSDs stay on bare metal. ## Proposal Run mon, mgr, mds and RGW in a hand-rolled `ceph-system` namespace (no Rook, no hostNetwork). OSDs reach the pods via flannel and kube-proxy. Mons get static PureLB LB IPs with `externalTrafficPolicy: Local`, routed by OSPF, so hosts that aren't k8s nodes yet can still reach them during the rebuild. The bootstrap tier must not depend on Ceph, Vault, cluster DNS or artifactapi pulls. 1. Move RGW to k8s: Deployment and Service (#583, needs terraform-artifactapi#78), then cert and HTTPRoute, then repoint `radosgw.service.consul`/`s3.ceph.unkin.net`, then retire ausyd1nxvm2115–2119. 2. Move the Incus workloads off prodnxsr0009–0013 and drop the host `cephfsvols` mounts, so every CephFS client is in k8s. 3. Add the `ceph-system` base: ceph.conf ConfigMap, toolbox pod, one PureLB Service per mon (Local). 4. Move MDS to a StatefulSet on the pod network (one cephx entity per ordinal). Keep 3 filesystems × (active + standby-replay) plus spare standbys, then stop the systemd MDS. 5. Rebuild prodnxsr0009–0012 one node at a time (OS disk only): - set noout, then `ceph mon remove` the node's mon - reinstall as `k8s::compute`, then `ceph-volume lvm activate --all` - set up openebs local, then add a k8s mon on its LB IP - update `mon_host` (puppet + CSI), unset noout, wait for HEALTH_OK 6. Before rebuilding prodnxsr0013, move mgr to a StatefulSet on the pod network and stop the systemd mgr. 7. Rebuild prodnxsr0013 the same way as step 5. 8. Clean up puppet: incus role ceph profiles, `mds.*`/`client.rgw.*` conf, `mon_initial_members`. 9. Move the bind resolvers off cephrbd onto openebs. 10. Rotate the RGW zone system key (independent). ## Acceptance - No ceph mon/mgr/mds/rgw systemd units or RGW containers remain. - 5 mons are in quorum on static LB IPs, and mgr/mds/rgw run in `ceph-system` under ArgoCD. - A cold start with Vault, artifactapi and cluster DNS down still brings Ceph up.
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#582