unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

Decommission the legacy standalone ceph-csi namespace #323

New Issue
Open
opened 2026-08-02 16:24:31 +10:00 by unkinben · 1 comment
unkinben commented 2026-08-02 16:24:31 +10:00
Owner
Copy Link
Copy Source

Problem
The privilege audit found a standalone ceph-csi namespace running a privileged CSI driver (csi-rbdplugin + provisioner) that is NOT in argocd-apps git and duplicates the managed csi-cephfs/csi-cephrbd deployments.

Proposal

  • confirm no PVs/StorageClasses bind to the legacy ceph-csi provisioner (the managed csi-cephrbd/csi-cephfs are authoritative)
  • remove the unmanaged namespace/workloads once confirmed unused

Acceptance

  • legacy ceph-csi gone; no volume regressions; only the git-managed CSI drivers remain.
**Problem** The privilege audit found a standalone `ceph-csi` namespace running a privileged CSI driver (csi-rbdplugin + provisioner) that is NOT in argocd-apps git and duplicates the managed csi-cephfs/csi-cephrbd deployments. **Proposal** - confirm no PVs/StorageClasses bind to the legacy ceph-csi provisioner (the managed csi-cephrbd/csi-cephfs are authoritative) - remove the unmanaged namespace/workloads once confirmed unused **Acceptance** - legacy ceph-csi gone; no volume regressions; only the git-managed CSI drivers remain.
unkinben commented 2026-08-08 18:31:51 +10:00
Author
Owner
Copy Link
Copy Source

Assessment: STILL REQUIRED — but this is not an argocd-apps change

Verified against live cluster (read-only) + git main.

Legacy ceph-csi is real, active, and unmanaged by Argo CD

  • ns ceph-csi (209d): privileged csi-rbdplugin daemonset 13/13 + csi-rbdplugin-provisioner 3/3, driver rbd.csi.ceph.com.
  • Not in argocd-apps git and no Argo Application exists for it. It is owned by a k3s server-manifest Addon — ns carries objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon, owner-name: ceph-csi-config, owner-namespace: kube-system.

It duplicates / conflicts with the managed driver

  • Managed csi-cephrbd (ceph-csi-rbd-csi-rbdplugin) also runs 13/13 and registers the same driver rbd.csi.ceph.com at the same kubelet path --kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock. Legacy csi-rbdplugin uses the identical --drivername=rbd.csi.ceph.com / /var/lib/kubelet/plugins/rbd.csi.ceph.com — so the two node plugins race for the same per-node registration socket (last writer wins).
  • Only 2 CSIDrivers exist cluster-wide: rbd.csi.ceph.com, cephfs.csi.ceph.com; managed csi-cephrbd/csi-cephfs (both in git under apps/*/csi-cephrbd, csi-cephfs, storage ApplicationSet) are authoritative.
  • PV usage: 90 PVs on rbd.csi.ceph.com, 11 on cephfs.csi.ceph.com. The pv.kubernetes.io/provisioned-by annotation is the driver name (not the namespace), so it can't attribute PVs to a specific provisioner deployment.

Why no PR here
kubectl delete ns ceph-csi alone will be reverted by the k3s addon reconciler. Decommission must be done at the k3s bootstrap layer, not argocd-apps:

  1. Remove the k3s server manifest ceph-csi-config (/var/lib/rancher/k3s/server/manifests/… on the server node(s) — likely Puppet/terraform-k3s managed) and delete Addon ceph-csi-config in kube-system.
  2. Delete orphaned cluster-scoped RBAC bound to ceph-csi SAs: ClusterRoleBindings ceph-csi-rbd-csi-rbdplugin-provisioner, rbd-csi-nodeplugin, rbd-csi-provisioner-role.
  3. Delete ns ceph-csi + leftover ReplicaSet csi-rbdplugin-provisioner-6796dff6f4.

Safety before deletion (needs privileged access I don't have — leases are RBAC-forbidden to my read-only token): confirm the managed csi-cephrbd provisioner holds the rbd.csi.ceph.com external-provisioner leader lease and its node plugins own the registration socket on all 13 nodes, then remove legacy so mounts/provisioning continue uninterrupted.

Recommendation: keep open; assign to whoever owns the k3s server-manifest/bootstrap (Puppet or terraform-k3s). Nothing to change in argocd-apps — the managed replacement is already in place; this is purely removing the pre-migration k3s addon.

### Assessment: STILL REQUIRED — but this is not an argocd-apps change Verified against live cluster (read-only) + git `main`. **Legacy `ceph-csi` is real, active, and unmanaged by Argo CD** - ns `ceph-csi` (209d): privileged `csi-rbdplugin` daemonset **13/13** + `csi-rbdplugin-provisioner` 3/3, driver `rbd.csi.ceph.com`. - Not in argocd-apps git and no Argo `Application` exists for it. It is owned by a **k3s server-manifest Addon** — ns carries `objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon`, `owner-name: ceph-csi-config`, `owner-namespace: kube-system`. **It duplicates / conflicts with the managed driver** - Managed `csi-cephrbd` (`ceph-csi-rbd-csi-rbdplugin`) also runs **13/13** and registers the **same** driver `rbd.csi.ceph.com` at the **same** kubelet path `--kubelet-registration-path=/var/lib/kubelet/plugins/rbd.csi.ceph.com/csi.sock`. Legacy `csi-rbdplugin` uses the identical `--drivername=rbd.csi.ceph.com` / `/var/lib/kubelet/plugins/rbd.csi.ceph.com` — so the two node plugins race for the same per-node registration socket (last writer wins). - Only 2 CSIDrivers exist cluster-wide: `rbd.csi.ceph.com`, `cephfs.csi.ceph.com`; managed `csi-cephrbd`/`csi-cephfs` (both in git under `apps/*/csi-cephrbd`, `csi-cephfs`, storage ApplicationSet) are authoritative. - PV usage: 90 PVs on `rbd.csi.ceph.com`, 11 on `cephfs.csi.ceph.com`. The `pv.kubernetes.io/provisioned-by` annotation is the driver name (not the namespace), so it can't attribute PVs to a specific provisioner deployment. **Why no PR here** `kubectl delete ns ceph-csi` alone will be reverted by the k3s addon reconciler. Decommission must be done at the k3s bootstrap layer, not argocd-apps: 1. Remove the k3s server manifest `ceph-csi-config` (`/var/lib/rancher/k3s/server/manifests/…` on the server node(s) — likely Puppet/terraform-k3s managed) and delete Addon `ceph-csi-config` in `kube-system`. 2. Delete orphaned cluster-scoped RBAC bound to `ceph-csi` SAs: ClusterRoleBindings `ceph-csi-rbd-csi-rbdplugin-provisioner`, `rbd-csi-nodeplugin`, `rbd-csi-provisioner-role`. 3. Delete ns `ceph-csi` + leftover ReplicaSet `csi-rbdplugin-provisioner-6796dff6f4`. **Safety before deletion (needs privileged access I don't have — leases are RBAC-forbidden to my read-only token):** confirm the managed `csi-cephrbd` provisioner holds the `rbd.csi.ceph.com` external-provisioner leader lease and its node plugins own the registration socket on all 13 nodes, then remove legacy so mounts/provisioning continue uninterrupted. **Recommendation:** keep open; assign to whoever owns the k3s server-manifest/bootstrap (Puppet or terraform-k3s). Nothing to change in argocd-apps — the managed replacement is already in place; this is purely removing the pre-migration k3s addon.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#323
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?