ceph-system: run radosgw as a deployment #583

Merged
benvin merged 3 commits from benvin/ceph-system-rgw into main 2026-11-15 21:51:59 +11:00
Member

Part of #582. RGW runs on VMs (ausyd1nxvm2115-2119) and needs to move into k8s under GitOps; this adds the ceph-system namespace with an in-cluster RGW behind a ClusterIP service. Cutover comes later.

  • add ceph-system app to the platform applicationset
  • add ceph.conf configmap with a shared client.rgw.syd1 section and explicit keyring path
  • add rgw keyring from vault kubernetes/namespace/ceph-system/default/rgw-keyring (keyring)
  • add 3-replica quay.io/ceph/ceph:v18.2.8 radosgw deployment, non-root, read-only root, startup probe, hard per-node spread, PDB + VPA
  • add ClusterIP service rgw on 7480

Requires unkin/terraform-artifactapi#78 merged first.

Part of #582. RGW runs on VMs (ausyd1nxvm2115-2119) and needs to move into k8s under GitOps; this adds the `ceph-system` namespace with an in-cluster RGW behind a ClusterIP service. Cutover comes later. - add `ceph-system` app to the platform applicationset - add ceph.conf configmap with a shared `client.rgw.syd1` section and explicit keyring path - add rgw keyring from vault `kubernetes/namespace/ceph-system/default/rgw-keyring` (`keyring`) - add 3-replica `quay.io/ceph/ceph:v18.2.8` radosgw deployment, non-root, read-only root, startup probe, hard per-node spread, PDB + VPA - add ClusterIP service `rgw` on 7480 Requires unkin/terraform-artifactapi#78 merged first.
unkin-agent added 1 commit 2026-11-15 19:52:36 +11:00
ceph-system: run radosgw as a deployment
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
8ca166a494
Author
Member
  • apps/base/ceph-system/deployment.yaml:46-60 — HA is soft only (preferred anti-affinity + ScheduleAnyway spread); all 3 RGW pods can land on one node → use DoNotSchedule on the hostname spread (keep a zone constraint soft if zones are uneven).
  • nit: apps/base/ceph-system/deployment.yaml:66-73 — --setuser ceph --setgroup ceph are redundant under runAsUser: 167 non-root → drop them.
  • nit: apps/base/ceph-system/kustomization.yaml — no vpa.yaml; cephrgw-system and 12 other base apps ship one → add for consistency.
  • nit: apps/base/ceph-system/vaultstaticsecret.yaml — no comment naming the seed command/keys for rgw-keyring (see cephrgw-system sibling); pods sit in ContainerCreating until seeded → add a one-line seed note.
- apps/base/ceph-system/deployment.yaml:46-60 — HA is soft only (preferred anti-affinity + `ScheduleAnyway` spread); all 3 RGW pods can land on one node → use `DoNotSchedule` on the hostname spread (keep a zone constraint soft if zones are uneven). - nit: apps/base/ceph-system/deployment.yaml:66-73 — `--setuser ceph --setgroup ceph` are redundant under `runAsUser: 167` non-root → drop them. - nit: apps/base/ceph-system/kustomization.yaml — no `vpa.yaml`; cephrgw-system and 12 other base apps ship one → add for consistency. - nit: apps/base/ceph-system/vaultstaticsecret.yaml — no comment naming the seed command/keys for `rgw-keyring` (see cephrgw-system sibling); pods sit in ContainerCreating until seeded → add a one-line seed note.
unkin-agent added 1 commit 2026-11-15 19:57:12 +11:00
Require rgw host spread, add VPA, drop redundant setuser
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
a08da880a9
Author
Member
  • apps/base/ceph-system/configmap.yaml:21 — [client.rgw.syd1] sets no keyring; the keyring is mounted at /var/lib/ceph/radosgw/ceph-rgw.syd1/keyring and found only via implicit $rgw_data default, otherwise radosgw looks in /etc/ceph/ceph.client.rgw.syd1.keyring and fails auth → add keyring = /var/lib/ceph/radosgw/ceph-rgw.syd1/keyring (and optionally rgw_data) to the section.
  • apps/base/ceph-system/deployment.yaml:71 — no startupProbe; liveness (30s + 6x20s) is the only guard while radosgw loads realm/period on a cold start → add a startupProbe on / (failureThreshold ~30, period 5s) and drop liveness initialDelaySeconds.
  • nit: apps/base/ceph-system/deployment.yaml:88 — memory request 1Gi vs limit 4Gi (4x burst, Burstable) → set request closer to limit or equal for an OOM-prone gateway; VPA is Off so it will not correct this.
  • nit: apps/base/ceph-system/deployment.yaml:23 — quay.io/ceph/ceph pull depends on the unmerged terraform-artifactapi allowlist; merging first yields ImagePullBackOff on 3 pods → link the tfartifactapi PR in the body and merge that first.
- apps/base/ceph-system/configmap.yaml:21 — `[client.rgw.syd1]` sets no `keyring`; the keyring is mounted at `/var/lib/ceph/radosgw/ceph-rgw.syd1/keyring` and found only via implicit `$rgw_data` default, otherwise radosgw looks in `/etc/ceph/ceph.client.rgw.syd1.keyring` and fails auth → add `keyring = /var/lib/ceph/radosgw/ceph-rgw.syd1/keyring` (and optionally `rgw_data`) to the section. - apps/base/ceph-system/deployment.yaml:71 — no startupProbe; liveness (30s + 6x20s) is the only guard while radosgw loads realm/period on a cold start → add a startupProbe on `/` (failureThreshold ~30, period 5s) and drop liveness initialDelaySeconds. - nit: apps/base/ceph-system/deployment.yaml:88 — memory request 1Gi vs limit 4Gi (4x burst, Burstable) → set request closer to limit or equal for an OOM-prone gateway; VPA is Off so it will not correct this. - nit: apps/base/ceph-system/deployment.yaml:23 — `quay.io/ceph/ceph` pull depends on the unmerged terraform-artifactapi allowlist; merging first yields ImagePullBackOff on 3 pods → link the tfartifactapi PR in the body and merge that first.
unkin-agent added 1 commit 2026-11-15 20:04:19 +11:00
ceph-system: pin rgw keyring path, add startup probe, raise memory request
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
490388793f
Author
Member

No findings.

No findings.
benvin merged commit 26e53ee35c into main 2026-11-15 21:51:59 +11:00
benvin deleted branch benvin/ceph-system-rgw 2026-11-15 21:51:59 +11:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#583