cephrgw-operator/docs/ceph-setup.md

Ceph setup: credentials and permissions the operator needs

cephrgw-operator never talks to RADOS or the RGW admin socket directly. It drives the Ceph manager dashboard REST API (the same API the web dashboard uses) at https://dashboard.ceph.unkin.net. Everything below is about giving the operator a dashboard login with enough RGW authority, and making sure the dashboard itself is wired to your RGW.

There are two credentials involved. Don't confuse them:

#	Credential	Who uses it	What it is
1	Dashboard login (username + password)	the operator → POST /api/auth	a dashboard account with the rgw-manager role
2	RGW admin connection	the dashboard → RGW	a radosgw system user (access/secret key) the dashboard is configured with

The operator only holds #1. #2 is what actually lets the dashboard create RGW users, buckets and bucket policies on the operator's behalf, so it must exist and be privileged.

---

1. Create the dashboard login for the operator

Create a dedicated dashboard user with the built-in rgw-manager role. That role grants full create/read/update/delete on the dashboard's rgw scope (users, buckets, policies) and nothing else — least privilege for this operator.

# Put the password in a file so it never lands in shell history.
printf '%s' 'REPLACE-WITH-A-STRONG-PASSWORD' > /tmp/cephrgw.pw

ceph dashboard ac-user-create k8s-cephrgw-operator -i /tmp/cephrgw.pw rgw-manager

rm -f /tmp/cephrgw.pw

If your Ceph version wants the arguments in a different order, check ceph dashboard ac-user-create -h. To confirm the role exists and what it grants:

ceph dashboard ac-role-show rgw-manager

Prefer rgw-manager over administrator. The operator only needs RGW authority; giving it full dashboard admin is unnecessary blast radius.

2. Make sure the dashboard can manage RGW

The dashboard performs RGW operations through a radosgw system user. On recent Ceph (Pacific and later) the mgr/dashboard module usually auto-discovers and configures this. Verify it first:

ceph dashboard get-rgw-api-access-key   # should print a key, not empty

If it is empty, create a system user and point the dashboard at it:

radosgw-admin user create \
  --uid=dashboard \
  --display-name="Ceph Dashboard" \
  --system

# Feed the returned keys to the dashboard.
radosgw-admin user info --uid=dashboard \
  | jq -r '.keys[0].access_key' > /tmp/ak
radosgw-admin user info --uid=dashboard \
  | jq -r '.keys[0].secret_key' > /tmp/sk

ceph dashboard set-rgw-api-access-key -i /tmp/ak
ceph dashboard set-rgw-api-secret-key -i /tmp/sk
rm -f /tmp/ak /tmp/sk

A --system user has the admin caps the dashboard needs to create/delete RGW users and buckets and to set bucket policies on any bucket. If you would rather not use --system, grant an equivalent admin cap set instead:

radosgw-admin caps add --uid=dashboard \
  --caps="users=*;buckets=*;metadata=*;usage=read;zone=read"

If the dashboard reaches RGW over TLS with a private CA, you may also need:

ceph dashboard set-rgw-api-ssl-verify true   # keep verification on in prod

3. Bucket policy support (read-only / non-owner read-write)

The operator enforces read-only and non-owner read-write grants by writing an S3 bucket policy through the dashboard's bucket API (the bucket_policy field on PUT /api/rgw/bucket/{name}). That field is available on Ceph Reef 18.2+ / Squid. On older releases bucket creation and owner (full) access still work, but policy-based grants will fail — upgrade the cluster, or only use owner credentials, if you are pre-Reef.

Check your version:

ceph versions | jq -r '.mon | keys[]'

4. (Optional) S3 endpoint for consumers

The operator can stamp the S3 endpoint into every credential Secret it writes (S3_ENDPOINT and BUCKET_HOST) so applications don't have to hard-code it. This is the RGW/S3 endpoint your clients use — not the dashboard URL. Provide it via CEPH_RGW_ENDPOINT (see below); if unset, those keys are simply omitted.

---

5. Give the operator its credentials (the cephrgw-credentials Secret)

The operator reads its configuration from environment variables, which the deployment sources (via envFrom) from a Secret named cephrgw-credentials in its namespace (cephrgw-system). The Secret data keys map 1:1 to the env vars:

Secret key	Required	Meaning
CEPH_DASHBOARD_URL	yes	dashboard base URL, e.g. https://dashboard.ceph.unkin.net
CEPH_DASHBOARD_USERNAME	yes	the rgw-manager account from step 1
CEPH_DASHBOARD_PASSWORD	yes	its password
CEPH_RGW_ENDPOINT	no	S3 endpoint written into consumer Secrets
CEPH_DASHBOARD_CA	no	PEM CA bundle to verify the dashboard TLS cert (inline)
CEPH_DASHBOARD_CA_FILE	no	path to a mounted CA file (alternative to the above)
CEPH_DASHBOARD_INSECURE	no	"true" to skip TLS verification (dev only)

Primary method: Vault + VSO

The Secret is not managed in GitOps; it is rendered from Vault by the Vault Secrets Operator (VSO). The argocd-apps cephrgw-system app ships a VaultAuth and a VaultStaticSecret that authenticate with the shared default Kubernetes auth role and render the KV path kubernetes/namespace/cephrgw-system/default/cephrgw-credentials into the cephrgw-credentials Secret. That path sits under the cluster's templated default policy (kv/data/kubernetes/namespace/<ns>/<sa>/*), so no dedicated Vault role or policy is required — you only seed the values:

vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
  CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
  CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
  CEPH_DASHBOARD_PASSWORD='REPLACE-WITH-A-STRONG-PASSWORD' \
  CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net

The keys under that KV path are copied verbatim into the Secret, so they must be named exactly as the table above. VSO refreshes the Secret every few minutes, and the deployment's reloader.stakater.com/auto: "true" annotation restarts the operator when it changes — so rotating the credential is just a new vault kv put, no manual rollout.

Fallback: a plain Secret

Outside this cluster (or for a quick kind test) you can create the Secret directly instead of using Vault:

kubectl -n cephrgw-system create secret generic cephrgw-credentials \
  --from-literal=CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
  --from-literal=CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
  --from-literal=CEPH_DASHBOARD_PASSWORD='REPLACE-WITH-A-STRONG-PASSWORD' \
  --from-literal=CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net

The operator does not care where the Secret comes from, only that those keys exist.

---

Quick verification

Once the Secret and dashboard account exist, a smoke test from your workstation:

# 1. Log in and capture a token.
TOKEN=$(curl -sk -X POST https://dashboard.ceph.unkin.net/api/auth \
  -H 'Accept: application/vnd.ceph.api.v1.0+json' \
  -H 'Content-Type: application/json' \
  -d '{"username":"k8s-cephrgw-operator","password":"REPLACE-WITH-A-STRONG-PASSWORD"}' \
  | jq -r .token)

# 2. List RGW users — a 200 with a JSON array means the role + RGW wiring work.
curl -sk https://dashboard.ceph.unkin.net/api/rgw/user \
  -H 'Accept: application/vnd.ceph.api.v1.0+json' \
  -H "Authorization: Bearer $TOKEN"

If step 1 fails the login/role is wrong (step 1–2 above); if step 1 works but step 2 returns 500/empty, the dashboard→RGW connection is not configured (step 2).