cephrgw-operator/docs/ceph-setup.md

Ceph setup: credentials and permissions the operator needs

cephrgw-operator never talks to RADOS or the RGW admin socket directly. It drives the Ceph manager dashboard REST API (the same API the web dashboard uses) at https://dashboard.ceph.unkin.net. Everything below is about giving the operator a dashboard login with enough RGW authority, and making sure the dashboard itself is wired to your RGW.

There are two credentials involved. Don't confuse them:

#	Credential	Who uses it	What it is
1	Dashboard login (username + password)	the operator → POST /api/auth	a dashboard account with the rgw-manager role
2	RGW admin connection	the dashboard → RGW	a radosgw system user (access/secret key) the dashboard is configured with

The operator only holds #1. #2 is what actually lets the dashboard create RGW users, buckets and bucket policies on the operator's behalf, so it must exist and be privileged.

---

1. Create the dashboard login for the operator

Create a dedicated dashboard user with the built-in rgw-manager role. That role grants full create/read/update/delete on the dashboard's rgw scope (users, buckets, policies) and nothing else — least privilege for this operator.

# Put the password in a file so it never lands in shell history.
printf '%s' 'REPLACE-WITH-A-STRONG-PASSWORD' > /tmp/cephrgw.pw

ceph dashboard ac-user-create k8s-cephrgw-operator -i /tmp/cephrgw.pw rgw-manager

rm -f /tmp/cephrgw.pw

If your Ceph version wants the arguments in a different order, check ceph dashboard ac-user-create -h. To confirm the role exists and what it grants:

ceph dashboard ac-role-show rgw-manager

Prefer rgw-manager over administrator. The operator only needs RGW authority; giving it full dashboard admin is unnecessary blast radius.

2. Make sure the dashboard can manage RGW

The dashboard performs RGW operations through a radosgw system user. On recent Ceph (Pacific and later) the mgr/dashboard module usually auto-discovers and configures this. Verify it first:

ceph dashboard get-rgw-api-access-key   # should print a key, not empty

If it is empty, create a system user and point the dashboard at it:

radosgw-admin user create \
  --uid=dashboard \
  --display-name="Ceph Dashboard" \
  --system

# Feed the returned keys to the dashboard.
radosgw-admin user info --uid=dashboard \
  | jq -r '.keys[0].access_key' > /tmp/ak
radosgw-admin user info --uid=dashboard \
  | jq -r '.keys[0].secret_key' > /tmp/sk

ceph dashboard set-rgw-api-access-key -i /tmp/ak
ceph dashboard set-rgw-api-secret-key -i /tmp/sk
rm -f /tmp/ak /tmp/sk

A --system user has the admin caps the dashboard needs to create/delete RGW users and buckets and to set bucket policies on any bucket. If you would rather not use --system, grant an equivalent admin cap set instead:

radosgw-admin caps add --uid=dashboard \
  --caps="users=*;buckets=*;metadata=*;usage=read;zone=read"

If the dashboard reaches RGW over TLS with a private CA, you may also need:

ceph dashboard set-rgw-api-ssl-verify true   # keep verification on in prod

3. Bucket policy support (read-only / non-owner read-write)

The operator enforces read-only and non-owner read-write grants by writing an S3 bucket policy through the dashboard's bucket API (the bucket_policy field on PUT /api/rgw/bucket/{name}). That field is available on Ceph Reef 18.2+ / Squid. On older releases bucket creation and owner (full) access still work, but policy-based grants will fail — upgrade the cluster, or only use owner credentials, if you are pre-Reef.

Check your version:

ceph versions | jq -r '.mon | keys[]'

4. (Optional) S3 endpoint for consumers

The operator can stamp the S3 endpoint into every credential Secret it writes (S3_ENDPOINT and BUCKET_HOST) so applications don't have to hard-code it. This is the RGW/S3 endpoint your clients use — not the dashboard URL. Provide it via CEPH_RGW_ENDPOINT (see below); if unset, those keys are simply omitted.

---

5. Give the operator its credentials (the cephrgw-credentials Secret)

The operator reads its configuration from environment variables, which the deployment sources from a Secret named cephrgw-credentials in its namespace (cephrgw-system). The Secret data keys map 1:1 to the env vars:

Secret key	Required	Meaning
CEPH_DASHBOARD_URL	yes	dashboard base URL, e.g. https://dashboard.ceph.unkin.net
CEPH_DASHBOARD_USERNAME	yes	the rgw-manager account from step 1
CEPH_DASHBOARD_PASSWORD	yes	its password
CEPH_RGW_ENDPOINT	no	S3 endpoint written into consumer Secrets
CEPH_DASHBOARD_CA	no	PEM CA bundle to verify the dashboard TLS cert (inline)
CEPH_DASHBOARD_CA_FILE	no	path to a mounted CA file (alternative to the above)
CEPH_DASHBOARD_INSECURE	no	"true" to skip TLS verification (dev only)

Create it directly:

kubectl -n cephrgw-system create secret generic cephrgw-credentials \
  --from-literal=CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
  --from-literal=CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
  --from-literal=CEPH_DASHBOARD_PASSWORD='REPLACE-WITH-A-STRONG-PASSWORD' \
  --from-literal=CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net

The deployment carries the reloader.stakater.com/auto: "true" annotation, so rotating this Secret triggers an automatic operator restart — no manual rollout needed.

Sourcing it from Vault (optional)

If you keep the password in Vault, sync it with a VaultStaticSecret (VSO is already running in vso-system) that renders into cephrgw-credentials with the keys above, instead of the plain kubectl create secret. The operator does not care where the Secret comes from, only that those keys exist.

---

Quick verification

Once the Secret and dashboard account exist, a smoke test from your workstation:

# 1. Log in and capture a token.
TOKEN=$(curl -sk -X POST https://dashboard.ceph.unkin.net/api/auth \
  -H 'Accept: application/vnd.ceph.api.v1.0+json' \
  -H 'Content-Type: application/json' \
  -d '{"username":"k8s-cephrgw-operator","password":"REPLACE-WITH-A-STRONG-PASSWORD"}' \
  | jq -r .token)

# 2. List RGW users — a 200 with a JSON array means the role + RGW wiring work.
curl -sk https://dashboard.ceph.unkin.net/api/rgw/user \
  -H 'Accept: application/vnd.ceph.api.v1.0+json' \
  -H "Authorization: Bearer $TOKEN"

If step 1 fails the login/role is wrong (step 1–2 above); if step 1 works but step 2 returns 500/empty, the dashboard→RGW connection is not configured (step 2).