cephrgw-operator/docs/ceph-setup.md

Ceph setup: credentials and permissions the operator needs

cephrgw-operator talks directly to radosgw, the same way the radosgw-admin CLI and S3 clients do — no manager dashboard involved. It uses two native Go libraries against the RGW endpoint (e.g. https://radosgw.service.consul:443):

• go-ceph rgw/admin → the RGW Admin Ops API (/admin/user, /admin/bucket), signed with the operator's access/secret key, for users, keys, quotas and bucket info/removal.
• aws-sdk-go-v2 → the S3 API, signed as each bucket's owner, for bucket creation, versioning, policy, tagging and object lock.

So there is exactly one credential to provision: a radosgw user with admin caps, plus its access/secret key.

---

1. Create the operator's RGW admin user

Create a dedicated radosgw user and give it the admin caps the operator needs. Only users and buckets caps are required (the operator never reads usage or metadata endpoints):

radosgw-admin user create \
  --uid=cephrgw-operator \
  --display-name="cephrgw-operator" \
  --caps="users=*;buckets=*"

# Grab its keys (these become CEPH_RGW_ACCESS_KEY / CEPH_RGW_SECRET_KEY):
radosgw-admin user info --uid=cephrgw-operator \
  | jq -r '.keys[0] | .access_key, .secret_key'

If the user already exists, add the caps instead:

radosgw-admin caps add --uid=cephrgw-operator --caps="users=*;buckets=*"

users=*;buckets=* lets the operator create/read/delete RGW users and read/ remove buckets through the Admin Ops API. Bucket creation and all bucket sub-resources (versioning, policy, tagging, object lock) go over the S3 API signed as the bucket owner, so they need no extra admin cap — every RGW user can manage its own buckets. The --system flag is not required.

2. Admin Ops API must be enabled on radosgw

The Admin Ops API is served by radosgw at the admin resource and is enabled by default. If your deployment has trimmed rgw_enable_apis, make sure it includes both s3 and admin:

rgw_enable_apis = s3, admin

Quick check from your workstation (a 403/AccessDenied still proves the endpoint is reachable and the API is on; a connection error means it is not):

curl -sk "https://radosgw.service.consul:443/admin/user?format=json"

3. Bucket policy support (read-only / non-owner read-write)

The operator enforces read-only and non-owner read-write grants by writing an S3 bucket policy (PutBucketPolicy). Bucket-policy support is available on Ceph Reef 18.2+ / Squid. On older releases bucket creation and owner (full) access still work, but policy-based grants will fail — upgrade the cluster, or only use owner credentials, if you are pre-Reef.

Check your version:

ceph versions | jq -r '.mon | keys[]'

4. (Optional) S3 endpoint for consumers

The operator can stamp the S3 endpoint into every credential Secret it writes (S3_ENDPOINT and BUCKET_HOST) so applications don't have to hard-code it. Provide it via CEPH_RGW_ENDPOINT (see below); if unset, those keys are simply omitted. This is also the default endpoint for the Admin Ops and S3 API calls when CEPH_RGW_ADMIN_ENDPOINT is not set separately.

---

5. Give the operator its credentials (the cephrgw-credentials Secret)

The operator reads its configuration from environment variables, which the deployment sources from a Secret named cephrgw-credentials in its namespace (cephrgw-system). The Secret data keys map 1:1 to the env vars:

Secret key	Required	Meaning
CEPH_RGW_ACCESS_KEY	yes	access key of the RGW admin user from step 1
CEPH_RGW_SECRET_KEY	yes	its secret key
CEPH_RGW_ENDPOINT	see note	S3 endpoint; written into consumer Secrets and used for API calls unless CEPH_RGW_ADMIN_ENDPOINT is set
CEPH_RGW_ADMIN_ENDPOINT	no	radosgw endpoint for the Admin Ops + S3 API calls, if it differs from the public CEPH_RGW_ENDPOINT
CEPH_RGW_REGION	no	SigV4 credential-scope region for S3 requests (default default)
CEPH_RGW_CA	no	PEM CA bundle to verify the radosgw TLS cert (inline)
CEPH_RGW_CA_FILE	no	path to a mounted CA file (alternative to the above)
CEPH_RGW_INSECURE	no	"true" to skip TLS verification (dev only)

At least one of CEPH_RGW_ENDPOINT or CEPH_RGW_ADMIN_ENDPOINT must be set — the API endpoint falls back to CEPH_RGW_ENDPOINT when the admin one is unset.

Primary method: Vault + VSO

The Secret is not managed in GitOps; it is rendered from Vault by the Vault Secrets Operator (VSO). The argocd-apps cephrgw-system app ships a VaultAuth and a VaultStaticSecret that authenticate with the shared default Kubernetes auth role and render the KV path kubernetes/namespace/cephrgw-system/default/cephrgw-credentials into the cephrgw-credentials Secret. That path sits under the cluster's templated default policy (kv/data/kubernetes/namespace/<ns>/<sa>/*), so no dedicated Vault role or policy is required — you only seed the values:

vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
  CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
  CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
  CEPH_RGW_ACCESS_KEY='REPLACE-WITH-ACCESS-KEY' \
  CEPH_RGW_SECRET_KEY='REPLACE-WITH-SECRET-KEY'

The keys under that KV path are copied verbatim into the Secret, so they must be named exactly as the table above. VSO refreshes the Secret every few minutes, and the deployment's reloader.stakater.com/auto: "true" annotation restarts the operator when it changes — so rotating the credential is just a new vault kv put, no manual rollout.

Fallback: a plain Secret

Outside this cluster (or for a quick kind test) you can create the Secret directly instead of using Vault:

kubectl -n cephrgw-system create secret generic cephrgw-credentials \
  --from-literal=CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
  --from-literal=CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
  --from-literal=CEPH_RGW_ACCESS_KEY='REPLACE-WITH-ACCESS-KEY' \
  --from-literal=CEPH_RGW_SECRET_KEY='REPLACE-WITH-SECRET-KEY'

The operator does not care where the Secret comes from, only that those keys exist.

---

Quick verification

Once the Secret and RGW admin user exist, a smoke test from your workstation using the operator's keys (this is the same Admin Ops call the operator's readiness Ping makes):

# Signing an Admin Ops request by hand is fiddly; the simplest proof is to use
# the AWS CLI configured with the operator's keys against the S3 endpoint:
AWS_ACCESS_KEY_ID=REPLACE-WITH-ACCESS-KEY \
AWS_SECRET_ACCESS_KEY=REPLACE-WITH-SECRET-KEY \
aws --endpoint-url https://s3.ceph.unkin.net s3 ls

A successful (even empty) listing proves the keys and endpoint work. If the operator logs initial radosgw authentication failed, the keys are wrong or the admin API is disabled (steps 1–2); if users are created but bucket policy grants fail, the cluster is likely pre-Reef (step 3).