cephrgw-operator/docs/ceph-setup.md

# Ceph setup: credentials and permissions the operator needs

`cephrgw-operator` never talks to RADOS or the RGW admin socket directly. It
drives the **Ceph manager dashboard REST API** (the same API the web dashboard
uses) at `https://dashboard.ceph.unkin.net`. Everything below is about giving
the operator a dashboard login with enough RGW authority, and making sure the
dashboard itself is wired to your RGW.

There are **two** credentials involved. Don't confuse them:

| # | Credential | Who uses it | What it is |
|---|------------|-------------|------------|
| 1 | Dashboard login (username + password) | the operator → `POST /api/auth` | a **dashboard account** with the `rgw-manager` role |
| 2 | RGW admin connection | the dashboard → RGW | a **radosgw system user** (access/secret key) the dashboard is configured with |

The operator only holds #1. #2 is what actually lets the dashboard create RGW
users, buckets and bucket policies on the operator's behalf, so it must exist
and be privileged.

---

## 1. Create the dashboard login for the operator

Create a dedicated dashboard user with the built-in **`rgw-manager`** role. That
role grants full create/read/update/delete on the dashboard's `rgw` scope
(users, buckets, policies) and nothing else — least privilege for this operator.

```bash
# Put the password in a file so it never lands in shell history.
printf '%s' 'REPLACE-WITH-A-STRONG-PASSWORD' > /tmp/cephrgw.pw

ceph dashboard ac-user-create k8s-cephrgw-operator -i /tmp/cephrgw.pw rgw-manager

rm -f /tmp/cephrgw.pw
```

If your Ceph version wants the arguments in a different order, check
`ceph dashboard ac-user-create -h`. To confirm the role exists and what it
grants:

```bash
ceph dashboard ac-role-show rgw-manager
```

> Prefer `rgw-manager` over `administrator`. The operator only needs RGW
> authority; giving it full dashboard admin is unnecessary blast radius.

## 2. Make sure the dashboard can manage RGW

The dashboard performs RGW operations through a **radosgw system user**. On
recent Ceph (Pacific and later) the mgr/dashboard module usually auto-discovers
and configures this. Verify it first:

```bash
ceph dashboard get-rgw-api-access-key   # should print a key, not empty
```

If it is empty, create a system user and point the dashboard at it:

```bash
radosgw-admin user create \
  --uid=dashboard \
  --display-name="Ceph Dashboard" \
  --system

# Feed the returned keys to the dashboard.
radosgw-admin user info --uid=dashboard \
  | jq -r '.keys[0].access_key' > /tmp/ak
radosgw-admin user info --uid=dashboard \
  | jq -r '.keys[0].secret_key' > /tmp/sk

ceph dashboard set-rgw-api-access-key -i /tmp/ak
ceph dashboard set-rgw-api-secret-key -i /tmp/sk
rm -f /tmp/ak /tmp/sk
```

A `--system` user has the admin caps the dashboard needs to create/delete RGW
users and buckets and to set bucket policies on any bucket. If you would rather
not use `--system`, grant an equivalent admin cap set instead:

```bash
radosgw-admin caps add --uid=dashboard \
  --caps="users=*;buckets=*;metadata=*;usage=read;zone=read"
```

If the dashboard reaches RGW over TLS with a private CA, you may also need:

```bash
ceph dashboard set-rgw-api-ssl-verify true   # keep verification on in prod
```

## 3. Bucket policy support (read-only / non-owner read-write)

The operator enforces `read-only` and non-owner `read-write` grants by writing
an **S3 bucket policy** through the dashboard's bucket API (the `bucket_policy`
field on `PUT /api/rgw/bucket/{name}`). That field is available on **Ceph Reef
18.2+ / Squid**. On older releases bucket creation and owner (`full`) access
still work, but policy-based grants will fail — upgrade the cluster, or only use
owner credentials, if you are pre-Reef.

Check your version:

```bash
ceph versions | jq -r '.mon | keys[]'
```

## 4. (Optional) S3 endpoint for consumers

The operator can stamp the S3 endpoint into every credential Secret it writes
(`S3_ENDPOINT` and `BUCKET_HOST`) so applications don't have to hard-code it.
This is the RGW/S3 endpoint your clients use — **not** the dashboard URL. Provide
it via `CEPH_RGW_ENDPOINT` (see below); if unset, those keys are simply omitted.

---

## 5. Give the operator its credentials (the `cephrgw-credentials` Secret)

The operator reads its configuration from environment variables, which the
deployment sources (via `envFrom`) from a Secret named **`cephrgw-credentials`**
in its namespace (`cephrgw-system`). The Secret data keys map 1:1 to the env
vars:

| Secret key | Required | Meaning |
|------------|----------|---------|
| `CEPH_DASHBOARD_URL` | yes | dashboard base URL, e.g. `https://dashboard.ceph.unkin.net` |
| `CEPH_DASHBOARD_USERNAME` | yes | the `rgw-manager` account from step 1 |
| `CEPH_DASHBOARD_PASSWORD` | yes | its password |
| `CEPH_RGW_ENDPOINT` | no | S3 endpoint written into consumer Secrets |
| `CEPH_DASHBOARD_CA` | no | PEM CA bundle to verify the dashboard TLS cert (inline) |
| `CEPH_DASHBOARD_CA_FILE` | no | path to a mounted CA file (alternative to the above) |
| `CEPH_DASHBOARD_INSECURE` | no | `"true"` to skip TLS verification (dev only) |

### Primary method: Vault + VSO

The Secret is not managed in GitOps; it is rendered from Vault by the Vault
Secrets Operator (VSO). The argocd-apps `cephrgw-system` app ships a `VaultAuth`
and a `VaultStaticSecret` that authenticate with the shared `default` Kubernetes
auth role and render the KV path
`kubernetes/namespace/cephrgw-system/default/cephrgw-credentials` into the
`cephrgw-credentials` Secret. That path sits under the cluster's templated
default policy (`kv/data/kubernetes/namespace/<ns>/<sa>/*`), so **no dedicated
Vault role or policy is required** — you only seed the values:

```bash
vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
  CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
  CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
  CEPH_DASHBOARD_PASSWORD='REPLACE-WITH-A-STRONG-PASSWORD' \
  CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net
```

The keys under that KV path are copied verbatim into the Secret, so they must be
named exactly as the table above. VSO refreshes the Secret every few minutes,
and the deployment's `reloader.stakater.com/auto: "true"` annotation restarts the
operator when it changes — so rotating the credential is just a new `vault kv
put`, no manual rollout.

### Fallback: a plain Secret

Outside this cluster (or for a quick `kind` test) you can create the Secret
directly instead of using Vault:

```bash
kubectl -n cephrgw-system create secret generic cephrgw-credentials \
  --from-literal=CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
  --from-literal=CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
  --from-literal=CEPH_DASHBOARD_PASSWORD='REPLACE-WITH-A-STRONG-PASSWORD' \
  --from-literal=CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net
```

The operator does not care where the Secret comes from, only that those keys
exist.

---

## Quick verification

Once the Secret and dashboard account exist, a smoke test from your workstation:

```bash
# 1. Log in and capture a token.
TOKEN=$(curl -sk -X POST https://dashboard.ceph.unkin.net/api/auth \
  -H 'Accept: application/vnd.ceph.api.v1.0+json' \
  -H 'Content-Type: application/json' \
  -d '{"username":"k8s-cephrgw-operator","password":"REPLACE-WITH-A-STRONG-PASSWORD"}' \
  | jq -r .token)

# 2. List RGW users — a 200 with a JSON array means the role + RGW wiring work.
curl -sk https://dashboard.ceph.unkin.net/api/rgw/user \
  -H 'Accept: application/vnd.ceph.api.v1.0+json' \
  -H "Authorization: Bearer $TOKEN"
```

If step 1 fails the login/role is wrong (step 1–2 above); if step 1 works but
step 2 returns 500/empty, the dashboard→RGW connection is not configured
(step 2).