benvin
1ea1713d6e
Adds a Kubernetes operator that provisions Ceph RGW (S3) buckets and access keys declaratively through the Ceph manager dashboard REST API. Three CRDs in group ceph.unkin.net/v1alpha1: - ObjectStoreUser: creates an RGW user, delivers its key pair to a Secret - Bucket: creates an S3 bucket owned by an ObjectStoreUser; owns the bucket's aggregate S3 policy (union of all BucketAccess grants) - BucketAccess: grants read-only/read-write/full access, provisioning a dedicated user (or reusing a referenced one) and delivering RW/RO keys The internal/ceph client wraps the dashboard /api/auth, /api/rgw/user and /api/rgw/bucket endpoints with lazy token auth and re-auth on 401. Bucket policies are rendered deterministically and applied via the bucket policy API (Reef 18.2+). Credentials come from the cephrgw-credentials Secret via env. Includes generated CRDs/RBAC, samples, kind manifests, Woodpecker CI, and docs/ceph-setup.md covering the required Ceph dashboard account, RGW wiring and permissions.
176 lines
6.7 KiB
Markdown
176 lines
6.7 KiB
Markdown
# Ceph setup: credentials and permissions the operator needs
|
||
|
||
`cephrgw-operator` never talks to RADOS or the RGW admin socket directly. It
|
||
drives the **Ceph manager dashboard REST API** (the same API the web dashboard
|
||
uses) at `https://dashboard.ceph.unkin.net`. Everything below is about giving
|
||
the operator a dashboard login with enough RGW authority, and making sure the
|
||
dashboard itself is wired to your RGW.
|
||
|
||
There are **two** credentials involved. Don't confuse them:
|
||
|
||
| # | Credential | Who uses it | What it is |
|
||
|---|------------|-------------|------------|
|
||
| 1 | Dashboard login (username + password) | the operator → `POST /api/auth` | a **dashboard account** with the `rgw-manager` role |
|
||
| 2 | RGW admin connection | the dashboard → RGW | a **radosgw system user** (access/secret key) the dashboard is configured with |
|
||
|
||
The operator only holds #1. #2 is what actually lets the dashboard create RGW
|
||
users, buckets and bucket policies on the operator's behalf, so it must exist
|
||
and be privileged.
|
||
|
||
---
|
||
|
||
## 1. Create the dashboard login for the operator
|
||
|
||
Create a dedicated dashboard user with the built-in **`rgw-manager`** role. That
|
||
role grants full create/read/update/delete on the dashboard's `rgw` scope
|
||
(users, buckets, policies) and nothing else — least privilege for this operator.
|
||
|
||
```bash
|
||
# Put the password in a file so it never lands in shell history.
|
||
printf '%s' 'REPLACE-WITH-A-STRONG-PASSWORD' > /tmp/cephrgw.pw
|
||
|
||
ceph dashboard ac-user-create k8s-cephrgw-operator -i /tmp/cephrgw.pw rgw-manager
|
||
|
||
rm -f /tmp/cephrgw.pw
|
||
```
|
||
|
||
If your Ceph version wants the arguments in a different order, check
|
||
`ceph dashboard ac-user-create -h`. To confirm the role exists and what it
|
||
grants:
|
||
|
||
```bash
|
||
ceph dashboard ac-role-show rgw-manager
|
||
```
|
||
|
||
> Prefer `rgw-manager` over `administrator`. The operator only needs RGW
|
||
> authority; giving it full dashboard admin is unnecessary blast radius.
|
||
|
||
## 2. Make sure the dashboard can manage RGW
|
||
|
||
The dashboard performs RGW operations through a **radosgw system user**. On
|
||
recent Ceph (Pacific and later) the mgr/dashboard module usually auto-discovers
|
||
and configures this. Verify it first:
|
||
|
||
```bash
|
||
ceph dashboard get-rgw-api-access-key # should print a key, not empty
|
||
```
|
||
|
||
If it is empty, create a system user and point the dashboard at it:
|
||
|
||
```bash
|
||
radosgw-admin user create \
|
||
--uid=dashboard \
|
||
--display-name="Ceph Dashboard" \
|
||
--system
|
||
|
||
# Feed the returned keys to the dashboard.
|
||
radosgw-admin user info --uid=dashboard \
|
||
| jq -r '.keys[0].access_key' > /tmp/ak
|
||
radosgw-admin user info --uid=dashboard \
|
||
| jq -r '.keys[0].secret_key' > /tmp/sk
|
||
|
||
ceph dashboard set-rgw-api-access-key -i /tmp/ak
|
||
ceph dashboard set-rgw-api-secret-key -i /tmp/sk
|
||
rm -f /tmp/ak /tmp/sk
|
||
```
|
||
|
||
A `--system` user has the admin caps the dashboard needs to create/delete RGW
|
||
users and buckets and to set bucket policies on any bucket. If you would rather
|
||
not use `--system`, grant an equivalent admin cap set instead:
|
||
|
||
```bash
|
||
radosgw-admin caps add --uid=dashboard \
|
||
--caps="users=*;buckets=*;metadata=*;usage=read;zone=read"
|
||
```
|
||
|
||
If the dashboard reaches RGW over TLS with a private CA, you may also need:
|
||
|
||
```bash
|
||
ceph dashboard set-rgw-api-ssl-verify true # keep verification on in prod
|
||
```
|
||
|
||
## 3. Bucket policy support (read-only / non-owner read-write)
|
||
|
||
The operator enforces `read-only` and non-owner `read-write` grants by writing
|
||
an **S3 bucket policy** through the dashboard's bucket API (the `bucket_policy`
|
||
field on `PUT /api/rgw/bucket/{name}`). That field is available on **Ceph Reef
|
||
18.2+ / Squid**. On older releases bucket creation and owner (`full`) access
|
||
still work, but policy-based grants will fail — upgrade the cluster, or only use
|
||
owner credentials, if you are pre-Reef.
|
||
|
||
Check your version:
|
||
|
||
```bash
|
||
ceph versions | jq -r '.mon | keys[]'
|
||
```
|
||
|
||
## 4. (Optional) S3 endpoint for consumers
|
||
|
||
The operator can stamp the S3 endpoint into every credential Secret it writes
|
||
(`S3_ENDPOINT` and `BUCKET_HOST`) so applications don't have to hard-code it.
|
||
This is the RGW/S3 endpoint your clients use — **not** the dashboard URL. Provide
|
||
it via `CEPH_RGW_ENDPOINT` (see below); if unset, those keys are simply omitted.
|
||
|
||
---
|
||
|
||
## 5. Give the operator its credentials (the `cephrgw-credentials` Secret)
|
||
|
||
The operator reads its configuration from environment variables, which the
|
||
deployment sources from a Secret named **`cephrgw-credentials`** in its namespace
|
||
(`cephrgw-system`). The Secret data keys map 1:1 to the env vars:
|
||
|
||
| Secret key | Required | Meaning |
|
||
|------------|----------|---------|
|
||
| `CEPH_DASHBOARD_URL` | yes | dashboard base URL, e.g. `https://dashboard.ceph.unkin.net` |
|
||
| `CEPH_DASHBOARD_USERNAME` | yes | the `rgw-manager` account from step 1 |
|
||
| `CEPH_DASHBOARD_PASSWORD` | yes | its password |
|
||
| `CEPH_RGW_ENDPOINT` | no | S3 endpoint written into consumer Secrets |
|
||
| `CEPH_DASHBOARD_CA` | no | PEM CA bundle to verify the dashboard TLS cert (inline) |
|
||
| `CEPH_DASHBOARD_CA_FILE` | no | path to a mounted CA file (alternative to the above) |
|
||
| `CEPH_DASHBOARD_INSECURE` | no | `"true"` to skip TLS verification (dev only) |
|
||
|
||
Create it directly:
|
||
|
||
```bash
|
||
kubectl -n cephrgw-system create secret generic cephrgw-credentials \
|
||
--from-literal=CEPH_DASHBOARD_URL=https://dashboard.ceph.unkin.net \
|
||
--from-literal=CEPH_DASHBOARD_USERNAME=k8s-cephrgw-operator \
|
||
--from-literal=CEPH_DASHBOARD_PASSWORD='REPLACE-WITH-A-STRONG-PASSWORD' \
|
||
--from-literal=CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net
|
||
```
|
||
|
||
The deployment carries the `reloader.stakater.com/auto: "true"` annotation, so
|
||
rotating this Secret triggers an automatic operator restart — no manual rollout
|
||
needed.
|
||
|
||
### Sourcing it from Vault (optional)
|
||
|
||
If you keep the password in Vault, sync it with a `VaultStaticSecret` (VSO is
|
||
already running in `vso-system`) that renders into `cephrgw-credentials` with
|
||
the keys above, instead of the plain `kubectl create secret`. The operator does
|
||
not care where the Secret comes from, only that those keys exist.
|
||
|
||
---
|
||
|
||
## Quick verification
|
||
|
||
Once the Secret and dashboard account exist, a smoke test from your workstation:
|
||
|
||
```bash
|
||
# 1. Log in and capture a token.
|
||
TOKEN=$(curl -sk -X POST https://dashboard.ceph.unkin.net/api/auth \
|
||
-H 'Accept: application/vnd.ceph.api.v1.0+json' \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"username":"k8s-cephrgw-operator","password":"REPLACE-WITH-A-STRONG-PASSWORD"}' \
|
||
| jq -r .token)
|
||
|
||
# 2. List RGW users — a 200 with a JSON array means the role + RGW wiring work.
|
||
curl -sk https://dashboard.ceph.unkin.net/api/rgw/user \
|
||
-H 'Accept: application/vnd.ceph.api.v1.0+json' \
|
||
-H "Authorization: Bearer $TOKEN"
|
||
```
|
||
|
||
If step 1 fails the login/role is wrong (step 1–2 above); if step 1 works but
|
||
step 2 returns 500/empty, the dashboard→RGW connection is not configured
|
||
(step 2).
|