unkinben
466514063a
The operator drove the Ceph manager dashboard REST API to manage RGW users, buckets and policies. That coupled it to a dashboard login, the dashboard's RGW wiring, and the dashboard's bucket API surface. Rebuild the Ceph integration to talk directly to radosgw the way the CLI does, using native Go libraries, while keeping every operator capability identical. The exported surface of internal/ceph is unchanged, so the three controllers and cmd/operator's structure are untouched (bar the CEPH_RGW_* config plumbing). - replace the internal/ceph client internals with github.com/ceph/go-ceph rgw/admin (Admin Ops API) for users, keys, quotas and bucket info/removal - add github.com/aws/aws-sdk-go-v2 S3 client for bucket create, versioning, policy, tagging and object lock, signed as the bucket owner - map go-ceph admin.ErrNoSuch*/ErrUserExists and smithy APIError codes into IsNotFound/IsConflict so controller create-vs-update branching is preserved - set S3 path-style addressing and WhenRequired checksum modes for RGW - delete the hand-rolled dashboard client, token auth and JSON plumbing - keep policy.go/BuildBucketPolicy/BuildTagJSON as pure builders - replace the client tests with NewClient validation and error-classifier tests - keep CGO_ENABLED=0 distroless: only go-ceph's pure-Go rgw/admin is imported - switch env/config to CEPH_RGW_* (endpoint, admin endpoint, access/secret key, region, CA, insecure) and update the deployment manifest - rewrite README and docs/ceph-setup.md for the single RGW admin user (caps users=*;buckets=*), keeping Vault/VSO as the primary credential source Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
169 lines
7.0 KiB
Markdown
169 lines
7.0 KiB
Markdown
# Ceph setup: credentials and permissions the operator needs
|
||
|
||
`cephrgw-operator` talks **directly to radosgw**, the same way the `radosgw-admin`
|
||
CLI and S3 clients do — no manager dashboard involved. It uses two native Go
|
||
libraries against the RGW endpoint (e.g. `https://radosgw.service.consul:443`):
|
||
|
||
- **go-ceph `rgw/admin`** → the RGW **Admin Ops API** (`/admin/user`,
|
||
`/admin/bucket`), signed with the operator's access/secret key, for users,
|
||
keys, quotas and bucket info/removal.
|
||
- **aws-sdk-go-v2** → the **S3 API**, signed as each bucket's **owner**, for
|
||
bucket creation, versioning, policy, tagging and object lock.
|
||
|
||
So there is exactly **one** credential to provision: a radosgw user with admin
|
||
caps, plus its access/secret key.
|
||
|
||
---
|
||
|
||
## 1. Create the operator's RGW admin user
|
||
|
||
Create a dedicated radosgw user and give it the admin caps the operator needs.
|
||
Only `users` and `buckets` caps are required (the operator never reads usage or
|
||
metadata endpoints):
|
||
|
||
```bash
|
||
radosgw-admin user create \
|
||
--uid=cephrgw-operator \
|
||
--display-name="cephrgw-operator" \
|
||
--caps="users=*;buckets=*"
|
||
|
||
# Grab its keys (these become CEPH_RGW_ACCESS_KEY / CEPH_RGW_SECRET_KEY):
|
||
radosgw-admin user info --uid=cephrgw-operator \
|
||
| jq -r '.keys[0] | .access_key, .secret_key'
|
||
```
|
||
|
||
If the user already exists, add the caps instead:
|
||
|
||
```bash
|
||
radosgw-admin caps add --uid=cephrgw-operator --caps="users=*;buckets=*"
|
||
```
|
||
|
||
> `users=*;buckets=*` lets the operator create/read/delete RGW users and read/
|
||
> remove buckets through the Admin Ops API. Bucket **creation** and all bucket
|
||
> sub-resources (versioning, policy, tagging, object lock) go over the S3 API
|
||
> signed as the bucket owner, so they need no extra admin cap — every RGW user
|
||
> can manage its own buckets. The `--system` flag is **not** required.
|
||
|
||
## 2. Admin Ops API must be enabled on radosgw
|
||
|
||
The Admin Ops API is served by radosgw at the `admin` resource and is enabled by
|
||
default. If your deployment has trimmed `rgw_enable_apis`, make sure it includes
|
||
both `s3` and `admin`:
|
||
|
||
```
|
||
rgw_enable_apis = s3, admin
|
||
```
|
||
|
||
Quick check from your workstation (a `403`/`AccessDenied` still proves the
|
||
endpoint is reachable and the API is on; a connection error means it is not):
|
||
|
||
```bash
|
||
curl -sk "https://radosgw.service.consul:443/admin/user?format=json"
|
||
```
|
||
|
||
## 3. Bucket policy support (read-only / non-owner read-write)
|
||
|
||
The operator enforces `read-only` and non-owner `read-write` grants by writing an
|
||
**S3 bucket policy** (`PutBucketPolicy`). Bucket-policy support is available on
|
||
**Ceph Reef 18.2+ / Squid**. On older releases bucket creation and owner
|
||
(`full`) access still work, but policy-based grants will fail — upgrade the
|
||
cluster, or only use owner credentials, if you are pre-Reef.
|
||
|
||
Check your version:
|
||
|
||
```bash
|
||
ceph versions | jq -r '.mon | keys[]'
|
||
```
|
||
|
||
## 4. (Optional) S3 endpoint for consumers
|
||
|
||
The operator can stamp the S3 endpoint into every credential Secret it writes
|
||
(`S3_ENDPOINT` and `BUCKET_HOST`) so applications don't have to hard-code it.
|
||
Provide it via `CEPH_RGW_ENDPOINT` (see below); if unset, those keys are simply
|
||
omitted. This is also the default endpoint for the Admin Ops and S3 API calls
|
||
when `CEPH_RGW_ADMIN_ENDPOINT` is not set separately.
|
||
|
||
---
|
||
|
||
## 5. Give the operator its credentials (the `cephrgw-credentials` Secret)
|
||
|
||
The operator reads its configuration from environment variables, which the
|
||
deployment sources from a Secret named **`cephrgw-credentials`** in its namespace
|
||
(`cephrgw-system`). The Secret data keys map 1:1 to the env vars:
|
||
|
||
| Secret key | Required | Meaning |
|
||
|------------|----------|---------|
|
||
| `CEPH_RGW_ACCESS_KEY` | yes | access key of the RGW admin user from step 1 |
|
||
| `CEPH_RGW_SECRET_KEY` | yes | its secret key |
|
||
| `CEPH_RGW_ENDPOINT` | see note | S3 endpoint; written into consumer Secrets and used for API calls unless `CEPH_RGW_ADMIN_ENDPOINT` is set |
|
||
| `CEPH_RGW_ADMIN_ENDPOINT` | no | radosgw endpoint for the Admin Ops + S3 API calls, if it differs from the public `CEPH_RGW_ENDPOINT` |
|
||
| `CEPH_RGW_REGION` | no | SigV4 credential-scope region for S3 requests (default `default`) |
|
||
| `CEPH_RGW_CA` | no | PEM CA bundle to verify the radosgw TLS cert (inline) |
|
||
| `CEPH_RGW_CA_FILE` | no | path to a mounted CA file (alternative to the above) |
|
||
| `CEPH_RGW_INSECURE` | no | `"true"` to skip TLS verification (dev only) |
|
||
|
||
> At least one of `CEPH_RGW_ENDPOINT` or `CEPH_RGW_ADMIN_ENDPOINT` must be set —
|
||
> the API endpoint falls back to `CEPH_RGW_ENDPOINT` when the admin one is unset.
|
||
|
||
### Primary method: Vault + VSO
|
||
|
||
The Secret is not managed in GitOps; it is rendered from Vault by the Vault
|
||
Secrets Operator (VSO). The argocd-apps `cephrgw-system` app ships a `VaultAuth`
|
||
and a `VaultStaticSecret` that authenticate with the shared `default` Kubernetes
|
||
auth role and render the KV path
|
||
`kubernetes/namespace/cephrgw-system/default/cephrgw-credentials` into the
|
||
`cephrgw-credentials` Secret. That path sits under the cluster's templated
|
||
default policy (`kv/data/kubernetes/namespace/<ns>/<sa>/*`), so **no dedicated
|
||
Vault role or policy is required** — you only seed the values:
|
||
|
||
```bash
|
||
vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
|
||
CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
|
||
CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
|
||
CEPH_RGW_ACCESS_KEY='REPLACE-WITH-ACCESS-KEY' \
|
||
CEPH_RGW_SECRET_KEY='REPLACE-WITH-SECRET-KEY'
|
||
```
|
||
|
||
The keys under that KV path are copied verbatim into the Secret, so they must be
|
||
named exactly as the table above. VSO refreshes the Secret every few minutes,
|
||
and the deployment's `reloader.stakater.com/auto: "true"` annotation restarts the
|
||
operator when it changes — so rotating the credential is just a new `vault kv
|
||
put`, no manual rollout.
|
||
|
||
### Fallback: a plain Secret
|
||
|
||
Outside this cluster (or for a quick `kind` test) you can create the Secret
|
||
directly instead of using Vault:
|
||
|
||
```bash
|
||
kubectl -n cephrgw-system create secret generic cephrgw-credentials \
|
||
--from-literal=CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
|
||
--from-literal=CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
|
||
--from-literal=CEPH_RGW_ACCESS_KEY='REPLACE-WITH-ACCESS-KEY' \
|
||
--from-literal=CEPH_RGW_SECRET_KEY='REPLACE-WITH-SECRET-KEY'
|
||
```
|
||
|
||
The operator does not care where the Secret comes from, only that those keys
|
||
exist.
|
||
|
||
---
|
||
|
||
## Quick verification
|
||
|
||
Once the Secret and RGW admin user exist, a smoke test from your workstation
|
||
using the operator's keys (this is the same Admin Ops call the operator's
|
||
readiness `Ping` makes):
|
||
|
||
```bash
|
||
# Signing an Admin Ops request by hand is fiddly; the simplest proof is to use
|
||
# the AWS CLI configured with the operator's keys against the S3 endpoint:
|
||
AWS_ACCESS_KEY_ID=REPLACE-WITH-ACCESS-KEY \
|
||
AWS_SECRET_ACCESS_KEY=REPLACE-WITH-SECRET-KEY \
|
||
aws --endpoint-url https://s3.ceph.unkin.net s3 ls
|
||
```
|
||
|
||
A successful (even empty) listing proves the keys and endpoint work. If the
|
||
operator logs `initial radosgw authentication failed`, the keys are wrong or the
|
||
`admin` API is disabled (steps 1–2); if users are created but bucket policy
|
||
grants fail, the cluster is likely pre-Reef (step 3).
|