benvin
1ea1713d6e
Adds a Kubernetes operator that provisions Ceph RGW (S3) buckets and access keys declaratively through the Ceph manager dashboard REST API. Three CRDs in group ceph.unkin.net/v1alpha1: - ObjectStoreUser: creates an RGW user, delivers its key pair to a Secret - Bucket: creates an S3 bucket owned by an ObjectStoreUser; owns the bucket's aggregate S3 policy (union of all BucketAccess grants) - BucketAccess: grants read-only/read-write/full access, provisioning a dedicated user (or reusing a referenced one) and delivering RW/RO keys The internal/ceph client wraps the dashboard /api/auth, /api/rgw/user and /api/rgw/bucket endpoints with lazy token auth and re-auth on 401. Bucket policies are rendered deterministically and applied via the bucket policy API (Reef 18.2+). Credentials come from the cephrgw-credentials Secret via env. Includes generated CRDs/RBAC, samples, kind manifests, Woodpecker CI, and docs/ceph-setup.md covering the required Ceph dashboard account, RGW wiring and permissions.
116 lines
4.7 KiB
Markdown
116 lines
4.7 KiB
Markdown
# cephrgw-operator
|
|
|
|
A Kubernetes operator that provisions Ceph RGW (S3) **buckets** and **access
|
|
keys** declaratively, driving the Ceph **manager dashboard REST API**. You
|
|
describe a bucket, its owner, and who may read or write it as custom resources;
|
|
the operator creates the RGW users and bucket, delivers the access/secret keys
|
|
into Kubernetes Secrets, and maintains the bucket's S3 policy.
|
|
|
|
It talks only to the dashboard API (e.g. `https://dashboard.ceph.unkin.net`) —
|
|
no RADOS access, no admin socket, no in-cluster Ceph required.
|
|
|
|
## Custom resources
|
|
|
|
| Kind | Short | Purpose |
|
|
|------|-------|---------|
|
|
| `ObjectStoreUser` | `osu` | An RGW S3 user. The operator creates it and writes its key pair into a Secret. |
|
|
| `Bucket` | `bkt` | An S3 bucket owned by an `ObjectStoreUser`. Owns the bucket's aggregate S3 policy. |
|
|
| `BucketAccess` | `ba` | Grants a user `read-only`, `read-write` or `full` access to a `Bucket`, delivering RW/RO keys. |
|
|
|
|
### How access levels work
|
|
|
|
The bucket **owner** (`Bucket.spec.ownerRef`) always has full control. Each
|
|
`BucketAccess` adds a principal to the bucket's S3 policy:
|
|
|
|
- `read-only` → `s3:GetObject`, `s3:ListBucket` and friends.
|
|
- `read-write` → read plus `s3:PutObject` / `s3:DeleteObject` / multipart.
|
|
- `full` → `s3:*` on the bucket and its objects.
|
|
|
|
If a `BucketAccess` omits `userRef`, the operator provisions a **dedicated** RGW
|
|
user for that grant and writes its keys into `spec.secretName` (default
|
|
`<name>-rgw`). If `userRef` names an existing `ObjectStoreUser`, that user's own
|
|
credential Secret is reused and only the policy is extended.
|
|
|
|
The `Bucket` controller renders the policy as the **union of every ready
|
|
`BucketAccess`** that targets it, so the result is convergent regardless of the
|
|
order objects are created or deleted. It watches `BucketAccess` and
|
|
`ObjectStoreUser`, re-reconciling the bucket whenever a grant or user changes.
|
|
|
|
```
|
|
ObjectStoreUser ──create user──▶ dashboard /api/rgw/user ──▶ Secret (AK/SK)
|
|
Bucket ──create bucket─▶ dashboard /api/rgw/bucket ─▶ owns S3 policy
|
|
BucketAccess ──ensure user───▶ dashboard /api/rgw/user ──▶ Secret (AK/SK, RW or RO)
|
|
└────── enqueues Bucket ──▶ PUT bucket_policy (aggregate)
|
|
```
|
|
|
|
## Credential Secrets
|
|
|
|
Every credential Secret carries the conventional keys, ready to mount straight
|
|
into a workload:
|
|
|
|
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
|
|
- `RGW_UID`
|
|
- `S3_ENDPOINT`, `BUCKET_HOST` (when `CEPH_RGW_ENDPOINT` is configured)
|
|
- `BUCKET_NAME` (on `BucketAccess` Secrets)
|
|
|
|
Secrets are owner-referenced by the resource that produced them, so they are
|
|
garbage-collected when the resource is deleted.
|
|
|
|
## Prerequisites
|
|
|
|
The operator needs a dashboard login with the `rgw-manager` role, a dashboard
|
|
that is wired to RGW, and (for `read-only`/non-owner `read-write` grants) Ceph
|
|
**Reef 18.2+ / Squid**. See **[docs/ceph-setup.md](docs/ceph-setup.md)** for the
|
|
exact commands and the `cephrgw-credentials` Secret schema.
|
|
|
|
## Quickstart
|
|
|
|
```sh
|
|
kubectl apply -f config/samples/00-owner-user.yaml
|
|
kubectl apply -f config/samples/01-bucket.yaml
|
|
kubectl apply -f config/samples/02-access-readonly.yaml
|
|
kubectl apply -f config/samples/03-access-readwrite.yaml
|
|
|
|
kubectl get osu,bkt,ba
|
|
kubectl get secret app-data-ro-rgw -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d
|
|
```
|
|
|
|
## Development
|
|
|
|
```sh
|
|
make generate # regenerate deepcopy, CRDs and RBAC from kubebuilder markers
|
|
make build # build the operator binary
|
|
make test # go test -race
|
|
make lint fmt # go vet / gofmt
|
|
```
|
|
|
|
### Local (kind)
|
|
|
|
```sh
|
|
kind create cluster --name cephrgw
|
|
docker build -t cephrgw-operator:dev -f Dockerfile.operator .
|
|
kind load docker-image cephrgw-operator:dev --name cephrgw
|
|
|
|
kubectl apply -f config/crd/bases/
|
|
kubectl apply -f hack/kind/manifests/ # edit the Secret first
|
|
```
|
|
|
|
## CI
|
|
|
|
Woodpecker runs `pre-commit` (gofmt + vet), `test`, and a dry-run image `build`
|
|
on pull requests; pushing a `v*` tag builds and pushes
|
|
`git.unkin.net/unkin/cephrgw-operator` to the Gitea registry. Bump a release
|
|
with `make patch|minor|major`.
|
|
|
|
## Notes & caveats
|
|
|
|
- **Policy clearing.** Removing the last `BucketAccess` asks the dashboard to
|
|
clear the bucket policy. Not every release honours an empty policy string; if
|
|
a stale policy lingers, clear it once by hand. Adding/replacing grants always
|
|
works.
|
|
- **Per-bucket quota.** `Bucket.spec.quota` is applied as the owner's default
|
|
bucket quota via the dashboard, which is per-owner rather than strictly
|
|
per-bucket. Use distinct owners if you need independent bucket quotas.
|
|
- **Immutability.** `bucketName`, an `ObjectStoreUser`'s `uid`, and object lock
|
|
are fixed at creation; changing them on an existing object has no effect.
|