unkinben
4b0430f0df
The BucketAccess model only offered three coarse levels (read-only/read-write/ full) applied to the whole bucket. Real grants often need to be scoped to a key prefix, limited to a source network or TLS, restricted to specific actions, or expressed as an arbitrary S3 statement. RGW (Reef 18.2+/Squid) honours the S3 bucket-policy features to do all of this; expose them on BucketAccess while keeping the level as the ergonomic default. - add BucketAccess spec fields: paths (key-prefix scoping), actions (action override), conditions (sourceIPs + secureTransportOnly), rawStatements (arbitrary S3 statements with the principal injected) - extend ceph.Grant + BuildBucketPolicy to render prefixed object resources, custom-action statements, S3 condition blocks, and raw statements, keeping output deterministic (sorted, stable sids) - translate the new spec fields into grants in the Bucket controller and fingerprint grants so distinct fine-grained BucketAccess objects no longer collapse on UID+level alone - regenerate deepcopy + CRDs; add config/samples/04-access-fine-grained.yaml - cover paths, action override, conditions, raw statements and determinism in policy_test.go; document the fields in the README Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
145 lines
6.1 KiB
Markdown
145 lines
6.1 KiB
Markdown
# cephrgw-operator
|
|
|
|
A Kubernetes operator that provisions Ceph RGW (S3) **buckets** and **access
|
|
keys** declaratively, talking **directly to radosgw** the way the CLI does. You
|
|
describe a bucket, its owner, and who may read or write it as custom resources;
|
|
the operator creates the RGW users and bucket, delivers the access/secret keys
|
|
into Kubernetes Secrets, and maintains the bucket's S3 policy.
|
|
|
|
It uses native Go libraries against radosgw (e.g.
|
|
`https://radosgw.service.consul:443`) — no manager dashboard, no RADOS access,
|
|
no admin socket, no in-cluster Ceph required:
|
|
|
|
- **[go-ceph](https://github.com/ceph/go-ceph) `rgw/admin`** drives the RGW
|
|
**Admin Ops API** (`/admin/...`) for users, keys, quotas and bucket
|
|
info/removal — pure Go, no cgo.
|
|
- **[aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2)** drives the **S3 API**
|
|
for bucket creation, versioning, policy, tagging and object lock — the
|
|
operations the Admin Ops API does not expose. These are signed as the bucket
|
|
**owner**, so the owner owns the bucket directly.
|
|
|
|
## Custom resources
|
|
|
|
| Kind | Short | Purpose |
|
|
|------|-------|---------|
|
|
| `ObjectStoreUser` | `osu` | An RGW S3 user. The operator creates it and writes its key pair into a Secret. |
|
|
| `Bucket` | `bkt` | An S3 bucket owned by an `ObjectStoreUser`. Owns the bucket's aggregate S3 policy. |
|
|
| `BucketAccess` | `ba` | Grants a user `read-only`, `read-write` or `full` access to a `Bucket`, delivering RW/RO keys. |
|
|
|
|
### How access levels work
|
|
|
|
The bucket **owner** (`Bucket.spec.ownerRef`) always has full control. Each
|
|
`BucketAccess` adds a principal to the bucket's S3 policy:
|
|
|
|
- `read-only` → `s3:GetObject`, `s3:ListBucket` and friends.
|
|
- `read-write` → read plus `s3:PutObject` / `s3:DeleteObject` / multipart.
|
|
- `full` → `s3:*` on the bucket and its objects.
|
|
|
|
If a `BucketAccess` omits `userRef`, the operator provisions a **dedicated** RGW
|
|
user for that grant and writes its keys into `spec.secretName` (default
|
|
`<name>-rgw`). If `userRef` names an existing `ObjectStoreUser`, that user's own
|
|
credential Secret is reused and only the policy is extended.
|
|
|
|
#### Fine-grained grants
|
|
|
|
The level is the ergonomic default; four optional fields on `BucketAccess`
|
|
refine it (see `config/samples/04-access-fine-grained.yaml`):
|
|
|
|
- `spec.paths` — scope object access to key prefixes; each becomes the resource
|
|
`<bucket>/<prefix>*`. The bucket-level `ListBucket` still spans the whole
|
|
bucket.
|
|
- `spec.actions` — grant exactly these S3 actions instead of the level's set (on
|
|
the bucket and its, optionally prefixed, objects).
|
|
- `spec.conditions` — `sourceIPs` (an `aws:SourceIp` CIDR allowlist) and
|
|
`secureTransportOnly` (require TLS).
|
|
- `spec.rawStatements` — an escape hatch of raw S3 policy statements
|
|
(`effect`/`actions`/`resources`/`conditions`) merged for this grant's
|
|
principal. When set, `level`, `actions`, `paths` and `conditions` are ignored;
|
|
resources without an `arn:` prefix are treated as bucket-relative key prefixes.
|
|
|
|
RGW honours S3 bucket policy on **Reef 18.2+ / Squid**; condition-key support is
|
|
a subset of AWS, so validate exotic conditions against your cluster.
|
|
|
|
The `Bucket` controller renders the policy as the **union of every ready
|
|
`BucketAccess`** that targets it, so the result is convergent regardless of the
|
|
order objects are created or deleted. It watches `BucketAccess` and
|
|
`ObjectStoreUser`, re-reconciling the bucket whenever a grant or user changes.
|
|
|
|
```
|
|
ObjectStoreUser ──admin PUT /admin/user──────▶ Secret (AK/SK)
|
|
Bucket ──S3 CreateBucket (as owner)─▶ owns S3 policy
|
|
BucketAccess ──admin PUT /admin/user──────▶ Secret (AK/SK, RW or RO)
|
|
└────── enqueues Bucket ──▶ S3 PutBucketPolicy (aggregate)
|
|
```
|
|
|
|
## Credential Secrets
|
|
|
|
Every credential Secret carries the conventional keys, ready to mount straight
|
|
into a workload:
|
|
|
|
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
|
|
- `RGW_UID`
|
|
- `S3_ENDPOINT`, `BUCKET_HOST` (when `CEPH_RGW_ENDPOINT` is set)
|
|
- `BUCKET_NAME` (on `BucketAccess` Secrets)
|
|
|
|
Secrets are owner-referenced by the resource that produced them, so they are
|
|
garbage-collected when the resource is deleted.
|
|
|
|
## Prerequisites
|
|
|
|
The operator needs an RGW user with admin caps (`users=*;buckets=*`) and its
|
|
access/secret key, the radosgw endpoint, and (for `read-only`/non-owner
|
|
`read-write` grants) Ceph **Reef 18.2+ / Squid**. See
|
|
**[docs/ceph-setup.md](docs/ceph-setup.md)** for the exact commands and the
|
|
`cephrgw-credentials` Secret schema.
|
|
|
|
## Quickstart
|
|
|
|
```sh
|
|
kubectl apply -f config/samples/00-owner-user.yaml
|
|
kubectl apply -f config/samples/01-bucket.yaml
|
|
kubectl apply -f config/samples/02-access-readonly.yaml
|
|
kubectl apply -f config/samples/03-access-readwrite.yaml
|
|
|
|
kubectl get osu,bkt,ba
|
|
kubectl get secret app-data-ro-rgw -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d
|
|
```
|
|
|
|
## Development
|
|
|
|
```sh
|
|
make generate # regenerate deepcopy, CRDs and RBAC from kubebuilder markers
|
|
make build # build the operator binary
|
|
make test # go test -race
|
|
make lint fmt # go vet / gofmt
|
|
```
|
|
|
|
### Local (kind)
|
|
|
|
```sh
|
|
kind create cluster --name cephrgw
|
|
docker build -t cephrgw-operator:dev -f Dockerfile.operator .
|
|
kind load docker-image cephrgw-operator:dev --name cephrgw
|
|
|
|
kubectl apply -f config/crd/bases/
|
|
kubectl apply -f hack/kind/manifests/ # edit the Secret first
|
|
```
|
|
|
|
## CI
|
|
|
|
Woodpecker runs `pre-commit` (gofmt + vet), `test`, and a dry-run image `build`
|
|
on pull requests; pushing a `v*` tag builds and pushes
|
|
`git.unkin.net/unkin/cephrgw-operator` to the Gitea registry. Bump a release
|
|
with `make patch|minor|major`.
|
|
|
|
## Notes & caveats
|
|
|
|
- **Policy clearing.** Removing the last `BucketAccess` issues an S3
|
|
`DeleteBucketPolicy`. A `NoSuchBucketPolicy` response is treated as already
|
|
clear. Adding/replacing grants always works.
|
|
- **Per-bucket quota.** `Bucket.spec.quota` is applied as the owner's default
|
|
bucket quota via the Admin Ops API, which is per-owner rather than strictly
|
|
per-bucket. Use distinct owners if you need independent bucket quotas.
|
|
- **Immutability.** `bucketName`, an `ObjectStoreUser`'s `uid`, and object lock
|
|
are fixed at creation; changing them on an existing object has no effect.
|