cephrgw-operator/README.md

cephrgw-operator

A Kubernetes operator that provisions Ceph RGW (S3) buckets and access keys declaratively, talking directly to radosgw the way the CLI does. You describe a bucket, its owner, and who may read or write it as custom resources; the operator creates the RGW users and bucket, delivers the access/secret keys into Kubernetes Secrets, and maintains the bucket's S3 policy.

It uses native Go libraries against radosgw (e.g. https://radosgw.service.consul:443) — no manager dashboard, no RADOS access, no admin socket, no in-cluster Ceph required:

• go-ceph rgw/admin drives the RGW Admin Ops API (/admin/...) for users, keys, quotas and bucket info/removal — pure Go, no cgo.
• aws-sdk-go-v2 drives the S3 API for bucket creation, versioning, policy, tagging and object lock — the operations the Admin Ops API does not expose. These are signed as the bucket owner, so the owner owns the bucket directly.

Custom resources

Kind	Short	Purpose
ObjectStoreUser	osu	An RGW S3 user. The operator creates it and writes its key pair into a Secret.
Bucket	bkt	An S3 bucket owned by an ObjectStoreUser. Owns the bucket's aggregate S3 policy.
BucketAccess	ba	Grants a user read-only, read-write or full access to a Bucket, delivering RW/RO keys.

How access levels work

The bucket owner (Bucket.spec.ownerRef) always has full control. Each BucketAccess adds a principal to the bucket's S3 policy:

• read-only → s3:GetObject, s3:ListBucket and friends.
• read-write → read plus s3:PutObject / s3:DeleteObject / multipart.
• full → s3:* on the bucket and its objects.

If a BucketAccess omits userRef, the operator provisions a dedicated RGW user for that grant and writes its keys into spec.secretName (default <name>-rgw). If userRef names an existing ObjectStoreUser, that user's own credential Secret is reused and only the policy is extended.

Fine-grained grants

The level is the ergonomic default; four optional fields on BucketAccess refine it (see config/samples/04-access-fine-grained.yaml):

• spec.paths — scope object access to key prefixes; each becomes the resource <bucket>/<prefix>*. The bucket-level ListBucket still spans the whole bucket.
• spec.actions — grant exactly these S3 actions instead of the level's set (on the bucket and its, optionally prefixed, objects).
• spec.conditions — sourceIPs (an aws:SourceIp CIDR allowlist) and secureTransportOnly (require TLS).
• spec.rawStatements — an escape hatch of raw S3 policy statements (effect/actions/resources/conditions) merged for this grant's principal. When set, level, actions, paths and conditions are ignored; resources without an arn: prefix are treated as bucket-relative key prefixes.

RGW honours S3 bucket policy on Reef 18.2+ / Squid; condition-key support is a subset of AWS, so validate exotic conditions against your cluster.

The Bucket controller renders the policy as the union of every ready BucketAccess that targets it, so the result is convergent regardless of the order objects are created or deleted. It watches BucketAccess and ObjectStoreUser, re-reconciling the bucket whenever a grant or user changes.

ObjectStoreUser ──admin PUT /admin/user──────▶ Secret (AK/SK)
Bucket          ──S3 CreateBucket (as owner)─▶ owns S3 policy
BucketAccess    ──admin PUT /admin/user──────▶ Secret (AK/SK, RW or RO)
                        └────── enqueues Bucket ──▶ S3 PutBucketPolicy (aggregate)

Credential Secrets

Every credential Secret carries the conventional keys, ready to mount straight into a workload:

• AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
• RGW_UID
• S3_ENDPOINT, BUCKET_HOST (when CEPH_RGW_ENDPOINT is set)
• BUCKET_NAME (on BucketAccess Secrets)

Secrets are owner-referenced by the resource that produced them, so they are garbage-collected when the resource is deleted.

Prerequisites

The operator needs an RGW user with admin caps (users=*;buckets=*) and its access/secret key, the radosgw endpoint, and (for read-only/non-owner read-write grants) Ceph Reef 18.2+ / Squid. See docs/ceph-setup.md for the exact commands and the cephrgw-credentials Secret schema.

Quickstart

kubectl apply -f config/samples/00-owner-user.yaml
kubectl apply -f config/samples/01-bucket.yaml
kubectl apply -f config/samples/02-access-readonly.yaml
kubectl apply -f config/samples/03-access-readwrite.yaml

kubectl get osu,bkt,ba
kubectl get secret app-data-ro-rgw -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d

Development

make generate   # regenerate deepcopy, CRDs and RBAC from kubebuilder markers
make build      # build the operator binary
make test       # go test -race
make lint fmt   # go vet / gofmt

Local (kind)

kind create cluster --name cephrgw
docker build -t cephrgw-operator:dev -f Dockerfile.operator .
kind load docker-image cephrgw-operator:dev --name cephrgw

kubectl apply -f config/crd/bases/
kubectl apply -f hack/kind/manifests/     # edit the Secret first

CI

Woodpecker runs pre-commit (gofmt + vet), test, and a dry-run image build on pull requests; pushing a v* tag builds and pushes git.unkin.net/unkin/cephrgw-operator to the Gitea registry. Bump a release with make patch|minor|major.

Notes & caveats

• Policy clearing. Removing the last BucketAccess issues an S3 DeleteBucketPolicy. A NoSuchBucketPolicy response is treated as already clear. Adding/replacing grants always works.
• Per-bucket quota. Bucket.spec.quota is applied as the owner's default bucket quota via the Admin Ops API, which is per-owner rather than strictly per-bucket. Use distinct owners if you need independent bucket quotas.
• Immutability. bucketName, an ObjectStoreUser's uid, and object lock are fixed at creation; changing them on an existing object has no effect.