21 Commits

Author SHA1 Message Date
benvin a89e38be64 Merge pull request 'ci: add buildkit_config CA trust for artifactapi push' (#10) from benvin/buildx-ca-config into main
Reviewed-on: #10
2026-08-15 18:47:28 +10:00
unkin-agent ede6604ab1 ci: add buildkit_config CA trust for artifactapi push
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-08-15 18:30:44 +10:00
benvin f60e69bb79 Merge pull request 'ci: use CA-baked plugin-docker-buildx image for artifactapi push' (#9) from benvin/buildx-ca-plugin-image into main
Reviewed-on: #9
2026-08-15 18:20:25 +10:00
unkin-agent 5f0b9f53ad ci: use CA-baked plugin-docker-buildx image for artifactapi push
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-08-15 18:03:55 +10:00
benvin 7f1f57eb8d Merge pull request 'ci: push images to artifactapi registry instead of gitea' (#8) from benvin/push-artifactapi into main
Reviewed-on: #8
2026-07-30 20:55:17 +10:00
unkinben d3b022d646 ci: push images to artifactapi registry instead of gitea
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Hard switch of the docker push target from the Gitea registry to the
artifactapi local docker registry (docker-internal); the Gitea VM and its
registry are being retired. Drops the droneci/DRONECI_PASSWORD creds since
artifactapi accepts unauthenticated in-cluster pushes. Also updates the README push note.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-30 00:34:59 +10:00
benvin 343d60cfcf Merge pull request 'Add immutable placement-target selection to Bucket' (#7) from benvin/placement-targets into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #7
2026-07-29 00:37:48 +10:00
unkinben c1b3ba1c34 Add immutable placement-target selection to Bucket
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
Buckets could not choose which RGW placement target (and thus durability
profile) backs them, so all data landed on the cluster default. The estate's
radosgw exposes two targets - default-placement (3x replicated) and ec (4+1
erasure-coded) - and archival workloads want ec.

- Validate spec.placementTarget: a DNS-ish pattern, 63-char cap, and a CEL
  self==oldSelf immutability rule (RGW fixes placement at bucket creation and
  cannot move a bucket between targets); make spec.zonegroup immutable too.
- Thread the target into the S3 CreateBucket LocationConstraint via the existing
  helper; an empty zonegroup yields ":<target>", selecting the local zonegroup
  so callers need not name the zonegroup api-name.
- Read the live placement_rule and zonegroup back from the Admin Ops bucket
  stats and surface them: status.placementTarget plus a Placement print column.
- Guard the controller: if a live bucket's placement differs from spec, set an
  Error phase with a PlacementImmutable reason instead of deleting/recreating.
- Cover locationConstraint construction, placement readback (httptest), and the
  placementConflict guard with tests; document targets and immutability in the
  README and add config/samples/06-bucket-ec.yaml.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-29 00:16:31 +10:00
benvin 52e183e6f3 Merge pull request 'Warn at startup when installed CRDs are stale or missing' (#6) from benvin/crd-version-warning into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #6
2026-07-25 22:59:42 +10:00
unkinben e7760b79f4 Warn at startup when installed CRDs are stale or missing
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The operator strict-decodes CR specs, so when the in-cluster CRDs lag the
running operator (e.g. CRDs pinned to v0.1.0 while the operator ran v0.3.0),
new spec fields fail to decode with no operator-side signal. This adds an
advisory startup check so the mismatch is visible in the operator logs.

- Add CheckCRDVersions: GET each owned CRD and verify it carries a
  version-sentinel spec field, logging a distinct WARNING per problem
  (missing CRD vs. present-but-stale schema); advisory only, never exits.
- Keep the sentinel list (buckets/managePolicy, objectstoreusers/
  retainOnDelete, bucketaccesses/rawStatements) in one place.
- Wire the check into operator startup after the manager config is available.
- Add apiextensions customresourcedefinitions get;list RBAC marker and
  regenerate config/rbac/role.yaml.
- Promote k8s.io/apiextensions-apiserver to a direct dependency.

Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
2026-07-25 22:42:03 +10:00
benvin 9bbaa2b8ba Merge pull request 'Support adopting existing radosgw buckets and users' (#5) from benvin/safe-adoption into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #5
2026-07-25 10:17:01 +10:00
unkinben 54d3e38223 Support adopting existing radosgw buckets and users
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The operator previously assumed it created every user and bucket it managed:
reconciling an existing resource could overwrite its user attributes or wipe its
bucket policy, and deleting a CRD always deleted the underlying RGW object (only
Bucket had retainOnDelete). That made taking over pre-existing radosgw state
unsafe. Make adoption first-class.

- add retainOnDelete to ObjectStoreUser and BucketAccess (dedicated users), so
  deleting the CRD orphans the RGW user instead of deleting it (symmetric with
  Bucket)
- merge bucket policy instead of replacing it: the operator marks its own
  statements with a cephrgwop* Sid and preserves any statement it does not own,
  so adopting a bucket with a hand-written policy keeps it; add Bucket
  managePolicy (default true) to opt out of policy management entirely
- only reconcile user attributes the spec sets: DisplayName when non-empty and
  Suspended is now an optional *bool, so adopting a user does not reset them
- record adoption: ObjectStoreUser/Bucket status.adopted (+ printcolumn) is true
  when the RGW object already existed on first reconcile
- add GetBucketPolicy + MergeBucketPolicy; keyed adoption detection off the
  status identity field so a Pending owner wait does not mislabel it
- regenerate CRDs/deepcopy; add docs/adoption.md and
  config/samples/05-adoption.yaml; cover the merge in policy_test.go

Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
2026-07-25 00:15:10 +10:00
benvin 619aa6751d Merge pull request 'Add fine-grained bucket access: paths, actions, conditions, raw' (#4) from benvin/bucketaccess-fine-grained-policy into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #4
2026-07-24 23:02:51 +10:00
unkinben 4b0430f0df Add fine-grained bucket access: paths, actions, conditions, raw
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The BucketAccess model only offered three coarse levels (read-only/read-write/
full) applied to the whole bucket. Real grants often need to be scoped to a key
prefix, limited to a source network or TLS, restricted to specific actions, or
expressed as an arbitrary S3 statement. RGW (Reef 18.2+/Squid) honours the S3
bucket-policy features to do all of this; expose them on BucketAccess while
keeping the level as the ergonomic default.

- add BucketAccess spec fields: paths (key-prefix scoping), actions (action
  override), conditions (sourceIPs + secureTransportOnly), rawStatements
  (arbitrary S3 statements with the principal injected)
- extend ceph.Grant + BuildBucketPolicy to render prefixed object resources,
  custom-action statements, S3 condition blocks, and raw statements, keeping
  output deterministic (sorted, stable sids)
- translate the new spec fields into grants in the Bucket controller and
  fingerprint grants so distinct fine-grained BucketAccess objects no longer
  collapse on UID+level alone
- regenerate deepcopy + CRDs; add config/samples/04-access-fine-grained.yaml
- cover paths, action override, conditions, raw statements and determinism in
  policy_test.go; document the fields in the README

Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
2026-07-24 22:48:20 +10:00
benvin 253105f914 Merge pull request 'Talk to radosgw directly via go-ceph + aws-sdk-go-v2' (#3) from benvin/go-ceph-native-client into main
Reviewed-on: #3
2026-07-24 22:43:18 +10:00
unkinben 466514063a Talk to radosgw directly via go-ceph + aws-sdk-go-v2
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The operator drove the Ceph manager dashboard REST API to manage RGW users,
buckets and policies. That coupled it to a dashboard login, the dashboard's RGW
wiring, and the dashboard's bucket API surface. Rebuild the Ceph integration to
talk directly to radosgw the way the CLI does, using native Go libraries, while
keeping every operator capability identical.

The exported surface of internal/ceph is unchanged, so the three controllers
and cmd/operator's structure are untouched (bar the CEPH_RGW_* config plumbing).

- replace the internal/ceph client internals with github.com/ceph/go-ceph
  rgw/admin (Admin Ops API) for users, keys, quotas and bucket info/removal
- add github.com/aws/aws-sdk-go-v2 S3 client for bucket create, versioning,
  policy, tagging and object lock, signed as the bucket owner
- map go-ceph admin.ErrNoSuch*/ErrUserExists and smithy APIError codes into
  IsNotFound/IsConflict so controller create-vs-update branching is preserved
- set S3 path-style addressing and WhenRequired checksum modes for RGW
- delete the hand-rolled dashboard client, token auth and JSON plumbing
- keep policy.go/BuildBucketPolicy/BuildTagJSON as pure builders
- replace the client tests with NewClient validation and error-classifier tests
- keep CGO_ENABLED=0 distroless: only go-ceph's pure-Go rgw/admin is imported
- switch env/config to CEPH_RGW_* (endpoint, admin endpoint, access/secret key,
  region, CA, insecure) and update the deployment manifest
- rewrite README and docs/ceph-setup.md for the single RGW admin user
  (caps users=*;buckets=*), keeping Vault/VSO as the primary credential source

Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
2026-07-24 22:36:10 +10:00
benvin ab21378f8f Merge pull request 'docs: make Vault/VSO the primary credential method' (#2) from benvin/docs-vault-primary into main
Reviewed-on: #2
2026-07-18 17:00:35 +10:00
benvin fa7d7281f0 docs: make Vault/VSO the primary credential method
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
Document sourcing cephrgw-credentials from Vault via VSO as the primary path,
using the shared default k8s auth role and the templated KV path
kubernetes/namespace/cephrgw-system/default/cephrgw-credentials (no dedicated
Vault role/policy needed). Keep the plain-Secret method as a fallback for
non-cluster/kind use.
2026-07-18 16:27:52 +10:00
benvin c7dcf29858 Merge pull request 'Initial cephrgw-operator implementation' (#1) from benvin/initial-operator into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #1
2026-07-18 11:26:35 +10:00
unkinben 41449d41c9 Merge auto-initialized repo
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2026-07-18 08:31:00 +10:00
benvin 1ea1713d6e Initial cephrgw-operator: Ceph RGW buckets & keys via dashboard API
Adds a Kubernetes operator that provisions Ceph RGW (S3) buckets and
access keys declaratively through the Ceph manager dashboard REST API.

Three CRDs in group ceph.unkin.net/v1alpha1:
- ObjectStoreUser: creates an RGW user, delivers its key pair to a Secret
- Bucket: creates an S3 bucket owned by an ObjectStoreUser; owns the
  bucket's aggregate S3 policy (union of all BucketAccess grants)
- BucketAccess: grants read-only/read-write/full access, provisioning a
  dedicated user (or reusing a referenced one) and delivering RW/RO keys

The internal/ceph client wraps the dashboard /api/auth, /api/rgw/user and
/api/rgw/bucket endpoints with lazy token auth and re-auth on 401. Bucket
policies are rendered deterministically and applied via the bucket
policy API (Reef 18.2+). Credentials come from the cephrgw-credentials
Secret via env. Includes generated CRDs/RBAC, samples, kind manifests,
Woodpecker CI, and docs/ceph-setup.md covering the required Ceph
dashboard account, RGW wiring and permissions.
2026-07-18 00:07:22 +10:00
49 changed files with 6350 additions and 1 deletions
+4
View File
@@ -0,0 +1,4 @@
/bin/
*.out
*.test
.env
+10
View File
@@ -0,0 +1,10 @@
when:
- event: pull_request
steps:
- name: docker-build-operator
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx:latest
settings:
repo: artifactapi.k8s.syd1.au.unkin.net/docker-internal/cephrgw-operator
dockerfile: Dockerfile.operator
dry_run: true
+17
View File
@@ -0,0 +1,17 @@
when:
- event: tag
ref: refs/tags/v*
steps:
- name: docker-operator
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx:latest
settings:
registry: artifactapi.k8s.syd1.au.unkin.net
repo: artifactapi.k8s.syd1.au.unkin.net/docker-internal/cephrgw-operator
dockerfile: Dockerfile.operator
buildkit_config: |
[registry."artifactapi.k8s.syd1.au.unkin.net"]
ca = ["/etc/docker/certs.d/artifactapi.k8s.syd1.au.unkin.net/ca.crt"]
tags:
- ${CI_COMMIT_TAG}
- latest
+9
View File
@@ -0,0 +1,9 @@
when:
- event: pull_request
steps:
- name: pre-commit
image: golang:1.25
commands:
- test -z "$(gofmt -l .)"
- go vet ./...
+8
View File
@@ -0,0 +1,8 @@
when:
- event: pull_request
steps:
- name: test
image: golang:1.25
commands:
- go test -race -count=1 ./api/... ./internal/...
+18
View File
@@ -0,0 +1,18 @@
FROM golang:1.25-alpine AS builder
RUN apk add --no-cache git
WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o cephrgw-operator ./cmd/operator
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /build/cephrgw-operator /usr/local/bin/cephrgw-operator
ENTRYPOINT ["cephrgw-operator"]
+57
View File
@@ -0,0 +1,57 @@
.PHONY: build test lint fmt generate manifests docker-operator clean tidy patch minor major
BINARY_OP := bin/cephrgw-operator
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo "0.0.0-dev")
build: tidy
go build -ldflags="-s -w" -o $(BINARY_OP) ./cmd/operator
test:
go test -race -count=1 ./api/... ./internal/...
lint:
go vet ./...
fmt:
gofmt -w .
CRD_BUNDLE := config/crd/install.yaml
## generate: regenerate deepcopy, CRDs and RBAC from kubebuilder markers, then
## bundle every CRD into a single applyable manifest ($(CRD_BUNDLE)) so it can
## be referenced by a stable raw URL.
generate:
controller-gen object paths="./api/..."
controller-gen crd paths="./api/..." output:crd:artifacts:config=config/crd/bases
controller-gen rbac:roleName=cephrgw-operator paths="./internal/controller/..." output:rbac:dir=config/rbac
printf '# Generated by "make generate". DO NOT EDIT.\n' > $(CRD_BUNDLE)
cat config/crd/bases/*.yaml >> $(CRD_BUNDLE)
manifests: generate
docker-operator:
docker build -t cephrgw-operator:$(VERSION) -f Dockerfile.operator .
clean:
rm -rf bin/
tidy:
go mod tidy
_LATEST := $(shell git tag --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$$' | head -1)
_BASE := $(if $(_LATEST),$(_LATEST),v0.0.0)
_MAJ := $(shell echo $(_BASE) | sed 's/^v//' | cut -d. -f1)
_MIN := $(shell echo $(_BASE) | sed 's/^v//' | cut -d. -f2)
_PAT := $(shell echo $(_BASE) | sed 's/^v//' | cut -d. -f3)
patch:
@NEW=v$(_MAJ).$(_MIN).$(shell expr $(_PAT) + 1); \
git tag $$NEW && echo "Tagged $$NEW" && git push origin $$NEW
minor:
@NEW=v$(_MAJ).$(shell expr $(_MIN) + 1).0; \
git tag $$NEW && echo "Tagged $$NEW" && git push origin $$NEW
major:
@NEW=v$(shell expr $(_MAJ) + 1).0.0; \
git tag $$NEW && echo "Tagged $$NEW" && git push origin $$NEW
+191 -1
View File
@@ -1,3 +1,193 @@
# cephrgw-operator
Kubernetes operator that provisions Ceph RGW (S3) buckets and access keys (RW/RO) from CRDs via the Ceph manager dashboard API
A Kubernetes operator that provisions Ceph RGW (S3) **buckets** and **access
keys** declaratively, talking **directly to radosgw** the way the CLI does. You
describe a bucket, its owner, and who may read or write it as custom resources;
the operator creates the RGW users and bucket, delivers the access/secret keys
into Kubernetes Secrets, and maintains the bucket's S3 policy.
It uses native Go libraries against radosgw (e.g.
`https://radosgw.service.consul:443`) — no manager dashboard, no RADOS access,
no admin socket, no in-cluster Ceph required:
- **[go-ceph](https://github.com/ceph/go-ceph) `rgw/admin`** drives the RGW
**Admin Ops API** (`/admin/...`) for users, keys, quotas and bucket
info/removal — pure Go, no cgo.
- **[aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2)** drives the **S3 API**
for bucket creation, versioning, policy, tagging and object lock — the
operations the Admin Ops API does not expose. These are signed as the bucket
**owner**, so the owner owns the bucket directly.
## Custom resources
| Kind | Short | Purpose |
|------|-------|---------|
| `ObjectStoreUser` | `osu` | An RGW S3 user. The operator creates it and writes its key pair into a Secret. |
| `Bucket` | `bkt` | An S3 bucket owned by an `ObjectStoreUser`. Owns the bucket's aggregate S3 policy. |
| `BucketAccess` | `ba` | Grants a user `read-only`, `read-write` or `full` access to a `Bucket`, delivering RW/RO keys. |
### How access levels work
The bucket **owner** (`Bucket.spec.ownerRef`) always has full control. Each
`BucketAccess` adds a principal to the bucket's S3 policy:
- `read-only` → `s3:GetObject`, `s3:ListBucket` and friends.
- `read-write` → read plus `s3:PutObject` / `s3:DeleteObject` / multipart.
- `full` → `s3:*` on the bucket and its objects.
If a `BucketAccess` omits `userRef`, the operator provisions a **dedicated** RGW
user for that grant and writes its keys into `spec.secretName` (default
`<name>-rgw`). If `userRef` names an existing `ObjectStoreUser`, that user's own
credential Secret is reused and only the policy is extended.
#### Fine-grained grants
The level is the ergonomic default; four optional fields on `BucketAccess`
refine it (see `config/samples/04-access-fine-grained.yaml`):
- `spec.paths` — scope object access to key prefixes; each becomes the resource
`<bucket>/<prefix>*`. The bucket-level `ListBucket` still spans the whole
bucket.
- `spec.actions` — grant exactly these S3 actions instead of the level's set (on
the bucket and its, optionally prefixed, objects).
- `spec.conditions` — `sourceIPs` (an `aws:SourceIp` CIDR allowlist) and
`secureTransportOnly` (require TLS).
- `spec.rawStatements` — an escape hatch of raw S3 policy statements
(`effect`/`actions`/`resources`/`conditions`) merged for this grant's
principal. When set, `level`, `actions`, `paths` and `conditions` are ignored;
resources without an `arn:` prefix are treated as bucket-relative key prefixes.
RGW honours S3 bucket policy on **Reef 18.2+ / Squid**; condition-key support is
a subset of AWS, so validate exotic conditions against your cluster.
### Placement targets
`Bucket.spec.placementTarget` selects the RGW **placement target** that backs the
bucket — i.e. which pools, and therefore which durability profile, store its
data. The valid values are cluster configuration, not a fixed set baked into the
operator. On this estate radosgw exposes two:
- `default-placement` — 3× replicated (the cluster default).
- `ec` — 4+1 erasure-coded (cheaper capacity, for bulk/archival data).
Leaving `placementTarget` empty keeps the current behaviour: the owning user's
`default_placement` (falling back to the zonegroup default). When set, the
operator threads it into the S3 `CreateBucket` `LocationConstraint` as
`<zonegroup>:<placementTarget>`; with `spec.zonegroup` empty (the default) that
is `:<placementTarget>`, which selects the local/master zonegroup with the given
placement — so you do not need to know the zonegroup's api-name to pick a target.
```yaml
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: raw-archive
spec:
ownerRef: logarchiver
placementTarget: ec # 4+1 erasure-coded pool
```
Placement is **immutable**: RGW fixes it at bucket creation and cannot move an
existing bucket between targets. The CRD rejects changing `placementTarget` (and
`zonegroup`) on an existing `Bucket`, and if a bucket already lives on a
different target than the spec requests (e.g. an adopted bucket, or a value
sneaked in around the CRD guard) the controller sets an `Error` phase with a
`PlacementImmutable` reason rather than ever deleting and recreating it. The
placement RGW actually stores the bucket on is reported in
`status.placementTarget` (and the `Placement` print column), so drift is visible.
See `config/samples/06-bucket-ec.yaml`.
### Adopting existing buckets and users
The operator can take over buckets/users that already exist in radosgw and hand
them back without deleting them. In short: matching CRDs manage the resource in
place (no recreation, keys reused, `status.adopted: true`), the bucket policy is
**merged** so an existing hand-written policy is preserved (`spec.managePolicy:
false` opts out entirely), and `spec.retainOnDelete` on `ObjectStoreUser` /
`Bucket` / `BucketAccess` orphans the RGW object instead of deleting it. See
**[docs/adoption.md](docs/adoption.md)** and
`config/samples/05-adoption.yaml`.
The `Bucket` controller renders the policy as the **union of every ready
`BucketAccess`** that targets it, so the result is convergent regardless of the
order objects are created or deleted. It watches `BucketAccess` and
`ObjectStoreUser`, re-reconciling the bucket whenever a grant or user changes.
```
ObjectStoreUser ──admin PUT /admin/user──────▶ Secret (AK/SK)
Bucket ──S3 CreateBucket (as owner)─▶ owns S3 policy
BucketAccess ──admin PUT /admin/user──────▶ Secret (AK/SK, RW or RO)
└────── enqueues Bucket ──▶ S3 PutBucketPolicy (aggregate)
```
## Credential Secrets
Every credential Secret carries the conventional keys, ready to mount straight
into a workload:
- `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
- `RGW_UID`
- `S3_ENDPOINT`, `BUCKET_HOST` (when `CEPH_RGW_ENDPOINT` is set)
- `BUCKET_NAME` (on `BucketAccess` Secrets)
Secrets are owner-referenced by the resource that produced them, so they are
garbage-collected when the resource is deleted.
## Prerequisites
The operator needs an RGW user with admin caps (`users=*;buckets=*`) and its
access/secret key, the radosgw endpoint, and (for `read-only`/non-owner
`read-write` grants) Ceph **Reef 18.2+ / Squid**. See
**[docs/ceph-setup.md](docs/ceph-setup.md)** for the exact commands and the
`cephrgw-credentials` Secret schema.
## Quickstart
```sh
kubectl apply -f config/samples/00-owner-user.yaml
kubectl apply -f config/samples/01-bucket.yaml
kubectl apply -f config/samples/02-access-readonly.yaml
kubectl apply -f config/samples/03-access-readwrite.yaml
kubectl get osu,bkt,ba
kubectl get secret app-data-ro-rgw -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d
```
## Development
```sh
make generate # regenerate deepcopy, CRDs and RBAC from kubebuilder markers
make build # build the operator binary
make test # go test -race
make lint fmt # go vet / gofmt
```
### Local (kind)
```sh
kind create cluster --name cephrgw
docker build -t cephrgw-operator:dev -f Dockerfile.operator .
kind load docker-image cephrgw-operator:dev --name cephrgw
kubectl apply -f config/crd/bases/
kubectl apply -f hack/kind/manifests/ # edit the Secret first
```
## CI
Woodpecker runs `pre-commit` (gofmt + vet), `test`, and a dry-run image `build`
on pull requests; pushing a `v*` tag builds and pushes
`artifactapi.k8s.syd1.au.unkin.net/docker-internal/cephrgw-operator` to the
artifactapi local docker registry. Bump a release
with `make patch|minor|major`.
## Notes & caveats
- **Policy clearing.** Removing the last `BucketAccess` issues an S3
`DeleteBucketPolicy`. A `NoSuchBucketPolicy` response is treated as already
clear. Adding/replacing grants always works.
- **Per-bucket quota.** `Bucket.spec.quota` is applied as the owner's default
bucket quota via the Admin Ops API, which is per-owner rather than strictly
per-bucket. Use distinct owners if you need independent bucket quotas.
- **Immutability.** `bucketName`, an `ObjectStoreUser`'s `uid`, and object lock
are fixed at creation; changing them on an existing object has no effect.
+172
View File
@@ -0,0 +1,172 @@
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// ObjectLockMode is the S3 object-lock retention mode.
// +kubebuilder:validation:Enum=GOVERNANCE;COMPLIANCE
type ObjectLockMode string
const (
ObjectLockGovernance ObjectLockMode = "GOVERNANCE"
ObjectLockCompliance ObjectLockMode = "COMPLIANCE"
)
// ObjectLock configures S3 object lock (WORM) on a bucket. Object lock can only
// be enabled at creation time and requires versioning.
type ObjectLock struct {
// Enabled turns on object lock for the bucket.
Enabled bool `json:"enabled"`
// Mode is the default retention mode applied to new objects.
// +optional
Mode ObjectLockMode `json:"mode,omitempty"`
// Days is the default retention period in days. Mutually exclusive with Years.
// +optional
Days *int32 `json:"days,omitempty"`
// Years is the default retention period in years. Mutually exclusive with Days.
// +optional
Years *int32 `json:"years,omitempty"`
}
// BucketSpec defines a Ceph RGW (S3) bucket owned by an ObjectStoreUser.
type BucketSpec struct {
// BucketName is the S3 bucket name. Defaults to metadata.name. Immutable.
// +optional
BucketName string `json:"bucketName,omitempty"`
// OwnerRef names the ObjectStoreUser (in this namespace) that owns the
// bucket. The owner always has full control; grant additional principals
// with BucketAccess objects.
OwnerRef string `json:"ownerRef"`
// Zonegroup optionally pins the bucket to a specific RGW zonegroup by its
// api-name. Empty (the default) uses the cluster's local/master zonegroup, so
// PlacementTarget selection works without naming the zonegroup. Immutable:
// RGW resolves the zonegroup at bucket creation and cannot move it afterwards.
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="zonegroup is immutable; RGW fixes it at bucket creation"
// +optional
Zonegroup string `json:"zonegroup,omitempty"`
// PlacementTarget optionally selects the RGW placement target that backs the
// bucket, choosing which pools (and thus replication/erasure profile) store
// its data. Empty (the default) uses the owning user's default_placement, or
// the zonegroup default. The valid values are cluster configuration, not a
// fixed set; on this estate the two configured targets are
// "default-placement" (3x replicated) and "ec" (4+1 erasure-coded).
//
// Immutable: RGW chooses the placement at bucket creation (from the S3
// LocationConstraint) and cannot move an existing bucket between placement
// targets. Set it on a fresh Bucket; changing it later is rejected, and if a
// pre-existing bucket is on a different placement the operator reports an
// error instead of recreating it.
// +kubebuilder:validation:MaxLength=63
// +kubebuilder:validation:Pattern=`^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$`
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="placementTarget is immutable; RGW cannot move a bucket between placement targets"
// +optional
PlacementTarget string `json:"placementTarget,omitempty"`
// Versioning enables S3 object versioning on the bucket.
// +optional
Versioning bool `json:"versioning,omitempty"`
// ObjectLock configures S3 object lock. Enabling it forces versioning on.
// +optional
ObjectLock *ObjectLock `json:"objectLock,omitempty"`
// Quota optionally applies a bucket-level quota.
// +optional
Quota *Quota `json:"quota,omitempty"`
// Tags are bucket tags (key/value) applied to the bucket.
// +optional
Tags map[string]string `json:"tags,omitempty"`
// ManagePolicy controls whether the operator manages the bucket's S3 policy
// from BucketAccess grants. When true (the default) the operator reconciles
// its own statements while preserving any statements it does not own, so it
// is safe to adopt a bucket that already has a policy. Set to false to leave
// the bucket policy entirely untouched (BucketAccess grants then have no
// effect on this bucket).
// +kubebuilder:default=true
// +optional
ManagePolicy *bool `json:"managePolicy,omitempty"`
// RetainOnDelete keeps the RGW bucket (and its objects) when the Bucket
// resource is deleted. By default the operator removes the empty bucket;
// it never purges objects unless PurgeOnDelete is also set.
// +optional
RetainOnDelete bool `json:"retainOnDelete,omitempty"`
// PurgeOnDelete deletes the bucket together with all objects it contains
// when the Bucket resource is removed. Dangerous; defaults to false.
// +optional
PurgeOnDelete bool `json:"purgeOnDelete,omitempty"`
}
// BucketStatus reports observed bucket state.
type BucketStatus struct {
// Phase is a coarse lifecycle summary (Pending/Ready/Error).
// +optional
Phase string `json:"phase,omitempty"`
// BucketName is the provisioned S3 bucket name.
// +optional
BucketName string `json:"bucketName,omitempty"`
// BucketID is the RGW internal bucket instance id.
// +optional
BucketID string `json:"bucketID,omitempty"`
// Owner is the RGW uid that owns the bucket.
// +optional
Owner string `json:"owner,omitempty"`
// PlacementTarget is the placement target RGW actually stores the bucket on,
// read back from the live bucket. It makes placement drift (a bucket landing
// on a different target than spec requested) visible.
// +optional
PlacementTarget string `json:"placementTarget,omitempty"`
// PolicyPrincipals is the number of extra principals granted via
// BucketAccess and reflected in the bucket policy.
// +optional
PolicyPrincipals int32 `json:"policyPrincipals,omitempty"`
// Adopted reports that the RGW bucket already existed when the operator
// first reconciled this resource (it was taken over, not created).
// +optional
Adopted bool `json:"adopted,omitempty"`
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +optional
// +listType=map
// +listMapKey=type
Conditions []metav1.Condition `json:"conditions,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=bkt
// +kubebuilder:printcolumn:name="Bucket",type=string,JSONPath=`.status.bucketName`
// +kubebuilder:printcolumn:name="Owner",type=string,JSONPath=`.status.owner`
// +kubebuilder:printcolumn:name="Placement",type=string,JSONPath=`.status.placementTarget`
// +kubebuilder:printcolumn:name="Grants",type=integer,JSONPath=`.status.policyPrincipals`
// +kubebuilder:printcolumn:name="Adopted",type=boolean,JSONPath=`.status.adopted`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// Bucket is a Ceph RGW S3 bucket.
type Bucket struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec BucketSpec `json:"spec,omitempty"`
Status BucketStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
// BucketList contains a list of Bucket.
type BucketList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []Bucket `json:"items"`
}
func init() {
SchemeBuilder.Register(&Bucket{}, &BucketList{})
}
+173
View File
@@ -0,0 +1,173 @@
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// AccessLevel is the level of access a principal is granted to a bucket. The
// operator translates it into an S3 bucket policy statement.
// +kubebuilder:validation:Enum=read-only;read-write;full
type AccessLevel string
const (
// AccessReadOnly grants object GET and bucket LIST (s3:GetObject,
// s3:ListBucket and friends).
AccessReadOnly AccessLevel = "read-only"
// AccessReadWrite grants read plus object PUT/DELETE and multipart.
AccessReadWrite AccessLevel = "read-write"
// AccessFull grants s3:* on the bucket and its objects.
AccessFull AccessLevel = "full"
)
// BucketAccessSpec grants an RGW user a level of access to a Bucket by
// maintaining a statement in the bucket's S3 policy. If UserRef is empty the
// operator provisions a dedicated user for this grant and writes its keys into
// a Secret; otherwise it grants an existing ObjectStoreUser.
type BucketAccessSpec struct {
// BucketRef names the Bucket (in this namespace) to grant access to.
BucketRef string `json:"bucketRef"`
// Level is the access level to grant.
Level AccessLevel `json:"level"`
// UserRef optionally names an existing ObjectStoreUser (in this namespace)
// to grant. When set, the operator does not create or delete a user and
// SecretName is ignored (that user already owns its own credential Secret).
// +optional
UserRef string `json:"userRef,omitempty"`
// UID overrides the id of the dedicated user created when UserRef is empty.
// Defaults to "<bucket>-<name>". Ignored when UserRef is set.
// +optional
UID string `json:"uid,omitempty"`
// SecretName is the Secret the operator writes credentials into for the
// dedicated user it creates (UserRef empty). Defaults to "<name>-rgw".
// +optional
SecretName string `json:"secretName,omitempty"`
// RetainOnDelete keeps the dedicated RGW user (created when UserRef is empty)
// instead of deleting it when this BucketAccess is removed. Ignored when
// UserRef is set (that user is never managed here). Defaults to false.
// +optional
RetainOnDelete bool `json:"retainOnDelete,omitempty"`
// Paths optionally scopes object-level access to these key prefixes within
// the bucket; each becomes the resource "<bucket>/<prefix>*". Empty grants
// the whole bucket. The bucket-level ListBucket action always applies to the
// whole bucket. Ignored when RawStatements is set.
// +optional
Paths []string `json:"paths,omitempty"`
// Actions optionally overrides the S3 actions granted by Level. When set,
// exactly these actions are granted, on the bucket and its (optionally
// prefixed) objects. Ignored when RawStatements is set.
// +optional
Actions []string `json:"actions,omitempty"`
// Conditions optionally restricts when the grant applies (e.g. source IPs,
// TLS required). Ignored when RawStatements is set.
// +optional
Conditions *AccessConditions `json:"conditions,omitempty"`
// RawStatements is an escape hatch for arbitrary S3 policy statements, merged
// into the bucket policy for this grant's principal. When set, Level,
// Actions, Paths and Conditions on this object are ignored; the operator only
// fills in the Principal (this grant's user) when a statement omits one.
// +optional
RawStatements []PolicyStatement `json:"rawStatements,omitempty"`
}
// AccessConditions restricts when a grant applies. Each field maps to an S3
// policy condition and, when several are set, all must hold (they are AND'd).
type AccessConditions struct {
// SourceIPs restricts the grant to requests from these CIDRs (or single
// addresses), via the S3 aws:SourceIp condition.
// +optional
SourceIPs []string `json:"sourceIPs,omitempty"`
// SecureTransportOnly requires the request to use TLS, via the S3
// aws:SecureTransport condition.
// +optional
SecureTransportOnly bool `json:"secureTransportOnly,omitempty"`
}
// PolicyStatement is a raw S3 bucket-policy statement, exposed for grants that
// need control beyond Level/Actions/Paths/Conditions.
type PolicyStatement struct {
// Sid is an optional statement id. The operator derives one when empty.
// +optional
Sid string `json:"sid,omitempty"`
// Effect is Allow or Deny. Defaults to Allow.
// +kubebuilder:validation:Enum=Allow;Deny
// +kubebuilder:default=Allow
// +optional
Effect string `json:"effect,omitempty"`
// Actions are the S3 actions the statement covers (e.g. s3:GetObject).
Actions []string `json:"actions"`
// Resources are S3 resource ARNs, or bucket-relative key prefixes when they
// do not start with "arn:". Empty means the whole bucket and its objects.
// +optional
Resources []string `json:"resources,omitempty"`
// Conditions is the raw S3 condition block: operator -> condition key ->
// values, e.g. {"IpAddress": {"aws:SourceIp": ["10.0.0.0/8"]}}.
// +optional
Conditions map[string]map[string][]string `json:"conditions,omitempty"`
}
// BucketAccessStatus reports observed grant state.
type BucketAccessStatus struct {
// Phase is a coarse lifecycle summary (Pending/Ready/Error).
// +optional
Phase string `json:"phase,omitempty"`
// UID is the RGW user id that was granted access.
// +optional
UID string `json:"uid,omitempty"`
// SecretName is the Secret holding the dedicated user's credentials, if any.
// +optional
SecretName string `json:"secretName,omitempty"`
// Bound reports whether the grant is reflected in the bucket policy.
// +optional
Bound bool `json:"bound,omitempty"`
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +optional
// +listType=map
// +listMapKey=type
Conditions []metav1.Condition `json:"conditions,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=ba
// +kubebuilder:printcolumn:name="Bucket",type=string,JSONPath=`.spec.bucketRef`
// +kubebuilder:printcolumn:name="Level",type=string,JSONPath=`.spec.level`
// +kubebuilder:printcolumn:name="UID",type=string,JSONPath=`.status.uid`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// BucketAccess grants an RGW user read-only, read-write or full access to a
// Bucket via the bucket's S3 policy.
type BucketAccess struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec BucketAccessSpec `json:"spec,omitempty"`
Status BucketAccessStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
// BucketAccessList contains a list of BucketAccess.
type BucketAccessList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []BucketAccess `json:"items"`
}
func init() {
SchemeBuilder.Register(&BucketAccess{}, &BucketAccessList{})
}
+27
View File
@@ -0,0 +1,27 @@
package v1alpha1
// Quota describes an RGW quota applied to a user or a bucket. A nil limit (or a
// negative value) means unlimited for that dimension.
type Quota struct {
// Enabled turns the quota on. When false the other fields are ignored and
// the quota is disabled on the target.
// +kubebuilder:default=true
// +optional
Enabled bool `json:"enabled,omitempty"`
// MaxSizeBytes caps the total size in bytes. Nil or negative means unlimited.
// +optional
MaxSizeBytes *int64 `json:"maxSizeBytes,omitempty"`
// MaxObjects caps the number of objects. Nil or negative means unlimited.
// +optional
MaxObjects *int64 `json:"maxObjects,omitempty"`
}
// SecretKeyRef points at a single key within a Secret in the same namespace.
type SecretKeyRef struct {
// Name is the Secret name.
Name string `json:"name"`
// Key is the data key within the Secret.
Key string `json:"key"`
}
+3
View File
@@ -0,0 +1,3 @@
// +kubebuilder:object:generate=true
// +groupName=ceph.unkin.net
package v1alpha1
+17
View File
@@ -0,0 +1,17 @@
package v1alpha1
import (
"k8s.io/apimachinery/pkg/runtime/schema"
"sigs.k8s.io/controller-runtime/pkg/scheme"
)
var (
// GroupVersion is the group/version used to register these objects.
GroupVersion = schema.GroupVersion{Group: "ceph.unkin.net", Version: "v1alpha1"}
// SchemeBuilder registers the API types with a runtime scheme.
SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
// AddToScheme adds the types in this group-version to the given scheme.
AddToScheme = SchemeBuilder.AddToScheme
)
+104
View File
@@ -0,0 +1,104 @@
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// ObjectStoreUserSpec defines a Ceph RGW (S3) user. The operator creates the
// user through the radosgw Admin Ops API and writes its generated access/secret
// key pair into a Kubernetes Secret. The key material is never stored on the
// resource itself.
type ObjectStoreUserSpec struct {
// UID is the RGW user id. Defaults to metadata.name. Immutable once created.
// +optional
UID string `json:"uid,omitempty"`
// DisplayName is the human-readable name for the user. Defaults to the UID.
// +optional
DisplayName string `json:"displayName,omitempty"`
// Email is an optional email address recorded on the user.
// +optional
Email string `json:"email,omitempty"`
// MaxBuckets caps how many buckets the user may own. A negative value
// disables bucket creation; 0 leaves the RGW default. Defaults to 1000.
// +kubebuilder:default=1000
// +optional
MaxBuckets *int32 `json:"maxBuckets,omitempty"`
// Suspended manages the user's suspended state: true suspends the user so
// its keys stop working, false resumes it. When unset the operator does not
// touch the suspended state (useful when adopting an existing user).
// +optional
Suspended *bool `json:"suspended,omitempty"`
// Quota optionally applies a user-level quota.
// +optional
Quota *Quota `json:"quota,omitempty"`
// SecretName is the Secret the operator writes the access/secret key into.
// Defaults to "<name>-rgw". The Secret holds AWS_ACCESS_KEY_ID,
// AWS_SECRET_ACCESS_KEY, BUCKET_HOST and the RGW uid.
// +optional
SecretName string `json:"secretName,omitempty"`
// RetainOnDelete keeps the RGW user (and its keys) when the ObjectStoreUser
// resource is deleted, instead of removing it. Set this before adopting an
// existing user you may later want to hand back. Defaults to false.
// +optional
RetainOnDelete bool `json:"retainOnDelete,omitempty"`
}
// ObjectStoreUserStatus reports observed user state.
type ObjectStoreUserStatus struct {
// Phase is a coarse lifecycle summary (Pending/Ready/Error).
// +optional
Phase string `json:"phase,omitempty"`
// UID is the RGW user id that was provisioned.
// +optional
UID string `json:"uid,omitempty"`
// SecretName is the Secret holding the user's credentials.
// +optional
SecretName string `json:"secretName,omitempty"`
// Adopted reports that the RGW user already existed when the operator first
// reconciled this resource (it was taken over, not created).
// +optional
Adopted bool `json:"adopted,omitempty"`
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +optional
// +listType=map
// +listMapKey=type
Conditions []metav1.Condition `json:"conditions,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=osu
// +kubebuilder:printcolumn:name="UID",type=string,JSONPath=`.status.uid`
// +kubebuilder:printcolumn:name="Secret",type=string,JSONPath=`.status.secretName`
// +kubebuilder:printcolumn:name="Adopted",type=boolean,JSONPath=`.status.adopted`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// ObjectStoreUser is a Ceph RGW S3 user whose keys are delivered into a Secret.
type ObjectStoreUser struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec ObjectStoreUserSpec `json:"spec,omitempty"`
Status ObjectStoreUserStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
// ObjectStoreUserList contains a list of ObjectStoreUser.
type ObjectStoreUserList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []ObjectStoreUser `json:"items"`
}
func init() {
SchemeBuilder.Register(&ObjectStoreUser{}, &ObjectStoreUserList{})
}
+494
View File
@@ -0,0 +1,494 @@
//go:build !ignore_autogenerated
// Code generated by controller-gen. DO NOT EDIT.
package v1alpha1
import (
"k8s.io/apimachinery/pkg/apis/meta/v1"
runtime "k8s.io/apimachinery/pkg/runtime"
)
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *AccessConditions) DeepCopyInto(out *AccessConditions) {
*out = *in
if in.SourceIPs != nil {
in, out := &in.SourceIPs, &out.SourceIPs
*out = make([]string, len(*in))
copy(*out, *in)
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AccessConditions.
func (in *AccessConditions) DeepCopy() *AccessConditions {
if in == nil {
return nil
}
out := new(AccessConditions)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Bucket) DeepCopyInto(out *Bucket) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
in.Spec.DeepCopyInto(&out.Spec)
in.Status.DeepCopyInto(&out.Status)
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Bucket.
func (in *Bucket) DeepCopy() *Bucket {
if in == nil {
return nil
}
out := new(Bucket)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *Bucket) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketAccess) DeepCopyInto(out *BucketAccess) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
in.Spec.DeepCopyInto(&out.Spec)
in.Status.DeepCopyInto(&out.Status)
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketAccess.
func (in *BucketAccess) DeepCopy() *BucketAccess {
if in == nil {
return nil
}
out := new(BucketAccess)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *BucketAccess) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketAccessList) DeepCopyInto(out *BucketAccessList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]BucketAccess, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketAccessList.
func (in *BucketAccessList) DeepCopy() *BucketAccessList {
if in == nil {
return nil
}
out := new(BucketAccessList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *BucketAccessList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketAccessSpec) DeepCopyInto(out *BucketAccessSpec) {
*out = *in
if in.Paths != nil {
in, out := &in.Paths, &out.Paths
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Actions != nil {
in, out := &in.Actions, &out.Actions
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = new(AccessConditions)
(*in).DeepCopyInto(*out)
}
if in.RawStatements != nil {
in, out := &in.RawStatements, &out.RawStatements
*out = make([]PolicyStatement, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketAccessSpec.
func (in *BucketAccessSpec) DeepCopy() *BucketAccessSpec {
if in == nil {
return nil
}
out := new(BucketAccessSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketAccessStatus) DeepCopyInto(out *BucketAccessStatus) {
*out = *in
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = make([]v1.Condition, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketAccessStatus.
func (in *BucketAccessStatus) DeepCopy() *BucketAccessStatus {
if in == nil {
return nil
}
out := new(BucketAccessStatus)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketList) DeepCopyInto(out *BucketList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]Bucket, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketList.
func (in *BucketList) DeepCopy() *BucketList {
if in == nil {
return nil
}
out := new(BucketList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *BucketList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketSpec) DeepCopyInto(out *BucketSpec) {
*out = *in
if in.ObjectLock != nil {
in, out := &in.ObjectLock, &out.ObjectLock
*out = new(ObjectLock)
(*in).DeepCopyInto(*out)
}
if in.Quota != nil {
in, out := &in.Quota, &out.Quota
*out = new(Quota)
(*in).DeepCopyInto(*out)
}
if in.Tags != nil {
in, out := &in.Tags, &out.Tags
*out = make(map[string]string, len(*in))
for key, val := range *in {
(*out)[key] = val
}
}
if in.ManagePolicy != nil {
in, out := &in.ManagePolicy, &out.ManagePolicy
*out = new(bool)
**out = **in
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketSpec.
func (in *BucketSpec) DeepCopy() *BucketSpec {
if in == nil {
return nil
}
out := new(BucketSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketStatus) DeepCopyInto(out *BucketStatus) {
*out = *in
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = make([]v1.Condition, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketStatus.
func (in *BucketStatus) DeepCopy() *BucketStatus {
if in == nil {
return nil
}
out := new(BucketStatus)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ObjectLock) DeepCopyInto(out *ObjectLock) {
*out = *in
if in.Days != nil {
in, out := &in.Days, &out.Days
*out = new(int32)
**out = **in
}
if in.Years != nil {
in, out := &in.Years, &out.Years
*out = new(int32)
**out = **in
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectLock.
func (in *ObjectLock) DeepCopy() *ObjectLock {
if in == nil {
return nil
}
out := new(ObjectLock)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ObjectStoreUser) DeepCopyInto(out *ObjectStoreUser) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
in.Spec.DeepCopyInto(&out.Spec)
in.Status.DeepCopyInto(&out.Status)
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectStoreUser.
func (in *ObjectStoreUser) DeepCopy() *ObjectStoreUser {
if in == nil {
return nil
}
out := new(ObjectStoreUser)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *ObjectStoreUser) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ObjectStoreUserList) DeepCopyInto(out *ObjectStoreUserList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]ObjectStoreUser, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectStoreUserList.
func (in *ObjectStoreUserList) DeepCopy() *ObjectStoreUserList {
if in == nil {
return nil
}
out := new(ObjectStoreUserList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *ObjectStoreUserList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ObjectStoreUserSpec) DeepCopyInto(out *ObjectStoreUserSpec) {
*out = *in
if in.MaxBuckets != nil {
in, out := &in.MaxBuckets, &out.MaxBuckets
*out = new(int32)
**out = **in
}
if in.Suspended != nil {
in, out := &in.Suspended, &out.Suspended
*out = new(bool)
**out = **in
}
if in.Quota != nil {
in, out := &in.Quota, &out.Quota
*out = new(Quota)
(*in).DeepCopyInto(*out)
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectStoreUserSpec.
func (in *ObjectStoreUserSpec) DeepCopy() *ObjectStoreUserSpec {
if in == nil {
return nil
}
out := new(ObjectStoreUserSpec)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ObjectStoreUserStatus) DeepCopyInto(out *ObjectStoreUserStatus) {
*out = *in
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = make([]v1.Condition, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectStoreUserStatus.
func (in *ObjectStoreUserStatus) DeepCopy() *ObjectStoreUserStatus {
if in == nil {
return nil
}
out := new(ObjectStoreUserStatus)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *PolicyStatement) DeepCopyInto(out *PolicyStatement) {
*out = *in
if in.Actions != nil {
in, out := &in.Actions, &out.Actions
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Resources != nil {
in, out := &in.Resources, &out.Resources
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = make(map[string]map[string][]string, len(*in))
for key, val := range *in {
var outVal map[string][]string
if val == nil {
(*out)[key] = nil
} else {
inVal := (*in)[key]
in, out := &inVal, &outVal
*out = make(map[string][]string, len(*in))
for key, val := range *in {
var outVal []string
if val == nil {
(*out)[key] = nil
} else {
inVal := (*in)[key]
in, out := &inVal, &outVal
*out = make([]string, len(*in))
copy(*out, *in)
}
(*out)[key] = outVal
}
}
(*out)[key] = outVal
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PolicyStatement.
func (in *PolicyStatement) DeepCopy() *PolicyStatement {
if in == nil {
return nil
}
out := new(PolicyStatement)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Quota) DeepCopyInto(out *Quota) {
*out = *in
if in.MaxSizeBytes != nil {
in, out := &in.MaxSizeBytes, &out.MaxSizeBytes
*out = new(int64)
**out = **in
}
if in.MaxObjects != nil {
in, out := &in.MaxObjects, &out.MaxObjects
*out = new(int64)
**out = **in
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Quota.
func (in *Quota) DeepCopy() *Quota {
if in == nil {
return nil
}
out := new(Quota)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *SecretKeyRef) DeepCopyInto(out *SecretKeyRef) {
*out = *in
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SecretKeyRef.
func (in *SecretKeyRef) DeepCopy() *SecretKeyRef {
if in == nil {
return nil
}
out := new(SecretKeyRef)
in.DeepCopyInto(out)
return out
}
+129
View File
@@ -0,0 +1,129 @@
package main
import (
"context"
"flag"
"os"
"time"
"k8s.io/apimachinery/pkg/runtime"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/healthz"
"sigs.k8s.io/controller-runtime/pkg/log/zap"
metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
cephv1alpha1 "git.unkin.net/unkin/cephrgw-operator/api/v1alpha1"
"git.unkin.net/unkin/cephrgw-operator/internal/ceph"
"git.unkin.net/unkin/cephrgw-operator/internal/controller"
)
var scheme = runtime.NewScheme()
func init() {
utilruntime.Must(clientgoscheme.AddToScheme(scheme))
utilruntime.Must(cephv1alpha1.AddToScheme(scheme))
}
func main() {
var metricsAddr, probeAddr string
var leaderElect bool
flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "metrics endpoint address")
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "health probe address")
flag.BoolVar(&leaderElect, "leader-elect", false, "enable leader election")
flag.Parse()
ctrl.SetLogger(zap.New(zap.UseDevMode(false)))
logger := ctrl.Log.WithName("setup")
cephCfg, endpoint, err := cephConfigFromEnv()
if err != nil {
logger.Error(err, "invalid radosgw configuration")
os.Exit(1)
}
cephClient, err := ceph.NewClient(cephCfg)
if err != nil {
logger.Error(err, "unable to build radosgw client")
os.Exit(1)
}
// Fail fast on obviously-broken credentials, but do not block startup on a
// transiently unreachable radosgw.
pingCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
if err := cephClient.Ping(pingCtx); err != nil {
logger.Error(err, "initial radosgw authentication failed; continuing and will retry per-reconcile")
}
cancel()
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
Metrics: metricsserver.Options{BindAddress: metricsAddr},
HealthProbeBindAddress: probeAddr,
LeaderElection: leaderElect,
LeaderElectionID: "cephrgw-operator",
})
if err != nil {
logger.Error(err, "unable to create manager")
os.Exit(1)
}
if err := controller.SetupAll(mgr, cephClient, endpoint); err != nil {
logger.Error(err, "unable to set up controllers")
os.Exit(1)
}
// Advisory only: warn (never exit) if the installed CRDs are missing or
// predate this operator's schema, which otherwise surfaces only as opaque
// strict-decode failures during reconcile.
crdCtx, crdCancel := context.WithTimeout(context.Background(), 15*time.Second)
controller.CheckCRDVersions(crdCtx, mgr.GetConfig())
crdCancel()
if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
logger.Error(err, "unable to set up health check")
os.Exit(1)
}
if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
logger.Error(err, "unable to set up ready check")
os.Exit(1)
}
logger.Info("starting cephrgw-operator")
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
logger.Error(err, "manager exited with error")
os.Exit(1)
}
}
// cephConfigFromEnv reads radosgw connection settings from the environment,
// which the deployment sources from the cephrgw-credentials Secret.
//
// The operator talks to the radosgw Admin Ops and S3 APIs at
// CEPH_RGW_ADMIN_ENDPOINT (falling back to CEPH_RGW_ENDPOINT). The returned
// endpoint string is the S3 endpoint written into consumer credential Secrets,
// which may differ (e.g. a public S3 name) from the API endpoint.
func cephConfigFromEnv() (ceph.Config, string, error) {
consumerEndpoint := os.Getenv("CEPH_RGW_ENDPOINT")
apiEndpoint := os.Getenv("CEPH_RGW_ADMIN_ENDPOINT")
if apiEndpoint == "" {
apiEndpoint = consumerEndpoint
}
cfg := ceph.Config{
Endpoint: apiEndpoint,
AccessKey: os.Getenv("CEPH_RGW_ACCESS_KEY"),
SecretKey: os.Getenv("CEPH_RGW_SECRET_KEY"),
Region: os.Getenv("CEPH_RGW_REGION"),
Insecure: os.Getenv("CEPH_RGW_INSECURE") == "true",
}
if f := os.Getenv("CEPH_RGW_CA_FILE"); f != "" {
b, err := os.ReadFile(f)
if err != nil {
return cfg, "", err
}
cfg.CACert = b
} else if inline := os.Getenv("CEPH_RGW_CA"); inline != "" {
cfg.CACert = []byte(inline)
}
return cfg, consumerEndpoint, nil
}
@@ -0,0 +1,269 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.17.3
name: bucketaccesses.ceph.unkin.net
spec:
group: ceph.unkin.net
names:
kind: BucketAccess
listKind: BucketAccessList
plural: bucketaccesses
shortNames:
- ba
singular: bucketaccess
scope: Namespaced
versions:
- additionalPrinterColumns:
- jsonPath: .spec.bucketRef
name: Bucket
type: string
- jsonPath: .spec.level
name: Level
type: string
- jsonPath: .status.uid
name: UID
type: string
- jsonPath: .status.phase
name: Phase
type: string
name: v1alpha1
schema:
openAPIV3Schema:
description: |-
BucketAccess grants an RGW user read-only, read-write or full access to a
Bucket via the bucket's S3 policy.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: |-
BucketAccessSpec grants an RGW user a level of access to a Bucket by
maintaining a statement in the bucket's S3 policy. If UserRef is empty the
operator provisions a dedicated user for this grant and writes its keys into
a Secret; otherwise it grants an existing ObjectStoreUser.
properties:
actions:
description: |-
Actions optionally overrides the S3 actions granted by Level. When set,
exactly these actions are granted, on the bucket and its (optionally
prefixed) objects. Ignored when RawStatements is set.
items:
type: string
type: array
bucketRef:
description: BucketRef names the Bucket (in this namespace) to grant
access to.
type: string
conditions:
description: |-
Conditions optionally restricts when the grant applies (e.g. source IPs,
TLS required). Ignored when RawStatements is set.
properties:
secureTransportOnly:
description: |-
SecureTransportOnly requires the request to use TLS, via the S3
aws:SecureTransport condition.
type: boolean
sourceIPs:
description: |-
SourceIPs restricts the grant to requests from these CIDRs (or single
addresses), via the S3 aws:SourceIp condition.
items:
type: string
type: array
type: object
level:
description: Level is the access level to grant.
enum:
- read-only
- read-write
- full
type: string
paths:
description: |-
Paths optionally scopes object-level access to these key prefixes within
the bucket; each becomes the resource "<bucket>/<prefix>*". Empty grants
the whole bucket. The bucket-level ListBucket action always applies to the
whole bucket. Ignored when RawStatements is set.
items:
type: string
type: array
rawStatements:
description: |-
RawStatements is an escape hatch for arbitrary S3 policy statements, merged
into the bucket policy for this grant's principal. When set, Level,
Actions, Paths and Conditions on this object are ignored; the operator only
fills in the Principal (this grant's user) when a statement omits one.
items:
description: |-
PolicyStatement is a raw S3 bucket-policy statement, exposed for grants that
need control beyond Level/Actions/Paths/Conditions.
properties:
actions:
description: Actions are the S3 actions the statement covers
(e.g. s3:GetObject).
items:
type: string
type: array
conditions:
additionalProperties:
additionalProperties:
items:
type: string
type: array
type: object
description: |-
Conditions is the raw S3 condition block: operator -> condition key ->
values, e.g. {"IpAddress": {"aws:SourceIp": ["10.0.0.0/8"]}}.
type: object
effect:
default: Allow
description: Effect is Allow or Deny. Defaults to Allow.
enum:
- Allow
- Deny
type: string
resources:
description: |-
Resources are S3 resource ARNs, or bucket-relative key prefixes when they
do not start with "arn:". Empty means the whole bucket and its objects.
items:
type: string
type: array
sid:
description: Sid is an optional statement id. The operator derives
one when empty.
type: string
required:
- actions
type: object
type: array
retainOnDelete:
description: |-
RetainOnDelete keeps the dedicated RGW user (created when UserRef is empty)
instead of deleting it when this BucketAccess is removed. Ignored when
UserRef is set (that user is never managed here). Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes credentials into for the
dedicated user it creates (UserRef empty). Defaults to "<name>-rgw".
type: string
uid:
description: |-
UID overrides the id of the dedicated user created when UserRef is empty.
Defaults to "<bucket>-<name>". Ignored when UserRef is set.
type: string
userRef:
description: |-
UserRef optionally names an existing ObjectStoreUser (in this namespace)
to grant. When set, the operator does not create or delete a user and
SecretName is ignored (that user already owns its own credential Secret).
type: string
required:
- bucketRef
- level
type: object
status:
description: BucketAccessStatus reports observed grant state.
properties:
bound:
description: Bound reports whether the grant is reflected in the bucket
policy.
type: boolean
conditions:
items:
description: Condition contains details for one aspect of the current
state of this API Resource.
properties:
lastTransitionTime:
description: |-
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: |-
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: |-
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: |-
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False, Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
observedGeneration:
format: int64
type: integer
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
secretName:
description: SecretName is the Secret holding the dedicated user's
credentials, if any.
type: string
uid:
description: UID is the RGW user id that was granted access.
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
@@ -0,0 +1,282 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.17.3
name: buckets.ceph.unkin.net
spec:
group: ceph.unkin.net
names:
kind: Bucket
listKind: BucketList
plural: buckets
shortNames:
- bkt
singular: bucket
scope: Namespaced
versions:
- additionalPrinterColumns:
- jsonPath: .status.bucketName
name: Bucket
type: string
- jsonPath: .status.owner
name: Owner
type: string
- jsonPath: .status.placementTarget
name: Placement
type: string
- jsonPath: .status.policyPrincipals
name: Grants
type: integer
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
name: v1alpha1
schema:
openAPIV3Schema:
description: Bucket is a Ceph RGW S3 bucket.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: BucketSpec defines a Ceph RGW (S3) bucket owned by an ObjectStoreUser.
properties:
bucketName:
description: BucketName is the S3 bucket name. Defaults to metadata.name.
Immutable.
type: string
managePolicy:
default: true
description: |-
ManagePolicy controls whether the operator manages the bucket's S3 policy
from BucketAccess grants. When true (the default) the operator reconciles
its own statements while preserving any statements it does not own, so it
is safe to adopt a bucket that already has a policy. Set to false to leave
the bucket policy entirely untouched (BucketAccess grants then have no
effect on this bucket).
type: boolean
objectLock:
description: ObjectLock configures S3 object lock. Enabling it forces
versioning on.
properties:
days:
description: Days is the default retention period in days. Mutually
exclusive with Years.
format: int32
type: integer
enabled:
description: Enabled turns on object lock for the bucket.
type: boolean
mode:
description: Mode is the default retention mode applied to new
objects.
enum:
- GOVERNANCE
- COMPLIANCE
type: string
years:
description: Years is the default retention period in years. Mutually
exclusive with Days.
format: int32
type: integer
required:
- enabled
type: object
ownerRef:
description: |-
OwnerRef names the ObjectStoreUser (in this namespace) that owns the
bucket. The owner always has full control; grant additional principals
with BucketAccess objects.
type: string
placementTarget:
description: |-
PlacementTarget optionally selects the RGW placement target that backs the
bucket, choosing which pools (and thus replication/erasure profile) store
its data. Empty (the default) uses the owning user's default_placement, or
the zonegroup default. The valid values are cluster configuration, not a
fixed set; on this estate the two configured targets are
"default-placement" (3x replicated) and "ec" (4+1 erasure-coded).
Immutable: RGW chooses the placement at bucket creation (from the S3
LocationConstraint) and cannot move an existing bucket between placement
targets. Set it on a fresh Bucket; changing it later is rejected, and if a
pre-existing bucket is on a different placement the operator reports an
error instead of recreating it.
maxLength: 63
pattern: ^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$
type: string
x-kubernetes-validations:
- message: placementTarget is immutable; RGW cannot move a bucket
between placement targets
rule: self == oldSelf
purgeOnDelete:
description: |-
PurgeOnDelete deletes the bucket together with all objects it contains
when the Bucket resource is removed. Dangerous; defaults to false.
type: boolean
quota:
description: Quota optionally applies a bucket-level quota.
properties:
enabled:
default: true
description: |-
Enabled turns the quota on. When false the other fields are ignored and
the quota is disabled on the target.
type: boolean
maxObjects:
description: MaxObjects caps the number of objects. Nil or negative
means unlimited.
format: int64
type: integer
maxSizeBytes:
description: MaxSizeBytes caps the total size in bytes. Nil or
negative means unlimited.
format: int64
type: integer
type: object
retainOnDelete:
description: |-
RetainOnDelete keeps the RGW bucket (and its objects) when the Bucket
resource is deleted. By default the operator removes the empty bucket;
it never purges objects unless PurgeOnDelete is also set.
type: boolean
tags:
additionalProperties:
type: string
description: Tags are bucket tags (key/value) applied to the bucket.
type: object
versioning:
description: Versioning enables S3 object versioning on the bucket.
type: boolean
zonegroup:
description: |-
Zonegroup optionally pins the bucket to a specific RGW zonegroup by its
api-name. Empty (the default) uses the cluster's local/master zonegroup, so
PlacementTarget selection works without naming the zonegroup. Immutable:
RGW resolves the zonegroup at bucket creation and cannot move it afterwards.
type: string
x-kubernetes-validations:
- message: zonegroup is immutable; RGW fixes it at bucket creation
rule: self == oldSelf
required:
- ownerRef
type: object
status:
description: BucketStatus reports observed bucket state.
properties:
adopted:
description: |-
Adopted reports that the RGW bucket already existed when the operator
first reconciled this resource (it was taken over, not created).
type: boolean
bucketID:
description: BucketID is the RGW internal bucket instance id.
type: string
bucketName:
description: BucketName is the provisioned S3 bucket name.
type: string
conditions:
items:
description: Condition contains details for one aspect of the current
state of this API Resource.
properties:
lastTransitionTime:
description: |-
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: |-
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: |-
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: |-
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False, Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
observedGeneration:
format: int64
type: integer
owner:
description: Owner is the RGW uid that owns the bucket.
type: string
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
placementTarget:
description: |-
PlacementTarget is the placement target RGW actually stores the bucket on,
read back from the live bucket. It makes placement drift (a bucket landing
on a different target than spec requested) visible.
type: string
policyPrincipals:
description: |-
PolicyPrincipals is the number of extra principals granted via
BucketAccess and reflected in the bucket policy.
format: int32
type: integer
type: object
type: object
served: true
storage: true
subresources:
status: {}
@@ -0,0 +1,203 @@
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.17.3
name: objectstoreusers.ceph.unkin.net
spec:
group: ceph.unkin.net
names:
kind: ObjectStoreUser
listKind: ObjectStoreUserList
plural: objectstoreusers
shortNames:
- osu
singular: objectstoreuser
scope: Namespaced
versions:
- additionalPrinterColumns:
- jsonPath: .status.uid
name: UID
type: string
- jsonPath: .status.secretName
name: Secret
type: string
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
name: v1alpha1
schema:
openAPIV3Schema:
description: ObjectStoreUser is a Ceph RGW S3 user whose keys are delivered
into a Secret.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: |-
ObjectStoreUserSpec defines a Ceph RGW (S3) user. The operator creates the
user through the radosgw Admin Ops API and writes its generated access/secret
key pair into a Kubernetes Secret. The key material is never stored on the
resource itself.
properties:
displayName:
description: DisplayName is the human-readable name for the user.
Defaults to the UID.
type: string
email:
description: Email is an optional email address recorded on the user.
type: string
maxBuckets:
default: 1000
description: |-
MaxBuckets caps how many buckets the user may own. A negative value
disables bucket creation; 0 leaves the RGW default. Defaults to 1000.
format: int32
type: integer
quota:
description: Quota optionally applies a user-level quota.
properties:
enabled:
default: true
description: |-
Enabled turns the quota on. When false the other fields are ignored and
the quota is disabled on the target.
type: boolean
maxObjects:
description: MaxObjects caps the number of objects. Nil or negative
means unlimited.
format: int64
type: integer
maxSizeBytes:
description: MaxSizeBytes caps the total size in bytes. Nil or
negative means unlimited.
format: int64
type: integer
type: object
retainOnDelete:
description: |-
RetainOnDelete keeps the RGW user (and its keys) when the ObjectStoreUser
resource is deleted, instead of removing it. Set this before adopting an
existing user you may later want to hand back. Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes the access/secret key into.
Defaults to "<name>-rgw". The Secret holds AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY, BUCKET_HOST and the RGW uid.
type: string
suspended:
description: |-
Suspended manages the user's suspended state: true suspends the user so
its keys stop working, false resumes it. When unset the operator does not
touch the suspended state (useful when adopting an existing user).
type: boolean
uid:
description: UID is the RGW user id. Defaults to metadata.name. Immutable
once created.
type: string
type: object
status:
description: ObjectStoreUserStatus reports observed user state.
properties:
adopted:
description: |-
Adopted reports that the RGW user already existed when the operator first
reconciled this resource (it was taken over, not created).
type: boolean
conditions:
items:
description: Condition contains details for one aspect of the current
state of this API Resource.
properties:
lastTransitionTime:
description: |-
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: |-
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: |-
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: |-
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False, Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
observedGeneration:
format: int64
type: integer
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
secretName:
description: SecretName is the Secret holding the user's credentials.
type: string
uid:
description: UID is the RGW user id that was provisioned.
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
+755
View File
@@ -0,0 +1,755 @@
# Generated by "make generate". DO NOT EDIT.
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.17.3
name: bucketaccesses.ceph.unkin.net
spec:
group: ceph.unkin.net
names:
kind: BucketAccess
listKind: BucketAccessList
plural: bucketaccesses
shortNames:
- ba
singular: bucketaccess
scope: Namespaced
versions:
- additionalPrinterColumns:
- jsonPath: .spec.bucketRef
name: Bucket
type: string
- jsonPath: .spec.level
name: Level
type: string
- jsonPath: .status.uid
name: UID
type: string
- jsonPath: .status.phase
name: Phase
type: string
name: v1alpha1
schema:
openAPIV3Schema:
description: |-
BucketAccess grants an RGW user read-only, read-write or full access to a
Bucket via the bucket's S3 policy.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: |-
BucketAccessSpec grants an RGW user a level of access to a Bucket by
maintaining a statement in the bucket's S3 policy. If UserRef is empty the
operator provisions a dedicated user for this grant and writes its keys into
a Secret; otherwise it grants an existing ObjectStoreUser.
properties:
actions:
description: |-
Actions optionally overrides the S3 actions granted by Level. When set,
exactly these actions are granted, on the bucket and its (optionally
prefixed) objects. Ignored when RawStatements is set.
items:
type: string
type: array
bucketRef:
description: BucketRef names the Bucket (in this namespace) to grant
access to.
type: string
conditions:
description: |-
Conditions optionally restricts when the grant applies (e.g. source IPs,
TLS required). Ignored when RawStatements is set.
properties:
secureTransportOnly:
description: |-
SecureTransportOnly requires the request to use TLS, via the S3
aws:SecureTransport condition.
type: boolean
sourceIPs:
description: |-
SourceIPs restricts the grant to requests from these CIDRs (or single
addresses), via the S3 aws:SourceIp condition.
items:
type: string
type: array
type: object
level:
description: Level is the access level to grant.
enum:
- read-only
- read-write
- full
type: string
paths:
description: |-
Paths optionally scopes object-level access to these key prefixes within
the bucket; each becomes the resource "<bucket>/<prefix>*". Empty grants
the whole bucket. The bucket-level ListBucket action always applies to the
whole bucket. Ignored when RawStatements is set.
items:
type: string
type: array
rawStatements:
description: |-
RawStatements is an escape hatch for arbitrary S3 policy statements, merged
into the bucket policy for this grant's principal. When set, Level,
Actions, Paths and Conditions on this object are ignored; the operator only
fills in the Principal (this grant's user) when a statement omits one.
items:
description: |-
PolicyStatement is a raw S3 bucket-policy statement, exposed for grants that
need control beyond Level/Actions/Paths/Conditions.
properties:
actions:
description: Actions are the S3 actions the statement covers
(e.g. s3:GetObject).
items:
type: string
type: array
conditions:
additionalProperties:
additionalProperties:
items:
type: string
type: array
type: object
description: |-
Conditions is the raw S3 condition block: operator -> condition key ->
values, e.g. {"IpAddress": {"aws:SourceIp": ["10.0.0.0/8"]}}.
type: object
effect:
default: Allow
description: Effect is Allow or Deny. Defaults to Allow.
enum:
- Allow
- Deny
type: string
resources:
description: |-
Resources are S3 resource ARNs, or bucket-relative key prefixes when they
do not start with "arn:". Empty means the whole bucket and its objects.
items:
type: string
type: array
sid:
description: Sid is an optional statement id. The operator derives
one when empty.
type: string
required:
- actions
type: object
type: array
retainOnDelete:
description: |-
RetainOnDelete keeps the dedicated RGW user (created when UserRef is empty)
instead of deleting it when this BucketAccess is removed. Ignored when
UserRef is set (that user is never managed here). Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes credentials into for the
dedicated user it creates (UserRef empty). Defaults to "<name>-rgw".
type: string
uid:
description: |-
UID overrides the id of the dedicated user created when UserRef is empty.
Defaults to "<bucket>-<name>". Ignored when UserRef is set.
type: string
userRef:
description: |-
UserRef optionally names an existing ObjectStoreUser (in this namespace)
to grant. When set, the operator does not create or delete a user and
SecretName is ignored (that user already owns its own credential Secret).
type: string
required:
- bucketRef
- level
type: object
status:
description: BucketAccessStatus reports observed grant state.
properties:
bound:
description: Bound reports whether the grant is reflected in the bucket
policy.
type: boolean
conditions:
items:
description: Condition contains details for one aspect of the current
state of this API Resource.
properties:
lastTransitionTime:
description: |-
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: |-
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: |-
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: |-
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False, Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
observedGeneration:
format: int64
type: integer
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
secretName:
description: SecretName is the Secret holding the dedicated user's
credentials, if any.
type: string
uid:
description: UID is the RGW user id that was granted access.
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.17.3
name: buckets.ceph.unkin.net
spec:
group: ceph.unkin.net
names:
kind: Bucket
listKind: BucketList
plural: buckets
shortNames:
- bkt
singular: bucket
scope: Namespaced
versions:
- additionalPrinterColumns:
- jsonPath: .status.bucketName
name: Bucket
type: string
- jsonPath: .status.owner
name: Owner
type: string
- jsonPath: .status.placementTarget
name: Placement
type: string
- jsonPath: .status.policyPrincipals
name: Grants
type: integer
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
name: v1alpha1
schema:
openAPIV3Schema:
description: Bucket is a Ceph RGW S3 bucket.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: BucketSpec defines a Ceph RGW (S3) bucket owned by an ObjectStoreUser.
properties:
bucketName:
description: BucketName is the S3 bucket name. Defaults to metadata.name.
Immutable.
type: string
managePolicy:
default: true
description: |-
ManagePolicy controls whether the operator manages the bucket's S3 policy
from BucketAccess grants. When true (the default) the operator reconciles
its own statements while preserving any statements it does not own, so it
is safe to adopt a bucket that already has a policy. Set to false to leave
the bucket policy entirely untouched (BucketAccess grants then have no
effect on this bucket).
type: boolean
objectLock:
description: ObjectLock configures S3 object lock. Enabling it forces
versioning on.
properties:
days:
description: Days is the default retention period in days. Mutually
exclusive with Years.
format: int32
type: integer
enabled:
description: Enabled turns on object lock for the bucket.
type: boolean
mode:
description: Mode is the default retention mode applied to new
objects.
enum:
- GOVERNANCE
- COMPLIANCE
type: string
years:
description: Years is the default retention period in years. Mutually
exclusive with Days.
format: int32
type: integer
required:
- enabled
type: object
ownerRef:
description: |-
OwnerRef names the ObjectStoreUser (in this namespace) that owns the
bucket. The owner always has full control; grant additional principals
with BucketAccess objects.
type: string
placementTarget:
description: |-
PlacementTarget optionally selects the RGW placement target that backs the
bucket, choosing which pools (and thus replication/erasure profile) store
its data. Empty (the default) uses the owning user's default_placement, or
the zonegroup default. The valid values are cluster configuration, not a
fixed set; on this estate the two configured targets are
"default-placement" (3x replicated) and "ec" (4+1 erasure-coded).
Immutable: RGW chooses the placement at bucket creation (from the S3
LocationConstraint) and cannot move an existing bucket between placement
targets. Set it on a fresh Bucket; changing it later is rejected, and if a
pre-existing bucket is on a different placement the operator reports an
error instead of recreating it.
maxLength: 63
pattern: ^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$
type: string
x-kubernetes-validations:
- message: placementTarget is immutable; RGW cannot move a bucket
between placement targets
rule: self == oldSelf
purgeOnDelete:
description: |-
PurgeOnDelete deletes the bucket together with all objects it contains
when the Bucket resource is removed. Dangerous; defaults to false.
type: boolean
quota:
description: Quota optionally applies a bucket-level quota.
properties:
enabled:
default: true
description: |-
Enabled turns the quota on. When false the other fields are ignored and
the quota is disabled on the target.
type: boolean
maxObjects:
description: MaxObjects caps the number of objects. Nil or negative
means unlimited.
format: int64
type: integer
maxSizeBytes:
description: MaxSizeBytes caps the total size in bytes. Nil or
negative means unlimited.
format: int64
type: integer
type: object
retainOnDelete:
description: |-
RetainOnDelete keeps the RGW bucket (and its objects) when the Bucket
resource is deleted. By default the operator removes the empty bucket;
it never purges objects unless PurgeOnDelete is also set.
type: boolean
tags:
additionalProperties:
type: string
description: Tags are bucket tags (key/value) applied to the bucket.
type: object
versioning:
description: Versioning enables S3 object versioning on the bucket.
type: boolean
zonegroup:
description: |-
Zonegroup optionally pins the bucket to a specific RGW zonegroup by its
api-name. Empty (the default) uses the cluster's local/master zonegroup, so
PlacementTarget selection works without naming the zonegroup. Immutable:
RGW resolves the zonegroup at bucket creation and cannot move it afterwards.
type: string
x-kubernetes-validations:
- message: zonegroup is immutable; RGW fixes it at bucket creation
rule: self == oldSelf
required:
- ownerRef
type: object
status:
description: BucketStatus reports observed bucket state.
properties:
adopted:
description: |-
Adopted reports that the RGW bucket already existed when the operator
first reconciled this resource (it was taken over, not created).
type: boolean
bucketID:
description: BucketID is the RGW internal bucket instance id.
type: string
bucketName:
description: BucketName is the provisioned S3 bucket name.
type: string
conditions:
items:
description: Condition contains details for one aspect of the current
state of this API Resource.
properties:
lastTransitionTime:
description: |-
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: |-
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: |-
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: |-
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False, Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
observedGeneration:
format: int64
type: integer
owner:
description: Owner is the RGW uid that owns the bucket.
type: string
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
placementTarget:
description: |-
PlacementTarget is the placement target RGW actually stores the bucket on,
read back from the live bucket. It makes placement drift (a bucket landing
on a different target than spec requested) visible.
type: string
policyPrincipals:
description: |-
PolicyPrincipals is the number of extra principals granted via
BucketAccess and reflected in the bucket policy.
format: int32
type: integer
type: object
type: object
served: true
storage: true
subresources:
status: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.17.3
name: objectstoreusers.ceph.unkin.net
spec:
group: ceph.unkin.net
names:
kind: ObjectStoreUser
listKind: ObjectStoreUserList
plural: objectstoreusers
shortNames:
- osu
singular: objectstoreuser
scope: Namespaced
versions:
- additionalPrinterColumns:
- jsonPath: .status.uid
name: UID
type: string
- jsonPath: .status.secretName
name: Secret
type: string
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
name: v1alpha1
schema:
openAPIV3Schema:
description: ObjectStoreUser is a Ceph RGW S3 user whose keys are delivered
into a Secret.
properties:
apiVersion:
description: |-
APIVersion defines the versioned schema of this representation of an object.
Servers should convert recognized schemas to the latest internal value, and
may reject unrecognized values.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
type: string
kind:
description: |-
Kind is a string value representing the REST resource this object represents.
Servers may infer this from the endpoint the client submits requests to.
Cannot be updated.
In CamelCase.
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
type: string
metadata:
type: object
spec:
description: |-
ObjectStoreUserSpec defines a Ceph RGW (S3) user. The operator creates the
user through the radosgw Admin Ops API and writes its generated access/secret
key pair into a Kubernetes Secret. The key material is never stored on the
resource itself.
properties:
displayName:
description: DisplayName is the human-readable name for the user.
Defaults to the UID.
type: string
email:
description: Email is an optional email address recorded on the user.
type: string
maxBuckets:
default: 1000
description: |-
MaxBuckets caps how many buckets the user may own. A negative value
disables bucket creation; 0 leaves the RGW default. Defaults to 1000.
format: int32
type: integer
quota:
description: Quota optionally applies a user-level quota.
properties:
enabled:
default: true
description: |-
Enabled turns the quota on. When false the other fields are ignored and
the quota is disabled on the target.
type: boolean
maxObjects:
description: MaxObjects caps the number of objects. Nil or negative
means unlimited.
format: int64
type: integer
maxSizeBytes:
description: MaxSizeBytes caps the total size in bytes. Nil or
negative means unlimited.
format: int64
type: integer
type: object
retainOnDelete:
description: |-
RetainOnDelete keeps the RGW user (and its keys) when the ObjectStoreUser
resource is deleted, instead of removing it. Set this before adopting an
existing user you may later want to hand back. Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes the access/secret key into.
Defaults to "<name>-rgw". The Secret holds AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY, BUCKET_HOST and the RGW uid.
type: string
suspended:
description: |-
Suspended manages the user's suspended state: true suspends the user so
its keys stop working, false resumes it. When unset the operator does not
touch the suspended state (useful when adopting an existing user).
type: boolean
uid:
description: UID is the RGW user id. Defaults to metadata.name. Immutable
once created.
type: string
type: object
status:
description: ObjectStoreUserStatus reports observed user state.
properties:
adopted:
description: |-
Adopted reports that the RGW user already existed when the operator first
reconciled this resource (it was taken over, not created).
type: boolean
conditions:
items:
description: Condition contains details for one aspect of the current
state of this API Resource.
properties:
lastTransitionTime:
description: |-
lastTransitionTime is the last time the condition transitioned from one status to another.
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
format: date-time
type: string
message:
description: |-
message is a human readable message indicating details about the transition.
This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: |-
observedGeneration represents the .metadata.generation that the condition was set based upon.
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
with respect to the current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: |-
reason contains a programmatic identifier indicating the reason for the condition's last transition.
Producers of specific condition types may define expected values and meanings for this field,
and whether the values are considered a guaranteed API.
The value should be a CamelCase string.
This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False, Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
observedGeneration:
format: int64
type: integer
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
secretName:
description: SecretName is the Secret holding the user's credentials.
type: string
uid:
description: UID is the RGW user id that was provisioned.
type: string
type: object
type: object
served: true
storage: true
subresources:
status: {}
+57
View File
@@ -0,0 +1,57 @@
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cephrgw-operator
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- get
- list
- apiGroups:
- ceph.unkin.net
resources:
- bucketaccesses
- buckets
- objectstoreusers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- ceph.unkin.net
resources:
- bucketaccesses/finalizers
- buckets/finalizers
- objectstoreusers/finalizers
verbs:
- update
- apiGroups:
- ceph.unkin.net
resources:
- bucketaccesses/status
- buckets/status
- objectstoreusers/status
verbs:
- get
- patch
- update
+13
View File
@@ -0,0 +1,13 @@
# The owner of a bucket. The operator creates an RGW user and writes its
# access/secret key pair into the Secret "app-owner-rgw".
apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
name: app-owner
namespace: default
spec:
displayName: "Application bucket owner"
maxBuckets: 50
quota:
enabled: true
maxSizeBytes: 107374182400 # 100 GiB
+16
View File
@@ -0,0 +1,16 @@
# A bucket owned by the app-owner user, with versioning enabled.
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: app-data
namespace: default
spec:
bucketName: app-data
ownerRef: app-owner
versioning: true
tags:
team: platform
env: prod
# By default the operator deletes the (empty) bucket when this object is
# removed. Set retainOnDelete: true to keep it, or purgeOnDelete: true to
# delete it together with all objects.
+13
View File
@@ -0,0 +1,13 @@
# Read-only credentials for the bucket. Because no userRef is given, the
# operator provisions a dedicated RGW user for this grant and writes its keys
# into the Secret "app-data-ro-rgw". The Bucket's S3 policy is updated to grant
# this user GetObject/ListBucket only.
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-ro
namespace: default
spec:
bucketRef: app-data
level: read-only
secretName: app-data-ro-rgw
+34
View File
@@ -0,0 +1,34 @@
# Read-write credentials for the bucket, delivered to a dedicated user and
# Secret "app-data-rw-rgw". The owner (app-owner) always retains full control;
# this grant is for a separate workload that needs to read and write objects
# but must not manage the bucket itself.
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-rw
namespace: default
spec:
bucketRef: app-data
level: read-write
secretName: app-data-rw-rgw
---
# Alternatively, grant an *existing* ObjectStoreUser access to the bucket by
# name. Here no user or Secret is created; the shared user's own credential
# Secret is used, and the bucket policy is extended to include it.
apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
name: analytics
namespace: default
spec:
displayName: "Analytics pipeline"
---
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-analytics
namespace: default
spec:
bucketRef: app-data
level: read-only
userRef: analytics
@@ -0,0 +1,78 @@
# Fine-grained grants. Each of these refines the coarse read-only/read-write/full
# levels with prefix scoping, action overrides, conditions, or raw statements.
# 1. Prefix-scoped read-write: this workload may read/write objects only under
# the "uploads/" and "tmp/" key prefixes (bucket-level ListBucket still spans
# the whole bucket).
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-uploader
namespace: default
spec:
bucketRef: app-data
level: read-write
secretName: app-data-uploader-rgw
paths:
- uploads/
- tmp/
---
# 2. Read-only from inside the cluster only: restrict the grant to a source CIDR
# and require TLS.
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-internal-ro
namespace: default
spec:
bucketRef: app-data
level: read-only
secretName: app-data-internal-ro-rgw
conditions:
sourceIPs:
- 10.0.0.0/8
secureTransportOnly: true
---
# 3. Explicit action set: grant exactly these actions instead of a level's
# canned set (level is still required but its actions are ignored).
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-getput
namespace: default
spec:
bucketRef: app-data
level: read-only
secretName: app-data-getput-rgw
actions:
- s3:GetObject
- s3:PutObject
---
# 4. Raw statements escape hatch: full control over the policy statement. Level,
# actions, paths and conditions are ignored; the operator only injects the
# Principal (this grant's user). Resources without an "arn:" prefix are
# treated as bucket-relative key prefixes.
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-raw
namespace: default
spec:
bucketRef: app-data
level: read-only
secretName: app-data-raw-rgw
rawStatements:
- effect: Allow
actions:
- s3:GetObject
resources:
- public/
- effect: Deny
actions:
- s3:DeleteObject
resources:
- locked/
conditions:
Bool:
aws:SecureTransport:
- "false"
+31
View File
@@ -0,0 +1,31 @@
# Adopting an existing radosgw user + bucket. The operator takes them over in
# place: no recreation, existing keys reused, existing bucket policy preserved.
# retainOnDelete keeps the RGW objects if these CRDs are later deleted.
# See docs/adoption.md.
apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
name: legacy-owner
namespace: default
spec:
# uid must match the existing RGW user id.
uid: legacy-owner
# Set maxBuckets to the existing user's limit (it otherwise defaults to 1000
# and would be applied). Leave displayName/suspended unset to keep them as-is.
maxBuckets: 1000
retainOnDelete: true
---
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: legacy-data
namespace: default
spec:
# bucketName must match the existing bucket.
bucketName: legacy-data
ownerRef: legacy-owner
retainOnDelete: true
# managePolicy defaults to true: the operator merges its BucketAccess grants
# into the existing policy, preserving statements it does not own. Set it to
# false to leave the bucket policy entirely under manual control.
managePolicy: true
+18
View File
@@ -0,0 +1,18 @@
# A bucket placed on the erasure-coded (4+1) placement target instead of the
# default 3x-replicated pool. Good for bulk/archival data where capacity matters
# more than the extra replica.
#
# placementTarget is immutable: RGW chooses the placement at bucket creation and
# cannot move an existing bucket between targets, so it can only be set on a
# fresh Bucket. The operator reports the live placement in status.placementTarget.
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: raw-archive
namespace: default
spec:
bucketName: raw-archive
ownerRef: app-owner
# Cluster-configured placement target. On this estate: "default-placement"
# (3x replicated) or "ec" (4+1 erasure-coded).
placementTarget: ec
+68
View File
@@ -0,0 +1,68 @@
# Adopting existing radosgw buckets and users
The operator can **take over** buckets and users that already exist in radosgw:
write CRDs that match them and it manages them in place instead of recreating
them. It can also **hand them back** without deleting the underlying RGW objects.
## What happens when you create a CRD for an existing resource
| CRD | On adoption |
|-----|-------------|
| `ObjectStoreUser` | The user is **not** recreated and its **existing keys are reused** (never rotated). `status.adopted` becomes `true`. Attributes are only changed if the spec sets them (see below). |
| `Bucket` | The bucket is **not** recreated. `status.adopted` becomes `true`. Versioning/tags/quota are only touched if the spec sets them. The policy is **merged**, not overwritten (see below). |
| `BucketAccess` | Adds the grant's statement to the bucket policy and (for a dedicated user) reuses that user's keys. |
`status.adopted` is also shown in `kubectl get osu` / `kubectl get bkt` under
the **ADOPTED** column.
### User attributes
To avoid clobbering an adopted user, the operator only sends attributes you set:
- `displayName` — left unchanged when empty.
- `suspended` — left unchanged when unset (it is an optional `*bool`; set it
explicitly to `true`/`false` to manage it).
- `email` — left unchanged when empty.
- `maxBuckets` — **defaults to `1000`** and is always applied. If the existing
user has a different limit you want to keep, set `maxBuckets` to match (or to
the value you want).
### Bucket policy is merged, not replaced
The operator owns only the policy statements it writes — they carry a
`cephrgwop…` statement id. On every reconcile it **preserves statements it does
not own** and reconciles only its own. So adopting a bucket that already has a
hand-written policy keeps that policy; your `BucketAccess` grants are added
alongside it.
- Reserve the `cephrgwop` prefix (and the legacy prefixes `full`, `custom`,
`raw`, `readonly*`, `readwrite*`) for the operator — do not name your own
statements with them, or they will be treated as operator-owned and replaced.
- To have the operator **never touch** a bucket's policy, set
`spec.managePolicy: false`. `BucketAccess` grants then have no effect on that
bucket.
## What happens when you delete the CRD
By default, deleting a CRD deletes the underlying RGW resource. Opt out per
resource to **orphan** it instead (the finalizer is dropped, the RGW object is
left in place):
| CRD | Default on delete | Keep the RGW object |
|-----|-------------------|---------------------|
| `ObjectStoreUser` | deletes the RGW user + keys | `spec.retainOnDelete: true` |
| `Bucket` | deletes the (empty) bucket | `spec.retainOnDelete: true` (and never set `purgeOnDelete`) |
| `BucketAccess` (dedicated user) | deletes the dedicated user | `spec.retainOnDelete: true` |
| `BucketAccess` (`userRef`) | only drops the policy statement | n/a (never manages that user) |
## Recommended adoption procedure
1. Create an `ObjectStoreUser` for each owner, setting `retainOnDelete: true`
and `maxBuckets` to the value you want. Confirm `ADOPTED=true`.
2. Create the `Bucket` (with `retainOnDelete: true`) referencing that owner.
With `managePolicy: true` (default) the existing policy is preserved.
3. Optionally add `BucketAccess` objects to model existing grants; they merge
into the policy. If you would rather keep managing the policy by hand, set
`managePolicy: false` on the Bucket.
4. To hand a resource back, delete its CRD — with `retainOnDelete: true` the RGW
bucket/user is left untouched.
+168
View File
@@ -0,0 +1,168 @@
# Ceph setup: credentials and permissions the operator needs
`cephrgw-operator` talks **directly to radosgw**, the same way the `radosgw-admin`
CLI and S3 clients do — no manager dashboard involved. It uses two native Go
libraries against the RGW endpoint (e.g. `https://radosgw.service.consul:443`):
- **go-ceph `rgw/admin`** → the RGW **Admin Ops API** (`/admin/user`,
`/admin/bucket`), signed with the operator's access/secret key, for users,
keys, quotas and bucket info/removal.
- **aws-sdk-go-v2** → the **S3 API**, signed as each bucket's **owner**, for
bucket creation, versioning, policy, tagging and object lock.
So there is exactly **one** credential to provision: a radosgw user with admin
caps, plus its access/secret key.
---
## 1. Create the operator's RGW admin user
Create a dedicated radosgw user and give it the admin caps the operator needs.
Only `users` and `buckets` caps are required (the operator never reads usage or
metadata endpoints):
```bash
radosgw-admin user create \
--uid=cephrgw-operator \
--display-name="cephrgw-operator" \
--caps="users=*;buckets=*"
# Grab its keys (these become CEPH_RGW_ACCESS_KEY / CEPH_RGW_SECRET_KEY):
radosgw-admin user info --uid=cephrgw-operator \
| jq -r '.keys[0] | .access_key, .secret_key'
```
If the user already exists, add the caps instead:
```bash
radosgw-admin caps add --uid=cephrgw-operator --caps="users=*;buckets=*"
```
> `users=*;buckets=*` lets the operator create/read/delete RGW users and read/
> remove buckets through the Admin Ops API. Bucket **creation** and all bucket
> sub-resources (versioning, policy, tagging, object lock) go over the S3 API
> signed as the bucket owner, so they need no extra admin cap — every RGW user
> can manage its own buckets. The `--system` flag is **not** required.
## 2. Admin Ops API must be enabled on radosgw
The Admin Ops API is served by radosgw at the `admin` resource and is enabled by
default. If your deployment has trimmed `rgw_enable_apis`, make sure it includes
both `s3` and `admin`:
```
rgw_enable_apis = s3, admin
```
Quick check from your workstation (a `403`/`AccessDenied` still proves the
endpoint is reachable and the API is on; a connection error means it is not):
```bash
curl -sk "https://radosgw.service.consul:443/admin/user?format=json"
```
## 3. Bucket policy support (read-only / non-owner read-write)
The operator enforces `read-only` and non-owner `read-write` grants by writing an
**S3 bucket policy** (`PutBucketPolicy`). Bucket-policy support is available on
**Ceph Reef 18.2+ / Squid**. On older releases bucket creation and owner
(`full`) access still work, but policy-based grants will fail — upgrade the
cluster, or only use owner credentials, if you are pre-Reef.
Check your version:
```bash
ceph versions | jq -r '.mon | keys[]'
```
## 4. (Optional) S3 endpoint for consumers
The operator can stamp the S3 endpoint into every credential Secret it writes
(`S3_ENDPOINT` and `BUCKET_HOST`) so applications don't have to hard-code it.
Provide it via `CEPH_RGW_ENDPOINT` (see below); if unset, those keys are simply
omitted. This is also the default endpoint for the Admin Ops and S3 API calls
when `CEPH_RGW_ADMIN_ENDPOINT` is not set separately.
---
## 5. Give the operator its credentials (the `cephrgw-credentials` Secret)
The operator reads its configuration from environment variables, which the
deployment sources from a Secret named **`cephrgw-credentials`** in its namespace
(`cephrgw-system`). The Secret data keys map 1:1 to the env vars:
| Secret key | Required | Meaning |
|------------|----------|---------|
| `CEPH_RGW_ACCESS_KEY` | yes | access key of the RGW admin user from step 1 |
| `CEPH_RGW_SECRET_KEY` | yes | its secret key |
| `CEPH_RGW_ENDPOINT` | see note | S3 endpoint; written into consumer Secrets and used for API calls unless `CEPH_RGW_ADMIN_ENDPOINT` is set |
| `CEPH_RGW_ADMIN_ENDPOINT` | no | radosgw endpoint for the Admin Ops + S3 API calls, if it differs from the public `CEPH_RGW_ENDPOINT` |
| `CEPH_RGW_REGION` | no | SigV4 credential-scope region for S3 requests (default `default`) |
| `CEPH_RGW_CA` | no | PEM CA bundle to verify the radosgw TLS cert (inline) |
| `CEPH_RGW_CA_FILE` | no | path to a mounted CA file (alternative to the above) |
| `CEPH_RGW_INSECURE` | no | `"true"` to skip TLS verification (dev only) |
> At least one of `CEPH_RGW_ENDPOINT` or `CEPH_RGW_ADMIN_ENDPOINT` must be set —
> the API endpoint falls back to `CEPH_RGW_ENDPOINT` when the admin one is unset.
### Primary method: Vault + VSO
The Secret is not managed in GitOps; it is rendered from Vault by the Vault
Secrets Operator (VSO). The argocd-apps `cephrgw-system` app ships a `VaultAuth`
and a `VaultStaticSecret` that authenticate with the shared `default` Kubernetes
auth role and render the KV path
`kubernetes/namespace/cephrgw-system/default/cephrgw-credentials` into the
`cephrgw-credentials` Secret. That path sits under the cluster's templated
default policy (`kv/data/kubernetes/namespace/<ns>/<sa>/*`), so **no dedicated
Vault role or policy is required** — you only seed the values:
```bash
vault kv put kv/kubernetes/namespace/cephrgw-system/default/cephrgw-credentials \
CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
CEPH_RGW_ACCESS_KEY='REPLACE-WITH-ACCESS-KEY' \
CEPH_RGW_SECRET_KEY='REPLACE-WITH-SECRET-KEY'
```
The keys under that KV path are copied verbatim into the Secret, so they must be
named exactly as the table above. VSO refreshes the Secret every few minutes,
and the deployment's `reloader.stakater.com/auto: "true"` annotation restarts the
operator when it changes — so rotating the credential is just a new `vault kv
put`, no manual rollout.
### Fallback: a plain Secret
Outside this cluster (or for a quick `kind` test) you can create the Secret
directly instead of using Vault:
```bash
kubectl -n cephrgw-system create secret generic cephrgw-credentials \
--from-literal=CEPH_RGW_ENDPOINT=https://s3.ceph.unkin.net \
--from-literal=CEPH_RGW_ADMIN_ENDPOINT=https://radosgw.service.consul:443 \
--from-literal=CEPH_RGW_ACCESS_KEY='REPLACE-WITH-ACCESS-KEY' \
--from-literal=CEPH_RGW_SECRET_KEY='REPLACE-WITH-SECRET-KEY'
```
The operator does not care where the Secret comes from, only that those keys
exist.
---
## Quick verification
Once the Secret and RGW admin user exist, a smoke test from your workstation
using the operator's keys (this is the same Admin Ops call the operator's
readiness `Ping` makes):
```bash
# Signing an Admin Ops request by hand is fiddly; the simplest proof is to use
# the AWS CLI configured with the operator's keys against the S3 endpoint:
AWS_ACCESS_KEY_ID=REPLACE-WITH-ACCESS-KEY \
AWS_SECRET_ACCESS_KEY=REPLACE-WITH-SECRET-KEY \
aws --endpoint-url https://s3.ceph.unkin.net s3 ls
```
A successful (even empty) listing proves the keys and endpoint work. If the
operator logs `initial radosgw authentication failed`, the keys are wrong or the
`admin` API is disabled (steps 1–2); if users are created but bucket policy
grants fail, the cluster is likely pre-Reef (step 3).
+81
View File
@@ -0,0 +1,81 @@
module git.unkin.net/unkin/cephrgw-operator
go 1.25.0
require (
github.com/aws/aws-sdk-go-v2 v1.43.0
github.com/aws/aws-sdk-go-v2/credentials v1.19.30
github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0
github.com/aws/smithy-go v1.27.4
github.com/ceph/go-ceph v0.40.0
k8s.io/api v0.34.4
k8s.io/apiextensions-apiserver v0.34.1
k8s.io/apimachinery v0.34.4
k8s.io/client-go v0.34.4
sigs.k8s.io/controller-runtime v0.22.4
)
require (
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/emicklei/go-restful/v3 v3.12.2 // indirect
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/btree v1.1.3 // indirect
github.com/google/gnostic-models v0.7.0 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.22.0 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.62.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/spf13/pflag v1.0.6 // indirect
github.com/x448/float16 v0.8.4 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.27.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/net v0.38.0 // indirect
golang.org/x/oauth2 v0.27.0 // indirect
golang.org/x/sync v0.12.0 // indirect
golang.org/x/sys v0.45.0 // indirect
golang.org/x/term v0.30.0 // indirect
golang.org/x/text v0.23.0 // indirect
golang.org/x/time v0.9.0 // indirect
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
google.golang.org/protobuf v1.36.5 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
sigs.k8s.io/structured-merge-diff/v6 v6.3.0 // indirect
sigs.k8s.io/yaml v1.6.0 // indirect
)
+236
View File
@@ -0,0 +1,236 @@
github.com/aws/aws-sdk-go-v2 v1.43.0 h1:fharf/WhbRAVZ1du0QL7roNFxZ6T/sWr+4Ni617bwSI=
github.com/aws/aws-sdk-go-v2 v1.43.0/go.mod h1:5pKeft2eJj+gElQ38Jqg4ibCqh+/AK33/0X3hip7IjM=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 h1:3IZY0XAJquT3aHzbkHfPzy4ACPcEjVG0x87KOwtpqGY=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14/go.mod h1:zwM6veDkhGgQFqkBy+uT28AAYpLu+uFMlPl+rCg/73E=
github.com/aws/aws-sdk-go-v2/config v1.32.22 h1:Vfvp7+fYKsVCADcWOEllqEV47aIBXhNchvyDFu1B5fY=
github.com/aws/aws-sdk-go-v2/config v1.32.22/go.mod h1:0+H+0nPKbvWltf5vSIGkApv+hGbaQ4FfwTjGIYQREcw=
github.com/aws/aws-sdk-go-v2/credentials v1.19.30 h1:TTCvvzFU6gXa4iJecNG/0F/B0oYTiazoRECr2XyLHrY=
github.com/aws/aws-sdk-go-v2/credentials v1.19.30/go.mod h1:jKxAp2AEncnliinzpgOSZDFv6+VjvWhjw/AtbfsWT9U=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 h1:kfVL5wAunCJycL6MOQ6aNh6PlAYEymflcjuKmrWUA0o=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31/go.mod h1:nWfRNDAppujCQgOUd43lKT4yeLv9z3nJ3bw1G3BgQKo=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 h1:Z8F3hfCY33IGpJjFAnv0wvtv1FIKj1GHmRDEYqy64tw=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31/go.mod h1:aVyUoytEyOViR6jhq6jula0xkc5NfBE2hgeF6BvOrao=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 h1:hyOxUyXdh3AyjE93gBgsfziJag9ACwcs+ZpDBLzi8mw=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31/go.mod h1:OERqI9k0draSLB8O8woxY3q25ZWTELRK4RRoLMuMZFo=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 h1:0MrUL35H/Y4kdFfItoR5jCgtDQ4Z/8LudAoIHRfA4hE=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32/go.mod h1:2tNZkuWz54arj8mHVf+8Y7cKkcD8Wr/fBpENgEXpjLc=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 h1:mbRIur/BiHK6SKPjoBIXSE/hJ6g6JGRLuxQy1jGjlN4=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13/go.mod h1:ITg9em2KbJx1s0y4aqRX5OYWG6HBZ5TVR//OdpEZ2CQ=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 h1:mdPwDQPqxlw9Sc62Nt15yjEcARaDbPXkjRYtXsUripo=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24/go.mod h1:ls5ytnwLTcQaUu32fMYXFI3MjpKuTwL840PAm9iqyEg=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 h1:w2SIhW92DZPFrSL4ksVCr8IYff5OZwIcxg8+95tzvAI=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31/go.mod h1:wAhpCQbkov+IcvjozJbd2xRCoZybUEHNkcFunssNACg=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 h1:jWXtZdCnhXa9sGFixRaU2AxT4DIVse9HS4E2f+/KwV0=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32/go.mod h1:9JS1UpfVvyD/ZPX8GsKb/Pq8scEM+7GP5fqh9SwH7po=
github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 h1:7QZWVJZWzHivHWIa+5TELLaBBkbuoj0GPwQtMlJ0sqk=
github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0/go.mod h1:fcvq5L7dK+5cQFicEJwpI6e6Wn8NY2i6yT5wRLYVc7s=
github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 h1:OHH5iTQvVGmfHjX/5Q+vFuA/Rf2x6/95aJ/75QCQSm4=
github.com/aws/aws-sdk-go-v2/service/signin v1.5.0/go.mod h1:mCF3AK9PpL49oOrhniUXWAfhVBVQ/XbytoE5eccZUIs=
github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 h1:CaJyYhxBE0M/HJX/YvSaSmQlsI91VHB0lKU8LtLxL3A=
github.com/aws/aws-sdk-go-v2/service/sso v1.33.0/go.mod h1:+e6BMRMPjBQoCw/WovYR9GLy2IU0z4Q77smOB1DraSg=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 h1:tC323YV77QdafeBr6LUhLDTsboyuyHLNRwAyCP44kGU=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0/go.mod h1:SfLK1sgviHmbI+MozR9iDwDjL4cdCVZtahsjoR+z7wg=
github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 h1:Pd6PNlp4t8PTXxqzstICl52Wsy78vpjFZ7PRUj44mJc=
github.com/aws/aws-sdk-go-v2/service/sts v1.45.0/go.mod h1:rmQ0TnHzuLPmabgjPcsywhsSOmaBDgzR4zvDxSPsGdg=
github.com/aws/smithy-go v1.27.4 h1:JQcphmBN4f0q/sPqXqROIItRNV/hy10cgu7CsFy616M=
github.com/aws/smithy-go v1.27.4/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/ceph/go-ceph v0.40.0 h1:Wz9WOX6i73Hz74mpwhTO9S6IyX3eFPv88VUc7FRMPRk=
github.com/ceph/go-ceph v0.40.0/go.mod h1:1oFtT/x/4y+teLsNiogdd/Kj81Gmrw6JM5w1bWnZoc4=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/emicklei/go-restful/v3 v3.12.2 h1:DhwDP0vY3k8ZzE0RunuJy8GhNpPL6zqLkDf9B/a0/xU=
github.com/emicklei/go-restful/v3 v3.12.2/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/evanphx/json-patch v0.5.2 h1:xVCHIVMUu1wtM/VkR9jVZ45N3FhZfYMMYGorLCR8P3k=
github.com/evanphx/json-patch v0.5.2/go.mod h1:ZWS5hhDbVDyob71nXKNL0+PWn6ToqBHMikGIFbs31qQ=
github.com/evanphx/json-patch/v5 v5.9.11 h1:/8HVnzMq13/3x9TPvjG08wUGqBTmZBsCWzjTM0wiaDU=
github.com/evanphx/json-patch/v5 v5.9.11/go.mod h1:3j+LviiESTElxA4p3EMKAB9HXj3/XEtnUf6OZxqIQTM=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM=
github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ=
github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg=
github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
github.com/go-openapi/jsonreference v0.20.2 h1:3sVjiK66+uXK/6oQ8xgcRKcFgQ5KXa2KvnJRumpMGbE=
github.com/go-openapi/jsonreference v0.20.2/go.mod h1:Bl1zwGIM8/wsvqjsOQLJ/SH+En5Ap4rVB5KVcIDZG2k=
github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14=
github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE=
github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo=
github.com/google/gnostic-models v0.7.0/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7OUGxBlw57miDrQ=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo=
github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee h1:W5t00kpgFdJifH4BDsTlE89Zl93FEloxaWZfGcifgq8=
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/onsi/ginkgo/v2 v2.22.0 h1:Yed107/8DjTr0lKCNt7Dn8yQ6ybuDRQoMGrNFKzMfHg=
github.com/onsi/ginkgo/v2 v2.22.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo=
github.com/onsi/gomega v1.36.1 h1:bJDPBO7ibjxcbHMgSCoo4Yj18UWbKDlLwX1x9sybDcw=
github.com/onsi/gomega v1.36.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ2Io=
github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkqml0W5zIY1I=
github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8=
golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
golang.org/x/oauth2 v0.27.0 h1:da9Vo7/tDv5RH/7nZDz1eMGS/q1Vv1N/7FCrBhI9I3M=
golang.org/x/oauth2 v0.27.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.12.0 h1:MHc5BpPuC30uJk597Ri8TV3CNZcTLu6B6z4lJy+g6Jw=
golang.org/x/sync v0.12.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.30.0 h1:PQ39fJZ+mfadBm0y5WlL4vlM7Sx1Hgf13sMIY2+QS9Y=
golang.org/x/term v0.30.0/go.mod h1:NYYFdzHoI5wRh/h5tDMdMqCqPJZEuNqVR5xJLd/n67g=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY=
golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY=
golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.26.0 h1:v/60pFQmzmT9ExmjDv2gGIfi3OqfKoEP6I5+umXlbnQ=
golang.org/x/tools v0.26.0/go.mod h1:TPVVj70c7JJ3WCazhD8OdXcZg/og+b9+tH/KxylGwH0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw=
gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY=
google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
k8s.io/api v0.34.4 h1:Z5hsoQcZ2yBjelb9j5JKzCVo9qv9XLkVm5llnqS4h+0=
k8s.io/api v0.34.4/go.mod h1:6SaGYuGPkMqqCgg8rPG/OQoCrhgSEV+wWn9v21fDP3o=
k8s.io/apiextensions-apiserver v0.34.1 h1:NNPBva8FNAPt1iSVwIE0FsdrVriRXMsaWFMqJbII2CI=
k8s.io/apiextensions-apiserver v0.34.1/go.mod h1:hP9Rld3zF5Ay2Of3BeEpLAToP+l4s5UlxiHfqRaRcMc=
k8s.io/apimachinery v0.34.4 h1:C5SiSzLEMyWIk53sSbnk0WlOOyqv/MFnWvuc/d6M+xc=
k8s.io/apimachinery v0.34.4/go.mod h1:/GwIlEcWuTX9zKIg2mbw0LRFIsXwrfoVxn+ef0X13lw=
k8s.io/client-go v0.34.4 h1:IXhvzFdm0e897kXtLbeyMpAGzontcShJ/gi/XCCsOLc=
k8s.io/client-go v0.34.4/go.mod h1:tXIVJTQabT5QRGlFdxZQFxrIhcGUPpKL5DAc4gSWTE8=
k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOPolHyvO06MXG5TUIj2mNAA=
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
sigs.k8s.io/controller-runtime v0.22.4 h1:GEjV7KV3TY8e+tJ2LCTxUTanW4z/FmNB7l327UfMq9A=
sigs.k8s.io/controller-runtime v0.22.4/go.mod h1:+QX1XUpTXN4mLoblf4tqr5CQcyHPAki2HLXqQMY6vh8=
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
sigs.k8s.io/randfill v1.0.0 h1:JfjMILfT8A6RbawdsK2JXGBR5AQVfd+9TbzrlneTyrU=
sigs.k8s.io/randfill v1.0.0/go.mod h1:XeLlZ/jmk4i1HRopwe7/aU3H5n1zNUcX6TM94b3QxOY=
sigs.k8s.io/structured-merge-diff/v6 v6.3.0 h1:jTijUJbW353oVOd9oTlifJqOGEkUw2jB/fXCbTiQEco=
sigs.k8s.io/structured-merge-diff/v6 v6.3.0/go.mod h1:M3W8sfWvn2HhQDIbGWj3S099YozAsymCo/wrT5ohRUE=
sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs=
sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4=
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: cephrgw-system
@@ -0,0 +1,80 @@
---
# radosgw admin credentials for local testing. Replace the values, or create the
# Secret out-of-band, before applying. Keys map 1:1 to the operator env vars.
# The access/secret key belong to an RGW user with admin caps
# (users=*, buckets=*, metadata=read).
apiVersion: v1
kind: Secret
metadata:
name: cephrgw-credentials
namespace: cephrgw-system
type: Opaque
stringData:
# radosgw endpoint the operator talks to (Admin Ops + S3 APIs).
CEPH_RGW_ADMIN_ENDPOINT: "https://radosgw.service.consul:443"
CEPH_RGW_ACCESS_KEY: "change-me"
CEPH_RGW_SECRET_KEY: "change-me"
# The S3 endpoint written into credential Secrets for consumers (may be a
# public name that differs from the API endpoint above).
CEPH_RGW_ENDPOINT: "https://s3.ceph.unkin.net"
# Optional: SigV4 credential-scope region (defaults to "default").
# CEPH_RGW_REGION: "default"
# Optional: set to "true" to skip TLS verification (dev only).
# CEPH_RGW_INSECURE: "true"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cephrgw-operator
namespace: cephrgw-system
labels:
app.kubernetes.io/name: cephrgw-operator
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: cephrgw-operator
template:
metadata:
labels:
app.kubernetes.io/name: cephrgw-operator
spec:
serviceAccountName: cephrgw-operator
securityContext:
runAsNonRoot: true
containers:
- name: operator
image: cephrgw-operator:dev
imagePullPolicy: IfNotPresent
args:
- --metrics-bind-address=:8080
- --health-probe-bind-address=:8081
- --leader-elect
envFrom:
- secretRef:
name: cephrgw-credentials
ports:
- containerPort: 8080
name: metrics
- containerPort: 8081
name: health
readinessProbe:
httpGet:
path: /readyz
port: 8081
livenessProbe:
httpGet:
path: /healthz
port: 8081
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 256Mi
+37
View File
@@ -0,0 +1,37 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephrgw-operator
namespace: cephrgw-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cephrgw-operator
rules:
- apiGroups: ["ceph.unkin.net"]
resources: ["*"]
verbs: ["*"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cephrgw-operator
subjects:
- kind: ServiceAccount
name: cephrgw-operator
namespace: cephrgw-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cephrgw-operator
+239
View File
@@ -0,0 +1,239 @@
package ceph
import (
"context"
"encoding/json"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/s3"
s3types "github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/ceph/go-ceph/rgw/admin"
)
// BucketInfo is the subset of an RGW bucket record the operator consumes, as
// returned by the Admin Ops API (GET /admin/bucket).
type BucketInfo struct {
Bucket string
Bid string
ID string
Owner string
// PlacementRule is the placement target RGW stores the bucket on (e.g.
// "default-placement" or "ec"), read from the Admin Ops bucket stats.
PlacementRule string
// Zonegroup is the RGW zonegroup id the bucket belongs to.
Zonegroup string
}
// Name returns the bucket name regardless of the field radosgw used.
func (b *BucketInfo) Name() string {
if b.Bucket != "" {
return b.Bucket
}
return b.Bid
}
// InstanceID returns the RGW bucket instance id.
func (b *BucketInfo) InstanceID() string { return b.ID }
// CreateBucketSpec describes a bucket to create.
type CreateBucketSpec struct {
Bucket string
OwnerUID string
Zonegroup string
PlacementTarget string
LockEnabled bool
LockMode string
LockDays *int32
LockYears *int32
}
// GetBucket fetches a bucket by name via the Admin Ops API, returning an error
// classified by IsNotFound (admin.ErrNoSuchBucket) when it does not exist.
func (c *Client) GetBucket(ctx context.Context, name string) (*BucketInfo, error) {
b, err := c.admin.GetBucketInfo(ctx, admin.Bucket{Bucket: name})
if err != nil {
return nil, err
}
return &BucketInfo{
Bucket: b.Bucket,
ID: b.ID,
Owner: b.Owner,
PlacementRule: b.PlacementRule,
Zonegroup: b.Zonegroup,
}, nil
}
// CreateBucket provisions a bucket owned by spec.OwnerUID. The Admin Ops API
// cannot create buckets, so the operator issues an S3 CreateBucket signed as the
// owner (which makes the owner the bucket owner directly).
func (c *Client) CreateBucket(ctx context.Context, spec CreateBucketSpec) (*BucketInfo, error) {
owner, err := c.asOwner(ctx, spec.OwnerUID)
if err != nil {
return nil, err
}
input := &s3.CreateBucketInput{Bucket: aws.String(spec.Bucket)}
if loc := locationConstraint(spec.Zonegroup, spec.PlacementTarget); loc != "" {
input.CreateBucketConfiguration = &s3types.CreateBucketConfiguration{
LocationConstraint: s3types.BucketLocationConstraint(loc),
}
}
if spec.LockEnabled {
input.ObjectLockEnabledForBucket = aws.Bool(true)
}
if _, err := c.s3.CreateBucket(ctx, input, owner); err != nil && !IsConflict(err) {
return nil, err
}
// Apply a default object-lock retention when requested.
if spec.LockEnabled && spec.LockMode != "" && (spec.LockDays != nil || spec.LockYears != nil) {
if err := c.setObjectLockDefault(ctx, owner, spec); err != nil {
return nil, err
}
}
return c.GetBucket(ctx, spec.Bucket)
}
// SetBucketVersioning enables or suspends S3 versioning on a bucket. bucketID is
// unused (kept for call-site stability).
func (c *Client) SetBucketVersioning(ctx context.Context, name, bucketID, ownerUID string, enabled bool) error {
owner, err := c.asOwner(ctx, ownerUID)
if err != nil {
return err
}
status := s3types.BucketVersioningStatusSuspended
if enabled {
status = s3types.BucketVersioningStatusEnabled
}
_, err = c.s3.PutBucketVersioning(ctx, &s3.PutBucketVersioningInput{
Bucket: aws.String(name),
VersioningConfiguration: &s3types.VersioningConfiguration{Status: status},
}, owner)
return err
}
// SetBucketPolicy replaces the S3 bucket policy. An empty policy clears it.
// bucketID is unused (kept for call-site stability).
func (c *Client) SetBucketPolicy(ctx context.Context, name, bucketID, ownerUID, policy string) error {
owner, err := c.asOwner(ctx, ownerUID)
if err != nil {
return err
}
if policy == "" {
_, err := c.s3.DeleteBucketPolicy(ctx, &s3.DeleteBucketPolicyInput{Bucket: aws.String(name)}, owner)
if IsNotFound(err) {
return nil
}
return err
}
_, err = c.s3.PutBucketPolicy(ctx, &s3.PutBucketPolicyInput{
Bucket: aws.String(name),
Policy: aws.String(policy),
}, owner)
return err
}
// GetBucketPolicy returns the bucket's current S3 policy JSON, or "" when it has
// none. It is signed as the bucket owner.
func (c *Client) GetBucketPolicy(ctx context.Context, name, ownerUID string) (string, error) {
owner, err := c.asOwner(ctx, ownerUID)
if err != nil {
return "", err
}
out, err := c.s3.GetBucketPolicy(ctx, &s3.GetBucketPolicyInput{Bucket: aws.String(name)}, owner)
if err != nil {
if IsNotFound(err) {
return "", nil
}
return "", err
}
if out.Policy == nil {
return "", nil
}
return *out.Policy, nil
}
// SetBucketTags replaces the bucket tag set. tagsJSON is the JSON produced by
// BuildTagJSON (a list of {"Key","Value"} objects). bucketID is unused (kept for
// call-site stability).
func (c *Client) SetBucketTags(ctx context.Context, name, bucketID, ownerUID, tagsJSON string) error {
owner, err := c.asOwner(ctx, ownerUID)
if err != nil {
return err
}
type tag struct {
Key string `json:"Key"`
Value string `json:"Value"`
}
var tags []tag
if tagsJSON != "" {
if err := json.Unmarshal([]byte(tagsJSON), &tags); err != nil {
return fmt.Errorf("ceph: parse bucket tags: %w", err)
}
}
if len(tags) == 0 {
_, err := c.s3.DeleteBucketTagging(ctx, &s3.DeleteBucketTaggingInput{Bucket: aws.String(name)}, owner)
if IsNotFound(err) {
return nil
}
return err
}
tagSet := make([]s3types.Tag, 0, len(tags))
for _, t := range tags {
tagSet = append(tagSet, s3types.Tag{Key: aws.String(t.Key), Value: aws.String(t.Value)})
}
_, err = c.s3.PutBucketTagging(ctx, &s3.PutBucketTaggingInput{
Bucket: aws.String(name),
Tagging: &s3types.Tagging{TagSet: tagSet},
}, owner)
return err
}
// DeleteBucket removes a bucket via the Admin Ops API. When purge is true its
// objects are deleted too. A NoSuchBucket response is treated as success.
func (c *Client) DeleteBucket(ctx context.Context, name string, purge bool) error {
err := c.admin.RemoveBucket(ctx, admin.Bucket{Bucket: name, PurgeObject: &purge})
if IsNotFound(err) {
return nil
}
return err
}
// setObjectLockDefault sets the bucket's default object-lock retention.
func (c *Client) setObjectLockDefault(ctx context.Context, owner func(*s3.Options), spec CreateBucketSpec) error {
_, err := c.s3.PutObjectLockConfiguration(ctx, &s3.PutObjectLockConfigurationInput{
Bucket: aws.String(spec.Bucket),
ObjectLockConfiguration: &s3types.ObjectLockConfiguration{
ObjectLockEnabled: s3types.ObjectLockEnabledEnabled,
Rule: &s3types.ObjectLockRule{
DefaultRetention: &s3types.DefaultRetention{
Mode: s3types.ObjectLockRetentionMode(spec.LockMode),
Days: spec.LockDays,
Years: spec.LockYears,
},
},
},
}, owner)
return err
}
// locationConstraint renders the RGW S3 CreateBucket LocationConstraint from a
// zonegroup api-name and a placement target. RGW's S3 create-bucket handler
// splits the value on the first ":" — the part before is the zonegroup api-name,
// the part after is the placement target id. An empty zonegroup (the common
// case) yields ":<placement>", which selects the local/master zonegroup with the
// given placement, so callers need not know the zonegroup's api-name to pick a
// placement target. Both empty yields "" (no constraint: user/zonegroup
// default). Placement empty with a zonegroup set yields just the zonegroup.
func locationConstraint(zonegroup, placement string) string {
loc := zonegroup
if placement != "" {
loc = zonegroup + ":" + placement
}
return loc
}
+68
View File
@@ -0,0 +1,68 @@
package ceph
import (
"context"
"net/http"
"net/http/httptest"
"testing"
)
func TestLocationConstraint(t *testing.T) {
cases := []struct {
name string
zonegroup string
placement string
want string
}{
{"both empty -> no constraint", "", "", ""},
{"placement only -> local zonegroup", "", "ec", ":ec"},
{"placement only default target", "", "default-placement", ":default-placement"},
{"zonegroup and placement", "default", "ec", "default:ec"},
{"zonegroup only", "default", "", "default"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := locationConstraint(tc.zonegroup, tc.placement); got != tc.want {
t.Errorf("locationConstraint(%q,%q)=%q want %q", tc.zonegroup, tc.placement, got, tc.want)
}
})
}
}
// TestGetBucketPlacement verifies GetBucket surfaces the placement target and
// zonegroup from the Admin Ops bucket-stats response, so the controller can
// detect placement drift.
func TestGetBucketPlacement(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"bucket": "raw-archive",
"id": "eae688bc-ee35-445d-9188-111b73c8b4a0.12345.1",
"owner": "logarchiver",
"zonegroup": "eae688bc-ee35-445d-9188-111b73c8b4a0",
"placement_rule": "ec"
}`))
}))
defer srv.Close()
c, err := NewClient(Config{Endpoint: srv.URL, AccessKey: "a", SecretKey: "s"})
if err != nil {
t.Fatalf("NewClient: %v", err)
}
info, err := c.GetBucket(context.Background(), "raw-archive")
if err != nil {
t.Fatalf("GetBucket: %v", err)
}
if info.PlacementRule != "ec" {
t.Errorf("PlacementRule=%q want %q", info.PlacementRule, "ec")
}
if info.Zonegroup != "eae688bc-ee35-445d-9188-111b73c8b4a0" {
t.Errorf("Zonegroup=%q unexpected", info.Zonegroup)
}
if info.Owner != "logarchiver" {
t.Errorf("Owner=%q want logarchiver", info.Owner)
}
if info.Name() != "raw-archive" {
t.Errorf("Name()=%q want raw-archive", info.Name())
}
}
+224
View File
@@ -0,0 +1,224 @@
// Package ceph is a small client for the Ceph RGW (radosgw) admin and S3 APIs,
// scoped to the user, bucket and policy operations the operator needs.
//
// It talks directly to radosgw (e.g. https://radosgw.service.consul:443) rather
// than the manager dashboard, via two native Go libraries:
//
// - github.com/ceph/go-ceph/rgw/admin drives the RGW Admin Ops API
// (/admin/...), signed with the operator's admin access/secret key, to
// manage users, keys, quotas and bucket info/removal.
// - github.com/aws/aws-sdk-go-v2/service/s3 drives the S3 API (/), signed as
// the bucket's owner, to create buckets and set versioning, tagging, policy
// and object lock — operations the Admin Ops API does not expose.
package ceph
import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"net/http"
"strings"
"sync"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
smithy "github.com/aws/smithy-go"
awshttp "github.com/aws/smithy-go/transport/http"
"github.com/ceph/go-ceph/rgw/admin"
)
// Config configures a Client.
type Config struct {
// Endpoint is the radosgw root, e.g. https://radosgw.service.consul:443.
Endpoint string
// AccessKey / SecretKey are the S3 credentials of an RGW user holding the
// admin caps the operator needs (users=*, buckets=*).
AccessKey string
SecretKey string
// Region is the SigV4 credential-scope region used for S3 requests. radosgw
// verifies the signature against whatever region the client used, so any
// consistent value works; defaults to "default". (The go-ceph admin client
// always signs its own requests with region "default".)
Region string
// CACert is an optional PEM bundle used to verify the radosgw TLS cert.
CACert []byte
// Insecure disables TLS verification (not recommended).
Insecure bool
// Timeout bounds each HTTP request. Defaults to 30s.
Timeout time.Duration
}
// Client talks to radosgw. It is safe for concurrent use.
type Client struct {
admin *admin.API
s3 *s3.Client
region string
// keyCache memoises owner uid -> S3 credentials (via the Admin Ops API) so
// per-owner S3 calls do not re-fetch keys on every reconcile.
mu sync.Mutex
keyCache map[string]aws.CredentialsProvider
}
// notFoundCodes and conflictCodes classify RGW/S3 error codes that surface only
// as a generic smithy.APIError (i.e. not a modeled S3 error type).
var notFoundCodes = map[string]bool{
"NoSuchUser": true, "NoSuchBucket": true, "NoSuchKey": true,
"NoSuchBucketPolicy": true, "NoSuchTagSet": true,
"NoSuchTagSetError": true, "NotFound": true,
}
var conflictCodes = map[string]bool{
"BucketAlreadyExists": true, "BucketAlreadyOwnedByYou": true, "UserAlreadyExists": true,
}
// IsNotFound reports whether err represents a missing user, bucket, key, policy
// or tag set, on either the admin or the S3 path.
func IsNotFound(err error) bool {
if err == nil {
return false
}
if errors.Is(err, admin.ErrNoSuchUser) || errors.Is(err, admin.ErrNoSuchBucket) ||
errors.Is(err, admin.ErrNoSuchKey) || errors.Is(err, admin.ErrNoSuchObject) {
return true
}
var apiErr smithy.APIError
if errors.As(err, &apiErr) && notFoundCodes[apiErr.ErrorCode()] {
return true
}
var respErr *awshttp.ResponseError
if errors.As(err, &respErr) && respErr.HTTPStatusCode() == http.StatusNotFound {
return true
}
return false
}
// IsConflict reports whether err represents an already-exists conflict on either
// the admin or the S3 path.
func IsConflict(err error) bool {
if err == nil {
return false
}
if errors.Is(err, admin.ErrUserExists) || errors.Is(err, admin.ErrEmailExists) ||
errors.Is(err, admin.ErrKeyExists) || errors.Is(err, admin.ErrBucketNotEmpty) {
return true
}
var apiErr smithy.APIError
if errors.As(err, &apiErr) && conflictCodes[apiErr.ErrorCode()] {
return true
}
var respErr *awshttp.ResponseError
if errors.As(err, &respErr) && respErr.HTTPStatusCode() == http.StatusConflict {
return true
}
return false
}
// NewClient validates cfg and builds a Client.
func NewClient(cfg Config) (*Client, error) {
endpoint := strings.TrimRight(cfg.Endpoint, "/")
if endpoint == "" {
return nil, fmt.Errorf("ceph: radosgw endpoint is required")
}
if cfg.AccessKey == "" || cfg.SecretKey == "" {
return nil, fmt.Errorf("ceph: radosgw admin access and secret key are required")
}
tlsCfg := &tls.Config{InsecureSkipVerify: cfg.Insecure} //nolint:gosec // opt-in via config
if len(cfg.CACert) > 0 {
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(cfg.CACert) {
return nil, fmt.Errorf("ceph: failed to parse CA certificate PEM")
}
tlsCfg.RootCAs = pool
}
timeout := cfg.Timeout
if timeout == 0 {
timeout = 30 * time.Second
}
region := cfg.Region
if region == "" {
region = "default"
}
httpClient := &http.Client{
Timeout: timeout,
Transport: &http.Transport{TLSClientConfig: tlsCfg},
}
adminAPI, err := admin.New(endpoint, cfg.AccessKey, cfg.SecretKey, httpClient)
if err != nil {
return nil, fmt.Errorf("ceph: build admin client: %w", err)
}
s3Client := s3.New(s3.Options{
Region: region,
Credentials: credentials.NewStaticCredentialsProvider(cfg.AccessKey, cfg.SecretKey, ""),
HTTPClient: httpClient,
BaseEndpoint: aws.String(endpoint),
// radosgw serves buckets path-style, not virtual-host style.
UsePathStyle: true,
// radosgw (pre-Reef backports) rejects the SDK's default CRC32 /
// aws-chunked integrity protections; only send checksums when the API
// requires them.
RequestChecksumCalculation: aws.RequestChecksumCalculationWhenRequired,
ResponseChecksumValidation: aws.ResponseChecksumValidationWhenRequired,
})
return &Client{
admin: adminAPI,
s3: s3Client,
region: region,
keyCache: map[string]aws.CredentialsProvider{},
}, nil
}
// asOwner returns a per-call S3 option that signs the request as the RGW user
// uid, looking up (and caching) the user's first key pair via the Admin Ops API.
// Signing S3 sub-resource operations as the bucket owner (rather than the admin
// user) makes the owner the bucket owner directly and keeps RGW's per-user S3
// authorization intact.
func (c *Client) asOwner(ctx context.Context, uid string) (func(*s3.Options), error) {
c.mu.Lock()
provider, ok := c.keyCache[uid]
c.mu.Unlock()
if !ok {
user, err := c.GetUser(ctx, uid)
if err != nil {
return nil, err
}
key, has := user.S3Key()
if !has {
return nil, fmt.Errorf("ceph: user %s has no S3 keys to sign bucket operations", uid)
}
provider = credentials.NewStaticCredentialsProvider(key.AccessKey, key.SecretKey, "")
c.mu.Lock()
c.keyCache[uid] = provider
c.mu.Unlock()
}
return func(o *s3.Options) { o.Credentials = provider }, nil
}
// forgetIdentity drops any cached S3 credentials for uid, e.g. after its keys
// may have changed.
func (c *Client) forgetIdentity(uid string) {
c.mu.Lock()
delete(c.keyCache, uid)
c.mu.Unlock()
}
// Ping verifies connectivity and that the admin credentials sign correctly. It
// asks the Admin Ops API for a sentinel user: a NoSuchUser answer still proves
// the request authenticated, so only transport/auth errors fail the check.
func (c *Client) Ping(ctx context.Context) error {
_, err := c.GetUser(ctx, "cephrgw-operator-ping-nonexistent")
if err == nil || IsNotFound(err) {
return nil
}
return err
}
+81
View File
@@ -0,0 +1,81 @@
package ceph
import (
"testing"
s3types "github.com/aws/aws-sdk-go-v2/service/s3/types"
smithy "github.com/aws/smithy-go"
"github.com/ceph/go-ceph/rgw/admin"
)
func TestNewClientValidation(t *testing.T) {
cases := []struct {
name string
cfg Config
wantErr bool
}{
{"ok", Config{Endpoint: "https://rgw:443", AccessKey: "a", SecretKey: "s"}, false},
{"no endpoint", Config{AccessKey: "a", SecretKey: "s"}, true},
{"no access key", Config{Endpoint: "https://rgw:443", SecretKey: "s"}, true},
{"no secret key", Config{Endpoint: "https://rgw:443", AccessKey: "a"}, true},
{"bad ca", Config{Endpoint: "https://rgw:443", AccessKey: "a", SecretKey: "s", CACert: []byte("not pem")}, true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
_, err := NewClient(tc.cfg)
if (err != nil) != tc.wantErr {
t.Fatalf("NewClient err=%v wantErr=%v", err, tc.wantErr)
}
})
}
}
func TestIsNotFound(t *testing.T) {
cases := []struct {
name string
err error
want bool
}{
{"nil", nil, false},
{"admin no such user", admin.ErrNoSuchUser, true},
{"admin no such bucket", admin.ErrNoSuchBucket, true},
{"admin no such key", admin.ErrNoSuchKey, true},
{"s3 no such bucket", &s3types.NoSuchBucket{}, true},
{"s3 no such key", &s3types.NoSuchKey{}, true},
{"generic no such bucket policy", &smithy.GenericAPIError{Code: "NoSuchBucketPolicy"}, true},
{"generic no such tag set", &smithy.GenericAPIError{Code: "NoSuchTagSet"}, true},
{"admin user exists is not notfound", admin.ErrUserExists, false},
{"unrelated", &smithy.GenericAPIError{Code: "AccessDenied"}, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := IsNotFound(tc.err); got != tc.want {
t.Errorf("IsNotFound(%v)=%v want %v", tc.err, got, tc.want)
}
})
}
}
func TestIsConflict(t *testing.T) {
cases := []struct {
name string
err error
want bool
}{
{"nil", nil, false},
{"admin user exists", admin.ErrUserExists, true},
{"admin bucket not empty", admin.ErrBucketNotEmpty, true},
{"s3 bucket already owned by you", &s3types.BucketAlreadyOwnedByYou{}, true},
{"s3 bucket already exists", &s3types.BucketAlreadyExists{}, true},
{"generic bucket already exists", &smithy.GenericAPIError{Code: "BucketAlreadyExists"}, true},
{"admin no such user is not conflict", admin.ErrNoSuchUser, false},
{"unrelated", &smithy.GenericAPIError{Code: "AccessDenied"}, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := IsConflict(tc.err); got != tc.want {
t.Errorf("IsConflict(%v)=%v want %v", tc.err, got, tc.want)
}
})
}
}
+398
View File
@@ -0,0 +1,398 @@
package ceph
import (
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
)
// Access levels mirrored from the API package to avoid an import cycle; the
// controllers translate their typed level into these strings.
const (
LevelReadOnly = "read-only"
LevelReadWrite = "read-write"
LevelFull = "full"
)
// managedSidPrefix marks statement ids the operator owns, so it can reconcile
// its own statements while preserving foreign ones when adopting a bucket that
// already has a policy. Do not reuse this prefix (or the legacy prefixes below)
// for statements you manage yourself.
const managedSidPrefix = "cephrgwop"
// legacyManagedSidPrefixes are the statement-id prefixes the operator emitted
// before managedSidPrefix existed; they are still recognised as operator-owned
// so upgrading does not duplicate statements.
var legacyManagedSidPrefixes = []string{
"readonlybkt", "readonlyobj", "readwritebkt", "readwriteobj", "full", "custom", "raw",
}
// GrantConditions restricts when a grant's statements apply. The zero value adds
// no conditions.
type GrantConditions struct {
// SourceIPs restricts the grant to these CIDRs (S3 aws:SourceIp).
SourceIPs []string
// SecureTransportOnly requires TLS (S3 aws:SecureTransport).
SecureTransportOnly bool
}
// RawStatement is a caller-supplied S3 policy statement for a grant.
type RawStatement struct {
Sid string
Effect string
Actions []string
Resources []string
Condition map[string]map[string][]string
}
// Grant couples an RGW user id with the access it should have on a bucket. The
// simple form is a Level; Paths, Actions and Conditions refine it, and Raw
// replaces it entirely with caller-supplied statements.
type Grant struct {
UID string
Level string
// Paths scopes object-level access to these key prefixes; empty = whole
// bucket.
Paths []string
// Actions overrides the level's action set; empty = derive from Level.
Actions []string
// Conditions optionally restricts when the grant applies.
Conditions *GrantConditions
// Raw, when non-empty, replaces Level/Actions/Paths/Conditions with these
// statements (the operator still fills in a Principal when one is omitted).
Raw []RawStatement
}
type policyDocument struct {
Version string `json:"Version"`
Statement []policyStatement `json:"Statement"`
}
type policyStatement struct {
Sid string `json:"Sid,omitempty"`
Effect string `json:"Effect"`
Principal map[string][]string `json:"Principal,omitempty"`
Action []string `json:"Action"`
Resource []string `json:"Resource"`
Condition map[string]map[string][]string `json:"Condition,omitempty"`
}
// bucket-level and object-level S3 actions per access level.
var bucketActions = map[string][]string{
LevelReadOnly: {
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketVersions",
},
LevelReadWrite: {
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketVersions",
"s3:ListBucketMultipartUploads",
},
}
var objectActions = map[string][]string{
LevelReadOnly: {
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetObjectTagging",
},
LevelReadWrite: {
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetObjectTagging",
"s3:PutObject",
"s3:PutObjectTagging",
"s3:DeleteObject",
"s3:DeleteObjectVersion",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts",
},
}
// BuildBucketPolicy renders a deterministic S3 bucket policy granting each
// principal its requested access. It returns "" when there are no grants so the
// caller can clear the policy.
func BuildBucketPolicy(bucket string, grants []Grant) (string, error) {
statements := buildStatements(bucket, grants)
if len(statements) == 0 {
return "", nil
}
b, err := json.Marshal(policyDocument{Version: "2012-10-17", Statement: statements})
if err != nil {
return "", err
}
return string(b), nil
}
// MergeBucketPolicy renders the operator's statements for grants and merges them
// into an existing policy, preserving any statement the operator does not own
// (identified by its Sid). It returns "" only when the merged policy would be
// empty, so a bucket adopted with a hand-written policy keeps that policy.
func MergeBucketPolicy(existing, bucket string, grants []Grant) (string, error) {
version := "2012-10-17"
var id string
var foreign []json.RawMessage
if strings.TrimSpace(existing) != "" {
var doc struct {
Version string `json:"Version"`
ID string `json:"Id,omitempty"`
Statement []json.RawMessage `json:"Statement"`
}
if err := json.Unmarshal([]byte(existing), &doc); err != nil {
return "", fmt.Errorf("parse existing bucket policy: %w", err)
}
if doc.Version != "" {
version = doc.Version
}
id = doc.ID
for _, raw := range doc.Statement {
var meta struct {
Sid string `json:"Sid"`
}
// Ignore unmarshal errors: a statement we cannot read the Sid of is
// treated as foreign and preserved verbatim.
_ = json.Unmarshal(raw, &meta)
if isManagedSid(meta.Sid) {
continue // operator-owned; re-rendered below
}
foreign = append(foreign, raw)
}
}
managed := buildStatements(bucket, grants)
if len(foreign) == 0 && len(managed) == 0 {
return "", nil
}
statements := make([]json.RawMessage, 0, len(foreign)+len(managed))
statements = append(statements, foreign...)
for _, st := range managed {
b, err := json.Marshal(st)
if err != nil {
return "", err
}
statements = append(statements, b)
}
b, err := json.Marshal(struct {
Version string `json:"Version"`
ID string `json:"Id,omitempty"`
Statement []json.RawMessage `json:"Statement"`
}{Version: version, ID: id, Statement: statements})
if err != nil {
return "", err
}
return string(b), nil
}
// isManagedSid reports whether a statement id was emitted by the operator.
func isManagedSid(s string) bool {
if strings.HasPrefix(s, managedSidPrefix) {
return true
}
for _, p := range legacyManagedSidPrefixes {
if strings.HasPrefix(s, p) {
return true
}
}
return false
}
// buildStatements renders the operator's statements for grants, sorted for
// deterministic output.
func buildStatements(bucket string, grants []Grant) []policyStatement {
if len(grants) == 0 {
return nil
}
sorted := make([]Grant, len(grants))
copy(sorted, grants)
sort.Slice(sorted, func(i, j int) bool {
if sorted[i].UID == sorted[j].UID {
return sorted[i].Level < sorted[j].Level
}
return sorted[i].UID < sorted[j].UID
})
bucketARN := "arn:aws:s3:::" + bucket
var statements []policyStatement
for _, g := range sorted {
statements = append(statements, statementsForGrant(bucketARN, g)...)
}
return statements
}
// statementsForGrant renders the policy statements for a single grant.
func statementsForGrant(bucketARN string, g Grant) []policyStatement {
principal := map[string][]string{"AWS": {"arn:aws:iam:::user/" + g.UID}}
if len(g.Raw) > 0 {
out := make([]policyStatement, 0, len(g.Raw))
for i, rs := range g.Raw {
st := policyStatement{
// Always operator-owned so a merge re-renders (not duplicates)
// it; any user-supplied Sid is folded into the managed id.
Sid: sid(firstNonEmpty(rs.Sid, "raw"+strconv.Itoa(i)), g.UID),
Effect: firstNonEmpty(rs.Effect, "Allow"),
Principal: principal,
Action: rs.Actions,
Resource: resolveResources(bucketARN, rs.Resources),
Condition: rs.Condition,
}
out = append(out, st)
}
return out
}
cond := buildCondition(g.Conditions)
objectARNs := objectResources(bucketARN, g.Paths)
if len(g.Actions) > 0 {
return []policyStatement{{
Sid: sid("custom", g.UID),
Effect: "Allow",
Principal: principal,
Action: g.Actions,
Resource: append([]string{bucketARN}, objectARNs...),
Condition: cond,
}}
}
if g.Level == LevelFull {
return []policyStatement{{
Sid: sid("full", g.UID),
Effect: "Allow",
Principal: principal,
Action: []string{"s3:*"},
Resource: append([]string{bucketARN}, objectARNs...),
Condition: cond,
}}
}
return []policyStatement{
{
Sid: sid(g.Level+"-bkt", g.UID),
Effect: "Allow",
Principal: principal,
Action: bucketActions[g.Level],
Resource: []string{bucketARN},
Condition: cond,
},
{
Sid: sid(g.Level+"-obj", g.UID),
Effect: "Allow",
Principal: principal,
Action: objectActions[g.Level],
Resource: objectARNs,
Condition: cond,
},
}
}
// objectResources renders the object-level resource ARNs for a grant: the whole
// bucket ("<bucket>/*") when no paths are given, or one "<bucket>/<prefix>*" per
// prefix (deduplicated and sorted for determinism).
func objectResources(bucketARN string, paths []string) []string {
if len(paths) == 0 {
return []string{bucketARN + "/*"}
}
seen := map[string]struct{}{}
out := make([]string, 0, len(paths))
for _, p := range paths {
p = strings.TrimPrefix(p, "/")
arn := bucketARN + "/" + p + "*"
if _, dup := seen[arn]; dup {
continue
}
seen[arn] = struct{}{}
out = append(out, arn)
}
sort.Strings(out)
return out
}
// resolveResources renders raw-statement resources: entries that already look
// like ARNs pass through verbatim; bucket-relative prefixes become
// "<bucket>/<prefix>*". An empty list defaults to the whole bucket and objects.
func resolveResources(bucketARN string, resources []string) []string {
if len(resources) == 0 {
return []string{bucketARN, bucketARN + "/*"}
}
out := make([]string, 0, len(resources))
for _, r := range resources {
switch {
case strings.HasPrefix(r, "arn:"):
out = append(out, r)
case r == "" || r == "/":
out = append(out, bucketARN+"/*")
default:
out = append(out, bucketARN+"/"+strings.TrimPrefix(r, "/")+"*")
}
}
return out
}
// buildCondition renders the S3 condition block for a grant, or nil when there
// is nothing to add.
func buildCondition(c *GrantConditions) map[string]map[string][]string {
if c == nil {
return nil
}
cond := map[string]map[string][]string{}
if len(c.SourceIPs) > 0 {
cond["IpAddress"] = map[string][]string{"aws:SourceIp": c.SourceIPs}
}
if c.SecureTransportOnly {
cond["Bool"] = map[string][]string{"aws:SecureTransport": {"true"}}
}
if len(cond) == 0 {
return nil
}
return cond
}
// sid builds a policy statement id that only contains characters S3 accepts,
// prefixed with managedSidPrefix so the operator can recognise its own
// statements when merging into an adopted bucket's policy.
func sid(prefix, uid string) string {
var b strings.Builder
b.WriteString(managedSidPrefix)
b.WriteString(strings.ReplaceAll(prefix, "-", ""))
for _, r := range uid {
switch {
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9':
b.WriteRune(r)
}
}
return b.String()
}
// BuildTagJSON renders bucket tags as a {Key,Value} JSON list, the intermediate
// form SetBucketTags parses and re-encodes into the S3 Tagging XML document.
func BuildTagJSON(tags map[string]string) (string, error) {
if len(tags) == 0 {
return "", nil
}
keys := make([]string, 0, len(tags))
for k := range tags {
keys = append(keys, k)
}
sort.Strings(keys)
type kv struct {
Key string `json:"Key"`
Value string `json:"Value"`
}
out := make([]kv, 0, len(keys))
for _, k := range keys {
out = append(out, kv{Key: k, Value: tags[k]})
}
b, err := json.Marshal(out)
if err != nil {
return "", err
}
return string(b), nil
}
+360
View File
@@ -0,0 +1,360 @@
package ceph
import (
"encoding/json"
"strings"
"testing"
)
func TestBuildBucketPolicyEmpty(t *testing.T) {
got, err := BuildBucketPolicy("data", nil)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got != "" {
t.Fatalf("expected empty policy for no grants, got %q", got)
}
}
func TestBuildBucketPolicyDeterministic(t *testing.T) {
a, err := BuildBucketPolicy("data", []Grant{
{UID: "reader", Level: LevelReadOnly},
{UID: "writer", Level: LevelReadWrite},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
b, err := BuildBucketPolicy("data", []Grant{
{UID: "writer", Level: LevelReadWrite},
{UID: "reader", Level: LevelReadOnly},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if a != b {
t.Fatalf("policy is order-dependent:\n a=%s\n b=%s", a, b)
}
}
func TestBuildBucketPolicyStructure(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "reader", Level: LevelReadOnly},
{UID: "admin", Level: LevelFull},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
var doc struct {
Version string `json:"Version"`
Statement []struct {
Effect string `json:"Effect"`
Principal map[string][]string `json:"Principal"`
Action []string `json:"Action"`
Resource []string `json:"Resource"`
} `json:"Statement"`
}
if err := json.Unmarshal([]byte(raw), &doc); err != nil {
t.Fatalf("policy is not valid JSON: %v\n%s", err, raw)
}
if doc.Version != "2012-10-17" {
t.Fatalf("unexpected version %q", doc.Version)
}
// read-only -> two statements (bucket + object); full -> one statement.
if len(doc.Statement) != 3 {
t.Fatalf("expected 3 statements, got %d", len(doc.Statement))
}
var sawFullWildcard, sawReaderPrincipal bool
for _, s := range doc.Statement {
if s.Effect != "Allow" {
t.Fatalf("expected Allow effect, got %q", s.Effect)
}
for _, a := range s.Action {
if a == "s3:*" {
sawFullWildcard = true
}
}
for _, p := range s.Principal["AWS"] {
if strings.HasSuffix(p, "user/reader") {
sawReaderPrincipal = true
}
}
}
if !sawFullWildcard {
t.Fatal("full grant did not produce an s3:* action")
}
if !sawReaderPrincipal {
t.Fatal("reader principal ARN missing")
}
}
// parsedPolicy is a fuller parse of a rendered policy for the fine-grained tests.
type parsedPolicy struct {
Statement []struct {
Sid string `json:"Sid"`
Effect string `json:"Effect"`
Principal map[string][]string `json:"Principal"`
Action []string `json:"Action"`
Resource []string `json:"Resource"`
Condition map[string]map[string][]string `json:"Condition"`
} `json:"Statement"`
}
func parsePolicy(t *testing.T, raw string) parsedPolicy {
t.Helper()
var doc parsedPolicy
if err := json.Unmarshal([]byte(raw), &doc); err != nil {
t.Fatalf("policy is not valid JSON: %v\n%s", err, raw)
}
return doc
}
func TestBuildBucketPolicyPaths(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "reader", Level: LevelReadOnly, Paths: []string{"team-a/", "/shared/inbox/"}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
var objResources []string
for _, s := range doc.Statement {
for _, a := range s.Action {
if a == "s3:GetObject" {
objResources = s.Resource
}
}
}
want := map[string]bool{
"arn:aws:s3:::data/shared/inbox/*": false,
"arn:aws:s3:::data/team-a/*": false,
}
if len(objResources) != len(want) {
t.Fatalf("expected %d object resources, got %v", len(want), objResources)
}
for _, r := range objResources {
if _, ok := want[r]; !ok {
t.Fatalf("unexpected object resource %q (leading slash not trimmed?)", r)
}
want[r] = true
}
for r, seen := range want {
if !seen {
t.Fatalf("missing object resource %q", r)
}
}
}
func TestBuildBucketPolicyActionsOverride(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "svc", Level: LevelReadOnly, Actions: []string{"s3:GetObject", "s3:PutObject"}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
if len(doc.Statement) != 1 {
t.Fatalf("expected a single custom-action statement, got %d", len(doc.Statement))
}
s := doc.Statement[0]
if len(s.Action) != 2 || s.Action[0] != "s3:GetObject" || s.Action[1] != "s3:PutObject" {
t.Fatalf("actions not taken verbatim: %v", s.Action)
}
// Custom-action statement lists both the bucket and object resources.
if len(s.Resource) != 2 || s.Resource[0] != "arn:aws:s3:::data" || s.Resource[1] != "arn:aws:s3:::data/*" {
t.Fatalf("unexpected resources: %v", s.Resource)
}
}
func TestBuildBucketPolicyConditions(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "reader", Level: LevelReadOnly, Conditions: &GrantConditions{
SourceIPs: []string{"10.0.0.0/8"},
SecureTransportOnly: true,
}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
for _, s := range doc.Statement {
if s.Condition == nil {
t.Fatalf("statement %q missing condition block", s.Sid)
}
if ip := s.Condition["IpAddress"]["aws:SourceIp"]; len(ip) != 1 || ip[0] != "10.0.0.0/8" {
t.Fatalf("unexpected SourceIp condition: %v", s.Condition["IpAddress"])
}
if tls := s.Condition["Bool"]["aws:SecureTransport"]; len(tls) != 1 || tls[0] != "true" {
t.Fatalf("unexpected SecureTransport condition: %v", s.Condition["Bool"])
}
}
}
func TestBuildBucketPolicyRawStatements(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "svc", Level: LevelReadOnly, Raw: []RawStatement{
{
Effect: "Deny",
Actions: []string{"s3:DeleteObject"},
Resources: []string{"locked/", "arn:aws:s3:::other/*"},
},
}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
if len(doc.Statement) != 1 {
t.Fatalf("expected 1 raw statement, got %d", len(doc.Statement))
}
s := doc.Statement[0]
if s.Effect != "Deny" {
t.Fatalf("raw effect not honoured: %q", s.Effect)
}
// The operator fills in the principal; bucket-relative prefixes are expanded
// while explicit ARNs pass through.
if len(s.Principal["AWS"]) != 1 || !strings.HasSuffix(s.Principal["AWS"][0], "user/svc") {
t.Fatalf("raw statement principal not injected: %v", s.Principal)
}
if len(s.Resource) != 2 || s.Resource[0] != "arn:aws:s3:::data/locked/*" || s.Resource[1] != "arn:aws:s3:::other/*" {
t.Fatalf("unexpected raw resources: %v", s.Resource)
}
}
func TestMergeBucketPolicyPreservesForeign(t *testing.T) {
// An existing policy with a foreign statement (unknown Sid, and a scalar
// condition value S3 allows but our typed struct does not model).
existing := `{"Version":"2012-10-17","Statement":[` +
`{"Sid":"AllowPublicRead","Effect":"Allow","Principal":"*","Action":["s3:GetObject"],` +
`"Resource":"arn:aws:s3:::data/public/*","Condition":{"Bool":{"aws:SecureTransport":"true"}}}]}`
merged, err := MergeBucketPolicy(existing, "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicyRaw(t, merged)
var sawForeign, sawManaged bool
for _, raw := range doc.Statement {
var s struct {
Sid string `json:"Sid"`
Condition map[string]map[string]any
}
if err := json.Unmarshal(raw, &s); err != nil {
t.Fatalf("statement not valid JSON: %v", err)
}
if s.Sid == "AllowPublicRead" {
sawForeign = true
// The scalar condition value must survive verbatim.
if v := s.Condition["Bool"]["aws:SecureTransport"]; v != "true" {
t.Fatalf("foreign scalar condition mangled: %v", s.Condition)
}
}
if isManagedSid(s.Sid) {
sawManaged = true
}
}
if !sawForeign {
t.Fatal("foreign statement was dropped")
}
if !sawManaged {
t.Fatal("operator statement missing from merge")
}
}
func TestMergeBucketPolicyReplacesManaged(t *testing.T) {
// Two rounds: an existing policy already carrying the operator's statements
// must not accumulate duplicates when re-merged.
first, err := MergeBucketPolicy("", "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
second, err := MergeBucketPolicy(first, "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if first != second {
t.Fatalf("re-merging its own policy was not idempotent:\n first=%s\nsecond=%s", first, second)
}
// A legacy (unprefixed) operator statement must also be recognised and
// replaced rather than preserved as foreign.
legacy := `{"Version":"2012-10-17","Statement":[` +
`{"Sid":"readonlybktreader","Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/reader"]},` +
`"Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::data"]}]}`
merged, err := MergeBucketPolicy(legacy, "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
for _, raw := range parsePolicyRaw(t, merged).Statement {
var s struct {
Sid string `json:"Sid"`
}
_ = json.Unmarshal(raw, &s)
if s.Sid == "readonlybktreader" {
t.Fatal("legacy operator statement was preserved instead of replaced")
}
}
}
func TestMergeBucketPolicyForeignOnlyKept(t *testing.T) {
existing := `{"Version":"2012-10-17","Statement":[` +
`{"Sid":"AllowPublicRead","Effect":"Allow","Principal":"*","Action":["s3:GetObject"],` +
`"Resource":"arn:aws:s3:::data/*"}]}`
// No grants: the operator adds nothing but must not wipe the foreign policy.
merged, err := MergeBucketPolicy(existing, "data", nil)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if merged == "" {
t.Fatal("merge cleared a policy that had a foreign statement")
}
if len(parsePolicyRaw(t, merged).Statement) != 1 {
t.Fatalf("expected the single foreign statement, got %s", merged)
}
}
func TestMergeBucketPolicyEmpty(t *testing.T) {
merged, err := MergeBucketPolicy("", "data", nil)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if merged != "" {
t.Fatalf("expected empty policy, got %q", merged)
}
}
// parsePolicyRaw parses a policy keeping statements as raw JSON.
func parsePolicyRaw(t *testing.T, raw string) struct {
Version string `json:"Version"`
Statement []json.RawMessage `json:"Statement"`
} {
t.Helper()
var doc struct {
Version string `json:"Version"`
Statement []json.RawMessage `json:"Statement"`
}
if err := json.Unmarshal([]byte(raw), &doc); err != nil {
t.Fatalf("policy is not valid JSON: %v\n%s", err, raw)
}
return doc
}
func TestBuildBucketPolicyFineGrainedDeterministic(t *testing.T) {
grants := []Grant{
{UID: "reader", Level: LevelReadOnly, Paths: []string{"a/", "b/"}},
{UID: "svc", Level: LevelReadWrite, Conditions: &GrantConditions{SecureTransportOnly: true}},
}
a, err := BuildBucketPolicy("data", grants)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
b, err := BuildBucketPolicy("data", []Grant{grants[1], grants[0]})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if a != b {
t.Fatalf("fine-grained policy is order-dependent:\n a=%s\n b=%s", a, b)
}
}
+176
View File
@@ -0,0 +1,176 @@
package ceph
import (
"context"
"github.com/ceph/go-ceph/rgw/admin"
)
// UserKey is an S3 access/secret key pair belonging to an RGW user.
type UserKey struct {
User string
AccessKey string
SecretKey string
}
// User is the subset of an RGW user record the operator consumes.
type User struct {
UID string
DisplayName string
Email string
MaxBuckets int
Suspended int
Keys []UserKey
}
// S3Key returns the first access/secret key pair, if any.
func (u *User) S3Key() (UserKey, bool) {
if len(u.Keys) == 0 {
return UserKey{}, false
}
return u.Keys[0], true
}
// UserSpec describes the desired state of an RGW user. A nil DisplayName/
// Suspended (empty string / nil pointer) leaves that attribute untouched on an
// existing user, so an adopted user is not mutated unless the fields are set.
type UserSpec struct {
UID string
DisplayName string
Email string
MaxBuckets *int32
Suspended *bool
}
// fromAdminUser converts a go-ceph admin.User into the subset the operator uses.
func fromAdminUser(u admin.User) *User {
out := &User{
UID: u.ID,
DisplayName: u.DisplayName,
Email: u.Email,
MaxBuckets: derefInt(u.MaxBuckets),
Suspended: derefInt(u.Suspended),
}
for _, k := range u.Keys {
out.Keys = append(out.Keys, UserKey{User: k.User, AccessKey: k.AccessKey, SecretKey: k.SecretKey})
}
return out
}
// GetUser fetches an RGW user by uid, returning an error classified by
// IsNotFound (admin.ErrNoSuchUser) when it does not exist.
func (c *Client) GetUser(ctx context.Context, uid string) (*User, error) {
u, err := c.admin.GetUser(ctx, admin.User{ID: uid})
if err != nil {
return nil, err
}
return fromAdminUser(u), nil
}
// CreateUser creates an RGW user, asking radosgw to generate an S3 key pair. The
// returned User carries the generated keys.
func (c *Client) CreateUser(ctx context.Context, spec UserSpec) (*User, error) {
u, err := c.admin.CreateUser(ctx, admin.User{
ID: spec.UID,
DisplayName: firstNonEmpty(spec.DisplayName, spec.UID),
Email: spec.Email,
MaxBuckets: int32PtrToIntPtr(spec.MaxBuckets),
Suspended: boolPtrToIntPtr(spec.Suspended),
GenerateKey: boolPtr(true),
})
if err != nil {
return nil, err
}
c.forgetIdentity(spec.UID)
return fromAdminUser(u), nil
}
// UpdateUser reconciles the mutable attributes of an existing RGW user. It only
// sends attributes the spec sets: an empty DisplayName or nil Suspended is left
// as-is, so reconciling (or adopting) a user does not clobber those fields.
func (c *Client) UpdateUser(ctx context.Context, spec UserSpec) (*User, error) {
req := admin.User{
ID: spec.UID,
Email: spec.Email,
MaxBuckets: int32PtrToIntPtr(spec.MaxBuckets),
Suspended: boolPtrToIntPtr(spec.Suspended),
}
if spec.DisplayName != "" {
req.DisplayName = spec.DisplayName
}
u, err := c.admin.ModifyUser(ctx, req)
if err != nil {
return nil, err
}
return fromAdminUser(u), nil
}
// DeleteUser removes an RGW user. A NoSuchUser response is treated as success.
func (c *Client) DeleteUser(ctx context.Context, uid string) error {
err := c.admin.RemoveUser(ctx, admin.User{ID: uid})
c.forgetIdentity(uid)
if IsNotFound(err) {
return nil
}
return err
}
// SetUserQuota applies a quota to a user. quotaType is "user" or "bucket" (the
// latter sets the per-bucket default for buckets the user owns). A nil or
// negative limit means unlimited for that dimension.
func (c *Client) SetUserQuota(ctx context.Context, uid, quotaType string, enabled bool, maxSizeBytes, maxObjects *int64) error {
maxSize := valueOr(maxSizeBytes, -1)
maxObj := valueOr(maxObjects, -1)
return c.admin.SetUserQuota(ctx, admin.QuotaSpec{
UID: uid,
QuotaType: quotaType,
Enabled: &enabled,
MaxSize: &maxSize,
MaxObjects: &maxObj,
})
}
func firstNonEmpty(vals ...string) string {
for _, v := range vals {
if v != "" {
return v
}
}
return ""
}
func valueOr(v *int64, fallback int64) int64 {
if v == nil || *v < 0 {
return fallback
}
return *v
}
func derefInt(p *int) int {
if p == nil {
return 0
}
return *p
}
func boolPtr(b bool) *bool { return &b }
func int32PtrToIntPtr(p *int32) *int {
if p == nil {
return nil
}
v := int(*p)
return &v
}
// boolPtrToIntPtr renders an optional bool as RGW's 0/1 int, or nil to omit.
func boolPtrToIntPtr(b *bool) *int {
if b == nil {
return nil
}
v := 0
if *b {
v = 1
}
return &v
}
+333
View File
@@ -0,0 +1,333 @@
package controller
import (
"context"
"encoding/json"
"fmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/handler"
"sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"git.unkin.net/unkin/cephrgw-operator/api/v1alpha1"
"git.unkin.net/unkin/cephrgw-operator/internal/ceph"
)
// BucketReconciler provisions RGW buckets and owns the bucket's S3 policy. It
// aggregates every BucketAccess that targets the bucket into a single policy
// document, so the policy stays convergent no matter the order of events.
type BucketReconciler struct {
client.Client
Scheme *runtime.Scheme
Ceph *ceph.Client
Endpoint string
}
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=buckets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=buckets/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=buckets/finalizers,verbs=update
func (r *BucketReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
var b v1alpha1.Bucket
if err := r.Get(ctx, req.NamespacedName, &b); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
bucketName := orDefault(b.Spec.BucketName, b.Name)
if !b.DeletionTimestamp.IsZero() {
if controllerutil.ContainsFinalizer(&b, finalizer) {
if !b.Spec.RetainOnDelete {
if err := r.Ceph.DeleteBucket(ctx, bucketName, b.Spec.PurgeOnDelete); err != nil {
return r.fail(ctx, &b, "DeleteFailed", err)
}
}
controllerutil.RemoveFinalizer(&b, finalizer)
if err := r.Update(ctx, &b); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
if controllerutil.AddFinalizer(&b, finalizer) {
if err := r.Update(ctx, &b); err != nil {
return ctrl.Result{}, err
}
}
// Resolve the owning user.
var owner v1alpha1.ObjectStoreUser
if err := r.Get(ctx, types.NamespacedName{Namespace: b.Namespace, Name: b.Spec.OwnerRef}, &owner); err != nil {
if apierrors.IsNotFound(err) {
return r.pending(ctx, &b, "OwnerMissing", fmt.Sprintf("waiting for ObjectStoreUser %q", b.Spec.OwnerRef))
}
return r.fail(ctx, &b, "OwnerLookupFailed", err)
}
if owner.Status.UID == "" || owner.Status.Phase != "Ready" {
return r.pending(ctx, &b, "OwnerNotReady", fmt.Sprintf("ObjectStoreUser %q not ready", b.Spec.OwnerRef))
}
ownerUID := owner.Status.UID
// Ensure the bucket exists. Record adoption once: whether the RGW bucket
// already existed the first time we reconciled this resource. status.BucketID
// is only set on a successful reconcile, so a Pending wait on the owner (or a
// transient failure) does not pollute the signal.
firstObserve := b.Status.BucketID == ""
info, err := r.Ceph.GetBucket(ctx, bucketName)
if ceph.IsNotFound(err) {
createSpec := ceph.CreateBucketSpec{
Bucket: bucketName,
OwnerUID: ownerUID,
Zonegroup: b.Spec.Zonegroup,
PlacementTarget: b.Spec.PlacementTarget,
}
if ol := b.Spec.ObjectLock; ol != nil && ol.Enabled {
createSpec.LockEnabled = true
createSpec.LockMode = string(ol.Mode)
createSpec.LockDays = ol.Days
createSpec.LockYears = ol.Years
}
info, err = r.Ceph.CreateBucket(ctx, createSpec)
if err != nil {
return r.fail(ctx, &b, "CreateFailed", err)
}
logger.Info("created bucket", "bucket", bucketName, "owner", ownerUID)
if firstObserve {
b.Status.Adopted = false
}
} else if err != nil {
return r.fail(ctx, &b, "LookupFailed", err)
} else if firstObserve {
b.Status.Adopted = true
logger.Info("adopted existing bucket", "bucket", bucketName, "owner", ownerUID)
}
bucketID := info.InstanceID()
// Placement is fixed at creation: RGW cannot move an existing bucket between
// placement targets. If the live bucket sits on a different target than the
// spec asks for (a changed spec, or an adopted bucket that predates the
// request), surface a clear error instead of ever deleting/recreating it. An
// empty PlacementTarget imposes no constraint.
if pc := placementConflict(b.Spec.PlacementTarget, info.PlacementRule); pc {
b.Status.PlacementTarget = info.PlacementRule
return r.fail(ctx, &b, "PlacementImmutable", fmt.Errorf(
"bucket %q is on placement target %q but spec requests %q; RGW cannot move a bucket between placement targets",
bucketName, info.PlacementRule, b.Spec.PlacementTarget))
}
// Versioning (forced on when object lock is enabled).
if b.Spec.Versioning || (b.Spec.ObjectLock != nil && b.Spec.ObjectLock.Enabled) {
if err := r.Ceph.SetBucketVersioning(ctx, bucketName, bucketID, ownerUID, true); err != nil {
return r.fail(ctx, &b, "VersioningFailed", err)
}
}
// Tags.
if len(b.Spec.Tags) > 0 {
tj, err := ceph.BuildTagJSON(b.Spec.Tags)
if err != nil {
return r.fail(ctx, &b, "TagsFailed", err)
}
if tj != "" {
if err := r.Ceph.SetBucketTags(ctx, bucketName, bucketID, ownerUID, tj); err != nil {
return r.fail(ctx, &b, "TagsFailed", err)
}
}
}
// Bucket default quota (applied to the owner).
if q := b.Spec.Quota; q != nil {
if err := r.Ceph.SetUserQuota(ctx, ownerUID, "bucket", q.Enabled, q.MaxSizeBytes, q.MaxObjects); err != nil {
return r.fail(ctx, &b, "QuotaFailed", err)
}
}
// Render and apply the aggregate S3 policy from all BucketAccess grants,
// unless the bucket opts out of policy management. The merge preserves any
// statements the operator does not own, so an adopted bucket keeps its
// existing policy.
principals := 0
if managePolicy(&b) {
grants, p, err := r.collectGrants(ctx, b.Namespace, b.Name)
if err != nil {
return r.fail(ctx, &b, "GrantsFailed", err)
}
existing, err := r.Ceph.GetBucketPolicy(ctx, bucketName, ownerUID)
if err != nil {
return r.fail(ctx, &b, "PolicyReadFailed", err)
}
policy, err := ceph.MergeBucketPolicy(existing, bucketName, grants)
if err != nil {
return r.fail(ctx, &b, "PolicyBuildFailed", err)
}
if err := r.Ceph.SetBucketPolicy(ctx, bucketName, bucketID, ownerUID, policy); err != nil {
return r.fail(ctx, &b, "PolicyFailed", err)
}
principals = p
}
b.Status.Phase = "Ready"
b.Status.BucketName = bucketName
b.Status.BucketID = bucketID
b.Status.Owner = ownerUID
b.Status.PlacementTarget = info.PlacementRule
b.Status.PolicyPrincipals = int32(principals)
b.Status.ObservedGeneration = b.Generation
setReady(&b.Status.Conditions, b.Generation, true, "Provisioned", "bucket provisioned")
if err := r.Status().Update(ctx, &b); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: requeueSteady}, nil
}
// collectGrants returns the deduplicated set of grants for a bucket, drawn from
// every ready, non-deleting BucketAccess that references it, plus the count of
// distinct principals.
func (r *BucketReconciler) collectGrants(ctx context.Context, namespace, bucketRefName string) ([]ceph.Grant, int, error) {
var list v1alpha1.BucketAccessList
if err := r.List(ctx, &list, client.InNamespace(namespace)); err != nil {
return nil, 0, err
}
seen := map[string]struct{}{}
principals := map[string]struct{}{}
var grants []ceph.Grant
for i := range list.Items {
ba := &list.Items[i]
if ba.Spec.BucketRef != bucketRefName {
continue
}
if !ba.DeletionTimestamp.IsZero() {
continue
}
if ba.Status.UID == "" {
continue
}
g := grantFromAccess(ba.Status.UID, ba)
key := grantKey(g)
if _, dup := seen[key]; dup {
continue
}
seen[key] = struct{}{}
principals[ba.Status.UID] = struct{}{}
grants = append(grants, g)
}
return grants, len(principals), nil
}
// grantFromAccess translates a BucketAccess spec into the ceph grant model,
// carrying the fine-grained scoping (paths, actions, conditions, raw statements).
func grantFromAccess(uid string, ba *v1alpha1.BucketAccess) ceph.Grant {
g := ceph.Grant{
UID: uid,
Level: string(ba.Spec.Level),
Paths: ba.Spec.Paths,
Actions: ba.Spec.Actions,
}
if c := ba.Spec.Conditions; c != nil {
g.Conditions = &ceph.GrantConditions{
SourceIPs: c.SourceIPs,
SecureTransportOnly: c.SecureTransportOnly,
}
}
for _, s := range ba.Spec.RawStatements {
g.Raw = append(g.Raw, ceph.RawStatement{
Sid: s.Sid,
Effect: s.Effect,
Actions: s.Actions,
Resources: s.Resources,
Condition: s.Conditions,
})
}
return g
}
// grantKey is a stable fingerprint of a grant used to collapse duplicate
// BucketAccess objects that would render identical policy statements.
func grantKey(g ceph.Grant) string {
b, _ := json.Marshal(g)
return string(b)
}
// managePolicy reports whether the operator should reconcile this bucket's S3
// policy. A nil ManagePolicy (the CRD default) is treated as true.
func managePolicy(b *v1alpha1.Bucket) bool {
return b.Spec.ManagePolicy == nil || *b.Spec.ManagePolicy
}
// placementConflict reports whether a bucket's live placement target violates
// the spec. An empty spec placement imposes no constraint (the bucket may sit on
// whatever default it was created with). Otherwise the live placement must match
// exactly, since RGW cannot move a bucket between placement targets.
func placementConflict(specPlacement, livePlacement string) bool {
if specPlacement == "" {
return false
}
return specPlacement != livePlacement
}
func (r *BucketReconciler) pending(ctx context.Context, b *v1alpha1.Bucket, reason, msg string) (ctrl.Result, error) {
b.Status.Phase = "Pending"
b.Status.ObservedGeneration = b.Generation
setReady(&b.Status.Conditions, b.Generation, false, reason, msg)
if err := r.Status().Update(ctx, b); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: requeueShort}, nil
}
func (r *BucketReconciler) fail(ctx context.Context, b *v1alpha1.Bucket, reason string, cause error) (ctrl.Result, error) {
b.Status.Phase = "Error"
b.Status.ObservedGeneration = b.Generation
setReady(&b.Status.Conditions, b.Generation, false, reason, cause.Error())
if err := r.Status().Update(ctx, b); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, cause
}
func (r *BucketReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&v1alpha1.Bucket{}).
Watches(&v1alpha1.BucketAccess{}, handler.EnqueueRequestsFromMapFunc(r.bucketForAccess)).
Watches(&v1alpha1.ObjectStoreUser{}, handler.EnqueueRequestsFromMapFunc(r.bucketsForOwner)).
Complete(r)
}
// bucketForAccess maps a BucketAccess change to its referenced Bucket.
func (r *BucketReconciler) bucketForAccess(_ context.Context, obj client.Object) []reconcile.Request {
ba, ok := obj.(*v1alpha1.BucketAccess)
if !ok || ba.Spec.BucketRef == "" {
return nil
}
return []reconcile.Request{{NamespacedName: types.NamespacedName{Namespace: ba.Namespace, Name: ba.Spec.BucketRef}}}
}
// bucketsForOwner maps an ObjectStoreUser change to every Bucket it owns.
func (r *BucketReconciler) bucketsForOwner(ctx context.Context, obj client.Object) []reconcile.Request {
osu, ok := obj.(*v1alpha1.ObjectStoreUser)
if !ok {
return nil
}
var list v1alpha1.BucketList
if err := r.List(ctx, &list, client.InNamespace(osu.Namespace)); err != nil {
return nil
}
var reqs []reconcile.Request
for i := range list.Items {
if list.Items[i].Spec.OwnerRef == osu.Name {
reqs = append(reqs, reconcile.Request{NamespacedName: types.NamespacedName{
Namespace: list.Items[i].Namespace, Name: list.Items[i].Name,
}})
}
}
return reqs
}
@@ -0,0 +1,27 @@
package controller
import "testing"
func TestPlacementConflict(t *testing.T) {
cases := []struct {
name string
specPlacement string
livePlacement string
want bool
}{
{"unset spec never conflicts", "", "default-placement", false},
{"unset spec unset live", "", "", false},
{"matching ec", "ec", "ec", false},
{"matching default", "default-placement", "default-placement", false},
{"ec requested but default live", "ec", "default-placement", true},
{"default requested but ec live", "default-placement", "ec", true},
{"spec set live empty", "ec", "", true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := placementConflict(tc.specPlacement, tc.livePlacement); got != tc.want {
t.Errorf("placementConflict(%q,%q)=%v want %v", tc.specPlacement, tc.livePlacement, got, tc.want)
}
})
}
}
@@ -0,0 +1,199 @@
package controller
import (
"context"
"fmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/handler"
"sigs.k8s.io/controller-runtime/pkg/log"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"git.unkin.net/unkin/cephrgw-operator/api/v1alpha1"
"git.unkin.net/unkin/cephrgw-operator/internal/ceph"
)
// BucketAccessReconciler ensures the principal for a grant exists (creating a
// dedicated RGW user when none is referenced) and delivers its keys. The bucket
// policy itself is owned and rendered by the Bucket controller, which watches
// BucketAccess objects.
type BucketAccessReconciler struct {
client.Client
Scheme *runtime.Scheme
Ceph *ceph.Client
Endpoint string
}
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=bucketaccesses,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=bucketaccesses/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=bucketaccesses/finalizers,verbs=update
func (r *BucketAccessReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
var ba v1alpha1.BucketAccess
if err := r.Get(ctx, req.NamespacedName, &ba); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
managed := ba.Spec.UserRef == ""
uid, err := r.resolveUID(ctx, &ba)
if err != nil {
return r.fail(ctx, &ba, "ResolveFailed", err)
}
if !ba.DeletionTimestamp.IsZero() {
if controllerutil.ContainsFinalizer(&ba, finalizer) {
// Only delete a user the operator created for this grant, and only
// when the grant does not ask to retain it.
if managed && uid != "" && !ba.Spec.RetainOnDelete {
if err := r.Ceph.DeleteUser(ctx, uid); err != nil {
return r.fail(ctx, &ba, "DeleteFailed", err)
}
}
controllerutil.RemoveFinalizer(&ba, finalizer)
if err := r.Update(ctx, &ba); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
if controllerutil.AddFinalizer(&ba, finalizer) {
if err := r.Update(ctx, &ba); err != nil {
return ctrl.Result{}, err
}
}
// Resolve the referenced Bucket so we can label the credential Secret and
// gate the grant on the bucket existing.
var bucket v1alpha1.Bucket
if err := r.Get(ctx, types.NamespacedName{Namespace: ba.Namespace, Name: ba.Spec.BucketRef}, &bucket); err != nil {
if apierrors.IsNotFound(err) {
return r.pending(ctx, &ba, "BucketMissing", fmt.Sprintf("waiting for Bucket %q", ba.Spec.BucketRef))
}
return r.fail(ctx, &ba, "BucketLookupFailed", err)
}
bucketName := orDefault(bucket.Status.BucketName, orDefault(bucket.Spec.BucketName, bucket.Name))
secretName := ba.Status.SecretName
if managed {
if uid == "" {
uid = fmt.Sprintf("%s-%s", ba.Spec.BucketRef, ba.Name)
}
secretName = orDefault(ba.Spec.SecretName, ba.Name+"-rgw")
user, err := r.ensureUser(ctx, uid)
if err != nil {
return r.fail(ctx, &ba, "UserFailed", err)
}
key, ok := user.S3Key()
if !ok {
return r.fail(ctx, &ba, "NoKeys", fmt.Errorf("user %s has no S3 keys", uid))
}
if err := upsertSecret(ctx, r.Client, r.Scheme, &ba, secretName, ba.Namespace,
credentialSecretData(key, uid, r.Endpoint, bucketName)); err != nil {
return r.fail(ctx, &ba, "SecretFailed", err)
}
}
ba.Status.Phase = "Ready"
ba.Status.UID = uid
ba.Status.SecretName = secretName
ba.Status.Bound = true
ba.Status.ObservedGeneration = ba.Generation
setReady(&ba.Status.Conditions, ba.Generation, true, "Granted",
fmt.Sprintf("%s access for %s applied to bucket %s", ba.Spec.Level, uid, bucketName))
if err := r.Status().Update(ctx, &ba); err != nil {
return ctrl.Result{}, err
}
logger.Info("bucket access reconciled", "bucket", bucketName, "uid", uid, "level", ba.Spec.Level)
return ctrl.Result{RequeueAfter: requeueSteady}, nil
}
// resolveUID returns the RGW uid this grant targets: the referenced
// ObjectStoreUser's provisioned uid, or the managed uid derived from the spec.
func (r *BucketAccessReconciler) resolveUID(ctx context.Context, ba *v1alpha1.BucketAccess) (string, error) {
if ba.Spec.UserRef == "" {
if ba.Spec.UID != "" {
return ba.Spec.UID, nil
}
// Derived lazily in Reconcile once we know it is not a deletion no-op.
return "", nil
}
var osu v1alpha1.ObjectStoreUser
if err := r.Get(ctx, types.NamespacedName{Namespace: ba.Namespace, Name: ba.Spec.UserRef}, &osu); err != nil {
return "", err
}
if osu.Status.UID == "" {
return "", fmt.Errorf("ObjectStoreUser %q not ready", ba.Spec.UserRef)
}
return osu.Status.UID, nil
}
func (r *BucketAccessReconciler) ensureUser(ctx context.Context, uid string) (*ceph.User, error) {
if _, err := r.Ceph.GetUser(ctx, uid); ceph.IsNotFound(err) {
if _, err := r.Ceph.CreateUser(ctx, ceph.UserSpec{UID: uid, DisplayName: uid}); err != nil {
return nil, err
}
} else if err != nil {
return nil, err
}
return r.Ceph.GetUser(ctx, uid)
}
func (r *BucketAccessReconciler) pending(ctx context.Context, ba *v1alpha1.BucketAccess, reason, msg string) (ctrl.Result, error) {
ba.Status.Phase = "Pending"
ba.Status.Bound = false
ba.Status.ObservedGeneration = ba.Generation
setReady(&ba.Status.Conditions, ba.Generation, false, reason, msg)
if err := r.Status().Update(ctx, ba); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: requeueShort}, nil
}
func (r *BucketAccessReconciler) fail(ctx context.Context, ba *v1alpha1.BucketAccess, reason string, cause error) (ctrl.Result, error) {
ba.Status.Phase = "Error"
ba.Status.Bound = false
ba.Status.ObservedGeneration = ba.Generation
setReady(&ba.Status.Conditions, ba.Generation, false, reason, cause.Error())
if err := r.Status().Update(ctx, ba); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, cause
}
func (r *BucketAccessReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&v1alpha1.BucketAccess{}).
Watches(&v1alpha1.ObjectStoreUser{}, handler.EnqueueRequestsFromMapFunc(r.accessForUser)).
Complete(r)
}
// accessForUser maps an ObjectStoreUser change to every BucketAccess that
// references it, so a grant binds as soon as its user becomes ready.
func (r *BucketAccessReconciler) accessForUser(ctx context.Context, obj client.Object) []reconcile.Request {
osu, ok := obj.(*v1alpha1.ObjectStoreUser)
if !ok {
return nil
}
var list v1alpha1.BucketAccessList
if err := r.List(ctx, &list, client.InNamespace(osu.Namespace)); err != nil {
return nil
}
var reqs []reconcile.Request
for i := range list.Items {
if list.Items[i].Spec.UserRef == osu.Name {
reqs = append(reqs, reconcile.Request{NamespacedName: types.NamespacedName{
Namespace: list.Items[i].Namespace, Name: list.Items[i].Name,
}})
}
}
return reqs
}
+89
View File
@@ -0,0 +1,89 @@
package controller
import (
"context"
apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
apiextensionsclient "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
ctrl "sigs.k8s.io/controller-runtime"
)
// The CRD version check reads CustomResourceDefinitions to compare the
// installed schema against what this operator expects.
//
// +kubebuilder:rbac:groups=apiextensions.k8s.io,resources=customresourcedefinitions,verbs=get;list
// crdSentinel names a CRD and a spec property that only exists in the schema
// version shipped alongside this operator build. If the installed CRD lacks the
// sentinel, its schema predates this operator and strict decoding of new spec
// fields will silently fail. Keep this list in one place so new fields are easy
// to register as sentinels.
type crdSentinel struct {
// crd is the metadata.name of the CustomResourceDefinition.
crd string
// specProperty is a key expected under
// .spec.versions[].schema.openAPIV3Schema.properties.spec.properties.
specProperty string
}
// crdSentinels is the authoritative list checked at startup. Extend it whenever
// a new spec field is added that older CRDs would reject.
var crdSentinels = []crdSentinel{
{crd: "buckets.ceph.unkin.net", specProperty: "managePolicy"},
{crd: "objectstoreusers.ceph.unkin.net", specProperty: "retainOnDelete"},
{crd: "bucketaccesses.ceph.unkin.net", specProperty: "rawStatements"},
}
// CheckCRDVersions verifies that every CRD this operator owns is installed and
// carries the schema fields this build expects. It is advisory only: it logs a
// distinct WARNING per problem and never returns an error or exits, so a stale
// or missing CRD cannot block startup.
func CheckCRDVersions(ctx context.Context, cfg *rest.Config) {
log := ctrl.Log.WithName("crd-version-check")
client, err := apiextensionsclient.NewForConfig(cfg)
if err != nil {
log.Error(err, "unable to build apiextensions client; skipping CRD version check")
return
}
for _, s := range crdSentinels {
crd, err := client.ApiextensionsV1().CustomResourceDefinitions().Get(ctx, s.crd, metav1.GetOptions{})
if apierrors.IsNotFound(err) {
log.Info("WARNING: CRD is not installed — apply the CRDs matching this operator version",
"crd", s.crd)
continue
}
if err != nil {
log.Error(err, "unable to read CRD; cannot verify it matches this operator version",
"crd", s.crd)
continue
}
if !crdHasSpecProperty(crd, s.specProperty) {
log.Info("WARNING: CRD is out of date — apply the CRDs matching this operator version",
"crd", s.crd, "missingField", "spec."+s.specProperty)
}
}
}
// crdHasSpecProperty reports whether any served/stored version of the CRD
// declares the given property under spec.
func crdHasSpecProperty(crd *apiextensionsv1.CustomResourceDefinition, property string) bool {
for _, v := range crd.Spec.Versions {
schema := v.Schema
if schema == nil || schema.OpenAPIV3Schema == nil {
continue
}
specSchema, ok := schema.OpenAPIV3Schema.Properties["spec"]
if !ok {
continue
}
if _, ok := specSchema.Properties[property]; ok {
return true
}
}
return false
}
+101
View File
@@ -0,0 +1,101 @@
package controller
import (
"context"
"net/url"
"time"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/meta"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"git.unkin.net/unkin/cephrgw-operator/internal/ceph"
)
// finalizer guards external RGW state (users, buckets, policy statements) so it
// is cleaned up before the Kubernetes object disappears.
const finalizer = "ceph.unkin.net/finalizer"
// requeueSteady is the resync interval for healthy objects; it lets the
// operator heal drift made directly against RGW.
const requeueSteady = 10 * time.Minute
// requeueShort backs off on transient "waiting for a dependency" states.
const requeueShort = 30 * time.Second
// setReady sets the standard Ready condition on a status conditions slice.
func setReady(conds *[]metav1.Condition, gen int64, ok bool, reason, msg string) {
status := metav1.ConditionFalse
if ok {
status = metav1.ConditionTrue
}
meta.SetStatusCondition(conds, metav1.Condition{
Type: "Ready",
Status: status,
ObservedGeneration: gen,
Reason: reason,
Message: truncate(msg, 32000),
})
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
func orDefault(v, def string) string {
if v != "" {
return v
}
return def
}
// upsertSecret creates or updates an owner-referenced Opaque Secret with data.
func upsertSecret(ctx context.Context, c client.Client, scheme *runtime.Scheme, owner client.Object, name, namespace string, data map[string][]byte) error {
sec := &corev1.Secret{}
sec.Name = name
sec.Namespace = namespace
_, err := controllerutil.CreateOrUpdate(ctx, c, sec, func() error {
sec.Type = corev1.SecretTypeOpaque
if sec.Data == nil {
sec.Data = map[string][]byte{}
}
for k, v := range data {
sec.Data[k] = v
}
return controllerutil.SetControllerReference(owner, sec, scheme)
})
return err
}
// credentialSecretData assembles the conventional S3/AWS credential keys.
func credentialSecretData(key ceph.UserKey, uid, endpoint, bucket string) map[string][]byte {
data := map[string][]byte{
"AWS_ACCESS_KEY_ID": []byte(key.AccessKey),
"AWS_SECRET_ACCESS_KEY": []byte(key.SecretKey),
"RGW_UID": []byte(uid),
}
if endpoint != "" {
data["S3_ENDPOINT"] = []byte(endpoint)
if host := hostOf(endpoint); host != "" {
data["BUCKET_HOST"] = []byte(host)
}
}
if bucket != "" {
data["BUCKET_NAME"] = []byte(bucket)
}
return data
}
func hostOf(endpoint string) string {
u, err := url.Parse(endpoint)
if err != nil || u.Host == "" {
return endpoint
}
return u.Host
}
@@ -0,0 +1,142 @@
package controller
import (
"context"
"fmt"
"k8s.io/apimachinery/pkg/runtime"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
"git.unkin.net/unkin/cephrgw-operator/api/v1alpha1"
"git.unkin.net/unkin/cephrgw-operator/internal/ceph"
)
// ObjectStoreUserReconciler provisions RGW users and delivers their keys.
type ObjectStoreUserReconciler struct {
client.Client
Scheme *runtime.Scheme
Ceph *ceph.Client
Endpoint string
}
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=objectstoreusers,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=objectstoreusers/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=ceph.unkin.net,resources=objectstoreusers/finalizers,verbs=update
// +kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch;create;update;patch;delete
func (r *ObjectStoreUserReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
logger := log.FromContext(ctx)
var osu v1alpha1.ObjectStoreUser
if err := r.Get(ctx, req.NamespacedName, &osu); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
uid := orDefault(osu.Spec.UID, osu.Name)
secretName := orDefault(osu.Spec.SecretName, osu.Name+"-rgw")
if !osu.DeletionTimestamp.IsZero() {
if controllerutil.ContainsFinalizer(&osu, finalizer) {
if osu.Spec.RetainOnDelete {
logger.Info("retaining RGW user on delete", "uid", uid)
} else if err := r.Ceph.DeleteUser(ctx, uid); err != nil {
return r.fail(ctx, &osu, "DeleteFailed", err)
}
controllerutil.RemoveFinalizer(&osu, finalizer)
if err := r.Update(ctx, &osu); err != nil {
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
if controllerutil.AddFinalizer(&osu, finalizer) {
if err := r.Update(ctx, &osu); err != nil {
return ctrl.Result{}, err
}
}
spec := ceph.UserSpec{
UID: uid,
DisplayName: osu.Spec.DisplayName,
Email: osu.Spec.Email,
MaxBuckets: osu.Spec.MaxBuckets,
Suspended: osu.Spec.Suspended,
}
// Record adoption once: whether the RGW user already existed the first time
// we reconciled this resource (taken over rather than created). status.UID is
// only set on a successful reconcile, so it is a clean "never provisioned"
// signal that transient failures do not pollute.
firstObserve := osu.Status.UID == ""
_, getErr := r.Ceph.GetUser(ctx, uid)
switch {
case ceph.IsNotFound(getErr):
if _, err := r.Ceph.CreateUser(ctx, spec); err != nil {
return r.fail(ctx, &osu, "CreateFailed", err)
}
logger.Info("created RGW user", "uid", uid)
if firstObserve {
osu.Status.Adopted = false
}
case getErr != nil:
return r.fail(ctx, &osu, "LookupFailed", getErr)
default:
if _, err := r.Ceph.UpdateUser(ctx, spec); err != nil {
return r.fail(ctx, &osu, "UpdateFailed", err)
}
if firstObserve {
osu.Status.Adopted = true
logger.Info("adopted existing RGW user", "uid", uid)
}
}
if q := osu.Spec.Quota; q != nil {
if err := r.Ceph.SetUserQuota(ctx, uid, "user", q.Enabled, q.MaxSizeBytes, q.MaxObjects); err != nil {
return r.fail(ctx, &osu, "QuotaFailed", err)
}
}
user, err := r.Ceph.GetUser(ctx, uid)
if err != nil {
return r.fail(ctx, &osu, "LookupFailed", err)
}
key, ok := user.S3Key()
if !ok {
return r.fail(ctx, &osu, "NoKeys", fmt.Errorf("user %s has no S3 keys", uid))
}
if err := upsertSecret(ctx, r.Client, r.Scheme, &osu, secretName, osu.Namespace,
credentialSecretData(key, uid, r.Endpoint, "")); err != nil {
return r.fail(ctx, &osu, "SecretFailed", err)
}
osu.Status.Phase = "Ready"
osu.Status.UID = uid
osu.Status.SecretName = secretName
osu.Status.ObservedGeneration = osu.Generation
setReady(&osu.Status.Conditions, osu.Generation, true, "Provisioned", "RGW user provisioned")
if err := r.Status().Update(ctx, &osu); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{RequeueAfter: requeueSteady}, nil
}
func (r *ObjectStoreUserReconciler) fail(ctx context.Context, osu *v1alpha1.ObjectStoreUser, reason string, cause error) (ctrl.Result, error) {
osu.Status.Phase = "Error"
osu.Status.ObservedGeneration = osu.Generation
setReady(&osu.Status.Conditions, osu.Generation, false, reason, cause.Error())
if err := r.Status().Update(ctx, osu); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, cause
}
func (r *ObjectStoreUserReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&v1alpha1.ObjectStoreUser{}).
Complete(r)
}
+36
View File
@@ -0,0 +1,36 @@
package controller
import (
ctrl "sigs.k8s.io/controller-runtime"
"git.unkin.net/unkin/cephrgw-operator/internal/ceph"
)
// SetupAll registers every controller with the manager.
func SetupAll(mgr ctrl.Manager, cephClient *ceph.Client, endpoint string) error {
if err := (&ObjectStoreUserReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Ceph: cephClient,
Endpoint: endpoint,
}).SetupWithManager(mgr); err != nil {
return err
}
if err := (&BucketReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Ceph: cephClient,
Endpoint: endpoint,
}).SetupWithManager(mgr); err != nil {
return err
}
if err := (&BucketAccessReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
Ceph: cephClient,
Endpoint: endpoint,
}).SetupWithManager(mgr); err != nil {
return err
}
return nil
}