15 Commits

Author SHA1 Message Date
benvin a89e38be64 Merge pull request 'ci: add buildkit_config CA trust for artifactapi push' (#10) from benvin/buildx-ca-config into main
Reviewed-on: #10
2026-08-15 18:47:28 +10:00
unkin-agent ede6604ab1 ci: add buildkit_config CA trust for artifactapi push
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-08-15 18:30:44 +10:00
benvin f60e69bb79 Merge pull request 'ci: use CA-baked plugin-docker-buildx image for artifactapi push' (#9) from benvin/buildx-ca-plugin-image into main
Reviewed-on: #9
2026-08-15 18:20:25 +10:00
unkin-agent 5f0b9f53ad ci: use CA-baked plugin-docker-buildx image for artifactapi push
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
2026-08-15 18:03:55 +10:00
benvin 7f1f57eb8d Merge pull request 'ci: push images to artifactapi registry instead of gitea' (#8) from benvin/push-artifactapi into main
Reviewed-on: #8
2026-07-30 20:55:17 +10:00
unkinben d3b022d646 ci: push images to artifactapi registry instead of gitea
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
Hard switch of the docker push target from the Gitea registry to the
artifactapi local docker registry (docker-internal); the Gitea VM and its
registry are being retired. Drops the droneci/DRONECI_PASSWORD creds since
artifactapi accepts unauthenticated in-cluster pushes. Also updates the README push note.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-30 00:34:59 +10:00
benvin 343d60cfcf Merge pull request 'Add immutable placement-target selection to Bucket' (#7) from benvin/placement-targets into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #7
2026-07-29 00:37:48 +10:00
unkinben c1b3ba1c34 Add immutable placement-target selection to Bucket
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
Buckets could not choose which RGW placement target (and thus durability
profile) backs them, so all data landed on the cluster default. The estate's
radosgw exposes two targets - default-placement (3x replicated) and ec (4+1
erasure-coded) - and archival workloads want ec.

- Validate spec.placementTarget: a DNS-ish pattern, 63-char cap, and a CEL
  self==oldSelf immutability rule (RGW fixes placement at bucket creation and
  cannot move a bucket between targets); make spec.zonegroup immutable too.
- Thread the target into the S3 CreateBucket LocationConstraint via the existing
  helper; an empty zonegroup yields ":<target>", selecting the local zonegroup
  so callers need not name the zonegroup api-name.
- Read the live placement_rule and zonegroup back from the Admin Ops bucket
  stats and surface them: status.placementTarget plus a Placement print column.
- Guard the controller: if a live bucket's placement differs from spec, set an
  Error phase with a PlacementImmutable reason instead of deleting/recreating.
- Cover locationConstraint construction, placement readback (httptest), and the
  placementConflict guard with tests; document targets and immutability in the
  README and add config/samples/06-bucket-ec.yaml.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-29 00:16:31 +10:00
benvin 52e183e6f3 Merge pull request 'Warn at startup when installed CRDs are stale or missing' (#6) from benvin/crd-version-warning into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #6
2026-07-25 22:59:42 +10:00
unkinben e7760b79f4 Warn at startup when installed CRDs are stale or missing
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The operator strict-decodes CR specs, so when the in-cluster CRDs lag the
running operator (e.g. CRDs pinned to v0.1.0 while the operator ran v0.3.0),
new spec fields fail to decode with no operator-side signal. This adds an
advisory startup check so the mismatch is visible in the operator logs.

- Add CheckCRDVersions: GET each owned CRD and verify it carries a
  version-sentinel spec field, logging a distinct WARNING per problem
  (missing CRD vs. present-but-stale schema); advisory only, never exits.
- Keep the sentinel list (buckets/managePolicy, objectstoreusers/
  retainOnDelete, bucketaccesses/rawStatements) in one place.
- Wire the check into operator startup after the manager config is available.
- Add apiextensions customresourcedefinitions get;list RBAC marker and
  regenerate config/rbac/role.yaml.
- Promote k8s.io/apiextensions-apiserver to a direct dependency.

Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
2026-07-25 22:42:03 +10:00
benvin 9bbaa2b8ba Merge pull request 'Support adopting existing radosgw buckets and users' (#5) from benvin/safe-adoption into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #5
2026-07-25 10:17:01 +10:00
unkinben 54d3e38223 Support adopting existing radosgw buckets and users
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The operator previously assumed it created every user and bucket it managed:
reconciling an existing resource could overwrite its user attributes or wipe its
bucket policy, and deleting a CRD always deleted the underlying RGW object (only
Bucket had retainOnDelete). That made taking over pre-existing radosgw state
unsafe. Make adoption first-class.

- add retainOnDelete to ObjectStoreUser and BucketAccess (dedicated users), so
  deleting the CRD orphans the RGW user instead of deleting it (symmetric with
  Bucket)
- merge bucket policy instead of replacing it: the operator marks its own
  statements with a cephrgwop* Sid and preserves any statement it does not own,
  so adopting a bucket with a hand-written policy keeps it; add Bucket
  managePolicy (default true) to opt out of policy management entirely
- only reconcile user attributes the spec sets: DisplayName when non-empty and
  Suspended is now an optional *bool, so adopting a user does not reset them
- record adoption: ObjectStoreUser/Bucket status.adopted (+ printcolumn) is true
  when the RGW object already existed on first reconcile
- add GetBucketPolicy + MergeBucketPolicy; keyed adoption detection off the
  status identity field so a Pending owner wait does not mislabel it
- regenerate CRDs/deepcopy; add docs/adoption.md and
  config/samples/05-adoption.yaml; cover the merge in policy_test.go

Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
2026-07-25 00:15:10 +10:00
benvin 619aa6751d Merge pull request 'Add fine-grained bucket access: paths, actions, conditions, raw' (#4) from benvin/bucketaccess-fine-grained-policy into main
ci/woodpecker/tag/docker Pipeline was successful
Reviewed-on: #4
2026-07-24 23:02:51 +10:00
unkinben 4b0430f0df Add fine-grained bucket access: paths, actions, conditions, raw
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
The BucketAccess model only offered three coarse levels (read-only/read-write/
full) applied to the whole bucket. Real grants often need to be scoped to a key
prefix, limited to a source network or TLS, restricted to specific actions, or
expressed as an arbitrary S3 statement. RGW (Reef 18.2+/Squid) honours the S3
bucket-policy features to do all of this; expose them on BucketAccess while
keeping the level as the ergonomic default.

- add BucketAccess spec fields: paths (key-prefix scoping), actions (action
  override), conditions (sourceIPs + secureTransportOnly), rawStatements
  (arbitrary S3 statements with the principal injected)
- extend ceph.Grant + BuildBucketPolicy to render prefixed object resources,
  custom-action statements, S3 condition blocks, and raw statements, keeping
  output deterministic (sorted, stable sids)
- translate the new spec fields into grants in the Bucket controller and
  fingerprint grants so distinct fine-grained BucketAccess objects no longer
  collapse on UID+level alone
- regenerate deepcopy + CRDs; add config/samples/04-access-fine-grained.yaml
- cover paths, action override, conditions, raw statements and determinism in
  policy_test.go; document the fields in the README

Claude-Session: https://claude.ai/code/session_016CEncETbf8cvy1PhsHfFHM
2026-07-24 22:48:20 +10:00
benvin 253105f914 Merge pull request 'Talk to radosgw directly via go-ceph + aws-sdk-go-v2' (#3) from benvin/go-ceph-native-client into main
Reviewed-on: #3
2026-07-24 22:43:18 +10:00
28 changed files with 1770 additions and 112 deletions
+2 -2
View File
@@ -3,8 +3,8 @@ when:
steps:
- name: docker-build-operator
image: woodpeckerci/plugin-docker-buildx
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx:latest
settings:
repo: git.unkin.net/unkin/cephrgw-operator
repo: artifactapi.k8s.syd1.au.unkin.net/docker-internal/cephrgw-operator
dockerfile: Dockerfile.operator
dry_run: true
+6 -6
View File
@@ -4,14 +4,14 @@ when:
steps:
- name: docker-operator
image: woodpeckerci/plugin-docker-buildx
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx:latest
settings:
registry: git.unkin.net
repo: git.unkin.net/unkin/cephrgw-operator
registry: artifactapi.k8s.syd1.au.unkin.net
repo: artifactapi.k8s.syd1.au.unkin.net/docker-internal/cephrgw-operator
dockerfile: Dockerfile.operator
username: droneci
password:
from_secret: DRONECI_PASSWORD
buildkit_config: |
[registry."artifactapi.k8s.syd1.au.unkin.net"]
ca = ["/etc/docker/certs.d/artifactapi.k8s.syd1.au.unkin.net/ca.crt"]
tags:
- ${CI_COMMIT_TAG}
- latest
+70 -1
View File
@@ -40,6 +40,74 @@ user for that grant and writes its keys into `spec.secretName` (default
`<name>-rgw`). If `userRef` names an existing `ObjectStoreUser`, that user's own
credential Secret is reused and only the policy is extended.
#### Fine-grained grants
The level is the ergonomic default; four optional fields on `BucketAccess`
refine it (see `config/samples/04-access-fine-grained.yaml`):
- `spec.paths` — scope object access to key prefixes; each becomes the resource
`<bucket>/<prefix>*`. The bucket-level `ListBucket` still spans the whole
bucket.
- `spec.actions` — grant exactly these S3 actions instead of the level's set (on
the bucket and its, optionally prefixed, objects).
- `spec.conditions``sourceIPs` (an `aws:SourceIp` CIDR allowlist) and
`secureTransportOnly` (require TLS).
- `spec.rawStatements` — an escape hatch of raw S3 policy statements
(`effect`/`actions`/`resources`/`conditions`) merged for this grant's
principal. When set, `level`, `actions`, `paths` and `conditions` are ignored;
resources without an `arn:` prefix are treated as bucket-relative key prefixes.
RGW honours S3 bucket policy on **Reef 18.2+ / Squid**; condition-key support is
a subset of AWS, so validate exotic conditions against your cluster.
### Placement targets
`Bucket.spec.placementTarget` selects the RGW **placement target** that backs the
bucket — i.e. which pools, and therefore which durability profile, store its
data. The valid values are cluster configuration, not a fixed set baked into the
operator. On this estate radosgw exposes two:
- `default-placement` — 3× replicated (the cluster default).
- `ec` — 4+1 erasure-coded (cheaper capacity, for bulk/archival data).
Leaving `placementTarget` empty keeps the current behaviour: the owning user's
`default_placement` (falling back to the zonegroup default). When set, the
operator threads it into the S3 `CreateBucket` `LocationConstraint` as
`<zonegroup>:<placementTarget>`; with `spec.zonegroup` empty (the default) that
is `:<placementTarget>`, which selects the local/master zonegroup with the given
placement — so you do not need to know the zonegroup's api-name to pick a target.
```yaml
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: raw-archive
spec:
ownerRef: logarchiver
placementTarget: ec # 4+1 erasure-coded pool
```
Placement is **immutable**: RGW fixes it at bucket creation and cannot move an
existing bucket between targets. The CRD rejects changing `placementTarget` (and
`zonegroup`) on an existing `Bucket`, and if a bucket already lives on a
different target than the spec requests (e.g. an adopted bucket, or a value
sneaked in around the CRD guard) the controller sets an `Error` phase with a
`PlacementImmutable` reason rather than ever deleting and recreating it. The
placement RGW actually stores the bucket on is reported in
`status.placementTarget` (and the `Placement` print column), so drift is visible.
See `config/samples/06-bucket-ec.yaml`.
### Adopting existing buckets and users
The operator can take over buckets/users that already exist in radosgw and hand
them back without deleting them. In short: matching CRDs manage the resource in
place (no recreation, keys reused, `status.adopted: true`), the bucket policy is
**merged** so an existing hand-written policy is preserved (`spec.managePolicy:
false` opts out entirely), and `spec.retainOnDelete` on `ObjectStoreUser` /
`Bucket` / `BucketAccess` orphans the RGW object instead of deleting it. See
**[docs/adoption.md](docs/adoption.md)** and
`config/samples/05-adoption.yaml`.
The `Bucket` controller renders the policy as the **union of every ready
`BucketAccess`** that targets it, so the result is convergent regardless of the
order objects are created or deleted. It watches `BucketAccess` and
@@ -109,7 +177,8 @@ kubectl apply -f hack/kind/manifests/ # edit the Secret first
Woodpecker runs `pre-commit` (gofmt + vet), `test`, and a dry-run image `build`
on pull requests; pushing a `v*` tag builds and pushes
`git.unkin.net/unkin/cephrgw-operator` to the Gitea registry. Bump a release
`artifactapi.k8s.syd1.au.unkin.net/docker-internal/cephrgw-operator` to the
artifactapi local docker registry. Bump a release
with `make patch|minor|major`.
## Notes & caveats
+41 -2
View File
@@ -40,11 +40,29 @@ type BucketSpec struct {
// with BucketAccess objects.
OwnerRef string `json:"ownerRef"`
// Zonegroup optionally pins the bucket to a specific RGW zonegroup.
// Zonegroup optionally pins the bucket to a specific RGW zonegroup by its
// api-name. Empty (the default) uses the cluster's local/master zonegroup, so
// PlacementTarget selection works without naming the zonegroup. Immutable:
// RGW resolves the zonegroup at bucket creation and cannot move it afterwards.
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="zonegroup is immutable; RGW fixes it at bucket creation"
// +optional
Zonegroup string `json:"zonegroup,omitempty"`
// PlacementTarget optionally selects a non-default placement target/pool.
// PlacementTarget optionally selects the RGW placement target that backs the
// bucket, choosing which pools (and thus replication/erasure profile) store
// its data. Empty (the default) uses the owning user's default_placement, or
// the zonegroup default. The valid values are cluster configuration, not a
// fixed set; on this estate the two configured targets are
// "default-placement" (3x replicated) and "ec" (4+1 erasure-coded).
//
// Immutable: RGW chooses the placement at bucket creation (from the S3
// LocationConstraint) and cannot move an existing bucket between placement
// targets. Set it on a fresh Bucket; changing it later is rejected, and if a
// pre-existing bucket is on a different placement the operator reports an
// error instead of recreating it.
// +kubebuilder:validation:MaxLength=63
// +kubebuilder:validation:Pattern=`^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$`
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="placementTarget is immutable; RGW cannot move a bucket between placement targets"
// +optional
PlacementTarget string `json:"placementTarget,omitempty"`
@@ -64,6 +82,16 @@ type BucketSpec struct {
// +optional
Tags map[string]string `json:"tags,omitempty"`
// ManagePolicy controls whether the operator manages the bucket's S3 policy
// from BucketAccess grants. When true (the default) the operator reconciles
// its own statements while preserving any statements it does not own, so it
// is safe to adopt a bucket that already has a policy. Set to false to leave
// the bucket policy entirely untouched (BucketAccess grants then have no
// effect on this bucket).
// +kubebuilder:default=true
// +optional
ManagePolicy *bool `json:"managePolicy,omitempty"`
// RetainOnDelete keeps the RGW bucket (and its objects) when the Bucket
// resource is deleted. By default the operator removes the empty bucket;
// it never purges objects unless PurgeOnDelete is also set.
@@ -90,10 +118,19 @@ type BucketStatus struct {
// Owner is the RGW uid that owns the bucket.
// +optional
Owner string `json:"owner,omitempty"`
// PlacementTarget is the placement target RGW actually stores the bucket on,
// read back from the live bucket. It makes placement drift (a bucket landing
// on a different target than spec requested) visible.
// +optional
PlacementTarget string `json:"placementTarget,omitempty"`
// PolicyPrincipals is the number of extra principals granted via
// BucketAccess and reflected in the bucket policy.
// +optional
PolicyPrincipals int32 `json:"policyPrincipals,omitempty"`
// Adopted reports that the RGW bucket already existed when the operator
// first reconciled this resource (it was taken over, not created).
// +optional
Adopted bool `json:"adopted,omitempty"`
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +optional
@@ -107,7 +144,9 @@ type BucketStatus struct {
// +kubebuilder:resource:shortName=bkt
// +kubebuilder:printcolumn:name="Bucket",type=string,JSONPath=`.status.bucketName`
// +kubebuilder:printcolumn:name="Owner",type=string,JSONPath=`.status.owner`
// +kubebuilder:printcolumn:name="Placement",type=string,JSONPath=`.status.placementTarget`
// +kubebuilder:printcolumn:name="Grants",type=integer,JSONPath=`.status.policyPrincipals`
// +kubebuilder:printcolumn:name="Adopted",type=boolean,JSONPath=`.status.adopted`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// Bucket is a Ceph RGW S3 bucket.
+72
View File
@@ -45,6 +45,78 @@ type BucketAccessSpec struct {
// dedicated user it creates (UserRef empty). Defaults to "<name>-rgw".
// +optional
SecretName string `json:"secretName,omitempty"`
// RetainOnDelete keeps the dedicated RGW user (created when UserRef is empty)
// instead of deleting it when this BucketAccess is removed. Ignored when
// UserRef is set (that user is never managed here). Defaults to false.
// +optional
RetainOnDelete bool `json:"retainOnDelete,omitempty"`
// Paths optionally scopes object-level access to these key prefixes within
// the bucket; each becomes the resource "<bucket>/<prefix>*". Empty grants
// the whole bucket. The bucket-level ListBucket action always applies to the
// whole bucket. Ignored when RawStatements is set.
// +optional
Paths []string `json:"paths,omitempty"`
// Actions optionally overrides the S3 actions granted by Level. When set,
// exactly these actions are granted, on the bucket and its (optionally
// prefixed) objects. Ignored when RawStatements is set.
// +optional
Actions []string `json:"actions,omitempty"`
// Conditions optionally restricts when the grant applies (e.g. source IPs,
// TLS required). Ignored when RawStatements is set.
// +optional
Conditions *AccessConditions `json:"conditions,omitempty"`
// RawStatements is an escape hatch for arbitrary S3 policy statements, merged
// into the bucket policy for this grant's principal. When set, Level,
// Actions, Paths and Conditions on this object are ignored; the operator only
// fills in the Principal (this grant's user) when a statement omits one.
// +optional
RawStatements []PolicyStatement `json:"rawStatements,omitempty"`
}
// AccessConditions restricts when a grant applies. Each field maps to an S3
// policy condition and, when several are set, all must hold (they are AND'd).
type AccessConditions struct {
// SourceIPs restricts the grant to requests from these CIDRs (or single
// addresses), via the S3 aws:SourceIp condition.
// +optional
SourceIPs []string `json:"sourceIPs,omitempty"`
// SecureTransportOnly requires the request to use TLS, via the S3
// aws:SecureTransport condition.
// +optional
SecureTransportOnly bool `json:"secureTransportOnly,omitempty"`
}
// PolicyStatement is a raw S3 bucket-policy statement, exposed for grants that
// need control beyond Level/Actions/Paths/Conditions.
type PolicyStatement struct {
// Sid is an optional statement id. The operator derives one when empty.
// +optional
Sid string `json:"sid,omitempty"`
// Effect is Allow or Deny. Defaults to Allow.
// +kubebuilder:validation:Enum=Allow;Deny
// +kubebuilder:default=Allow
// +optional
Effect string `json:"effect,omitempty"`
// Actions are the S3 actions the statement covers (e.g. s3:GetObject).
Actions []string `json:"actions"`
// Resources are S3 resource ARNs, or bucket-relative key prefixes when they
// do not start with "arn:". Empty means the whole bucket and its objects.
// +optional
Resources []string `json:"resources,omitempty"`
// Conditions is the raw S3 condition block: operator -> condition key ->
// values, e.g. {"IpAddress": {"aws:SourceIp": ["10.0.0.0/8"]}}.
// +optional
Conditions map[string]map[string][]string `json:"conditions,omitempty"`
}
// BucketAccessStatus reports observed grant state.
+15 -2
View File
@@ -27,9 +27,11 @@ type ObjectStoreUserSpec struct {
// +optional
MaxBuckets *int32 `json:"maxBuckets,omitempty"`
// Suspended, when true, suspends the user so its keys stop working.
// Suspended manages the user's suspended state: true suspends the user so
// its keys stop working, false resumes it. When unset the operator does not
// touch the suspended state (useful when adopting an existing user).
// +optional
Suspended bool `json:"suspended,omitempty"`
Suspended *bool `json:"suspended,omitempty"`
// Quota optionally applies a user-level quota.
// +optional
@@ -40,6 +42,12 @@ type ObjectStoreUserSpec struct {
// AWS_SECRET_ACCESS_KEY, BUCKET_HOST and the RGW uid.
// +optional
SecretName string `json:"secretName,omitempty"`
// RetainOnDelete keeps the RGW user (and its keys) when the ObjectStoreUser
// resource is deleted, instead of removing it. Set this before adopting an
// existing user you may later want to hand back. Defaults to false.
// +optional
RetainOnDelete bool `json:"retainOnDelete,omitempty"`
}
// ObjectStoreUserStatus reports observed user state.
@@ -53,6 +61,10 @@ type ObjectStoreUserStatus struct {
// SecretName is the Secret holding the user's credentials.
// +optional
SecretName string `json:"secretName,omitempty"`
// Adopted reports that the RGW user already existed when the operator first
// reconciled this resource (it was taken over, not created).
// +optional
Adopted bool `json:"adopted,omitempty"`
// +optional
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// +optional
@@ -66,6 +78,7 @@ type ObjectStoreUserStatus struct {
// +kubebuilder:resource:shortName=osu
// +kubebuilder:printcolumn:name="UID",type=string,JSONPath=`.status.uid`
// +kubebuilder:printcolumn:name="Secret",type=string,JSONPath=`.status.secretName`
// +kubebuilder:printcolumn:name="Adopted",type=boolean,JSONPath=`.status.adopted`
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// ObjectStoreUser is a Ceph RGW S3 user whose keys are delivered into a Secret.
+105 -1
View File
@@ -9,6 +9,26 @@ import (
runtime "k8s.io/apimachinery/pkg/runtime"
)
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *AccessConditions) DeepCopyInto(out *AccessConditions) {
*out = *in
if in.SourceIPs != nil {
in, out := &in.SourceIPs, &out.SourceIPs
*out = make([]string, len(*in))
copy(*out, *in)
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AccessConditions.
func (in *AccessConditions) DeepCopy() *AccessConditions {
if in == nil {
return nil
}
out := new(AccessConditions)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Bucket) DeepCopyInto(out *Bucket) {
*out = *in
@@ -41,7 +61,7 @@ func (in *BucketAccess) DeepCopyInto(out *BucketAccess) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
out.Spec = in.Spec
in.Spec.DeepCopyInto(&out.Spec)
in.Status.DeepCopyInto(&out.Status)
}
@@ -98,6 +118,28 @@ func (in *BucketAccessList) DeepCopyObject() runtime.Object {
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *BucketAccessSpec) DeepCopyInto(out *BucketAccessSpec) {
*out = *in
if in.Paths != nil {
in, out := &in.Paths, &out.Paths
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Actions != nil {
in, out := &in.Actions, &out.Actions
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = new(AccessConditions)
(*in).DeepCopyInto(*out)
}
if in.RawStatements != nil {
in, out := &in.RawStatements, &out.RawStatements
*out = make([]PolicyStatement, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketAccessSpec.
@@ -184,6 +226,11 @@ func (in *BucketSpec) DeepCopyInto(out *BucketSpec) {
(*out)[key] = val
}
}
if in.ManagePolicy != nil {
in, out := &in.ManagePolicy, &out.ManagePolicy
*out = new(bool)
**out = **in
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new BucketSpec.
@@ -310,6 +357,11 @@ func (in *ObjectStoreUserSpec) DeepCopyInto(out *ObjectStoreUserSpec) {
*out = new(int32)
**out = **in
}
if in.Suspended != nil {
in, out := &in.Suspended, &out.Suspended
*out = new(bool)
**out = **in
}
if in.Quota != nil {
in, out := &in.Quota, &out.Quota
*out = new(Quota)
@@ -349,6 +401,58 @@ func (in *ObjectStoreUserStatus) DeepCopy() *ObjectStoreUserStatus {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *PolicyStatement) DeepCopyInto(out *PolicyStatement) {
*out = *in
if in.Actions != nil {
in, out := &in.Actions, &out.Actions
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Resources != nil {
in, out := &in.Resources, &out.Resources
*out = make([]string, len(*in))
copy(*out, *in)
}
if in.Conditions != nil {
in, out := &in.Conditions, &out.Conditions
*out = make(map[string]map[string][]string, len(*in))
for key, val := range *in {
var outVal map[string][]string
if val == nil {
(*out)[key] = nil
} else {
inVal := (*in)[key]
in, out := &inVal, &outVal
*out = make(map[string][]string, len(*in))
for key, val := range *in {
var outVal []string
if val == nil {
(*out)[key] = nil
} else {
inVal := (*in)[key]
in, out := &inVal, &outVal
*out = make([]string, len(*in))
copy(*out, *in)
}
(*out)[key] = outVal
}
}
(*out)[key] = outVal
}
}
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PolicyStatement.
func (in *PolicyStatement) DeepCopy() *PolicyStatement {
if in == nil {
return nil
}
out := new(PolicyStatement)
in.DeepCopyInto(out)
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Quota) DeepCopyInto(out *Quota) {
*out = *in
+7
View File
@@ -73,6 +73,13 @@ func main() {
os.Exit(1)
}
// Advisory only: warn (never exit) if the installed CRDs are missing or
// predate this operator's schema, which otherwise surfaces only as opaque
// strict-decode failures during reconcile.
crdCtx, crdCancel := context.WithTimeout(context.Background(), 15*time.Second)
controller.CheckCRDVersions(crdCtx, mgr.GetConfig())
crdCancel()
if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
logger.Error(err, "unable to set up health check")
os.Exit(1)
@@ -60,10 +60,36 @@ spec:
operator provisions a dedicated user for this grant and writes its keys into
a Secret; otherwise it grants an existing ObjectStoreUser.
properties:
actions:
description: |-
Actions optionally overrides the S3 actions granted by Level. When set,
exactly these actions are granted, on the bucket and its (optionally
prefixed) objects. Ignored when RawStatements is set.
items:
type: string
type: array
bucketRef:
description: BucketRef names the Bucket (in this namespace) to grant
access to.
type: string
conditions:
description: |-
Conditions optionally restricts when the grant applies (e.g. source IPs,
TLS required). Ignored when RawStatements is set.
properties:
secureTransportOnly:
description: |-
SecureTransportOnly requires the request to use TLS, via the S3
aws:SecureTransport condition.
type: boolean
sourceIPs:
description: |-
SourceIPs restricts the grant to requests from these CIDRs (or single
addresses), via the S3 aws:SourceIp condition.
items:
type: string
type: array
type: object
level:
description: Level is the access level to grant.
enum:
@@ -71,6 +97,71 @@ spec:
- read-write
- full
type: string
paths:
description: |-
Paths optionally scopes object-level access to these key prefixes within
the bucket; each becomes the resource "<bucket>/<prefix>*". Empty grants
the whole bucket. The bucket-level ListBucket action always applies to the
whole bucket. Ignored when RawStatements is set.
items:
type: string
type: array
rawStatements:
description: |-
RawStatements is an escape hatch for arbitrary S3 policy statements, merged
into the bucket policy for this grant's principal. When set, Level,
Actions, Paths and Conditions on this object are ignored; the operator only
fills in the Principal (this grant's user) when a statement omits one.
items:
description: |-
PolicyStatement is a raw S3 bucket-policy statement, exposed for grants that
need control beyond Level/Actions/Paths/Conditions.
properties:
actions:
description: Actions are the S3 actions the statement covers
(e.g. s3:GetObject).
items:
type: string
type: array
conditions:
additionalProperties:
additionalProperties:
items:
type: string
type: array
type: object
description: |-
Conditions is the raw S3 condition block: operator -> condition key ->
values, e.g. {"IpAddress": {"aws:SourceIp": ["10.0.0.0/8"]}}.
type: object
effect:
default: Allow
description: Effect is Allow or Deny. Defaults to Allow.
enum:
- Allow
- Deny
type: string
resources:
description: |-
Resources are S3 resource ARNs, or bucket-relative key prefixes when they
do not start with "arn:". Empty means the whole bucket and its objects.
items:
type: string
type: array
sid:
description: Sid is an optional statement id. The operator derives
one when empty.
type: string
required:
- actions
type: object
type: array
retainOnDelete:
description: |-
RetainOnDelete keeps the dedicated RGW user (created when UserRef is empty)
instead of deleting it when this BucketAccess is removed. Ignored when
UserRef is set (that user is never managed here). Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes credentials into for the
+54 -4
View File
@@ -23,9 +23,15 @@ spec:
- jsonPath: .status.owner
name: Owner
type: string
- jsonPath: .status.placementTarget
name: Placement
type: string
- jsonPath: .status.policyPrincipals
name: Grants
type: integer
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
@@ -58,6 +64,16 @@ spec:
description: BucketName is the S3 bucket name. Defaults to metadata.name.
Immutable.
type: string
managePolicy:
default: true
description: |-
ManagePolicy controls whether the operator manages the bucket's S3 policy
from BucketAccess grants. When true (the default) the operator reconciles
its own statements while preserving any statements it does not own, so it
is safe to adopt a bucket that already has a policy. Set to false to leave
the bucket policy entirely untouched (BucketAccess grants then have no
effect on this bucket).
type: boolean
objectLock:
description: ObjectLock configures S3 object lock. Enabling it forces
versioning on.
@@ -92,9 +108,26 @@ spec:
with BucketAccess objects.
type: string
placementTarget:
description: PlacementTarget optionally selects a non-default placement
target/pool.
description: |-
PlacementTarget optionally selects the RGW placement target that backs the
bucket, choosing which pools (and thus replication/erasure profile) store
its data. Empty (the default) uses the owning user's default_placement, or
the zonegroup default. The valid values are cluster configuration, not a
fixed set; on this estate the two configured targets are
"default-placement" (3x replicated) and "ec" (4+1 erasure-coded).
Immutable: RGW chooses the placement at bucket creation (from the S3
LocationConstraint) and cannot move an existing bucket between placement
targets. Set it on a fresh Bucket; changing it later is rejected, and if a
pre-existing bucket is on a different placement the operator reports an
error instead of recreating it.
maxLength: 63
pattern: ^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$
type: string
x-kubernetes-validations:
- message: placementTarget is immutable; RGW cannot move a bucket
between placement targets
rule: self == oldSelf
purgeOnDelete:
description: |-
PurgeOnDelete deletes the bucket together with all objects it contains
@@ -135,15 +168,26 @@ spec:
description: Versioning enables S3 object versioning on the bucket.
type: boolean
zonegroup:
description: Zonegroup optionally pins the bucket to a specific RGW
zonegroup.
description: |-
Zonegroup optionally pins the bucket to a specific RGW zonegroup by its
api-name. Empty (the default) uses the cluster's local/master zonegroup, so
PlacementTarget selection works without naming the zonegroup. Immutable:
RGW resolves the zonegroup at bucket creation and cannot move it afterwards.
type: string
x-kubernetes-validations:
- message: zonegroup is immutable; RGW fixes it at bucket creation
rule: self == oldSelf
required:
- ownerRef
type: object
status:
description: BucketStatus reports observed bucket state.
properties:
adopted:
description: |-
Adopted reports that the RGW bucket already existed when the operator
first reconciled this resource (it was taken over, not created).
type: boolean
bucketID:
description: BucketID is the RGW internal bucket instance id.
type: string
@@ -218,6 +262,12 @@ spec:
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
placementTarget:
description: |-
PlacementTarget is the placement target RGW actually stores the bucket on,
read back from the live bucket. It makes placement drift (a bucket landing
on a different target than spec requested) visible.
type: string
policyPrincipals:
description: |-
PolicyPrincipals is the number of extra principals granted via
@@ -23,6 +23,9 @@ spec:
- jsonPath: .status.secretName
name: Secret
type: string
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
@@ -52,7 +55,7 @@ spec:
spec:
description: |-
ObjectStoreUserSpec defines a Ceph RGW (S3) user. The operator creates the
user through the Ceph dashboard API and writes its generated access/secret
user through the radosgw Admin Ops API and writes its generated access/secret
key pair into a Kubernetes Secret. The key material is never stored on the
resource itself.
properties:
@@ -90,6 +93,12 @@ spec:
format: int64
type: integer
type: object
retainOnDelete:
description: |-
RetainOnDelete keeps the RGW user (and its keys) when the ObjectStoreUser
resource is deleted, instead of removing it. Set this before adopting an
existing user you may later want to hand back. Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes the access/secret key into.
@@ -97,8 +106,10 @@ spec:
AWS_SECRET_ACCESS_KEY, BUCKET_HOST and the RGW uid.
type: string
suspended:
description: Suspended, when true, suspends the user so its keys stop
working.
description: |-
Suspended manages the user's suspended state: true suspends the user so
its keys stop working, false resumes it. When unset the operator does not
touch the suspended state (useful when adopting an existing user).
type: boolean
uid:
description: UID is the RGW user id. Defaults to metadata.name. Immutable
@@ -108,6 +119,11 @@ spec:
status:
description: ObjectStoreUserStatus reports observed user state.
properties:
adopted:
description: |-
Adopted reports that the RGW user already existed when the operator first
reconciled this resource (it was taken over, not created).
type: boolean
conditions:
items:
description: Condition contains details for one aspect of the current
+164 -7
View File
@@ -61,10 +61,36 @@ spec:
operator provisions a dedicated user for this grant and writes its keys into
a Secret; otherwise it grants an existing ObjectStoreUser.
properties:
actions:
description: |-
Actions optionally overrides the S3 actions granted by Level. When set,
exactly these actions are granted, on the bucket and its (optionally
prefixed) objects. Ignored when RawStatements is set.
items:
type: string
type: array
bucketRef:
description: BucketRef names the Bucket (in this namespace) to grant
access to.
type: string
conditions:
description: |-
Conditions optionally restricts when the grant applies (e.g. source IPs,
TLS required). Ignored when RawStatements is set.
properties:
secureTransportOnly:
description: |-
SecureTransportOnly requires the request to use TLS, via the S3
aws:SecureTransport condition.
type: boolean
sourceIPs:
description: |-
SourceIPs restricts the grant to requests from these CIDRs (or single
addresses), via the S3 aws:SourceIp condition.
items:
type: string
type: array
type: object
level:
description: Level is the access level to grant.
enum:
@@ -72,6 +98,71 @@ spec:
- read-write
- full
type: string
paths:
description: |-
Paths optionally scopes object-level access to these key prefixes within
the bucket; each becomes the resource "<bucket>/<prefix>*". Empty grants
the whole bucket. The bucket-level ListBucket action always applies to the
whole bucket. Ignored when RawStatements is set.
items:
type: string
type: array
rawStatements:
description: |-
RawStatements is an escape hatch for arbitrary S3 policy statements, merged
into the bucket policy for this grant's principal. When set, Level,
Actions, Paths and Conditions on this object are ignored; the operator only
fills in the Principal (this grant's user) when a statement omits one.
items:
description: |-
PolicyStatement is a raw S3 bucket-policy statement, exposed for grants that
need control beyond Level/Actions/Paths/Conditions.
properties:
actions:
description: Actions are the S3 actions the statement covers
(e.g. s3:GetObject).
items:
type: string
type: array
conditions:
additionalProperties:
additionalProperties:
items:
type: string
type: array
type: object
description: |-
Conditions is the raw S3 condition block: operator -> condition key ->
values, e.g. {"IpAddress": {"aws:SourceIp": ["10.0.0.0/8"]}}.
type: object
effect:
default: Allow
description: Effect is Allow or Deny. Defaults to Allow.
enum:
- Allow
- Deny
type: string
resources:
description: |-
Resources are S3 resource ARNs, or bucket-relative key prefixes when they
do not start with "arn:". Empty means the whole bucket and its objects.
items:
type: string
type: array
sid:
description: Sid is an optional statement id. The operator derives
one when empty.
type: string
required:
- actions
type: object
type: array
retainOnDelete:
description: |-
RetainOnDelete keeps the dedicated RGW user (created when UserRef is empty)
instead of deleting it when this BucketAccess is removed. Ignored when
UserRef is set (that user is never managed here). Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes credentials into for the
@@ -202,9 +293,15 @@ spec:
- jsonPath: .status.owner
name: Owner
type: string
- jsonPath: .status.placementTarget
name: Placement
type: string
- jsonPath: .status.policyPrincipals
name: Grants
type: integer
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
@@ -237,6 +334,16 @@ spec:
description: BucketName is the S3 bucket name. Defaults to metadata.name.
Immutable.
type: string
managePolicy:
default: true
description: |-
ManagePolicy controls whether the operator manages the bucket's S3 policy
from BucketAccess grants. When true (the default) the operator reconciles
its own statements while preserving any statements it does not own, so it
is safe to adopt a bucket that already has a policy. Set to false to leave
the bucket policy entirely untouched (BucketAccess grants then have no
effect on this bucket).
type: boolean
objectLock:
description: ObjectLock configures S3 object lock. Enabling it forces
versioning on.
@@ -271,9 +378,26 @@ spec:
with BucketAccess objects.
type: string
placementTarget:
description: PlacementTarget optionally selects a non-default placement
target/pool.
description: |-
PlacementTarget optionally selects the RGW placement target that backs the
bucket, choosing which pools (and thus replication/erasure profile) store
its data. Empty (the default) uses the owning user's default_placement, or
the zonegroup default. The valid values are cluster configuration, not a
fixed set; on this estate the two configured targets are
"default-placement" (3x replicated) and "ec" (4+1 erasure-coded).
Immutable: RGW chooses the placement at bucket creation (from the S3
LocationConstraint) and cannot move an existing bucket between placement
targets. Set it on a fresh Bucket; changing it later is rejected, and if a
pre-existing bucket is on a different placement the operator reports an
error instead of recreating it.
maxLength: 63
pattern: ^[a-zA-Z0-9]([a-zA-Z0-9._-]*[a-zA-Z0-9])?$
type: string
x-kubernetes-validations:
- message: placementTarget is immutable; RGW cannot move a bucket
between placement targets
rule: self == oldSelf
purgeOnDelete:
description: |-
PurgeOnDelete deletes the bucket together with all objects it contains
@@ -314,15 +438,26 @@ spec:
description: Versioning enables S3 object versioning on the bucket.
type: boolean
zonegroup:
description: Zonegroup optionally pins the bucket to a specific RGW
zonegroup.
description: |-
Zonegroup optionally pins the bucket to a specific RGW zonegroup by its
api-name. Empty (the default) uses the cluster's local/master zonegroup, so
PlacementTarget selection works without naming the zonegroup. Immutable:
RGW resolves the zonegroup at bucket creation and cannot move it afterwards.
type: string
x-kubernetes-validations:
- message: zonegroup is immutable; RGW fixes it at bucket creation
rule: self == oldSelf
required:
- ownerRef
type: object
status:
description: BucketStatus reports observed bucket state.
properties:
adopted:
description: |-
Adopted reports that the RGW bucket already existed when the operator
first reconciled this resource (it was taken over, not created).
type: boolean
bucketID:
description: BucketID is the RGW internal bucket instance id.
type: string
@@ -397,6 +532,12 @@ spec:
phase:
description: Phase is a coarse lifecycle summary (Pending/Ready/Error).
type: string
placementTarget:
description: |-
PlacementTarget is the placement target RGW actually stores the bucket on,
read back from the live bucket. It makes placement drift (a bucket landing
on a different target than spec requested) visible.
type: string
policyPrincipals:
description: |-
PolicyPrincipals is the number of extra principals granted via
@@ -434,6 +575,9 @@ spec:
- jsonPath: .status.secretName
name: Secret
type: string
- jsonPath: .status.adopted
name: Adopted
type: boolean
- jsonPath: .status.phase
name: Phase
type: string
@@ -463,7 +607,7 @@ spec:
spec:
description: |-
ObjectStoreUserSpec defines a Ceph RGW (S3) user. The operator creates the
user through the Ceph dashboard API and writes its generated access/secret
user through the radosgw Admin Ops API and writes its generated access/secret
key pair into a Kubernetes Secret. The key material is never stored on the
resource itself.
properties:
@@ -501,6 +645,12 @@ spec:
format: int64
type: integer
type: object
retainOnDelete:
description: |-
RetainOnDelete keeps the RGW user (and its keys) when the ObjectStoreUser
resource is deleted, instead of removing it. Set this before adopting an
existing user you may later want to hand back. Defaults to false.
type: boolean
secretName:
description: |-
SecretName is the Secret the operator writes the access/secret key into.
@@ -508,8 +658,10 @@ spec:
AWS_SECRET_ACCESS_KEY, BUCKET_HOST and the RGW uid.
type: string
suspended:
description: Suspended, when true, suspends the user so its keys stop
working.
description: |-
Suspended manages the user's suspended state: true suspends the user so
its keys stop working, false resumes it. When unset the operator does not
touch the suspended state (useful when adopting an existing user).
type: boolean
uid:
description: UID is the RGW user id. Defaults to metadata.name. Immutable
@@ -519,6 +671,11 @@ spec:
status:
description: ObjectStoreUserStatus reports observed user state.
properties:
adopted:
description: |-
Adopted reports that the RGW user already existed when the operator first
reconciled this resource (it was taken over, not created).
type: boolean
conditions:
items:
description: Condition contains details for one aspect of the current
+7
View File
@@ -16,6 +16,13 @@ rules:
- patch
- update
- watch
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- get
- list
- apiGroups:
- ceph.unkin.net
resources:
@@ -0,0 +1,78 @@
# Fine-grained grants. Each of these refines the coarse read-only/read-write/full
# levels with prefix scoping, action overrides, conditions, or raw statements.
# 1. Prefix-scoped read-write: this workload may read/write objects only under
# the "uploads/" and "tmp/" key prefixes (bucket-level ListBucket still spans
# the whole bucket).
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-uploader
namespace: default
spec:
bucketRef: app-data
level: read-write
secretName: app-data-uploader-rgw
paths:
- uploads/
- tmp/
---
# 2. Read-only from inside the cluster only: restrict the grant to a source CIDR
# and require TLS.
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-internal-ro
namespace: default
spec:
bucketRef: app-data
level: read-only
secretName: app-data-internal-ro-rgw
conditions:
sourceIPs:
- 10.0.0.0/8
secureTransportOnly: true
---
# 3. Explicit action set: grant exactly these actions instead of a level's
# canned set (level is still required but its actions are ignored).
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-getput
namespace: default
spec:
bucketRef: app-data
level: read-only
secretName: app-data-getput-rgw
actions:
- s3:GetObject
- s3:PutObject
---
# 4. Raw statements escape hatch: full control over the policy statement. Level,
# actions, paths and conditions are ignored; the operator only injects the
# Principal (this grant's user). Resources without an "arn:" prefix are
# treated as bucket-relative key prefixes.
apiVersion: ceph.unkin.net/v1alpha1
kind: BucketAccess
metadata:
name: app-data-raw
namespace: default
spec:
bucketRef: app-data
level: read-only
secretName: app-data-raw-rgw
rawStatements:
- effect: Allow
actions:
- s3:GetObject
resources:
- public/
- effect: Deny
actions:
- s3:DeleteObject
resources:
- locked/
conditions:
Bool:
aws:SecureTransport:
- "false"
+31
View File
@@ -0,0 +1,31 @@
# Adopting an existing radosgw user + bucket. The operator takes them over in
# place: no recreation, existing keys reused, existing bucket policy preserved.
# retainOnDelete keeps the RGW objects if these CRDs are later deleted.
# See docs/adoption.md.
apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
name: legacy-owner
namespace: default
spec:
# uid must match the existing RGW user id.
uid: legacy-owner
# Set maxBuckets to the existing user's limit (it otherwise defaults to 1000
# and would be applied). Leave displayName/suspended unset to keep them as-is.
maxBuckets: 1000
retainOnDelete: true
---
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: legacy-data
namespace: default
spec:
# bucketName must match the existing bucket.
bucketName: legacy-data
ownerRef: legacy-owner
retainOnDelete: true
# managePolicy defaults to true: the operator merges its BucketAccess grants
# into the existing policy, preserving statements it does not own. Set it to
# false to leave the bucket policy entirely under manual control.
managePolicy: true
+18
View File
@@ -0,0 +1,18 @@
# A bucket placed on the erasure-coded (4+1) placement target instead of the
# default 3x-replicated pool. Good for bulk/archival data where capacity matters
# more than the extra replica.
#
# placementTarget is immutable: RGW chooses the placement at bucket creation and
# cannot move an existing bucket between targets, so it can only be set on a
# fresh Bucket. The operator reports the live placement in status.placementTarget.
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: raw-archive
namespace: default
spec:
bucketName: raw-archive
ownerRef: app-owner
# Cluster-configured placement target. On this estate: "default-placement"
# (3x replicated) or "ec" (4+1 erasure-coded).
placementTarget: ec
+68
View File
@@ -0,0 +1,68 @@
# Adopting existing radosgw buckets and users
The operator can **take over** buckets and users that already exist in radosgw:
write CRDs that match them and it manages them in place instead of recreating
them. It can also **hand them back** without deleting the underlying RGW objects.
## What happens when you create a CRD for an existing resource
| CRD | On adoption |
|-----|-------------|
| `ObjectStoreUser` | The user is **not** recreated and its **existing keys are reused** (never rotated). `status.adopted` becomes `true`. Attributes are only changed if the spec sets them (see below). |
| `Bucket` | The bucket is **not** recreated. `status.adopted` becomes `true`. Versioning/tags/quota are only touched if the spec sets them. The policy is **merged**, not overwritten (see below). |
| `BucketAccess` | Adds the grant's statement to the bucket policy and (for a dedicated user) reuses that user's keys. |
`status.adopted` is also shown in `kubectl get osu` / `kubectl get bkt` under
the **ADOPTED** column.
### User attributes
To avoid clobbering an adopted user, the operator only sends attributes you set:
- `displayName` — left unchanged when empty.
- `suspended` — left unchanged when unset (it is an optional `*bool`; set it
explicitly to `true`/`false` to manage it).
- `email` — left unchanged when empty.
- `maxBuckets`**defaults to `1000`** and is always applied. If the existing
user has a different limit you want to keep, set `maxBuckets` to match (or to
the value you want).
### Bucket policy is merged, not replaced
The operator owns only the policy statements it writes — they carry a
`cephrgwop…` statement id. On every reconcile it **preserves statements it does
not own** and reconciles only its own. So adopting a bucket that already has a
hand-written policy keeps that policy; your `BucketAccess` grants are added
alongside it.
- Reserve the `cephrgwop` prefix (and the legacy prefixes `full`, `custom`,
`raw`, `readonly*`, `readwrite*`) for the operator — do not name your own
statements with them, or they will be treated as operator-owned and replaced.
- To have the operator **never touch** a bucket's policy, set
`spec.managePolicy: false`. `BucketAccess` grants then have no effect on that
bucket.
## What happens when you delete the CRD
By default, deleting a CRD deletes the underlying RGW resource. Opt out per
resource to **orphan** it instead (the finalizer is dropped, the RGW object is
left in place):
| CRD | Default on delete | Keep the RGW object |
|-----|-------------------|---------------------|
| `ObjectStoreUser` | deletes the RGW user + keys | `spec.retainOnDelete: true` |
| `Bucket` | deletes the (empty) bucket | `spec.retainOnDelete: true` (and never set `purgeOnDelete`) |
| `BucketAccess` (dedicated user) | deletes the dedicated user | `spec.retainOnDelete: true` |
| `BucketAccess` (`userRef`) | only drops the policy statement | n/a (never manages that user) |
## Recommended adoption procedure
1. Create an `ObjectStoreUser` for each owner, setting `retainOnDelete: true`
and `maxBuckets` to the value you want. Confirm `ADOPTED=true`.
2. Create the `Bucket` (with `retainOnDelete: true`) referencing that owner.
With `managePolicy: true` (default) the existing policy is preserved.
3. Optionally add `BucketAccess` objects to model existing grants; they merge
into the policy. If you would rather keep managing the policy by hand, set
`managePolicy: false` on the Bucket.
4. To hand a resource back, delete its CRD — with `retainOnDelete: true` the RGW
bucket/user is left untouched.
+1 -1
View File
@@ -9,6 +9,7 @@ require (
github.com/aws/smithy-go v1.27.4
github.com/ceph/go-ceph v0.40.0
k8s.io/api v0.34.4
k8s.io/apiextensions-apiserver v0.34.1
k8s.io/apimachinery v0.34.4
k8s.io/client-go v0.34.4
sigs.k8s.io/controller-runtime v0.22.4
@@ -70,7 +71,6 @@ require (
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/apiextensions-apiserver v0.34.1 // indirect
k8s.io/klog/v2 v2.130.1 // indirect
k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
+40 -3
View File
@@ -18,6 +18,11 @@ type BucketInfo struct {
Bid string
ID string
Owner string
// PlacementRule is the placement target RGW stores the bucket on (e.g.
// "default-placement" or "ec"), read from the Admin Ops bucket stats.
PlacementRule string
// Zonegroup is the RGW zonegroup id the bucket belongs to.
Zonegroup string
}
// Name returns the bucket name regardless of the field radosgw used.
@@ -50,7 +55,13 @@ func (c *Client) GetBucket(ctx context.Context, name string) (*BucketInfo, error
if err != nil {
return nil, err
}
return &BucketInfo{Bucket: b.Bucket, ID: b.ID, Owner: b.Owner}, nil
return &BucketInfo{
Bucket: b.Bucket,
ID: b.ID,
Owner: b.Owner,
PlacementRule: b.PlacementRule,
Zonegroup: b.Zonegroup,
}, nil
}
// CreateBucket provisions a bucket owned by spec.OwnerUID. The Admin Ops API
@@ -125,6 +136,26 @@ func (c *Client) SetBucketPolicy(ctx context.Context, name, bucketID, ownerUID,
return err
}
// GetBucketPolicy returns the bucket's current S3 policy JSON, or "" when it has
// none. It is signed as the bucket owner.
func (c *Client) GetBucketPolicy(ctx context.Context, name, ownerUID string) (string, error) {
owner, err := c.asOwner(ctx, ownerUID)
if err != nil {
return "", err
}
out, err := c.s3.GetBucketPolicy(ctx, &s3.GetBucketPolicyInput{Bucket: aws.String(name)}, owner)
if err != nil {
if IsNotFound(err) {
return "", nil
}
return "", err
}
if out.Policy == nil {
return "", nil
}
return *out.Policy, nil
}
// SetBucketTags replaces the bucket tag set. tagsJSON is the JSON produced by
// BuildTagJSON (a list of {"Key","Value"} objects). bucketID is unused (kept for
// call-site stability).
@@ -191,8 +222,14 @@ func (c *Client) setObjectLockDefault(ctx context.Context, owner func(*s3.Option
return err
}
// locationConstraint renders the RGW LocationConstraint from a zonegroup and
// placement target ("<zonegroup>:<placement>"), or "" for default placement.
// locationConstraint renders the RGW S3 CreateBucket LocationConstraint from a
// zonegroup api-name and a placement target. RGW's S3 create-bucket handler
// splits the value on the first ":" — the part before is the zonegroup api-name,
// the part after is the placement target id. An empty zonegroup (the common
// case) yields ":<placement>", which selects the local/master zonegroup with the
// given placement, so callers need not know the zonegroup's api-name to pick a
// placement target. Both empty yields "" (no constraint: user/zonegroup
// default). Placement empty with a zonegroup set yields just the zonegroup.
func locationConstraint(zonegroup, placement string) string {
loc := zonegroup
if placement != "" {
+68
View File
@@ -0,0 +1,68 @@
package ceph
import (
"context"
"net/http"
"net/http/httptest"
"testing"
)
func TestLocationConstraint(t *testing.T) {
cases := []struct {
name string
zonegroup string
placement string
want string
}{
{"both empty -> no constraint", "", "", ""},
{"placement only -> local zonegroup", "", "ec", ":ec"},
{"placement only default target", "", "default-placement", ":default-placement"},
{"zonegroup and placement", "default", "ec", "default:ec"},
{"zonegroup only", "default", "", "default"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := locationConstraint(tc.zonegroup, tc.placement); got != tc.want {
t.Errorf("locationConstraint(%q,%q)=%q want %q", tc.zonegroup, tc.placement, got, tc.want)
}
})
}
}
// TestGetBucketPlacement verifies GetBucket surfaces the placement target and
// zonegroup from the Admin Ops bucket-stats response, so the controller can
// detect placement drift.
func TestGetBucketPlacement(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"bucket": "raw-archive",
"id": "eae688bc-ee35-445d-9188-111b73c8b4a0.12345.1",
"owner": "logarchiver",
"zonegroup": "eae688bc-ee35-445d-9188-111b73c8b4a0",
"placement_rule": "ec"
}`))
}))
defer srv.Close()
c, err := NewClient(Config{Endpoint: srv.URL, AccessKey: "a", SecretKey: "s"})
if err != nil {
t.Fatalf("NewClient: %v", err)
}
info, err := c.GetBucket(context.Background(), "raw-archive")
if err != nil {
t.Fatalf("GetBucket: %v", err)
}
if info.PlacementRule != "ec" {
t.Errorf("PlacementRule=%q want %q", info.PlacementRule, "ec")
}
if info.Zonegroup != "eae688bc-ee35-445d-9188-111b73c8b4a0" {
t.Errorf("Zonegroup=%q unexpected", info.Zonegroup)
}
if info.Owner != "logarchiver" {
t.Errorf("Owner=%q want logarchiver", info.Owner)
}
if info.Name() != "raw-archive" {
t.Errorf("Name()=%q want raw-archive", info.Name())
}
}
+278 -46
View File
@@ -2,7 +2,9 @@ package ceph
import (
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
)
@@ -14,10 +16,53 @@ const (
LevelFull = "full"
)
// Grant couples an RGW user id with the access level to grant it on a bucket.
// managedSidPrefix marks statement ids the operator owns, so it can reconcile
// its own statements while preserving foreign ones when adopting a bucket that
// already has a policy. Do not reuse this prefix (or the legacy prefixes below)
// for statements you manage yourself.
const managedSidPrefix = "cephrgwop"
// legacyManagedSidPrefixes are the statement-id prefixes the operator emitted
// before managedSidPrefix existed; they are still recognised as operator-owned
// so upgrading does not duplicate statements.
var legacyManagedSidPrefixes = []string{
"readonlybkt", "readonlyobj", "readwritebkt", "readwriteobj", "full", "custom", "raw",
}
// GrantConditions restricts when a grant's statements apply. The zero value adds
// no conditions.
type GrantConditions struct {
// SourceIPs restricts the grant to these CIDRs (S3 aws:SourceIp).
SourceIPs []string
// SecureTransportOnly requires TLS (S3 aws:SecureTransport).
SecureTransportOnly bool
}
// RawStatement is a caller-supplied S3 policy statement for a grant.
type RawStatement struct {
Sid string
Effect string
Actions []string
Resources []string
Condition map[string]map[string][]string
}
// Grant couples an RGW user id with the access it should have on a bucket. The
// simple form is a Level; Paths, Actions and Conditions refine it, and Raw
// replaces it entirely with caller-supplied statements.
type Grant struct {
UID string
Level string
// Paths scopes object-level access to these key prefixes; empty = whole
// bucket.
Paths []string
// Actions overrides the level's action set; empty = derive from Level.
Actions []string
// Conditions optionally restricts when the grant applies.
Conditions *GrantConditions
// Raw, when non-empty, replaces Level/Actions/Paths/Conditions with these
// statements (the operator still fills in a Principal when one is omitted).
Raw []RawStatement
}
type policyDocument struct {
@@ -26,11 +71,12 @@ type policyDocument struct {
}
type policyStatement struct {
Sid string `json:"Sid"`
Effect string `json:"Effect"`
Principal map[string][]string `json:"Principal"`
Action []string `json:"Action"`
Resource []string `json:"Resource"`
Sid string `json:"Sid,omitempty"`
Effect string `json:"Effect"`
Principal map[string][]string `json:"Principal,omitempty"`
Action []string `json:"Action"`
Resource []string `json:"Resource"`
Condition map[string]map[string][]string `json:"Condition,omitempty"`
}
// bucket-level and object-level S3 actions per access level.
@@ -68,13 +114,101 @@ var objectActions = map[string][]string{
}
// BuildBucketPolicy renders a deterministic S3 bucket policy granting each
// principal its requested level. It returns "" when there are no grants so the
// principal its requested access. It returns "" when there are no grants so the
// caller can clear the policy.
func BuildBucketPolicy(bucket string, grants []Grant) (string, error) {
if len(grants) == 0 {
statements := buildStatements(bucket, grants)
if len(statements) == 0 {
return "", nil
}
b, err := json.Marshal(policyDocument{Version: "2012-10-17", Statement: statements})
if err != nil {
return "", err
}
return string(b), nil
}
// MergeBucketPolicy renders the operator's statements for grants and merges them
// into an existing policy, preserving any statement the operator does not own
// (identified by its Sid). It returns "" only when the merged policy would be
// empty, so a bucket adopted with a hand-written policy keeps that policy.
func MergeBucketPolicy(existing, bucket string, grants []Grant) (string, error) {
version := "2012-10-17"
var id string
var foreign []json.RawMessage
if strings.TrimSpace(existing) != "" {
var doc struct {
Version string `json:"Version"`
ID string `json:"Id,omitempty"`
Statement []json.RawMessage `json:"Statement"`
}
if err := json.Unmarshal([]byte(existing), &doc); err != nil {
return "", fmt.Errorf("parse existing bucket policy: %w", err)
}
if doc.Version != "" {
version = doc.Version
}
id = doc.ID
for _, raw := range doc.Statement {
var meta struct {
Sid string `json:"Sid"`
}
// Ignore unmarshal errors: a statement we cannot read the Sid of is
// treated as foreign and preserved verbatim.
_ = json.Unmarshal(raw, &meta)
if isManagedSid(meta.Sid) {
continue // operator-owned; re-rendered below
}
foreign = append(foreign, raw)
}
}
managed := buildStatements(bucket, grants)
if len(foreign) == 0 && len(managed) == 0 {
return "", nil
}
statements := make([]json.RawMessage, 0, len(foreign)+len(managed))
statements = append(statements, foreign...)
for _, st := range managed {
b, err := json.Marshal(st)
if err != nil {
return "", err
}
statements = append(statements, b)
}
b, err := json.Marshal(struct {
Version string `json:"Version"`
ID string `json:"Id,omitempty"`
Statement []json.RawMessage `json:"Statement"`
}{Version: version, ID: id, Statement: statements})
if err != nil {
return "", err
}
return string(b), nil
}
// isManagedSid reports whether a statement id was emitted by the operator.
func isManagedSid(s string) bool {
if strings.HasPrefix(s, managedSidPrefix) {
return true
}
for _, p := range legacyManagedSidPrefixes {
if strings.HasPrefix(s, p) {
return true
}
}
return false
}
// buildStatements renders the operator's statements for grants, sorted for
// deterministic output.
func buildStatements(bucket string, grants []Grant) []policyStatement {
if len(grants) == 0 {
return nil
}
sorted := make([]Grant, len(grants))
copy(sorted, grants)
sort.Slice(sorted, func(i, j int) bool {
@@ -85,50 +219,148 @@ func BuildBucketPolicy(bucket string, grants []Grant) (string, error) {
})
bucketARN := "arn:aws:s3:::" + bucket
objectARN := bucketARN + "/*"
doc := policyDocument{Version: "2012-10-17"}
var statements []policyStatement
for _, g := range sorted {
principal := map[string][]string{"AWS": {"arn:aws:iam:::user/" + g.UID}}
switch g.Level {
case LevelFull:
doc.Statement = append(doc.Statement, policyStatement{
Sid: sid("full", g.UID),
Effect: "Allow",
Principal: principal,
Action: []string{"s3:*"},
Resource: []string{bucketARN, objectARN},
})
default:
doc.Statement = append(doc.Statement,
policyStatement{
Sid: sid(g.Level+"-bkt", g.UID),
Effect: "Allow",
Principal: principal,
Action: bucketActions[g.Level],
Resource: []string{bucketARN},
},
policyStatement{
Sid: sid(g.Level+"-obj", g.UID),
Effect: "Allow",
Principal: principal,
Action: objectActions[g.Level],
Resource: []string{objectARN},
},
)
}
statements = append(statements, statementsForGrant(bucketARN, g)...)
}
b, err := json.Marshal(doc)
if err != nil {
return "", err
}
return string(b), nil
return statements
}
// sid builds a policy statement id that only contains characters S3 accepts.
// statementsForGrant renders the policy statements for a single grant.
func statementsForGrant(bucketARN string, g Grant) []policyStatement {
principal := map[string][]string{"AWS": {"arn:aws:iam:::user/" + g.UID}}
if len(g.Raw) > 0 {
out := make([]policyStatement, 0, len(g.Raw))
for i, rs := range g.Raw {
st := policyStatement{
// Always operator-owned so a merge re-renders (not duplicates)
// it; any user-supplied Sid is folded into the managed id.
Sid: sid(firstNonEmpty(rs.Sid, "raw"+strconv.Itoa(i)), g.UID),
Effect: firstNonEmpty(rs.Effect, "Allow"),
Principal: principal,
Action: rs.Actions,
Resource: resolveResources(bucketARN, rs.Resources),
Condition: rs.Condition,
}
out = append(out, st)
}
return out
}
cond := buildCondition(g.Conditions)
objectARNs := objectResources(bucketARN, g.Paths)
if len(g.Actions) > 0 {
return []policyStatement{{
Sid: sid("custom", g.UID),
Effect: "Allow",
Principal: principal,
Action: g.Actions,
Resource: append([]string{bucketARN}, objectARNs...),
Condition: cond,
}}
}
if g.Level == LevelFull {
return []policyStatement{{
Sid: sid("full", g.UID),
Effect: "Allow",
Principal: principal,
Action: []string{"s3:*"},
Resource: append([]string{bucketARN}, objectARNs...),
Condition: cond,
}}
}
return []policyStatement{
{
Sid: sid(g.Level+"-bkt", g.UID),
Effect: "Allow",
Principal: principal,
Action: bucketActions[g.Level],
Resource: []string{bucketARN},
Condition: cond,
},
{
Sid: sid(g.Level+"-obj", g.UID),
Effect: "Allow",
Principal: principal,
Action: objectActions[g.Level],
Resource: objectARNs,
Condition: cond,
},
}
}
// objectResources renders the object-level resource ARNs for a grant: the whole
// bucket ("<bucket>/*") when no paths are given, or one "<bucket>/<prefix>*" per
// prefix (deduplicated and sorted for determinism).
func objectResources(bucketARN string, paths []string) []string {
if len(paths) == 0 {
return []string{bucketARN + "/*"}
}
seen := map[string]struct{}{}
out := make([]string, 0, len(paths))
for _, p := range paths {
p = strings.TrimPrefix(p, "/")
arn := bucketARN + "/" + p + "*"
if _, dup := seen[arn]; dup {
continue
}
seen[arn] = struct{}{}
out = append(out, arn)
}
sort.Strings(out)
return out
}
// resolveResources renders raw-statement resources: entries that already look
// like ARNs pass through verbatim; bucket-relative prefixes become
// "<bucket>/<prefix>*". An empty list defaults to the whole bucket and objects.
func resolveResources(bucketARN string, resources []string) []string {
if len(resources) == 0 {
return []string{bucketARN, bucketARN + "/*"}
}
out := make([]string, 0, len(resources))
for _, r := range resources {
switch {
case strings.HasPrefix(r, "arn:"):
out = append(out, r)
case r == "" || r == "/":
out = append(out, bucketARN+"/*")
default:
out = append(out, bucketARN+"/"+strings.TrimPrefix(r, "/")+"*")
}
}
return out
}
// buildCondition renders the S3 condition block for a grant, or nil when there
// is nothing to add.
func buildCondition(c *GrantConditions) map[string]map[string][]string {
if c == nil {
return nil
}
cond := map[string]map[string][]string{}
if len(c.SourceIPs) > 0 {
cond["IpAddress"] = map[string][]string{"aws:SourceIp": c.SourceIPs}
}
if c.SecureTransportOnly {
cond["Bool"] = map[string][]string{"aws:SecureTransport": {"true"}}
}
if len(cond) == 0 {
return nil
}
return cond
}
// sid builds a policy statement id that only contains characters S3 accepts,
// prefixed with managedSidPrefix so the operator can recognise its own
// statements when merging into an adopted bucket's policy.
func sid(prefix, uid string) string {
var b strings.Builder
b.WriteString(managedSidPrefix)
b.WriteString(strings.ReplaceAll(prefix, "-", ""))
for _, r := range uid {
switch {
+270
View File
@@ -88,3 +88,273 @@ func TestBuildBucketPolicyStructure(t *testing.T) {
t.Fatal("reader principal ARN missing")
}
}
// parsedPolicy is a fuller parse of a rendered policy for the fine-grained tests.
type parsedPolicy struct {
Statement []struct {
Sid string `json:"Sid"`
Effect string `json:"Effect"`
Principal map[string][]string `json:"Principal"`
Action []string `json:"Action"`
Resource []string `json:"Resource"`
Condition map[string]map[string][]string `json:"Condition"`
} `json:"Statement"`
}
func parsePolicy(t *testing.T, raw string) parsedPolicy {
t.Helper()
var doc parsedPolicy
if err := json.Unmarshal([]byte(raw), &doc); err != nil {
t.Fatalf("policy is not valid JSON: %v\n%s", err, raw)
}
return doc
}
func TestBuildBucketPolicyPaths(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "reader", Level: LevelReadOnly, Paths: []string{"team-a/", "/shared/inbox/"}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
var objResources []string
for _, s := range doc.Statement {
for _, a := range s.Action {
if a == "s3:GetObject" {
objResources = s.Resource
}
}
}
want := map[string]bool{
"arn:aws:s3:::data/shared/inbox/*": false,
"arn:aws:s3:::data/team-a/*": false,
}
if len(objResources) != len(want) {
t.Fatalf("expected %d object resources, got %v", len(want), objResources)
}
for _, r := range objResources {
if _, ok := want[r]; !ok {
t.Fatalf("unexpected object resource %q (leading slash not trimmed?)", r)
}
want[r] = true
}
for r, seen := range want {
if !seen {
t.Fatalf("missing object resource %q", r)
}
}
}
func TestBuildBucketPolicyActionsOverride(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "svc", Level: LevelReadOnly, Actions: []string{"s3:GetObject", "s3:PutObject"}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
if len(doc.Statement) != 1 {
t.Fatalf("expected a single custom-action statement, got %d", len(doc.Statement))
}
s := doc.Statement[0]
if len(s.Action) != 2 || s.Action[0] != "s3:GetObject" || s.Action[1] != "s3:PutObject" {
t.Fatalf("actions not taken verbatim: %v", s.Action)
}
// Custom-action statement lists both the bucket and object resources.
if len(s.Resource) != 2 || s.Resource[0] != "arn:aws:s3:::data" || s.Resource[1] != "arn:aws:s3:::data/*" {
t.Fatalf("unexpected resources: %v", s.Resource)
}
}
func TestBuildBucketPolicyConditions(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "reader", Level: LevelReadOnly, Conditions: &GrantConditions{
SourceIPs: []string{"10.0.0.0/8"},
SecureTransportOnly: true,
}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
for _, s := range doc.Statement {
if s.Condition == nil {
t.Fatalf("statement %q missing condition block", s.Sid)
}
if ip := s.Condition["IpAddress"]["aws:SourceIp"]; len(ip) != 1 || ip[0] != "10.0.0.0/8" {
t.Fatalf("unexpected SourceIp condition: %v", s.Condition["IpAddress"])
}
if tls := s.Condition["Bool"]["aws:SecureTransport"]; len(tls) != 1 || tls[0] != "true" {
t.Fatalf("unexpected SecureTransport condition: %v", s.Condition["Bool"])
}
}
}
func TestBuildBucketPolicyRawStatements(t *testing.T) {
raw, err := BuildBucketPolicy("data", []Grant{
{UID: "svc", Level: LevelReadOnly, Raw: []RawStatement{
{
Effect: "Deny",
Actions: []string{"s3:DeleteObject"},
Resources: []string{"locked/", "arn:aws:s3:::other/*"},
},
}},
})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicy(t, raw)
if len(doc.Statement) != 1 {
t.Fatalf("expected 1 raw statement, got %d", len(doc.Statement))
}
s := doc.Statement[0]
if s.Effect != "Deny" {
t.Fatalf("raw effect not honoured: %q", s.Effect)
}
// The operator fills in the principal; bucket-relative prefixes are expanded
// while explicit ARNs pass through.
if len(s.Principal["AWS"]) != 1 || !strings.HasSuffix(s.Principal["AWS"][0], "user/svc") {
t.Fatalf("raw statement principal not injected: %v", s.Principal)
}
if len(s.Resource) != 2 || s.Resource[0] != "arn:aws:s3:::data/locked/*" || s.Resource[1] != "arn:aws:s3:::other/*" {
t.Fatalf("unexpected raw resources: %v", s.Resource)
}
}
func TestMergeBucketPolicyPreservesForeign(t *testing.T) {
// An existing policy with a foreign statement (unknown Sid, and a scalar
// condition value S3 allows but our typed struct does not model).
existing := `{"Version":"2012-10-17","Statement":[` +
`{"Sid":"AllowPublicRead","Effect":"Allow","Principal":"*","Action":["s3:GetObject"],` +
`"Resource":"arn:aws:s3:::data/public/*","Condition":{"Bool":{"aws:SecureTransport":"true"}}}]}`
merged, err := MergeBucketPolicy(existing, "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
doc := parsePolicyRaw(t, merged)
var sawForeign, sawManaged bool
for _, raw := range doc.Statement {
var s struct {
Sid string `json:"Sid"`
Condition map[string]map[string]any
}
if err := json.Unmarshal(raw, &s); err != nil {
t.Fatalf("statement not valid JSON: %v", err)
}
if s.Sid == "AllowPublicRead" {
sawForeign = true
// The scalar condition value must survive verbatim.
if v := s.Condition["Bool"]["aws:SecureTransport"]; v != "true" {
t.Fatalf("foreign scalar condition mangled: %v", s.Condition)
}
}
if isManagedSid(s.Sid) {
sawManaged = true
}
}
if !sawForeign {
t.Fatal("foreign statement was dropped")
}
if !sawManaged {
t.Fatal("operator statement missing from merge")
}
}
func TestMergeBucketPolicyReplacesManaged(t *testing.T) {
// Two rounds: an existing policy already carrying the operator's statements
// must not accumulate duplicates when re-merged.
first, err := MergeBucketPolicy("", "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
second, err := MergeBucketPolicy(first, "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if first != second {
t.Fatalf("re-merging its own policy was not idempotent:\n first=%s\nsecond=%s", first, second)
}
// A legacy (unprefixed) operator statement must also be recognised and
// replaced rather than preserved as foreign.
legacy := `{"Version":"2012-10-17","Statement":[` +
`{"Sid":"readonlybktreader","Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/reader"]},` +
`"Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::data"]}]}`
merged, err := MergeBucketPolicy(legacy, "data", []Grant{{UID: "reader", Level: LevelReadOnly}})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
for _, raw := range parsePolicyRaw(t, merged).Statement {
var s struct {
Sid string `json:"Sid"`
}
_ = json.Unmarshal(raw, &s)
if s.Sid == "readonlybktreader" {
t.Fatal("legacy operator statement was preserved instead of replaced")
}
}
}
func TestMergeBucketPolicyForeignOnlyKept(t *testing.T) {
existing := `{"Version":"2012-10-17","Statement":[` +
`{"Sid":"AllowPublicRead","Effect":"Allow","Principal":"*","Action":["s3:GetObject"],` +
`"Resource":"arn:aws:s3:::data/*"}]}`
// No grants: the operator adds nothing but must not wipe the foreign policy.
merged, err := MergeBucketPolicy(existing, "data", nil)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if merged == "" {
t.Fatal("merge cleared a policy that had a foreign statement")
}
if len(parsePolicyRaw(t, merged).Statement) != 1 {
t.Fatalf("expected the single foreign statement, got %s", merged)
}
}
func TestMergeBucketPolicyEmpty(t *testing.T) {
merged, err := MergeBucketPolicy("", "data", nil)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if merged != "" {
t.Fatalf("expected empty policy, got %q", merged)
}
}
// parsePolicyRaw parses a policy keeping statements as raw JSON.
func parsePolicyRaw(t *testing.T, raw string) struct {
Version string `json:"Version"`
Statement []json.RawMessage `json:"Statement"`
} {
t.Helper()
var doc struct {
Version string `json:"Version"`
Statement []json.RawMessage `json:"Statement"`
}
if err := json.Unmarshal([]byte(raw), &doc); err != nil {
t.Fatalf("policy is not valid JSON: %v\n%s", err, raw)
}
return doc
}
func TestBuildBucketPolicyFineGrainedDeterministic(t *testing.T) {
grants := []Grant{
{UID: "reader", Level: LevelReadOnly, Paths: []string{"a/", "b/"}},
{UID: "svc", Level: LevelReadWrite, Conditions: &GrantConditions{SecureTransportOnly: true}},
}
a, err := BuildBucketPolicy("data", grants)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
b, err := BuildBucketPolicy("data", []Grant{grants[1], grants[0]})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if a != b {
t.Fatalf("fine-grained policy is order-dependent:\n a=%s\n b=%s", a, b)
}
}
+24 -13
View File
@@ -31,13 +31,15 @@ func (u *User) S3Key() (UserKey, bool) {
return u.Keys[0], true
}
// UserSpec describes the desired state of an RGW user.
// UserSpec describes the desired state of an RGW user. A nil DisplayName/
// Suspended (empty string / nil pointer) leaves that attribute untouched on an
// existing user, so an adopted user is not mutated unless the fields are set.
type UserSpec struct {
UID string
DisplayName string
Email string
MaxBuckets *int32
Suspended bool
Suspended *bool
}
// fromAdminUser converts a go-ceph admin.User into the subset the operator uses.
@@ -73,7 +75,7 @@ func (c *Client) CreateUser(ctx context.Context, spec UserSpec) (*User, error) {
DisplayName: firstNonEmpty(spec.DisplayName, spec.UID),
Email: spec.Email,
MaxBuckets: int32PtrToIntPtr(spec.MaxBuckets),
Suspended: boolToIntPtr(spec.Suspended),
Suspended: boolPtrToIntPtr(spec.Suspended),
GenerateKey: boolPtr(true),
})
if err != nil {
@@ -83,15 +85,20 @@ func (c *Client) CreateUser(ctx context.Context, spec UserSpec) (*User, error) {
return fromAdminUser(u), nil
}
// UpdateUser reconciles the mutable attributes of an existing RGW user.
// UpdateUser reconciles the mutable attributes of an existing RGW user. It only
// sends attributes the spec sets: an empty DisplayName or nil Suspended is left
// as-is, so reconciling (or adopting) a user does not clobber those fields.
func (c *Client) UpdateUser(ctx context.Context, spec UserSpec) (*User, error) {
u, err := c.admin.ModifyUser(ctx, admin.User{
ID: spec.UID,
DisplayName: firstNonEmpty(spec.DisplayName, spec.UID),
Email: spec.Email,
MaxBuckets: int32PtrToIntPtr(spec.MaxBuckets),
Suspended: boolToIntPtr(spec.Suspended),
})
req := admin.User{
ID: spec.UID,
Email: spec.Email,
MaxBuckets: int32PtrToIntPtr(spec.MaxBuckets),
Suspended: boolPtrToIntPtr(spec.Suspended),
}
if spec.DisplayName != "" {
req.DisplayName = spec.DisplayName
}
u, err := c.admin.ModifyUser(ctx, req)
if err != nil {
return nil, err
}
@@ -156,9 +163,13 @@ func int32PtrToIntPtr(p *int32) *int {
return &v
}
func boolToIntPtr(b bool) *int {
// boolPtrToIntPtr renders an optional bool as RGW's 0/1 int, or nil to omit.
func boolPtrToIntPtr(b *bool) *int {
if b == nil {
return nil
}
v := 0
if b {
if *b {
v = 1
}
return &v
+101 -14
View File
@@ -2,6 +2,7 @@ package controller
import (
"context"
"encoding/json"
"fmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
@@ -76,7 +77,11 @@ func (r *BucketReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
}
ownerUID := owner.Status.UID
// Ensure the bucket exists.
// Ensure the bucket exists. Record adoption once: whether the RGW bucket
// already existed the first time we reconciled this resource. status.BucketID
// is only set on a successful reconcile, so a Pending wait on the owner (or a
// transient failure) does not pollute the signal.
firstObserve := b.Status.BucketID == ""
info, err := r.Ceph.GetBucket(ctx, bucketName)
if ceph.IsNotFound(err) {
createSpec := ceph.CreateBucketSpec{
@@ -96,11 +101,29 @@ func (r *BucketReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
return r.fail(ctx, &b, "CreateFailed", err)
}
logger.Info("created bucket", "bucket", bucketName, "owner", ownerUID)
if firstObserve {
b.Status.Adopted = false
}
} else if err != nil {
return r.fail(ctx, &b, "LookupFailed", err)
} else if firstObserve {
b.Status.Adopted = true
logger.Info("adopted existing bucket", "bucket", bucketName, "owner", ownerUID)
}
bucketID := info.InstanceID()
// Placement is fixed at creation: RGW cannot move an existing bucket between
// placement targets. If the live bucket sits on a different target than the
// spec asks for (a changed spec, or an adopted bucket that predates the
// request), surface a clear error instead of ever deleting/recreating it. An
// empty PlacementTarget imposes no constraint.
if pc := placementConflict(b.Spec.PlacementTarget, info.PlacementRule); pc {
b.Status.PlacementTarget = info.PlacementRule
return r.fail(ctx, &b, "PlacementImmutable", fmt.Errorf(
"bucket %q is on placement target %q but spec requests %q; RGW cannot move a bucket between placement targets",
bucketName, info.PlacementRule, b.Spec.PlacementTarget))
}
// Versioning (forced on when object lock is enabled).
if b.Spec.Versioning || (b.Spec.ObjectLock != nil && b.Spec.ObjectLock.Enabled) {
if err := r.Ceph.SetBucketVersioning(ctx, bucketName, bucketID, ownerUID, true); err != nil {
@@ -128,23 +151,35 @@ func (r *BucketReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctr
}
}
// Render and apply the aggregate S3 policy from all BucketAccess grants.
grants, principals, err := r.collectGrants(ctx, b.Namespace, b.Name)
if err != nil {
return r.fail(ctx, &b, "GrantsFailed", err)
}
policy, err := ceph.BuildBucketPolicy(bucketName, grants)
if err != nil {
return r.fail(ctx, &b, "PolicyBuildFailed", err)
}
if err := r.Ceph.SetBucketPolicy(ctx, bucketName, bucketID, ownerUID, policy); err != nil {
return r.fail(ctx, &b, "PolicyFailed", err)
// Render and apply the aggregate S3 policy from all BucketAccess grants,
// unless the bucket opts out of policy management. The merge preserves any
// statements the operator does not own, so an adopted bucket keeps its
// existing policy.
principals := 0
if managePolicy(&b) {
grants, p, err := r.collectGrants(ctx, b.Namespace, b.Name)
if err != nil {
return r.fail(ctx, &b, "GrantsFailed", err)
}
existing, err := r.Ceph.GetBucketPolicy(ctx, bucketName, ownerUID)
if err != nil {
return r.fail(ctx, &b, "PolicyReadFailed", err)
}
policy, err := ceph.MergeBucketPolicy(existing, bucketName, grants)
if err != nil {
return r.fail(ctx, &b, "PolicyBuildFailed", err)
}
if err := r.Ceph.SetBucketPolicy(ctx, bucketName, bucketID, ownerUID, policy); err != nil {
return r.fail(ctx, &b, "PolicyFailed", err)
}
principals = p
}
b.Status.Phase = "Ready"
b.Status.BucketName = bucketName
b.Status.BucketID = bucketID
b.Status.Owner = ownerUID
b.Status.PlacementTarget = info.PlacementRule
b.Status.PolicyPrincipals = int32(principals)
b.Status.ObservedGeneration = b.Generation
setReady(&b.Status.Conditions, b.Generation, true, "Provisioned", "bucket provisioned")
@@ -176,17 +211,69 @@ func (r *BucketReconciler) collectGrants(ctx context.Context, namespace, bucketR
if ba.Status.UID == "" {
continue
}
key := ba.Status.UID + "|" + string(ba.Spec.Level)
g := grantFromAccess(ba.Status.UID, ba)
key := grantKey(g)
if _, dup := seen[key]; dup {
continue
}
seen[key] = struct{}{}
principals[ba.Status.UID] = struct{}{}
grants = append(grants, ceph.Grant{UID: ba.Status.UID, Level: string(ba.Spec.Level)})
grants = append(grants, g)
}
return grants, len(principals), nil
}
// grantFromAccess translates a BucketAccess spec into the ceph grant model,
// carrying the fine-grained scoping (paths, actions, conditions, raw statements).
func grantFromAccess(uid string, ba *v1alpha1.BucketAccess) ceph.Grant {
g := ceph.Grant{
UID: uid,
Level: string(ba.Spec.Level),
Paths: ba.Spec.Paths,
Actions: ba.Spec.Actions,
}
if c := ba.Spec.Conditions; c != nil {
g.Conditions = &ceph.GrantConditions{
SourceIPs: c.SourceIPs,
SecureTransportOnly: c.SecureTransportOnly,
}
}
for _, s := range ba.Spec.RawStatements {
g.Raw = append(g.Raw, ceph.RawStatement{
Sid: s.Sid,
Effect: s.Effect,
Actions: s.Actions,
Resources: s.Resources,
Condition: s.Conditions,
})
}
return g
}
// grantKey is a stable fingerprint of a grant used to collapse duplicate
// BucketAccess objects that would render identical policy statements.
func grantKey(g ceph.Grant) string {
b, _ := json.Marshal(g)
return string(b)
}
// managePolicy reports whether the operator should reconcile this bucket's S3
// policy. A nil ManagePolicy (the CRD default) is treated as true.
func managePolicy(b *v1alpha1.Bucket) bool {
return b.Spec.ManagePolicy == nil || *b.Spec.ManagePolicy
}
// placementConflict reports whether a bucket's live placement target violates
// the spec. An empty spec placement imposes no constraint (the bucket may sit on
// whatever default it was created with). Otherwise the live placement must match
// exactly, since RGW cannot move a bucket between placement targets.
func placementConflict(specPlacement, livePlacement string) bool {
if specPlacement == "" {
return false
}
return specPlacement != livePlacement
}
func (r *BucketReconciler) pending(ctx context.Context, b *v1alpha1.Bucket, reason, msg string) (ctrl.Result, error) {
b.Status.Phase = "Pending"
b.Status.ObservedGeneration = b.Generation
@@ -0,0 +1,27 @@
package controller
import "testing"
func TestPlacementConflict(t *testing.T) {
cases := []struct {
name string
specPlacement string
livePlacement string
want bool
}{
{"unset spec never conflicts", "", "default-placement", false},
{"unset spec unset live", "", "", false},
{"matching ec", "ec", "ec", false},
{"matching default", "default-placement", "default-placement", false},
{"ec requested but default live", "ec", "default-placement", true},
{"default requested but ec live", "default-placement", "ec", true},
{"spec set live empty", "ec", "", true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := placementConflict(tc.specPlacement, tc.livePlacement); got != tc.want {
t.Errorf("placementConflict(%q,%q)=%v want %v", tc.specPlacement, tc.livePlacement, got, tc.want)
}
})
}
}
@@ -49,8 +49,9 @@ func (r *BucketAccessReconciler) Reconcile(ctx context.Context, req ctrl.Request
if !ba.DeletionTimestamp.IsZero() {
if controllerutil.ContainsFinalizer(&ba, finalizer) {
// Only delete a user the operator created for this grant.
if managed && uid != "" {
// Only delete a user the operator created for this grant, and only
// when the grant does not ask to retain it.
if managed && uid != "" && !ba.Spec.RetainOnDelete {
if err := r.Ceph.DeleteUser(ctx, uid); err != nil {
return r.fail(ctx, &ba, "DeleteFailed", err)
}
+89
View File
@@ -0,0 +1,89 @@
package controller
import (
"context"
apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
apiextensionsclient "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/rest"
ctrl "sigs.k8s.io/controller-runtime"
)
// The CRD version check reads CustomResourceDefinitions to compare the
// installed schema against what this operator expects.
//
// +kubebuilder:rbac:groups=apiextensions.k8s.io,resources=customresourcedefinitions,verbs=get;list
// crdSentinel names a CRD and a spec property that only exists in the schema
// version shipped alongside this operator build. If the installed CRD lacks the
// sentinel, its schema predates this operator and strict decoding of new spec
// fields will silently fail. Keep this list in one place so new fields are easy
// to register as sentinels.
type crdSentinel struct {
// crd is the metadata.name of the CustomResourceDefinition.
crd string
// specProperty is a key expected under
// .spec.versions[].schema.openAPIV3Schema.properties.spec.properties.
specProperty string
}
// crdSentinels is the authoritative list checked at startup. Extend it whenever
// a new spec field is added that older CRDs would reject.
var crdSentinels = []crdSentinel{
{crd: "buckets.ceph.unkin.net", specProperty: "managePolicy"},
{crd: "objectstoreusers.ceph.unkin.net", specProperty: "retainOnDelete"},
{crd: "bucketaccesses.ceph.unkin.net", specProperty: "rawStatements"},
}
// CheckCRDVersions verifies that every CRD this operator owns is installed and
// carries the schema fields this build expects. It is advisory only: it logs a
// distinct WARNING per problem and never returns an error or exits, so a stale
// or missing CRD cannot block startup.
func CheckCRDVersions(ctx context.Context, cfg *rest.Config) {
log := ctrl.Log.WithName("crd-version-check")
client, err := apiextensionsclient.NewForConfig(cfg)
if err != nil {
log.Error(err, "unable to build apiextensions client; skipping CRD version check")
return
}
for _, s := range crdSentinels {
crd, err := client.ApiextensionsV1().CustomResourceDefinitions().Get(ctx, s.crd, metav1.GetOptions{})
if apierrors.IsNotFound(err) {
log.Info("WARNING: CRD is not installed — apply the CRDs matching this operator version",
"crd", s.crd)
continue
}
if err != nil {
log.Error(err, "unable to read CRD; cannot verify it matches this operator version",
"crd", s.crd)
continue
}
if !crdHasSpecProperty(crd, s.specProperty) {
log.Info("WARNING: CRD is out of date — apply the CRDs matching this operator version",
"crd", s.crd, "missingField", "spec."+s.specProperty)
}
}
}
// crdHasSpecProperty reports whether any served/stored version of the CRD
// declares the given property under spec.
func crdHasSpecProperty(crd *apiextensionsv1.CustomResourceDefinition, property string) bool {
for _, v := range crd.Spec.Versions {
schema := v.Schema
if schema == nil || schema.OpenAPIV3Schema == nil {
continue
}
specSchema, ok := schema.OpenAPIV3Schema.Properties["spec"]
if !ok {
continue
}
if _, ok := specSchema.Properties[property]; ok {
return true
}
}
return false
}
@@ -40,7 +40,9 @@ func (r *ObjectStoreUserReconciler) Reconcile(ctx context.Context, req ctrl.Requ
if !osu.DeletionTimestamp.IsZero() {
if controllerutil.ContainsFinalizer(&osu, finalizer) {
if err := r.Ceph.DeleteUser(ctx, uid); err != nil {
if osu.Spec.RetainOnDelete {
logger.Info("retaining RGW user on delete", "uid", uid)
} else if err := r.Ceph.DeleteUser(ctx, uid); err != nil {
return r.fail(ctx, &osu, "DeleteFailed", err)
}
controllerutil.RemoveFinalizer(&osu, finalizer)
@@ -65,17 +67,31 @@ func (r *ObjectStoreUserReconciler) Reconcile(ctx context.Context, req ctrl.Requ
Suspended: osu.Spec.Suspended,
}
if _, err := r.Ceph.GetUser(ctx, uid); ceph.IsNotFound(err) {
// Record adoption once: whether the RGW user already existed the first time
// we reconciled this resource (taken over rather than created). status.UID is
// only set on a successful reconcile, so it is a clean "never provisioned"
// signal that transient failures do not pollute.
firstObserve := osu.Status.UID == ""
_, getErr := r.Ceph.GetUser(ctx, uid)
switch {
case ceph.IsNotFound(getErr):
if _, err := r.Ceph.CreateUser(ctx, spec); err != nil {
return r.fail(ctx, &osu, "CreateFailed", err)
}
logger.Info("created RGW user", "uid", uid)
} else if err != nil {
return r.fail(ctx, &osu, "LookupFailed", err)
} else {
if firstObserve {
osu.Status.Adopted = false
}
case getErr != nil:
return r.fail(ctx, &osu, "LookupFailed", getErr)
default:
if _, err := r.Ceph.UpdateUser(ctx, spec); err != nil {
return r.fail(ctx, &osu, "UpdateFailed", err)
}
if firstObserve {
osu.Status.Adopted = true
logger.Info("adopted existing RGW user", "uid", uid)
}
}
if q := osu.Spec.Quota; q != nil {