Implement autobackup-operator (PVC + CNPG controllers, tests, CI) #1

2026-08-14T00:09:16+10:00

unkin-agent commented

2026-08-14 00:09:16 +10:00

Annotation-driven backups (backups.unkin.net/schedule + destination) for PVCs and CloudNativePG Clusters need their S3 bucket and backup-schedule resources provisioned consistently, with no new CRDs. This adds the initial operator.

How:

Adds a PVC controller: provisions cephrgw ObjectStoreUser/Bucket/BucketAccess, auto-generates a restic repo-password Secret, and creates a k8up Schedule scoped to the PVC via spec.backup.volumes[].persistentVolumeClaim.claimName.
Adds a CNPG Cluster controller: provisions the same bucket stack, idempotently patches spec.backup.barmanObjectStore (leaves a user-set destinationPath alone with a Warning event) and creates a ScheduledBackup.
Resolves destinations via a ConfigMap lookup table, requeues until BucketAccess is Ready before creating schedule resources, owner-references created resources and retains bucket data by default.
Adds schedule-mapping helpers (k8up 5-field/shortcut pass-through; CNPG 6-field seconds-first) and deterministic length-bounded name derivation.
Adds unit tests (schedule mapping, name derivation, destination resolution) and envtest controller tests for both paths.
Adds kubebuilder-generated RBAC, a distroless/nonroot Dockerfile, Woodpecker lint/test/build pipelines and a tag-triggered image push to artifactapi docker-internal, a version-bump Makefile, and deploy manifests.

Notes:

Uses typed cephrgw + k8up APIs; CNPG is handled as unstructured to avoid vendoring its heavy module graph (barman-cloud, cnpg-i/gRPC, prometheus-operator).
Dependencies are vendored so the container build and CI need no access to the private cephrgw-operator module.
CI steps set serviceAccountName autobackup-operator-ci (needs a matching ServiceAccount added in argocd-apps).

Annotation-driven backups (backups.unkin.net/schedule + destination) for PVCs and CloudNativePG Clusters need their S3 bucket and backup-schedule resources provisioned consistently, with no new CRDs. This adds the initial operator. How: - Adds a PVC controller: provisions cephrgw ObjectStoreUser/Bucket/BucketAccess, auto-generates a restic repo-password Secret, and creates a k8up Schedule scoped to the PVC via spec.backup.volumes[].persistentVolumeClaim.claimName. - Adds a CNPG Cluster controller: provisions the same bucket stack, idempotently patches spec.backup.barmanObjectStore (leaves a user-set destinationPath alone with a Warning event) and creates a ScheduledBackup. - Resolves destinations via a ConfigMap lookup table, requeues until BucketAccess is Ready before creating schedule resources, owner-references created resources and retains bucket data by default. - Adds schedule-mapping helpers (k8up 5-field/shortcut pass-through; CNPG 6-field seconds-first) and deterministic length-bounded name derivation. - Adds unit tests (schedule mapping, name derivation, destination resolution) and envtest controller tests for both paths. - Adds kubebuilder-generated RBAC, a distroless/nonroot Dockerfile, Woodpecker lint/test/build pipelines and a tag-triggered image push to artifactapi docker-internal, a version-bump Makefile, and deploy manifests. Notes: - Uses typed cephrgw + k8up APIs; CNPG is handled as unstructured to avoid vendoring its heavy module graph (barman-cloud, cnpg-i/gRPC, prometheus-operator). - Dependencies are vendored so the container build and CI need no access to the private cephrgw-operator module. - CI steps set serviceAccountName autobackup-operator-ci (needs a matching ServiceAccount added in argocd-apps).

unkin-agent added 1 commit 2026-08-14 00:09:17 +10:00

Implement autobackup-operator controllers, tests, CI and packaging 9da206dc7c

PVCs and CloudNativePG Clusters need S3 buckets and backup schedules
provisioned consistently. This operator watches the
backups.unkin.net/{schedule,destination} annotations on those objects and
provisions everything needed to back them up, with no new CRDs.

- Add a PVC controller that provisions cephrgw ObjectStoreUser/Bucket/BucketAccess,
  auto-generates a restic repo-password Secret and creates a k8up Schedule scoped
  to the PVC via spec.backup.volumes[].persistentVolumeClaim.claimName.
- Add a CNPG Cluster controller that provisions the same bucket stack, idempotently
  patches spec.backup.barmanObjectStore (leaving a user-set destinationPath alone
  with a Warning event) and creates a ScheduledBackup.
- Resolve destinations through a ConfigMap lookup table; requeue until the
  BucketAccess is Ready before creating schedule resources; own-reference created
  resources and retain bucket data by default.
- Add schedule-mapping helpers (k8up 5-field/shortcut pass-through, CNPG 6-field
  seconds-first) and deterministic, length-bounded name derivation.
- Add unit tests (schedule mapping, name derivation, destination resolution) and
  envtest controller tests for both paths, wiring the external CRDs into envtest.
- Add kubebuilder-generated RBAC, a Dockerfile (distroless/nonroot), Woodpecker
  lint/test/build pipelines and a tag-triggered image push to the artifactapi
  docker-internal registry, plus a version-bump Makefile and deploy manifests.

unkin-agent commented

2026-08-14 00:17:47 +10:00

Review (advisory; not merging). Code is correct against the real target CRDs (k8up Schedule backend.s3/repoPasswordSecretRef/backup.volumes[].persistentVolumeClaim.claimName; CNPG barmanObjectStore + retentionPolicy at spec.backup.retentionPolicy; ScheduledBackup cluster.name/schedule/method/backupOwnerReference all verified against schemas), build+vet clean, and unit + envtest controller tests pass and genuinely assert the wiring (incl. requeue-until-Ready gating and the non-clobber destinationPath conflict path with event). A few findings:

HIGH — leader election will crash on startup. config/manager/manager.yaml runs with --leader-elect (LeaderElectionID autobackup-operator), but the RBAC grants no coordination.k8s.io/leases permission and there is no leader-election Role anywhere. controller-runtime`s default lease lock will be forbidden creating/updating its Lease and the manager will exit. Add the standard kubebuilder leader-election Role (leases: get;list;watch;create;update;patch;delete + events) bound to the SA in the operator namespace, or add those verbs to the ClusterRole.

MED — Go toolchain mismatch. go.mod says go 1.26.5 but the CI steps use golang:1.25 and Dockerfile.operator uses golang:1.25-alpine. With GOTOOLCHAIN=auto this silently downloads the 1.26.5 toolchain at build/test time (breaking the Dockerfile`s explicit "no network needed" hermetic claim and making the image pins moot); with no egress it fails. Bump the CI + Dockerfile base images to golang:1.26, or relax the go directive.

MED — tag release push has no registry credentials. .woodpecker/docker.yaml pushes to artifactapi.k8s.syd1.au.unkin.net/docker-internal with no username/password/secret. Confirm anonymous in-cluster push is allowed, otherwise wire a Vault/woodpecker push secret.

LOW — PVC controller has no annotation predicate, so every unannotated PVC in the cluster triggers a reconcile + a NotFound-ignored Schedule delete each resync (minor churn). Cron validator also rejects alphabetic month/day names (MON/JAN) that k8up + robfig/cron accept.

Atomicity: fine as a single bootstrap — both controllers share the bucket-stack/destinations/naming machinery and cant be meaningfully split; the huge diff is almost entirely vendor/. Not a split candidate.

Review (advisory; not merging). Code is correct against the real target CRDs (k8up Schedule backend.s3/repoPasswordSecretRef/backup.volumes[].persistentVolumeClaim.claimName; CNPG barmanObjectStore + retentionPolicy at spec.backup.retentionPolicy; ScheduledBackup cluster.name/schedule/method/backupOwnerReference all verified against schemas), build+vet clean, and unit + envtest controller tests pass and genuinely assert the wiring (incl. requeue-until-Ready gating and the non-clobber destinationPath conflict path with event). A few findings: HIGH — leader election will crash on startup. `config/manager/manager.yaml` runs with `--leader-elect` (LeaderElectionID `autobackup-operator`), but the RBAC grants no `coordination.k8s.io/leases` permission and there is no leader-election Role anywhere. controller-runtime`s default lease lock will be forbidden creating/updating its Lease and the manager will exit. Add the standard kubebuilder leader-election Role (leases: get;list;watch;create;update;patch;delete + events) bound to the SA in the operator namespace, or add those verbs to the ClusterRole. MED — Go toolchain mismatch. go.mod says `go 1.26.5` but the CI steps use `golang:1.25` and Dockerfile.operator uses `golang:1.25-alpine`. With GOTOOLCHAIN=auto this silently downloads the 1.26.5 toolchain at build/test time (breaking the Dockerfile`s explicit "no network needed" hermetic claim and making the image pins moot); with no egress it fails. Bump the CI + Dockerfile base images to golang:1.26, or relax the go directive. MED — tag release push has no registry credentials. `.woodpecker/docker.yaml` pushes to artifactapi.k8s.syd1.au.unkin.net/docker-internal with no username/password/secret. Confirm anonymous in-cluster push is allowed, otherwise wire a Vault/woodpecker push secret. LOW — PVC controller has no annotation predicate, so every unannotated PVC in the cluster triggers a reconcile + a NotFound-ignored Schedule delete each resync (minor churn). Cron validator also rejects alphabetic month/day names (MON/JAN) that k8up + robfig/cron accept. Atomicity: fine as a single bootstrap — both controllers share the bucket-stack/destinations/naming machinery and cant be meaningfully split; the huge diff is almost entirely vendor/. Not a split candidate.

unkin-agent added 1 commit 2026-08-14 00:34:28 +10:00

Fix leader-election RBAC, Go version drift, PVC churn, cron names, apply order

ci/woodpecker/push/test Pipeline was successful

Details

ci/woodpecker/pr/build Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/test Pipeline was successful

Details

a3339a30b5

Addresses PR review + acceptance findings:

- Add leader-election RBAC: namespaced Role (coordination leases +
  events) and RoleBinding to the operator ServiceAccount, so controllers
  actually start under --leader-elect instead of looping on "leases
  forbidden". Verified in a kind cluster: lease acquired, both
  controllers start workers.
- Align Go versions: bump Dockerfile.operator to golang:1.26-alpine and
  CI images to golang:1.26 to match go.mod (go 1.26.5); set
  GOTOOLCHAIN=local so the image build stays hermetic (no toolchain
  download).
- PVC controller: add an annotation predicate so only PVCs carrying (or
  transitioning off of) backups.unkin.net/schedule enqueue, eliminating
  reconcile churn from unannotated PVCs while keeping the teardown path.
- Cron validator: accept alphabetic month/day-of-week names (MON, JAN,
  MON-FRI) that k8up and CNPG's robfig/cron accept, still rejecting
  unknown names; add unit tests for both mapping paths.
- Deploy ordering: extract the Namespace into its own manifest and add a
  config/kustomization.yaml so `kubectl apply -k config` creates the
  namespace first; document it in the README.
- Note the credential-less push precedent (jellyfin-ha) in docker.yaml.

unkin-agent commented

2026-08-14 00:34:47 +10:00

Pushed fixes for the review + acceptance findings (head a3339a3):

Leader-election RBAC (HIGH): added a namespaced Role (coordination.k8s.io/leases + events) and RoleBinding to the operator SA (config/rbac/leader_election_role*.yaml). Verified in a kind cluster with --leader-elect on: lease acquired, both controllers Starting workers, no more leases ... forbidden.
Go version drift (HIGH): Dockerfile.operator → golang:1.26-alpine, CI images → golang:1.26 to match go.mod (go 1.26.5), plus GOTOOLCHAIN=local so the image build is hermetic. docker build -f Dockerfile.operator . succeeds with no toolchain download.
Tag push creds (MED): verified against jellyfin-ha — it pushes credential-less to the same artifactapi docker-internal registry (in-cluster runner has push access). Kept the approach; added a comment noting the precedent.
PVC reconcile churn (LOW): added an annotation predicate so only PVCs carrying (or transitioning off) backups.unkin.net/schedule enqueue; teardown-on-removal still fires. Unit test added.
Cron validator (LOW): now accepts alphabetic month/day names (MON, JAN, MON-FRI) that k8up/robfig accept, still rejecting unknown names; tests added for both the k8up and CNPG paths.
Apply order (MINOR): extracted the Namespace into its own manifest and added config/kustomization.yaml (Namespace first); kubectl apply -k config now works clean. Documented in README.

go build/go vet/gofmt/go test -race ./... (incl. envtest) all green.

Pushed fixes for the review + acceptance findings (head a3339a3): - **Leader-election RBAC (HIGH):** added a namespaced Role (`coordination.k8s.io/leases` + events) and RoleBinding to the operator SA (`config/rbac/leader_election_role*.yaml`). Verified in a kind cluster with `--leader-elect` on: lease acquired, both controllers `Starting workers`, no more `leases ... forbidden`. - **Go version drift (HIGH):** `Dockerfile.operator` → `golang:1.26-alpine`, CI images → `golang:1.26` to match `go.mod` (go 1.26.5), plus `GOTOOLCHAIN=local` so the image build is hermetic. `docker build -f Dockerfile.operator .` succeeds with no toolchain download. - **Tag push creds (MED):** verified against jellyfin-ha — it pushes credential-less to the same artifactapi `docker-internal` registry (in-cluster runner has push access). Kept the approach; added a comment noting the precedent. - **PVC reconcile churn (LOW):** added an annotation predicate so only PVCs carrying (or transitioning off) `backups.unkin.net/schedule` enqueue; teardown-on-removal still fires. Unit test added. - **Cron validator (LOW):** now accepts alphabetic month/day names (`MON`, `JAN`, `MON-FRI`) that k8up/robfig accept, still rejecting unknown names; tests added for both the k8up and CNPG paths. - **Apply order (MINOR):** extracted the Namespace into its own manifest and added `config/kustomization.yaml` (Namespace first); `kubectl apply -k config` now works clean. Documented in README. `go build`/`go vet`/`gofmt`/`go test -race ./...` (incl. envtest) all green.

benvin merged commit 1ddaf14d57 into main

2026-08-14 20:31:42 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/autobackup-operator#1