unkinben
3c2bdf307a
## Why None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for every cluster, plus code-forward restore docs. ## What - **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2, 30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected `vault-ca-cert` (`endpointCA`). - **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is hardcoded**) and a staggered nightly `ScheduledBackup`. - **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can validate the new CRs. - **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`). ## Design decisions (answers to the open questions) **One bucket for all, or per-database?** → **Per-database (one bucket + owner user per cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only *within the same namespace*), and CNPG reads its S3 credential Secret from its *own* namespace. A single shared bucket would require either cross-namespace bucket refs (unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the simplest correct topology and needs zero manual seeding. Each user owns exactly one bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess` grant needed. **Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart `cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree `barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`). ## Schedule / retention (defaults — Ben to adjust) | App | Cluster | Bucket | Nightly base backup | | --- | --- | --- | --- | | authentik | postgres | cnpg-authentik | 01:00 | | litellm | litellm-postgres | cnpg-litellm | 01:20 | | artifactapi | postgres | cnpg-artifactapi | 01:40 | | woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 | | puppet | puppet-postgres | cnpg-puppet | 02:20 | | encapi | postgres | cnpg-encapi | 02:40 | | paperclip | paperclip-postgres | cnpg-paperclip | 03:00 | | grafana | postgres | cnpg-grafana | 03:20 | Retention is **30d** across the board — flagged as a default to tune per cluster. Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once. ## Validation - `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph schemas) pass on all 8 affected overlays (paperclip validated at base — it has no overlay yet). ceph CRs resolve their schemas (`Skipped: 0`). - `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.). - Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated `cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) — pre-existing, reproduces on `origin/main`, unrelated to this change. ## Notes / caveats - No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json` files, which merge cleanly whichever lands first. - Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The only prerequisite is the operator being healthy (it is, in `cephrgw-system`). ## Follow-ups - Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs). - Tune per-cluster retention / schedule if the defaults don't fit. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #298 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
56 lines
2.0 KiB
YAML
56 lines
2.0 KiB
YAML
---
|
|
# Ceph RGW (S3) backup target for the grafana CNPG cluster, provisioned by the
|
|
# in-estate cephrgw-operator. One dedicated bucket + owner user per cluster:
|
|
# cephrgw CRs are namespace-scoped and CNPG reads its S3 credential Secret from
|
|
# its own namespace, so backups are per-database rather than one shared bucket.
|
|
apiVersion: ceph.unkin.net/v1alpha1
|
|
kind: ObjectStoreUser
|
|
metadata:
|
|
name: cnpg-grafana-backup
|
|
namespace: grafana
|
|
spec:
|
|
displayName: "CNPG backup owner (grafana)"
|
|
# RGW users are global; keep the uid namespace-qualified so it never collides.
|
|
uid: cnpg-grafana-backup
|
|
maxBuckets: 5
|
|
# Operator writes AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ RGW_UID,
|
|
# S3_ENDPOINT) into this Secret; the Cluster's barmanObjectStore consumes it.
|
|
secretName: cnpg-grafana-backup-s3
|
|
# Keep the RGW user (and thus the keys) if this CR is ever deleted, so an
|
|
# in-flight restore can still reach the archive.
|
|
retainOnDelete: true
|
|
---
|
|
apiVersion: ceph.unkin.net/v1alpha1
|
|
kind: Bucket
|
|
metadata:
|
|
name: cnpg-grafana
|
|
namespace: grafana
|
|
spec:
|
|
bucketName: cnpg-grafana
|
|
# The owner user has full control of its own bucket (read + write), which is
|
|
# all the backup/restore identity needs — no extra BucketAccess grant.
|
|
ownerRef: cnpg-grafana-backup
|
|
versioning: false
|
|
tags:
|
|
app: grafana
|
|
purpose: cnpg-backup
|
|
# Never drop the backups if the CR is removed; retire buckets by hand.
|
|
retainOnDelete: true
|
|
---
|
|
# Nightly base backup. Continuous WAL archiving is always-on via the Cluster's
|
|
# spec.backup.barmanObjectStore; this schedules the periodic full backup that
|
|
# WAL is layered on top of. Schedules are staggered across clusters so the 8
|
|
# base backups do not hit RGW at once (CNPG cron is 6-field, seconds first).
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: ScheduledBackup
|
|
metadata:
|
|
name: cnpg-grafana-nightly
|
|
namespace: grafana
|
|
spec:
|
|
schedule: "0 20 3 * * *"
|
|
immediate: false
|
|
backupOwnerReference: self
|
|
method: barmanObjectStore
|
|
cluster:
|
|
name: postgres
|