unkinben
3c2bdf307a
## Why None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for every cluster, plus code-forward restore docs. ## What - **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2, 30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected `vault-ca-cert` (`endpointCA`). - **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is hardcoded**) and a staggered nightly `ScheduledBackup`. - **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can validate the new CRs. - **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`). ## Design decisions (answers to the open questions) **One bucket for all, or per-database?** → **Per-database (one bucket + owner user per cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only *within the same namespace*), and CNPG reads its S3 credential Secret from its *own* namespace. A single shared bucket would require either cross-namespace bucket refs (unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the simplest correct topology and needs zero manual seeding. Each user owns exactly one bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess` grant needed. **Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart `cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree `barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`). ## Schedule / retention (defaults — Ben to adjust) | App | Cluster | Bucket | Nightly base backup | | --- | --- | --- | --- | | authentik | postgres | cnpg-authentik | 01:00 | | litellm | litellm-postgres | cnpg-litellm | 01:20 | | artifactapi | postgres | cnpg-artifactapi | 01:40 | | woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 | | puppet | puppet-postgres | cnpg-puppet | 02:20 | | encapi | postgres | cnpg-encapi | 02:40 | | paperclip | paperclip-postgres | cnpg-paperclip | 03:00 | | grafana | postgres | cnpg-grafana | 03:20 | Retention is **30d** across the board — flagged as a default to tune per cluster. Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once. ## Validation - `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph schemas) pass on all 8 affected overlays (paperclip validated at base — it has no overlay yet). ceph CRs resolve their schemas (`Skipped: 0`). - `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.). - Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated `cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) — pre-existing, reproduces on `origin/main`, unrelated to this change. ## Notes / caveats - No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json` files, which merge cleanly whichever lands first. - Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The only prerequisite is the operator being healthy (it is, in `cephrgw-system`). ## Follow-ups - Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs). - Tune per-cluster retention / schedule if the defaults don't fit. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv Reviewed-on: #298 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
88 lines
2.3 KiB
YAML
88 lines
2.3 KiB
YAML
---
|
|
apiVersion: postgresql.cnpg.io/v1
|
|
kind: Cluster
|
|
metadata:
|
|
name: postgres
|
|
namespace: grafana
|
|
spec:
|
|
affinity:
|
|
podAntiAffinityType: preferred
|
|
backup:
|
|
# 30-day retention (DEFAULT — adjust per cluster if needed). Enforced by CNPG
|
|
# against the object store on each successful base backup.
|
|
retentionPolicy: 30d
|
|
barmanObjectStore:
|
|
# Dedicated per-cluster Ceph RGW bucket (cephrgw-operator provisions it).
|
|
destinationPath: s3://cnpg-grafana
|
|
endpointURL: https://s3.ceph.unkin.net
|
|
# radosgw serves a Vault-PKI cert; trust the internal CA (reflected into
|
|
# every namespace as the vault-ca-cert Secret).
|
|
endpointCA:
|
|
name: vault-ca-cert
|
|
key: ca.crt
|
|
# Keys minted by the ObjectStoreUser in cnpg_backup.yaml; never hardcoded.
|
|
s3Credentials:
|
|
accessKeyId:
|
|
name: cnpg-grafana-backup-s3
|
|
key: AWS_ACCESS_KEY_ID
|
|
secretAccessKey:
|
|
name: cnpg-grafana-backup-s3
|
|
key: AWS_SECRET_ACCESS_KEY
|
|
# Path prefix within the bucket; keep stable across restores (see docs).
|
|
serverName: grafana
|
|
data:
|
|
compression: bzip2
|
|
jobs: 2
|
|
wal:
|
|
compression: zstd
|
|
maxParallel: 2
|
|
bootstrap:
|
|
initdb:
|
|
database: grafana
|
|
encoding: UTF8
|
|
localeCType: C
|
|
localeCollate: C
|
|
owner: grafana
|
|
secret:
|
|
name: postgres-credentials
|
|
enablePDB: true
|
|
enableSuperuserAccess: false
|
|
failoverDelay: 0
|
|
imageName: ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie
|
|
instances: 2
|
|
logLevel: info
|
|
monitoring:
|
|
customQueriesConfigMap:
|
|
- key: queries
|
|
name: cnpg-default-monitoring
|
|
disableDefaultQueries: false
|
|
enablePodMonitor: false
|
|
postgresql:
|
|
parameters:
|
|
max_connections: "200"
|
|
shared_buffers: 128MB
|
|
primaryUpdateMethod: restart
|
|
primaryUpdateStrategy: unsupervised
|
|
replicationSlots:
|
|
highAvailability:
|
|
enabled: true
|
|
slotPrefix: _cnpg_
|
|
synchronizeReplicas:
|
|
enabled: true
|
|
updateInterval: 30
|
|
resources:
|
|
limits:
|
|
cpu: 500m
|
|
memory: 512Mi
|
|
requests:
|
|
cpu: 50m
|
|
memory: 256Mi
|
|
smartShutdownTimeout: 180
|
|
startDelay: 3600
|
|
stopDelay: 1800
|
|
storage:
|
|
resizeInUseVolumes: true
|
|
size: 10Gi
|
|
storageClass: cephrbd-fast-delete
|
|
switchoverDelay: 3600
|