Add S3 backups to all CNPG Postgres clusters (#298)

## Why

None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A
lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable
data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip
and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for
every cluster, plus code-forward restore docs.

## What

- **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous
  WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2,
  30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected
  `vault-ca-cert` (`endpointCA`).
- **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator
  provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is
  hardcoded**) and a staggered nightly `ScheduledBackup`.
- **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can
  validate the new CRs.
- **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`).

## Design decisions (answers to the open questions)

**One bucket for all, or per-database?** → **Per-database (one bucket + owner user per
cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only
*within the same namespace*), and CNPG reads its S3 credential Secret from its *own*
namespace. A single shared bucket would require either cross-namespace bucket refs
(unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator
mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the
simplest correct topology and needs zero manual seeding. Each user owns exactly one
bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess`
grant needed.

**Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart
`cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree
`barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not
deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the
in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`).

## Schedule / retention (defaults — Ben to adjust)

| App | Cluster | Bucket | Nightly base backup |
| --- | --- | --- | --- |
| authentik | postgres | cnpg-authentik | 01:00 |
| litellm | litellm-postgres | cnpg-litellm | 01:20 |
| artifactapi | postgres | cnpg-artifactapi | 01:40 |
| woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 |
| puppet | puppet-postgres | cnpg-puppet | 02:20 |
| encapi | postgres | cnpg-encapi | 02:40 |
| paperclip | paperclip-postgres | cnpg-paperclip | 03:00 |
| grafana | postgres | cnpg-grafana | 03:20 |

Retention is **30d** across the board — flagged as a default to tune per cluster.
Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once.

## Validation

- `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph
  schemas) pass on all 8 affected overlays (paperclip validated at base — it has no
  overlay yet). ceph CRs resolve their schemas (`Skipped: 0`).
- `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.).
- Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated
  `cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) —
  pre-existing, reproduces on `origin/main`, unrelated to this change.

## Notes / caveats

- No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging
  PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json`
  files, which merge cleanly whichever lands first.
- Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The
  only prerequisite is the operator being healthy (it is, in `cephrgw-system`).

## Follow-ups

- Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs).
- Tune per-cluster retention / schedule if the defaults don't fit.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #298
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>

apps/base/authentik/cnpg_backup.yaml

@@ -0,0 +1,55 @@
+---
+# Ceph RGW (S3) backup target for the authentik CNPG cluster, provisioned by the
+# in-estate cephrgw-operator. One dedicated bucket + owner user per cluster:
+# cephrgw CRs are namespace-scoped and CNPG reads its S3 credential Secret from
+# its own namespace, so backups are per-database rather than one shared bucket.
+apiVersion: ceph.unkin.net/v1alpha1
+kind: ObjectStoreUser
+metadata:
+  name: cnpg-authentik-backup
+  namespace: authentik
+spec:
+  displayName: "CNPG backup owner (authentik)"
+  # RGW users are global; keep the uid namespace-qualified so it never collides.
+  uid: cnpg-authentik-backup
+  maxBuckets: 5
+  # Operator writes AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ RGW_UID,
+  # S3_ENDPOINT) into this Secret; the Cluster's barmanObjectStore consumes it.
+  secretName: cnpg-authentik-backup-s3
+  # Keep the RGW user (and thus the keys) if this CR is ever deleted, so an
+  # in-flight restore can still reach the archive.
+  retainOnDelete: true
+---
+apiVersion: ceph.unkin.net/v1alpha1
+kind: Bucket
+metadata:
+  name: cnpg-authentik
+  namespace: authentik
+spec:
+  bucketName: cnpg-authentik
+  # The owner user has full control of its own bucket (read + write), which is
+  # all the backup/restore identity needs — no extra BucketAccess grant.
+  ownerRef: cnpg-authentik-backup
+  versioning: false
+  tags:
+    app: authentik
+    purpose: cnpg-backup
+  # Never drop the backups if the CR is removed; retire buckets by hand.
+  retainOnDelete: true
+---
+# Nightly base backup. Continuous WAL archiving is always-on via the Cluster's
+# spec.backup.barmanObjectStore; this schedules the periodic full backup that
+# WAL is layered on top of. Schedules are staggered across clusters so the 8
+# base backups do not hit RGW at once (CNPG cron is 6-field, seconds first).
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: cnpg-authentik-nightly
+  namespace: authentik
+spec:
+  schedule: "0 0 1 * * *"
+  immediate: false
+  backupOwnerReference: self
+  method: barmanObjectStore
+  cluster:
+    name: postgres

apps/base/authentik/cnpg_cluster.yaml

@@ -7,6 +7,35 @@ metadata:
 spec:
   affinity:
     podAntiAffinityType: preferred
+  backup:
+    # 30-day retention (DEFAULT — adjust per cluster if needed). Enforced by CNPG
+    # against the object store on each successful base backup.
+    retentionPolicy: 30d
+    barmanObjectStore:
+      # Dedicated per-cluster Ceph RGW bucket (cephrgw-operator provisions it).
+      destinationPath: s3://cnpg-authentik
+      endpointURL: https://s3.ceph.unkin.net
+      # radosgw serves a Vault-PKI cert; trust the internal CA (reflected into
+      # every namespace as the vault-ca-cert Secret).
+      endpointCA:
+        name: vault-ca-cert
+        key: ca.crt
+      # Keys minted by the ObjectStoreUser in cnpg_backup.yaml; never hardcoded.
+      s3Credentials:
+        accessKeyId:
+          name: cnpg-authentik-backup-s3
+          key: AWS_ACCESS_KEY_ID
+        secretAccessKey:
+          name: cnpg-authentik-backup-s3
+          key: AWS_SECRET_ACCESS_KEY
+      # Path prefix within the bucket; keep stable across restores (see docs).
+      serverName: authentik
+      data:
+        compression: bzip2
+        jobs: 2
+      wal:
+        compression: zstd
+        maxParallel: 2
   bootstrap:
     initdb:
       database: authentik

apps/base/authentik/kustomization.yaml

@@ -4,6 +4,7 @@ kind: Kustomization
 
 resources:
   - cnpg_cluster.yaml
+  - cnpg_backup.yaml
   - cnpg_pooler.yaml
   - gateway.yaml
   - httproute.yaml