Add S3 backups to all CNPG Postgres clusters

None of the CNPG clusters had any backup configured, so a lost PVC or a bad
migration meant permanent data loss. This adds continuous WAL archiving plus a
nightly base backup to Ceph RGW for every cluster, with restore docs.

- Add spec.backup.barmanObjectStore (in-tree; operator is CNPG 1.28, Barman
  Cloud Plugin not deployed) to each cnpg_cluster.yaml: WAL zstd, base bzip2,
  30-day retention, endpointCA via the reflected vault-ca-cert.
- Add cnpg_backup.yaml per app: a cephrgw ObjectStoreUser + Bucket (one
  dedicated s3://cnpg-<app> bucket and owner user per cluster, since cephrgw CRs
  and the CNPG credential Secret are namespace-scoped) and a staggered nightly
  ScheduledBackup. Credentials are minted by the operator; nothing is hardcoded.
- Add the three ceph.unkin.net CRD schemas so kubeconform can validate the CRs.
- Add docs/ (README index, cnpg-backups.md, cnpg-restore.md) covering config and
  full/PITR restore procedures.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

apps/base/artifactapi/cnpg_backup.yaml

@@ -0,0 +1,55 @@
+---
+# Ceph RGW (S3) backup target for the artifactapi CNPG cluster, provisioned by the
+# in-estate cephrgw-operator. One dedicated bucket + owner user per cluster:
+# cephrgw CRs are namespace-scoped and CNPG reads its S3 credential Secret from
+# its own namespace, so backups are per-database rather than one shared bucket.
+apiVersion: ceph.unkin.net/v1alpha1
+kind: ObjectStoreUser
+metadata:
+  name: cnpg-artifactapi-backup
+  namespace: artifactapi
+spec:
+  displayName: "CNPG backup owner (artifactapi)"
+  # RGW users are global; keep the uid namespace-qualified so it never collides.
+  uid: cnpg-artifactapi-backup
+  maxBuckets: 5
+  # Operator writes AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ RGW_UID,
+  # S3_ENDPOINT) into this Secret; the Cluster's barmanObjectStore consumes it.
+  secretName: cnpg-artifactapi-backup-s3
+  # Keep the RGW user (and thus the keys) if this CR is ever deleted, so an
+  # in-flight restore can still reach the archive.
+  retainOnDelete: true
+---
+apiVersion: ceph.unkin.net/v1alpha1
+kind: Bucket
+metadata:
+  name: cnpg-artifactapi
+  namespace: artifactapi
+spec:
+  bucketName: cnpg-artifactapi
+  # The owner user has full control of its own bucket (read + write), which is
+  # all the backup/restore identity needs — no extra BucketAccess grant.
+  ownerRef: cnpg-artifactapi-backup
+  versioning: false
+  tags:
+    app: artifactapi
+    purpose: cnpg-backup
+  # Never drop the backups if the CR is removed; retire buckets by hand.
+  retainOnDelete: true
+---
+# Nightly base backup. Continuous WAL archiving is always-on via the Cluster's
+# spec.backup.barmanObjectStore; this schedules the periodic full backup that
+# WAL is layered on top of. Schedules are staggered across clusters so the 8
+# base backups do not hit RGW at once (CNPG cron is 6-field, seconds first).
+apiVersion: postgresql.cnpg.io/v1
+kind: ScheduledBackup
+metadata:
+  name: cnpg-artifactapi-nightly
+  namespace: artifactapi
+spec:
+  schedule: "0 40 1 * * *"
+  immediate: false
+  backupOwnerReference: self
+  method: barmanObjectStore
+  cluster:
+    name: postgres

apps/base/artifactapi/cnpg_cluster.yaml

@@ -7,6 +7,35 @@ metadata:
 spec:
   affinity:
     podAntiAffinityType: preferred
+  backup:
+    # 30-day retention (DEFAULT — adjust per cluster if needed). Enforced by CNPG
+    # against the object store on each successful base backup.
+    retentionPolicy: 30d
+    barmanObjectStore:
+      # Dedicated per-cluster Ceph RGW bucket (cephrgw-operator provisions it).
+      destinationPath: s3://cnpg-artifactapi
+      endpointURL: https://s3.ceph.unkin.net
+      # radosgw serves a Vault-PKI cert; trust the internal CA (reflected into
+      # every namespace as the vault-ca-cert Secret).
+      endpointCA:
+        name: vault-ca-cert
+        key: ca.crt
+      # Keys minted by the ObjectStoreUser in cnpg_backup.yaml; never hardcoded.
+      s3Credentials:
+        accessKeyId:
+          name: cnpg-artifactapi-backup-s3
+          key: AWS_ACCESS_KEY_ID
+        secretAccessKey:
+          name: cnpg-artifactapi-backup-s3
+          key: AWS_SECRET_ACCESS_KEY
+      # Path prefix within the bucket; keep stable across restores (see docs).
+      serverName: artifactapi
+      data:
+        compression: bzip2
+        jobs: 2
+      wal:
+        compression: zstd
+        maxParallel: 2
   bootstrap:
     initdb:
       database: artifacts

apps/base/artifactapi/kustomization.yaml

@@ -7,6 +7,7 @@ resources:
   - api-hpa.yaml
   - configmap.yaml
   - cnpg_cluster.yaml
+  - cnpg_backup.yaml
   - cnpg_pooler.yaml
   - gateway.yaml
   - httproute.yaml