argocd-apps/docs/cnpg-backups.md

CNPG backups

Every CNPG Postgres cluster in this repo backs up to a dedicated Ceph RGW (S3) bucket: continuous WAL archiving plus a staggered nightly base backup, 30-day retention, compressed. Buckets and their S3 keys are provisioned by the in-estate cephrgw-operator — nothing is seeded by hand. Because the operator's CRs are namespace-scoped and CNPG reads its credential Secret from its own namespace, the topology is one bucket per cluster (s3://cnpg-<app>), not one shared bucket.

The backup mechanism is CNPG's in-tree barmanObjectStore (the deployed operator is v1.28; the Barman Cloud Plugin is not deployed — see the migration note at the bottom).

Where it lives

Per cluster, two files under apps/base/<app>/:

• cnpg_cluster.yaml — the spec.backup stanza (WAL archiving + retention).
• cnpg_backup.yaml — the ObjectStoreUser + Bucket (RGW provisioning) and the nightly ScheduledBackup.

The backup stanza (spec.backup in the Cluster)

spec:
  backup:
    retentionPolicy: 30d              # default; adjust per cluster
    barmanObjectStore:
      destinationPath: s3://cnpg-<app>
      endpointURL: https://s3.ceph.unkin.net
      endpointCA:                     # trust the internal Vault PKI CA
        name: vault-ca-cert           # reflected into every namespace
        key: ca.crt
      s3Credentials:
        accessKeyId:
          name: cnpg-<app>-backup-s3  # minted by the ObjectStoreUser
          key: AWS_ACCESS_KEY_ID
        secretAccessKey:
          name: cnpg-<app>-backup-s3
          key: AWS_SECRET_ACCESS_KEY
      serverName: <app>               # path prefix inside the bucket
      data:
        compression: bzip2            # base backup (zstd not supported here)
        jobs: 2
      wal:
        compression: zstd             # WAL segments
        maxParallel: 2

Setting spec.backup.barmanObjectStore turns on continuous WAL archiving immediately (CNPG points archive_command at the object store). The nightly base backup is a separate object:

The nightly base backup (ScheduledBackup)

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: cnpg-<app>-nightly
  namespace: <app>
spec:
  schedule: "0 0 1 * * *"   # 6-field cron, SECONDS first (01:00:00 daily)
  immediate: false
  backupOwnerReference: self
  method: barmanObjectStore
  cluster:
    name: <cluster-name>

Schedules are staggered so the base backups don't hit RGW at once:

App	Cluster	Bucket	Nightly (local)
authentik	postgres	cnpg-authentik	01:00
litellm	litellm-postgres	cnpg-litellm	01:20
artifactapi	postgres	cnpg-artifactapi	01:40
woodpecker	woodpecker-postgres	cnpg-woodpecker	02:00
puppet	puppet-postgres	cnpg-puppet	02:20
encapi	postgres	cnpg-encapi	02:40
paperclip	paperclip-postgres	cnpg-paperclip	03:00
grafana	postgres	cnpg-grafana	03:20

Where the credentials come from

The ObjectStoreUser in cnpg_backup.yaml tells cephrgw-operator to mint an RGW user and write its keys into a Secret; the Bucket makes that user the bucket owner (full read/write on its own bucket). No keys are ever committed.

apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
  name: cnpg-<app>-backup
  namespace: <app>
spec:
  uid: cnpg-<app>-backup           # RGW users are global; keep it unique
  secretName: cnpg-<app>-backup-s3 # -> AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
  retainOnDelete: true
---
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
  name: cnpg-<app>
  namespace: <app>
spec:
  bucketName: cnpg-<app>
  ownerRef: cnpg-<app>-backup
  retainOnDelete: true

Confirm the operator provisioned everything:

kubectl -n <app> get objectstoreuser,bucket
kubectl -n <app> get secret cnpg-<app>-backup-s3 \
  -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d; echo

Checking backup status

# Cluster health + first-recoverability point (WAL archiving working?)
kubectl -n <app> get cluster <cluster-name> \
  -o jsonpath='{.status.firstRecoverabilityPoint}{"\n"}'

# Backups taken so far
kubectl -n <app> get backups.postgresql.cnpg.io

# With the cnpg kubectl plugin (richer view, shows archiving + last backup)
kubectl cnpg status <cluster-name> -n <app>

# Kick a one-off backup right now (verify the whole path end to end)
kubectl cnpg backup <cluster-name> -n <app>

# Look at what actually landed in the bucket
aws --endpoint-url https://s3.ceph.unkin.net s3 ls s3://cnpg-<app>/<app>/

Follow-up: Barman Cloud Plugin migration

CNPG 1.26+ deprecates the in-tree barmanObjectStore in favour of the Barman Cloud Plugin (removal is planned for a future major). The plugin is not deployed today, so this repo stays on the in-tree mechanism, which is fully functional on 1.28. Migrating means deploying the plugin and moving each cluster's config to an ObjectStore CR + plugins: reference — track that as separate work.