The cluster preloads no statement-statistics library, so there is no per-query cost attribution in postgres and slow query paths have to be inferred from application-side metrics instead of read straight out of the database.
- preload `pg_stat_statements`
- set `pg_stat_statements.max` and `.track`, which is what makes CNPG manage the extension and create it in every database
Requires a postgres restart. Stacked on `benvin/authentik-cnpg-resources`.
Reviewed-on: #475
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The 500m CPU limit is a 50ms CFS quota per 100ms period, so the postgres pods are throttled on every burst even at ~0.01 cores average and each query pays that latency. 128MB of `shared_buffers` and a 256MB cache estimate also under-serve the planner on the joins authentik issues on its hot read paths.
- raise resources to requests `500m`/`1Gi`, limits `2`/`2Gi`
- raise `shared_buffers` to 512MB and `effective_cache_size` to 1536MB
- hold the post-incident memory headroom multiple over `shared_buffers`
Rolling restart with switchover. Stacked on `benvin/authentik-hot-standby-feedback`.
Reviewed-on: #474
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Authentik serves multi-second API reads from the CNPG hot standbys. Those reads outlive `max_standby_streaming_delay`, so recovery cancels them with `canceling statement due to conflict with recovery`, which authentik surfaces as HTTP 500 — enough to break a terraform apply mid-run.
- set `hot_standby_feedback` on so replicas report their oldest xmin to the primary and long reads stop being cancelled
- SIGHUP reload only, no restart or switchover
- retained-dead-tuple cost is negligible on a ~155MB database
Reviewed-on: #473
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Authentik's intermittent API 500s (which failed two terraform-authentik CI runs today) were traced to its CNPG postgres replicas being OOMKilled: 512Mi limits leave no headroom over shared_buffers 128MB + max_connections 200, both OOM events matched the 500 bursts to the second, and the session-pinned RO pooler turns each replica death into a batch of severed read connections. The primary is at 84% of its limit and is next.
- raise the authentik CNPG memory limit from 512Mi to 1Gi and request from 256Mi to 512Mi
Follow-up candidates (not in this PR): RO pooler poolMode session→transaction to shrink the blast radius of a replica loss; revisit max_connections/shared_buffers sizing.
Reviewed-on: #300
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A
lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable
data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip
and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for
every cluster, plus code-forward restore docs.
## What
- **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous
WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2,
30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected
`vault-ca-cert` (`endpointCA`).
- **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator
provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is
hardcoded**) and a staggered nightly `ScheduledBackup`.
- **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can
validate the new CRs.
- **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`).
## Design decisions (answers to the open questions)
**One bucket for all, or per-database?** → **Per-database (one bucket + owner user per
cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only
*within the same namespace*), and CNPG reads its S3 credential Secret from its *own*
namespace. A single shared bucket would require either cross-namespace bucket refs
(unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator
mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the
simplest correct topology and needs zero manual seeding. Each user owns exactly one
bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess`
grant needed.
**Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart
`cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree
`barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not
deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the
in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`).
## Schedule / retention (defaults — Ben to adjust)
| App | Cluster | Bucket | Nightly base backup |
| --- | --- | --- | --- |
| authentik | postgres | cnpg-authentik | 01:00 |
| litellm | litellm-postgres | cnpg-litellm | 01:20 |
| artifactapi | postgres | cnpg-artifactapi | 01:40 |
| woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 |
| puppet | puppet-postgres | cnpg-puppet | 02:20 |
| encapi | postgres | cnpg-encapi | 02:40 |
| paperclip | paperclip-postgres | cnpg-paperclip | 03:00 |
| grafana | postgres | cnpg-grafana | 03:20 |
Retention is **30d** across the board — flagged as a default to tune per cluster.
Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once.
## Validation
- `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph
schemas) pass on all 8 affected overlays (paperclip validated at base — it has no
overlay yet). ceph CRs resolve their schemas (`Skipped: 0`).
- `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.).
- Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated
`cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) —
pre-existing, reproduces on `origin/main`, unrelated to this change.
## Notes / caveats
- No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging
PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json`
files, which merge cleanly whichever lands first.
- Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The
only prerequisite is the operator being healthy (it is, in `cephrgw-system`).
## Follow-ups
- Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs).
- Tune per-cluster retention / schedule if the defaults don't fit.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #298
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben