## Why
None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A
lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable
data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip
and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for
every cluster, plus code-forward restore docs.
## What
- **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous
WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2,
30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected
`vault-ca-cert` (`endpointCA`).
- **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator
provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is
hardcoded**) and a staggered nightly `ScheduledBackup`.
- **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can
validate the new CRs.
- **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`).
## Design decisions (answers to the open questions)
**One bucket for all, or per-database?** → **Per-database (one bucket + owner user per
cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only
*within the same namespace*), and CNPG reads its S3 credential Secret from its *own*
namespace. A single shared bucket would require either cross-namespace bucket refs
(unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator
mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the
simplest correct topology and needs zero manual seeding. Each user owns exactly one
bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess`
grant needed.
**Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart
`cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree
`barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not
deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the
in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`).
## Schedule / retention (defaults — Ben to adjust)
| App | Cluster | Bucket | Nightly base backup |
| --- | --- | --- | --- |
| authentik | postgres | cnpg-authentik | 01:00 |
| litellm | litellm-postgres | cnpg-litellm | 01:20 |
| artifactapi | postgres | cnpg-artifactapi | 01:40 |
| woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 |
| puppet | puppet-postgres | cnpg-puppet | 02:20 |
| encapi | postgres | cnpg-encapi | 02:40 |
| paperclip | paperclip-postgres | cnpg-paperclip | 03:00 |
| grafana | postgres | cnpg-grafana | 03:20 |
Retention is **30d** across the board — flagged as a default to tune per cluster.
Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once.
## Validation
- `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph
schemas) pass on all 8 affected overlays (paperclip validated at base — it has no
overlay yet). ceph CRs resolve their schemas (`Skipped: 0`).
- `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.).
- Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated
`cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) —
pre-existing, reproduces on `origin/main`, unrelated to this change.
## Notes / caveats
- No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging
PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json`
files, which merge cleanly whichever lands first.
- Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The
only prerequisite is the operator being healthy (it is, in `cephrgw-system`).
## Follow-ups
- Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs).
- Tune per-cluster retention / schedule if the defaults don't fit.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #298
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Turns on right-sizing telemetry for the whole estate. Adds a `VerticalPodAutoscaler` with `updateMode: "Off"` (recommendation-only, advise mode) for every Deployment and StatefulSet in `apps/base`. Off mode never evicts or mutates pods, so this is purely observational: the VPA recommender (added in the vpa-system PR) publishes suggested requests/limits in each VPA's status, and nothing acts on them until someone deliberately flips a mode.
## Changes
- Add one `vpa.yaml` per app under `apps/base/<app>/` containing a `<workload>-vpa` VerticalPodAutoscaler for each workload, and register it in that app's `kustomization.yaml`.
- Coverage: 17 workloads across 11 apps.
- age-api (age-api), artifactapi (api, redis, ui), authentik (redis), bind-system (bind-operator), cephrgw-system (cephrgw-operator), encapi (encapi), kanidm (kanidm StatefulSet), litellm (litellm, redis), paperclip (paperclip), pdbmux (pdbmux), puppet (puppetboard, puppetdb, puppetserver-compiler, puppetserver-master).
## Skipped (intentionally)
- **CNPG `Cluster` objects** (artifactapi, authentik, encapi, grafana, litellm, paperclip, puppet, woodpecker) — Postgres is managed by CloudNativePG, not a VPA target.
- **CronJobs** (puppet g10k/generate-types, reposync x4) — not VPA-able.
## HPA / VPA caveat
api, ui (artifactapi), litellm, and all four puppet deployments also carry an HPA. With `updateMode: "Off"` there is no conflict today (VPA only recommends). VPA objects targeting these carry an inline comment: do **not** flip to `Auto`/`Initial` while the HPA still autoscales on CPU/memory, or the two controllers will fight over the same resource. Move the HPA to a custom/non-resource metric first.
## Verification
- `kubectl kustomize` over every touched overlay: 9/11 overlays PASS rendering all their VPAs; 2 failures are pre-existing and unrelated to this change — `authentik` fails on a local helm-tooling flag error identically on origin/main, and `paperclip` has no `apps/overlays/au-syd1/paperclip` directory yet. Both apps' base kustomizations build clean and render their VPA.
## Merge gate
- **Requires the vpa-system CRDs PR (argocd-apps #281) to merge first.** These manifests use `autoscaling.k8s.io/v1 VerticalPodAutoscaler`; without the CRDs installed, ArgoCD sync fails on an unknown kind.
Reviewed-on: #283
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Pairs with terraform-authentik#10: request the litellm_role scope (Authentik
emits the computed role claim) and read it via GENERIC_USER_ROLE_ATTRIBUTE so
akP-litellm-admin -> proxy_admin, akP-litellm-user -> internal_user.
App-side of the LiteLLM Authentik onboarding (terraform-authentik#8). Configures
LiteLLM's generic OIDC SSO against Authentik.
- VaultStaticSecret oauth-credentials: surfaces the OIDC client secret (same
secret Authentik sets on the provider) as a k8s Secret.
- Deployment: GENERIC_CLIENT_SECRET from that Secret.
- litellm-env: GENERIC_CLIENT_ID, authorization/token/userinfo endpoints, scope,
and PROXY_BASE_URL (required for SSO). reloader restarts on secret/config change.
## Summary
- Replace `Ingress` (nginx) with `Gateway` + `HTTPRoute` using `traefik-internal` GatewayClass
- TLS terminated at the Gateway listener; cert-manager provisions the certificate via `vault-issuer`
- external-dns annotations moved to the Gateway
## Test plan
- [ ] ArgoCD syncs the litellm app cleanly
- [ ] cert-manager issues the `litellm-tls` certificate
- [ ] external-dns creates the DNS record
- [ ] `https://litellm.k8s.syd1.au.unkin.net` is reachable
Reviewed-on: #134
Deploys LiteLLM proxy with CNPG PostgreSQL (3-instance HA), PgBouncer
pooler, and Redis cache. Introduces a dedicated aitooling AppProject and
ApplicationSet to keep AI tooling services separate from platform infra.
Reviewed-on: #94
unkinben