The public authentik gateway (identity.unkin.net) now uses traefik-external
and removes external-dns management, matching the arrproxy external gateway
convention. DNS remains managed out-of-band via bind-operator. The internal
authentik-internal gateway stays on traefik-internal.
The single authentik Gateway kept cert-manager.io annotations (vault-issuer,
common-name identity.k8s.syd1.au.unkin.net) while its public identity.unkin.net
https listener referenced the reflector-managed wildcard-unkin-net-tls. The
cert-manager gateway-shim mints a Certificate per distinct listener secretName
from Gateway-level annotations, so it would create a competing vault-issuer
Certificate for wildcard-unkin-net-tls and fight the emberstack reflector over
that secret.
Split into two Gateways mirroring the cheeztv/fafflix convention:
- authentik (public): identity.unkin.net listeners on wildcard-unkin-net-tls,
no cert-manager annotations, external-dns publishes identity.unkin.net.
- authentik-internal: identity.k8s.syd1.au.unkin.net listeners on authentik-tls
with the vault-issuer cert-manager annotations, external-dns publishes the
cluster host. Both target 198.18.200.4.
Update HTTPRoutes to attach each hostname's redirect + app route to the
matching gateway's http/https sections.
why: after #417 stood up the central Let's Encrypt *.unkin.net wildcard
(Certificate wildcard-unkin-net, secret wildcard-unkin-net-tls, emberstack-
reflected), the remaining single-label public hosts should terminate on the
real publicly-trusted leaf instead of the internal Vault-PKI CA. A *.unkin.net
wildcard only matches single-label subdomains, so cluster hostnames under
*.k8s.syd1.au.unkin.net stay on vault-issuer.
- certificate_wildcard-unkin-net: reflect the secret into arrstack, authentik
and gitea (append to both reflector namespace lists)
- arrstack: point the public arrproxy gateway (arrstack.unkin.net) at
wildcard-unkin-net-tls and drop its vault-issuer annotations
- authentik: point the identity.unkin.net listener at wildcard-unkin-net-tls;
the identity.k8s.syd1.au.unkin.net listeners keep authentik-tls, so the
gateway common-name is retargeted to the cluster host
- gitea: secret reflected only; git.unkin.net cutover deferred (no manifest
change), same deferred pattern as fafflix
## Why
No redis/valkey instance managed in this repo exports metrics today — the six caches (artifactapi, authentik, litellm, gitea, netbox, jellyfin) are blind spots in VictoriaMetrics.
## What
- Enables the valkey-operator exporter sidecar on the jellyfin ValkeyCluster (`exporter.enabled: true`); overrides the operator's default bare-dockerhub image (`oliver006/redis_exporter:v1.80.0`) with the artifactapi-proxied pin. The operator manages a dedicated `_exporter` ACL user for it.
- Adds a `metrics-exporter` sidecar (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/oliver006/redis_exporter:v1.89.0`, :9121, port name `metrics`, req 32Mi/25m lim 64Mi/100m) to the five raw Deployments: artifactapi/authentik/litellm redis and gitea/netbox valkey. gitea/netbox sidecars get the same restrictive securityContext as their valkey container.
- Adds the `monitoring.unkin.net/redis-exporter: "true"` opt-in pod label to all five pod templates.
- Adds `apps/base/observability/vmpodscrape-redis.yaml` with two VMPodScrapes (any namespace, port `metrics`): `redis-exporters` selecting the opt-in label, and `valkey-operator-clusters` selecting `app.kubernetes.io/managed-by: valkey-operator` + `app.kubernetes.io/component: valkey-node` — the ValkeyCluster CR has no pod-label passthrough (verified against the v0.5.0 CRD/source), so operator pods cannot carry the opt-in label.
## Notes
- No instance runs with `requirepass`/auth, so no REDIS_PASSWORD wiring is needed on the raw Deployments; the jellyfin exporter auth is operator-managed.
- Validated: `kustomize build` on every touched base + au-syd1 overlay (helm overlays with `--enable-helm`) and `kubeconform -strict -ignore-missing-schemas` (68 valid, 0 invalid).
Reviewed-on: #398
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Final step of the ec migration: the old buckets were purged in #315, so the operator will now create fresh ones on the ec placement target. Restores the nine Bucket CRs with placementTarget: ec and retainOnDelete: true (purge disabled again for safety).
- re-add the nine cnpg backup Bucket resources on ec, retainOnDelete: true
Reviewed-on: #316
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Step 2 of the ec placement migration: with purge-on-delete now allowed (#313), removing the Bucket CRs makes the operator delete the underlying RGW buckets and their objects, freeing the names to be recreated on ec in step 3.
- remove the nine cnpg backup Bucket resources
- keep ObjectStoreUser and ScheduledBackup so the backup-s3 secrets and schedules survive
Reviewed-on: #315
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Step 1 of moving the CNPG backup buckets to ec placement: RGW can't move an existing bucket, so they must be dropped and recreated. This lets the operator actually delete the buckets (with their objects) when the CRs are removed in step 2.
- set retainOnDelete: false and purgeOnDelete: true on all nine cnpg backup Bucket CRs
- leave ObjectStoreUser/BucketAccess untouched so the backup-s3 secrets persist
Reviewed-on: #313
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
cephrgw-operator v0.4.0 adds immutable placementTarget selection on Buckets; the nine CNPG backup buckets should live on the ec (4/1) placement instead of 3-replica. Existing buckets cannot change placement, so after this merges the buckets get deleted and recreated on ec and fresh base backups are triggered (day-old backups are accepted losses, per Ben).
- bump cephrgw-operator image and CRD ref to v0.4.0
- add placementTarget: ec to all nine cnpg backup Bucket CRs
Reviewed-on: #312
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Authentik's intermittent API 500s (which failed two terraform-authentik CI runs today) were traced to its CNPG postgres replicas being OOMKilled: 512Mi limits leave no headroom over shared_buffers 128MB + max_connections 200, both OOM events matched the 500 bursts to the second, and the session-pinned RO pooler turns each replica death into a batch of severed read connections. The primary is at 84% of its limit and is next.
- raise the authentik CNPG memory limit from 512Mi to 1Gi and request from 256Mi to 512Mi
Follow-up candidates (not in this PR): RO pooler poolMode session→transaction to shrink the blast radius of a replica loss; revisit max_connections/shared_buffers sizing.
Reviewed-on: #300
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A
lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable
data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip
and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for
every cluster, plus code-forward restore docs.
## What
- **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous
WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2,
30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected
`vault-ca-cert` (`endpointCA`).
- **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator
provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is
hardcoded**) and a staggered nightly `ScheduledBackup`.
- **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can
validate the new CRs.
- **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`).
## Design decisions (answers to the open questions)
**One bucket for all, or per-database?** → **Per-database (one bucket + owner user per
cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only
*within the same namespace*), and CNPG reads its S3 credential Secret from its *own*
namespace. A single shared bucket would require either cross-namespace bucket refs
(unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator
mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the
simplest correct topology and needs zero manual seeding. Each user owns exactly one
bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess`
grant needed.
**Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart
`cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree
`barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not
deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the
in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`).
## Schedule / retention (defaults — Ben to adjust)
| App | Cluster | Bucket | Nightly base backup |
| --- | --- | --- | --- |
| authentik | postgres | cnpg-authentik | 01:00 |
| litellm | litellm-postgres | cnpg-litellm | 01:20 |
| artifactapi | postgres | cnpg-artifactapi | 01:40 |
| woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 |
| puppet | puppet-postgres | cnpg-puppet | 02:20 |
| encapi | postgres | cnpg-encapi | 02:40 |
| paperclip | paperclip-postgres | cnpg-paperclip | 03:00 |
| grafana | postgres | cnpg-grafana | 03:20 |
Retention is **30d** across the board — flagged as a default to tune per cluster.
Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once.
## Validation
- `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph
schemas) pass on all 8 affected overlays (paperclip validated at base — it has no
overlay yet). ceph CRs resolve their schemas (`Skipped: 0`).
- `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.).
- Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated
`cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) —
pre-existing, reproduces on `origin/main`, unrelated to this change.
## Notes / caveats
- No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging
PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json`
files, which merge cleanly whichever lands first.
- Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The
only prerequisite is the operator being healthy (it is, in `cephrgw-system`).
## Follow-ups
- Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs).
- Tune per-cluster retention / schedule if the defaults don't fit.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #298
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Turns on right-sizing telemetry for the whole estate. Adds a `VerticalPodAutoscaler` with `updateMode: "Off"` (recommendation-only, advise mode) for every Deployment and StatefulSet in `apps/base`. Off mode never evicts or mutates pods, so this is purely observational: the VPA recommender (added in the vpa-system PR) publishes suggested requests/limits in each VPA's status, and nothing acts on them until someone deliberately flips a mode.
## Changes
- Add one `vpa.yaml` per app under `apps/base/<app>/` containing a `<workload>-vpa` VerticalPodAutoscaler for each workload, and register it in that app's `kustomization.yaml`.
- Coverage: 17 workloads across 11 apps.
- age-api (age-api), artifactapi (api, redis, ui), authentik (redis), bind-system (bind-operator), cephrgw-system (cephrgw-operator), encapi (encapi), kanidm (kanidm StatefulSet), litellm (litellm, redis), paperclip (paperclip), pdbmux (pdbmux), puppet (puppetboard, puppetdb, puppetserver-compiler, puppetserver-master).
## Skipped (intentionally)
- **CNPG `Cluster` objects** (artifactapi, authentik, encapi, grafana, litellm, paperclip, puppet, woodpecker) — Postgres is managed by CloudNativePG, not a VPA target.
- **CronJobs** (puppet g10k/generate-types, reposync x4) — not VPA-able.
## HPA / VPA caveat
api, ui (artifactapi), litellm, and all four puppet deployments also carry an HPA. With `updateMode: "Off"` there is no conflict today (VPA only recommends). VPA objects targeting these carry an inline comment: do **not** flip to `Auto`/`Initial` while the HPA still autoscales on CPU/memory, or the two controllers will fight over the same resource. Move the HPA to a custom/non-resource metric first.
## Verification
- `kubectl kustomize` over every touched overlay: 9/11 overlays PASS rendering all their VPAs; 2 failures are pre-existing and unrelated to this change — `authentik` fails on a local helm-tooling flag error identically on origin/main, and `paperclip` has no `apps/overlays/au-syd1/paperclip` directory yet. Both apps' base kustomizations build clean and render their VPA.
## Merge gate
- **Requires the vpa-system CRDs PR (argocd-apps #281) to merge first.** These manifests use `autoscaling.k8s.io/v1 VerticalPodAutoscaler`; without the CRDs installed, ArgoCD sync fails on an unknown kind.
Reviewed-on: #283
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben
benvin