## Why
None of the 8 CNPG Postgres clusters in this repo had **any** backup configured. A
lost PVC, a fat-fingered migration, or a bad app deploy meant permanent, unrecoverable
data loss for authentik, litellm, artifactapi, woodpecker, puppet, encapi, paperclip
and grafana. This adds continuous WAL archiving + a nightly base backup to Ceph RGW for
every cluster, plus code-forward restore docs.
## What
- **`spec.backup.barmanObjectStore`** on each `cnpg_cluster.yaml` — turns on continuous
WAL archiving to `s3://cnpg-<app>`, WAL compressed with zstd, base backups with bzip2,
30-day retention. TLS to `s3.ceph.unkin.net` is trusted via the reflected
`vault-ca-cert` (`endpointCA`).
- **`cnpg_backup.yaml`** per app — a cephrgw `ObjectStoreUser` + `Bucket` (operator
provisions the bucket and mints the S3 key into `cnpg-<app>-backup-s3`; **nothing is
hardcoded**) and a staggered nightly `ScheduledBackup`.
- **`schemas/ceph.unkin.net/*.json`** — the three cephrgw CRD schemas so kubeconform can
validate the new CRs.
- **`docs/`** — new docs folder (README index + `cnpg-backups.md` + `cnpg-restore.md`).
## Design decisions (answers to the open questions)
**One bucket for all, or per-database?** → **Per-database (one bucket + owner user per
cluster).** The cephrgw CRDs are namespace-scoped (`BucketRef`/`OwnerRef` resolve only
*within the same namespace*), and CNPG reads its S3 credential Secret from its *own*
namespace. A single shared bucket would require either cross-namespace bucket refs
(unsupported) or hand-copying the S3 secret into all 8 namespaces (defeats "operator
mints the keys"). Per-namespace `s3://cnpg-<app>` with a dedicated owner user is the
simplest correct topology and needs zero manual seeding. Each user owns exactly one
bucket, so owner-level (full) access is already tightly scoped — no extra `BucketAccess`
grant needed.
**Backup mechanism.** The deployed CNPG operator is **v1.28** (helm chart
`cloudnative-pg-0.27.0`, appVersion 1.28.0). 1.26+ deprecates the in-tree
`barmanObjectStore` in favour of the Barman Cloud Plugin, but the plugin is **not
deployed**, and `barmanObjectStore` is still fully functional on 1.28. So this uses the
in-tree mechanism. Migrating to the plugin is a follow-up (noted in `docs/cnpg-backups.md`).
## Schedule / retention (defaults — Ben to adjust)
| App | Cluster | Bucket | Nightly base backup |
| --- | --- | --- | --- |
| authentik | postgres | cnpg-authentik | 01:00 |
| litellm | litellm-postgres | cnpg-litellm | 01:20 |
| artifactapi | postgres | cnpg-artifactapi | 01:40 |
| woodpecker | woodpecker-postgres | cnpg-woodpecker | 02:00 |
| puppet | puppet-postgres | cnpg-puppet | 02:20 |
| encapi | postgres | cnpg-encapi | 02:40 |
| paperclip | paperclip-postgres | cnpg-paperclip | 03:00 |
| grafana | postgres | cnpg-grafana | 03:20 |
Retention is **30d** across the board — flagged as a default to tune per cluster.
Schedules are staggered 20 min apart so 8 base backups don't hit RGW at once.
## Validation
- `kustomize build --enable-helm` + `kubeconform` (repo CI args, incl. the new ceph
schemas) pass on all 8 affected overlays (paperclip validated at base — it has no
overlay yet). ceph CRs resolve their schemas (`Skipped: 0`).
- `pre-commit run` passes on all changed files (yamllint, no-plain-secrets, etc.).
- Note: a full `ci/validate-apps.sh` run aborts locally on the unrelated
`cattle-system` overlay (`chart requires kubeVersion < 1.35 vs host helm v1.36.0`) —
pre-existing, reproduces on `origin/main`, unrelated to this change.
## Notes / caveats
- No overlap with the woodpecker chart-bump PR (#297, overlay files only) or the logging
PR (#296) beyond the three **identical** generated `schemas/ceph.unkin.net/*.json`
files, which merge cleanly whichever lands first.
- Credentials: no manual seeding — the cephrgw-operator mints the RGW user + keys. The
only prerequisite is the operator being healthy (it is, in `cephrgw-system`).
## Follow-ups
- Barman Cloud Plugin migration (deploy plugin, move clusters to `ObjectStore` CRs).
- Tune per-cluster retention / schedule if the defaults don't fit.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #298
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Turns on right-sizing telemetry for the whole estate. Adds a `VerticalPodAutoscaler` with `updateMode: "Off"` (recommendation-only, advise mode) for every Deployment and StatefulSet in `apps/base`. Off mode never evicts or mutates pods, so this is purely observational: the VPA recommender (added in the vpa-system PR) publishes suggested requests/limits in each VPA's status, and nothing acts on them until someone deliberately flips a mode.
## Changes
- Add one `vpa.yaml` per app under `apps/base/<app>/` containing a `<workload>-vpa` VerticalPodAutoscaler for each workload, and register it in that app's `kustomization.yaml`.
- Coverage: 17 workloads across 11 apps.
- age-api (age-api), artifactapi (api, redis, ui), authentik (redis), bind-system (bind-operator), cephrgw-system (cephrgw-operator), encapi (encapi), kanidm (kanidm StatefulSet), litellm (litellm, redis), paperclip (paperclip), pdbmux (pdbmux), puppet (puppetboard, puppetdb, puppetserver-compiler, puppetserver-master).
## Skipped (intentionally)
- **CNPG `Cluster` objects** (artifactapi, authentik, encapi, grafana, litellm, paperclip, puppet, woodpecker) — Postgres is managed by CloudNativePG, not a VPA target.
- **CronJobs** (puppet g10k/generate-types, reposync x4) — not VPA-able.
## HPA / VPA caveat
api, ui (artifactapi), litellm, and all four puppet deployments also carry an HPA. With `updateMode: "Off"` there is no conflict today (VPA only recommends). VPA objects targeting these carry an inline comment: do **not** flip to `Auto`/`Initial` while the HPA still autoscales on CPU/memory, or the two controllers will fight over the same resource. Move the HPA to a custom/non-resource metric first.
## Verification
- `kubectl kustomize` over every touched overlay: 9/11 overlays PASS rendering all their VPAs; 2 failures are pre-existing and unrelated to this change — `authentik` fails on a local helm-tooling flag error identically on origin/main, and `paperclip` has no `apps/overlays/au-syd1/paperclip` directory yet. Both apps' base kustomizations build clean and render their VPA.
## Merge gate
- **Requires the vpa-system CRDs PR (argocd-apps #281) to merge first.** These manifests use `autoscaling.k8s.io/v1 VerticalPodAutoscaler`; without the CRDs installed, ArgoCD sync fails on an unknown kind.
Reviewed-on: #283
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the two new UI features released in v3.7.7: per-repo usage-instruction panels (artifactapi#105) and direct-download links for local repo files (artifactapi#106).
- Bumps artifactapi api image to git.unkin.net/unkin/artifactapi:v3.7.7
- Bumps artifactapi ui image to git.unkin.net/unkin/artifactapi-ui:v3.7.7
Both v3.7.7 images are confirmed published to the registry; kustomize builds clean.
Reviewed-on: #278
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Roll out artifactapi `v3.7.5`, which ships the local docker registry (artifactapi#103): local `docker` repos now serve the Docker Registry HTTP API V2 for push and pull.
## Changes
- `apps/base/artifactapi/api-deployment.yaml`: `artifactapi` image `v3.7.4` → `v3.7.5`
- `apps/base/artifactapi/ui-deployment.yaml`: `artifactapi-ui` image `v3.7.4` → `v3.7.5`
## Heads-up (follow-up needed)
The API HPA runs `minReplicas: 2`. Local-docker **chunked** blob uploads keep the upload session in-memory per replica, so a real `docker push` (POST → PATCH → PUT across replicas, no session affinity) can intermittently 404 with `BLOB_UPLOAD_UNKNOWN`. Monolithic pushes are unaffected. Recommend a follow-up to make upload sessions replica-independent (S3-backed) or add session affinity for `/v2/*/blobs/uploads/` before relying on pushes in anger.
Reviewed-on: #231
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
artifactapi `v3.7.4` images are built and pushed; au-syd1 is on `v3.7.3`. This rolls forward to ship the terraform provider registry.
## Changes
- `api-deployment`: `artifactapi` `v3.7.3` → `v3.7.4`
- `ui-deployment`: `artifactapi-ui` `v3.7.3` → `v3.7.4`
## What's new in v3.7.4
- Local terraform repos are now a real provider registry: `/.well-known/terraform.json` + `providers.v1` versions/download with GPG-signed SHA256SUMS (#102).
- The signing key self-provisions in the DB (`signing_keys` table) — no K8s secret to mount, so no deployment wiring needed.
Once synced, `terraform init` against `source = "artifactapi.k8s.syd1.au.unkin.net/<repo>/<type>"` works.
Reviewed-on: #218
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
artifactapi images \`v3.7.3\` are built and pushed to the registry, but au-syd1 is still running \`v3.6.5\`. This rolls the deployment forward to pick up the recent fixes.
## Changes
- \`api-deployment\`: \`artifactapi\` \`v3.6.5\` → \`v3.7.3\`
- \`ui-deployment\`: \`artifactapi-ui\` \`v3.6.5\` → \`v3.7.3\`
Included in v3.7.x since v3.6.5:
- Local-repo files now appear in the cached-objects UI (#99).
- Evicting a local RPM prunes its repodata metadata (#100).
- The bare domain redirects to the web UI at /ui (#101).
Reviewed-on: #215
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Fixes helm chart URL path duplication for same-host repos (stakater).
Reviewed-on: #207
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Includes Docker Accept header forwarding, Content-Type fix, nginx base path fix, and version endpoint fix.
Reviewed-on: #206
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Includes Docker Bearer token auth (#60) and UI BASE_PATH build_args fix (#59).
Reviewed-on: #205
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Rebuilds UI with BASE_PATH=/ui so assets serve under /ui/.
Reviewed-on: #204
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Bumps API and UI images from v3.5.0 to v3.6.0.
Reviewed-on: #203
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The UI now serves under /ui (artifactapi#58). Health probes need /ui instead of /.
Reviewed-on: #202
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Route /ui → UI service, everything else → API service.
Replaces the growing list of per-prefix rules (/api, /v2, /health) with a single catch-all to the API. No more needing to add a route rule every time the API adds a new top-level path.
Reviewed-on: #201
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The v3 route migration (#198) split routes into /api → API and / → UI, but /v2/ (Docker Registry V2 API) and /health now hit the UI catch-all instead of the API backend.
This breaks `docker pull artifactapi.k8s.syd1.au.unkin.net/...` with context deadline exceeded.
Adds /v2 and /health prefix rules before the UI catch-all.
Reviewed-on: #200
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
update the environment secret reference to match what has been
deployed. this prevents a containerconfigerror
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #199
What changed:
- Adds new v3 API and UI deployments (separate api-deployment.yaml, ui-deployment.yaml) alongside the existing monolithic artifactapi-deployment.yaml
- Adds CNPG PostgreSQL cluster + pooler to replace the standalone postgres deployment
- Adds new api-env configmap, new Vault secrets (postgres-credentials, environment), and a second VaultAuth (default1)
- Adds new services targeting the split api and ui selectors
- Adds HPAs for both new deployments
- Updates kustomization to include all new resources
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #197
attempted to let claude deploy a new version of artifactory with
terrible results. this change is to remove that mess so I can start
again.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #196
just-enough to test terraform deployment and begin migration. have
change to cnpg for the database and a new bucket for storage
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #192
Part of #155 (prerequisite for open-webui deployment PR #172).
## Summary
- Adds `^open-webui/open-webui` to the `ghcr` remote's `immutable_patterns` in `remote-docker.yaml` so version-pinned open-webui image pulls are cached indefinitely through artifactapi
## Test plan
- artifactapi serves `ghcr.io/open-webui/open-webui:<version>` with `X-Artifact-Source: cache` on second fetch
Reviewed-on: #173
## Summary
- Adds `open-policy-agent/conftest/.*/conftest_.*_Linux_x86_64.tar.gz$` to the `github` remote immutable patterns in artifactapi
## Why
conftest v0.68.2 (https://github.com/open-policy-agent/conftest/releases/tag/v0.68.2) is now used for OPA policy checks in CI (see #167). Caching the release tarball in artifactapi reduces external dependency on GitHub during builds.
Reviewed-on: #168
## Summary
- Adds `group: gateway.networking.k8s.io` and `kind: Gateway` to all `parentRefs` entries
- Adds `group: ""`, `kind: Service`, and `weight: 1` to all `backendRefs` entries
- Affects 9 HTTPRoute files across artifactapi, cattle-system, consul, kanidm, litellm, paperclip, puppet, and vault
## Why
ArgoCD diffs the desired manifest against the live Kubernetes object. The Gateway API controller defaults these fields when creating/updating objects, so the live state always has them — causing persistent OutOfSync for every HTTPRoute. Same root cause as #153 (certificateRefs).
## Test plan
- [ ] All affected ArgoCD applications show Synced after merge
Reviewed-on: #162
## Summary
- Removes `^kanidm/` from the `ghcr` remote immutable_patterns
- Adds `^kanidm/` to the `dockerhub` remote immutable_patterns
## Why
`kanidm/server` is published on Docker Hub, not ghcr.io. Pulling via the `ghcr` cache was failing with 403 on anonymous token fetch → 502 Bad Gateway.
## Test plan
- [ ] `docker pull artifactapi.k8s.syd1.au.unkin.net/dockerhub/kanidm/server:1.10.3` succeeds after artifactapi redeploys
Reviewed-on: #160
## Summary
- Adds \`stalwartlabs/webadmin/releases/latest/download/webadmin.zip\` to \`mutable_patterns\` in the \`github\` generic remote so the stalwart webadmin UI can be fetched through artifactapi rather than directly from GitHub.
## Notes
- Uses \`mutable_patterns\` (not \`immutable\`) because \`releases/latest\` resolves to whichever release is current and changes over time.
- Access URL: \`https://artifactapi.k8s.syd1.au.unkin.net/generic/github/stalwartlabs/webadmin/releases/latest/download/webadmin.zip\`
Reviewed-on: #157
The Gateway API admission server defaults certificateRefs[].group to ""
when it is omitted. ArgoCD diffed the desired state (no group field) against
the live state (group: "") and flagged every gateway as out of sync.
Fix: explicitly set group: "" in all certificateRefs entries so the
rendered manifest matches the API server's canonical form exactly.
Affected: artifactapi, cattle-system, consul, litellm, paperclip,
puppet (puppetboard + puppetdb), vault.
Reviewed-on: #153
## Summary
- Adds \`^hashicorp/consul\` and \`^hashicorp/vault\` to the dockerhub immutable_patterns in artifactapi's remote-docker.yaml
- Replaces the more specific \`^hashicorp/vault-secrets-operator\` pattern since \`^hashicorp/vault\` subsumes it
- Required for the benvin/vault and benvin/consul branches (vault:2.0.1 and consul:1.22.7)
## Test plan
- [ ] Verify artifactapi accepts requests for hashicorp/vault and hashicorp/consul images after merge
Reviewed-on: #147
Add port 80 HTTP listener and redirect HTTPRoute to artifactapi,
cattle-system (rancher), litellm, paperclip, and puppetboard — restoring
the redirect behaviour that existed on the previous nginx/traefik Ingress
resources.
Reviewed-on: #145
## Summary
- Replace `Ingress` (nginx) with `Gateway` + `HTTPRoute` using `traefik-internal` GatewayClass
- TLS terminated at the Gateway listener; cert-manager provisions the certificate via `vault-issuer`
- external-dns annotations moved to the Gateway
## Notes
The original Ingress had nginx-specific annotations (`proxy-body-size: 10g`, `proxy-read-timeout: 600`) which are not portable to Gateway API. These can be re-introduced via a Traefik `Middleware` CRD if needed.
## Test plan
- [ ] ArgoCD syncs the app cleanly
- [ ] cert-manager issues the `artifactapi-tls` certificate
- [ ] external-dns creates the DNS record
- [ ] `https://artifactapi.k8s.syd1.au.unkin.net` is reachable
Reviewed-on: #129
Adds immutable patterns for yannh/kubeconform and kubernetes-sigs/kustomize
to fix 403 Forbidden errors when downloading their Linux amd64 releases.
Reviewed-on: #123
Split monolithic remotes.yaml into per-type-package files under
resources/conf.d/ to align with artifactapi v2.7.1 directory loading.
Updated schema: virtuals/locals use dedicated top-level keys, type field
removed. Added helm remotes for all kustomize helmCharts repos and
OCI patterns to docker remotes. CONFIG_PATH now points to the directory.
Reviewed-on: #92
unkinben
benvin