Use full postgres image on minimal CNPG clusters so WAL archiving works #304

2026-07-28T22:22:16+10:00

unkinben commented

2026-07-28 22:22:16 +10:00

Why

The CNPG buckets are empty after PR #298. Live diagnosis found two independent
causes; this PR fixes one of them.

litellm, puppet and paperclip run the CNPG 17-minimal-trixie image, which
omits the barman-cloud CLI. In-tree barmanObjectStore archiving shells out to
barman-cloud-wal-archive, so their archiver dies immediately:

ContinuousArchiving=False :: unexpected failure invoking barman-cloud-wal-archive:
exec: "barman-cloud-wal-archive": executable file not found in $PATH

(verified on the live puppet-postgres primary: which barman-cloud-wal-archive →
not found; on a -system cluster it resolves to /usr/local/bin/barman-cloud-wal-archive).

How

Switch those three clusters from 17-minimal-trixie to 17-system-trixie — the
-system variant already used by the other six clusters, which bundles the
barman-cloud tools. Tag confirmed present upstream (ghcr manifest HTTP 200).

- imageName: ghcr.io/cloudnative-pg/postgresql:17-minimal-trixie
+ imageName: ghcr.io/cloudnative-pg/postgresql:17-system-trixie

CNPG applies this as a rolling image update (switchover, no data change).

Not fixed here (separate, primary blocker)

The other six clusters (full image, barman present) fail with a TLS trust
error — the reflected vault-ca-cert bundle carries a stale intermediate CA,
so barman can't verify s3.ceph.unkin.net:

SSL: CERTIFICATE_VERIFY_FAILED ... certificate signature failure

That is a shared trust-anchor refresh (likely owned by terraform-k8s /
config/certificates/secret.yaml, managed-by: terragrunt), handled separately —
it also gates litellm/puppet once they have barman. See the investigation report.

Validation

kustomize build --enable-helm + kubeconform pass on litellm, puppet
overlays and the paperclip base (paperclip has no overlay yet).
pre-commit run passes on all changed files.

Follow-ups

Longer term, the Barman Cloud Plugin (sidecar) would let minimal images keep
their size while still archiving — track with the plugin migration.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why The CNPG buckets are empty after PR #298. Live diagnosis found **two** independent causes; this PR fixes one of them. `litellm`, `puppet` and `paperclip` run the CNPG `17-minimal-trixie` image, which omits the `barman-cloud` CLI. In-tree `barmanObjectStore` archiving shells out to `barman-cloud-wal-archive`, so their archiver dies immediately: ``` ContinuousArchiving=False :: unexpected failure invoking barman-cloud-wal-archive: exec: "barman-cloud-wal-archive": executable file not found in $PATH ``` (verified on the live `puppet-postgres` primary: `which barman-cloud-wal-archive` → not found; on a `-system` cluster it resolves to `/usr/local/bin/barman-cloud-wal-archive`). ## How Switch those three clusters from `17-minimal-trixie` to `17-system-trixie` — the `-system` variant already used by the other six clusters, which bundles the barman-cloud tools. Tag confirmed present upstream (ghcr manifest HTTP 200). ``` - imageName: ghcr.io/cloudnative-pg/postgresql:17-minimal-trixie + imageName: ghcr.io/cloudnative-pg/postgresql:17-system-trixie ``` CNPG applies this as a rolling image update (switchover, no data change). ## Not fixed here (separate, primary blocker) The other six clusters (full image, barman present) fail with a **TLS trust** error — the reflected `vault-ca-cert` bundle carries a **stale intermediate CA**, so barman can't verify `s3.ceph.unkin.net`: ``` SSL: CERTIFICATE_VERIFY_FAILED ... certificate signature failure ``` That is a shared trust-anchor refresh (likely owned by terraform-k8s / `config/certificates/secret.yaml`, `managed-by: terragrunt`), handled separately — it also gates litellm/puppet once they have barman. See the investigation report. ## Validation - `kustomize build --enable-helm` + `kubeconform` pass on `litellm`, `puppet` overlays and the `paperclip` base (paperclip has no overlay yet). - `pre-commit run` passes on all changed files. ## Follow-ups - Longer term, the Barman Cloud Plugin (sidecar) would let minimal images keep their size while still archiving — track with the plugin migration. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

unkinben added 1 commit 2026-07-28 22:22:16 +10:00

Use full postgres image on minimal CNPG clusters so WAL archiving works

ci/woodpecker/pr/vector-test Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/kubeconform Pipeline was successful

Details

3e5d01d584

PR #298 added barmanObjectStore WAL archiving to every CNPG cluster, but
litellm, puppet and paperclip run the `17-minimal-trixie` image, which omits
the barman-cloud CLI. Their archiver dies with `exec: "barman-cloud-wal-archive":
executable file not found in $PATH`, so their buckets stay empty.

Switch those three to `17-system-trixie` (the -system variant already used by
the other clusters), which bundles the barman-cloud tools the in-tree backup
mechanism shells out to. This is a rolling image change; CNPG performs it as a
switchover with no data change.

Note: litellm and puppet will only start archiving once the separate stale
vault-ca-cert trust-anchor issue is also fixed (tracked separately) — that
blocks TLS to s3.ceph.unkin.net for all clusters.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

benvin merged commit e0eeeb6b04 into main

2026-07-28 23:11:30 +10:00

benvin deleted branch benvin/cnpg-backup-fixes

2026-07-28 23:11:30 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/argocd-apps#304