puppetdb: isolate only the stockpile queue per pod (fix #340 crashloop) #342

2026-08-08T20:14:04+10:00

unkinben commented

2026-08-08 20:14:04 +10:00

Why

PR #340 made the entire PuppetDB vardir (/opt/puppetlabs/server/data/puppetdb) per-pod via subPathExpr: $(POD_NAME). The vardir also holds the SSL keypair under certs/, so every freshly-rolled pod got an empty certs/ and crash-looped:

(/ssl.sh) Error: No keypair on disk and CA already has signed certificate for 'openvoxdb'

All puppetdb pods share certname openvoxdb, so a pod with no local keypair can neither reuse the cert nor re-request one the CA already signed. Result: puppetdb went CrashLoopBackOff after #340 synced. Only the command queue (stockpile/) actually needs per-pod isolation; certs/ (and logs/) were correctly shared before.

Change

Revert the whole-vardir subPathExpr; the vardir mount is shared again (certs + logs behave as before #340).
Nest-mount the same PVC at /opt/puppetlabs/server/data/puppetdb/stockpile with subPathExpr: stockpile/$(POD_NAME), so each pod gets its own command queue at <pvc>/stockpile/<pod-name>/ while everything else stays shared.
The create-log-dir init container (root) now also mkdirs and chowns the per-pod stockpile dir to uid/gid 999.

Approach

Still subPathExpr with pod name, but scoped to the stockpile subdirectory instead of the whole vardir — the minimal correct fix that both restores service and delivers unique per-pod command queues. Persistent per-pod (nested subPath on the durable RWX PVC), not emptyDir. PuppetDB's stockpile default (<vardir>/stockpile) is unchanged; the k8s mount simply backs that path per-pod, so app config and mount agree with no configmap change.

Replica count: HPA min 2 / max 5 (unchanged).

Restores service (certs shared again) and keeps the per-pod queue fix from #340.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

## Why PR #340 made the **entire** PuppetDB vardir (`/opt/puppetlabs/server/data/puppetdb`) per-pod via `subPathExpr: $(POD_NAME)`. The vardir also holds the SSL keypair under `certs/`, so every freshly-rolled pod got an empty `certs/` and crash-looped: ``` (/ssl.sh) Error: No keypair on disk and CA already has signed certificate for 'openvoxdb' ``` All puppetdb pods share certname `openvoxdb`, so a pod with no local keypair can neither reuse the cert nor re-request one the CA already signed. Result: puppetdb went CrashLoopBackOff after #340 synced. Only the command queue (`stockpile/`) actually needs per-pod isolation; `certs/` (and `logs/`) were correctly shared before. ## Change - Revert the whole-vardir `subPathExpr`; the vardir mount is shared again (certs + logs behave as before #340). - Nest-mount the same PVC at `/opt/puppetlabs/server/data/puppetdb/stockpile` with `subPathExpr: stockpile/$(POD_NAME)`, so each pod gets its own command queue at `<pvc>/stockpile/<pod-name>/` while everything else stays shared. - The `create-log-dir` init container (root) now also `mkdir`s and `chown`s the per-pod stockpile dir to uid/gid 999. ## Approach Still **subPathExpr with pod name**, but scoped to the stockpile subdirectory instead of the whole vardir — the minimal correct fix that both restores service and delivers unique per-pod command queues. Persistent per-pod (nested subPath on the durable RWX PVC), not emptyDir. PuppetDB's stockpile default (`<vardir>/stockpile`) is unchanged; the k8s mount simply backs that path per-pod, so app config and mount agree with no configmap change. Replica count: HPA min 2 / max 5 (unchanged). Restores service (certs shared again) and keeps the per-pod queue fix from #340. https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben added 1 commit 2026-08-08 20:14:04 +10:00

puppetdb: isolate only the stockpile queue per pod, not the whole vardir

ci/woodpecker/pr/vector-test Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/kubeconform Pipeline was successful

Details

7aed499b8c

PR #340 made the entire vardir (/opt/puppetlabs/server/data/puppetdb)
per-pod via subPathExpr. That also isolated the SSL keypair under
vardir/certs, so freshly-rolled pods had an empty certs dir and crashed:
"No keypair on disk and CA already has signed certificate for openvoxdb"
(all pods share certname openvoxdb). puppetdb went CrashLoopBackOff.

Keep the vardir shared (certs + logs as before) and make only the
command queue per-pod: nest-mount the same PVC at vardir/stockpile with
subPathExpr stockpile/$(POD_NAME). The init container pre-creates and
chowns the per-pod stockpile dir.

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben merged commit c722df415a into main

2026-08-08 20:16:13 +10:00

benvin deleted branch benvin/puppetdb-stockpile-only-queue

2026-08-08 22:11:21 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/argocd-apps#342