unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

puppetdb: unique per-pod command-queue directory #340

Merged
unkinben merged 1 commits from benvin/puppetdb-per-pod-queue into main 2026-08-08 20:04:18 +10:00
Conversation 0 Commits 1 Files Changed 1
+11
unkinben commented 2026-08-08 19:58:01 +10:00
Owner
Copy Link
Copy Source

Why

PuppetDB runs as a Deployment (HPA minReplicas: 2, maxReplicas: 5) with its vardir /opt/puppetlabs/server/data/puppetdb backed by a single ReadWriteMany cephfs PVC (puppetserver-puppetdb-claim). Every replica mounts the same path, so all pods share one stockpile/ command queue. PuppetDB's stockpile holds unprocessed commands and is not safe for concurrent writers from multiple processes — shared, the pods corrupt each other's queue/stockpile.

Change

  • Add POD_NAME via the downward API (fieldRef: metadata.name) to the puppetdb container and the create-log-dir init container.
  • Mount puppetdb-storage with subPathExpr: $(POD_NAME) on both, so each pod gets its own vardir subtree (<pvc>/<pod-name>/) — hence its own stockpile/ command queue — while still living on the shared, persistent RWX volume.

Approach

Chose subPathExpr with pod name over converting to a StatefulSet: it is the minimal correct fix for a Deployment on a shared RWX PVC, keeps the existing HPA (which targets a Deployment) intact, and keeps the queue persistent per pod (not an emptyDir), so unprocessed commands survive restarts. PuppetDB's vardir default is unchanged (/opt/puppetlabs/server/data/puppetdb); the container still sees the mount at that path, so the app config and the k8s mount agree with no configmap change. The stockpile resolves to <vardir>/stockpile, now per-pod.

Current replica count: HPA min 2 / max 5 (actively multi-pod, so the corruption is live, not just a future scaling risk).

Note: at rollout, pods move from the shared root of the PVC to per-pod subdirs; any commands sitting in the old shared stockpile/ are orphaned once. This is a one-time cutover — the authoritative data is in Postgres and agents resubmit on their next run.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

## Why PuppetDB runs as a Deployment (HPA `minReplicas: 2`, `maxReplicas: 5`) with its vardir `/opt/puppetlabs/server/data/puppetdb` backed by a single **ReadWriteMany** cephfs PVC (`puppetserver-puppetdb-claim`). Every replica mounts the same path, so all pods share one `stockpile/` command queue. PuppetDB's stockpile holds unprocessed commands and is not safe for concurrent writers from multiple processes — shared, the pods corrupt each other's queue/stockpile. ## Change - Add `POD_NAME` via the downward API (`fieldRef: metadata.name`) to the puppetdb container and the `create-log-dir` init container. - Mount `puppetdb-storage` with `subPathExpr: $(POD_NAME)` on both, so each pod gets its own vardir subtree (`<pvc>/<pod-name>/`) — hence its own `stockpile/` command queue — while still living on the shared, persistent RWX volume. ## Approach Chose **subPathExpr with pod name** over converting to a StatefulSet: it is the minimal correct fix for a Deployment on a shared RWX PVC, keeps the existing HPA (which targets a Deployment) intact, and keeps the queue persistent per pod (not an emptyDir), so unprocessed commands survive restarts. PuppetDB's `vardir` default is unchanged (`/opt/puppetlabs/server/data/puppetdb`); the container still sees the mount at that path, so the app config and the k8s mount agree with no configmap change. The stockpile resolves to `<vardir>/stockpile`, now per-pod. Current replica count: HPA min 2 / max 5 (actively multi-pod, so the corruption is live, not just a future scaling risk). Note: at rollout, pods move from the shared root of the PVC to per-pod subdirs; any commands sitting in the old shared `stockpile/` are orphaned once. This is a one-time cutover — the authoritative data is in Postgres and agents resubmit on their next run. https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
unkinben added 1 commit 2026-08-08 19:58:02 +10:00
puppetdb: give each pod a unique command-queue directory
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
5bb5be15f3 
PuppetDB runs as a Deployment (HPA min 2 / max 5) with its vardir
/opt/puppetlabs/server/data/puppetdb backed by a shared RWX cephfs PVC.
All replicas mount the same path, so they share the stockpile/ command
queue and corrupt each other's unprocessed commands.

Mount the shared PVC with subPathExpr: $(POD_NAME) so each pod gets its
own vardir (and thus its own stockpile queue) on the shared volume.

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
unkinben merged commit 4c7c97ab80 into main 2026-08-08 20:04:18 +10:00
benvin deleted branch benvin/puppetdb-per-pod-queue 2026-08-08 22:23:04 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#340
Delete Branch "benvin/puppetdb-per-pod-queue"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?