Fix nats-bootstrap: run from /tmp so the nats CLI works under readOnlyRootFS #311

2026-07-29T22:41:53+10:00

unkinben commented

2026-07-29 22:41:53 +10:00

Why

Final-mile bringup: after #301/#306/#308 the auth chain was fixed and logs flowed, but the nats-bootstrap PostSync hook failed with:

nats: error: could not pick a Stream to operate on: ... could not load schema { ... }: stat .: permission denied

The nats CLI stats its working directory when loading response-validation schemas. Under the Job's readOnlyRootFilesystem: true + runAsUser: 1000, the nats-box image's default WORKDIR isn't accessible to uid 1000, so every nats stream/consumer call errored. (A throwaway pod using default securityContext worked, which is why manual stream creation succeeded.)

Consequence: the PostSync hook never completes → logging-logging stays OutOfSync. The LOGS stream + consumers persist in JetStream once created, so log flow is unaffected — but GitOps convergence is blocked and the hook would keep retrying.

What

Set workingDir: /tmp on the bootstrap container (the writable emptyDir already mounted for HOME). The nats CLI can then stat/operate normally.

Verified on the live cluster: a nats-box pod with the Job's exact restrictive securityContext + workingDir: /tmp runs nats stream info LOGS cleanly (fails without it).

Note (separate, pre-existing)

There is also a first-deploy ordering deadlock: the nats-bootstrap PostSync hook runs only after the Sync-phase resources are healthy, but the vector consumer Deployments can't become healthy until the hook creates the LOGS stream. On this deploy I broke the deadlock by creating the stream/consumers manually (idempotent with the Job); the stream now persists so it won't recur on normal re-syncs, but a fresh cluster / PVC loss would hit it again. A durable fix (sync-waves so bootstrap runs after NATS but before the consumers) is worth a follow-up — flagged, not included here to keep this fix minimal.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

## Why Final-mile bringup: after #301/#306/#308 the auth chain was fixed and logs flowed, but the `nats-bootstrap` PostSync hook **failed** with: ``` nats: error: could not pick a Stream to operate on: ... could not load schema { ... }: stat .: permission denied ``` The nats CLI stats its **working directory** when loading response-validation schemas. Under the Job's `readOnlyRootFilesystem: true` + `runAsUser: 1000`, the nats-box image's default WORKDIR isn't accessible to uid 1000, so every `nats stream/consumer` call errored. (A throwaway pod using default securityContext worked, which is why manual stream creation succeeded.) Consequence: the PostSync hook never completes → `logging-logging` stays **OutOfSync**. The `LOGS` stream + consumers persist in JetStream once created, so log flow is unaffected — but GitOps convergence is blocked and the hook would keep retrying. ## What Set `workingDir: /tmp` on the bootstrap container (the writable emptyDir already mounted for `HOME`). The nats CLI can then stat/operate normally. **Verified on the live cluster:** a nats-box pod with the Job's exact restrictive securityContext + `workingDir: /tmp` runs `nats stream info LOGS` cleanly (fails without it). ## Note (separate, pre-existing) There is also a first-deploy ordering deadlock: the `nats-bootstrap` PostSync hook runs only after the Sync-phase resources are healthy, but the vector consumer Deployments can't become healthy until the hook creates the `LOGS` stream. On this deploy I broke the deadlock by creating the stream/consumers manually (idempotent with the Job); the stream now persists so it won't recur on normal re-syncs, but a fresh cluster / PVC loss would hit it again. A durable fix (sync-waves so bootstrap runs after NATS but before the consumers) is worth a follow-up — flagged, not included here to keep this fix minimal. https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

unkinben added 1 commit 2026-07-29 22:41:54 +10:00

Fix nats-bootstrap: run from /tmp so the nats CLI works under readOnlyRootFS

ci/woodpecker/pr/vector-test Pipeline was successful

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/kubeconform Pipeline was successful

Details

d9eb13c3e6

The nats-bootstrap PostSync Job failed at deploy time with
"could not load schema ... stat .: permission denied". The nats CLI stats its
working directory when loading response-validation schemas, and under the Job's
readOnlyRootFilesystem + runAsUser 1000 the image's default WORKDIR is not
accessible. Set workingDir: /tmp (the writable emptyDir already mounted for
HOME) so the CLI can stat/operate. Verified against the live cluster: a nats-box
pod with the exact restrictive securityContext + workingDir: /tmp runs
`nats stream info` cleanly.

Without this the PostSync hook never completes, so the logging-logging app
stays OutOfSync (the LOGS stream/consumers persist in JetStream once created, so
log flow is unaffected, but GitOps convergence is blocked).

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv

benvin merged commit 72c259a2a0 into main

2026-07-30 00:11:16 +10:00

benvin deleted branch benvin/logging-deploy-fixes-3

2026-07-30 00:11:16 +10:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/argocd-apps#311