Fix nats-bootstrap: run from /tmp so the nats CLI works under readOnlyRootFS (#311)

## Why

Final-mile bringup: after #301/#306/#308 the auth chain was fixed and logs flowed, but the `nats-bootstrap` PostSync hook **failed** with:
```
nats: error: could not pick a Stream to operate on: ... could not load schema { ... }: stat .: permission denied
```
The nats CLI stats its **working directory** when loading response-validation schemas. Under the Job's `readOnlyRootFilesystem: true` + `runAsUser: 1000`, the nats-box image's default WORKDIR isn't accessible to uid 1000, so every `nats stream/consumer` call errored. (A throwaway pod using default securityContext worked, which is why manual stream creation succeeded.)

Consequence: the PostSync hook never completes → `logging-logging` stays **OutOfSync**. The `LOGS` stream + consumers persist in JetStream once created, so log flow is unaffected — but GitOps convergence is blocked and the hook would keep retrying.

## What

Set `workingDir: /tmp` on the bootstrap container (the writable emptyDir already mounted for `HOME`). The nats CLI can then stat/operate normally.

**Verified on the live cluster:** a nats-box pod with the Job's exact restrictive securityContext + `workingDir: /tmp` runs `nats stream info LOGS` cleanly (fails without it).

## Note (separate, pre-existing)

There is also a first-deploy ordering deadlock: the `nats-bootstrap` PostSync hook runs only after the Sync-phase resources are healthy, but the vector consumer Deployments can't become healthy until the hook creates the `LOGS` stream. On this deploy I broke the deadlock by creating the stream/consumers manually (idempotent with the Job); the stream now persists so it won't recur on normal re-syncs, but a fresh cluster / PVC loss would hit it again. A durable fix (sync-waves so bootstrap runs after NATS but before the consumers) is worth a follow-up — flagged, not included here to keep this fix minimal.

https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #311
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>

This commit was merged in pull request #311.

This commit is contained in:

Ben Vincent

2026-07-30 00:11:16 +10:00

committed by

BenVincent

parent 9c10b9096a

commit 72c259a2a0

1 changed files with 5 additions and 0 deletions

									
										apps/base/logging/nats-bootstrap-job.yaml
									
		+5
		
												View File
												
				@@ -59,6 +59,11 @@ spec:

				      containers:

				        - name: nats-bootstrap

				          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/natsio/nats-box:0.18.0

				          # nats CLI stats the working directory when loading its response

				          # schemas; under readOnlyRootFilesystem + runAsUser 1000 the image's

				          # default WORKDIR is not accessible ("stat .: permission denied"), so

				          # run from the writable /tmp emptyDir.

				          workingDir: /tmp

				          securityContext:

				            allowPrivilegeEscalation: false

				            readOnlyRootFilesystem: true