security: reduce privilege in puppet namespace workloads (root init containers, redundant caps, APE) #307

New Issue

2026-07-29T19:26:42+10:00

unkinben commented

2026-07-29 19:26:42 +10:00

Problem

A cluster-wide security sweep (kubectl get pods -A -o json, read-only) flagged the puppet namespace as the only Argo-managed application whose privilege is lazy/reducible rather than intrinsically required (CNI, ceph-csi, purelb, vault IPC_LOCK, traefik NET_BIND_SERVICE, node-feature-discovery, intel-gpu are all legitimately privileged and out of scope). All findings trace to raw manifests under apps/base/puppet/. Concretely:

deployment_puppetboard.yaml — cert-generator init container runs as root with escalation enabled.
securityContext: { runAsUser: 0, runAsGroup: 0, allowPrivilegeEscalation: true } (lines 132-135). The container only generates an SSL keypair into the puppetboard-certs PVC and finishes with chown -R 1000:1000. The main puppetboard container is already fully hardened (runAsNonRoot: true, uid 1000, allowPrivilegeEscalation: false, capabilities.drop: [all]). Nothing in the init script is setuid, so allowPrivilegeEscalation: true grants escalation for no reason, and root is only used to satisfy the trailing chown.
deployment_puppetdb.yaml — create-log-dir init container runs as root only to chown. runAsUser: 0 (line 109) exists solely to mkdir … && chown 999:999 …/logs. The pod has no fsGroup, so the volume is not group-owned by 999 and the chown is the workaround.
Redundant / over-broad added capabilities on the OpenVox containers. puppetdb, puppetserver-master, and puppetserver-compiler main containers, both perms-and-dirs init containers, and the generate-types CronJob all drop: [all] then add: the same cap set listed twice — once CAP_-prefixed and once bare (e.g. CAP_CHOWN and CHOWN). Kubernetes normalises both spellings to the identical kernel capability, so every add-list is 100% duplicated noise. The server/compiler/cronjob lists also include AUDIT_WRITE, which OpenVox does not use (no auditd writes).
Root main containers on the OpenVox workloads. puppetdb/puppetserver-master/puppetserver-compiler main containers set no runAsUser, so they run as the image default (root) and rely on CHOWN/SETUID/SETGID/DAC_OVERRIDE for the entrypoint's privilege-drop to the puppet user. The perms-and-dirs init already chowns the data dirs to puppet:puppet and fsGroup: 999 is set, so running the main process directly as uid 999 (removing the need for any added caps) is likely feasible.

Why it matters: these are the Puppet control-plane pods (CA keys, eyaml keys, compiled catalogs). Root + allowPrivilegeEscalation + broad caps widen the blast radius of any RCE in puppetboard/puppetserver and violate the estate's baseline (non-root, allowPrivilegeEscalation: false, least-capability). Findings 1 and 2 are class A (privilege removable outright); 3 and 4 are class B (reducible).

Proposal

All changes are in apps/base/puppet/:

deployment_puppetboard.yaml: add pod-level spec.template.spec.securityContext.fsGroup: 1000; change the cert-generator init securityContext to runAsUser: 1000, runAsGroup: 1000, runAsNonRoot: true, allowPrivilegeEscalation: false, capabilities.drop: [all]; delete the trailing chown -R 1000:1000 ${CERT_DIR} line (redundant once fsGroup owns the PVC).
deployment_puppetdb.yaml: add pod-level securityContext.fsGroup: 999; drop the create-log-dir init container to runAsUser: 999 / runAsGroup: 999 / runAsNonRoot: true / allowPrivilegeEscalation: false and remove the chown 999:999 from its args (log dir inherits the group via fsGroup).
All five OpenVox add-lists (deployment_puppetdb.yaml, deployment_puppetserver-master.yaml, deployment_puppetserver-compiler.yaml main + perms-and-dirs, cronjob_generate-types.yaml): delete the duplicate CAP_-prefixed entries (keep one canonical spelling), and drop AUDIT_WRITE.
OpenVox main containers (puppetdb, puppetserver-master, puppetserver-compiler): attempt runAsUser: 999, runAsNonRoot: true, capabilities.drop: [all] with no adds; gate on the entrypoint tolerating a non-root start (OpenVox images check id -u before dropping). If a workload regresses, fall back to keeping the minimal CHOWN/SETUID/SETGID/DAC_OVERRIDE/FOWNER set (de-duplicated) for that container only and document why in a comment.

Acceptance

kubectl get pods -n puppet -o json shows: puppetboard cert-generator and puppetdb create-log-dir running as non-root with allowPrivilegeEscalation: false; no container in the namespace carries a duplicated or AUDIT_WRITE capability.
puppetboard serves its UI (pod Ready, / returns 200 via its httproute); puppetdb reaches running at /status/v1/simple; puppetserver master and a compiler reach running; the generate-types CronJob completes without error.
Any container that must remain root retains only the minimal justified capability set with an explanatory comment (class-B fallback), and no container is privileged or has allowPrivilegeEscalation: true.
make / kubeconform + the repo's pre-commit and Woodpecker CI are green.

## Problem A cluster-wide security sweep (`kubectl get pods -A -o json`, read-only) flagged the `puppet` namespace as the only Argo-managed application whose privilege is *lazy/reducible* rather than intrinsically required (CNI, ceph-csi, purelb, vault IPC_LOCK, traefik NET_BIND_SERVICE, node-feature-discovery, intel-gpu are all legitimately privileged and out of scope). All findings trace to raw manifests under `apps/base/puppet/`. Concretely: 1. **`deployment_puppetboard.yaml` — `cert-generator` init container runs as root with escalation enabled.** `securityContext: { runAsUser: 0, runAsGroup: 0, allowPrivilegeEscalation: true }` (lines 132-135). The container only generates an SSL keypair into the `puppetboard-certs` PVC and finishes with `chown -R 1000:1000`. The *main* puppetboard container is already fully hardened (`runAsNonRoot: true`, uid 1000, `allowPrivilegeEscalation: false`, `capabilities.drop: [all]`). Nothing in the init script is setuid, so `allowPrivilegeEscalation: true` grants escalation for no reason, and root is only used to satisfy the trailing `chown`. 2. **`deployment_puppetdb.yaml` — `create-log-dir` init container runs as root only to chown.** `runAsUser: 0` (line 109) exists solely to `mkdir … && chown 999:999 …/logs`. The pod has **no** `fsGroup`, so the volume is not group-owned by 999 and the chown is the workaround. 3. **Redundant / over-broad added capabilities on the OpenVox containers.** `puppetdb`, `puppetserver-master`, and `puppetserver-compiler` main containers, both `perms-and-dirs` init containers, and the `generate-types` CronJob all `drop: [all]` then `add:` the same cap set listed **twice** — once `CAP_`-prefixed and once bare (e.g. `CAP_CHOWN` **and** `CHOWN`). Kubernetes normalises both spellings to the identical kernel capability, so every add-list is 100% duplicated noise. The server/compiler/cronjob lists also include `AUDIT_WRITE`, which OpenVox does not use (no auditd writes). 4. **Root main containers on the OpenVox workloads.** `puppetdb`/`puppetserver-master`/`puppetserver-compiler` main containers set no `runAsUser`, so they run as the image default (root) and rely on `CHOWN`/`SETUID`/`SETGID`/`DAC_OVERRIDE` for the entrypoint's privilege-drop to the `puppet` user. The `perms-and-dirs` init already chowns the data dirs to `puppet:puppet` and `fsGroup: 999` is set, so running the main process directly as uid 999 (removing the need for any added caps) is likely feasible. **Why it matters:** these are the Puppet control-plane pods (CA keys, eyaml keys, compiled catalogs). Root + `allowPrivilegeEscalation` + broad caps widen the blast radius of any RCE in puppetboard/puppetserver and violate the estate's baseline (non-root, `allowPrivilegeEscalation: false`, least-capability). Findings 1 and 2 are class **A** (privilege removable outright); 3 and 4 are class **B** (reducible). ## Proposal All changes are in `apps/base/puppet/`: - **`deployment_puppetboard.yaml`**: add pod-level `spec.template.spec.securityContext.fsGroup: 1000`; change the `cert-generator` init `securityContext` to `runAsUser: 1000`, `runAsGroup: 1000`, `runAsNonRoot: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: [all]`; delete the trailing `chown -R 1000:1000 ${CERT_DIR}` line (redundant once `fsGroup` owns the PVC). - **`deployment_puppetdb.yaml`**: add pod-level `securityContext.fsGroup: 999`; drop the `create-log-dir` init container to `runAsUser: 999 / runAsGroup: 999 / runAsNonRoot: true / allowPrivilegeEscalation: false` and remove the `chown 999:999` from its args (log dir inherits the group via `fsGroup`). - **All five OpenVox add-lists** (`deployment_puppetdb.yaml`, `deployment_puppetserver-master.yaml`, `deployment_puppetserver-compiler.yaml` main + `perms-and-dirs`, `cronjob_generate-types.yaml`): delete the duplicate `CAP_`-prefixed entries (keep one canonical spelling), and drop `AUDIT_WRITE`. - **OpenVox main containers** (`puppetdb`, `puppetserver-master`, `puppetserver-compiler`): attempt `runAsUser: 999`, `runAsNonRoot: true`, `capabilities.drop: [all]` with **no** adds; gate on the entrypoint tolerating a non-root start (OpenVox images check `id -u` before dropping). If a workload regresses, fall back to keeping the minimal `CHOWN`/`SETUID`/`SETGID`/`DAC_OVERRIDE`/`FOWNER` set (de-duplicated) for that container only and document why in a comment. ## Acceptance - `kubectl get pods -n puppet -o json` shows: puppetboard `cert-generator` and puppetdb `create-log-dir` running as non-root with `allowPrivilegeEscalation: false`; no container in the namespace carries a duplicated or `AUDIT_WRITE` capability. - puppetboard serves its UI (pod Ready, `/` returns 200 via its httproute); puppetdb reaches `running` at `/status/v1/simple`; puppetserver master and a compiler reach `running`; the `generate-types` CronJob completes without error. - Any container that must remain root retains only the minimal justified capability set with an explanatory comment (class-B fallback), and no container is `privileged` or has `allowPrivilegeEscalation: true`. - `make` / kubeconform + the repo's pre-commit and Woodpecker CI are green.

unkinben referenced this issue from a commit

2026-08-01 01:32:00 +10:00

puppet: reduce privilege in namespace workloads (#307)

unkinben referenced a pull request that will close this issue

2026-08-01 01:32:15 +10:00

puppet: reduce privilege in namespace workloads (#307) #319

unkinben referenced this issue from a commit

2026-08-08 22:19:44 +10:00

puppet: reduce privilege in namespace workloads (#307)

benvin closed this issue

2026-08-08 22:36:44 +10:00

benvin referenced this issue from a commit