Files
argocd-apps/apps/base/puppet/deployment_puppetboard.yaml
T
unkinben ba7a1a9509 Enable Reloader secret watching, scope existing auto to configmap-only (#326) (#339)
## Why

The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret.

## Changes

- Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched.
- Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads.
- Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`.
- Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf.
- Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`).

## Safety review (secret-only / CA workloads)

`vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded.

- `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated.
- `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`).
- `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only.
- `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added.

CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`).

## Notes / coordination

- Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression).
- Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here.

## Validation

- `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains.
- `uvx pre-commit run --all-files` — all hooks pass.

Closes #326

---------

Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #339
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-08-08 19:35:31 +10:00

171 lines
5.6 KiB
YAML

---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: puppetboard
app.kubernetes.io/instance: puppetserver
app.kubernetes.io/name: puppetserver
app.kubernetes.io/version: 8.8.0
name: puppetboard
namespace: puppet
spec:
selector:
matchLabels:
app.kubernetes.io/component: puppetboard
app.kubernetes.io/name: puppetserver
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
annotations:
configmap.reloader.stakater.com/auto: "true"
labels:
app.kubernetes.io/component: puppetboard
app.kubernetes.io/instance: puppetserver
app.kubernetes.io/name: puppetserver
app.kubernetes.io/version: 8.8.0
spec:
enableServiceLinks: false
initContainers:
- name: wait-puppetserver
image: curlimages/curl:8.11.1
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
echo 'Waiting for puppetserver to become ready...'
until printf "." && curl --silent --fail --insecure 'https://puppetca:8140/status/v1/simple' | grep -q '^running$'; do
sleep 2;
done;
echo 'Puppetserver OK ✓'
resources:
limits:
cpu: 20m
memory: 32Mi
requests:
cpu: 20m
memory: 32Mi
- name: cert-generator
image: git.unkin.net/unkin/almalinux9-base:20260606
imagePullPolicy: IfNotPresent
command:
- sh
- -c
- |
set -e
# Set the hostname for the certificate
HOSTNAME="puppetboard"
CERT_DIR="/opt/puppetboard/ssl"
# Create certificate directory
mkdir -p ${CERT_DIR}
# Check if certificates already exist
if [ -f "${CERT_DIR}/${HOSTNAME}.pem" ] && [ -f "${CERT_DIR}/${HOSTNAME}.key" ] && [ -f "${CERT_DIR}/ca.pem" ]; then
echo "Certificates already exist for ${HOSTNAME}, skipping generation"
exit 0
fi
# Request certificate from Puppet CA for Puppetboard
echo "Requesting certificate for ${HOSTNAME} from puppetca service"
# Generate private key
openssl genrsa -out ${CERT_DIR}/${HOSTNAME}.key 2048
# Create certificate signing request (CSR)
openssl req -new -key ${CERT_DIR}/${HOSTNAME}.key \
-out /tmp/${HOSTNAME}.csr \
-subj "/CN=${HOSTNAME}"
# Submit CSR to Puppet CA
echo "Submitting certificate request to Puppet CA..."
curl -X PUT \
--insecure \
--data-binary @/tmp/${HOSTNAME}.csr \
-H "Content-Type: text/plain" \
https://puppetca:8140/puppet-ca/v1/certificate_request/${HOSTNAME}
# Wait for certificate to be signed (poll the CA)
echo "Waiting for certificate to be signed..."
for i in {1..30}; do
if curl --insecure -f -s https://puppetca:8140/puppet-ca/v1/certificate/${HOSTNAME} > ${CERT_DIR}/${HOSTNAME}.pem; then
echo "Certificate received for ${HOSTNAME}"
break
fi
echo "Attempt $i: Certificate not ready yet, waiting 10 seconds..."
sleep 10
done
# Verify we got the certificate
if [ ! -f "${CERT_DIR}/${HOSTNAME}.pem" ] || [ ! -s "${CERT_DIR}/${HOSTNAME}.pem" ]; then
echo "Failed to obtain certificate for ${HOSTNAME}"
exit 1
fi
# Get CA certificate
curl --insecure -f https://puppetca:8140/puppet-ca/v1/certificate/ca > ${CERT_DIR}/ca.pem
# Set appropriate permissions
chmod 644 ${CERT_DIR}/${HOSTNAME}.pem
chmod 600 ${CERT_DIR}/${HOSTNAME}.key
chmod 644 ${CERT_DIR}/ca.pem
# Change ownership to puppetboard user (1000:1000)
chown -R 1000:1000 ${CERT_DIR}
echo "Certificate generation completed for ${HOSTNAME}"
volumeMounts:
- name: puppetboard-certs
mountPath: /opt/puppetboard/ssl
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 50m
memory: 64Mi
securityContext:
runAsUser: 0
runAsGroup: 0
allowPrivilegeEscalation: true
containers:
- name: puppetboard
image: ghcr.io/voxpupuli/puppetboard:7.0.1
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
name: puppetboard
envFrom:
- configMapRef:
name: puppetboard-config
- secretRef:
name: puppetboard-secrets
resources:
requests:
memory: 350Mi
cpu: 50m
limits:
memory: 1024Mi
cpu: 500m
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
allowPrivilegeEscalation: false
capabilities:
drop:
- all
volumeMounts:
- name: puppetboard-certs
mountPath: /opt/puppetboard/ssl
readOnly: true
volumes:
- name: puppetboard-certs
persistentVolumeClaim:
claimName: puppetboard-certs