The internal intermediate CA re-key broke CA consumers because Reloader
ignored Secrets and could not restart on the vault-ca-cert Secret. Enabling
generic secret watching naively would restart every workload on each
Vault/VSO secret rotation, so this scopes existing auto annotations to
ConfigMaps and makes secret reload opt-in per Secret.
- set reloader ignoreSecrets: false so Secrets are watched
- convert every generic reloader.stakater.com/auto to the configmap-only
configmap.reloader.stakater.com/auto form (22 annotations, 19 files)
- add explicit secret.reloader.stakater.com/reload: "vault-ca-cert" to the CA
consumers (artifactapi api, cephrgw-operator, puppetserver master+compiler,
litellm, logarchiver) so CA rotation restarts them
- add secret.reloader.stakater.com/reload: "kanidm-tls" so the cert-manager
leaf renewal rolls kanidm
- add docs/ca-rotation.md runbook and index it
Closes#326
Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
## Why
The puppet-on-k8s compilers classify nodes with a uv/python ENC script (`encapi-enc`). Each fresh compiler pod resolves the script's python dependencies on first invocation, and that resolution fails on cold pods (observed exits 135/2), breaking puppet agent catalog compilation. `encapic` (git.unkin.net/unkin/encapic) is a stdlib-only Go replacement with no runtime dependency resolution — a behavioural drop-in whose output matches the python script byte-for-byte.
## Changes
- Points the compiler `external_nodes` at `/opt/bin/encapic`.
- Reworks the `setup-shared-bins` init container to `curl` the encapic `v0.1.0` `encapic_linux_amd64` release binary (sha256-verified against the published `.sha256`, installed mode 0755) into the shared bins dir, instead of copying the python script and installing uv.
- Removes the `puppet-encapi-enc` configmap generator, its volume and mount, and the `resources/encapi-enc` script. uv was consumed solely by that script (grep of `apps/base/puppet` confirms no other consumer), so its installation is removed too.
`kubectl kustomize apps/overlays/au-syd1/puppet` builds clean.
## Merge gate
Do not merge until the encapic `v0.1.0` release assets exist:
`https://git.unkin.net/unkin/encapic/releases/download/v0.1.0/encapic_linux_amd64` (+ `.sha256`). The init container pulls them at pod start.
## Rollback
Revert this PR to restore the `encapi-enc` configmap script + uv install and repoint `external_nodes`.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #277
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The k8s puppetserver compilers classify nodes via an exec ENC that today queries legacy Cobbler (`https://cobbler.main.unkin.net`) over TLS. `encapi` now runs in-cluster and exposes a cobbler-wire-compatible endpoint (`GET /cblr/svc/op/puppet/hostname/<certname>`), a drop-in for the Cobbler URL. This cuts the puppet-on-k8s ENC over from Cobbler to encapi — a prerequisite for migrating VM agents onto puppet-on-k8s.
## Changes
- Rename the ENC script `resources/cobbler-enc` -> `resources/encapi-enc`, and its configmap `puppet-cobbler-enc` -> `puppet-encapi-enc` (kustomization configMapGenerator + deployment volume, initContainer copy path, and volumeMount subPath).
- Point `external_nodes` in the compiler `puppet.conf` at `/opt/bin/encapi-enc`.
- Target the in-cluster encapi service `http://encapi.encapi.svc.cluster.local` (plain HTTP), overridable via the `ENCAPI_URL` env var.
- Drop the `/opt/vault-ca-cert.crt` verify for the ENC request (no TLS needed in-cluster).
- Leave the response normalization identical: classes coerced to a list, `enc_role`/`enc_env` params set, `environment` stripped when it equals `testing`.
Verified with `kubectl kustomize apps/overlays/au-syd1/puppet` (builds clean, exit 0); the generated `puppet-encapi-enc` configmap contains the new URL and env var.
## 🚨 Merge gate
**Do not merge until encapi is seeded** (terraform-incus `benvin/encapi-seed` PR applied). An empty encapi means every node resolves to a 404. On 404 the ENC script exits non-zero, so puppet fails the compile rather than classifying the node with zero classes — nodes will fail to run until they exist in encapi. Seed encapi first so real nodes classify correctly; only unknown nodes should 404.
Reviewed-on: #272
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Add support for installing additional Ruby gems via custom entrypoint script.
The script is mounted as a ConfigMap into /container-custom-entrypoint.d/
and will be executed during Puppetserver container startup.
Reviewed-on: #63
- Mount vault-ca-cert secret at /opt/vault-ca-cert.crt in both deployments
- Update cobbler-enc script to use correct CA certificate path
- Resolves OSError about missing TLS CA certificate bundle
Reviewed-on: #62
- Add puppet-shared-bins PVC (10GB) for shared binaries
- Mount /opt/bin in both compiler and master deployments
- Add init container to install uv binary and cobbler script to shared volume
- Update cobbler-enc to use absolute path and uv cache directory
- Configure puppet.conf to reference cobbler-enc from /opt/bin
Reviewed-on: #61
Ben Vincent
unkinben
benvin