Pin puppet master to a single Recreate replica
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful

The puppet MASTER is the singleton CA/master: a second master, even
transiently during a rolling update, races on CA/cert signing and shared
state. It was previously driven by an HPA (minReplicas 2, maxReplicas 5)
with a RollingUpdate strategy, so 2-5 masters could coexist and a rollout
would briefly run old+new masters against the same shared CA PVC.

- Set puppetserver-master replicas to 1 and strategy to Recreate so the
  old master always terminates before a new one starts.
- Remove the puppetserver-masters-autoscaler HPA (and its kustomization
  entry) that forced 2-5 master replicas.
- Refresh the master VPA note to reflect the pinned-singleton, no-HPA state.

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
This commit is contained in:
Ben Vincent
2026-08-08 19:58:42 +10:00
parent ba7a1a9509
commit a5f3b18e86
4 changed files with 5 additions and 42 deletions
@@ -12,12 +12,13 @@ metadata:
name: puppetserver-master
namespace: puppet
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: puppetserver
app.kubernetes.io/name: puppetserver
strategy:
type: RollingUpdate
type: Recreate
template:
metadata:
annotations:
@@ -1,37 +0,0 @@
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
labels:
app.kubernetes.io/component: puppetserver
app.kubernetes.io/instance: puppetserver
app.kubernetes.io/name: puppetserver
app.kubernetes.io/version: 8.8.0
name: puppetserver-masters-autoscaler
namespace: puppet
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: puppetserver-master
minReplicas: 2
maxReplicas: 5
metrics:
- resource:
name: cpu
target:
averageUtilization: 75
type: Utilization
type: Resource
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Percent
value: 50
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 25
periodSeconds: 60
-1
View File
@@ -23,7 +23,6 @@ resources:
- deployment_puppetdb.yaml
- deployment_puppetserver-master.yaml
- horizontalpodautoscaler_puppetserver-compilers-autoscaler.yaml
- horizontalpodautoscaler_puppetserver-masters-autoscaler.yaml
- horizontalpodautoscaler_puppetserver-puppetboard-autoscaler.yaml
- horizontalpodautoscaler_puppetserver-puppetdb-autoscaler.yaml
- gateway_puppetboard.yaml
+3 -3
View File
@@ -52,9 +52,9 @@ kind: VerticalPodAutoscaler
metadata:
name: puppetserver-master-vpa
namespace: puppet
# NOTE: this workload also has an HPA. updateMode Off is recommendation-only
# and does not act, so there is no HPA/VPA conflict today. Do not flip to Auto/
# Initial without first moving the HPA off CPU/memory (VPA owns those under Auto).
# NOTE: the master is a pinned single replica (Recreate, no HPA) so the CA/master
# never coexists. updateMode Off keeps this recommendation-only; do not flip to
# Auto/Initial, which would evict and briefly recreate the singleton pod.
spec:
targetRef:
apiVersion: apps/v1