rke2: taint control-plane nodes NoSchedule #517

Merged

unkinben merged 1 commits from benvin/rke2-control-plane-taint into develop 2026-08-08 23:19:12 +10:00

Conversation 0 Commits 1 Files Changed 1

+3

unkinben commented 2026-08-08 23:14:30 +10:00

Owner

Copy Link

Copy Source

Why

• The 3 control-plane/etcd/master nodes (prodnxsr0001/0002/0003, roles::infra::k8s::control) carry NO taints, so regular workloads schedule onto the controllers.
• Controllers should run only control-plane and tolerating workloads.

How

• Add node-taint: "node-role.kubernetes.io/control-plane=true:NoSchedule" to rke2::config_hash in hieradata/roles/infra/k8s/control.yaml — control role only; compute/agent nodes untouched.
• rke2::config_hash is a deep-merge key, so the taint layers onto the existing control config and renders into the /etc/rancher/rke2/config.yaml node-taint: list.

Operational nuances (read before converging)

• Stagger the control-plane restart — one node at a time. Puppet's rke2::service subscribes to File[config_file], so changing config.yaml restarts rke2-server. On a 3-node etcd control plane a near-simultaneous restart of all three risks a brief quorum loss. Let Puppet converge/restart the controllers one at a time, confirming etcd quorum (e.g. rke2 etcd-snapshot health / etcdctl endpoint health) between each.

• A one-time manual taint IS required. RKE2 node-taint (kubelet --register-with-taints) is applied only at node registration — per the RKE2 docs the option "only add[s] labels and/or taints at registration time, and can only be added once and not removed after that through rke2 commands." The RKE2 version in use (1.33.13~rke2r2, pinned in modules/rke2/manifests/params.pp) does not reconcile the taint onto already-registered nodes on restart. The config change only guarantees the taint on any future re-registration (node rebuilt/removed+rejoined). Apply the taint once, from a privileged kubeconfig context:

  kubectl taint nodes prodnxsr0001 prodnxsr0002 prodnxsr0003 \
    node-role.kubernetes.io/control-plane=true:NoSchedule
  

  Confirm the exact node names first with kubectl get nodes (RKE2 uses the node hostname; may be short or FQDN).

• NoSchedule, not NoExecute. Running pods are not evicted; only new scheduling is blocked. Workloads already on the controllers stay until they are rescheduled elsewhere.

• Nothing critical gets stranded. The static control-plane pods (etcd/kube-apiserver/controller-manager/scheduler) bypass scheduling and are unaffected by taints; the canal CNI DaemonSet tolerates all taints (operator: Exists) so it still runs on the controllers. Everything else (CoreDNS, ingress, metrics-server, estate workloads) has compute/agent nodes to land on. Operators should verify any per-node DaemonSet they care about on controllers (e.g. metrics exporters) carries the standard node-role.kubernetes.io/control-plane:NoSchedule toleration, or it will stop scheduling to the controllers.

## Why - The 3 control-plane/etcd/master nodes (prodnxsr0001/0002/0003, `roles::infra::k8s::control`) carry NO taints, so regular workloads schedule onto the controllers. - Controllers should run only control-plane and tolerating workloads. ## How - Add `node-taint: "node-role.kubernetes.io/control-plane=true:NoSchedule"` to `rke2::config_hash` in `hieradata/roles/infra/k8s/control.yaml` — control role only; compute/agent nodes untouched. - `rke2::config_hash` is a deep-merge key, so the taint layers onto the existing control config and renders into the `/etc/rancher/rke2/config.yaml` `node-taint:` list. ## Operational nuances (read before converging) - **Stagger the control-plane restart — one node at a time.** Puppet's `rke2::service` subscribes to `File[config_file]`, so changing `config.yaml` restarts `rke2-server`. On a 3-node etcd control plane a near-simultaneous restart of all three risks a brief quorum loss. Let Puppet converge/restart the controllers **one at a time**, confirming etcd quorum (e.g. `rke2 etcd-snapshot` health / `etcdctl endpoint health`) between each. - **A one-time manual taint IS required.** RKE2 `node-taint` (kubelet `--register-with-taints`) is applied **only at node registration** — per the RKE2 docs the option "only add[s] labels and/or taints at registration time, and can only be added once and not removed after that through rke2 commands." The RKE2 version in use (`1.33.13~rke2r2`, pinned in `modules/rke2/manifests/params.pp`) does **not** reconcile the taint onto already-registered nodes on restart. The config change only guarantees the taint on any future re-registration (node rebuilt/removed+rejoined). Apply the taint once, from a privileged kubeconfig context: ``` kubectl taint nodes prodnxsr0001 prodnxsr0002 prodnxsr0003 \ node-role.kubernetes.io/control-plane=true:NoSchedule ``` Confirm the exact node names first with `kubectl get nodes` (RKE2 uses the node hostname; may be short or FQDN). - **NoSchedule, not NoExecute.** Running pods are not evicted; only new scheduling is blocked. Workloads already on the controllers stay until they are rescheduled elsewhere. - **Nothing critical gets stranded.** The static control-plane pods (etcd/kube-apiserver/controller-manager/scheduler) bypass scheduling and are unaffected by taints; the canal CNI DaemonSet tolerates all taints (`operator: Exists`) so it still runs on the controllers. Everything else (CoreDNS, ingress, metrics-server, estate workloads) has compute/agent nodes to land on. Operators should verify any per-node DaemonSet they care about on controllers (e.g. metrics exporters) carries the standard `node-role.kubernetes.io/control-plane:NoSchedule` toleration, or it will stop scheduling to the controllers.

unkinben added 1 commit 2026-08-08 23:14:31 +10:00

rke2: taint control-plane nodes NoSchedule ... b85b7c207e

## Why
- The 3 control-plane/etcd nodes (prodnxsr0001/0002/0003, roles::infra::k8s::control) carry NO taints, so regular workloads schedule onto the controllers.
- Controllers should run only control-plane and tolerating workloads.

## How
- Add `node-taint: node-role.kubernetes.io/control-plane=true:NoSchedule` to `rke2::config_hash` in `hieradata/roles/infra/k8s/control.yaml` (control role only; compute/agent nodes untouched).
- `rke2::config_hash` deep-merges, so the taint layers onto the existing control config and renders into `/etc/rancher/rke2/config.yaml` node-taint list.

unkinben merged commit 62dff97c98 into develop 2026-08-08 23:19:12 +10:00

unkinben deleted branch benvin/rke2-control-plane-taint 2026-08-08 23:19:13 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/puppet-prod#517