From 10854b65012fd4a9aa9733b8a6025d3b3ca862ee Mon Sep 17 00:00:00 2001 From: Ben Vincent Date: Sat, 8 Aug 2026 18:18:55 +1000 Subject: [PATCH] fix: preload correct rke2 airgap images to break CNI bootstrap deadlock (#512) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why Four newly-provisioned el9_8 compute nodes (prodnxsr0014/0015/0018/0019) hang with canal's kube-flannel container in `ImagePullBackOff`: the flannel VXLAN overlay never comes up, so the node can't reach any in-cluster `198.18.200.x` LoadBalancer VIP. Root cause is a stale airgap-preload version. The nodes boot rke2 **v1.33.11+rke2r1** from the rolling `latest/1.33` repo, whose canal manifest requests `hardened-flannel:v0.28.4` / `hardened-calico:v3.31.5` (build20260415). But `rke2::install` pinned the preloaded bundle to **v1.33.4+rke2r1** (flannel v0.27.2 / calico v3.30.2), so those images were never on disk. containerd then falls back to the `docker.io` mirror (artifactapi, `disable-default-registry-endpoint: true`), reachable only via the pod-overlay VIP that requires the very flannel image being pulled — a bootstrap deadlock. Older nodes (0001-0008) are versionlocked at 1.33.4 and still match their original bundle, so they were unaffected. ## Changes - Bump pinned `rke2_version` `1.33.4` -> `1.33.11` so the versionlock, RPM `ensure`, and preloaded bundle all line up with the canal image tags the running binary requests. The default `rke2-images.linux-amd64.tar.zst` bundle already contains the canal CNI images (it is RKE2's default CNI), so no extra tarball is needed. - Wire the airgap archive `source` to the `container_archive_source` class parameter (previously declared in the module but never consumed). The module keeps its generic upstream default; the artifactapi override (the pre-CNI-reachable source, same BGP/physical path the rke2 yum repos already use) lives in the k8s role hiera as `rke2::container_archive_source`. Applies to servers and agents alike (`rke2::install` runs for both) and preloads `before => Service`, so the bundle lands before rke2 starts. Notes: - The `latest/1.33` repo is rolling, so the pinned version must be maintained as the repo advances; a follow-up to pin the yum channel to a fixed patch would remove the drift entirely. - No terraform-artifactapi change is required. (If a canal-only preload were ever wanted, the github generic remote allowlist would need `rancher/rke2/.*/rke2-images-canal.linux-amd64.tar.zst$` added — but the default bundle already carries those images, so it is unnecessary.) https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT Reviewed-on: https://git.unkin.net/unkin/puppet-prod/pulls/512 Co-authored-by: Ben Vincent Co-committed-by: Ben Vincent --- hieradata/roles/infra/k8s.yaml | 2 ++ modules/rke2/manifests/install.pp | 4 ++-- modules/rke2/manifests/params.pp | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/hieradata/roles/infra/k8s.yaml b/hieradata/roles/infra/k8s.yaml index d8c9bcb..b91b86a 100644 --- a/hieradata/roles/infra/k8s.yaml +++ b/hieradata/roles/infra/k8s.yaml @@ -11,6 +11,8 @@ hiera_include: # manage rke2 rke2::bootstrap_node: prodnxsr0001.main.unkin.net rke2::join_url: https://join-k8s.service.consul:9345 +# pull the airgap image bundle via artifactapi (host-reachable pre-CNI), not github directly +rke2::container_archive_source: 'https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/rancher/rke2/releases/download' rke2::manage_registries: true rke2::registries: docker.io: diff --git a/modules/rke2/manifests/install.pp b/modules/rke2/manifests/install.pp index f15764d..7c5b7b3 100644 --- a/modules/rke2/manifests/install.pp +++ b/modules/rke2/manifests/install.pp @@ -27,10 +27,10 @@ class rke2::install ( before => Service["rke2-${node_type}"], } - # download required archive of containers + # preload the airgap bundle (has the default canal CNI images) so canal starts from disk, not the mirror VIP that needs flannel first archive { '/var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64.tar.zst': ensure => present, - source => "https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/rancher/rke2/releases/download/v${rke2_version}%2B${rke2_release}/rke2-images.linux-amd64.tar.zst", + source => "${container_archive_source}/v${rke2_version}%2B${rke2_release}/rke2-images.linux-amd64.tar.zst", require => [ Package["rke2-${node_type}"], File['/var/lib/rancher/rke2/agent/images'], diff --git a/modules/rke2/manifests/params.pp b/modules/rke2/manifests/params.pp index 4c110dc..9d5a443 100644 --- a/modules/rke2/manifests/params.pp +++ b/modules/rke2/manifests/params.pp @@ -1,7 +1,7 @@ # rke2 params class rke2::params ( Enum['server', 'agent'] $node_type = 'agent', - String $rke2_version = '1.33.4', + String $rke2_version = '1.33.11', String $rke2_release = 'rke2r1', Stdlib::Absolutepath $config_file = '/etc/rancher/rke2/config.yaml', Hash $config_hash = {},