fix: preload correct rke2 airgap images to break CNI bootstrap deadlock (#512)

## Why

Four newly-provisioned el9_8 compute nodes (prodnxsr0014/0015/0018/0019) hang with canal's kube-flannel container in `ImagePullBackOff`: the flannel VXLAN overlay never comes up, so the node can't reach any in-cluster `198.18.200.x` LoadBalancer VIP.

Root cause is a stale airgap-preload version. The nodes boot rke2 **v1.33.11+rke2r1** from the rolling `latest/1.33` repo, whose canal manifest requests `hardened-flannel:v0.28.4` / `hardened-calico:v3.31.5` (build20260415). But `rke2::install` pinned the preloaded bundle to **v1.33.4+rke2r1** (flannel v0.27.2 / calico v3.30.2), so those images were never on disk. containerd then falls back to the `docker.io` mirror (artifactapi, `disable-default-registry-endpoint: true`), reachable only via the pod-overlay VIP that requires the very flannel image being pulled — a bootstrap deadlock. Older nodes (0001-0008) are versionlocked at 1.33.4 and still match their original bundle, so they were unaffected.

## Changes

- Bump pinned `rke2_version` `1.33.4` -> `1.33.11` so the versionlock, RPM `ensure`, and preloaded bundle all line up with the canal image tags the running binary requests. The default `rke2-images.linux-amd64.tar.zst` bundle already contains the canal CNI images (it is RKE2's default CNI), so no extra tarball is needed.
- Wire the airgap archive `source` to the `container_archive_source` class parameter (previously declared in the module but never consumed). The module keeps its generic upstream default; the artifactapi override (the pre-CNI-reachable source, same BGP/physical path the rke2 yum repos already use) lives in the k8s role hiera as `rke2::container_archive_source`.

Applies to servers and agents alike (`rke2::install` runs for both) and preloads `before => Service`, so the bundle lands before rke2 starts.

Notes:
- The `latest/1.33` repo is rolling, so the pinned version must be maintained as the repo advances; a follow-up to pin the yum channel to a fixed patch would remove the drift entirely.
- No terraform-artifactapi change is required. (If a canal-only preload were ever wanted, the github generic remote allowlist would need `rancher/rke2/.*/rke2-images-canal.linux-amd64.tar.zst$` added — but the default bundle already carries those images, so it is unnecessary.)

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #512
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #512.
This commit is contained in:
2026-08-08 18:18:55 +10:00
committed by BenVincent
parent 895ca5b1cb
commit 10854b6501
3 changed files with 5 additions and 3 deletions
+2
View File
@@ -11,6 +11,8 @@ hiera_include:
# manage rke2 # manage rke2
rke2::bootstrap_node: prodnxsr0001.main.unkin.net rke2::bootstrap_node: prodnxsr0001.main.unkin.net
rke2::join_url: https://join-k8s.service.consul:9345 rke2::join_url: https://join-k8s.service.consul:9345
# pull the airgap image bundle via artifactapi (host-reachable pre-CNI), not github directly
rke2::container_archive_source: 'https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/rancher/rke2/releases/download'
rke2::manage_registries: true rke2::manage_registries: true
rke2::registries: rke2::registries:
docker.io: docker.io:
+2 -2
View File
@@ -27,10 +27,10 @@ class rke2::install (
before => Service["rke2-${node_type}"], before => Service["rke2-${node_type}"],
} }
# download required archive of containers # preload the airgap bundle (has the default canal CNI images) so canal starts from disk, not the mirror VIP that needs flannel first
archive { '/var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64.tar.zst': archive { '/var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64.tar.zst':
ensure => present, ensure => present,
source => "https://artifactapi.k8s.syd1.au.unkin.net/api/v1/remote/github/rancher/rke2/releases/download/v${rke2_version}%2B${rke2_release}/rke2-images.linux-amd64.tar.zst", source => "${container_archive_source}/v${rke2_version}%2B${rke2_release}/rke2-images.linux-amd64.tar.zst",
require => [ require => [
Package["rke2-${node_type}"], Package["rke2-${node_type}"],
File['/var/lib/rancher/rke2/agent/images'], File['/var/lib/rancher/rke2/agent/images'],
+1 -1
View File
@@ -1,7 +1,7 @@
# rke2 params # rke2 params
class rke2::params ( class rke2::params (
Enum['server', 'agent'] $node_type = 'agent', Enum['server', 'agent'] $node_type = 'agent',
String $rke2_version = '1.33.4', String $rke2_version = '1.33.11',
String $rke2_release = 'rke2r1', String $rke2_release = 'rke2r1',
Stdlib::Absolutepath $config_file = '/etc/rancher/rke2/config.yaml', Stdlib::Absolutepath $config_file = '/etc/rancher/rke2/config.yaml',
Hash $config_hash = {}, Hash $config_hash = {},