fix: preload correct rke2 airgap images to break CNI bootstrap deadlock #512

Merged

benvin merged 2 commits from benvin/rke2-airgap-images into develop 2026-08-08 18:18:56 +10:00

Conversation 0 Commits 2 Files Changed 3

+5 -3

unkinben commented 2026-08-08 17:52:19 +10:00

Owner

Copy Link

Copy Source

Why

Four newly-provisioned el9_8 compute nodes (prodnxsr0014/0015/0018/0019) hang with canal's kube-flannel container in ImagePullBackOff: the flannel VXLAN overlay never comes up, so the node can't reach any in-cluster 198.18.200.x LoadBalancer VIP.

Root cause is a stale airgap-preload version. The nodes boot rke2 v1.33.11+rke2r1 from the rolling latest/1.33 repo, whose canal manifest requests hardened-flannel:v0.28.4 / hardened-calico:v3.31.5 (build20260415). But rke2::install pinned the preloaded bundle to v1.33.4+rke2r1 (flannel v0.27.2 / calico v3.30.2), so those images were never on disk. containerd then falls back to the docker.io mirror (artifactapi, disable-default-registry-endpoint: true), reachable only via the pod-overlay VIP that requires the very flannel image being pulled — a bootstrap deadlock. Older nodes (0001-0008) are versionlocked at 1.33.4 and still match their original bundle, so they were unaffected.

Changes

• Bump pinned rke2_version 1.33.4 -> 1.33.11 so the versionlock, RPM ensure, and preloaded bundle all line up with the canal image tags the running binary requests. The default rke2-images.linux-amd64.tar.zst bundle already contains the canal CNI images (it is RKE2's default CNI), so no extra tarball is needed.
• Wire the airgap archive source to the container_archive_source class parameter (previously declared in the module but never consumed). The module keeps its generic upstream default; the artifactapi override (the pre-CNI-reachable source, same BGP/physical path the rke2 yum repos already use) lives in the k8s role hiera as rke2::container_archive_source.

Applies to servers and agents alike (rke2::install runs for both) and preloads before => Service, so the bundle lands before rke2 starts.

Notes:

• The latest/1.33 repo is rolling, so the pinned version must be maintained as the repo advances; a follow-up to pin the yum channel to a fixed patch would remove the drift entirely.
• No terraform-artifactapi change is required. (If a canal-only preload were ever wanted, the github generic remote allowlist would need rancher/rke2/.*/rke2-images-canal.linux-amd64.tar.zst$ added — but the default bundle already carries those images, so it is unnecessary.)

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

## Why Four newly-provisioned el9_8 compute nodes (prodnxsr0014/0015/0018/0019) hang with canal's kube-flannel container in `ImagePullBackOff`: the flannel VXLAN overlay never comes up, so the node can't reach any in-cluster `198.18.200.x` LoadBalancer VIP. Root cause is a stale airgap-preload version. The nodes boot rke2 **v1.33.11+rke2r1** from the rolling `latest/1.33` repo, whose canal manifest requests `hardened-flannel:v0.28.4` / `hardened-calico:v3.31.5` (build20260415). But `rke2::install` pinned the preloaded bundle to **v1.33.4+rke2r1** (flannel v0.27.2 / calico v3.30.2), so those images were never on disk. containerd then falls back to the `docker.io` mirror (artifactapi, `disable-default-registry-endpoint: true`), reachable only via the pod-overlay VIP that requires the very flannel image being pulled — a bootstrap deadlock. Older nodes (0001-0008) are versionlocked at 1.33.4 and still match their original bundle, so they were unaffected. ## Changes - Bump pinned `rke2_version` `1.33.4` -> `1.33.11` so the versionlock, RPM `ensure`, and preloaded bundle all line up with the canal image tags the running binary requests. The default `rke2-images.linux-amd64.tar.zst` bundle already contains the canal CNI images (it is RKE2's default CNI), so no extra tarball is needed. - Wire the airgap archive `source` to the `container_archive_source` class parameter (previously declared in the module but never consumed). The module keeps its generic upstream default; the artifactapi override (the pre-CNI-reachable source, same BGP/physical path the rke2 yum repos already use) lives in the k8s role hiera as `rke2::container_archive_source`. Applies to servers and agents alike (`rke2::install` runs for both) and preloads `before => Service`, so the bundle lands before rke2 starts. Notes: - The `latest/1.33` repo is rolling, so the pinned version must be maintained as the repo advances; a follow-up to pin the yum channel to a fixed patch would remove the drift entirely. - No terraform-artifactapi change is required. (If a canal-only preload were ever wanted, the github generic remote allowlist would need `rancher/rke2/.*/rke2-images-canal.linux-amd64.tar.zst$` added — but the default bundle already carries those images, so it is unnecessary.) https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben added 1 commit 2026-08-08 18:11:31 +10:00

fix: preload correct rke2 airgap images to break CNI bootstrap deadlock ... c39caeb8bb

New el9_8 nodes boot rke2 v1.33.11 (flannel v0.28.4 / calico v3.31.5) from
the rolling latest/1.33 repo, but the airgap preload pinned v1.33.4
(flannel v0.27.2 / calico v3.30.2), so canal's images were never on disk and
containerd fell back to the artifactapi mirror VIP, which is unreachable until
the flannel overlay it would provide is up. Classic bootstrap deadlock.

- bump pinned rke2_version 1.33.4 -> 1.33.11 so the versionlock, RPM and
  preloaded bundle all match the canal image tags the running binary requests
  (the default rke2-images bundle already contains the canal CNI images)
- wire the airgap archive source to the container_archive_source parameter
  (previously declared but unused) and refresh its default to the artifactapi
  github remote, so the pre-CNI-reachable source is hiera-overridable per node

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben force-pushed benvin/rke2-airgap-images from 408b363e05 to c39caeb8bb 2026-08-08 18:11:31 +10:00 Compare

unkinben added 1 commit 2026-08-08 18:14:31 +10:00

refactor: keep rke2 archive source in role hiera, not module default ... 1194b71e9c

- revert rke2::params container_archive_source to the upstream github default
  so the module stays generic and portable
- set the artifactapi override in the k8s role hiera
  (rke2::container_archive_source) where the estate-specific, pre-CNI-reachable
  source belongs

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

benvin merged commit 10854b6501 into develop 2026-08-08 18:18:56 +10:00

benvin deleted branch benvin/rke2-airgap-images 2026-08-08 18:18:56 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/puppet-prod#512