## Why
Every puppet run on k8s nodes (e.g. prodnxsr0002) fails on `Package[rke2-server]` and stops applying the rest of the catalog, so the nodes stop receiving all further package/config updates:
```
change from '1.33.4~rke2r1-1.el9' to '1.33.11~rke2r1' failed: Could not update:
dnf upgrade rke2-server-1.33.11~rke2r1 returned 1:
Problem: problem with installed package rke2-common-1.33.13~rke2r2-0.el9.x86_64
- package rke2-server-1.33.11~rke2r1 requires rke2-common = 1.33.11~rke2r1, but none of the providers can be installed
- cannot install the best update candidate for package rke2-server
```
`rke2::install` versionlocks only `rke2-server`/`rke2-agent`, never their strict (`= version`) `rke2-common` dependency. `rke2-common` is served from the rolling `rancher-rke2-1.33-latest` channel, whose head is now `1.33.13~rke2r2`, so `rke2-common` drifted up to `1.33.13~rke2r2` while the pin (`#512`) sat at `1.33.11~rke2r1`. `dnf upgrade` cannot downgrade the newer `rke2-common` to satisfy the older server, so the transaction fails. This is the rolling-channel drift `#512` flagged as needing follow-up.
## How
- Versionlock `rke2-common` to the same `${rke2_version}~${rke2_release}` as the server/agent, so the rolling channel can no longer drift the dependency ahead of the pin.
- Bump `rke2_version` `1.33.11` -> `1.33.13` and `rke2_release` `rke2r1` -> `rke2r2` to match the current channel head and the already-drifted installed `rke2-common`, so the pinned server/agent, both versionlocks, and the preloaded airgap bundle resolve in one transaction.
Verified against the live artifactapi rke2 remote: `rke2-server-1.33.13~rke2r2-0.el9.x86_64.rpm` and the `v1.33.13+rke2r2` `rke2-images.linux-amd64.tar.zst` airgap bundle both serve HTTP 200.
Reviewed-on: #516
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
A package versionlock is only effective if it is applied **before** the package resource is evaluated. When the package is upgraded/installed first and locked afterwards, dnf can pull an unpinned version and the lock then pins whatever landed. This is the exact drift that let new RKE2 nodes boot 1.33.11 instead of the pinned version, and it also matters because a global `dnf update -y` autoupdate timer (`profiles::yum::autoupdater`) relies on the versionlock plugin already having the lock written.
## Changes
- `rke2::install` orders the versionlock ahead of the package: adds `before => Package["rke2-${node_type}"]` so the lock exists before install/upgrade.
- `profiles::puppet::agent` (RedHat branch) adds `before => Package[$use_package]` on the versionlock for the same reason.
- Leaves `profiles::gitea::runner` and `profiles::vault::server` unchanged: both already order the versionlock ahead of the package (`before => Package` / `before => Class['vault']`).
- Changes ordering only; locked versions (including the RKE2 pin) are untouched. No dependency cycle is introduced (packages hold no back-reference to the versionlock).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #514
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
LLDP is the only source of physical switch/port topology in the estate: which switch and which port each machine is cabled to exists nowhere else (not in intent YAML, not in any other fact, not in DHCP/DNS). NetBox needs it to record device interface -> switch/port connections. This installs and enables lldpd on physical hosts and exposes its neighbour data as a structured `lldp` fact for the terraform-infra pdbmux backfill to consume (linked issue below).
## Changes
- Add `profiles::lldpd`: install the `lldpd` package and enable+start the service (it ships disabled), the service subscribing to the package.
- Assign it physicals-only via `hiera_include` in `hieradata/virtual/physical.yaml` (`facts.virtual == 'physical'`), merged `unique` with the common `hiera_include`. VMs never receive it.
- Add the `lldp` custom fact (`modules/libs/lib/facter/lldp.rb`): parse `lldpctl -f json0` into a per-interface map of `{neighbor_chassis_name, neighbor_chassis_mac, neighbor_chassis_descr, neighbor_port_id, neighbor_port_descr, vlan_id, vlan_name}`, skipping interfaces with no neighbour. Confined to physical Linux hosts that have `lldpctl` and a live lldpd socket; returns an empty hash on any error so it can never break a puppet run.
json0 is used because it array-wraps every node regardless of cardinality, so one neighbour and many neighbours parse identically (plain `keyvalue` folds the neighbour SysName into the key path; plain `json` collapses single-element arrays into objects).
## Sample fact output (prodnxsr0019, enp1s0)
```json
{
"enp1s0": {
"neighbor_chassis_name": "sg3429x-m2-02",
"neighbor_chassis_mac": "b8:fb:b3:a7:f9:5d",
"neighbor_chassis_descr": "48-Port Gigabit Smart Managed Pro Switch",
"neighbor_port_id": "24",
"neighbor_port_descr": "two-gigabitEthernet 1/0/11",
"vlan_id": "201",
"vlan_name": "fabric_common_02"
}
}
```
## Consumer
NetBox population of the switch/port cabling is tracked in terraform-infra: unkin/terraform-infra#7 (extend the pdbmux backfill / PR #6 to read this fact and emit the connection).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #513
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Four newly-provisioned el9_8 compute nodes (prodnxsr0014/0015/0018/0019) hang with canal's kube-flannel container in `ImagePullBackOff`: the flannel VXLAN overlay never comes up, so the node can't reach any in-cluster `198.18.200.x` LoadBalancer VIP.
Root cause is a stale airgap-preload version. The nodes boot rke2 **v1.33.11+rke2r1** from the rolling `latest/1.33` repo, whose canal manifest requests `hardened-flannel:v0.28.4` / `hardened-calico:v3.31.5` (build20260415). But `rke2::install` pinned the preloaded bundle to **v1.33.4+rke2r1** (flannel v0.27.2 / calico v3.30.2), so those images were never on disk. containerd then falls back to the `docker.io` mirror (artifactapi, `disable-default-registry-endpoint: true`), reachable only via the pod-overlay VIP that requires the very flannel image being pulled — a bootstrap deadlock. Older nodes (0001-0008) are versionlocked at 1.33.4 and still match their original bundle, so they were unaffected.
## Changes
- Bump pinned `rke2_version` `1.33.4` -> `1.33.11` so the versionlock, RPM `ensure`, and preloaded bundle all line up with the canal image tags the running binary requests. The default `rke2-images.linux-amd64.tar.zst` bundle already contains the canal CNI images (it is RKE2's default CNI), so no extra tarball is needed.
- Wire the airgap archive `source` to the `container_archive_source` class parameter (previously declared in the module but never consumed). The module keeps its generic upstream default; the artifactapi override (the pre-CNI-reachable source, same BGP/physical path the rke2 yum repos already use) lives in the k8s role hiera as `rke2::container_archive_source`.
Applies to servers and agents alike (`rke2::install` runs for both) and preloads `before => Service`, so the bundle lands before rke2 starts.
Notes:
- The `latest/1.33` repo is rolling, so the pinned version must be maintained as the repo advances; a follow-up to pin the yum channel to a fixed patch would remove the drift entirely.
- No terraform-artifactapi change is required. (If a canal-only preload were ever wanted, the github generic remote allowlist would need `rancher/rke2/.*/rke2-images-canal.linux-amd64.tar.zst$` added — but the default bundle already carries those images, so it is unnecessary.)
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #512
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
A forthcoming profile needs to set the I/O scheduler to noop/none on the disks backing ceph OSDs, so Puppet must know which block devices those are. This fact surfaces them by inspecting LVM PVs whose volume group belongs to ceph.
- add ceph_osd_devices fact returning the sorted array of PV device paths whose VG name starts with ceph-
- confine the fact to physical Linux hosts with pvs installed so it never resolves on VMs
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #504
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- Replace static `registries.yaml` with EPP template driven by `rke2::registries` hash
- Add `disable-default-registry-endpoint: true` to all mirrors — RKE2 will only use artifactapi and never fall back to upstream registries
- Registry configuration now fully managed via hiera data (`roles/infra/k8s.yaml`)
Reviewed-on: #474
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Add/Remove the registries.yaml file based on the manage_registries
boolean. We are leaving it on default=false now as the artifactapi
server was broken.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #472
## Summary
- Adds `Unkin::Ceph::Utils` facter module detecting ceph service instances via `systemctl list-units`, exposing `is_ceph_mon`, `is_ceph_mgr`, `is_ceph_mds`, `is_ceph_osd` booleans and a `ceph_services` hash of unit names
- Adds `profiles::ceph::mon`, `mgr`, `mds`, `osd` — each with `Boolean $ensure_running` that iterates discovered service instances and manages them as running and enabled
- Works across incus nodes (mon/mgr/mds/osd) and k8s compute/control nodes (osd only); verified on prodnxsr0001 which correctly reports `is_ceph_osd: true` and `ceph_services: {osd: [ceph-osd@5]}`
## Test plan
- [x] Noop deploy against prodnxsr0001.main.unkin.net passed cleanly
- [x] `ceph_services` fact returns correct service map
- [x] `is_ceph_osd` returns `True`, `is_ceph_mon` returns `False` as expected
- [x] Test on an incus/ceph node with mon/mgr/mds services
Reviewed-on: #459
Replace deprecated dalen-puppetdbquery module with native puppetdb_query
function using PQL syntax to resolve URI.escape compatibility issues.
This is required to migrated to Puppet 8 (and kubernetes).
Changes:
- Remove dalen-puppetdbquery dependency from Puppetfile
- Replace query_nodes() calls with puppetdb_query() using PQL syntax
- Update 27 function calls across 18 Puppet manifests
- Maintain equivalent functionality with improved compatibility
Reviewed-on: #457
- add module to manage externaldns bind for k8s
- add infra::dns::externaldns role
- add 198.18.19.20 as anycast for k8s external-dns service
Reviewed-on: #428
- add SMTP submission listener on port 587 with TLS requirement
- configure HAProxy frontend/backend for submission with send-proxy-v2 support
- add send-proxy-v2 support to all listeners
- add dynamic HAProxy node discovery for proxy trusted networks
- use service hostname instead of node FQDN for autoconfig/autodiscover
- remove redundant IMAP/IMAPS/SMTP alt-names from TLS certificates
- update VRRP CNAME configuration to use mail.main.unkin.net
Reviewed-on: #425
- add fact to export vault public cert from agents
- add fact to export list of trusted incus client certs
- add method for incus clients to export their client cert to be trusted
Reviewed-on: #406
- replace default incus certificates with vault-generated ephemeral certificates
- configure incus service to restart on certificate changes
Reviewed-on: #405
- only install a base config
- wait for 3 masters before deploying helm charts
- remove cluster-domain
- manage nginx ingres via rke2 helmconfig
Reviewed-on: #403
- cattle-system namespace is created earlier than helm
- leave namespaces.yaml to manage cattle-system namespace (required
before installing helm/rancher)
Reviewed-on: #399
This change will install rancher, purelb and cert-manager, then
configure a dmz and common ip pool to be used by loadbalancers. The
nginx ingres controller is configured to use 198.18.200.0 (common) and
announce the ip from all nodes so that it becomes an anycast ip in ospf.
- manage the install of rancher, purelb and cert-manager
- add rancher ingress routes
- add nginx externalip/loadBalancer
Reviewed-on: #395
- enable changing the source address for learned ospf routes
- this enables the loopback0 interface to be used as a default src address
- ensure k8s nodes use loopback0 as default src
- ensure incus nodes use loopback0 as default src
Reviewed-on: #388
- create module class for journald clients
- ensure module class it used on all hosts
- use consul service address for insert/journald
Reviewed-on: #377
- manage vmstorage package, service and environment file
- manage vmselect package, service and environment file
- manage vminsert package, service and environment file
- manage vmagent package, service and environment file
- manage options for vmstorage, vmselect, vminsert, vmagent role
Reviewed-on: #363
- enable enough ceph/frr to join to cephfs
- notify sshd when restarting the network
- update ssh principals to include all ssh interfaces
Reviewed-on: #362
unkinben
benvin