## Why
Plugin 0.2.0 adds per-role method scoping; the binary must land on the vault nodes before the catalog sha256 bump in terraform-vault.
## How
- Bump the `openbao-plugin-secrets-arrstack` pin in `hieradata/roles/infra/storage/vault.yaml` from `0.1.0` to `0.2.0`.
Reviewed-on: #522
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The new `openbao-plugin-secrets-arrstack` OpenBao secrets engine mints dynamic per-service arrproxy machine tokens for the arrstack (Sonarr/Radarr/Prowlarr) fleet. Its v0.1.0 RPM is now published in artifactapi (rpm-internal), so the OpenBao VM nodes can install the plugin binary into `/opt/openbao-plugins`.
## Changes
- Add `openbao-plugin-secrets-arrstack` pinned to `0.1.0` to `profiles::packages::include` in `hieradata/roles/infra/storage/vault.yaml`, matching the existing pinned sibling plugins (litellm, gpg, rancher, gitea, apptoken, netbox, ghp).
This PR only installs the RPM binary on the nodes. Catalog registration is handled separately by terraform-vault (`config/plugins/*.yaml`).
## Reference
Plugin binary sha256 (used by the terraform-vault catalog registration, separate PR):
```
f8ee60ca7ba14819976acb7dc4cfb6799e3e8da8f871d0bb2bd18d1d9e537972
```
Reviewed-on: #521
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: step 1 of adding the ghp Vault secrets engine — the plugin binary must be on the OpenBao nodes before terraform-vault registers it in the catalog (the catalog sha256 pins this exact build).
How: pin `openbao-plugin-secrets-ghp` ensure 0.1.0 in hieradata/roles/infra/storage/vault.yaml profiles::packages::include, alongside the other secrets plugins.
Ordering: this merges + puppet-applies first; then the config-write policy; then the terraform-vault engine resources (terraform-vault#121).
Reviewed-on: #520
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The litellm secrets engine on the OpenBao cluster died with `rpc Unavailable / dial unix /tmp/pluginNNN: no such file` (terraform-vault#112), fixed only by a manual `sys/plugins/reload/backend`.
Root cause (post-incident log audit): go-plugin puts each plugin's control socket under the process TMPDIR — `/tmp/pluginNNN` — and `vault.service` runs without PrivateTmp. The daily `systemd-tmpfiles-clean` reaps aged `/tmp` files; bao ran a single systemd invocation for 3+ weeks, so the socket long outlived the `/tmp` cleanup age and got deleted out from under the still-running plugin process (no panic/OOM/signal/exit in the bao journal — the process was healthy, just unreachable). The risk is shared by every OpenBao plugin (gpg, rancher, gitea, ...), not just litellm.
Supersedes the earlier tmpfiles-exclude approach (#508, closed) with the permanent fix: move the sockets off `/tmp` entirely.
## Change
- Add a `vault.service` drop-in (`systemd::manage_dropin`) that sets `Environment=TMPDIR=/run/vault-plugins` and `RuntimeDirectory=vault-plugins` (mode 0700).
- Point plugin sockets at `/run` (tmpfs, no age-based cleanup); `RuntimeDirectory` creates/owns the dir per service start.
- Notify a vault service restart (module handles daemon-reload) so the new TMPDIR takes effect and plugins respawn with sockets under `/run`.
## Heads-up
Puppet rolls a **bao restart per node** when this lands (the drop-in notifies `Service['vault']`). With auto-unseal (this cluster runs `profiles::vault::unseal`) it is a rolling non-event; if any node relies on manual unseal it will come back **sealed** and need unsealing. Merge consciously / stagger if needed.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #509
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
terraform-vault #115 registers `vault-plugin-secrets-netbox` in the OpenBao plugin catalog pinned to the released v0.1.0 binary's sha256. OpenBao only registers/mounts a plugin whose binary is physically present in the plugin_directory (`/opt/openbao-plugins`); the `openbao-plugin-secrets-netbox` RPM is published on rpm-internal but not yet installed on the bao nodes, so #115 fails to register until Puppet lays the binary down. The `openbao-plugin-secrets-apptoken` binary is likewise published but missing from the bao package list, deferred from the apptoken engine's earlier deploy follow-up.
## Changes
- Add `openbao-plugin-secrets-netbox` (pinned `0.1.0`) to the bao node `profiles::packages::include` list
- Add `openbao-plugin-secrets-apptoken` (pinned `0.1.0`) to the same list, alongside the existing gpg/litellm/rancher/gitea plugin RPMs
- Both install to `/opt/openbao-plugins`, matching the plugin_directory OpenBao reads and the sha256 registered in terraform-vault
## Ordering
This must apply (Puppet converge on the `roles::infra::storage::vault` nodes) BEFORE terraform-vault #115 is applied, otherwise #115's catalog registration fails on a missing binary.
Reviewed-on: #519
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.
## How hosts are identified (osd-only vs mon/mgr/mds)
The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):
- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.
The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.
## Changes
- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.
## Live-cluster safety review (please read before merge)
This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.
- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
- osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
- mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
- prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.
Rendered samples:
- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.
Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Every puppet run on k8s nodes (e.g. prodnxsr0002) fails on `Package[rke2-server]` and stops applying the rest of the catalog, so the nodes stop receiving all further package/config updates:
```
change from '1.33.4~rke2r1-1.el9' to '1.33.11~rke2r1' failed: Could not update:
dnf upgrade rke2-server-1.33.11~rke2r1 returned 1:
Problem: problem with installed package rke2-common-1.33.13~rke2r2-0.el9.x86_64
- package rke2-server-1.33.11~rke2r1 requires rke2-common = 1.33.11~rke2r1, but none of the providers can be installed
- cannot install the best update candidate for package rke2-server
```
`rke2::install` versionlocks only `rke2-server`/`rke2-agent`, never their strict (`= version`) `rke2-common` dependency. `rke2-common` is served from the rolling `rancher-rke2-1.33-latest` channel, whose head is now `1.33.13~rke2r2`, so `rke2-common` drifted up to `1.33.13~rke2r2` while the pin (`#512`) sat at `1.33.11~rke2r1`. `dnf upgrade` cannot downgrade the newer `rke2-common` to satisfy the older server, so the transaction fails. This is the rolling-channel drift `#512` flagged as needing follow-up.
## How
- Versionlock `rke2-common` to the same `${rke2_version}~${rke2_release}` as the server/agent, so the rolling channel can no longer drift the dependency ahead of the pin.
- Bump `rke2_version` `1.33.11` -> `1.33.13` and `rke2_release` `rke2r1` -> `rke2r2` to match the current channel head and the already-drifted installed `rke2-common`, so the pinned server/agent, both versionlocks, and the preloaded airgap bundle resolve in one transaction.
Verified against the live artifactapi rke2 remote: `rke2-server-1.33.13~rke2r2-0.el9.x86_64.rpm` and the `v1.33.13+rke2r2` `rke2-images.linux-amd64.tar.zst` airgap bundle both serve HTTP 200.
Reviewed-on: #516
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Ceph OSDs manage their own I/O ordering, so the kernel scheduler on the backing disks just adds overhead. The original intent was to set those disks to the `noop` scheduler. The whole OSD fleet (k8s + incus nodes) runs AlmaLinux 9 on blk-mq kernels (5.14), where the equivalent of `noop` is `none`.
## Changes
- Add `profiles::ceph::osd_scheduler`, rendering a udev rule from the `ceph_osd_devices` fact (PR #504) that pins `queue/scheduler` to `none` on each OSD disk.
- Reload udev and trigger the matched block devices so the setting applies immediately; the udev rule keeps it set across reboots and device re-add.
- No-op when the fact is absent/empty, so VMs and non-OSD hosts are untouched.
- Include the class from `profiles::ceph::osd` so it lands only on OSD hosts.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #505
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
A package versionlock is only effective if it is applied **before** the package resource is evaluated. When the package is upgraded/installed first and locked afterwards, dnf can pull an unpinned version and the lock then pins whatever landed. This is the exact drift that let new RKE2 nodes boot 1.33.11 instead of the pinned version, and it also matters because a global `dnf update -y` autoupdate timer (`profiles::yum::autoupdater`) relies on the versionlock plugin already having the lock written.
## Changes
- `rke2::install` orders the versionlock ahead of the package: adds `before => Package["rke2-${node_type}"]` so the lock exists before install/upgrade.
- `profiles::puppet::agent` (RedHat branch) adds `before => Package[$use_package]` on the versionlock for the same reason.
- Leaves `profiles::gitea::runner` and `profiles::vault::server` unchanged: both already order the versionlock ahead of the package (`before => Package` / `before => Class['vault']`).
- Changes ordering only; locked versions (including the RKE2 pin) are untouched. No dependency cycle is introduced (packages hold no back-reference to the versionlock).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #514
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
LLDP is the only source of physical switch/port topology in the estate: which switch and which port each machine is cabled to exists nowhere else (not in intent YAML, not in any other fact, not in DHCP/DNS). NetBox needs it to record device interface -> switch/port connections. This installs and enables lldpd on physical hosts and exposes its neighbour data as a structured `lldp` fact for the terraform-infra pdbmux backfill to consume (linked issue below).
## Changes
- Add `profiles::lldpd`: install the `lldpd` package and enable+start the service (it ships disabled), the service subscribing to the package.
- Assign it physicals-only via `hiera_include` in `hieradata/virtual/physical.yaml` (`facts.virtual == 'physical'`), merged `unique` with the common `hiera_include`. VMs never receive it.
- Add the `lldp` custom fact (`modules/libs/lib/facter/lldp.rb`): parse `lldpctl -f json0` into a per-interface map of `{neighbor_chassis_name, neighbor_chassis_mac, neighbor_chassis_descr, neighbor_port_id, neighbor_port_descr, vlan_id, vlan_name}`, skipping interfaces with no neighbour. Confined to physical Linux hosts that have `lldpctl` and a live lldpd socket; returns an empty hash on any error so it can never break a puppet run.
json0 is used because it array-wraps every node regardless of cardinality, so one neighbour and many neighbours parse identically (plain `keyvalue` folds the neighbour SysName into the key path; plain `json` collapses single-element arrays into objects).
## Sample fact output (prodnxsr0019, enp1s0)
```json
{
"enp1s0": {
"neighbor_chassis_name": "sg3429x-m2-02",
"neighbor_chassis_mac": "b8:fb:b3:a7:f9:5d",
"neighbor_chassis_descr": "48-Port Gigabit Smart Managed Pro Switch",
"neighbor_port_id": "24",
"neighbor_port_descr": "two-gigabitEthernet 1/0/11",
"vlan_id": "201",
"vlan_name": "fabric_common_02"
}
}
```
## Consumer
NetBox population of the switch/port cabling is tracked in terraform-infra: unkin/terraform-infra#7 (extend the pdbmux backfill / PR #6 to read this fact and emit the connection).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #513
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Four newly-provisioned el9_8 compute nodes (prodnxsr0014/0015/0018/0019) hang with canal's kube-flannel container in `ImagePullBackOff`: the flannel VXLAN overlay never comes up, so the node can't reach any in-cluster `198.18.200.x` LoadBalancer VIP.
Root cause is a stale airgap-preload version. The nodes boot rke2 **v1.33.11+rke2r1** from the rolling `latest/1.33` repo, whose canal manifest requests `hardened-flannel:v0.28.4` / `hardened-calico:v3.31.5` (build20260415). But `rke2::install` pinned the preloaded bundle to **v1.33.4+rke2r1** (flannel v0.27.2 / calico v3.30.2), so those images were never on disk. containerd then falls back to the `docker.io` mirror (artifactapi, `disable-default-registry-endpoint: true`), reachable only via the pod-overlay VIP that requires the very flannel image being pulled — a bootstrap deadlock. Older nodes (0001-0008) are versionlocked at 1.33.4 and still match their original bundle, so they were unaffected.
## Changes
- Bump pinned `rke2_version` `1.33.4` -> `1.33.11` so the versionlock, RPM `ensure`, and preloaded bundle all line up with the canal image tags the running binary requests. The default `rke2-images.linux-amd64.tar.zst` bundle already contains the canal CNI images (it is RKE2's default CNI), so no extra tarball is needed.
- Wire the airgap archive `source` to the `container_archive_source` class parameter (previously declared in the module but never consumed). The module keeps its generic upstream default; the artifactapi override (the pre-CNI-reachable source, same BGP/physical path the rke2 yum repos already use) lives in the k8s role hiera as `rke2::container_archive_source`.
Applies to servers and agents alike (`rke2::install` runs for both) and preloads `before => Service`, so the bundle lands before rke2 starts.
Notes:
- The `latest/1.33` repo is rolling, so the pinned version must be maintained as the repo advances; a follow-up to pin the yum channel to a fixed patch would remove the drift entirely.
- No terraform-artifactapi change is required. (If a canal-only preload were ever wanted, the github generic remote allowlist would need `rancher/rke2/.*/rke2-images-canal.linux-amd64.tar.zst$` added — but the default bundle already carries those images, so it is unnecessary.)
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #512
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
must change management interfaces for new nodes to start from 21, since
.14 to .20 were used for various anycast services
really need ipam
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #511
# Managed by puppet (profiles::ceph::osd_scheduler).
# Set the I/O scheduler to <%= @scheduler %> on ceph OSD block devices.
<% @kernel_names.sort.each do |dev| -%>
ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="<%= dev %>", ATTR{queue/scheduler}="<%= @scheduler %>"
<% end -%>
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
unkin-agent
unkinben
benvin