The edge gateways sit in BinaryLane VPCs with no consul datacentre and a public interface, so the consul agent cannot join and sshd must not serve the internet.
- add an enable toggle to profiles::consul::client, off for the gateway role
- disable node_exporter on the gateway role, as it registers via consul
- restrict sshd AllowUsers to the edge VPCs (198.18.30.0/24, 198.18.31.0/24) and the lab (198.18.0.0/16, 10.10.12.200/32)
Stacked on #554.
Reviewed-on: #555
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
BinaryLane edge hosts have a public primary IP, so region/environment/country/zone resolve to unknown even though their VPC NIC sits in a known subnet.
- match the primary IP first, unchanged for existing hosts
- otherwise scan interface bindings in name order, skipping lo, dum* and wg*
- add modules/libs/test/subnet_facts_check.rb (`ruby modules/libs/test/subnet_facts_check.rb`)
Reviewed-on: #557
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The mail gateway config was copied from an eFa appliance and predates the edge design: unfiltered/dead DNSBLs, a MailScanner milter, no local filtering, and no edge identity.
- filter postscreen DNSBL codes, add mailspike/psbl, drop sorbs/swl; dnswl clients skip greet tests
- run local rspamd (deterministic checks, redis bayes, edge.cf rules fetched each puppet run) as a tempfail milter
- set a 50MB size limit; drop qmqpd, 3DES and eFa leftovers
- relay unkin.net and main.unkin.net to [mail-in.main.unkin.net]:25 with verified TLS and recipient probing
- set myhostname and vault cert SANs per node (mta1/mta2.edge.unkin.net)
Reviewed-on: #552
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The BinaryLane mail edge DMZ VPCs need environment, region, country and zone facts like every other lab subnet.
- map 198.18.30.0/24 to prod/syd2/au/dmz (binarylane syd dmz vpc)
- map 198.18.31.0/24 to prod/mel1/au/dmz (binarylane mel dmz vpc)
Reviewed-on: #556
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Edge gateways need cheap deterministic in-session filtering behind the postfix milter, with no AI and no ClamAV on 1GB VMs.
- add a signature-checked rspamd yum repo via artifactapi, enabled for the gateway role
- add profiles::rspamd::edge managing the package and local.d files from hiera
- fetch rule files without blocking the service: empty placeholder, swap only on a successful changed fetch, then reload
- run a self-scanning milter proxy on 11332; reject at 13, header at 6, no greylisting, subject rewrite or gpt
- add a local 128MB redis for bayes autolearn; load SpamAssassin rules from rspamd-rules
- bypass rspamd for mail relayed out by the smarthost
Stacked on #552; requires terraform-artifactapi #63 applied first.
Reviewed-on: #553
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
WAN federation through mesh gateways makes Consul verify server certs against `server.<dc>.consul` and `<node_name>.server.<dc>.consul`. Node names are FQDNs, so each server needs its exact SAN. This adds the SANs ahead of Consul server TLS, which a stacked follow-up enables once the reissued certs are verified.
- add `server.<country>-<region>.consul` to consul server alt_names
- add `<fqdn>.server.<country>-<region>.consul` to consul server alt_names
Reviewed-on: #547
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Move the k8s roles to the puppet-on-k8s compilers. Their role code (rke2, frrouting, ceph) has no exported resources or PuppetDB queries; the base consul lookup is covered by the static syd1 server list.
- enable profiles::puppet::migrate in the shared roles::infra::k8s hieradata (compute, control, node)
Reviewed-on: #545
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Incus nodes still run against the VM puppetservers; move them to the k8s compilers as part of the puppet-on-k8s migration.
- enable profiles::puppet::migrate for roles::infra::incus::node
Reviewed-on: #546
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Hosts compiled on the k8s puppetservers can't see the consul servers in PuppetDB, so the members lookup returns nothing and syd1 clients get no retry_join targets.
- set a static consul server list for syd1 and disable the members lookup there
- keep the PuppetDB members lookup for all other regions
Reviewed-on: #544
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
nzbget defaulted bind_address and port from sonarr::params, which is not loaded on nzbget-only nodes, so catalog compilation failed with "Unknown variable".
- default bind_address/port from nzbget::params
Reviewed-on: #543
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The ausyd1nxvm2120 canary is healthy on the puppet-on-k8s servers, so move the next role wave across: the media nodes (ausyd1nxvm2045-2051: nzbget, sonarr, radarr, lidarr, readarr, prowlarr, jellyfin).
- enable `profiles::puppet::migrate` in `roles/apps/media.yaml`
Reviewed-on: #540
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The haproxy2 VMs are being decommissioned and the site names they front move to bind-internal DNSRecord CRs in argocd-apps. dns-updater on these hosts deletes records that leave its records file, so the CNAMEs drop once this applies.
- empty profiles::haproxy::dns::vrrp_cnames for haproxy2
- empty profiles::haproxy::dns::cnames for haproxy2
Reviewed-on: #542
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Effective rp_filter is max(all, <iface>). Setting `all`/`default` to 0 is not enough after a reboot: `/usr/lib/sysctl.d/50-redhat.conf` sets `net.ipv4.conf.*.rp_filter = 1`, which udev applies to every interface as it is created, so routed asymmetric traffic gets dropped once shorewall goes.
- add `net.ipv4.conf.*.rp_filter: 0` to the router role, overriding the vendor glob
- disable `enforce` for it, since `sysctl -n` cannot read glob keys
Reviewed-on: #539
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
certmanager and sshsignhost are now Go binaries released as RPMs, and their packaged paths collide with the venv install these helper classes manage.
- Drop the pyvenv, pip, rendered script and /usr/local/bin symlink resources
- Delete the now-unused Python script templates
- Keep rendering /opt/<tool>/config.yaml, unchanged ownership and mode
- Nest certmanager's output_path under vault:, where the binary reads it
- Drop output_path from sshsignhost's config; the binary has no such key
- Pin certmanager to 0.2.0 and sshsignhost to 0.1.0 on the puppet master role
- Point sshsignhost at the sshca mount and signhost role, documented in doc/vault
Reviewed-on: #524
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
WireGuard on the router is configured by hand, so its tunnels are not reproducible from code. This adds a module to manage it from hieradata.
- add `wireguard` class to install wireguard-tools and manage interfaces from a hash
- add `wireguard::interface` to render `/etc/wireguard/<iface>.conf` (0600) and enable `wg-quick@<iface>`
- keep private and preshared keys `Sensitive` end to end (`wireguard::interfaces` lookup_options `convert_to: Sensitive`, typed peer Struct)
- without `private_key`, generate `/etc/wireguard/<iface>.key` (0600) only if absent and load it via PostUp, so the key never rotates
- apply config changes with `wg syncconf` instead of restarting the tunnel
Reviewed-on: #538
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The router runs dnsmasq as a DNS forwarder and DHCP relay from hand-edited config, so its state is not reproducible from code.
- add `dnsmasq` module installing the package, rendering `/etc/dnsmasq.conf` from class params and running the service
- cover listen addresses/interfaces, bind mode, upstream servers, no-resolv, cache-size, domain-needed, bogus-priv, per-domain forwards and dhcp-relay
- add raw `options` lines for anything else
- add `purge_config_dir` (default off) to remove unmanaged `/etc/dnsmasq.d` files
Reviewed-on: #537
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
prodnxsr0020 runs FRR/OSPF hand-configured; bring its routing config under puppet without touching interfaces, firewall or dnsmasq.
- add roles::infra::network::router (base + frrouting + frr_exporter)
- enable ip_forward and disable rp_filter via sysctl::base
- add prodnxsr0020 OSPF config (dum0, dum1, bond0.201; src 198.18.21.160)
- pin dns, consul and router-id to dum0 instead of the WAN-facing primary IP
- listen sshd on 127.0.0.1 and dum0 only, knocking out the common WAN primary IP
- keep resolv.conf on the local dnsmasq (127.0.0.1)
Reviewed-on: #536
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Router loopbacks in 198.18.2.0/24 (e.g. prodnxsr0020, 198.18.2.160) match no subnet entry, so they get unknown environment/region/country facts and are not autosigned.
- add 198.18.2.0/24 to subnet_facts as prod/syd1/au/common
- add 198.18.2.0/24 to puppet master autosign subnet_ranges
Reviewed-on: #535
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Estate journald ingestion has been down since 00:33Z. `systemd-journal-upload` appends `:19532/upload` to the configured URL whenever that URL carries no explicit port, so the portless k8s endpoint became `/insert/journald:19532/upload`, which vlinsert rejects as an unsupported path. The previous consul URL only worked because `:9428` was explicit.
- pin `victorialogs::client::journald::inserturl` to port 443
Requests then land on `/insert/journald/upload`, which vlinsert accepts.
Reviewed-on: #533
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The estate ships journald to the VM VictoriaLogs cluster, which is being left to age out rather than grow. New log capacity lands in k8s, so clients need to point there while the VM cluster keeps serving historical queries until its retention lapses.
- repoint victorialogs::client::journald::inserturl at https://logs-ingest.k8s.syd1.au.unkin.net/insert/journald
VMs already trust the issuing CA via the system bundle, so journal-upload needs no TLS change.
Requires the k8s VLCluster deployed and serving /insert/journald first.
Reviewed-on: #531
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The Kubernetes-hosted haproxy edge probes the arr/nzbget backends on `/consul/health`, but its traffic arrives SNATed from the node ranges rather than the DMZ edge, so nginx returns 403 and every backend health-checks down.
- Allow `198.18.21.0/24` (2.5gbe physical), `198.18.15.0/24` and `198.18.19.0/24` (node loopbacks) on the `arrstack_web_healthcheck` location.
- Keep `198.18.24.0/24` so the existing DMZ edge stays healthy through cutover.
---------
Co-authored-by: unkin-agent <agent@unkin.net>
Reviewed-on: #532
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Catalog compilation moved to the k8s puppetserver compilers, which sign host certificates against the terraform-managed `sshca` mount. Clients only trust the legacy `ssh-host-signer` CA, so every re-signed node (ausyd1nxvm2120 already) presents a certificate nothing accepts, and knownhosts emits no plain host-key fallback.
- Add a second `@cert-authority *` entry for the `sshca` public key to `profiles::ssh::knownhosts::lines`.
- Keep the legacy entry untouched so legacy-signed hosts still verify.
Reviewed-on: #530
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Facter runs under Puppet's vendored Ruby, which reads its own bundled CA file and never the system trust store. The ENC fact now fetches over HTTPS, so every node fails certificate verification and falls back to its cached value.
- set ca_file on the request to the vaultca anchor bundle
- keep VERIFY_PEER on, and fall through to the existing cache path when the anchor is absent
Reviewed-on: #529
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The enc_role and enc_env facts still resolve against Cobbler on every agent, the last Cobbler dependency in the classification path now that the master-side ENC runs encapic-enc.
- Point the fact at https://encapi.k8s.syd1.au.unkin.net
- Rename the module and its messages from Cobbler to encapi
Cache file, TTL and fallback-to-cache failure behaviour are unchanged.
Reviewed-on: #528
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The VM puppet masters are the last part of the classification path still calling Cobbler; the k8s compilers already classify through encapi and the encapic RPM is installed on all six masters.
- Point `profiles::puppet::server::external_nodes` at `/usr/bin/encapic-enc` for `roles::infra::puppet::master`.
- Drop the stale comment about external_nodes still using cobbler-enc.
`profiles::puppet::cobbler_enc` stays in place so the revert is one hiera line.
Depends on the encapic 0.2.0 install (#525).
Reviewed-on: #527
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A package pinned in hieradata right after its RPM lands in artifactapi is invisible to dnf until the host's 1h cached metadata expires, so the first Puppet run after a release cannot find the version.
- Set `metadata_expire` 60s on `rpm-internal`/`rpm-vendor` and their per-release variants for AlmaLinux and Fedora
- Leave upstream mirrors on the 1h default
- Drop the stale expiry note in `profiles::dns::updater`
Reviewed-on: #526
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The VM masters classify through the cobbler ENC while the k8s compilers
already use encapi. Install the client ahead of that cutover;
external_nodes still points at cobbler-enc, so classification is unchanged.
- pin the encapic package to 0.2.0 via profiles::packages::include
- add profiles::puppet::encapic managing /etc/encapic/encapic.conf from a
hiera-driven ENCAPI_URL, ordered after Package['encapic'] so the config is
written once the RPM that owns the path is installed
- include the class from profiles::puppet::puppetmaster
Requires encapic 0.2.0 in the rpm-internal repo.
Reviewed-on: #525
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Plugin 0.2.0 adds per-role method scoping; the binary must land on the vault nodes before the catalog sha256 bump in terraform-vault.
## How
- Bump the `openbao-plugin-secrets-arrstack` pin in `hieradata/roles/infra/storage/vault.yaml` from `0.1.0` to `0.2.0`.
Reviewed-on: #522
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The new `openbao-plugin-secrets-arrstack` OpenBao secrets engine mints dynamic per-service arrproxy machine tokens for the arrstack (Sonarr/Radarr/Prowlarr) fleet. Its v0.1.0 RPM is now published in artifactapi (rpm-internal), so the OpenBao VM nodes can install the plugin binary into `/opt/openbao-plugins`.
## Changes
- Add `openbao-plugin-secrets-arrstack` pinned to `0.1.0` to `profiles::packages::include` in `hieradata/roles/infra/storage/vault.yaml`, matching the existing pinned sibling plugins (litellm, gpg, rancher, gitea, apptoken, netbox, ghp).
This PR only installs the RPM binary on the nodes. Catalog registration is handled separately by terraform-vault (`config/plugins/*.yaml`).
## Reference
Plugin binary sha256 (used by the terraform-vault catalog registration, separate PR):
```
f8ee60ca7ba14819976acb7dc4cfb6799e3e8da8f871d0bb2bd18d1d9e537972
```
Reviewed-on: #521
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: step 1 of adding the ghp Vault secrets engine — the plugin binary must be on the OpenBao nodes before terraform-vault registers it in the catalog (the catalog sha256 pins this exact build).
How: pin `openbao-plugin-secrets-ghp` ensure 0.1.0 in hieradata/roles/infra/storage/vault.yaml profiles::packages::include, alongside the other secrets plugins.
Ordering: this merges + puppet-applies first; then the config-write policy; then the terraform-vault engine resources (terraform-vault#121).
Reviewed-on: #520
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The litellm secrets engine on the OpenBao cluster died with `rpc Unavailable / dial unix /tmp/pluginNNN: no such file` (terraform-vault#112), fixed only by a manual `sys/plugins/reload/backend`.
Root cause (post-incident log audit): go-plugin puts each plugin's control socket under the process TMPDIR — `/tmp/pluginNNN` — and `vault.service` runs without PrivateTmp. The daily `systemd-tmpfiles-clean` reaps aged `/tmp` files; bao ran a single systemd invocation for 3+ weeks, so the socket long outlived the `/tmp` cleanup age and got deleted out from under the still-running plugin process (no panic/OOM/signal/exit in the bao journal — the process was healthy, just unreachable). The risk is shared by every OpenBao plugin (gpg, rancher, gitea, ...), not just litellm.
Supersedes the earlier tmpfiles-exclude approach (#508, closed) with the permanent fix: move the sockets off `/tmp` entirely.
## Change
- Add a `vault.service` drop-in (`systemd::manage_dropin`) that sets `Environment=TMPDIR=/run/vault-plugins` and `RuntimeDirectory=vault-plugins` (mode 0700).
- Point plugin sockets at `/run` (tmpfs, no age-based cleanup); `RuntimeDirectory` creates/owns the dir per service start.
- Notify a vault service restart (module handles daemon-reload) so the new TMPDIR takes effect and plugins respawn with sockets under `/run`.
## Heads-up
Puppet rolls a **bao restart per node** when this lands (the drop-in notifies `Service['vault']`). With auto-unseal (this cluster runs `profiles::vault::unseal`) it is a rolling non-event; if any node relies on manual unseal it will come back **sealed** and need unsealing. Merge consciously / stagger if needed.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #509
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
terraform-vault #115 registers `vault-plugin-secrets-netbox` in the OpenBao plugin catalog pinned to the released v0.1.0 binary's sha256. OpenBao only registers/mounts a plugin whose binary is physically present in the plugin_directory (`/opt/openbao-plugins`); the `openbao-plugin-secrets-netbox` RPM is published on rpm-internal but not yet installed on the bao nodes, so #115 fails to register until Puppet lays the binary down. The `openbao-plugin-secrets-apptoken` binary is likewise published but missing from the bao package list, deferred from the apptoken engine's earlier deploy follow-up.
## Changes
- Add `openbao-plugin-secrets-netbox` (pinned `0.1.0`) to the bao node `profiles::packages::include` list
- Add `openbao-plugin-secrets-apptoken` (pinned `0.1.0`) to the same list, alongside the existing gpg/litellm/rancher/gitea plugin RPMs
- Both install to `/opt/openbao-plugins`, matching the plugin_directory OpenBao reads and the sha256 registered in terraform-vault
## Ordering
This must apply (Puppet converge on the `roles::infra::storage::vault` nodes) BEFORE terraform-vault #115 is applied, otherwise #115's catalog registration fails on a missing binary.
Reviewed-on: #519
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.
## How hosts are identified (osd-only vs mon/mgr/mds)
The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):
- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.
The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.
## Changes
- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.
## Live-cluster safety review (please read before merge)
This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.
- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
- osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
- mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
- prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.
Rendered samples:
- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.
Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Every puppet run on k8s nodes (e.g. prodnxsr0002) fails on `Package[rke2-server]` and stops applying the rest of the catalog, so the nodes stop receiving all further package/config updates:
```
change from '1.33.4~rke2r1-1.el9' to '1.33.11~rke2r1' failed: Could not update:
dnf upgrade rke2-server-1.33.11~rke2r1 returned 1:
Problem: problem with installed package rke2-common-1.33.13~rke2r2-0.el9.x86_64
- package rke2-server-1.33.11~rke2r1 requires rke2-common = 1.33.11~rke2r1, but none of the providers can be installed
- cannot install the best update candidate for package rke2-server
```
`rke2::install` versionlocks only `rke2-server`/`rke2-agent`, never their strict (`= version`) `rke2-common` dependency. `rke2-common` is served from the rolling `rancher-rke2-1.33-latest` channel, whose head is now `1.33.13~rke2r2`, so `rke2-common` drifted up to `1.33.13~rke2r2` while the pin (`#512`) sat at `1.33.11~rke2r1`. `dnf upgrade` cannot downgrade the newer `rke2-common` to satisfy the older server, so the transaction fails. This is the rolling-channel drift `#512` flagged as needing follow-up.
## How
- Versionlock `rke2-common` to the same `${rke2_version}~${rke2_release}` as the server/agent, so the rolling channel can no longer drift the dependency ahead of the pin.
- Bump `rke2_version` `1.33.11` -> `1.33.13` and `rke2_release` `rke2r1` -> `rke2r2` to match the current channel head and the already-drifted installed `rke2-common`, so the pinned server/agent, both versionlocks, and the preloaded airgap bundle resolve in one transaction.
Verified against the live artifactapi rke2 remote: `rke2-server-1.33.13~rke2r2-0.el9.x86_64.rpm` and the `v1.33.13+rke2r2` `rke2-images.linux-amd64.tar.zst` airgap bundle both serve HTTP 200.
Reviewed-on: #516
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Ceph OSDs manage their own I/O ordering, so the kernel scheduler on the backing disks just adds overhead. The original intent was to set those disks to the `noop` scheduler. The whole OSD fleet (k8s + incus nodes) runs AlmaLinux 9 on blk-mq kernels (5.14), where the equivalent of `noop` is `none`.
## Changes
- Add `profiles::ceph::osd_scheduler`, rendering a udev rule from the `ceph_osd_devices` fact (PR #504) that pins `queue/scheduler` to `none` on each OSD disk.
- Reload udev and trigger the matched block devices so the setting applies immediately; the udev rule keeps it set across reboots and device re-add.
- No-op when the fact is absent/empty, so VMs and non-OSD hosts are untouched.
- Include the class from `profiles::ceph::osd` so it lands only on OSD hosts.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #505
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
A package versionlock is only effective if it is applied **before** the package resource is evaluated. When the package is upgraded/installed first and locked afterwards, dnf can pull an unpinned version and the lock then pins whatever landed. This is the exact drift that let new RKE2 nodes boot 1.33.11 instead of the pinned version, and it also matters because a global `dnf update -y` autoupdate timer (`profiles::yum::autoupdater`) relies on the versionlock plugin already having the lock written.
## Changes
- `rke2::install` orders the versionlock ahead of the package: adds `before => Package["rke2-${node_type}"]` so the lock exists before install/upgrade.
- `profiles::puppet::agent` (RedHat branch) adds `before => Package[$use_package]` on the versionlock for the same reason.
- Leaves `profiles::gitea::runner` and `profiles::vault::server` unchanged: both already order the versionlock ahead of the package (`before => Package` / `before => Class['vault']`).
- Changes ordering only; locked versions (including the RKE2 pin) are untouched. No dependency cycle is introduced (packages hold no back-reference to the versionlock).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #514
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
LLDP is the only source of physical switch/port topology in the estate: which switch and which port each machine is cabled to exists nowhere else (not in intent YAML, not in any other fact, not in DHCP/DNS). NetBox needs it to record device interface -> switch/port connections. This installs and enables lldpd on physical hosts and exposes its neighbour data as a structured `lldp` fact for the terraform-infra pdbmux backfill to consume (linked issue below).
## Changes
- Add `profiles::lldpd`: install the `lldpd` package and enable+start the service (it ships disabled), the service subscribing to the package.
- Assign it physicals-only via `hiera_include` in `hieradata/virtual/physical.yaml` (`facts.virtual == 'physical'`), merged `unique` with the common `hiera_include`. VMs never receive it.
- Add the `lldp` custom fact (`modules/libs/lib/facter/lldp.rb`): parse `lldpctl -f json0` into a per-interface map of `{neighbor_chassis_name, neighbor_chassis_mac, neighbor_chassis_descr, neighbor_port_id, neighbor_port_descr, vlan_id, vlan_name}`, skipping interfaces with no neighbour. Confined to physical Linux hosts that have `lldpctl` and a live lldpd socket; returns an empty hash on any error so it can never break a puppet run.
json0 is used because it array-wraps every node regardless of cardinality, so one neighbour and many neighbours parse identically (plain `keyvalue` folds the neighbour SysName into the key path; plain `json` collapses single-element arrays into objects).
## Sample fact output (prodnxsr0019, enp1s0)
```json
{
"enp1s0": {
"neighbor_chassis_name": "sg3429x-m2-02",
"neighbor_chassis_mac": "b8:fb:b3:a7:f9:5d",
"neighbor_chassis_descr": "48-Port Gigabit Smart Managed Pro Switch",
"neighbor_port_id": "24",
"neighbor_port_descr": "two-gigabitEthernet 1/0/11",
"vlan_id": "201",
"vlan_name": "fabric_common_02"
}
}
```
## Consumer
NetBox population of the switch/port cabling is tracked in terraform-infra: unkin/terraform-infra#7 (extend the pdbmux backfill / PR #6 to read this fact and emit the connection).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #513
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin-agent
unkinben