The estate ships journald to the VM VictoriaLogs cluster, which is being left to age out rather than grow. New log capacity lands in k8s, so clients need to point there while the VM cluster keeps serving historical queries until its retention lapses.
- repoint victorialogs::client::journald::inserturl at https://logs-ingest.k8s.syd1.au.unkin.net/insert/journald
VMs already trust the issuing CA via the system bundle, so journal-upload needs no TLS change.
Requires the k8s VLCluster deployed and serving /insert/journald first.
Reviewed-on: #531
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The Kubernetes-hosted haproxy edge probes the arr/nzbget backends on `/consul/health`, but its traffic arrives SNATed from the node ranges rather than the DMZ edge, so nginx returns 403 and every backend health-checks down.
- Allow `198.18.21.0/24` (2.5gbe physical), `198.18.15.0/24` and `198.18.19.0/24` (node loopbacks) on the `arrstack_web_healthcheck` location.
- Keep `198.18.24.0/24` so the existing DMZ edge stays healthy through cutover.
---------
Co-authored-by: unkin-agent <agent@unkin.net>
Reviewed-on: #532
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Catalog compilation moved to the k8s puppetserver compilers, which sign host certificates against the terraform-managed `sshca` mount. Clients only trust the legacy `ssh-host-signer` CA, so every re-signed node (ausyd1nxvm2120 already) presents a certificate nothing accepts, and knownhosts emits no plain host-key fallback.
- Add a second `@cert-authority *` entry for the `sshca` public key to `profiles::ssh::knownhosts::lines`.
- Keep the legacy entry untouched so legacy-signed hosts still verify.
Reviewed-on: #530
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The VM puppet masters are the last part of the classification path still calling Cobbler; the k8s compilers already classify through encapi and the encapic RPM is installed on all six masters.
- Point `profiles::puppet::server::external_nodes` at `/usr/bin/encapic-enc` for `roles::infra::puppet::master`.
- Drop the stale comment about external_nodes still using cobbler-enc.
`profiles::puppet::cobbler_enc` stays in place so the revert is one hiera line.
Depends on the encapic 0.2.0 install (#525).
Reviewed-on: #527
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A package pinned in hieradata right after its RPM lands in artifactapi is invisible to dnf until the host's 1h cached metadata expires, so the first Puppet run after a release cannot find the version.
- Set `metadata_expire` 60s on `rpm-internal`/`rpm-vendor` and their per-release variants for AlmaLinux and Fedora
- Leave upstream mirrors on the 1h default
- Drop the stale expiry note in `profiles::dns::updater`
Reviewed-on: #526
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The VM masters classify through the cobbler ENC while the k8s compilers
already use encapi. Install the client ahead of that cutover;
external_nodes still points at cobbler-enc, so classification is unchanged.
- pin the encapic package to 0.2.0 via profiles::packages::include
- add profiles::puppet::encapic managing /etc/encapic/encapic.conf from a
hiera-driven ENCAPI_URL, ordered after Package['encapic'] so the config is
written once the RPM that owns the path is installed
- include the class from profiles::puppet::puppetmaster
Requires encapic 0.2.0 in the rpm-internal repo.
Reviewed-on: #525
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Plugin 0.2.0 adds per-role method scoping; the binary must land on the vault nodes before the catalog sha256 bump in terraform-vault.
## How
- Bump the `openbao-plugin-secrets-arrstack` pin in `hieradata/roles/infra/storage/vault.yaml` from `0.1.0` to `0.2.0`.
Reviewed-on: #522
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The new `openbao-plugin-secrets-arrstack` OpenBao secrets engine mints dynamic per-service arrproxy machine tokens for the arrstack (Sonarr/Radarr/Prowlarr) fleet. Its v0.1.0 RPM is now published in artifactapi (rpm-internal), so the OpenBao VM nodes can install the plugin binary into `/opt/openbao-plugins`.
## Changes
- Add `openbao-plugin-secrets-arrstack` pinned to `0.1.0` to `profiles::packages::include` in `hieradata/roles/infra/storage/vault.yaml`, matching the existing pinned sibling plugins (litellm, gpg, rancher, gitea, apptoken, netbox, ghp).
This PR only installs the RPM binary on the nodes. Catalog registration is handled separately by terraform-vault (`config/plugins/*.yaml`).
## Reference
Plugin binary sha256 (used by the terraform-vault catalog registration, separate PR):
```
f8ee60ca7ba14819976acb7dc4cfb6799e3e8da8f871d0bb2bd18d1d9e537972
```
Reviewed-on: #521
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: step 1 of adding the ghp Vault secrets engine — the plugin binary must be on the OpenBao nodes before terraform-vault registers it in the catalog (the catalog sha256 pins this exact build).
How: pin `openbao-plugin-secrets-ghp` ensure 0.1.0 in hieradata/roles/infra/storage/vault.yaml profiles::packages::include, alongside the other secrets plugins.
Ordering: this merges + puppet-applies first; then the config-write policy; then the terraform-vault engine resources (terraform-vault#121).
Reviewed-on: #520
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
terraform-vault #115 registers `vault-plugin-secrets-netbox` in the OpenBao plugin catalog pinned to the released v0.1.0 binary's sha256. OpenBao only registers/mounts a plugin whose binary is physically present in the plugin_directory (`/opt/openbao-plugins`); the `openbao-plugin-secrets-netbox` RPM is published on rpm-internal but not yet installed on the bao nodes, so #115 fails to register until Puppet lays the binary down. The `openbao-plugin-secrets-apptoken` binary is likewise published but missing from the bao package list, deferred from the apptoken engine's earlier deploy follow-up.
## Changes
- Add `openbao-plugin-secrets-netbox` (pinned `0.1.0`) to the bao node `profiles::packages::include` list
- Add `openbao-plugin-secrets-apptoken` (pinned `0.1.0`) to the same list, alongside the existing gpg/litellm/rancher/gitea plugin RPMs
- Both install to `/opt/openbao-plugins`, matching the plugin_directory OpenBao reads and the sha256 registered in terraform-vault
## Ordering
This must apply (Puppet converge on the `roles::infra::storage::vault` nodes) BEFORE terraform-vault #115 is applied, otherwise #115's catalog registration fails on a missing binary.
Reviewed-on: #519
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.
## How hosts are identified (osd-only vs mon/mgr/mds)
The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):
- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.
The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.
## Changes
- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.
## Live-cluster safety review (please read before merge)
This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.
- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
- osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
- mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
- prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.
Rendered samples:
- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.
Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
LLDP is the only source of physical switch/port topology in the estate: which switch and which port each machine is cabled to exists nowhere else (not in intent YAML, not in any other fact, not in DHCP/DNS). NetBox needs it to record device interface -> switch/port connections. This installs and enables lldpd on physical hosts and exposes its neighbour data as a structured `lldp` fact for the terraform-infra pdbmux backfill to consume (linked issue below).
## Changes
- Add `profiles::lldpd`: install the `lldpd` package and enable+start the service (it ships disabled), the service subscribing to the package.
- Assign it physicals-only via `hiera_include` in `hieradata/virtual/physical.yaml` (`facts.virtual == 'physical'`), merged `unique` with the common `hiera_include`. VMs never receive it.
- Add the `lldp` custom fact (`modules/libs/lib/facter/lldp.rb`): parse `lldpctl -f json0` into a per-interface map of `{neighbor_chassis_name, neighbor_chassis_mac, neighbor_chassis_descr, neighbor_port_id, neighbor_port_descr, vlan_id, vlan_name}`, skipping interfaces with no neighbour. Confined to physical Linux hosts that have `lldpctl` and a live lldpd socket; returns an empty hash on any error so it can never break a puppet run.
json0 is used because it array-wraps every node regardless of cardinality, so one neighbour and many neighbours parse identically (plain `keyvalue` folds the neighbour SysName into the key path; plain `json` collapses single-element arrays into objects).
## Sample fact output (prodnxsr0019, enp1s0)
```json
{
"enp1s0": {
"neighbor_chassis_name": "sg3429x-m2-02",
"neighbor_chassis_mac": "b8:fb:b3:a7:f9:5d",
"neighbor_chassis_descr": "48-Port Gigabit Smart Managed Pro Switch",
"neighbor_port_id": "24",
"neighbor_port_descr": "two-gigabitEthernet 1/0/11",
"vlan_id": "201",
"vlan_name": "fabric_common_02"
}
}
```
## Consumer
NetBox population of the switch/port cabling is tracked in terraform-infra: unkin/terraform-infra#7 (extend the pdbmux backfill / PR #6 to read this fact and emit the connection).
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #513
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Four newly-provisioned el9_8 compute nodes (prodnxsr0014/0015/0018/0019) hang with canal's kube-flannel container in `ImagePullBackOff`: the flannel VXLAN overlay never comes up, so the node can't reach any in-cluster `198.18.200.x` LoadBalancer VIP.
Root cause is a stale airgap-preload version. The nodes boot rke2 **v1.33.11+rke2r1** from the rolling `latest/1.33` repo, whose canal manifest requests `hardened-flannel:v0.28.4` / `hardened-calico:v3.31.5` (build20260415). But `rke2::install` pinned the preloaded bundle to **v1.33.4+rke2r1** (flannel v0.27.2 / calico v3.30.2), so those images were never on disk. containerd then falls back to the `docker.io` mirror (artifactapi, `disable-default-registry-endpoint: true`), reachable only via the pod-overlay VIP that requires the very flannel image being pulled — a bootstrap deadlock. Older nodes (0001-0008) are versionlocked at 1.33.4 and still match their original bundle, so they were unaffected.
## Changes
- Bump pinned `rke2_version` `1.33.4` -> `1.33.11` so the versionlock, RPM `ensure`, and preloaded bundle all line up with the canal image tags the running binary requests. The default `rke2-images.linux-amd64.tar.zst` bundle already contains the canal CNI images (it is RKE2's default CNI), so no extra tarball is needed.
- Wire the airgap archive `source` to the `container_archive_source` class parameter (previously declared in the module but never consumed). The module keeps its generic upstream default; the artifactapi override (the pre-CNI-reachable source, same BGP/physical path the rke2 yum repos already use) lives in the k8s role hiera as `rke2::container_archive_source`.
Applies to servers and agents alike (`rke2::install` runs for both) and preloads `before => Service`, so the bundle lands before rke2 starts.
Notes:
- The `latest/1.33` repo is rolling, so the pinned version must be maintained as the repo advances; a follow-up to pin the yum channel to a fixed patch would remove the drift entirely.
- No terraform-artifactapi change is required. (If a canal-only preload were ever wanted, the github generic remote allowlist would need `rancher/rke2/.*/rke2-images-canal.linux-amd64.tar.zst$` added — but the default bundle already carries those images, so it is unnecessary.)
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #512
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
must change management interfaces for new nodes to start from 21, since
.14 to .20 were used for various anycast services
really need ipam
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #511
AlmaLinux hosts still pulled in-house RPMs from the legacy `unkin` repo hosted on Gitea (git.query.consul), and a second legacy Gitea repo `unkinben` was enabled fleet-wide. This moves AlmaLinux hosts fully onto the artifactapi local rpm repos so all hosts get the in-house/vendored set for their EL release, matching the artifactapi-is-the-real-registry model already used for baseos/epel/etc., and removes both legacy Gitea yum repos.
Depends on unkin/terraform-artifactapi#15 (creates the general `rpm-vendor` local repo); that should apply before this reaches hosts.
### RPM source migration
- retarget `rpm-vendor` to the distribution-agnostic `local/rpm-vendor` repo
- add `rpm-internal-release`/`rpm-vendor-release`, resolved from `os.release.major` so el8 hosts pull `-el8`, el9 hosts `-el9`, etc.
- remove the legacy `unkin` yumrepo definition (its content now comes from rpm-internal/rpm-vendor)
### unkinben removal
- remove the `unkinben` yumrepo definition from `AlmaLinux/all_releases.yaml`
- remove the `unkinben` `ensure: present` override from the jellyfin media role (its only consumer)
- `purge => true` on the yum profile then cleans `/etc/yum.repos.d/unkin.repo` from hosts
**Heads-up for reviewer:** the jellyfin role installs `jellyfin-web`, `jellyfin-server`, and `jellyfin-ffmpeg-bin` from unkinben. These packages exist **only** in unkinben, so once it is removed they will remain installed but **stop receiving updates** until they are republished into artifactapi (e.g. `rpm-internal-elN`). Jellyfin's other deps (`SDL2`, `ffmpeg`, `ffmpeg-devel`) come from rpmfusion and are unaffected. The `unkinben` *generic* archive downloads used by the *arr modules (sonarr/radarr/prowlarr/lidarr/readarr) and glauth are direct HTTP fetches, not the yum repo, and are untouched.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #496
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
We want to manage Fedora 44 hosts with this Puppet codebase, starting with the `base` role only. Fedora reuses the RedHat osfamily code paths (dnf/yum, crypto-policies, firewalld, openvox), so this adds the Fedora-specific hieradata, repositories, and gates needed for `base` to compile and apply, mirroring how AlmaLinux is wired and consuming the artifactapi `rpm-internal-f<major>` / `rpm-vendor-f<major>` local repos. Everything is keyed off `facts.os.release.major` so a future Fedora release only needs its artifactapi local repos created, not new hieradata.
## How
- Add `hieradata/os/Fedora/all_releases.yaml`:
- Define dnf repos via `profiles::yum::global::repos`: `fedora` and `updates` proxied through artifactapi's `fedora` remote, plus the artifactapi locals `rpm-internal-f%{major}` and `rpm-vendor-f%{major}`; GPG via the on-disk `fedora-gpg-keys`; metalink/mirrorlist cleared so only the artifactapi baseurl is used.
- Set `crypto_policies::policy: DEFAULT`, `profiles::puppet::agent::openvox_enable: true`, and `lm-sensors:📦 lm_sensors`.
- Fix up the base package set for Fedora: exclude `p7zip`/`dstat`/`iotop` (absent on Fedora) and include `7zip` and `iotop-c`, plus the same `crypto-policies-scripts`/`lzo`/`policycoreutils`/`unar`/`xz` additions AlmaLinux carries.
- Wire `profiles::fedora::base` via `hiera_include`.
- Add `profiles::fedora::base` (ensures NetworkManager enabled) as the Fedora analogue of `profiles::almalinux::base`; deliberately minimal so it can grow into workstation/laptop use later.
- Make `profiles::puppet::agent` select the OpenVox distribution path (`fedora/<major>` on Fedora, `el/<major>` elsewhere); AlmaLinux/Debian behaviour is unchanged.
## Validation
`puppet-lint`, puppet manifest validate, and `yamllint` all pass via the repo's pre-commit hooks on the changed files.
## Note for reviewer
OpenVox does not yet publish a Fedora 44 build (`openvox7/fedora/` currently has only 36/40/41), so `openvox-agent` will 404 until upstream publishes f44 or a build is placed in `rpm-internal-f44`. The Puppet code produces the correct path for when that exists; installing the agent is a prerequisite for a Fedora 44 host to actually run.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #497
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new Gitea token secrets engine (`vault-plugin-secrets-gitea`, released v0.1.0) needs its plugin binary present on the OpenBao servers before terraform-vault can register it in the plugin catalog and mount it. This is the same install step already done for the rancher, gpg, and litellm secrets plugins.
## Change
- Add `openbao-plugin-secrets-gitea` to `profiles::packages::include` on the vault storage role, pinned to `0.1.0` so the on-disk binary stays in lockstep with the sha256 that terraform-vault pins in its plugin catalog entry (`config/plugins/vault-plugin-secrets-gitea.yaml`).
## Order
This is the first step. It must merge (and Puppet must run on the vault nodes to lay down the binary) before the terraform-vault engine PR is applied, or OpenBao cannot launch the plugin.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #498
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
During the k8s DNS cutover the `dns` module dual-writes every host's records two
ways: (a) LEGACY exported `@@concat::fragment`s collected onto the VM puppet DNS
master into `/var/named/sources/<zone>.conf`, and (b) NEW nsupdate/RFC2136 to the
k8s bind-authoritative cluster (`manage_nsupdate`). Both were gated on so the
legacy path stayed as a safety net.
The legacy path is no longer in any client's resolution path:
- **#490** repointed `profiles::dns::base::nameservers` from the VM resolver
anycast `198.18.19.16` to the k8s **bind-resolvers** LB `198.18.200.7`. Every
puppet client now resolves there.
- The k8s bind-resolvers forward `unkin.net`, `main.unkin.net`, and all
`18.198.in-addr.arpa` reverse zones to the k8s **bind-authoritative** cluster
(`198.18.200.6`), which is populated purely by the nsupdate path — the same
`profiles::dns::record` defines, `manage_nsupdate` branch.
- The VM master's assembled `/var/named/sources` files are therefore orphaned:
nothing resolves against them anymore.
This flips `profiles::dns::updater::manage_export: false` to stop nodes exporting
the legacy fragments. `manage_nsupdate` stays on.
## What changes
- `hieradata/common.yaml`: `profiles::dns::updater::manage_export` `true` -> `false`.
## Verification evidence
Pulled all **498** exported `Concat::Fragment` resources from PuppetDB (239 A,
232 PTR, 27 CNAME; 490 unique fqdn/type/value) and `dig`'d each against the k8s
authoritative read endpoint `198.18.200.7` (the dynamic path's serving side):
| Result | Count |
|---|---|
| MATCH | 447 |
| MISSING-from-dynamic | 38 |
| DIFFERENT-value | 5 |
Both non-match buckets are accounted for and are **not coverage gaps**:
- **38 MISSING** are all records the dynamic path deliberately filters via
`profiles::dns::updater::deny_ranges`
(`198.18.200.0/24`, `10.42.0.0/16`, ...) plus reverse zones the authoritative
cluster isn't even authoritative for: `*-kube-lb0` (198.18.200.x),
`*-flannel.1` (10.42.x.x pod CIDR), and `*-nomad`/`*-brcom1`
(172.26.64.x / 10.255.255.x). These are exactly the k8s/LB noise the deny
ranges exist to keep out of the authoritative zones. Verified every one falls
in a denied/non-authoritative range.
- **5 DIFFERENT** are stale legacy duplicates that the dynamic path holds *more*
correctly:
- 4x `au-syd1-prod-halb.main.unkin.net` A: four ceph nodes (2069-2073) each
self-register the shared name with their own primary IP (haproxy::dns
exports `networking.ip` per host, a legacy collision). Dynamic resolves it to
the single VIP `198.18.24.60`.
- 1x `dashboard.ceph.unkin.net` CNAME: the ceph node self-publishes this to
`lb1.unkin.net.` via its own nsupdate (per the k8s zones.yaml note); the
exported VRRP-name fragment is stale.
Conclusion: the dynamic path fully covers the served record set, and the serving
path no longer depends on the exported files.
## Collector-behavior note (what the flip actually does)
`profiles::dns::zone` realizes the fragments with an exported-resource collector
(`Concat::Fragment <<| target == ... |>>`). Turning off export purges the
fragments from PuppetDB, so on the VM master concat rebuilds each
`/var/named/sources/<zone>.conf` down to just the header (SOA+NS+glue) as nodes
converge. That is **cosmetic**: no client resolves against the VM master anymore
(#490), and the k8s authoritative zones are unaffected (fed by nsupdate, not
these files).
## Rollback
Set `profiles::dns::updater::manage_export: true` again and let puppet converge;
nodes re-export their fragments and the VM-master zone files repopulate.
## Follow-up (after soak)
Deleting the `@@concat::fragment` / `Concat::Fragment <<||>>` collector code in
`profiles::dns::record` / `profiles::dns::zone` (and the VM `profiles::dns::master`
zone assembly) is a separate cleanup PR once this has soaked and the VM master is
confirmed decommissionable.
Reviewed-on: #495
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
First live agent migration to the k8s puppetserver, using the single-boolean mechanism from #493 on the lowest-risk host (roles::base test host).
- Adds hieradata/nodes/ausyd1nxvm2120.main.unkin.net.yaml with profiles::puppet::migrate::enabled: true
On merge: the node's next run (from the old master) rewrites puppet.conf to the k8s server/ca_server with a fresh ssl-k8s ssldir; the run after enrols against the k8s CA (autosigned) and compiles from the k8s compilers (ENC=encapi). Verify: node appears in the k8s PuppetDB with a fresh report. Rollback: revert this file — legacy ssldir/cert remain intact.
Reviewed-on: #494
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`ensure: present`/`latest` lets the plugin binaries drift from the sha256 pinned in the terraform-vault catalog (`config/plugins/*.yaml`). On the next OpenBao restart, a drifted binary fails the sha check and the plugin won't launch — a latent footgun (hit exactly this with rancher on `ensure: latest`).
## Changes
Pin each secrets plugin to the version whose binary matches its registered catalog sha (all verified against the RPMs in rpm-internal):
- `openbao-plugin-secrets-litellm`: **0.1.1** (sha 2263ebcb…)
- `openbao-plugin-secrets-gpg`: **0.1.0** (sha 0e92d740…)
- `openbao-plugin-secrets-rancher`: **0.1.1** (sha 9e597cd9…; was `ensure: latest`)
All three are no-op on the binary (installed versions already match) — this just locks them so a future release can't silently upgrade the binary out of lockstep with the catalog.
`openbao-plugins` (base bundle) left unpinned — its version couldn't be verified from the tooling side and it tracks the openbao package, not a catalog sha.
## Note
To upgrade a plugin in future: bump the RPM version here **and** the catalog sha256 in terraform-vault in the same change, then `vault write sys/plugins/reload/backend plugin=<name>`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #492
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- change the default anycast nameserver to bind-internal resolvers in
kubernetes
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #490
## Why
`s3.ceph.unkin.net` is the RGW S3 endpoint name used by cephrgw-operator consumers. A companion argocd-apps PR adds it as a CNAME to `radosgw.service.consul`. The radosgw hosts front RGW with nginx (443 -> radosgw:7480) using the vault-issued cert, so **both** the certificate and nginx must recognise the name.
## Changes (hieradata/roles/infra/ceph/rgw.yaml)
- Add `s3.ceph.unkin.net` to `profiles::pki::vault::alt_names` — puts the name in the radosgw cert SAN (renewal re-triggers on the alt-names change).
- Add `s3.ceph.unkin.net` to `profiles::nginx::simpleproxy::nginx_aliases` — nginx answers for that server_name.
### nginx: yes, needed
radosgw sits behind `profiles::nginx::simpleproxy` (listens 443, proxies to :7480) sharing the same vault cert, so the alias is required for nginx to serve the hostname — not just the cert SAN.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #487
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The vault nodes have rancher plugin v0.1.0, which mints non-functional tokens (returns `status.value` not `status.bearerToken`; every cred/rotation 401s). v0.1.1 fixes it. `ensure_packages` defaults to `present`, which won't upgrade an already-installed package, so pin to `latest` to pull v0.1.1.
## Changes
- `openbao-plugin-secrets-rancher: { ensure: latest }` on the vault storage role.
## Note
`latest` will also pull future releases automatically — keep the terraform-vault catalog sha256 (`config/plugins/vault-plugin-secrets-rancher.yaml`) bumped in lockstep, or a vault restart after a future upgrade will fail the sha check. Happy to pin to an exact version instead if you'd prefer lockstep-by-pin.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #488
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Repoints the `rpmfusion-free` and `rpmfusion-nonfree` yum repos from the retired `packagerepo.service.consul` mirror to the artifactapi RPM Fusion remotes.
dnf on media nodes (jellyfin, nzbget) failed with `Cannot download repomd.xml: All mirrors were tried`, which blocked package installs such as `dns-updater`.
Changes:
- `rpmfusion-free` baseurl → `.../api/v1/remote/rpmfusion-free/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-free/el/RPM-GPG-KEY-rpmfusion-free-el-$major`
- `rpmfusion-nonfree` baseurl → `.../api/v1/remote/rpmfusion-nonfree/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-nonfree/el/RPM-GPG-KEY-rpmfusion-nonfree-el-$major`
Depends on terraform-artifactapi PR #7 (adds the rpmfusion remotes) being applied first.
https://claude.ai/code/session_01VZ81u2JtjvaPq8Z3EDTSMb
Reviewed-on: #486
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: rpmbuilder's vendored third-party RPMs are moving from Gitea's RPM registry to the artifactapi rpm-vendor local repos. Managed AlmaLinux hosts need the matching yum repo to install them; this mirrors the existing rpm-internal repo and selects the per-EL-release repo via os.release.major.
Changes:
- Add rpm-vendor to profiles::yum::global::repos in hieradata/os/AlmaLinux/all_releases.yaml, pointing at .../api/v1/local/rpm-vendor-el<major>/ (el8/el9), gpgcheck disabled and mirrorlist absent, matching the rpm-internal repo definition.
Depends on terraform-artifactapi PR #6 (creates the rpm-vendor-el8/el9 repos).
Reviewed-on: #484
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Rancher token secrets engine needs its plugin binary present in the OpenBao plugin directory before terraform-vault can register/mount it. Mirrors #480 (gpg).
## Changes
- Add `openbao-plugin-secrets-rancher` to `profiles::packages::include` on the vault storage role. Installs the RPM (from rpm-internal) to `/opt/openbao-plugins/vault-plugin-secrets-rancher`.
## Merge order
Part 3 of 4 — merge first (binary must exist on disk before the terraform-vault plugin-import #92 registers it with a matching sha256).
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #483
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).
- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.
Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.
Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
enable installing locally packaged rpms with no specific linux release
- add the rpm-internal repo from artifactapi
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #476
## Summary
- Replace static `registries.yaml` with EPP template driven by `rke2::registries` hash
- Add `disable-default-registry-endpoint: true` to all mirrors — RKE2 will only use artifactapi and never fall back to upstream registries
- Registry configuration now fully managed via hiera data (`roles/infra/k8s.yaml`)
Reviewed-on: #474
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Pin grafana package version to 13.0.2 via a new version parameter on
profiles::metrics::grafana, wired through to the puppet-grafana class.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #470
- update release to install to 1.26.2
- change base_url to artifactapi
- update releases/checksums
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #469
## Summary
- Adds `Unkin::Ceph::Utils` facter module detecting ceph service instances via `systemctl list-units`, exposing `is_ceph_mon`, `is_ceph_mgr`, `is_ceph_mds`, `is_ceph_osd` booleans and a `ceph_services` hash of unit names
- Adds `profiles::ceph::mon`, `mgr`, `mds`, `osd` — each with `Boolean $ensure_running` that iterates discovered service instances and manages them as running and enabled
- Works across incus nodes (mon/mgr/mds/osd) and k8s compute/control nodes (osd only); verified on prodnxsr0001 which correctly reports `is_ceph_osd: true` and `ceph_services: {osd: [ceph-osd@5]}`
## Test plan
- [x] Noop deploy against prodnxsr0001.main.unkin.net passed cleanly
- [x] `ceph_services` fact returns correct service map
- [x] `is_ceph_osd` returns `True`, `is_ceph_mon` returns `False` as expected
- [x] Test on an incus/ceph node with mon/mgr/mds services
Reviewed-on: #459
rebuilding router, taking the chance to not mess up ip ranges. I did
have 198.18.21.0/24 and 198.18.21.160/27 and 198.18.21.192/27 all on
differnt interfaces.
- update IP's that can reach bind view for main.unkin.net
- keep both for intermediate period
Reviewed-on: #460
need to separate the permissions inside vault into different groups, one
per-permission.
- add group for each kubernetes role in vault
Reviewed-on: #449
unkin-agent
unkinben
benvin