go-plugin gives each secrets plugin a control socket at /tmp/pluginNNN.
The daily systemd-tmpfiles-clean deletes /tmp files older than the OS
policy, which severs the socket of a long-lived plugin while its process
keeps running. Every request then fails with rpc Unavailable / dial unix
/tmp/pluginNNN: no such file, until a plugin reload respawns it. This hit
the litellm engine (terraform-vault#112) but the risk is shared by every
OpenBao plugin (gpg, rancher, gitea, ...).
Ship a tmpfiles.d exclude (x /tmp/plugin*) on all vault servers so plugin
sockets survive the cleaner. Takes effect on the next clean, no restart.
Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
A forthcoming profile needs to set the I/O scheduler to noop/none on the disks backing ceph OSDs, so Puppet must know which block devices those are. This fact surfaces them by inspecting LVM PVs whose volume group belongs to ceph.
- add ceph_osd_devices fact returning the sorted array of PV device paths whose VG name starts with ceph-
- confine the fact to physical Linux hosts with pvs installed so it never resolves on VMs
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #504
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
An out-of-band OpenJDK upgrade (java-17-openjdk 17.0.18 -> 17.0.19, delivered by the AlmaLinux repo migration in #496) removed the old versioned JAVA_HOME while the puppetserver JVMs kept running against the now-deleted files. The running JVM re-execs `jspawnhelper` from its original (deleted) JAVA_HOME on every posix_spawn, so `ProcessBuilder` fails with `error=2, No such file or directory`. That broke the exec ENC (`/opt/cobbler-enc/cobbler-enc`) and 500'd every catalog compile across all 6 masters, failing 136/143 nodes. The masters could not self-heal because nothing restarts the JVM after a java upgrade. Incident was resolved by manually restarting puppetserver on all 6 masters.
## Change
- Add `/usr/local/bin/puppetserver_jvm_guard.sh`: restarts puppetserver if a running JVM (`puppet-server-release.jar`) is executing from deleted binaries.
- Add a `puppetserver-jvm-guard` systemd timer (every 5 min, mirrors the existing generate-types timer pattern) that runs the guard, so any future JVM/library upgrade recovers automatically.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #499
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
AlmaLinux hosts still pulled in-house RPMs from the legacy `unkin` repo hosted on Gitea (git.query.consul), and a second legacy Gitea repo `unkinben` was enabled fleet-wide. This moves AlmaLinux hosts fully onto the artifactapi local rpm repos so all hosts get the in-house/vendored set for their EL release, matching the artifactapi-is-the-real-registry model already used for baseos/epel/etc., and removes both legacy Gitea yum repos.
Depends on unkin/terraform-artifactapi#15 (creates the general `rpm-vendor` local repo); that should apply before this reaches hosts.
### RPM source migration
- retarget `rpm-vendor` to the distribution-agnostic `local/rpm-vendor` repo
- add `rpm-internal-release`/`rpm-vendor-release`, resolved from `os.release.major` so el8 hosts pull `-el8`, el9 hosts `-el9`, etc.
- remove the legacy `unkin` yumrepo definition (its content now comes from rpm-internal/rpm-vendor)
### unkinben removal
- remove the `unkinben` yumrepo definition from `AlmaLinux/all_releases.yaml`
- remove the `unkinben` `ensure: present` override from the jellyfin media role (its only consumer)
- `purge => true` on the yum profile then cleans `/etc/yum.repos.d/unkin.repo` from hosts
**Heads-up for reviewer:** the jellyfin role installs `jellyfin-web`, `jellyfin-server`, and `jellyfin-ffmpeg-bin` from unkinben. These packages exist **only** in unkinben, so once it is removed they will remain installed but **stop receiving updates** until they are republished into artifactapi (e.g. `rpm-internal-elN`). Jellyfin's other deps (`SDL2`, `ffmpeg`, `ffmpeg-devel`) come from rpmfusion and are unaffected. The `unkinben` *generic* archive downloads used by the *arr modules (sonarr/radarr/prowlarr/lidarr/readarr) and glauth are direct HTTP fetches, not the yum repo, and are untouched.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #496
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
We want to manage Fedora 44 hosts with this Puppet codebase, starting with the `base` role only. Fedora reuses the RedHat osfamily code paths (dnf/yum, crypto-policies, firewalld, openvox), so this adds the Fedora-specific hieradata, repositories, and gates needed for `base` to compile and apply, mirroring how AlmaLinux is wired and consuming the artifactapi `rpm-internal-f<major>` / `rpm-vendor-f<major>` local repos. Everything is keyed off `facts.os.release.major` so a future Fedora release only needs its artifactapi local repos created, not new hieradata.
## How
- Add `hieradata/os/Fedora/all_releases.yaml`:
- Define dnf repos via `profiles::yum::global::repos`: `fedora` and `updates` proxied through artifactapi's `fedora` remote, plus the artifactapi locals `rpm-internal-f%{major}` and `rpm-vendor-f%{major}`; GPG via the on-disk `fedora-gpg-keys`; metalink/mirrorlist cleared so only the artifactapi baseurl is used.
- Set `crypto_policies::policy: DEFAULT`, `profiles::puppet::agent::openvox_enable: true`, and `lm-sensors:📦 lm_sensors`.
- Fix up the base package set for Fedora: exclude `p7zip`/`dstat`/`iotop` (absent on Fedora) and include `7zip` and `iotop-c`, plus the same `crypto-policies-scripts`/`lzo`/`policycoreutils`/`unar`/`xz` additions AlmaLinux carries.
- Wire `profiles::fedora::base` via `hiera_include`.
- Add `profiles::fedora::base` (ensures NetworkManager enabled) as the Fedora analogue of `profiles::almalinux::base`; deliberately minimal so it can grow into workstation/laptop use later.
- Make `profiles::puppet::agent` select the OpenVox distribution path (`fedora/<major>` on Fedora, `el/<major>` elsewhere); AlmaLinux/Debian behaviour is unchanged.
## Validation
`puppet-lint`, puppet manifest validate, and `yamllint` all pass via the repo's pre-commit hooks on the changed files.
## Note for reviewer
OpenVox does not yet publish a Fedora 44 build (`openvox7/fedora/` currently has only 36/40/41), so `openvox-agent` will 404 until upstream publishes f44 or a build is placed in `rpm-internal-f44`. The Puppet code produces the correct path for when that exists; installing the agent is a prerequisite for a Fedora 44 host to actually run.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #497
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The new Gitea token secrets engine (`vault-plugin-secrets-gitea`, released v0.1.0) needs its plugin binary present on the OpenBao servers before terraform-vault can register it in the plugin catalog and mount it. This is the same install step already done for the rancher, gpg, and litellm secrets plugins.
## Change
- Add `openbao-plugin-secrets-gitea` to `profiles::packages::include` on the vault storage role, pinned to `0.1.0` so the on-disk binary stays in lockstep with the sha256 that terraform-vault pins in its plugin catalog entry (`config/plugins/vault-plugin-secrets-gitea.yaml`).
## Order
This is the first step. It must merge (and Puppet must run on the vault nodes to lay down the binary) before the terraform-vault engine PR is applied, or OpenBao cannot launch the plugin.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #498
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
During the k8s DNS cutover the `dns` module dual-writes every host's records two
ways: (a) LEGACY exported `@@concat::fragment`s collected onto the VM puppet DNS
master into `/var/named/sources/<zone>.conf`, and (b) NEW nsupdate/RFC2136 to the
k8s bind-authoritative cluster (`manage_nsupdate`). Both were gated on so the
legacy path stayed as a safety net.
The legacy path is no longer in any client's resolution path:
- **#490** repointed `profiles::dns::base::nameservers` from the VM resolver
anycast `198.18.19.16` to the k8s **bind-resolvers** LB `198.18.200.7`. Every
puppet client now resolves there.
- The k8s bind-resolvers forward `unkin.net`, `main.unkin.net`, and all
`18.198.in-addr.arpa` reverse zones to the k8s **bind-authoritative** cluster
(`198.18.200.6`), which is populated purely by the nsupdate path — the same
`profiles::dns::record` defines, `manage_nsupdate` branch.
- The VM master's assembled `/var/named/sources` files are therefore orphaned:
nothing resolves against them anymore.
This flips `profiles::dns::updater::manage_export: false` to stop nodes exporting
the legacy fragments. `manage_nsupdate` stays on.
## What changes
- `hieradata/common.yaml`: `profiles::dns::updater::manage_export` `true` -> `false`.
## Verification evidence
Pulled all **498** exported `Concat::Fragment` resources from PuppetDB (239 A,
232 PTR, 27 CNAME; 490 unique fqdn/type/value) and `dig`'d each against the k8s
authoritative read endpoint `198.18.200.7` (the dynamic path's serving side):
| Result | Count |
|---|---|
| MATCH | 447 |
| MISSING-from-dynamic | 38 |
| DIFFERENT-value | 5 |
Both non-match buckets are accounted for and are **not coverage gaps**:
- **38 MISSING** are all records the dynamic path deliberately filters via
`profiles::dns::updater::deny_ranges`
(`198.18.200.0/24`, `10.42.0.0/16`, ...) plus reverse zones the authoritative
cluster isn't even authoritative for: `*-kube-lb0` (198.18.200.x),
`*-flannel.1` (10.42.x.x pod CIDR), and `*-nomad`/`*-brcom1`
(172.26.64.x / 10.255.255.x). These are exactly the k8s/LB noise the deny
ranges exist to keep out of the authoritative zones. Verified every one falls
in a denied/non-authoritative range.
- **5 DIFFERENT** are stale legacy duplicates that the dynamic path holds *more*
correctly:
- 4x `au-syd1-prod-halb.main.unkin.net` A: four ceph nodes (2069-2073) each
self-register the shared name with their own primary IP (haproxy::dns
exports `networking.ip` per host, a legacy collision). Dynamic resolves it to
the single VIP `198.18.24.60`.
- 1x `dashboard.ceph.unkin.net` CNAME: the ceph node self-publishes this to
`lb1.unkin.net.` via its own nsupdate (per the k8s zones.yaml note); the
exported VRRP-name fragment is stale.
Conclusion: the dynamic path fully covers the served record set, and the serving
path no longer depends on the exported files.
## Collector-behavior note (what the flip actually does)
`profiles::dns::zone` realizes the fragments with an exported-resource collector
(`Concat::Fragment <<| target == ... |>>`). Turning off export purges the
fragments from PuppetDB, so on the VM master concat rebuilds each
`/var/named/sources/<zone>.conf` down to just the header (SOA+NS+glue) as nodes
converge. That is **cosmetic**: no client resolves against the VM master anymore
(#490), and the k8s authoritative zones are unaffected (fed by nsupdate, not
these files).
## Rollback
Set `profiles::dns::updater::manage_export: true` again and let puppet converge;
nodes re-export their fragments and the VM-master zone files repopulate.
## Follow-up (after soak)
Deleting the `@@concat::fragment` / `Concat::Fragment <<||>>` collector code in
`profiles::dns::record` / `profiles::dns::zone` (and the VM `profiles::dns::master`
zone assembly) is a separate cleanup PR once this has soaked and the VM master is
confirmed decommissionable.
Reviewed-on: #495
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
First live agent migration to the k8s puppetserver, using the single-boolean mechanism from #493 on the lowest-risk host (roles::base test host).
- Adds hieradata/nodes/ausyd1nxvm2120.main.unkin.net.yaml with profiles::puppet::migrate::enabled: true
On merge: the node's next run (from the old master) rewrites puppet.conf to the k8s server/ca_server with a fresh ssl-k8s ssldir; the run after enrols against the k8s CA (autosigned) and compiles from the k8s compilers (ENC=encapi). Verify: node appears in the k8s PuppetDB with a fresh report. Rollback: revert this file — legacy ssldir/cert remain intact.
Reviewed-on: #494
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Migrate the VM puppet agents off the legacy VM puppetmasters onto the new puppet-on-kubernetes servers, one wave at a time and reversible without re-enrolment.
Changing `server`/`ca_server` alone is insufficient: each agent's ssldir holds a client certificate signed by the **old** CA, which the new k8s CA neither trusts nor recognises. Migrated nodes switch to a **fresh ssldir**, so the agent generates a new key + CSR on its next run (autosigned immediately by the k8s CA, which runs `autosign = true`), pulls its catalog from the new compilers, and leaves the old CA-signed creds intact on disk for rollback.
The reconfiguration catalog is compiled by the **old** server (the node still points there when it applies the change). `profiles::puppet::client` rewrites `puppet.conf` with the new endpoints + ssldir and notifies `Service['puppet']`, so the very next agent run lands on k8s.
**Single-boolean design (reworked):** cutover is now driven by one hiera key. `profiles::puppet::migrate` ships wired into `profiles::base` with `enabled => false`, so every node evaluates it but nothing changes until the boolean is flipped. `profiles::puppet::client` includes migrate and folds the migrate endpoints into the effective values it renders. Flip `profiles::puppet::migrate::enabled: true` at any hiera layer (node/role/common) to migrate a target -- no other keys required.
## Changes
- `profiles::base` (`site/profiles/manifests/base.pp`): `include profiles::puppet::migrate` alongside the other puppet profiles (ships disabled).
- `profiles::puppet::migrate` (`site/profiles/manifests/puppet/migrate.pp`): unchanged params (`enabled=false`, `new_server`, `new_ca_server`, `new_ssldir`); when enabled and the node is not a puppetmaster it owns the fresh ssldir directory. Class-header runbook rewritten for the single-boolean flow. No longer opt-in via `hiera_include`.
- `profiles::puppet::client` (`site/profiles/manifests/puppet/client.pp`): `include profiles::puppet::migrate` and compute `$effective_server` / `$effective_ca_server` / `$effective_report_server` / `$effective_ssldir`, feeding those to the template. Keeps the existing optional `$ssldir` / `$report_server` params as manual overrides.
- `puppet.conf.erb`: render from the `@effective_*` vars.
## Precedence
- **ssldir / report_server** (`Optional`, default `undef`): explicit `profiles::puppet::client::ssldir` / `::report_server` in hiera wins; else the migrate-derived value when `enabled`; else the legacy default (ssldir omitted -> puppet default; report_server -> effective server).
- **server / ca_server** (`String`, no undef sentinel): the migrate endpoint wins while `enabled => true`; otherwise the client param (legacy default unless set explicitly). To pin a bespoke server, leave `enabled => false` for that target and set `profiles::puppet::client::server` directly.
- No include cycle: migrate.pp never references client.pp; its guard uses `$facts['enc_role']`.
## Runbook
- **Canary (one node)** -> `hieradata/nodes/<certname>.yaml`: `profiles::puppet::migrate::enabled: true`
- **Wave (a role)** -> `hieradata/roles/<t1>/<t2>[/<t3>].yaml`: `profiles::puppet::migrate::enabled: true`
- **Estate flip** -> `hieradata/common.yaml`: `profiles::puppet::migrate::enabled: true`
- **Rollback** -> remove the key (or set `false`) at that layer; next run rewrites puppet.conf back to the legacy server + legacy ssldir (original CA-signed cert intact).
- The k8s endpoints (`new_server` / `new_ca_server` / `new_ssldir`) are `profiles::puppet::migrate` param defaults; override them in hiera if the k8s names change.
## Verification
- Render comparison: unmigrated node renders a **byte-identical** `puppet.conf` to `develop` (diff empty). Migrated render (only `enabled: true`) emits `server`/`ca_server`/`report_server` = k8s endpoints and `ssldir = /etc/puppetlabs/puppet/ssl-k8s`.
- `puppet-lint` (repo args `--no-80chars-check --no-documentation-check --no-puppet_url_without_modules-check --fail-on-warnings`): clean on all 3 manifests.
- `puppet parser validate`: OK. ERB syntax: OK. Repo pre-commit hooks (erb-validate, puppet-lint, puppet-validate): all Passed.
Reviewed-on: #493
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`ensure: present`/`latest` lets the plugin binaries drift from the sha256 pinned in the terraform-vault catalog (`config/plugins/*.yaml`). On the next OpenBao restart, a drifted binary fails the sha check and the plugin won't launch — a latent footgun (hit exactly this with rancher on `ensure: latest`).
## Changes
Pin each secrets plugin to the version whose binary matches its registered catalog sha (all verified against the RPMs in rpm-internal):
- `openbao-plugin-secrets-litellm`: **0.1.1** (sha 2263ebcb…)
- `openbao-plugin-secrets-gpg`: **0.1.0** (sha 0e92d740…)
- `openbao-plugin-secrets-rancher`: **0.1.1** (sha 9e597cd9…; was `ensure: latest`)
All three are no-op on the binary (installed versions already match) — this just locks them so a future release can't silently upgrade the binary out of lockstep with the catalog.
`openbao-plugins` (base bundle) left unpinned — its version couldn't be verified from the tooling side and it tracks the openbao package, not a catalog sha.
## Note
To upgrade a plugin in future: bump the RPM version here **and** the catalog sha256 in terraform-vault in the same change, then `vault write sys/plugins/reload/backend plugin=<name>`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #492
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The cobbler master exports `cobbler.main.unkin.net` as a CNAME whose value is the bare hostname (`ausyd1nxvm2098`). The legacy puppet DNS master loaded this into a zone file under `$ORIGIN main.unkin.net.`, so BIND completed the bare label to `ausyd1nxvm2098.main.unkin.net.` and it resolved. `dns-updater` renders RFC2136 updates with no `$ORIGIN`, so the same bare label becomes root-absolute (`ausyd1nxvm2098.`) and dead-ends in NXDOMAIN — the record exists on the k8s authoritative servers (and via the resolvers at 198.18.200.7) but has no reachable address. This is the same failure already documented and fixed for haproxy in `profiles::haproxy::dns`.
Changes:
- Set the cobbler CNAME `value` to the fully-qualified `${facts.networking.fqdn}.` (trailing dot) so the target is origin-independent and resolves identically on both the legacy zone-file export and the dns-updater nsupdate path.
- Add an explanatory comment mirroring the one in `profiles::haproxy::dns` to prevent reintroducing a bare label.
Reviewed-on: #491
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- change the default anycast nameserver to bind-internal resolvers in
kubernetes
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #490
## Why
The halb publishes service CNAMEs (`git.unkin.net`, `dashboard.ceph.unkin.net`, and every other halb-fronted name) with a **bare target label**, e.g.:
```
git.unkin.net. 300 IN CNAME au-syd1-prod-halb-vrrp.
```
`dns-updater` builds the RR with `dns.NewRR` and **no $ORIGIN**, so a bare value becomes root-absolute (`au-syd1-prod-halb-vrrp.`) and dead-ends in NXDOMAIN. Once the k8s bind became authoritative for these zones, that broke resolution of every halb-fronted service (git, the Ceph dashboard, ...). `au-syd1-prod-halb-vrrp.unkin.net` / `.main.unkin.net` resolve fine (198.18.19.17) — only the CNAME target was truncated.
## Change (`site/profiles/manifests/haproxy/dns.pp`)
Emit **fully-qualified** CNAME targets (trailing dot), the FQDN form `dns-updater` expects (per its own test fixture `au-syd1-prod-halb.main.unkin.net.`):
- vrrp cnames: `${location_environment}-halb-vrrp.${domain}.`
- non-vrrp cnames: `${location_environment}-halb.${domain}.`
The matching A records are already published in `main.unkin.net`/`unkin.net` just above, so the targets resolve to the VIP.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #489
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`s3.ceph.unkin.net` is the RGW S3 endpoint name used by cephrgw-operator consumers. A companion argocd-apps PR adds it as a CNAME to `radosgw.service.consul`. The radosgw hosts front RGW with nginx (443 -> radosgw:7480) using the vault-issued cert, so **both** the certificate and nginx must recognise the name.
## Changes (hieradata/roles/infra/ceph/rgw.yaml)
- Add `s3.ceph.unkin.net` to `profiles::pki::vault::alt_names` — puts the name in the radosgw cert SAN (renewal re-triggers on the alt-names change).
- Add `s3.ceph.unkin.net` to `profiles::nginx::simpleproxy::nginx_aliases` — nginx answers for that server_name.
### nginx: yes, needed
radosgw sits behind `profiles::nginx::simpleproxy` (listens 443, proxies to :7480) sharing the same vault cert, so the alias is required for nginx to serve the hostname — not just the cert SAN.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #487
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The vault nodes have rancher plugin v0.1.0, which mints non-functional tokens (returns `status.value` not `status.bearerToken`; every cred/rotation 401s). v0.1.1 fixes it. `ensure_packages` defaults to `present`, which won't upgrade an already-installed package, so pin to `latest` to pull v0.1.1.
## Changes
- `openbao-plugin-secrets-rancher: { ensure: latest }` on the vault storage role.
## Note
`latest` will also pull future releases automatically — keep the terraform-vault catalog sha256 (`config/plugins/vault-plugin-secrets-rancher.yaml`) bumped in lockstep, or a vault restart after a future upgrade will fail the sha check. Happy to pin to an exact version instead if you'd prefer lockstep-by-pin.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #488
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Repoints the `rpmfusion-free` and `rpmfusion-nonfree` yum repos from the retired `packagerepo.service.consul` mirror to the artifactapi RPM Fusion remotes.
dnf on media nodes (jellyfin, nzbget) failed with `Cannot download repomd.xml: All mirrors were tried`, which blocked package installs such as `dns-updater`.
Changes:
- `rpmfusion-free` baseurl → `.../api/v1/remote/rpmfusion-free/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-free/el/RPM-GPG-KEY-rpmfusion-free-el-$major`
- `rpmfusion-nonfree` baseurl → `.../api/v1/remote/rpmfusion-nonfree/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-nonfree/el/RPM-GPG-KEY-rpmfusion-nonfree-el-$major`
Depends on terraform-artifactapi PR #7 (adds the rpmfusion remotes) being applied first.
https://claude.ai/code/session_01VZ81u2JtjvaPq8Z3EDTSMb
Reviewed-on: #486
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Configures the dns-updater record filter (dns-updater#3) so k8s/LB/internal addresses stay out of the authoritative zones.
## Why
Hosts publish records the authoritative server should not hold — `*-flannel.1 A 10.42.x`, `*-kube-lb0 A 198.18.200.x`, and the matching reverse PTRs (`200.18.198`, `2.42.10`) — which the daemon WARNs `NOTAUTH`/`bad authentication` on.
## Changes
- `DNS_UPDATER_DENY_RANGES=198.18.199.0/24,198.18.200.0/24,10.42.0.0/16,10.43.0.0/16` (pod CIDR, service CIDR, LB VIP pool, .199).
- New class params: `deny_ranges` / `allow_ranges` / `deny_domains` / `allow_domains`.
- `package_ensure` → `latest` so hosts pick up the filter release (rpm-internal `metadata_expire` is 1h, so no thrash).
## Sequencing
Needs dns-updater **#3** merged + a new tag (v0.2.0) so the filter env vars are honored. The env keys are ignored by v0.1.0, so this is safe to merge first — it takes effect when hosts upgrade to v0.2.0.
Reviewed-on: #485
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: rpmbuilder's vendored third-party RPMs are moving from Gitea's RPM registry to the artifactapi rpm-vendor local repos. Managed AlmaLinux hosts need the matching yum repo to install them; this mirrors the existing rpm-internal repo and selects the per-EL-release repo via os.release.major.
Changes:
- Add rpm-vendor to profiles::yum::global::repos in hieradata/os/AlmaLinux/all_releases.yaml, pointing at .../api/v1/local/rpm-vendor-el<major>/ (el8/el9), gpgcheck disabled and mirrorlist absent, matching the rpm-internal repo definition.
Depends on terraform-artifactapi PR #6 (creates the rpm-vendor-el8/el9 repos).
Reviewed-on: #484
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Rancher token secrets engine needs its plugin binary present in the OpenBao plugin directory before terraform-vault can register/mount it. Mirrors #480 (gpg).
## Changes
- Add `openbao-plugin-secrets-rancher` to `profiles::packages::include` on the vault storage role. Installs the RPM (from rpm-internal) to `/opt/openbao-plugins/vault-plugin-secrets-rancher`.
## Merge order
Part 3 of 4 — merge first (binary must exist on disk before the terraform-vault plugin-import #92 registers it with a matching sha256).
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #483
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Replace the `profiles::dns::updater` shell mechanism (`dns-update.sh` + `dns-update.path`/`.service` + the in-run `exec`) with the packaged **dns-updater** daemon. The daemon watches the records file (inotify) and network interfaces and pushes TSIG-signed RFC2136 updates to BIND natively — with structured per-zone RCODEs and a status API/facter fact, so failures like the recent `invalid owner name: empty label` / NOTZONE surface directly instead of as opaque nsupdate stderr.
## Changes
- Install the `dns-updater` package; manage `/etc/dns-updater/env`.
- Run the packaged `dns-updater.service`, restarting **only** on env/key change — records-file edits are picked up by the daemon`s own inotify watch, so no service churn on record changes.
- Keep the `concat` records file and the TSIG key file unchanged (same paths/format).
- Ensure the old `/usr/local/bin/dns-update` + `dns-update.path`/`.service` units are absent.
- Drop the now-dead `dns-update.{sh,service,path}.epp` templates.
## Sequencing — HOLD
Do not merge until the `dns-updater` RPM is published to artifactapi `rpm-internal` (needs terraform-git#33 to create the repo, then the daemon code pushed + tagged so Woodpecker builds the RPM). Merging before the package exists makes `package { dns-updater }` fail on every host.
Supersedes the interim shell fix in #481 (which stays valid until this rolls out). Keeps the file as the desired-state interface (puppet owns desired records; the daemon reconciles + reports).
Reviewed-on: #482
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
6 hosts (ausyd1nxvm2069-2073, 2098) ended up with a reverse PTR in bind-authoritative but **no forward A record**, and the `unkin.net` service records (git/grafana/auth/fafflix, all published by the halb host 2069) never landed at all.
VictoriaLogs (`dns-update-apply` on 2069 & 2070) shows the cause:
```
dns-update: nsupdate to 198.18.200.9 failed
invalid owner name: empty label
syntax error
```
`profiles::dns::record` publishes some records whose name is already fully-qualified (trailing dot) — e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`, `dashboard.ceph.unkin.net.`, and the halb CNAMEs. The `dns-update` script `fqdn()` unconditionally appended the zone, producing `…net..main.unkin.net.` — the `..` is an empty label, which nsupdate rejects, failing the entire per-zone `send`. The reverse-PTR send is sorted first and its name is always relative, so it still applied — hence "PTR but no A".
## Change
`fqdn()` now handles three cases:
- `@`/empty → zone apex (unchanged)
- name ending in `.` → already FQDN, used verbatim (**the fix**)
- otherwise → relative, append `.zone.` (unchanged)
Verified against all record shapes (relative host, apex, FQDN CNAME, reverse label) — no more `..`.
## After merge
Once puppet re-runs on the affected hosts their `main.unkin.net`/`unkin.net` updates succeed, filling in the missing A records and the `unkin.net` service zone. Pairs with argocd-apps#260 (adds the `ceph.unkin.net` zone so `dashboard.ceph.unkin.net` does not then hit NOTZONE).
Reviewed-on: #481
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).
- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.
Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.
Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
enable installing locally packaged rpms with no specific linux release
- add the rpm-internal repo from artifactapi
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #476
## Summary
- Replace static `registries.yaml` with EPP template driven by `rke2::registries` hash
- Add `disable-default-registry-endpoint: true` to all mirrors — RKE2 will only use artifactapi and never fall back to upstream registries
- Registry configuration now fully managed via hiera data (`roles/infra/k8s.yaml`)
Reviewed-on: #474
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-06-29 22:30:48 +10:00
39 changed files with 808 additions and 94 deletions
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
unkinben
benvin