## Why
`ensure: present`/`latest` lets the plugin binaries drift from the sha256 pinned in the terraform-vault catalog (`config/plugins/*.yaml`). On the next OpenBao restart, a drifted binary fails the sha check and the plugin won't launch — a latent footgun (hit exactly this with rancher on `ensure: latest`).
## Changes
Pin each secrets plugin to the version whose binary matches its registered catalog sha (all verified against the RPMs in rpm-internal):
- `openbao-plugin-secrets-litellm`: **0.1.1** (sha 2263ebcb…)
- `openbao-plugin-secrets-gpg`: **0.1.0** (sha 0e92d740…)
- `openbao-plugin-secrets-rancher`: **0.1.1** (sha 9e597cd9…; was `ensure: latest`)
All three are no-op on the binary (installed versions already match) — this just locks them so a future release can't silently upgrade the binary out of lockstep with the catalog.
`openbao-plugins` (base bundle) left unpinned — its version couldn't be verified from the tooling side and it tracks the openbao package, not a catalog sha.
## Note
To upgrade a plugin in future: bump the RPM version here **and** the catalog sha256 in terraform-vault in the same change, then `vault write sys/plugins/reload/backend plugin=<name>`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #492
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The cobbler master exports `cobbler.main.unkin.net` as a CNAME whose value is the bare hostname (`ausyd1nxvm2098`). The legacy puppet DNS master loaded this into a zone file under `$ORIGIN main.unkin.net.`, so BIND completed the bare label to `ausyd1nxvm2098.main.unkin.net.` and it resolved. `dns-updater` renders RFC2136 updates with no `$ORIGIN`, so the same bare label becomes root-absolute (`ausyd1nxvm2098.`) and dead-ends in NXDOMAIN — the record exists on the k8s authoritative servers (and via the resolvers at 198.18.200.7) but has no reachable address. This is the same failure already documented and fixed for haproxy in `profiles::haproxy::dns`.
Changes:
- Set the cobbler CNAME `value` to the fully-qualified `${facts.networking.fqdn}.` (trailing dot) so the target is origin-independent and resolves identically on both the legacy zone-file export and the dns-updater nsupdate path.
- Add an explanatory comment mirroring the one in `profiles::haproxy::dns` to prevent reintroducing a bare label.
Reviewed-on: #491
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- change the default anycast nameserver to bind-internal resolvers in
kubernetes
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #490
## Why
The halb publishes service CNAMEs (`git.unkin.net`, `dashboard.ceph.unkin.net`, and every other halb-fronted name) with a **bare target label**, e.g.:
```
git.unkin.net. 300 IN CNAME au-syd1-prod-halb-vrrp.
```
`dns-updater` builds the RR with `dns.NewRR` and **no $ORIGIN**, so a bare value becomes root-absolute (`au-syd1-prod-halb-vrrp.`) and dead-ends in NXDOMAIN. Once the k8s bind became authoritative for these zones, that broke resolution of every halb-fronted service (git, the Ceph dashboard, ...). `au-syd1-prod-halb-vrrp.unkin.net` / `.main.unkin.net` resolve fine (198.18.19.17) — only the CNAME target was truncated.
## Change (`site/profiles/manifests/haproxy/dns.pp`)
Emit **fully-qualified** CNAME targets (trailing dot), the FQDN form `dns-updater` expects (per its own test fixture `au-syd1-prod-halb.main.unkin.net.`):
- vrrp cnames: `${location_environment}-halb-vrrp.${domain}.`
- non-vrrp cnames: `${location_environment}-halb.${domain}.`
The matching A records are already published in `main.unkin.net`/`unkin.net` just above, so the targets resolve to the VIP.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #489
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`s3.ceph.unkin.net` is the RGW S3 endpoint name used by cephrgw-operator consumers. A companion argocd-apps PR adds it as a CNAME to `radosgw.service.consul`. The radosgw hosts front RGW with nginx (443 -> radosgw:7480) using the vault-issued cert, so **both** the certificate and nginx must recognise the name.
## Changes (hieradata/roles/infra/ceph/rgw.yaml)
- Add `s3.ceph.unkin.net` to `profiles::pki::vault::alt_names` — puts the name in the radosgw cert SAN (renewal re-triggers on the alt-names change).
- Add `s3.ceph.unkin.net` to `profiles::nginx::simpleproxy::nginx_aliases` — nginx answers for that server_name.
### nginx: yes, needed
radosgw sits behind `profiles::nginx::simpleproxy` (listens 443, proxies to :7480) sharing the same vault cert, so the alias is required for nginx to serve the hostname — not just the cert SAN.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #487
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The vault nodes have rancher plugin v0.1.0, which mints non-functional tokens (returns `status.value` not `status.bearerToken`; every cred/rotation 401s). v0.1.1 fixes it. `ensure_packages` defaults to `present`, which won't upgrade an already-installed package, so pin to `latest` to pull v0.1.1.
## Changes
- `openbao-plugin-secrets-rancher: { ensure: latest }` on the vault storage role.
## Note
`latest` will also pull future releases automatically — keep the terraform-vault catalog sha256 (`config/plugins/vault-plugin-secrets-rancher.yaml`) bumped in lockstep, or a vault restart after a future upgrade will fail the sha check. Happy to pin to an exact version instead if you'd prefer lockstep-by-pin.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #488
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Repoints the `rpmfusion-free` and `rpmfusion-nonfree` yum repos from the retired `packagerepo.service.consul` mirror to the artifactapi RPM Fusion remotes.
dnf on media nodes (jellyfin, nzbget) failed with `Cannot download repomd.xml: All mirrors were tried`, which blocked package installs such as `dns-updater`.
Changes:
- `rpmfusion-free` baseurl → `.../api/v1/remote/rpmfusion-free/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-free/el/RPM-GPG-KEY-rpmfusion-free-el-$major`
- `rpmfusion-nonfree` baseurl → `.../api/v1/remote/rpmfusion-nonfree/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-nonfree/el/RPM-GPG-KEY-rpmfusion-nonfree-el-$major`
Depends on terraform-artifactapi PR #7 (adds the rpmfusion remotes) being applied first.
https://claude.ai/code/session_01VZ81u2JtjvaPq8Z3EDTSMb
Reviewed-on: #486
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Configures the dns-updater record filter (dns-updater#3) so k8s/LB/internal addresses stay out of the authoritative zones.
## Why
Hosts publish records the authoritative server should not hold — `*-flannel.1 A 10.42.x`, `*-kube-lb0 A 198.18.200.x`, and the matching reverse PTRs (`200.18.198`, `2.42.10`) — which the daemon WARNs `NOTAUTH`/`bad authentication` on.
## Changes
- `DNS_UPDATER_DENY_RANGES=198.18.199.0/24,198.18.200.0/24,10.42.0.0/16,10.43.0.0/16` (pod CIDR, service CIDR, LB VIP pool, .199).
- New class params: `deny_ranges` / `allow_ranges` / `deny_domains` / `allow_domains`.
- `package_ensure` → `latest` so hosts pick up the filter release (rpm-internal `metadata_expire` is 1h, so no thrash).
## Sequencing
Needs dns-updater **#3** merged + a new tag (v0.2.0) so the filter env vars are honored. The env keys are ignored by v0.1.0, so this is safe to merge first — it takes effect when hosts upgrade to v0.2.0.
Reviewed-on: #485
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: rpmbuilder's vendored third-party RPMs are moving from Gitea's RPM registry to the artifactapi rpm-vendor local repos. Managed AlmaLinux hosts need the matching yum repo to install them; this mirrors the existing rpm-internal repo and selects the per-EL-release repo via os.release.major.
Changes:
- Add rpm-vendor to profiles::yum::global::repos in hieradata/os/AlmaLinux/all_releases.yaml, pointing at .../api/v1/local/rpm-vendor-el<major>/ (el8/el9), gpgcheck disabled and mirrorlist absent, matching the rpm-internal repo definition.
Depends on terraform-artifactapi PR #6 (creates the rpm-vendor-el8/el9 repos).
Reviewed-on: #484
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Rancher token secrets engine needs its plugin binary present in the OpenBao plugin directory before terraform-vault can register/mount it. Mirrors #480 (gpg).
## Changes
- Add `openbao-plugin-secrets-rancher` to `profiles::packages::include` on the vault storage role. Installs the RPM (from rpm-internal) to `/opt/openbao-plugins/vault-plugin-secrets-rancher`.
## Merge order
Part 3 of 4 — merge first (binary must exist on disk before the terraform-vault plugin-import #92 registers it with a matching sha256).
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #483
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Replace the `profiles::dns::updater` shell mechanism (`dns-update.sh` + `dns-update.path`/`.service` + the in-run `exec`) with the packaged **dns-updater** daemon. The daemon watches the records file (inotify) and network interfaces and pushes TSIG-signed RFC2136 updates to BIND natively — with structured per-zone RCODEs and a status API/facter fact, so failures like the recent `invalid owner name: empty label` / NOTZONE surface directly instead of as opaque nsupdate stderr.
## Changes
- Install the `dns-updater` package; manage `/etc/dns-updater/env`.
- Run the packaged `dns-updater.service`, restarting **only** on env/key change — records-file edits are picked up by the daemon`s own inotify watch, so no service churn on record changes.
- Keep the `concat` records file and the TSIG key file unchanged (same paths/format).
- Ensure the old `/usr/local/bin/dns-update` + `dns-update.path`/`.service` units are absent.
- Drop the now-dead `dns-update.{sh,service,path}.epp` templates.
## Sequencing — HOLD
Do not merge until the `dns-updater` RPM is published to artifactapi `rpm-internal` (needs terraform-git#33 to create the repo, then the daemon code pushed + tagged so Woodpecker builds the RPM). Merging before the package exists makes `package { dns-updater }` fail on every host.
Supersedes the interim shell fix in #481 (which stays valid until this rolls out). Keeps the file as the desired-state interface (puppet owns desired records; the daemon reconciles + reports).
Reviewed-on: #482
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
6 hosts (ausyd1nxvm2069-2073, 2098) ended up with a reverse PTR in bind-authoritative but **no forward A record**, and the `unkin.net` service records (git/grafana/auth/fafflix, all published by the halb host 2069) never landed at all.
VictoriaLogs (`dns-update-apply` on 2069 & 2070) shows the cause:
```
dns-update: nsupdate to 198.18.200.9 failed
invalid owner name: empty label
syntax error
```
`profiles::dns::record` publishes some records whose name is already fully-qualified (trailing dot) — e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`, `dashboard.ceph.unkin.net.`, and the halb CNAMEs. The `dns-update` script `fqdn()` unconditionally appended the zone, producing `…net..main.unkin.net.` — the `..` is an empty label, which nsupdate rejects, failing the entire per-zone `send`. The reverse-PTR send is sorted first and its name is always relative, so it still applied — hence "PTR but no A".
## Change
`fqdn()` now handles three cases:
- `@`/empty → zone apex (unchanged)
- name ending in `.` → already FQDN, used verbatim (**the fix**)
- otherwise → relative, append `.zone.` (unchanged)
Verified against all record shapes (relative host, apex, FQDN CNAME, reverse label) — no more `..`.
## After merge
Once puppet re-runs on the affected hosts their `main.unkin.net`/`unkin.net` updates succeed, filling in the missing A records and the `unkin.net` service zone. Pairs with argocd-apps#260 (adds the `ceph.unkin.net` zone so `dashboard.ceph.unkin.net` does not then hit NOTZONE).
Reviewed-on: #481
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).
- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.
Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.
Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
enable installing locally packaged rpms with no specific linux release
- add the rpm-internal repo from artifactapi
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #476
## Summary
- Replace static `registries.yaml` with EPP template driven by `rke2::registries` hash
- Add `disable-default-registry-endpoint: true` to all mirrors — RKE2 will only use artifactapi and never fall back to upstream registries
- Registry configuration now fully managed via hiera data (`roles/infra/k8s.yaml`)
Reviewed-on: #474
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Add/Remove the registries.yaml file based on the manage_registries
boolean. We are leaving it on default=false now as the artifactapi
server was broken.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #472
Pin grafana package version to 13.0.2 via a new version parameter on
profiles::metrics::grafana, wired through to the puppet-grafana class.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #470
- update release to install to 1.26.2
- change base_url to artifactapi
- update releases/checksums
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #469
currently, all woodpecker jobs jam onto one host, and have no resource
limits resulting in one kubernetes host suddenly maxing its cpu
- ensure we allocate resources for each woodpecker job
Reviewed-on: #463
## Summary
- Adds `Unkin::Ceph::Utils` facter module detecting ceph service instances via `systemctl list-units`, exposing `is_ceph_mon`, `is_ceph_mgr`, `is_ceph_mds`, `is_ceph_osd` booleans and a `ceph_services` hash of unit names
- Adds `profiles::ceph::mon`, `mgr`, `mds`, `osd` — each with `Boolean $ensure_running` that iterates discovered service instances and manages them as running and enabled
- Works across incus nodes (mon/mgr/mds/osd) and k8s compute/control nodes (osd only); verified on prodnxsr0001 which correctly reports `is_ceph_osd: true` and `ceph_services: {osd: [ceph-osd@5]}`
## Test plan
- [x] Noop deploy against prodnxsr0001.main.unkin.net passed cleanly
- [x] `ceph_services` fact returns correct service map
- [x] `is_ceph_osd` returns `True`, `is_ceph_mon` returns `False` as expected
- [x] Test on an incus/ceph node with mon/mgr/mds services
Reviewed-on: #459
rebuilding router, taking the chance to not mess up ip ranges. I did
have 198.18.21.0/24 and 198.18.21.160/27 and 198.18.21.192/27 all on
differnt interfaces.
- update IP's that can reach bind view for main.unkin.net
- keep both for intermediate period
Reviewed-on: #460
Replace deprecated dalen-puppetdbquery module with native puppetdb_query
function using PQL syntax to resolve URI.escape compatibility issues.
This is required to migrated to Puppet 8 (and kubernetes).
Changes:
- Remove dalen-puppetdbquery dependency from Puppetfile
- Replace query_nodes() calls with puppetdb_query() using PQL syntax
- Update 27 function calls across 18 Puppet manifests
- Maintain equivalent functionality with improved compatibility
Reviewed-on: #457
split all pre-commit checks into individual workflows, so that
woodpecker spawns a container/job for each. this vastly improves the
time it takes for CI to complete checks for puppet
- create per-pre-commit-check pre-commit config files
- create per-pre-commit-check woodpecker workflows
Reviewed-on: #455
need to separate the permissions inside vault into different groups, one
per-permission.
- add group for each kubernetes role in vault
Reviewed-on: #449
unkinben
benvin