rpmbuilder's vendored third-party RPMs are moving from Gitea's RPM registry
to the artifactapi rpm-vendor local repos. Managed AlmaLinux hosts need the
matching yum repo to install them; this mirrors the existing rpm-internal
repo and selects the per-EL-release repo via os.release.major.
- Add rpm-vendor to profiles::yum::global::repos in
hieradata/os/AlmaLinux/all_releases.yaml, pointing at
.../api/v1/local/rpm-vendor-el<major>/ (el8/el9), gpgcheck disabled and
mirrorlist absent, matching the rpm-internal repo definition.
## Why
Replace the `profiles::dns::updater` shell mechanism (`dns-update.sh` + `dns-update.path`/`.service` + the in-run `exec`) with the packaged **dns-updater** daemon. The daemon watches the records file (inotify) and network interfaces and pushes TSIG-signed RFC2136 updates to BIND natively — with structured per-zone RCODEs and a status API/facter fact, so failures like the recent `invalid owner name: empty label` / NOTZONE surface directly instead of as opaque nsupdate stderr.
## Changes
- Install the `dns-updater` package; manage `/etc/dns-updater/env`.
- Run the packaged `dns-updater.service`, restarting **only** on env/key change — records-file edits are picked up by the daemon`s own inotify watch, so no service churn on record changes.
- Keep the `concat` records file and the TSIG key file unchanged (same paths/format).
- Ensure the old `/usr/local/bin/dns-update` + `dns-update.path`/`.service` units are absent.
- Drop the now-dead `dns-update.{sh,service,path}.epp` templates.
## Sequencing — HOLD
Do not merge until the `dns-updater` RPM is published to artifactapi `rpm-internal` (needs terraform-git#33 to create the repo, then the daemon code pushed + tagged so Woodpecker builds the RPM). Merging before the package exists makes `package { dns-updater }` fail on every host.
Supersedes the interim shell fix in #481 (which stays valid until this rolls out). Keeps the file as the desired-state interface (puppet owns desired records; the daemon reconciles + reports).
Reviewed-on: #482
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
6 hosts (ausyd1nxvm2069-2073, 2098) ended up with a reverse PTR in bind-authoritative but **no forward A record**, and the `unkin.net` service records (git/grafana/auth/fafflix, all published by the halb host 2069) never landed at all.
VictoriaLogs (`dns-update-apply` on 2069 & 2070) shows the cause:
```
dns-update: nsupdate to 198.18.200.9 failed
invalid owner name: empty label
syntax error
```
`profiles::dns::record` publishes some records whose name is already fully-qualified (trailing dot) — e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`, `dashboard.ceph.unkin.net.`, and the halb CNAMEs. The `dns-update` script `fqdn()` unconditionally appended the zone, producing `…net..main.unkin.net.` — the `..` is an empty label, which nsupdate rejects, failing the entire per-zone `send`. The reverse-PTR send is sorted first and its name is always relative, so it still applied — hence "PTR but no A".
## Change
`fqdn()` now handles three cases:
- `@`/empty → zone apex (unchanged)
- name ending in `.` → already FQDN, used verbatim (**the fix**)
- otherwise → relative, append `.zone.` (unchanged)
Verified against all record shapes (relative host, apex, FQDN CNAME, reverse label) — no more `..`.
## After merge
Once puppet re-runs on the affected hosts their `main.unkin.net`/`unkin.net` updates succeed, filling in the missing A records and the `unkin.net` service zone. Pairs with argocd-apps#260 (adds the `ceph.unkin.net` zone so `dashboard.ceph.unkin.net` does not then hit NOTZONE).
Reviewed-on: #481
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).
- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.
Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.
Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
enable installing locally packaged rpms with no specific linux release
- add the rpm-internal repo from artifactapi
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #476
## Summary
- Replace static `registries.yaml` with EPP template driven by `rke2::registries` hash
- Add `disable-default-registry-endpoint: true` to all mirrors — RKE2 will only use artifactapi and never fall back to upstream registries
- Registry configuration now fully managed via hiera data (`roles/infra/k8s.yaml`)
Reviewed-on: #474
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Add/Remove the registries.yaml file based on the manage_registries
boolean. We are leaving it on default=false now as the artifactapi
server was broken.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #472
Pin grafana package version to 13.0.2 via a new version parameter on
profiles::metrics::grafana, wired through to the puppet-grafana class.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #470
- update release to install to 1.26.2
- change base_url to artifactapi
- update releases/checksums
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #469
currently, all woodpecker jobs jam onto one host, and have no resource
limits resulting in one kubernetes host suddenly maxing its cpu
- ensure we allocate resources for each woodpecker job
Reviewed-on: #463
## Summary
- Adds `Unkin::Ceph::Utils` facter module detecting ceph service instances via `systemctl list-units`, exposing `is_ceph_mon`, `is_ceph_mgr`, `is_ceph_mds`, `is_ceph_osd` booleans and a `ceph_services` hash of unit names
- Adds `profiles::ceph::mon`, `mgr`, `mds`, `osd` — each with `Boolean $ensure_running` that iterates discovered service instances and manages them as running and enabled
- Works across incus nodes (mon/mgr/mds/osd) and k8s compute/control nodes (osd only); verified on prodnxsr0001 which correctly reports `is_ceph_osd: true` and `ceph_services: {osd: [ceph-osd@5]}`
## Test plan
- [x] Noop deploy against prodnxsr0001.main.unkin.net passed cleanly
- [x] `ceph_services` fact returns correct service map
- [x] `is_ceph_osd` returns `True`, `is_ceph_mon` returns `False` as expected
- [x] Test on an incus/ceph node with mon/mgr/mds services
Reviewed-on: #459
rebuilding router, taking the chance to not mess up ip ranges. I did
have 198.18.21.0/24 and 198.18.21.160/27 and 198.18.21.192/27 all on
differnt interfaces.
- update IP's that can reach bind view for main.unkin.net
- keep both for intermediate period
Reviewed-on: #460
Replace deprecated dalen-puppetdbquery module with native puppetdb_query
function using PQL syntax to resolve URI.escape compatibility issues.
This is required to migrated to Puppet 8 (and kubernetes).
Changes:
- Remove dalen-puppetdbquery dependency from Puppetfile
- Replace query_nodes() calls with puppetdb_query() using PQL syntax
- Update 27 function calls across 18 Puppet manifests
- Maintain equivalent functionality with improved compatibility
Reviewed-on: #457
split all pre-commit checks into individual workflows, so that
woodpecker spawns a container/job for each. this vastly improves the
time it takes for CI to complete checks for puppet
- create per-pre-commit-check pre-commit config files
- create per-pre-commit-check woodpecker workflows
Reviewed-on: #455
need to separate the permissions inside vault into different groups, one
per-permission.
- add group for each kubernetes role in vault
Reviewed-on: #449
unkinben
benvin