## Why
The halb publishes service CNAMEs (`git.unkin.net`, `dashboard.ceph.unkin.net`, and every other halb-fronted name) with a **bare target label**, e.g.:
```
git.unkin.net. 300 IN CNAME au-syd1-prod-halb-vrrp.
```
`dns-updater` builds the RR with `dns.NewRR` and **no $ORIGIN**, so a bare value becomes root-absolute (`au-syd1-prod-halb-vrrp.`) and dead-ends in NXDOMAIN. Once the k8s bind became authoritative for these zones, that broke resolution of every halb-fronted service (git, the Ceph dashboard, ...). `au-syd1-prod-halb-vrrp.unkin.net` / `.main.unkin.net` resolve fine (198.18.19.17) — only the CNAME target was truncated.
## Change (`site/profiles/manifests/haproxy/dns.pp`)
Emit **fully-qualified** CNAME targets (trailing dot), the FQDN form `dns-updater` expects (per its own test fixture `au-syd1-prod-halb.main.unkin.net.`):
- vrrp cnames: `${location_environment}-halb-vrrp.${domain}.`
- non-vrrp cnames: `${location_environment}-halb.${domain}.`
The matching A records are already published in `main.unkin.net`/`unkin.net` just above, so the targets resolve to the VIP.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #489
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`s3.ceph.unkin.net` is the RGW S3 endpoint name used by cephrgw-operator consumers. A companion argocd-apps PR adds it as a CNAME to `radosgw.service.consul`. The radosgw hosts front RGW with nginx (443 -> radosgw:7480) using the vault-issued cert, so **both** the certificate and nginx must recognise the name.
## Changes (hieradata/roles/infra/ceph/rgw.yaml)
- Add `s3.ceph.unkin.net` to `profiles::pki::vault::alt_names` — puts the name in the radosgw cert SAN (renewal re-triggers on the alt-names change).
- Add `s3.ceph.unkin.net` to `profiles::nginx::simpleproxy::nginx_aliases` — nginx answers for that server_name.
### nginx: yes, needed
radosgw sits behind `profiles::nginx::simpleproxy` (listens 443, proxies to :7480) sharing the same vault cert, so the alias is required for nginx to serve the hostname — not just the cert SAN.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #487
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The vault nodes have rancher plugin v0.1.0, which mints non-functional tokens (returns `status.value` not `status.bearerToken`; every cred/rotation 401s). v0.1.1 fixes it. `ensure_packages` defaults to `present`, which won't upgrade an already-installed package, so pin to `latest` to pull v0.1.1.
## Changes
- `openbao-plugin-secrets-rancher: { ensure: latest }` on the vault storage role.
## Note
`latest` will also pull future releases automatically — keep the terraform-vault catalog sha256 (`config/plugins/vault-plugin-secrets-rancher.yaml`) bumped in lockstep, or a vault restart after a future upgrade will fail the sha check. Happy to pin to an exact version instead if you'd prefer lockstep-by-pin.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #488
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Repoints the `rpmfusion-free` and `rpmfusion-nonfree` yum repos from the retired `packagerepo.service.consul` mirror to the artifactapi RPM Fusion remotes.
dnf on media nodes (jellyfin, nzbget) failed with `Cannot download repomd.xml: All mirrors were tried`, which blocked package installs such as `dns-updater`.
Changes:
- `rpmfusion-free` baseurl → `.../api/v1/remote/rpmfusion-free/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-free/el/RPM-GPG-KEY-rpmfusion-free-el-$major`
- `rpmfusion-nonfree` baseurl → `.../api/v1/remote/rpmfusion-nonfree/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-nonfree/el/RPM-GPG-KEY-rpmfusion-nonfree-el-$major`
Depends on terraform-artifactapi PR #7 (adds the rpmfusion remotes) being applied first.
https://claude.ai/code/session_01VZ81u2JtjvaPq8Z3EDTSMb
Reviewed-on: #486
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Configures the dns-updater record filter (dns-updater#3) so k8s/LB/internal addresses stay out of the authoritative zones.
## Why
Hosts publish records the authoritative server should not hold — `*-flannel.1 A 10.42.x`, `*-kube-lb0 A 198.18.200.x`, and the matching reverse PTRs (`200.18.198`, `2.42.10`) — which the daemon WARNs `NOTAUTH`/`bad authentication` on.
## Changes
- `DNS_UPDATER_DENY_RANGES=198.18.199.0/24,198.18.200.0/24,10.42.0.0/16,10.43.0.0/16` (pod CIDR, service CIDR, LB VIP pool, .199).
- New class params: `deny_ranges` / `allow_ranges` / `deny_domains` / `allow_domains`.
- `package_ensure` → `latest` so hosts pick up the filter release (rpm-internal `metadata_expire` is 1h, so no thrash).
## Sequencing
Needs dns-updater **#3** merged + a new tag (v0.2.0) so the filter env vars are honored. The env keys are ignored by v0.1.0, so this is safe to merge first — it takes effect when hosts upgrade to v0.2.0.
Reviewed-on: #485
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: rpmbuilder's vendored third-party RPMs are moving from Gitea's RPM registry to the artifactapi rpm-vendor local repos. Managed AlmaLinux hosts need the matching yum repo to install them; this mirrors the existing rpm-internal repo and selects the per-EL-release repo via os.release.major.
Changes:
- Add rpm-vendor to profiles::yum::global::repos in hieradata/os/AlmaLinux/all_releases.yaml, pointing at .../api/v1/local/rpm-vendor-el<major>/ (el8/el9), gpgcheck disabled and mirrorlist absent, matching the rpm-internal repo definition.
Depends on terraform-artifactapi PR #6 (creates the rpm-vendor-el8/el9 repos).
Reviewed-on: #484
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Rancher token secrets engine needs its plugin binary present in the OpenBao plugin directory before terraform-vault can register/mount it. Mirrors #480 (gpg).
## Changes
- Add `openbao-plugin-secrets-rancher` to `profiles::packages::include` on the vault storage role. Installs the RPM (from rpm-internal) to `/opt/openbao-plugins/vault-plugin-secrets-rancher`.
## Merge order
Part 3 of 4 — merge first (binary must exist on disk before the terraform-vault plugin-import #92 registers it with a matching sha256).
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #483
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Replace the `profiles::dns::updater` shell mechanism (`dns-update.sh` + `dns-update.path`/`.service` + the in-run `exec`) with the packaged **dns-updater** daemon. The daemon watches the records file (inotify) and network interfaces and pushes TSIG-signed RFC2136 updates to BIND natively — with structured per-zone RCODEs and a status API/facter fact, so failures like the recent `invalid owner name: empty label` / NOTZONE surface directly instead of as opaque nsupdate stderr.
## Changes
- Install the `dns-updater` package; manage `/etc/dns-updater/env`.
- Run the packaged `dns-updater.service`, restarting **only** on env/key change — records-file edits are picked up by the daemon`s own inotify watch, so no service churn on record changes.
- Keep the `concat` records file and the TSIG key file unchanged (same paths/format).
- Ensure the old `/usr/local/bin/dns-update` + `dns-update.path`/`.service` units are absent.
- Drop the now-dead `dns-update.{sh,service,path}.epp` templates.
## Sequencing — HOLD
Do not merge until the `dns-updater` RPM is published to artifactapi `rpm-internal` (needs terraform-git#33 to create the repo, then the daemon code pushed + tagged so Woodpecker builds the RPM). Merging before the package exists makes `package { dns-updater }` fail on every host.
Supersedes the interim shell fix in #481 (which stays valid until this rolls out). Keeps the file as the desired-state interface (puppet owns desired records; the daemon reconciles + reports).
Reviewed-on: #482
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
6 hosts (ausyd1nxvm2069-2073, 2098) ended up with a reverse PTR in bind-authoritative but **no forward A record**, and the `unkin.net` service records (git/grafana/auth/fafflix, all published by the halb host 2069) never landed at all.
VictoriaLogs (`dns-update-apply` on 2069 & 2070) shows the cause:
```
dns-update: nsupdate to 198.18.200.9 failed
invalid owner name: empty label
syntax error
```
`profiles::dns::record` publishes some records whose name is already fully-qualified (trailing dot) — e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`, `dashboard.ceph.unkin.net.`, and the halb CNAMEs. The `dns-update` script `fqdn()` unconditionally appended the zone, producing `…net..main.unkin.net.` — the `..` is an empty label, which nsupdate rejects, failing the entire per-zone `send`. The reverse-PTR send is sorted first and its name is always relative, so it still applied — hence "PTR but no A".
## Change
`fqdn()` now handles three cases:
- `@`/empty → zone apex (unchanged)
- name ending in `.` → already FQDN, used verbatim (**the fix**)
- otherwise → relative, append `.zone.` (unchanged)
Verified against all record shapes (relative host, apex, FQDN CNAME, reverse label) — no more `..`.
## After merge
Once puppet re-runs on the affected hosts their `main.unkin.net`/`unkin.net` updates succeed, filling in the missing A records and the `unkin.net` service zone. Pairs with argocd-apps#260 (adds the `ceph.unkin.net` zone so `dashboard.ceph.unkin.net` does not then hit NOTZONE).
Reviewed-on: #481
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).
- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.
Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.
Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
enable installing locally packaged rpms with no specific linux release
- add the rpm-internal repo from artifactapi
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #476
## Summary
- Replace static `registries.yaml` with EPP template driven by `rke2::registries` hash
- Add `disable-default-registry-endpoint: true` to all mirrors — RKE2 will only use artifactapi and never fall back to upstream registries
- Registry configuration now fully managed via hiera data (`roles/infra/k8s.yaml`)
Reviewed-on: #474
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Add/Remove the registries.yaml file based on the manage_registries
boolean. We are leaving it on default=false now as the artifactapi
server was broken.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #472
Pin grafana package version to 13.0.2 via a new version parameter on
profiles::metrics::grafana, wired through to the puppet-grafana class.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #470
- update release to install to 1.26.2
- change base_url to artifactapi
- update releases/checksums
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #469
currently, all woodpecker jobs jam onto one host, and have no resource
limits resulting in one kubernetes host suddenly maxing its cpu
- ensure we allocate resources for each woodpecker job
Reviewed-on: #463
## Summary
- Adds `Unkin::Ceph::Utils` facter module detecting ceph service instances via `systemctl list-units`, exposing `is_ceph_mon`, `is_ceph_mgr`, `is_ceph_mds`, `is_ceph_osd` booleans and a `ceph_services` hash of unit names
- Adds `profiles::ceph::mon`, `mgr`, `mds`, `osd` — each with `Boolean $ensure_running` that iterates discovered service instances and manages them as running and enabled
- Works across incus nodes (mon/mgr/mds/osd) and k8s compute/control nodes (osd only); verified on prodnxsr0001 which correctly reports `is_ceph_osd: true` and `ceph_services: {osd: [ceph-osd@5]}`
## Test plan
- [x] Noop deploy against prodnxsr0001.main.unkin.net passed cleanly
- [x] `ceph_services` fact returns correct service map
- [x] `is_ceph_osd` returns `True`, `is_ceph_mon` returns `False` as expected
- [x] Test on an incus/ceph node with mon/mgr/mds services
Reviewed-on: #459
rebuilding router, taking the chance to not mess up ip ranges. I did
have 198.18.21.0/24 and 198.18.21.160/27 and 198.18.21.192/27 all on
differnt interfaces.
- update IP's that can reach bind view for main.unkin.net
- keep both for intermediate period
Reviewed-on: #460
Replace deprecated dalen-puppetdbquery module with native puppetdb_query
function using PQL syntax to resolve URI.escape compatibility issues.
This is required to migrated to Puppet 8 (and kubernetes).
Changes:
- Remove dalen-puppetdbquery dependency from Puppetfile
- Replace query_nodes() calls with puppetdb_query() using PQL syntax
- Update 27 function calls across 18 Puppet manifests
- Maintain equivalent functionality with improved compatibility
Reviewed-on: #457
split all pre-commit checks into individual workflows, so that
woodpecker spawns a container/job for each. this vastly improves the
time it takes for CI to complete checks for puppet
- create per-pre-commit-check pre-commit config files
- create per-pre-commit-check woodpecker workflows
Reviewed-on: #455
need to separate the permissions inside vault into different groups, one
per-permission.
- add group for each kubernetes role in vault
Reviewed-on: #449
unkinben
benvin