Finish the artifactapi migration by dropping the last legacy Gitea-hosted
(git.query.consul) yum repo. The yum profile's purge => true then removes
/etc/yum.repos.d/unkin.repo from hosts. The jellyfin media role was the only
consumer; its jellyfin-* packages (jellyfin-web, jellyfin-server,
jellyfin-ffmpeg-bin) live only in unkinben and will stop receiving updates
until republished into artifactapi (e.g. rpm-internal-elN). Its other deps
(SDL2, ffmpeg, ffmpeg-devel) come from rpmfusion and are unaffected.
- remove the unkinben yumrepo definition from AlmaLinux/all_releases.yaml
- remove the unkinben ensure: present override from the jellyfin role
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
AlmaLinux hosts still pulled in-house RPMs from the legacy unkin repo on
Gitea (git.query.consul). Point them at the artifactapi local rpm repos
instead, adding both the distribution-agnostic and per-EL-major variants of
rpm-internal and rpm-vendor so hosts get everything published for their
release. The unkinben personal repo is left untouched because the jellyfin
media role still enables it and its content is not on artifactapi yet.
- retarget rpm-vendor to the distribution-agnostic local/rpm-vendor repo
- add rpm-internal-release/rpm-vendor-release, resolved from the host's
os.release.major (el8 hosts -> -el8, el9 -> -el9, etc.)
- remove the legacy unkin yumrepo definition
Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
## Why
During the k8s DNS cutover the `dns` module dual-writes every host's records two
ways: (a) LEGACY exported `@@concat::fragment`s collected onto the VM puppet DNS
master into `/var/named/sources/<zone>.conf`, and (b) NEW nsupdate/RFC2136 to the
k8s bind-authoritative cluster (`manage_nsupdate`). Both were gated on so the
legacy path stayed as a safety net.
The legacy path is no longer in any client's resolution path:
- **#490** repointed `profiles::dns::base::nameservers` from the VM resolver
anycast `198.18.19.16` to the k8s **bind-resolvers** LB `198.18.200.7`. Every
puppet client now resolves there.
- The k8s bind-resolvers forward `unkin.net`, `main.unkin.net`, and all
`18.198.in-addr.arpa` reverse zones to the k8s **bind-authoritative** cluster
(`198.18.200.6`), which is populated purely by the nsupdate path — the same
`profiles::dns::record` defines, `manage_nsupdate` branch.
- The VM master's assembled `/var/named/sources` files are therefore orphaned:
nothing resolves against them anymore.
This flips `profiles::dns::updater::manage_export: false` to stop nodes exporting
the legacy fragments. `manage_nsupdate` stays on.
## What changes
- `hieradata/common.yaml`: `profiles::dns::updater::manage_export` `true` -> `false`.
## Verification evidence
Pulled all **498** exported `Concat::Fragment` resources from PuppetDB (239 A,
232 PTR, 27 CNAME; 490 unique fqdn/type/value) and `dig`'d each against the k8s
authoritative read endpoint `198.18.200.7` (the dynamic path's serving side):
| Result | Count |
|---|---|
| MATCH | 447 |
| MISSING-from-dynamic | 38 |
| DIFFERENT-value | 5 |
Both non-match buckets are accounted for and are **not coverage gaps**:
- **38 MISSING** are all records the dynamic path deliberately filters via
`profiles::dns::updater::deny_ranges`
(`198.18.200.0/24`, `10.42.0.0/16`, ...) plus reverse zones the authoritative
cluster isn't even authoritative for: `*-kube-lb0` (198.18.200.x),
`*-flannel.1` (10.42.x.x pod CIDR), and `*-nomad`/`*-brcom1`
(172.26.64.x / 10.255.255.x). These are exactly the k8s/LB noise the deny
ranges exist to keep out of the authoritative zones. Verified every one falls
in a denied/non-authoritative range.
- **5 DIFFERENT** are stale legacy duplicates that the dynamic path holds *more*
correctly:
- 4x `au-syd1-prod-halb.main.unkin.net` A: four ceph nodes (2069-2073) each
self-register the shared name with their own primary IP (haproxy::dns
exports `networking.ip` per host, a legacy collision). Dynamic resolves it to
the single VIP `198.18.24.60`.
- 1x `dashboard.ceph.unkin.net` CNAME: the ceph node self-publishes this to
`lb1.unkin.net.` via its own nsupdate (per the k8s zones.yaml note); the
exported VRRP-name fragment is stale.
Conclusion: the dynamic path fully covers the served record set, and the serving
path no longer depends on the exported files.
## Collector-behavior note (what the flip actually does)
`profiles::dns::zone` realizes the fragments with an exported-resource collector
(`Concat::Fragment <<| target == ... |>>`). Turning off export purges the
fragments from PuppetDB, so on the VM master concat rebuilds each
`/var/named/sources/<zone>.conf` down to just the header (SOA+NS+glue) as nodes
converge. That is **cosmetic**: no client resolves against the VM master anymore
(#490), and the k8s authoritative zones are unaffected (fed by nsupdate, not
these files).
## Rollback
Set `profiles::dns::updater::manage_export: true` again and let puppet converge;
nodes re-export their fragments and the VM-master zone files repopulate.
## Follow-up (after soak)
Deleting the `@@concat::fragment` / `Concat::Fragment <<||>>` collector code in
`profiles::dns::record` / `profiles::dns::zone` (and the VM `profiles::dns::master`
zone assembly) is a separate cleanup PR once this has soaked and the VM master is
confirmed decommissionable.
Reviewed-on: #495
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
First live agent migration to the k8s puppetserver, using the single-boolean mechanism from #493 on the lowest-risk host (roles::base test host).
- Adds hieradata/nodes/ausyd1nxvm2120.main.unkin.net.yaml with profiles::puppet::migrate::enabled: true
On merge: the node's next run (from the old master) rewrites puppet.conf to the k8s server/ca_server with a fresh ssl-k8s ssldir; the run after enrols against the k8s CA (autosigned) and compiles from the k8s compilers (ENC=encapi). Verify: node appears in the k8s PuppetDB with a fresh report. Rollback: revert this file — legacy ssldir/cert remain intact.
Reviewed-on: #494
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
`ensure: present`/`latest` lets the plugin binaries drift from the sha256 pinned in the terraform-vault catalog (`config/plugins/*.yaml`). On the next OpenBao restart, a drifted binary fails the sha check and the plugin won't launch — a latent footgun (hit exactly this with rancher on `ensure: latest`).
## Changes
Pin each secrets plugin to the version whose binary matches its registered catalog sha (all verified against the RPMs in rpm-internal):
- `openbao-plugin-secrets-litellm`: **0.1.1** (sha 2263ebcb…)
- `openbao-plugin-secrets-gpg`: **0.1.0** (sha 0e92d740…)
- `openbao-plugin-secrets-rancher`: **0.1.1** (sha 9e597cd9…; was `ensure: latest`)
All three are no-op on the binary (installed versions already match) — this just locks them so a future release can't silently upgrade the binary out of lockstep with the catalog.
`openbao-plugins` (base bundle) left unpinned — its version couldn't be verified from the tooling side and it tracks the openbao package, not a catalog sha.
## Note
To upgrade a plugin in future: bump the RPM version here **and** the catalog sha256 in terraform-vault in the same change, then `vault write sys/plugins/reload/backend plugin=<name>`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #492
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- change the default anycast nameserver to bind-internal resolvers in
kubernetes
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #490
## Why
`s3.ceph.unkin.net` is the RGW S3 endpoint name used by cephrgw-operator consumers. A companion argocd-apps PR adds it as a CNAME to `radosgw.service.consul`. The radosgw hosts front RGW with nginx (443 -> radosgw:7480) using the vault-issued cert, so **both** the certificate and nginx must recognise the name.
## Changes (hieradata/roles/infra/ceph/rgw.yaml)
- Add `s3.ceph.unkin.net` to `profiles::pki::vault::alt_names` — puts the name in the radosgw cert SAN (renewal re-triggers on the alt-names change).
- Add `s3.ceph.unkin.net` to `profiles::nginx::simpleproxy::nginx_aliases` — nginx answers for that server_name.
### nginx: yes, needed
radosgw sits behind `profiles::nginx::simpleproxy` (listens 443, proxies to :7480) sharing the same vault cert, so the alias is required for nginx to serve the hostname — not just the cert SAN.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #487
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The vault nodes have rancher plugin v0.1.0, which mints non-functional tokens (returns `status.value` not `status.bearerToken`; every cred/rotation 401s). v0.1.1 fixes it. `ensure_packages` defaults to `present`, which won't upgrade an already-installed package, so pin to `latest` to pull v0.1.1.
## Changes
- `openbao-plugin-secrets-rancher: { ensure: latest }` on the vault storage role.
## Note
`latest` will also pull future releases automatically — keep the terraform-vault catalog sha256 (`config/plugins/vault-plugin-secrets-rancher.yaml`) bumped in lockstep, or a vault restart after a future upgrade will fail the sha check. Happy to pin to an exact version instead if you'd prefer lockstep-by-pin.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #488
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Repoints the `rpmfusion-free` and `rpmfusion-nonfree` yum repos from the retired `packagerepo.service.consul` mirror to the artifactapi RPM Fusion remotes.
dnf on media nodes (jellyfin, nzbget) failed with `Cannot download repomd.xml: All mirrors were tried`, which blocked package installs such as `dns-updater`.
Changes:
- `rpmfusion-free` baseurl → `.../api/v1/remote/rpmfusion-free/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-free/el/RPM-GPG-KEY-rpmfusion-free-el-$major`
- `rpmfusion-nonfree` baseurl → `.../api/v1/remote/rpmfusion-nonfree/el/updates/$major/$arch/`, gpgkey → `.../rpmfusion-nonfree/el/RPM-GPG-KEY-rpmfusion-nonfree-el-$major`
Depends on terraform-artifactapi PR #7 (adds the rpmfusion remotes) being applied first.
https://claude.ai/code/session_01VZ81u2JtjvaPq8Z3EDTSMb
Reviewed-on: #486
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: rpmbuilder's vendored third-party RPMs are moving from Gitea's RPM registry to the artifactapi rpm-vendor local repos. Managed AlmaLinux hosts need the matching yum repo to install them; this mirrors the existing rpm-internal repo and selects the per-EL-release repo via os.release.major.
Changes:
- Add rpm-vendor to profiles::yum::global::repos in hieradata/os/AlmaLinux/all_releases.yaml, pointing at .../api/v1/local/rpm-vendor-el<major>/ (el8/el9), gpgcheck disabled and mirrorlist absent, matching the rpm-internal repo definition.
Depends on terraform-artifactapi PR #6 (creates the rpm-vendor-el8/el9 repos).
Reviewed-on: #484
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Rancher token secrets engine needs its plugin binary present in the OpenBao plugin directory before terraform-vault can register/mount it. Mirrors #480 (gpg).
## Changes
- Add `openbao-plugin-secrets-rancher` to `profiles::packages::include` on the vault storage role. Installs the RPM (from rpm-internal) to `/opt/openbao-plugins/vault-plugin-secrets-rancher`.
## Merge order
Part 3 of 4 — merge first (binary must exist on disk before the terraform-vault plugin-import #92 registers it with a matching sha256).
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #483
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).
- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.
Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.
Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
enable installing locally packaged rpms with no specific linux release
- add the rpm-internal repo from artifactapi
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #476
## Summary
- Replace static `registries.yaml` with EPP template driven by `rke2::registries` hash
- Add `disable-default-registry-endpoint: true` to all mirrors — RKE2 will only use artifactapi and never fall back to upstream registries
- Registry configuration now fully managed via hiera data (`roles/infra/k8s.yaml`)
Reviewed-on: #474
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Pin grafana package version to 13.0.2 via a new version parameter on
profiles::metrics::grafana, wired through to the puppet-grafana class.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #470
- update release to install to 1.26.2
- change base_url to artifactapi
- update releases/checksums
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #469
## Summary
- Adds `Unkin::Ceph::Utils` facter module detecting ceph service instances via `systemctl list-units`, exposing `is_ceph_mon`, `is_ceph_mgr`, `is_ceph_mds`, `is_ceph_osd` booleans and a `ceph_services` hash of unit names
- Adds `profiles::ceph::mon`, `mgr`, `mds`, `osd` — each with `Boolean $ensure_running` that iterates discovered service instances and manages them as running and enabled
- Works across incus nodes (mon/mgr/mds/osd) and k8s compute/control nodes (osd only); verified on prodnxsr0001 which correctly reports `is_ceph_osd: true` and `ceph_services: {osd: [ceph-osd@5]}`
## Test plan
- [x] Noop deploy against prodnxsr0001.main.unkin.net passed cleanly
- [x] `ceph_services` fact returns correct service map
- [x] `is_ceph_osd` returns `True`, `is_ceph_mon` returns `False` as expected
- [x] Test on an incus/ceph node with mon/mgr/mds services
Reviewed-on: #459
rebuilding router, taking the chance to not mess up ip ranges. I did
have 198.18.21.0/24 and 198.18.21.160/27 and 198.18.21.192/27 all on
differnt interfaces.
- update IP's that can reach bind view for main.unkin.net
- keep both for intermediate period
Reviewed-on: #460
need to separate the permissions inside vault into different groups, one
per-permission.
- add group for each kubernetes role in vault
Reviewed-on: #449
saving artifacts are breaking in some actions as the runner will switch
between different git hosts. using haproxy will ensure the same backend
is always hit via stick-tables and cookies
- ensure runners use haproxy to reach git
we now package act_runner now, lets use the rpm
- change installation method to rpm instead of curl + untar
- add capability to versionlock act_runner
- fix paths to act_runner
- remove manually installed act_runner
Reviewed-on: #432
- update root password in common.eyaml
- add missing param to the accounts::root manifest
- remove if block as undef sshkeys has same effect
Reviewed-on: #429
- add module to manage externaldns bind for k8s
- add infra::dns::externaldns role
- add 198.18.19.20 as anycast for k8s external-dns service
Reviewed-on: #428
- add SMTP submission listener on port 587 with TLS requirement
- configure HAProxy frontend/backend for submission with send-proxy-v2 support
- add send-proxy-v2 support to all listeners
- add dynamic HAProxy node discovery for proxy trusted networks
- use service hostname instead of node FQDN for autoconfig/autodiscover
- remove redundant IMAP/IMAPS/SMTP alt-names from TLS certificates
- update VRRP CNAME configuration to use mail.main.unkin.net
Reviewed-on: #425
unkinben
benvin