## Why
`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.
## How hosts are identified (osd-only vs mon/mgr/mds)
The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):
- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.
The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.
## Changes
- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.
## Live-cluster safety review (please read before merge)
This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.
- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
- osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
- mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
- prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.
Rendered samples:
- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.
Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Ceph OSDs manage their own I/O ordering, so the kernel scheduler on the backing disks just adds overhead. The original intent was to set those disks to the `noop` scheduler. The whole OSD fleet (k8s + incus nodes) runs AlmaLinux 9 on blk-mq kernels (5.14), where the equivalent of `noop` is `none`.
## Changes
- Add `profiles::ceph::osd_scheduler`, rendering a udev rule from the `ceph_osd_devices` fact (PR #504) that pins `queue/scheduler` to `none` on each OSD disk.
- Reload udev and trigger the matched block devices so the setting applies immediately; the udev rule keeps it set across reboots and device re-add.
- No-op when the fact is absent/empty, so VMs and non-OSD hosts are untouched.
- Include the class from `profiles::ceph::osd` so it lands only on OSD hosts.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #505
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Migrate the VM puppet agents off the legacy VM puppetmasters onto the new puppet-on-kubernetes servers, one wave at a time and reversible without re-enrolment.
Changing `server`/`ca_server` alone is insufficient: each agent's ssldir holds a client certificate signed by the **old** CA, which the new k8s CA neither trusts nor recognises. Migrated nodes switch to a **fresh ssldir**, so the agent generates a new key + CSR on its next run (autosigned immediately by the k8s CA, which runs `autosign = true`), pulls its catalog from the new compilers, and leaves the old CA-signed creds intact on disk for rollback.
The reconfiguration catalog is compiled by the **old** server (the node still points there when it applies the change). `profiles::puppet::client` rewrites `puppet.conf` with the new endpoints + ssldir and notifies `Service['puppet']`, so the very next agent run lands on k8s.
**Single-boolean design (reworked):** cutover is now driven by one hiera key. `profiles::puppet::migrate` ships wired into `profiles::base` with `enabled => false`, so every node evaluates it but nothing changes until the boolean is flipped. `profiles::puppet::client` includes migrate and folds the migrate endpoints into the effective values it renders. Flip `profiles::puppet::migrate::enabled: true` at any hiera layer (node/role/common) to migrate a target -- no other keys required.
## Changes
- `profiles::base` (`site/profiles/manifests/base.pp`): `include profiles::puppet::migrate` alongside the other puppet profiles (ships disabled).
- `profiles::puppet::migrate` (`site/profiles/manifests/puppet/migrate.pp`): unchanged params (`enabled=false`, `new_server`, `new_ca_server`, `new_ssldir`); when enabled and the node is not a puppetmaster it owns the fresh ssldir directory. Class-header runbook rewritten for the single-boolean flow. No longer opt-in via `hiera_include`.
- `profiles::puppet::client` (`site/profiles/manifests/puppet/client.pp`): `include profiles::puppet::migrate` and compute `$effective_server` / `$effective_ca_server` / `$effective_report_server` / `$effective_ssldir`, feeding those to the template. Keeps the existing optional `$ssldir` / `$report_server` params as manual overrides.
- `puppet.conf.erb`: render from the `@effective_*` vars.
## Precedence
- **ssldir / report_server** (`Optional`, default `undef`): explicit `profiles::puppet::client::ssldir` / `::report_server` in hiera wins; else the migrate-derived value when `enabled`; else the legacy default (ssldir omitted -> puppet default; report_server -> effective server).
- **server / ca_server** (`String`, no undef sentinel): the migrate endpoint wins while `enabled => true`; otherwise the client param (legacy default unless set explicitly). To pin a bespoke server, leave `enabled => false` for that target and set `profiles::puppet::client::server` directly.
- No include cycle: migrate.pp never references client.pp; its guard uses `$facts['enc_role']`.
## Runbook
- **Canary (one node)** -> `hieradata/nodes/<certname>.yaml`: `profiles::puppet::migrate::enabled: true`
- **Wave (a role)** -> `hieradata/roles/<t1>/<t2>[/<t3>].yaml`: `profiles::puppet::migrate::enabled: true`
- **Estate flip** -> `hieradata/common.yaml`: `profiles::puppet::migrate::enabled: true`
- **Rollback** -> remove the key (or set `false`) at that layer; next run rewrites puppet.conf back to the legacy server + legacy ssldir (original CA-signed cert intact).
- The k8s endpoints (`new_server` / `new_ca_server` / `new_ssldir`) are `profiles::puppet::migrate` param defaults; override them in hiera if the k8s names change.
## Verification
- Render comparison: unmigrated node renders a **byte-identical** `puppet.conf` to `develop` (diff empty). Migrated render (only `enabled: true`) emits `server`/`ca_server`/`report_server` = k8s endpoints and `ssldir = /etc/puppetlabs/puppet/ssl-k8s`.
- `puppet-lint` (repo args `--no-80chars-check --no-documentation-check --no-puppet_url_without_modules-check --fail-on-warnings`): clean on all 3 manifests.
- `puppet parser validate`: OK. ERB syntax: OK. Repo pre-commit hooks (erb-validate, puppet-lint, puppet-validate): all Passed.
Reviewed-on: #493
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Replace the `profiles::dns::updater` shell mechanism (`dns-update.sh` + `dns-update.path`/`.service` + the in-run `exec`) with the packaged **dns-updater** daemon. The daemon watches the records file (inotify) and network interfaces and pushes TSIG-signed RFC2136 updates to BIND natively — with structured per-zone RCODEs and a status API/facter fact, so failures like the recent `invalid owner name: empty label` / NOTZONE surface directly instead of as opaque nsupdate stderr.
## Changes
- Install the `dns-updater` package; manage `/etc/dns-updater/env`.
- Run the packaged `dns-updater.service`, restarting **only** on env/key change — records-file edits are picked up by the daemon`s own inotify watch, so no service churn on record changes.
- Keep the `concat` records file and the TSIG key file unchanged (same paths/format).
- Ensure the old `/usr/local/bin/dns-update` + `dns-update.path`/`.service` units are absent.
- Drop the now-dead `dns-update.{sh,service,path}.epp` templates.
## Sequencing — HOLD
Do not merge until the `dns-updater` RPM is published to artifactapi `rpm-internal` (needs terraform-git#33 to create the repo, then the daemon code pushed + tagged so Woodpecker builds the RPM). Merging before the package exists makes `package { dns-updater }` fail on every host.
Supersedes the interim shell fix in #481 (which stays valid until this rolls out). Keeps the file as the desired-state interface (puppet owns desired records; the daemon reconciles + reports).
Reviewed-on: #482
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
6 hosts (ausyd1nxvm2069-2073, 2098) ended up with a reverse PTR in bind-authoritative but **no forward A record**, and the `unkin.net` service records (git/grafana/auth/fafflix, all published by the halb host 2069) never landed at all.
VictoriaLogs (`dns-update-apply` on 2069 & 2070) shows the cause:
```
dns-update: nsupdate to 198.18.200.9 failed
invalid owner name: empty label
syntax error
```
`profiles::dns::record` publishes some records whose name is already fully-qualified (trailing dot) — e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`, `dashboard.ceph.unkin.net.`, and the halb CNAMEs. The `dns-update` script `fqdn()` unconditionally appended the zone, producing `…net..main.unkin.net.` — the `..` is an empty label, which nsupdate rejects, failing the entire per-zone `send`. The reverse-PTR send is sorted first and its name is always relative, so it still applied — hence "PTR but no A".
## Change
`fqdn()` now handles three cases:
- `@`/empty → zone apex (unchanged)
- name ending in `.` → already FQDN, used verbatim (**the fix**)
- otherwise → relative, append `.zone.` (unchanged)
Verified against all record shapes (relative host, apex, FQDN CNAME, reverse label) — no more `..`.
## After merge
Once puppet re-runs on the affected hosts their `main.unkin.net`/`unkin.net` updates succeed, filling in the missing A records and the `unkin.net` service zone. Pairs with argocd-apps#260 (adds the `ceph.unkin.net` zone so `dashboard.ceph.unkin.net` does not then hit NOTZONE).
Reviewed-on: #481
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
saving artifacts are breaking in some actions as the runner will switch
between different git hosts. using haproxy will ensure the same backend
is always hit via stick-tables and cookies
- ensure runners use haproxy to reach git
we now package act_runner now, lets use the rpm
- change installation method to rpm instead of curl + untar
- add capability to versionlock act_runner
- fix paths to act_runner
- remove manually installed act_runner
Reviewed-on: #432
- refactor profiles::postfix::gateway as parameterized class
- move base postfix parameters, transports, and virtuals to hiera for flexibility
- convert SMTP restrictions to arrays for better readability using join()
- add postscreen enable/disable boolean with conditional master.cf configuration
- add per-domain TLS policy maps (smtp_tls_policy_maps)
- convert alias_maps to array parameter for flexibility
- convert all postfix map files to ERB templates with parameter hashes
- add map parameters: sender_canonical_maps, sender_access_maps, relay_recipients_maps,
relay_domains_maps, recipient_canonical_maps, recipient_access_maps, postscreen_access_maps, helo_access_maps
- move default map data to hiera while keeping parameters as empty hashes by default
This approach balances flexibility with data-driven configuration, allowing
easy customization through parameters while keeping transport/virtual maps
and default map data in hiera for role-specific overrides.
Reviewed-on: #416
- ensure the autopromoter removes hardlinks/replicas for repos older
than the current promoted monthly
- this is to reduce MDS load for ceph, as hardlinks require memory
Reviewed-on: #393
- start managing ceph configuration file
- manage ceph-radosgw
- merge the ceph::conf and ceph::node profiles
- ensure the ceph repos exist
- mange nginx frontend and consul service
Reviewed-on: #380
- manage the unkin.net domain
- ensure forwarding for unkin.net
- split domain from cname list and set zone correctly
- add fafflix to cnames list for haproxy2
Reviewed-on: #347
- add group members to jupyterhub_user
- add svc_jupyterhub user for ldap binding
- paramatarise all ldap fields required
- manage the notebook data directory
- add nodejs module to use npm package provider
- add jupyterhub role
- add class to configure the jupyterhub instance
- add ldap groups
- add nginx simpleproxy
- ensure docker is configured
- create runner user/group
- deploy config.yaml from hiera hash
- install runner from url
- register the runner with the gitea instance
- manage the act_runner service
- add ceph class to manage ceph client configuration/packages
- add cephfs define for mounting volumes
- add ceph keyring define to manage secrets used to mount cephfs
- ensure a database, user and credential is created for each grafana node
- ensure all databases for a region are included in a mariadb cluster
- refine params with stdlib types
- manage python script/venv to sign ssh host certificates
- add approle_id to puppetmaster eyaml files
- add class to sign ssh-rsa host keys
- add facts to check if the current principals match the desired principals
unkinben