5 Commits

Author SHA1 Message Date
unkinben 1e0ad2743f Add rpm-vendor yum repository
ci/woodpecker/pr/puppet-lint Pipeline was successful
ci/woodpecker/pr/ruby-validate Pipeline was successful
ci/woodpecker/pr/yamllint Pipeline was successful
ci/woodpecker/pr/bolt-validate Pipeline was successful
ci/woodpecker/pr/erb-validate Pipeline was successful
ci/woodpecker/pr/epp-validate Pipeline was successful
ci/woodpecker/pr/puppet-validate Pipeline was successful
ci/woodpecker/pr/ruby-check Pipeline was successful
rpmbuilder's vendored third-party RPMs are moving from Gitea's RPM registry
to the artifactapi rpm-vendor local repos. Managed AlmaLinux hosts need the
matching yum repo to install them; this mirrors the existing rpm-internal
repo and selects the per-EL-release repo via os.release.major.

- Add rpm-vendor to profiles::yum::global::repos in
  hieradata/os/AlmaLinux/all_releases.yaml, pointing at
  .../api/v1/local/rpm-vendor-el<major>/ (el8/el9), gpgcheck disabled and
  mirrorlist absent, matching the rpm-internal repo definition.
2026-07-18 00:09:08 +10:00
unkinben d71e221049 dns: deploy dns-updater daemon in place of the shell nsupdate script [HOLD: needs dns-updater RPM published] (#482)
ci/woodpecker/pr/ruby-check Pipeline was canceled
ci/woodpecker/pr/puppet-validate Pipeline was canceled
ci/woodpecker/pr/ruby-validate Pipeline was successful
ci/woodpecker/pr/puppet-lint Pipeline was successful
ci/woodpecker/pr/yamllint Pipeline was successful
ci/woodpecker/pr/erb-validate Pipeline was successful
ci/woodpecker/pr/bolt-validate Pipeline was successful
ci/woodpecker/pr/epp-validate Pipeline was successful
## Why
Replace the `profiles::dns::updater` shell mechanism (`dns-update.sh` + `dns-update.path`/`.service` + the in-run `exec`) with the packaged **dns-updater** daemon. The daemon watches the records file (inotify) and network interfaces and pushes TSIG-signed RFC2136 updates to BIND natively — with structured per-zone RCODEs and a status API/facter fact, so failures like the recent `invalid owner name: empty label` / NOTZONE surface directly instead of as opaque nsupdate stderr.

## Changes
- Install the `dns-updater` package; manage `/etc/dns-updater/env`.
- Run the packaged `dns-updater.service`, restarting **only** on env/key change — records-file edits are picked up by the daemon`s own inotify watch, so no service churn on record changes.
- Keep the `concat` records file and the TSIG key file unchanged (same paths/format).
- Ensure the old `/usr/local/bin/dns-update` + `dns-update.path`/`.service` units are absent.
- Drop the now-dead `dns-update.{sh,service,path}.epp` templates.

## Sequencing — HOLD
Do not merge until the `dns-updater` RPM is published to artifactapi `rpm-internal` (needs terraform-git#33 to create the repo, then the daemon code pushed + tagged so Woodpecker builds the RPM). Merging before the package exists makes `package { dns-updater }` fail on every host.

Supersedes the interim shell fix in #481 (which stays valid until this rolls out). Keeps the file as the desired-state interface (puppet owns desired records; the daemon reconciles + reports).

Reviewed-on: #482
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-17 23:35:00 +10:00
unkinben 0de3ac2a0b dns: fix dns-update fqdn() double-appending zone to FQDN records (#481)
## Why
6 hosts (ausyd1nxvm2069-2073, 2098) ended up with a reverse PTR in bind-authoritative but **no forward A record**, and the `unkin.net` service records (git/grafana/auth/fafflix, all published by the halb host 2069) never landed at all.

VictoriaLogs (`dns-update-apply` on 2069 & 2070) shows the cause:
```
dns-update: nsupdate to 198.18.200.9 failed
invalid owner name: empty label
syntax error
```
`profiles::dns::record` publishes some records whose name is already fully-qualified (trailing dot) — e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`, `dashboard.ceph.unkin.net.`, and the halb CNAMEs. The `dns-update` script `fqdn()` unconditionally appended the zone, producing `…net..main.unkin.net.` — the `..` is an empty label, which nsupdate rejects, failing the entire per-zone `send`. The reverse-PTR send is sorted first and its name is always relative, so it still applied — hence "PTR but no A".

## Change
`fqdn()` now handles three cases:
- `@`/empty → zone apex (unchanged)
- name ending in `.` → already FQDN, used verbatim (**the fix**)
- otherwise → relative, append `.zone.` (unchanged)

Verified against all record shapes (relative host, apex, FQDN CNAME, reverse label) — no more `..`.

## After merge
Once puppet re-runs on the affected hosts their `main.unkin.net`/`unkin.net` updates succeed, filling in the missing A records and the `unkin.net` service zone. Pairs with argocd-apps#260 (adds the `ceph.unkin.net` zone so `dashboard.ceph.unkin.net` does not then hit NOTZONE).

Reviewed-on: #481
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-17 22:50:28 +10:00
unkinben 0702676da6 Install openbao-plugin-secrets-gpg on vault nodes (#480)
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).

- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.

Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.

Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-16 22:48:08 +10:00
unkinben 88fcb97ad1 dns: nsupdate host records to the authoritative server (#475)
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.

## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.

## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.

## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.

## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).

## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).

Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 22:23:18 +10:00
7 changed files with 58 additions and 96 deletions
+2
View File
@@ -146,6 +146,8 @@ lookup_options:
strategy: deep strategy: deep
profiles::etcd::node::initial_cluster_token: profiles::etcd::node::initial_cluster_token:
convert_to: Sensitive convert_to: Sensitive
profiles::dns::updater::key_secret:
convert_to: Sensitive
sysctl::base::values: sysctl::base::values:
merge: merge:
strategy: deep strategy: deep
+7
View File
@@ -85,6 +85,13 @@ profiles::yum::global::repos:
baseurl: https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/rpm-internal/ baseurl: https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/rpm-internal/
gpgcheck: false gpgcheck: false
mirrorlist: absent mirrorlist: absent
rpm-vendor:
name: rpm-vendor
descr: rpm-vendor repository
target: /etc/yum.repos.d/rpm-vendor.repo
baseurl: https://artifactapi.k8s.syd1.au.unkin.net/api/v1/local/rpm-vendor-el%{facts.os.release.major}/
gpgcheck: false
mirrorlist: absent
# Additional repositories - default to absent, roles can override with ensure: present # Additional repositories - default to absent, roles can override with ensure: present
# FRRouting repositories # FRRouting repositories
+1
View File
@@ -29,3 +29,4 @@ profiles::nginx::simpleproxy::proxy_path: '/'
profiles::packages::include: profiles::packages::include:
openbao-plugins: {} openbao-plugins: {}
openbao-plugin-secrets-litellm: {} openbao-plugin-secrets-litellm: {}
openbao-plugin-secrets-gpg: {}
+48 -21
View File
@@ -23,6 +23,12 @@ class profiles::dns::updater (
Stdlib::AbsolutePath $state_dir = '/var/lib/dns-updater', Stdlib::AbsolutePath $state_dir = '/var/lib/dns-updater',
Stdlib::AbsolutePath $config_dir = '/etc/dns-updater', Stdlib::AbsolutePath $config_dir = '/etc/dns-updater',
Stdlib::AbsolutePath $master_basedir = lookup('profiles::dns::master::basedir'), Stdlib::AbsolutePath $master_basedir = lookup('profiles::dns::master::basedir'),
# dns-updater daemon (replaces the dns-update shell script).
String $package_ensure = 'installed',
Stdlib::AbsolutePath $api_socket = '/run/dns-updater/api.sock',
String $resync = '10m',
Enum['debug', 'info', 'warn', 'error'] $log_level = 'info',
Boolean $watch_interfaces = true,
) { ) {
$state_file = "${state_dir}/applied" $state_file = "${state_dir}/applied"
@@ -91,36 +97,57 @@ class profiles::dns::updater (
})), })),
} }
file { '/usr/local/bin/dns-update': # dns-updater daemon: watches the records file (inotify) and network
# interfaces, pushes TSIG-signed RFC2136 updates to $server natively.
package { 'dns-updater':
ensure => $package_ensure,
}
$env_content = @("ENV")
# Managed by puppet (profiles::dns::updater).
DNS_UPDATER_SERVER=${server}
DNS_UPDATER_KEY_FILE=${key_file}
DNS_UPDATER_RECORDS_FILE=${records_file}
DNS_UPDATER_STATE_FILE=${state_file}
DNS_UPDATER_API=${api_socket}
DNS_UPDATER_RESYNC=${resync}
DNS_UPDATER_WATCH_INTERFACES=${watch_interfaces}
DNS_UPDATER_LOG_LEVEL=${log_level}
| ENV
file { "${config_dir}/env":
ensure => file, ensure => file,
owner => 'root', owner => 'root',
group => 'root', group => 'root',
mode => '0755', mode => '0644',
content => epp('profiles/dns/dns-update.sh.epp', { content => $env_content,
'server' => $server, require => [File[$config_dir], Package['dns-updater']],
'key_file' => $key_file, }
'records_file' => $records_file,
'state_file' => $state_file, # Restart only on config/key change; records-file changes are picked up by
}), # the daemon's own inotify watch, so no service churn on record edits.
service { 'dns-updater':
ensure => running,
enable => true,
subscribe => [File["${config_dir}/env"], File[$key_file]],
require => [Package['dns-updater'], Concat[$records_file], File[$key_file]],
}
# Retire the previous shell-based mechanism.
file { '/usr/local/bin/dns-update':
ensure => absent,
} }
systemd::unit_file { 'dns-update.service': systemd::unit_file { 'dns-update.service':
content => epp('profiles/dns/dns-update.service.epp', { 'script' => '/usr/local/bin/dns-update' }), ensure => absent,
active => false,
enable => false,
} }
# The .path unit watches the records file and triggers the service.
systemd::unit_file { 'dns-update.path': systemd::unit_file { 'dns-update.path':
content => epp('profiles/dns/dns-update.path.epp', { 'records_file' => $records_file }), ensure => absent,
active => true, active => false,
enable => true, enable => false,
}
# Also apply within the puppet run whenever the records change.
exec { 'dns-update-apply':
command => '/usr/local/bin/dns-update',
refreshonly => true,
subscribe => Concat[$records_file],
require => [File['/usr/local/bin/dns-update'], File[$key_file]],
} }
} }
} }
@@ -1,10 +0,0 @@
<%- | String $records_file | -%>
[Unit]
Description=Watch the DNS records file and apply changes
[Path]
PathModified=<%= $records_file %>
Unit=dns-update.service
[Install]
WantedBy=multi-user.target
@@ -1,9 +0,0 @@
<%- | String $script | -%>
[Unit]
Description=Apply host DNS records via nsupdate
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=<%= $script %>
@@ -1,56 +0,0 @@
<%- | String $server, String $key_file, String $records_file, String $state_file | -%>
#!/bin/bash
# Managed by puppet (profiles::dns::updater). Applies this host's records to the
# authoritative DNS server via TSIG nsupdate. Only the delta since the last
# successful run is sent; removed records are deleted.
set -euo pipefail
SERVER="<%= $server %>"
KEYFILE="<%= $key_file %>"
RECORDS="<%= $records_file %>"
STATE="<%= $state_file %>"
[ -f "$RECORDS" ] || exit 0
touch "$STATE"
# Format per line: zone|name|type|ttl|value (name is relative to zone, or @).
desired="$(grep -vE '^[[:space:]]*(#|$)' "$RECORDS" | sort -u || true)"
applied="$(grep -vE '^[[:space:]]*(#|$)' "$STATE" 2>/dev/null | sort -u || true)"
[ "$desired" = "$applied" ] && exit 0
fqdn() { # name zone
if [ -z "$1" ] || [ "$1" = "@" ]; then printf '%s.' "$2"; else printf '%s.%s.' "$1" "$2"; fi
}
msg="$(mktemp)"
trap 'rm -f "$msg"' EXIT
printf 'server %s\n' "$SERVER" >> "$msg"
# Process per zone so each UPDATE message targets a single zone.
zones="$(printf '%s\n%s\n' "$desired" "$applied" | cut -d'|' -f1 | sort -u | grep -v '^$' || true)"
for zone in $zones; do
printf 'zone %s.\n' "$zone" >> "$msg"
# Additions/updates: replace the RRset for every desired record in this zone.
printf '%s\n' "$desired" | awk -F'|' -v z="$zone" 'NF>=5 && $1==z' | \
while IFS='|' read -r z name type ttl value; do
f="$(fqdn "$name" "$z")"
printf 'update delete %s %s\n' "$f" "$type" >> "$msg"
printf 'update add %s %s %s %s\n' "$f" "$ttl" "$type" "$value" >> "$msg"
done
# Deletions: records present last run but gone now.
comm -23 <(printf '%s\n' "$applied") <(printf '%s\n' "$desired") | \
awk -F'|' -v z="$zone" 'NF>=5 && $1==z' | \
while IFS='|' read -r z name type ttl value; do
f="$(fqdn "$name" "$z")"
printf 'update delete %s %s %s\n' "$f" "$type" "$value" >> "$msg"
done
printf 'send\n' >> "$msg"
done
if nsupdate -k "$KEYFILE" "$msg"; then
printf '%s\n' "$desired" > "$STATE"
else
echo "dns-update: nsupdate to ${SERVER} failed" >&2
exit 1
fi