5 Commits

Author SHA1 Message Date
Ben Vincent fc7716a0a2 Install openbao-plugin-secrets-rancher on vault nodes
ci/woodpecker/pr/puppet-lint Pipeline was successful
ci/woodpecker/pr/bolt-validate Pipeline was successful
ci/woodpecker/pr/yamllint Pipeline was successful
ci/woodpecker/pr/erb-validate Pipeline was successful
ci/woodpecker/pr/ruby-validate Pipeline was successful
ci/woodpecker/pr/epp-validate Pipeline was successful
ci/woodpecker/pr/ruby-check Pipeline was successful
ci/woodpecker/pr/puppet-validate Pipeline was successful
Lays down the rancher secrets-engine plugin binary in the OpenBao plugin
directory (/opt/openbao-plugins/vault-plugin-secrets-rancher) so terraform-vault
can register and mount it. Package comes from the rpm-internal repo.

- Add openbao-plugin-secrets-rancher to profiles::packages::include on the vault
  storage role.
2026-07-17 23:40:08 +10:00
unkinben d71e221049 dns: deploy dns-updater daemon in place of the shell nsupdate script [HOLD: needs dns-updater RPM published] (#482)
ci/woodpecker/pr/ruby-check Pipeline was canceled
ci/woodpecker/pr/puppet-validate Pipeline was canceled
ci/woodpecker/pr/ruby-validate Pipeline was successful
ci/woodpecker/pr/puppet-lint Pipeline was successful
ci/woodpecker/pr/yamllint Pipeline was successful
ci/woodpecker/pr/erb-validate Pipeline was successful
ci/woodpecker/pr/bolt-validate Pipeline was successful
ci/woodpecker/pr/epp-validate Pipeline was successful
## Why
Replace the `profiles::dns::updater` shell mechanism (`dns-update.sh` + `dns-update.path`/`.service` + the in-run `exec`) with the packaged **dns-updater** daemon. The daemon watches the records file (inotify) and network interfaces and pushes TSIG-signed RFC2136 updates to BIND natively — with structured per-zone RCODEs and a status API/facter fact, so failures like the recent `invalid owner name: empty label` / NOTZONE surface directly instead of as opaque nsupdate stderr.

## Changes
- Install the `dns-updater` package; manage `/etc/dns-updater/env`.
- Run the packaged `dns-updater.service`, restarting **only** on env/key change — records-file edits are picked up by the daemon`s own inotify watch, so no service churn on record changes.
- Keep the `concat` records file and the TSIG key file unchanged (same paths/format).
- Ensure the old `/usr/local/bin/dns-update` + `dns-update.path`/`.service` units are absent.
- Drop the now-dead `dns-update.{sh,service,path}.epp` templates.

## Sequencing — HOLD
Do not merge until the `dns-updater` RPM is published to artifactapi `rpm-internal` (needs terraform-git#33 to create the repo, then the daemon code pushed + tagged so Woodpecker builds the RPM). Merging before the package exists makes `package { dns-updater }` fail on every host.

Supersedes the interim shell fix in #481 (which stays valid until this rolls out). Keeps the file as the desired-state interface (puppet owns desired records; the daemon reconciles + reports).

Reviewed-on: #482
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-17 23:35:00 +10:00
unkinben 0de3ac2a0b dns: fix dns-update fqdn() double-appending zone to FQDN records (#481)
## Why
6 hosts (ausyd1nxvm2069-2073, 2098) ended up with a reverse PTR in bind-authoritative but **no forward A record**, and the `unkin.net` service records (git/grafana/auth/fafflix, all published by the halb host 2069) never landed at all.

VictoriaLogs (`dns-update-apply` on 2069 & 2070) shows the cause:
```
dns-update: nsupdate to 198.18.200.9 failed
invalid owner name: empty label
syntax error
```
`profiles::dns::record` publishes some records whose name is already fully-qualified (trailing dot) — e.g. `au-syd1-pve.main.unkin.net.`, `cobbler.main.unkin.net.`, `dashboard.ceph.unkin.net.`, and the halb CNAMEs. The `dns-update` script `fqdn()` unconditionally appended the zone, producing `…net..main.unkin.net.` — the `..` is an empty label, which nsupdate rejects, failing the entire per-zone `send`. The reverse-PTR send is sorted first and its name is always relative, so it still applied — hence "PTR but no A".

## Change
`fqdn()` now handles three cases:
- `@`/empty → zone apex (unchanged)
- name ending in `.` → already FQDN, used verbatim (**the fix**)
- otherwise → relative, append `.zone.` (unchanged)

Verified against all record shapes (relative host, apex, FQDN CNAME, reverse label) — no more `..`.

## After merge
Once puppet re-runs on the affected hosts their `main.unkin.net`/`unkin.net` updates succeed, filling in the missing A records and the `unkin.net` service zone. Pairs with argocd-apps#260 (adds the `ceph.unkin.net` zone so `dashboard.ceph.unkin.net` does not then hit NOTZONE).

Reviewed-on: #481
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-17 22:50:28 +10:00
unkinben 0702676da6 Install openbao-plugin-secrets-gpg on vault nodes (#480)
Deploy the GPG/OpenPGP secrets engine to the OpenBao (vault-role) cluster by installing its plugin RPM into `/opt/openbao-plugins`, mirroring the existing `openbao-plugin-secrets-litellm` deployment (#479).

- Add `openbao-plugin-secrets-gpg` to `profiles::packages::include` in the vault role hiera. The RPM ships from artifactapi `rpm-internal` (built on the [vault-plugin-secrets-gpg](https://git.unkin.net/unkin/vault-plugin-secrets-gpg) v0.1.0 tag) and lands the plugin binary in the node's configured `plugin_directory`.

Registering + enabling the secrets backend (`plugin register` / `secrets enable`) is a follow-up terraform-vault change, matching how litellm is wired.

Reviewed-on: #480
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-16 22:48:08 +10:00
unkinben 88fcb97ad1 dns: nsupdate host records to the authoritative server (#475)
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.

## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.

## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.

## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.

## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).

## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).

Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-12 22:23:18 +10:00
6 changed files with 52 additions and 96 deletions
+2
View File
@@ -146,6 +146,8 @@ lookup_options:
strategy: deep
profiles::etcd::node::initial_cluster_token:
convert_to: Sensitive
profiles::dns::updater::key_secret:
convert_to: Sensitive
sysctl::base::values:
merge:
strategy: deep
+2
View File
@@ -29,3 +29,5 @@ profiles::nginx::simpleproxy::proxy_path: '/'
profiles::packages::include:
openbao-plugins: {}
openbao-plugin-secrets-litellm: {}
openbao-plugin-secrets-gpg: {}
openbao-plugin-secrets-rancher: {}
+48 -21
View File
@@ -23,6 +23,12 @@ class profiles::dns::updater (
Stdlib::AbsolutePath $state_dir = '/var/lib/dns-updater',
Stdlib::AbsolutePath $config_dir = '/etc/dns-updater',
Stdlib::AbsolutePath $master_basedir = lookup('profiles::dns::master::basedir'),
# dns-updater daemon (replaces the dns-update shell script).
String $package_ensure = 'installed',
Stdlib::AbsolutePath $api_socket = '/run/dns-updater/api.sock',
String $resync = '10m',
Enum['debug', 'info', 'warn', 'error'] $log_level = 'info',
Boolean $watch_interfaces = true,
) {
$state_file = "${state_dir}/applied"
@@ -91,36 +97,57 @@ class profiles::dns::updater (
})),
}
file { '/usr/local/bin/dns-update':
# dns-updater daemon: watches the records file (inotify) and network
# interfaces, pushes TSIG-signed RFC2136 updates to $server natively.
package { 'dns-updater':
ensure => $package_ensure,
}
$env_content = @("ENV")
# Managed by puppet (profiles::dns::updater).
DNS_UPDATER_SERVER=${server}
DNS_UPDATER_KEY_FILE=${key_file}
DNS_UPDATER_RECORDS_FILE=${records_file}
DNS_UPDATER_STATE_FILE=${state_file}
DNS_UPDATER_API=${api_socket}
DNS_UPDATER_RESYNC=${resync}
DNS_UPDATER_WATCH_INTERFACES=${watch_interfaces}
DNS_UPDATER_LOG_LEVEL=${log_level}
| ENV
file { "${config_dir}/env":
ensure => file,
owner => 'root',
group => 'root',
mode => '0755',
content => epp('profiles/dns/dns-update.sh.epp', {
'server' => $server,
'key_file' => $key_file,
'records_file' => $records_file,
'state_file' => $state_file,
}),
mode => '0644',
content => $env_content,
require => [File[$config_dir], Package['dns-updater']],
}
# Restart only on config/key change; records-file changes are picked up by
# the daemon's own inotify watch, so no service churn on record edits.
service { 'dns-updater':
ensure => running,
enable => true,
subscribe => [File["${config_dir}/env"], File[$key_file]],
require => [Package['dns-updater'], Concat[$records_file], File[$key_file]],
}
# Retire the previous shell-based mechanism.
file { '/usr/local/bin/dns-update':
ensure => absent,
}
systemd::unit_file { 'dns-update.service':
content => epp('profiles/dns/dns-update.service.epp', { 'script' => '/usr/local/bin/dns-update' }),
ensure => absent,
active => false,
enable => false,
}
# The .path unit watches the records file and triggers the service.
systemd::unit_file { 'dns-update.path':
content => epp('profiles/dns/dns-update.path.epp', { 'records_file' => $records_file }),
active => true,
enable => true,
}
# Also apply within the puppet run whenever the records change.
exec { 'dns-update-apply':
command => '/usr/local/bin/dns-update',
refreshonly => true,
subscribe => Concat[$records_file],
require => [File['/usr/local/bin/dns-update'], File[$key_file]],
ensure => absent,
active => false,
enable => false,
}
}
}
@@ -1,10 +0,0 @@
<%- | String $records_file | -%>
[Unit]
Description=Watch the DNS records file and apply changes
[Path]
PathModified=<%= $records_file %>
Unit=dns-update.service
[Install]
WantedBy=multi-user.target
@@ -1,9 +0,0 @@
<%- | String $script | -%>
[Unit]
Description=Apply host DNS records via nsupdate
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=<%= $script %>
@@ -1,56 +0,0 @@
<%- | String $server, String $key_file, String $records_file, String $state_file | -%>
#!/bin/bash
# Managed by puppet (profiles::dns::updater). Applies this host's records to the
# authoritative DNS server via TSIG nsupdate. Only the delta since the last
# successful run is sent; removed records are deleted.
set -euo pipefail
SERVER="<%= $server %>"
KEYFILE="<%= $key_file %>"
RECORDS="<%= $records_file %>"
STATE="<%= $state_file %>"
[ -f "$RECORDS" ] || exit 0
touch "$STATE"
# Format per line: zone|name|type|ttl|value (name is relative to zone, or @).
desired="$(grep -vE '^[[:space:]]*(#|$)' "$RECORDS" | sort -u || true)"
applied="$(grep -vE '^[[:space:]]*(#|$)' "$STATE" 2>/dev/null | sort -u || true)"
[ "$desired" = "$applied" ] && exit 0
fqdn() { # name zone
if [ -z "$1" ] || [ "$1" = "@" ]; then printf '%s.' "$2"; else printf '%s.%s.' "$1" "$2"; fi
}
msg="$(mktemp)"
trap 'rm -f "$msg"' EXIT
printf 'server %s\n' "$SERVER" >> "$msg"
# Process per zone so each UPDATE message targets a single zone.
zones="$(printf '%s\n%s\n' "$desired" "$applied" | cut -d'|' -f1 | sort -u | grep -v '^$' || true)"
for zone in $zones; do
printf 'zone %s.\n' "$zone" >> "$msg"
# Additions/updates: replace the RRset for every desired record in this zone.
printf '%s\n' "$desired" | awk -F'|' -v z="$zone" 'NF>=5 && $1==z' | \
while IFS='|' read -r z name type ttl value; do
f="$(fqdn "$name" "$z")"
printf 'update delete %s %s\n' "$f" "$type" >> "$msg"
printf 'update add %s %s %s %s\n' "$f" "$ttl" "$type" "$value" >> "$msg"
done
# Deletions: records present last run but gone now.
comm -23 <(printf '%s\n' "$applied") <(printf '%s\n' "$desired") | \
awk -F'|' -v z="$zone" 'NF>=5 && $1==z' | \
while IFS='|' read -r z name type ttl value; do
f="$(fqdn "$name" "$z")"
printf 'update delete %s %s %s\n' "$f" "$type" "$value" >> "$msg"
done
printf 'send\n' >> "$msg"
done
if nsupdate -k "$KEYFILE" "$msg"; then
printf '%s\n' "$desired" > "$STATE"
else
echo "dns-update: nsupdate to ${SERVER} failed" >&2
exit 1
fi