Estate journald ingestion has been down since 00:33Z. `systemd-journal-upload` appends `:19532/upload` to the configured URL whenever that URL carries no explicit port, so the portless k8s endpoint became `/insert/journald:19532/upload`, which vlinsert rejects as an unsupported path. The previous consul URL only worked because `:9428` was explicit.
- pin `victorialogs::client::journald::inserturl` to port 443
Requests then land on `/insert/journald/upload`, which vlinsert accepts.
Reviewed-on: #533
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The estate ships journald to the VM VictoriaLogs cluster, which is being left to age out rather than grow. New log capacity lands in k8s, so clients need to point there while the VM cluster keeps serving historical queries until its retention lapses.
- repoint victorialogs::client::journald::inserturl at https://logs-ingest.k8s.syd1.au.unkin.net/insert/journald
VMs already trust the issuing CA via the system bundle, so journal-upload needs no TLS change.
Requires the k8s VLCluster deployed and serving /insert/journald first.
Reviewed-on: #531
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Catalog compilation moved to the k8s puppetserver compilers, which sign host certificates against the terraform-managed `sshca` mount. Clients only trust the legacy `ssh-host-signer` CA, so every re-signed node (ausyd1nxvm2120 already) presents a certificate nothing accepts, and knownhosts emits no plain host-key fallback.
- Add a second `@cert-authority *` entry for the `sshca` public key to `profiles::ssh::knownhosts::lines`.
- Keep the legacy entry untouched so legacy-signed hosts still verify.
Reviewed-on: #530
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
`/etc/ceph/ceph.conf` on the prodnxsr ceph cluster (fsid `de96a98f`) is currently unmanaged and hand-maintained via the runbook (`profiles::ceph::client::manage_ceph_conf` is `false` on the k8s and incus roles). This brings it under Puppet, drives it from a single source of truth in hiera, and normalizes `public_network` to every ceph host's `/32`.
## How hosts are identified (osd-only vs mon/mgr/mds)
The authoritative, declarative signal is the role's `hiera_include` list (ENC `role` -> role hiera):
- **osd-only hosts** = `roles::infra::k8s::*` (prodnxsr0001-0008 and prodnxsr0014-0019): include `profiles::ceph::osd` + `::client`, NOT `::mds` -> `[global]` only.
- **mon/mgr/mds hosts** = `roles::infra::incus::node` (prodnxsr0009-0013): also include `profiles::ceph::mds` -> `[global]` + `[mds]` + `[mds.*]`.
The template keys on `profiles::ceph::client::render_mds_config`, set `true` only in the incus node role hiera (the single role that includes `profiles::ceph::mds`). The `is_ceph_mds` fact is a runtime fallback, not used here.
## Changes
- Add cluster topology as a single source of truth in `hieradata/common.yaml`: `cluster_public_ips` (all 19 ceph host `/32`s, 198.18.23.1-19), `mon_initial_members` (the 5 mons), and the `mds_instances` map (two mds daemons per mon/mgr/mds host).
- Rework `profiles/ceph/client.conf.erb` to render `[global]` always and the `[mds]`/`[mds.*]` sections only when `render_mds_config` is true, preserving the live two-space indent, exact key order, and blank-line separators between mds sections.
- Extend `profiles::ceph::client` with topology + tunable params (sample values as defaults) and gate `[mds]` on `render_mds_config`.
- Drop the hard `Package[ceph-common]` dependency when the class does not manage the package (cephadm / `profiles::packages` deliver it on the k8s and incus hosts; k8s hosts do not declare `ceph-common`, so an unconditional require would fail to compile).
- Enable `manage_ceph_conf: true` on `roles/infra/k8s.yaml`, `roles/infra/k8s/node.yaml`, and `roles/infra/incus/node.yaml`; add `render_mds_config: true` to the incus node role. prodnxsr0014-0019 run `roles::infra::k8s::compute` (confirmed live via PuppetDB; `is_ceph_osd` true, `is_ceph_mds` false), so they are already covered by `k8s.yaml` and get the `[global]`-only variant with no mds sections.
## Live-cluster safety review (please read before merge)
This touches a running storage cluster's config. The rendered content is byte-equivalent to the live file **except** for the one deliberate `public_network` normalization below.
- **Intended change:** `public_network` is normalized to all 19 ceph host `/32`s on every host (198.18.23.1-19). Adoption is now a **single `public_network`-line change on every cluster host**, no longer a no-op on the osd hosts:
- osd hosts prodnxsr0001-0008 currently list `.1-.13` live -> first convergence adds `.14-.19`.
- mon/mgr/mds hosts prodnxsr0009-0013 currently list only `.9-.13` live -> first convergence adds `.1-.8` and `.14-.19`.
- prodnxsr0014-0019 (newly joined OSD hosts) get the file created with the full `[global]` variant.
- **Verified byte diff against the live files** (`/etc/ceph/ceph.conf` pulled from prodnxsr0008 and prodnxsr0009): each differs by **exactly one line** (the expanded `public_network`); on prodnxsr0009 the `[mds]` + `[mds.*]` sections and their blank-line separators are byte-identical to live. Rendering prodnxsr0009 with its current live `.9-.13` value reproduces the live file byte-for-byte, confirming the format match.
- RGW hosts (`roles::infra::ceph::rgw`, ausyd1nxvm2115-2119) include only `profiles::ceph::rgw` -> `profiles::ceph::conf`; they do not include `profiles::ceph::client`, so their variant is untouched and there is no duplicate `File[/etc/ceph/ceph.conf]`.
Rendered samples:
- osd-only (prodnxsr0008): `[global]` with all 19 `/32`s in `public_network`, nothing else.
- mon/mgr/mds (prodnxsr0009): the same `[global]` plus `[mds]` and `[mds.<host>-1|2]` for prodnxsr0009-0013, blank-line separated.
Reviewed-on: #515
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
During the k8s DNS cutover the `dns` module dual-writes every host's records two
ways: (a) LEGACY exported `@@concat::fragment`s collected onto the VM puppet DNS
master into `/var/named/sources/<zone>.conf`, and (b) NEW nsupdate/RFC2136 to the
k8s bind-authoritative cluster (`manage_nsupdate`). Both were gated on so the
legacy path stayed as a safety net.
The legacy path is no longer in any client's resolution path:
- **#490** repointed `profiles::dns::base::nameservers` from the VM resolver
anycast `198.18.19.16` to the k8s **bind-resolvers** LB `198.18.200.7`. Every
puppet client now resolves there.
- The k8s bind-resolvers forward `unkin.net`, `main.unkin.net`, and all
`18.198.in-addr.arpa` reverse zones to the k8s **bind-authoritative** cluster
(`198.18.200.6`), which is populated purely by the nsupdate path — the same
`profiles::dns::record` defines, `manage_nsupdate` branch.
- The VM master's assembled `/var/named/sources` files are therefore orphaned:
nothing resolves against them anymore.
This flips `profiles::dns::updater::manage_export: false` to stop nodes exporting
the legacy fragments. `manage_nsupdate` stays on.
## What changes
- `hieradata/common.yaml`: `profiles::dns::updater::manage_export` `true` -> `false`.
## Verification evidence
Pulled all **498** exported `Concat::Fragment` resources from PuppetDB (239 A,
232 PTR, 27 CNAME; 490 unique fqdn/type/value) and `dig`'d each against the k8s
authoritative read endpoint `198.18.200.7` (the dynamic path's serving side):
| Result | Count |
|---|---|
| MATCH | 447 |
| MISSING-from-dynamic | 38 |
| DIFFERENT-value | 5 |
Both non-match buckets are accounted for and are **not coverage gaps**:
- **38 MISSING** are all records the dynamic path deliberately filters via
`profiles::dns::updater::deny_ranges`
(`198.18.200.0/24`, `10.42.0.0/16`, ...) plus reverse zones the authoritative
cluster isn't even authoritative for: `*-kube-lb0` (198.18.200.x),
`*-flannel.1` (10.42.x.x pod CIDR), and `*-nomad`/`*-brcom1`
(172.26.64.x / 10.255.255.x). These are exactly the k8s/LB noise the deny
ranges exist to keep out of the authoritative zones. Verified every one falls
in a denied/non-authoritative range.
- **5 DIFFERENT** are stale legacy duplicates that the dynamic path holds *more*
correctly:
- 4x `au-syd1-prod-halb.main.unkin.net` A: four ceph nodes (2069-2073) each
self-register the shared name with their own primary IP (haproxy::dns
exports `networking.ip` per host, a legacy collision). Dynamic resolves it to
the single VIP `198.18.24.60`.
- 1x `dashboard.ceph.unkin.net` CNAME: the ceph node self-publishes this to
`lb1.unkin.net.` via its own nsupdate (per the k8s zones.yaml note); the
exported VRRP-name fragment is stale.
Conclusion: the dynamic path fully covers the served record set, and the serving
path no longer depends on the exported files.
## Collector-behavior note (what the flip actually does)
`profiles::dns::zone` realizes the fragments with an exported-resource collector
(`Concat::Fragment <<| target == ... |>>`). Turning off export purges the
fragments from PuppetDB, so on the VM master concat rebuilds each
`/var/named/sources/<zone>.conf` down to just the header (SOA+NS+glue) as nodes
converge. That is **cosmetic**: no client resolves against the VM master anymore
(#490), and the k8s authoritative zones are unaffected (fed by nsupdate, not
these files).
## Rollback
Set `profiles::dns::updater::manage_export: true` again and let puppet converge;
nodes re-export their fragments and the VM-master zone files repopulate.
## Follow-up (after soak)
Deleting the `@@concat::fragment` / `Concat::Fragment <<||>>` collector code in
`profiles::dns::record` / `profiles::dns::zone` (and the VM `profiles::dns::master`
zone assembly) is a separate cleanup PR once this has soaked and the VM master is
confirmed decommissionable.
Reviewed-on: #495
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- change the default anycast nameserver to bind-internal resolvers in
kubernetes
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #490
Replaces the exported-resources → puppet DNS master zone-file flow with per-host RFC2136 dynamic updates against the k8s **bind-authoritative** write endpoint (198.18.200.9). The master no longer manages zone files.
## Design
Each node assembles its DNS records into a local concat file; a systemd `.path` unit watches it and runs `dns-update` (nsupdate) on change — exactly the watch-a-file model requested.
## Changes
- **profiles::dns::updater** (new): concat records file + TSIG key file + `dns-update` script + `dns-update.service` (oneshot) + `dns-update.path` (watcher). The script sends only the delta since last run and deletes removed records, grouped per zone.
- **profiles::dns::record**: writes a local concat fragment (`zone|name|type|ttl|value`) instead of exporting `@@concat::fragment` to the master.
- **profiles::dns::base**: includes `profiles::dns::updater` (all nodes).
- **hiera**: `profiles::dns::updater` server/key_name/algorithm in common.yaml.
## Inert until keyed
The updater does nothing until `profiles::dns::updater::key_secret` (TSIG) is set in eyaml — records are assembled but not applied, so nodes are safe before the key exists.
## Prerequisites (k8s side, separate)
1. The `bind-authoritative` zones must set `dynamicUpdate: true` + an `updateKeyRef` (a client-update BindTSIGKey) so they accept these updates.
2. The TSIG key must be shared: the operator-generated key value goes into eyaml here (or the planned Vault-sync feature bridges it).
## Validated
puppet parser/epp validate, puppet-lint, and a functional test of the generated per-zone nsupdate message (replace + delete-removed).
Reviewed-on: #475
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
- refactor profiles::postfix::gateway as parameterized class
- move base postfix parameters, transports, and virtuals to hiera for flexibility
- convert SMTP restrictions to arrays for better readability using join()
- add postscreen enable/disable boolean with conditional master.cf configuration
- add per-domain TLS policy maps (smtp_tls_policy_maps)
- convert alias_maps to array parameter for flexibility
- convert all postfix map files to ERB templates with parameter hashes
- add map parameters: sender_canonical_maps, sender_access_maps, relay_recipients_maps,
relay_domains_maps, recipient_canonical_maps, recipient_access_maps, postscreen_access_maps, helo_access_maps
- move default map data to hiera while keeping parameters as empty hashes by default
This approach balances flexibility with data-driven configuration, allowing
easy customization through parameters while keeping transport/virtual maps
and default map data in hiera for role-specific overrides.
Reviewed-on: #416
- change from puppet-agent to openvox-agent
- upgrade version from 7.34 to 7.36
- ensure workflow of: Yumrepo -> dnf-makecache -> Package
Reviewed-on: #408
- start managing ceph configuration file
- manage ceph-radosgw
- merge the ceph::conf and ceph::node profiles
- ensure the ceph repos exist
- mange nginx frontend and consul service
Reviewed-on: #380
- create module class for journald clients
- ensure module class it used on all hosts
- use consul service address for insert/journald
Reviewed-on: #377
- vault 18.2 rpm produced by rpmbuilder repo
- ensure the /etc/vault directory is managed
- ensure service file is managed by puppet
- ensure package comes from unkin repo (not hashicorp)
- disable_mlock as unprivileged containers cannot use mlock
Reviewed-on: https://git.query.consul/unkinben/puppet-prod/pulls/264
- upgrade to puppet-consul changed default install method to archive
- ensure package method is used
- dont manage the repo, consul is packaged by rpmbuilder
Reviewed-on: https://git.query.consul/unkinben/puppet-prod/pulls/233
- update keepalived module to 5.1.0
- add keepalived::vrrp::* to be deep merged in hiera
- add vrrp dns configuration
- add vrrp instance/script to halb in syd1
- moved certbot into its own module
- added fact to list available certificates
- created systemd timer to rsync data to $data_dir/pub
- ensure the $data_dir/pub exists
- manage selinux for nginx
- refactor sonarr locations to generalised locations
- set locations to be deep merged
- updated hiera_include statements for media and media subroles
- added eyaml entries for all ldap credentials
- add ceph class to manage ceph client configuration/packages
- add cephfs define for mounting volumes
- add ceph keyring define to manage secrets used to mount cephfs
- ensure a database, user and credential is created for each grafana node
- ensure all databases for a region are included in a mariadb cluster
- refine params with stdlib types
unkin-agent
unkinben
benvin