Backfill NetBox reality from pdbmux #6

2026-08-06T23:15:46+10:00

unkinben commented

2026-08-06 23:15:46 +10:00

Closes #1.

Automates the PuppetDB reality backfill into NetBox. Per the issue's latest
comment this queries pdbmux (the PuppetDB multiplexer) rather than PuppetDB
directly: pdbmux merges the old (Consul) and new (k8s) PuppetDBs, needs no auth,
and its HTTPS gateway (pdbmux.k8s.syd1.au.unkin.net) is reachable from
CI/workstations — the reachability gap that blocked the original in-cluster-only
script. The agreed design is a generator producing reviewable committed output,
not a live data source or CronJob.

Changes:

Add tools/backfill (Go, no external deps): queries pdbmux
/pdb/query/v4/facts for prodnxsr0001-0013 and emits per-host reality —
serial/model/UUID, every recordable interface (real NICs plus
overlay/loopback/kube-lb) with MAC and CIDR IPs, and CPU/RAM/disk inventory.
Filter reality to what NetBox should hold: keep physical NICs always and
virtual interfaces only when they carry a routable address (drops the ephemeral
Calico cali* veths and lo, keeps flannel.1/kube-lb0/loopback*); skip
Ceph RBD and other virtual block devices; take interface names from Facter,
never assume them.
Emit deterministic, idempotent, yamllint-clean YAML into
config/au/syd1/reality/<host>.yaml, generated for all 13 existing physicals.
Extend modules/infra: add a reality variable and reality.tf creating
netbox_device_interface / netbox_mac_address / netbox_ip_address /
netbox_inventory_item and the device serial. Reality wires into NetBox only
for hosts that also have an intent devices/<host>.yaml.
Load reality/*.yaml in the terragrunt env; add make backfill; add a
go vet/go test woodpecker job; remove the in-cluster-only Python script.

Validation:

go vet + go test -race green (mock-pdbmux httptest, real captured fact
fixture, golden emit, mask/filter cases).
tofu validate on modules/infra succeeds; the reality variable type-checks
against all 13 generated files and the reality.tf locals produce the expected
interface/MAC/IP/inventory maps under the intent guard.
repo pre-commit passes (tofu fmt, terragrunt hcl fmt, yamllint, eof/whitespace).

Dependencies / notes:

Targets benvin/initial-infra; PR #5 (which introduces this repo's module and
layout) is not yet merged. Rebase onto main once #5 lands.
The 13 hosts' intent devices are seeded separately (#2); until that merges each
host's reality/*.yaml is committed and reviewed but produces no NetBox
resources (guarded by intent presence). No reboots required (acceptance).

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

Closes #1. Automates the PuppetDB reality backfill into NetBox. Per the issue's latest comment this queries **pdbmux** (the PuppetDB multiplexer) rather than PuppetDB directly: pdbmux merges the old (Consul) and new (k8s) PuppetDBs, needs no auth, and its HTTPS gateway (`pdbmux.k8s.syd1.au.unkin.net`) is reachable from CI/workstations — the reachability gap that blocked the original in-cluster-only script. The agreed design is a generator producing reviewable committed output, not a live data source or CronJob. Changes: - Add `tools/backfill` (Go, no external deps): queries pdbmux `/pdb/query/v4/facts` for prodnxsr0001-0013 and emits per-host reality — serial/model/UUID, every recordable interface (real NICs plus overlay/loopback/kube-lb) with MAC and CIDR IPs, and CPU/RAM/disk inventory. - Filter reality to what NetBox should hold: keep physical NICs always and virtual interfaces only when they carry a routable address (drops the ephemeral Calico `cali*` veths and `lo`, keeps `flannel.1`/`kube-lb0`/`loopback*`); skip Ceph RBD and other virtual block devices; take interface names from Facter, never assume them. - Emit deterministic, idempotent, yamllint-clean YAML into `config/au/syd1/reality/<host>.yaml`, generated for all 13 existing physicals. - Extend `modules/infra`: add a `reality` variable and `reality.tf` creating `netbox_device_interface` / `netbox_mac_address` / `netbox_ip_address` / `netbox_inventory_item` and the device `serial`. Reality wires into NetBox only for hosts that also have an intent `devices/<host>.yaml`. - Load `reality/*.yaml` in the terragrunt env; add `make backfill`; add a `go vet`/`go test` woodpecker job; remove the in-cluster-only Python script. Validation: - `go vet` + `go test -race` green (mock-pdbmux httptest, real captured fact fixture, golden emit, mask/filter cases). - `tofu validate` on `modules/infra` succeeds; the `reality` variable type-checks against all 13 generated files and the `reality.tf` locals produce the expected interface/MAC/IP/inventory maps under the intent guard. - repo pre-commit passes (tofu fmt, terragrunt hcl fmt, yamllint, eof/whitespace). Dependencies / notes: - Targets `benvin/initial-infra`; PR #5 (which introduces this repo's module and layout) is not yet merged. Rebase onto `main` once #5 lands. - The 13 hosts' intent devices are seeded separately (#2); until that merges each host's `reality/*.yaml` is committed and reviewed but produces no NetBox resources (guarded by intent presence). No reboots required (acceptance). https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben added 1 commit 2026-08-06 23:15:47 +10:00

Backfill NetBox reality from pdbmux

ci/woodpecker/pr/plan Pipeline failed

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/backfill-test Pipeline was successful

Details

585d32b15c

Automate the PuppetDB reality import (issue #1) by querying pdbmux — the
PuppetDB multiplexer whose HTTPS gateway is reachable from CI/workstations,
unlike raw PuppetDB — instead of PuppetDB directly, and shaping the result to
the NetBox reality side the devices module reconciles.

- Add tools/backfill (Go): query pdbmux /pdb/query/v4/facts for the 13 existing
  physicals and emit per-host reality YAML — serial/model/UUID, every recordable
  interface (real NICs plus overlay/loopback/kube-lb) with MAC and CIDR IPs, and
  CPU/RAM/disk inventory. Filter ephemeral Calico veths and Ceph RBD volumes;
  take interface names from Facter, never assume them.
- Emit deterministic, idempotent, yamllint-clean output into
  config/au/syd1/reality/<host>.yaml, generated for prodnxsr0001-0013.
- Extend modules/infra with a reality variable and reality.tf creating
  netbox_device_interface/netbox_mac_address/netbox_ip_address/
  netbox_inventory_item and device serial; wire reality only for hosts that also
  have an intent device.
- Load reality/*.yaml in the terragrunt env; add `make backfill`; add a
  go vet/test woodpecker job; drop the in-cluster-only Python script.

Closes #1

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

unkinben referenced this pull request

2026-08-08 18:33:52 +10:00

pdbmux backfill: consume the puppet lldp fact to populate NetBox switch/port cabling #7

ci/woodpecker/pr/plan Pipeline failed

Details

ci/woodpecker/pr/pre-commit Pipeline was successful

Details

ci/woodpecker/pr/backfill-test Pipeline was successful

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin benvin/pdbmux-backfill:benvin/pdbmux-backfill

git checkout benvin/pdbmux-backfill

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/terraform-infra#6