Discovery boot flow for unknown devices → auto-populate NetBox #3

New Issue

2026-08-05T00:54:40+10:00

unkinben commented

2026-08-05 00:54:40 +10:00

Problem

bootapi resolves a PXE-booting machine by MAC/hostname against NetBox and serves /ipxe; an unknown MAC currently falls through to the fallback iPXE (fallback-shell/fallback-local). So a brand-new machine cannot bootstrap itself into NetBox — a human must pre-create the device + interface MAC (today done transitionally via terraform-infra bootstrap_mac). There is also no source of switch/port topology anywhere in the estate (no LLDP fact in PuppetDB, none in Cobbler); it can only be captured by the machine at boot.

Proposal

Add a discovery path to the /ipxe decision:

Decision (bootapi): unknown MAC (NetBox lookup miss) → serve a discovery iPXE chain instead of the fallback. Known-but-staged/unprovisioned devices already flowing normally are unaffected. Reuse the existing NetBox-miss branch + gate/metrics (bootapi_netbox_lookups_total{result=miss}, ipxeGated).
Discovery image (see bootapi-templates + bootapi-images): boots a small live image that, in a %pre-style stage, collects hardware reality:
- dmidecode → system serial, product name/model, UUID, board, chassis type (asset tag if type-3 populated — usually blank on these OptiPlex).
- All physical NICs by enumeration (ip -j link), name + MAC + link state. Never assume enp2s0: interface names vary by model (OptiPlex 3060 enumerates its onboard NIC as enp1s0, 3050/7080 as enp2s0) — key on MAC / lowest-numbered physical NIC.
- LLDP neighbours via lldpd/lldpctl -f json → switch chassis-id/sysname + port. This is the only way to obtain topology and is the highest-value new capability.
- Optional inventory: CPU, RAM, disks.
NetBox writes (bootapi endpoint, e.g. POST /discover): match to an existing declared device by primary MAC or serial; if matched, populate reality (create real interfaces + netbox_mac_address objects, replace the transitional bootstrap interface, set primary IP, record serial/UUID/inventory, store LLDP neighbour as a cable/connection or a custom field). If unmatched, create an inventory/staged device for a human to adopt. bootapi owns the NetBox write (it already holds NetBox creds); the discovery image only POSTs collected JSON.
Reboot into normal flow: after a successful discover POST, the device is now known → next iPXE cycle resolves to its normal provisioning profile (chainload reboot).

Acceptance

Unknown MAC at /ipxe serves the discovery chain (not the fallback); covered by a test.
The discovery image POSTs serial + model + UUID + every NIC (name/MAC/link) + LLDP neighbour(s) as JSON to bootapi.
bootapi matches by primary MAC or serial, writes interfaces/MACs/primary-IP/serial/LLDP into NetBox, and replaces any transitional bootstrap interface.
After discovery, a reboot resolves the host to its normal profile and it provisions without human NetBox edits.
Interface handling is MAC/enumeration-keyed; no hardcoded interface name anywhere (regression-tested against the 3060 enp1s0 case).

Cross-refs

bootapi-templates: discovery iPXE template + %pre collection script + catalog entry (companion issue).
terraform-infra: retires the transitional bootstrap_mac once discovery lands; NetBox stays IP/interface-authoritative.

## Problem bootapi resolves a PXE-booting machine by MAC/hostname against NetBox and serves `/ipxe`; an **unknown MAC** currently falls through to the fallback iPXE (`fallback-shell`/`fallback-local`). So a brand-new machine cannot bootstrap itself into NetBox — a human must pre-create the device + interface MAC (today done transitionally via terraform-infra `bootstrap_mac`). There is also **no source of switch/port topology anywhere** in the estate (no LLDP fact in PuppetDB, none in Cobbler); it can only be captured by the machine at boot. ## Proposal Add a discovery path to the `/ipxe` decision: 1. **Decision (bootapi):** unknown MAC (NetBox lookup miss) → serve a **discovery** iPXE chain instead of the fallback. Known-but-`staged`/unprovisioned devices already flowing normally are unaffected. Reuse the existing NetBox-miss branch + gate/metrics (`bootapi_netbox_lookups_total{result=miss}`, `ipxeGated`). 2. **Discovery image (see bootapi-templates + bootapi-images):** boots a small live image that, in a `%pre`-style stage, collects hardware reality: - `dmidecode` → system serial, product name/model, UUID, board, chassis type (asset tag if type-3 populated — usually blank on these OptiPlex). - **All physical NICs by enumeration** (`ip -j link`), name + MAC + link state. Never assume `enp2s0`: interface names vary by model (OptiPlex 3060 enumerates its onboard NIC as `enp1s0`, 3050/7080 as `enp2s0`) — key on MAC / lowest-numbered physical NIC. - **LLDP neighbours** via `lldpd`/`lldpctl -f json` → switch chassis-id/sysname + port. This is the only way to obtain topology and is the highest-value new capability. - Optional inventory: CPU, RAM, disks. 3. **NetBox writes (bootapi endpoint, e.g. `POST /discover`):** match to an existing declared device by **primary MAC or serial**; if matched, populate reality (create real interfaces + `netbox_mac_address` objects, replace the transitional `bootstrap` interface, set primary IP, record serial/UUID/inventory, store LLDP neighbour as a cable/connection or a custom field). If unmatched, create an `inventory`/`staged` device for a human to adopt. bootapi owns the NetBox write (it already holds NetBox creds); the discovery image only POSTs collected JSON. 4. **Reboot into normal flow:** after a successful discover POST, the device is now known → next iPXE cycle resolves to its normal provisioning profile (chainload reboot). ## Acceptance - Unknown MAC at `/ipxe` serves the discovery chain (not the fallback); covered by a test. - The discovery image POSTs serial + model + UUID + every NIC (name/MAC/link) + LLDP neighbour(s) as JSON to bootapi. - bootapi matches by primary MAC or serial, writes interfaces/MACs/primary-IP/serial/LLDP into NetBox, and replaces any transitional `bootstrap` interface. - After discovery, a reboot resolves the host to its normal profile and it provisions without human NetBox edits. - Interface handling is MAC/enumeration-keyed; no hardcoded interface name anywhere (regression-tested against the 3060 `enp1s0` case). ## Cross-refs - bootapi-templates: discovery iPXE template + `%pre` collection script + catalog entry (companion issue). - terraform-infra: retires the transitional `bootstrap_mac` once discovery lands; NetBox stays IP/interface-authoritative.

unkinben commented

2026-08-05 00:55:02 +10:00