Initial bootapi: NetBox-driven PXE/kickstart boot service

bootapi replaces Cobbler's PXE/kickstart side. It resolves a PXE-booting host from NetBox (by MAC or hostname), renders an iPXE boot script and a kickstart from Go text/templates, and serves them over HTTP. The ENC half already moved to encapi; this covers the provisioning/boot half. What's here: - cmd/bootapi + internal/{config,model,netbox,render,server}; embedded default templates under templates/ (AlmaLinux 9 + Fedora kickstarts, iPXE boot + unknown-MAC fallbacks) ported from Cobbler's boot/bootstrap contract. - NetBox client (v4.x API) behind a Resolver interface with a short-TTL cache; tested against httptest fixtures using real NetBox JSON shapes. - chi HTTP server: /ipxe/{mac}, /boot/ipxe?mac=, /ks/{ident}, healthz/readyz, Prometheus /metrics. Unknown MAC -> safe fallback iPXE (200), unknown KS -> 404. - Secrets (root pw hash, ssh keys) injected at render time from env/Vault, never NetBox. Config is env-based per estate convention. - Makefile (build/test/lint/docker + patch/minor/major), Dockerfile (distroless), .woodpecker (pre-commit, golangci-lint v2 + go test -race, docker build on PR; image push + Gitea binary release on v* tag), docs/ and example config. go build/vet clean, go test -race green, golangci-lint v2 clean, pre-commit clean. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
2026-07-28 17:20:06 +10:00
parent 98e69d2fcb
commit 274c480b09
37 changed files with 3290 additions and 1 deletions
@@ -0,0 +1,77 @@
+# Template data model
+
+Every kickstart and iPXE template is rendered with Go `text/template` against a
+single flat value. This is the exact, stable contract template authors code
+against. It is assembled in `internal/render.dataFor` from a NetBox device
+(`internal/netbox`) plus render-time config (env/Vault).
+
+## Kickstart templates (`*.ks.tmpl`)
+
+| Field | Type | Source | Notes |
+|-------|------|--------|-------|
+| `.Hostname` | string | NetBox device name | short name, e.g. `web01` |
+| `.Domain` | string | NetBox CF `domain`, else `BOOTAPI_DOMAIN` | |
+| `.FQDN` | string | derived | `Hostname.Domain` (or `Hostname` if no domain) |
+| `.Platform` | string | NetBox platform slug | e.g. `almalinux9` — primary template-selection key |
+| `.OSFamily` | string | derived from platform | e.g. `almalinux` |
+| `.OSVersion` | string | derived from platform | e.g. `9` |
+| `.Arch` | string | fixed `x86_64` (today) | |
+| `.Role` | string | NetBox device role slug | e.g. `kubernetes-worker` |
+| `.Interfaces` | `[]Interface` | NetBox interfaces + IPs | primary interface sorted first |
+| `.PrimaryInterface` | `*Interface` | derived | the NIC carrying the primary IP (or first) |
+| `.PrimaryIP` | string | NetBox device `primary_ip` | address only, no prefix |
+| `.Nameservers` | `[]string` | NetBox CF `nameservers`, else `BOOTAPI_NAMESERVERS` | |
+| `.RootPasswordHash` | string | **render-time** (`BOOTAPI_ROOT_PASSWORD_HASH[_FILE]`) | crypt(3) hash; empty ⇒ lock root. **Never** from NetBox — see [security.md](security.md) |
+| `.SSHAuthorizedKeys` | `[]string` | **render-time** (`BOOTAPI_SSH_AUTHORIZED_KEYS`) | |
+| `.PuppetServer` | string | `BOOTAPI_PUPPET_SERVER` | default `puppet.query.consul` |
+| `.PuppetCAServer` | string | `BOOTAPI_PUPPET_CA_SERVER` | default `puppetca.query.consul` |
+| `.BaseURL` | string | `BOOTAPI_BASE_URL` | bootapi's own URL |
+| `.BootBaseURL` | string | `BOOTAPI_BOOT_BASE_URL` | OS install-tree base |
+| `.KickstartURL` | string | derived | `BaseURL/ks/Hostname` |
+| `.Custom` | `map[string]any` | **all** NetBox custom fields, verbatim | escape hatch for site-specific knobs without a code change |
+
+### `Interface`
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `.Name` | string | NetBox interface name, e.g. `eth0` |
+| `.MAC` | string | normalized lower-case colon form |
+| `.IP` | string | address only (empty ⇒ no IP; skip in the network stanza) |
+| `.PrefixLen` | int | CIDR length, e.g. `24` |
+| `.Netmask` | string | dotted-quad, e.g. `255.255.255.0` |
+| `.Gateway` | string | per-IP CF `gateway`, else device CF `gateway`, else empty |
+| `.VLAN` | int | untagged VLAN id, or 0 |
+| `.Primary` | bool | true for the NIC with the primary IP |
+
+## iPXE templates (`*.ipxe.tmpl`)
+
+Rendered with everything above **plus**:
+
+| Field | Type | Notes |
+|-------|------|-------|
+| `.KernelURL` | string | `BootBaseURL/images/pxeboot/vmlinuz` (empty if `BootBaseURL` unset) |
+| `.InitrdURL` | string | `BootBaseURL/images/pxeboot/initrd.img` |
+
+The fallback templates (`fallback-local`, `fallback-shell`) are rendered with an
+empty value — they take no host data by design.
+
+## NetBox custom fields bootapi reads
+
+Define these on the *device* (or, where noted, the *IP address*) in NetBox.
+All are optional; sensible fallbacks apply.
+
+| Custom field | On | Effect |
+|--------------|----|--------|
+| `domain` | device | DNS domain; overrides `BOOTAPI_DOMAIN` |
+| `gateway` | device / IP address | default gateway (IP-level wins) |
+| `nameservers` | device | comma-separated resolvers; overrides `BOOTAPI_NAMESERVERS` |
+| `provision_template` | device | force a specific template name (see below) |
+
+## Template selection precedence
+
+`SelectKickstart` picks the first template name that exists, in order:
+
+1. `provision_template` custom field (exact template name)
+2. `.Platform` slug (e.g. `almalinux9`)
+3. `.OSFamily` (e.g. `almalinux`, or `fedora`)
+4. `BOOTAPI_DEFAULT_TEMPLATE` (default `almalinux9`)
@@ -0,0 +1,79 @@
+# Deploying bootapi
+
+> The actual argocd-apps deployment is a **follow-up task** and is intentionally
+> not part of this repo. This document is the spec for that follow-up plus the
+> DHCP change the estate needs.
+
+bootapi is a stateless HTTP service. It mirrors encapi's deployment shape: a Go
+binary in a distroless image, config from env, secrets from Vault via the Vault
+Secrets Operator (VSO).
+
+## Container image
+
+`git.unkin.net/unkin/bootapi:<tag>` (built + pushed by `.woodpecker/docker.yaml`
+on a `v*` tag). Also mirror to the artifactapi local docker registry if desired.
+
+## Kubernetes wiring (argocd-apps follow-up)
+
+Create `apps/base/bootapi/` following the argocd-apps `AGENTS.md` pattern:
+
+1. **namespace** `bootapi`.
+2. **VaultAuth** (`default`) — kubernetes method, mount `k8s/au/syd1`, role
+   `default`, SA `default` (copy netbox's `vaultauth.yaml`).
+3. **VaultStaticSecret** → k8s Secret `bootapi-secrets`, from Vault kv path
+   `kubernetes/namespace/bootapi/default/bootapi-secrets` with keys:
+   - `netbox_token` — a **dedicated, read-only** NetBox API token for bootapi
+     (create a `bootapi` NetBox user/token via terraform-netbox rather than
+     reusing the seeded superuser token at
+     `kv/kubernetes/namespace/netbox/default/netbox-superuser`).
+   - `root_password_hash` — crypt(3) hash for the installed root account
+     (the successor to Cobbler's eyaml `default_password_crypted`).
+   - `ssh_authorized_keys` — optional, newline-separated.
+4. **ConfigMap** `bootapi-templates` (optional) — override `*.ks.tmpl` /
+   `*.ipxe.tmpl`, mounted at `BOOTAPI_TEMPLATE_DIR=/etc/bootapi/templates`. Omit
+   to use the embedded defaults. Annotate the Deployment with
+   `reloader.stakater.com/auto: "true"` so template edits roll the pods.
+5. **Deployment** — image above, env from `config.example.env`, secret keys wired
+   as `BOOTAPI_NETBOX_TOKEN_FILE`/`BOOTAPI_ROOT_PASSWORD_HASH_FILE` (mount the
+   Secret) or `...FROM secretKeyRef`. Least-privilege securityContext
+   (`runAsNonRoot`, `drop: [all]`). Baseline resources: requests `512Mi`/`1`,
+   limits `2Gi`/`2` cpu.
+6. **Service** `bootapi` (ClusterIP, port 80 → 8000) plus a **LoadBalancer** (or
+   Gateway HTTPRoute) reachable by PXE clients at a stable address/hostname —
+   this is what DHCP points at. Reuse the Vault-issued TLS the Cobbler vhost used
+   if you terminate TLS at a gateway; note that iPXE fetches are plain HTTP, so a
+   plain HTTP listener on the PXE VLAN is required either way.
+7. Register in `argocd/applicationsets/platform.yaml` (`apps/overlays/*/bootapi`)
+   and the platform AppProject destinations.
+
+### Cross-repo dependencies (per estate conventions)
+
+- **argocd-apps**: add a `serviceaccount_*` under `apps/base/woodpecker/` if the
+  bootapi pipelines need a dedicated SA (they use `default` today).
+- **terraform-vault**: add the k8s auth role + kv policy granting the `bootapi`
+  namespace read on `kv/kubernetes/namespace/bootapi/default/*`.
+- **terraform-netbox**: create the read-only `bootapi` NetBox token and seed it
+  (plus `root_password_hash`) into the Vault kv path above.
+
+## DHCP change (the cutover)
+
+Cobbler advertised itself at anycast `198.18.19.19` as the DHCP `next-server`,
+with `filename "/ipxe.efi"` (UEFI arch 7/9) or `/undionly.kpxe` (BIOS arch 0).
+Today those are set in `puppet-prod` hieradata
+`hieradata/roles/infra/dhcp/server.yaml` (`pools.*.pxeserver` and the
+`UEFI-64`/`Legacy` dhcp classes).
+
+To cut a subnet over to bootapi, repoint DHCP for that pool:
+
+- `next-server` → bootapi's LB IP (or keep the `198.18.19.19` anycast and move
+  the anycast advertisement to bootapi's node/LB).
+- `filename` → the iPXE binary as before (`/ipxe.efi` / `/undionly.kpxe`); bootapi
+  does not serve the NBP itself. The chained iPXE must then be told to fetch
+  bootapi's script — either bake `chain http://<bootapi>/ipxe/${net0/mac}` into
+  the site iPXE binary/embedded script, or set DHCP option 67 to that URL for
+  iPXE user-class requests. This replaces Cobbler's
+  `chain http://${next-server}/cblr/svc/op/gpxe/mac/${net0/mac}`.
+
+Roll one pool at a time (the PXE subnets are `198.18.13.0/24`–`198.18.17.0/24`);
+Puppet autosign already trusts those subnets and `*.main.unkin.net`, so a host
+installed via bootapi checks in exactly as before.
@@ -0,0 +1,68 @@
+# bootapi HTTP endpoints
+
+bootapi speaks plain HTTP. It is fronted by the same Vault-issued TLS the Cobbler
+server used; the booting firmware reaches it at the DHCP `next-server` (see
+[deployment.md](deployment.md)).
+
+## The PXE flow
+
+```
+DHCP  ── next-server + filename (ipxe.efi / undionly.kpxe) ──▶  firmware loads iPXE
+iPXE  ── GET /ipxe/<mac> ───────────────────────────────────▶  bootapi renders a boot script
+boot  ── kernel + initrd + inst.ks=<BASE_URL>/ks/<host> ────▶  Anaconda fetches the kickstart
+KS    ── GET /ks/<host> ────────────────────────────────────▶  bootapi renders the kickstart
+```
+
+This mirrors Cobbler, which chained iPXE to `/cblr/svc/op/gpxe/mac/<mac>` and
+served a per-system script carrying `inst.ks=`.
+
+## Endpoints
+
+| Method | Path | Purpose |
+|--------|------|---------|
+| GET | `/ipxe/{mac}` | iPXE boot script for the host owning `{mac}`. `{mac}` may use `:`/`-`/`.` separators or be bare hex; a trailing `.ipxe` is stripped. |
+| GET | `/boot/ipxe?mac=...` | Query-string alias of `/ipxe/{mac}` (some firmware finds this shape easier to template). |
+| GET | `/ks/{ident}` | Rendered kickstart. `{ident}` is a MAC (auto-detected) or a hostname; trailing `.ks`/`.cfg` is stripped. |
+| GET | `/healthz` | Liveness: always `200 ok`. |
+| GET | `/readyz` | Readiness: `200` once templates parsed. Does **not** probe NetBox (a NetBox outage still lets iPXE serve the safe fallback). |
+| GET | `/metrics` | Prometheus metrics (see below). |
+
+## Host identification
+
+A booting host is identified by the **MAC** of the NIC it PXE-booted from
+(`/ipxe/{mac}`), which bootapi resolves via NetBox
+`GET /api/dcim/interfaces/?mac_address=<mac>` → device → primary IP, platform,
+role, interfaces. `/ks/{ident}` additionally accepts a **hostname** (NetBox
+device name), for hand-testing and for installers that template the hostname
+into the kickstart URL.
+
+## Error behavior (important, and deliberate)
+
+The two endpoints fail **differently** on an unknown host, because the cost of a
+wrong answer differs:
+
+- **`/ipxe/{mac}` never returns 404.** iPXE needs a syntactically valid script or
+  the boot chain simply errors. An unknown MAC — or *any* NetBox error — returns
+  HTTP 200 with the **fallback script** selected by `BOOTAPI_UNKNOWN_MAC_FALLBACK`:
+  - `local` (default): `sanboot` the local disk. Safe: a machine that PXE-booted
+    by accident (or a NetBox blip) just boots its installed OS; a genuinely new
+    machine loops back to PXE next time, by which point NetBox should know it. We
+    deliberately do **not** start an installer for a machine we can't identify —
+    that could wipe a production box.
+  - `shell`: drop to an interactive iPXE shell so an operator racking a new box
+    can read `${net0/mac}` and register it. Opt-in; unsafe as a default because
+    it halts the boot.
+- **`/ks/{ident}` returns 404** for an unknown host (and 502 on a NetBox error).
+  By the time Anaconda fetches the kickstart it has already committed to
+  installing; a clear failure is safer than serving an empty or wrong kickstart.
+
+## Metrics
+
+All on `/metrics`, prefix `bootapi_`:
+
+- `bootapi_http_requests_total{endpoint,status}` — endpoint = `ipxe|ks|healthz|readyz`, status = `2xx|3xx|4xx|5xx`.
+- `bootapi_render_total{kind,result}` — kind = `kickstart|ipxe`, result = `ok|error`.
+- `bootapi_netbox_lookups_total{field,result}` — field = `mac|name`, result = `ok|notfound|error`.
+- `bootapi_netbox_lookup_duration_seconds{field}` — histogram.
+- `bootapi_netbox_cache_hits_total` / `bootapi_netbox_cache_misses_total`.
+- standard Go/process collectors.
@@ -0,0 +1,49 @@
+# Security: secrets in kickstarts
+
+A kickstart is fetched over the network by an unauthenticated installer and can
+embed real secrets: the root password hash, SSH keys, bootstrap tokens, repo
+credentials. bootapi's rule is:
+
+**NetBox holds identity and topology, never secrets. Secrets are injected at
+render time from Vault/env.**
+
+## What goes where
+
+| Value | Where it lives | How it reaches the template |
+|-------|----------------|-----------------------------|
+| hostname, domain, IPs, MACs, VLANs, gateway, platform, role | NetBox | `internal/netbox` → `model.Host` |
+| template selection knobs (`provision_template`, `nameservers`, `gateway`) | NetBox custom fields | `.Custom` / typed fields |
+| **root password hash** | Vault → `BOOTAPI_ROOT_PASSWORD_HASH[_FILE]` | `.RootPasswordHash` |
+| **SSH authorized keys** | Vault → `BOOTAPI_SSH_AUTHORIZED_KEYS` | `.SSHAuthorizedKeys` |
+| puppet CA/server names | env (not secret) | `.PuppetServer` / `.PuppetCAServer` |
+
+`BOOTAPI_ROOT_PASSWORD_HASH_FILE` and `BOOTAPI_NETBOX_TOKEN_FILE` let the values
+arrive as Vault-mounted files rather than env, which is the k8s norm (see
+[deployment.md](deployment.md)). If no root hash is configured, the default
+templates emit `rootpw --lock` rather than a blank/guessable password.
+
+This matches the pre-bootapi setup, where the root hash was Cobbler's eyaml
+`default_password_crypted` injected into `settings.yaml` — an operator-managed
+secret, never in the NetBox/inventory layer.
+
+## Exposure notes
+
+- Kickstarts are served over **HTTP** to the installer, so treat any embedded
+  secret as visible to anything on the provisioning VLAN. Keep bootapi's
+  kickstart endpoint on the trusted PXE network, exactly as Cobbler's was.
+- The root password hash *is* in the rendered kickstart by necessity (Anaconda
+  needs it). Prefer SSH-key login + a locked or strong-random root password, and
+  rotate the hash in Vault as normal.
+- The **puppet bootstrap uses no long-lived token**: the host generates a CSR and
+  the puppetmaster autosigns it based on source subnet + `*.main.unkin.net`
+  (unchanged from Cobbler). So the kickstart carries no puppet secret.
+
+## Follow-up: per-template Vault lookups
+
+Today all render-time secrets are process-wide env/files (one root hash, one key
+set for the fleet), which covers the current estate. If per-host or per-role
+secrets are ever needed (e.g. a distinct bootstrap token per role), the seam is
+`internal/render.dataFor`: add a Vault kv fetch keyed by host/role there, behind
+an interface, the same way encapi's `internal/distro` resolver injects per-host
+params behind an interface. Tracked as a follow-up, not implemented, to avoid
+giving bootapi broad Vault read scope before it's needed.
@@ -0,0 +1,71 @@
+# Authoring templates
+
+bootapi ships an embedded default set and lets you override or extend it.
+
+## Where templates live
+
+- **Embedded defaults**: `templates/kickstart/*.ks.tmpl` and
+  `templates/ipxe/*.ipxe.tmpl`, compiled into the binary (`templates/embed.go`).
+- **Overrides**: any directory pointed to by `BOOTAPI_TEMPLATE_DIR`. Files there
+  with the same base name **replace** the embedded one; new names **add** to the
+  set. In Kubernetes this is a ConfigMap mount (see [deployment.md](deployment.md)).
+
+## Naming
+
+- Kickstart: `<name>.ks.tmpl` → registered as template `<name>`.
+- iPXE: `<name>.ipxe.tmpl` → registered as template `<name>`.
+
+`<name>` is what template selection matches against (platform slug, OS family,
+`provision_template`, or the configured default — see
+[data-model.md](data-model.md#template-selection-precedence)).
+
+Reserved iPXE names bootapi renders directly:
+- `boot` — the per-host boot script (`/ipxe/{mac}` for a known host).
+- `fallback-local`, `fallback-shell` — unknown-MAC fallbacks.
+
+## Engine and functions
+
+Standard Go `text/template`. Available funcs: `join`, `upper`, `lower`,
+`default` (`{{ default "x" .Maybe }}` → `.Maybe` unless empty). The data model is
+in [data-model.md](data-model.md).
+
+Example network stanza (iterate interfaces, skip those without an IP, set the
+hostname on the primary):
+
+```gotemplate
+{{- $primary := .PrimaryInterface }}
+{{- range .Interfaces }}
+{{- if .IP }}
+network --bootproto=static --device={{ .MAC }} --ip={{ .IP }} --netmask={{ .Netmask }}{{ if .Gateway }} --gateway={{ .Gateway }}{{ end }}{{ range $.Nameservers }} --nameserver={{ . }}{{ end }}{{ if and $primary (eq .MAC $primary.MAC) }} --hostname={{ $.FQDN }}{{ end }} --activate
+{{- end }}
+{{- end }}
+```
+
+## What the default AlmaLinux template does (ported from Cobbler)
+
+The literal `.ks` bodies from the old Cobbler server are not in version control
+(they lived in `/var/lib/cobbler/{templates,snippets}` on the Cobbler host). The
+embedded `almalinux9.ks.tmpl` reproduces the **contract** that estate relied on:
+
+- static per-interface networking from NetBox, hostname on the primary NIC;
+- `rootpw --iscrypted` from the render-time hash (Cobbler's
+  `default_password_crypted`), or `--lock` when unset;
+- a minimal package set + `openssh-server`, `chrony`;
+- a `%post` that installs the Puppet agent, points it at `puppet.query.consul` /
+  `puppetca.query.consul`, and enables it — handing off to the existing
+  `profiles::firstrun` Puppet bootstrap and autosign, exactly as the Cobbler
+  kickstart did.
+
+Adjust partitioning, package sets and repos to taste; keep the puppet `%post`
+handoff so a freshly-installed host still checks in and converges.
+
+## Testing a template locally
+
+```bash
+BOOTAPI_NETBOX_URL=... BOOTAPI_NETBOX_TOKEN=... \
+BOOTAPI_BASE_URL=http://localhost:8000 \
+BOOTAPI_BOOT_BASE_URL=http://mirror/almalinux/9 \
+BOOTAPI_TEMPLATE_DIR=./mytemplates ./bin/bootapi &
+curl -s localhost:8000/ks/web01        # rendered kickstart
+curl -s localhost:8000/ipxe/aa:bb:cc:00:11:22
+```