add POST /logs installer log relay to VictoriaLogs
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/build Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful

A host being PXE-discovered or installed is not in Kubernetes, so vlagent
cannot collect its logs and a failed install leaves no record; the installer
environment has no internal-CA trust or credentials for the HTTPS log ingest,
and bootapi is already the plain-HTTP broker it can reach.

- add POST /logs, token-guarded like POST /provisioned, relaying ndjson to
  vlinsert's jsonline endpoint keyed on serial+phase
- stamp observed source IP and resolved NetBox device name into extra_fields
- return 202 on a sink failure so logs never block an install
- add BOOTAPI_VLINSERT_URL/_TIMEOUT and bootapi_log_relay metrics
This commit is contained in:
2026-10-03 19:44:27 +10:00
parent 0f0fb7fa8e
commit 3c77895788
9 changed files with 416 additions and 5 deletions
+30
View File
@@ -35,6 +35,7 @@ of the install.
| GET | `/boot/ipxe?mac=...` | Query-string alias of `/ipxe/{mac}`. |
| GET | `/ks/{ident}` | Rendered kickstart. `{ident}` is a MAC (auto-detected) or a hostname; trailing `.ks`/`.cfg` is stripped. |
| POST | `/provisioned/{ident}` | End-of-kickstart callback; clears `pxe_enabled` in NetBox. **Token-guarded** (`Authorization: Bearer <BOOTAPI_PROVISION_TOKEN>`). |
| POST | `/logs` | Relays a host's newline-delimited JSON install logs to VictoriaLogs. **Token-guarded** (same token). |
| GET | `/healthz` | Liveness: always `200 ok`. |
| GET | `/readyz` | Readiness: `200` once templates parsed. Does **not** probe NetBox. |
| GET | `/metrics` | Prometheus metrics (see below). |
@@ -85,6 +86,33 @@ bad/missing token, `404` for an unknown host, `503` when no
failure. The default kickstart templates call it from `%post` over plain HTTP
(the token authenticates the call; no CA trust needed at install time).
## The installer log relay
`POST /logs` exists because a host being PXE-discovered or installed is not in
Kubernetes, so the cluster's vlagent cannot collect its logs — if an install
fails, the only record dies with the machine. That environment also has no
internal-CA trust and no credentials for the HTTPS-only log ingest, and bootapi
is already the plain-HTTP broker it can reach, so bootapi forwards for it.
- Body: newline-delimited JSON, one log record per line (`application/x-ndjson`
or `application/json`), capped at **1 MiB**.
- Auth: the same `BOOTAPI_PROVISION_TOKEN` as `/provisioned`, same fail-closed
behavior — `503` when no token (or no `BOOTAPI_VLINSERT_URL`) is configured,
`401` on a bad/missing token.
- Forwarded as one short-timeout POST to
`{BOOTAPI_VLINSERT_URL}/insert/jsonline?_stream_fields=serial,phase&_msg_field=msg&_time_field=time`.
`serial` and `phase` are constant for a run and low-cardinality, so they key
the stream; MAC is per-NIC and goes in `extra_fields` so it stays searchable
without multiplying streams.
- `extra_fields` carries only what bootapi *observes* rather than what the
client claims: the request's source IP (`src_ip`), plus the resolved NetBox
device name (`device`) when `?mac=` resolves. Resolution is optional — a miss
just omits the field.
- Responses: `202` once the batch is accepted, `400` on an empty or oversized
body. A vlinsert failure is logged and counted but **still returns `202`** —
no retries, no buffering: a host must never block its install because the log
sink is down.
## Metrics
All on `/metrics`, prefix `bootapi_`:
@@ -95,6 +123,8 @@ All on `/metrics`, prefix `bootapi_`:
- `bootapi_netbox_lookup_duration_seconds{field}` — histogram.
- `bootapi_netbox_cache_hits_total` / `bootapi_netbox_cache_misses_total`.
- `bootapi_provisioned_total{result}` — result = `ok|unauthorized|notfound|error|disabled`.
- `bootapi_log_relay_total{result}` — result = `ok|error|unauthorized|disabled`.
- `bootapi_log_relay_lines_total` — installer log lines successfully relayed.
- `bootapi_ipxe_gated_total` — known hosts served local-boot because `pxe_enabled=false`.
- `bootapi_template_sync_total` / `bootapi_template_sync_failures_total` / `bootapi_template_generation` — template git-sync (see [template-authoring.md](template-authoring.md)).
- standard Go/process collectors.