add POST /logs installer log relay to VictoriaLogs
A host being PXE-discovered or installed is not in Kubernetes, so vlagent cannot collect its logs and a failed install leaves no record; the installer environment has no internal-CA trust or credentials for the HTTPS log ingest, and bootapi is already the plain-HTTP broker it can reach. - add POST /logs, token-guarded like POST /provisioned, relaying ndjson to vlinsert's jsonline endpoint keyed on serial+phase - stamp observed source IP and resolved NetBox device name into extra_fields - return 202 on a sink failure so logs never block an install - add BOOTAPI_VLINSERT_URL/_TIMEOUT and bootapi_log_relay metrics
This commit is contained in:
@@ -35,6 +35,7 @@ of the install.
|
||||
| GET | `/boot/ipxe?mac=...` | Query-string alias of `/ipxe/{mac}`. |
|
||||
| GET | `/ks/{ident}` | Rendered kickstart. `{ident}` is a MAC (auto-detected) or a hostname; trailing `.ks`/`.cfg` is stripped. |
|
||||
| POST | `/provisioned/{ident}` | End-of-kickstart callback; clears `pxe_enabled` in NetBox. **Token-guarded** (`Authorization: Bearer <BOOTAPI_PROVISION_TOKEN>`). |
|
||||
| POST | `/logs` | Relays a host's newline-delimited JSON install logs to VictoriaLogs. **Token-guarded** (same token). |
|
||||
| GET | `/healthz` | Liveness: always `200 ok`. |
|
||||
| GET | `/readyz` | Readiness: `200` once templates parsed. Does **not** probe NetBox. |
|
||||
| GET | `/metrics` | Prometheus metrics (see below). |
|
||||
@@ -85,6 +86,33 @@ bad/missing token, `404` for an unknown host, `503` when no
|
||||
failure. The default kickstart templates call it from `%post` over plain HTTP
|
||||
(the token authenticates the call; no CA trust needed at install time).
|
||||
|
||||
## The installer log relay
|
||||
|
||||
`POST /logs` exists because a host being PXE-discovered or installed is not in
|
||||
Kubernetes, so the cluster's vlagent cannot collect its logs — if an install
|
||||
fails, the only record dies with the machine. That environment also has no
|
||||
internal-CA trust and no credentials for the HTTPS-only log ingest, and bootapi
|
||||
is already the plain-HTTP broker it can reach, so bootapi forwards for it.
|
||||
|
||||
- Body: newline-delimited JSON, one log record per line (`application/x-ndjson`
|
||||
or `application/json`), capped at **1 MiB**.
|
||||
- Auth: the same `BOOTAPI_PROVISION_TOKEN` as `/provisioned`, same fail-closed
|
||||
behavior — `503` when no token (or no `BOOTAPI_VLINSERT_URL`) is configured,
|
||||
`401` on a bad/missing token.
|
||||
- Forwarded as one short-timeout POST to
|
||||
`{BOOTAPI_VLINSERT_URL}/insert/jsonline?_stream_fields=serial,phase&_msg_field=msg&_time_field=time`.
|
||||
`serial` and `phase` are constant for a run and low-cardinality, so they key
|
||||
the stream; MAC is per-NIC and goes in `extra_fields` so it stays searchable
|
||||
without multiplying streams.
|
||||
- `extra_fields` carries only what bootapi *observes* rather than what the
|
||||
client claims: the request's source IP (`src_ip`), plus the resolved NetBox
|
||||
device name (`device`) when `?mac=` resolves. Resolution is optional — a miss
|
||||
just omits the field.
|
||||
- Responses: `202` once the batch is accepted, `400` on an empty or oversized
|
||||
body. A vlinsert failure is logged and counted but **still returns `202`** —
|
||||
no retries, no buffering: a host must never block its install because the log
|
||||
sink is down.
|
||||
|
||||
## Metrics
|
||||
|
||||
All on `/metrics`, prefix `bootapi_`:
|
||||
@@ -95,6 +123,8 @@ All on `/metrics`, prefix `bootapi_`:
|
||||
- `bootapi_netbox_lookup_duration_seconds{field}` — histogram.
|
||||
- `bootapi_netbox_cache_hits_total` / `bootapi_netbox_cache_misses_total`.
|
||||
- `bootapi_provisioned_total{result}` — result = `ok|unauthorized|notfound|error|disabled`.
|
||||
- `bootapi_log_relay_total{result}` — result = `ok|error|unauthorized|disabled`.
|
||||
- `bootapi_log_relay_lines_total` — installer log lines successfully relayed.
|
||||
- `bootapi_ipxe_gated_total` — known hosts served local-boot because `pxe_enabled=false`.
|
||||
- `bootapi_template_sync_total` / `bootapi_template_sync_failures_total` / `bootapi_template_generation` — template git-sync (see [template-authoring.md](template-authoring.md)).
|
||||
- standard Go/process collectors.
|
||||
|
||||
Reference in New Issue
Block a user