unkinben
9b3fa83dac
Container entrypoints were rendered as `fmt.Sprintf` shell strings in Go, so every entrypoint fix needed a full operator release, nothing was shellcheckable, and the escaping was a hazard. How: - Add a `kea-init` initContainer that finalises the per-pod config and bounded-waits for HA peer DNS, replacing the in-entrypoint retry. It hardens the shared run dir to 0750, substitutes `this-server-name` from the pod ordinal (`POD_NAME` via the downward API), stages both configs into the shared emptyDir, and gates on `kea-dhcp4 -t` (60x2s) — failing loud after the cap so the kubelet restarts it instead of starting a doomed server. - Run the main kea-dhcp4 / kea-ctrl-agent containers with kea exec'd directly, dropping both wrapper shells. - Replace the two `fmt.Sprintf` entrypoints with a single committed `internal/kea/scripts/init.sh` embedded via `go:embed` and parameterised entirely by env vars — no Go string interpolation. - Add a shellcheck step to the pre-commit pipeline. Test: - Assert the pod shape: one kea-init initContainer, POD_NAME from the downward API, main containers exec kea directly, and the ConfigMap carries init.sh (not the old per-container entrypoints). - Assert init.sh hardens the socket dir, gates on `kea-dhcp4 -t`, fails loud after the cap, and is free of fmt verbs. - shellcheck the embedded script.
97 lines
5.2 KiB
Markdown
97 lines
5.2 KiB
Markdown
# kea-operator
|
||
|
||
A Kubernetes operator that runs [ISC Kea](https://www.isc.org/kea/) DHCPv4
|
||
servers to replace the legacy ISC `dhcpd` VM used for PXE-booting physical
|
||
hosts. Modelled on the sibling `bind-operator`.
|
||
|
||
Namespace: `dhcp-system`. API group: `kea.unkin.net/v1alpha1`.
|
||
|
||
## Custom resources
|
||
|
||
| Kind | Purpose |
|
||
|------|---------|
|
||
| **KeaCluster** | Spawns a StatefulSet of `kea-dhcp4` + `kea-ctrl-agent` pods, an HA control channel, and an anycast DHCP `Service`. Holds global config (domain, lease times, NTP, PXE option-defs). |
|
||
| **KeaSubnet** | One DHCPv4 subnet referenced to a KeaCluster (`clusterRef`). CIDR, optional pools, routers, DNS, domain, next-server. Pool-less subnets are declared so relayed requests are still serviced. |
|
||
| **KeaClientClass** | PXE boot class matched on client architecture (option 93), e.g. `legacy` → `/undionly.kpxe`, `uefi-64` → `/ipxe.efi`. |
|
||
| **KeaAPI** | Spawns the Terraform-friendly REST API service (see below). |
|
||
|
||
The **KeaCluster** controller lists the matching subnets and client classes,
|
||
renders a deterministic `kea-dhcp4.conf` (+ `kea-ctrl-agent.conf`) into a
|
||
ConfigMap, and rolls the StatefulSet via a config-hash annotation stamped on the
|
||
pod template (identical trick to bind-operator). Ready pods are additionally
|
||
hot-reloaded best-effort via the kea-ctrl-agent REST control channel
|
||
(`config-reload`), analogous to `rndc reconfig`.
|
||
|
||
## HA design
|
||
|
||
**Mode: hot-standby, memfile lease DB (non-persistent).** PXE leases are
|
||
ephemeral, low-volume, and drawn from tiny pools (`.200–.220`). Hot-standby
|
||
keeps a single active primary answering all DHCP with a warm secondary that
|
||
syncs leases over the HA control channel and auto-promotes on primary failure —
|
||
avoiding the split-pool double-allocation two uncoordinated servers would cause.
|
||
Load-balancing mode is also selectable via `spec.ha.mode`.
|
||
|
||
Kea HA needs a **stable per-peer identity** (each server must know which peer it
|
||
is, and peers reference each other by stable URL). That is exactly why this
|
||
operator (like bind-operator) uses a **StatefulSet** rather than a bare
|
||
Deployment: pods get stable ordinals (`<cluster>-0`, `<cluster>-1`) and headless
|
||
DNS, an initContainer derives `this-server-name` from the ordinal, and the peer
|
||
URLs are DNS names (never pod IPs, so the config hash never loops).
|
||
|
||
Startup preconditions live in a `kea-init` initContainer (a committed,
|
||
shellcheck-clean `internal/kea/scripts/init.sh` embedded via `go:embed` and
|
||
parameterised by env vars — no shell is interpolated in Go). It hardens the
|
||
shared run dir to `0750`, substitutes `this-server-name` from the pod ordinal,
|
||
stages both configs into the shared `emptyDir`, and bounded-waits for the HA
|
||
peer DNS to resolve (`kea-dhcp4 -t`, failing loud after the cap so the kubelet
|
||
restarts it). The main `kea-dhcp4` and `kea-ctrl-agent` containers then exec kea
|
||
directly with no wrapper shell.
|
||
|
||
### Anycast routing caveat (deployment follow-up)
|
||
|
||
The DHCP `Service` is a `LoadBalancer` intended to receive a PureLB anycast IP;
|
||
routers relay unicast to it. Under hot-standby the standby does not answer while
|
||
the primary is up, so the anycast VIP should be pinned to the active peer. Pin
|
||
it operationally (PureLB local traffic policy / active-peer endpoint selection)
|
||
before production cutover. See follow-ups.
|
||
|
||
## REST API (KeaAPI)
|
||
|
||
A separate binary (`cmd/api`) spawned by the `KeaAPI` CRD. It is a Terraform-
|
||
friendly CRUD facade over the `KeaSubnet` / `KeaClientClass` CRs — an
|
||
alternative to argocd-managed CRs. Contract mirrors `encapi`:
|
||
|
||
- `PUT/GET/DELETE /api/v1/subnets/{name}` and `.../clientclasses/{name}`, plus
|
||
list endpoints. Stable client-supplied IDs (the CR name, from the URL path).
|
||
- `PUT` is an idempotent upsert returning the canonical object (200); `DELETE`
|
||
returns 204; `GET` on a missing resource returns 404 (drives provider drift
|
||
handling). Error bodies are `{"error": "..."}`.
|
||
- Auth: bearer token (constant-time compare, fail-closed) from `KEA_API_TOKEN`,
|
||
sourced from a k8s Secret the operator generates if absent (or pre-seeds).
|
||
|
||
The Terraform provider is a **separate queued task**; the API is designed so
|
||
wrapping it is trivial (full-object PUT, canonical GET, 404 semantics).
|
||
|
||
## Build & release
|
||
|
||
- `make test` — `go test -race` over `./api/... ./internal/...`.
|
||
- `make generate` — regenerates deepcopy, CRDs, RBAC, and the `install.yaml`
|
||
bundle (all committed).
|
||
- `make patch|minor|major` — tags `vX.Y.Z` and pushes; the tag triggers the
|
||
`.woodpecker/docker.yaml` image builds.
|
||
|
||
Images: `kea-operator` and `kea-api` are distroless Go binaries. The Kea
|
||
workload image (`Dockerfile.kea`) is built from AlmaLinux (reachable via the
|
||
artifactapi dockerhub remote) + EPEL kea packages.
|
||
|
||
## Follow-ups (not in this repo yet)
|
||
|
||
- **argocd-apps**: a `kea-operator-ci` ServiceAccount for the woodpecker CI
|
||
steps, and the operator deployment manifests.
|
||
- **artifactapi**: the Kea image build pulls kea RPMs from EPEL; if the CI build
|
||
network cannot reach EPEL, add an artifactapi rpm remote (or vendor kea via
|
||
rpmbuilder) and point `Dockerfile.kea` at it.
|
||
- **terraform-provider-kea**: wrap the KeaAPI REST contract.
|
||
- **Vault**: issue ephemeral API bearer tokens instead of a static k8s Secret.
|
||
- **Anycast**: pin the anycast VIP to the active HA peer (PureLB tuning).
|