unkinben
66ae5f5f3c
## Why
kea-dhcp4 crash-loops at HA hook load: kea 2.6's HA hook parses each peer url host as an IP literal and never resolves DNS, so the StatefulSet headless hostnames are rejected ("Failed to convert string to address ..."). Verified in-cluster that only an IP works (short name, FQDN both fail; `kea-dhcp4 -t` does not exercise this, which is why the v0.1.3 wait did not catch it). Pod IPs cannot be baked into the config because they change on restart and would roll-loop the StatefulSet via the config hash.
## How
- create one ClusterIP Service per HA peer, selecting the pod by its statefulset.kubernetes.io/pod-name label, with publishNotReadyAddresses so peers are routable during bootstrap
- render each HA peer url as its peer Service ClusterIP (a stable IP literal, safe in the config hash); reconcile Services before the ConfigMap and requeue until the ClusterIPs are allocated
kea-operator
A Kubernetes operator that runs ISC Kea DHCPv4
servers to replace the legacy ISC dhcpd VM used for PXE-booting physical
hosts. Modelled on the sibling bind-operator.
Namespace: dhcp-system. API group: kea.unkin.net/v1alpha1.
Custom resources
| Kind | Purpose |
|---|---|
| KeaCluster | Spawns a StatefulSet of kea-dhcp4 + kea-ctrl-agent pods, an HA control channel, and an anycast DHCP Service. Holds global config (domain, lease times, NTP, PXE option-defs). |
| KeaSubnet | One DHCPv4 subnet referenced to a KeaCluster (clusterRef). CIDR, optional pools, routers, DNS, domain, next-server. Pool-less subnets are declared so relayed requests are still serviced. |
| KeaClientClass | PXE boot class matched on client architecture (option 93), e.g. legacy → /undionly.kpxe, uefi-64 → /ipxe.efi. |
| KeaAPI | Spawns the Terraform-friendly REST API service (see below). |
The KeaCluster controller lists the matching subnets and client classes,
renders a deterministic kea-dhcp4.conf (+ kea-ctrl-agent.conf) into a
ConfigMap, and rolls the StatefulSet via a config-hash annotation stamped on the
pod template (identical trick to bind-operator). Ready pods are additionally
hot-reloaded best-effort via the kea-ctrl-agent REST control channel
(config-reload), analogous to rndc reconfig.
HA design
Mode: hot-standby, memfile lease DB (non-persistent). PXE leases are
ephemeral, low-volume, and drawn from tiny pools (.200–.220). Hot-standby
keeps a single active primary answering all DHCP with a warm secondary that
syncs leases over the HA control channel and auto-promotes on primary failure —
avoiding the split-pool double-allocation two uncoordinated servers would cause.
Load-balancing mode is also selectable via spec.ha.mode.
Kea HA needs a stable per-peer identity (each server must know which peer it
is, and peers reference each other by stable URL). That is exactly why this
operator (like bind-operator) uses a StatefulSet rather than a bare
Deployment: pods get stable ordinals (<cluster>-0, <cluster>-1) and headless
DNS, an initContainer derives this-server-name from the ordinal, and the peer
URLs are DNS names (never pod IPs, so the config hash never loops).
Startup preconditions live in a kea-init initContainer (a committed,
shellcheck-clean internal/kea/scripts/init.sh embedded via go:embed and
parameterised by env vars — no shell is interpolated in Go). It hardens the
shared run dir to 0750, substitutes this-server-name from the pod ordinal,
stages both configs into the shared emptyDir, and bounded-waits for the HA
peer DNS to resolve (kea-dhcp4 -t, failing loud after the cap so the kubelet
restarts it). The main kea-dhcp4 and kea-ctrl-agent containers then exec kea
directly with no wrapper shell.
Anycast routing caveat (deployment follow-up)
The DHCP Service is a LoadBalancer intended to receive a PureLB anycast IP;
routers relay unicast to it. Under hot-standby the standby does not answer while
the primary is up, so the anycast VIP should be pinned to the active peer. Pin
it operationally (PureLB local traffic policy / active-peer endpoint selection)
before production cutover. See follow-ups.
REST API (KeaAPI)
A separate binary (cmd/api) spawned by the KeaAPI CRD. It is a Terraform-
friendly CRUD facade over the KeaSubnet / KeaClientClass CRs — an
alternative to argocd-managed CRs. Contract mirrors encapi:
PUT/GET/DELETE /api/v1/subnets/{name}and.../clientclasses/{name}, plus list endpoints. Stable client-supplied IDs (the CR name, from the URL path).PUTis an idempotent upsert returning the canonical object (200);DELETEreturns 204;GETon a missing resource returns 404 (drives provider drift handling). Error bodies are{"error": "..."}.- Auth: bearer token (constant-time compare, fail-closed) from
KEA_API_TOKEN, sourced from a k8s Secret the operator generates if absent (or pre-seeds).
The Terraform provider is a separate queued task; the API is designed so wrapping it is trivial (full-object PUT, canonical GET, 404 semantics).
Build & release
make test—go test -raceover./api/... ./internal/....make generate— regenerates deepcopy, CRDs, RBAC, and theinstall.yamlbundle (all committed).make patch|minor|major— tagsvX.Y.Zand pushes; the tag triggers the.woodpecker/docker.yamlimage builds.
Images: kea-operator and kea-api are distroless Go binaries. The Kea
workload image (Dockerfile.kea) is built from AlmaLinux (reachable via the
artifactapi dockerhub remote) + EPEL kea packages.
Follow-ups (not in this repo yet)
- argocd-apps: a
kea-operator-ciServiceAccount for the woodpecker CI steps, and the operator deployment manifests. - artifactapi: the Kea image build pulls kea RPMs from EPEL; if the CI build
network cannot reach EPEL, add an artifactapi rpm remote (or vendor kea via
rpmbuilder) and point
Dockerfile.keaat it. - terraform-provider-kea: wrap the KeaAPI REST contract.
- Vault: issue ephemeral API bearer tokens instead of a static k8s Secret.
- Anycast: pin the anycast VIP to the active HA peer (PureLB tuning).