Move the au-syd1 haproxy edge into Kubernetes #485

Merged

benvin merged 5 commits from benvin/haproxy-app into main 2026-09-26 18:23:00 +10:00

Conversation 1 Commits 5 Files Changed 11

+720

unkin-agent commented 2026-09-26 14:15:30 +10:00

Member

Copy Link

Copy Source

The au-syd1 edge proxy runs on a hand-managed LXD container outside the cluster, with no HA and no shared config source.

• Add apps/base/haproxy/: 3 replicas behind the DMZ LoadBalancer 198.18.199.1, config from a ConfigMap, wildcard certs from reflected secrets.
• Keep source IPs via externalTrafficPolicy: Local; sessionAffinity: ClientIP stands in for the stick-table peers a Deployment cannot name.
• Drain on shutdown: hard-stop-after 2m, a preStop SIGUSR1 soft-stop, 150s grace.
• Bind the stats listener to 127.0.0.1 so it is port-forward only.
• Register the app in the platform project and ApplicationSet.

The au-syd1 edge proxy runs on a hand-managed LXD container outside the cluster, with no HA and no shared config source. - Add `apps/base/haproxy/`: 3 replicas behind the DMZ LoadBalancer 198.18.199.1, config from a ConfigMap, wildcard certs from reflected secrets. - Keep source IPs via `externalTrafficPolicy: Local`; `sessionAffinity: ClientIP` stands in for the stick-table peers a Deployment cannot name. - Drain on shutdown: `hard-stop-after 2m`, a preStop SIGUSR1 soft-stop, 150s grace. - Bind the stats listener to 127.0.0.1 so it is port-forward only. - Register the app in the platform project and ApplicationSet.

unkin-agent commented 2026-09-26 14:27:31 +10:00

Author

Member

Copy Link

Copy Source

• apps/base/haproxy/configmap.yaml:187,199 — be_ausyd1pve_api and be_ausyd1pve_web have zero server lines. On the VM these are filled by exported Haproxy::Balancermember from profiles::proxmox::weblb (port 8006, cookie <hostname> ssl verify none check inter 2s rise 3 fall 2; api has no cookie). As written au-syd1-pve.main.unkin.net and au-syd1-pve-api.main.unkin.net 503 → add the pve node servers.
• apps/base/haproxy/configmap.yaml:306 — be_letsencrypt is pinned to one host 198.18.25.3:8888, but certbot::init includes certbot::haproxy so every certbot client in au-syd1-prod exports an :8888 member. HTTP-01 renewals for the remaining VM certs break when that single host is down → list all certbot members or drop the backend if VM certbot is going away.
• apps/base/haproxy/configmap.yaml:480-484 — listen stats binds 0.0.0.0:9090 with stats auth admin:admin committed in a ConfigMap; Puppet bound it to the node IP inside the DMZ, in-pod it is reachable from every pod in the cluster → drop the listener (metrics already on :8405) or move the credential to a Secret.
• apps/base/haproxy/configmap.yaml:120-133 — haproxy -c on 3.2.24 emits "acl 'acl_X' will never match because it only involves keywords that are incompatible with 'frontend http-response header rule'" for all 14 acl_*, so no X-Frame-Options header is ever set. Carried over from Puppet, but silently dead → set the host into a var in the request phase and test the var, or emit the header unconditionally.
• apps/base/haproxy/deployment.yaml — no preStop/terminationGracePeriodSeconds. HAProxy treats SIGTERM as an immediate stop (soft stop is SIGUSR1), so every ConfigMap/cert reload cuts in-flight IMAP/SMTP sessions on all 3 replicas → add a preStop that sends SIGUSR1 and raise the grace period past timeout client 5m, or set hard-stop-after.
• nit: apps/base/haproxy/configmap.yaml:159 — fe_metrics silently loses the Puppet ssl crt /etc/pki/tls/vault/certificate.pem bind; the exporter is now plaintext and unauthenticated on the pod network. Fine for a VMPodScrape, but it is not in the PR description.
• nit: server addresses are frozen in the ConfigMap where Puppet re-rendered them from exported facts on every run; a VM renumber now silently 503s with no signal outside the stats page.

- `apps/base/haproxy/configmap.yaml:187,199` — `be_ausyd1pve_api` and `be_ausyd1pve_web` have zero `server` lines. On the VM these are filled by exported `Haproxy::Balancermember` from `profiles::proxmox::weblb` (port 8006, `cookie <hostname> ssl verify none check inter 2s rise 3 fall 2`; api has no cookie). As written `au-syd1-pve.main.unkin.net` and `au-syd1-pve-api.main.unkin.net` 503 → add the pve node servers. - `apps/base/haproxy/configmap.yaml:306` — `be_letsencrypt` is pinned to one host `198.18.25.3:8888`, but `certbot::init` includes `certbot::haproxy` so every certbot client in au-syd1-prod exports an `:8888` member. HTTP-01 renewals for the remaining VM certs break when that single host is down → list all certbot members or drop the backend if VM certbot is going away. - `apps/base/haproxy/configmap.yaml:480-484` — `listen stats` binds `0.0.0.0:9090` with `stats auth admin:admin` committed in a ConfigMap; Puppet bound it to the node IP inside the DMZ, in-pod it is reachable from every pod in the cluster → drop the listener (metrics already on :8405) or move the credential to a Secret. - `apps/base/haproxy/configmap.yaml:120-133` — `haproxy -c` on 3.2.24 emits "acl 'acl_X' will never match because it only involves keywords that are incompatible with 'frontend http-response header rule'" for all 14 `acl_*`, so no `X-Frame-Options` header is ever set. Carried over from Puppet, but silently dead → set the host into a var in the request phase and test the var, or emit the header unconditionally. - `apps/base/haproxy/deployment.yaml` — no `preStop`/`terminationGracePeriodSeconds`. HAProxy treats SIGTERM as an immediate stop (soft stop is SIGUSR1), so every ConfigMap/cert reload cuts in-flight IMAP/SMTP sessions on all 3 replicas → add a `preStop` that sends SIGUSR1 and raise the grace period past `timeout client 5m`, or set `hard-stop-after`. - nit: `apps/base/haproxy/configmap.yaml:159` — `fe_metrics` silently loses the Puppet `ssl crt /etc/pki/tls/vault/certificate.pem` bind; the exporter is now plaintext and unauthenticated on the pod network. Fine for a VMPodScrape, but it is not in the PR description. - nit: server addresses are frozen in the ConfigMap where Puppet re-rendered them from exported facts on every run; a VM renumber now silently 503s with no signal outside the stats page.

benvin changed target branch from benvin/haproxy-wildcard-certs to main 2026-09-26 16:38:32 +10:00

benvin added 3 commits 2026-09-26 16:38:33 +10:00

add *.main.unkin.net and *.ceph.unkin.net wildcard certificates ae3edd9db5

move the au-syd1 haproxy edge into kubernetes 4fcee63925

drain haproxy on shutdown and keep the stats listener on loopback 2cbac81015

unkin-agent added 1 commit 2026-09-26 16:42:06 +10:00

merge main into haproxy branch 02ed176563

unkin-agent added 1 commit 2026-09-26 17:45:43 +10:00

drop proxmox routing be5270e7a9

benvin merged commit 4190785389 into main 2026-09-26 18:23:00 +10:00

benvin deleted branch benvin/haproxy-app 2026-09-26 18:23:01 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/argocd-apps#485