From 7a1e8351a7200a16fbd6b687adb31b0b61be6e6c Mon Sep 17 00:00:00 2001 From: Unkin Agent Date: Thu, 13 Aug 2026 23:19:54 +1000 Subject: [PATCH] ghp: serve plain HTTP behind the gateway (fix redirect loop) (#361) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://ghp.unkin.net/ 308-loops onto itself. The traefik gateway terminates TLS and forwards cleartext to the ghp Service port 80 -> container :8080, but :8080 was `GHP_SERVER_HTTP_LISTEN` — ghp's http->https **308 redirect** listener. So ghp bounced every request back to https, the gateway re-forwarded it to :8080, and it looped forever. ## Root cause (confirmed against ghp source) `internal/server/server.go` `Run()` picks the serving mode: ```go hasTLS := s.cfg.Server.HTTPSListen != "" || (systemd socket + certs) if hasTLS { return s.serveTLS(...) } // app on HTTPSListen (TLS); HTTPListen = 308 redirect; Listen IGNORED return s.servePlain(...) // app (full handler: mgmt UI + API) on Listen, cleartext ``` It is **strictly either/or**. `serveTLS` serves the app on `HTTPSListen` and gives `HTTPListen` only `httpsRedirectHandler()` (`redirect.go`: `http.StatusPermanentRedirect` = 308). `servePlain` serves the same full handler on `Listen` in cleartext (`createListener()` uses `cfg.Server.Listen`). The mgmt UI is the same `handler` in both modes, so it IS served on the plain `Listen` port. Behind a TLS-terminating gateway that forwards cleartext to :8080, ghp therefore has to run in **plain mode**. Keeping `GHP_SERVER_HTTPS_LISTEN` would keep `hasTLS` true, leave `GHP_SERVER_LISTEN` ignored, and nothing would serve cleartext on :8080. ## Change - **configmap**: drop `GHP_SERVER_HTTPS_LISTEN` and `GHP_SERVER_HTTP_LISTEN`; set `GHP_SERVER_LISTEN: ":8080"` so :8080 SERVES the app; add `GHP_SERVER_TRUST_PROXY_HEADERS: "true"` so ghp trusts the gateway's `X-Forwarded-*`/`Forwarded` for scheme/host (`GHP_SERVER_BASE_URL` already set). - **deployment + vmservicescrape**: the metrics server only wraps TLS when `hasTLS` is true (`Run()` gates `loadTLSConfig` on `hasTLS`); in plain mode it is cleartext, so the `/metrics` liveness/readiness probes and the VMServiceScrape switch from HTTPS/https to HTTP/http. Service, HTTPRoute and Gateway are unchanged. configmap+deployment carry the stakater reloader annotation, so pods roll on the change. ## Deviation from the brief The brief said to keep `GHP_SERVER_HTTPS_LISTEN: ":8443"`. Source shows that is incompatible with serving cleartext on :8080 (the two modes are mutually exclusive), so this drops it. The unused `GHP_TLS_CERT_FILE`/`KEY_FILE`, the `tls` volume, and containerPort 8443 are left in place (harmless) for an easy revert to TLS mode. The alternative — gateway -> Service 443 -> :8443 with a BackendTLSPolicy — is the bigger change flagged in the brief and is NOT taken here. Validated: `kustomize build apps/overlays/au-syd1/ghp` clean, kubeconform 0 invalid/0 errors, pre-commit clean. Not applied. --------- Co-authored-by: unkin-agent Reviewed-on: https://git.unkin.net/unkin/argocd-apps/pulls/361 Co-authored-by: Unkin Agent Co-committed-by: Unkin Agent --- apps/base/ghp/configmap.yaml | 16 ++++++++++++++-- apps/base/ghp/deployment.yaml | 10 ++++++++-- apps/base/ghp/vmservicescrape.yaml | 8 +++----- 3 files changed, 25 insertions(+), 9 deletions(-) diff --git a/apps/base/ghp/configmap.yaml b/apps/base/ghp/configmap.yaml index ca0ae45..f9c30b1 100644 --- a/apps/base/ghp/configmap.yaml +++ b/apps/base/ghp/configmap.yaml @@ -6,9 +6,21 @@ metadata: namespace: ghp data: GHP_DATABASE_DRIVER: postgres + # ghp runs in plain-HTTP reverse-proxy mode: the traefik gateway terminates TLS + # for ghp.unkin.net and forwards cleartext to container :8080, where ghp SERVES + # the full app (mgmt UI + API) via GHP_SERVER_LISTEN. + # + # Do NOT set GHP_SERVER_HTTPS_LISTEN here. Any non-empty https_listen flips ghp + # into TLS-only mode (server.go Run(): hasTLS -> serveTLS): the app is served on + # :8443 and http_listen becomes a 308 http->https REDIRECT listener, while + # GHP_SERVER_LISTEN is ignored entirely. The gateway forwarding cleartext into + # that redirect listener on :8080 was the ghp.unkin.net -> ghp.unkin.net 308 loop. + # # nonroot cannot bind <1024; listen high and remap in the Service. - GHP_SERVER_HTTPS_LISTEN: ":8443" - GHP_SERVER_HTTP_LISTEN: ":8080" + GHP_SERVER_LISTEN: ":8080" + # Behind the TLS-terminating gateway: trust its X-Forwarded-* / Forwarded headers + # for scheme/host when generating absolute URLs (GHP_SERVER_BASE_URL is also set). + GHP_SERVER_TRUST_PROXY_HEADERS: "true" GHP_METRICS_LISTEN: ":9136" GHP_METRICS_ENABLED: "true" GHP_SERVER_BASE_URL: https://ghp.unkin.net diff --git a/apps/base/ghp/deployment.yaml b/apps/base/ghp/deployment.yaml index 1a7f5b2..e4619b7 100644 --- a/apps/base/ghp/deployment.yaml +++ b/apps/base/ghp/deployment.yaml @@ -102,7 +102,10 @@ spec: httpGet: path: /metrics port: metrics - scheme: HTTPS + # Plain HTTP: ghp only serves metrics over TLS in TLS mode + # (hasTLS). In reverse-proxy/plain mode the metrics server is + # cleartext, so probe with HTTP. + scheme: HTTP initialDelaySeconds: 30 periodSeconds: 30 successThreshold: 1 @@ -112,7 +115,10 @@ spec: httpGet: path: /metrics port: metrics - scheme: HTTPS + # Plain HTTP: ghp only serves metrics over TLS in TLS mode + # (hasTLS). In reverse-proxy/plain mode the metrics server is + # cleartext, so probe with HTTP. + scheme: HTTP initialDelaySeconds: 10 periodSeconds: 5 successThreshold: 1 diff --git a/apps/base/ghp/vmservicescrape.yaml b/apps/base/ghp/vmservicescrape.yaml index e5ca485..ea6a7a8 100644 --- a/apps/base/ghp/vmservicescrape.yaml +++ b/apps/base/ghp/vmservicescrape.yaml @@ -16,8 +16,6 @@ spec: endpoints: - port: metrics path: /metrics - scheme: https - # ghp serves metrics over TLS with an internal-CA cert; skip verification - # since the scrape targets a pod IP the cert SANs do not cover. - tlsConfig: - insecureSkipVerify: true + # ghp runs in plain reverse-proxy mode (no GHP_SERVER_HTTPS_LISTEN), so the + # metrics server is cleartext HTTP rather than TLS. Scrape over http. + scheme: http