Compare commits

..

11 Commits

Author SHA1 Message Date
unkin-agent 6e0ac5fdbf Set VictoriaLogs cluster image version to v1.52.0
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
2026-09-26 23:43:56 +10:00
unkin-agent 74c092040a Pin VictoriaLogs cluster image version to v1.50.0
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
2026-09-26 23:39:52 +10:00
unkin-agent acf074ca3c Size vlstorage at 250Gi per node on cephrbd-fast-delete
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
2026-09-26 23:35:28 +10:00
unkin-agent f193c7e75a Size vlstorage for a 180d log store
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
- raise the vlstorage PVCs to 500Gi and the disk cap to 440GiB per node
- move vlstorage onto cephrbd-fast-retain
2026-09-26 19:02:34 +10:00
unkin-agent efdfd3eb4b Add a VictoriaLogs cluster and point logs-ingest at it
ci/woodpecker/pr/vector-test Pipeline was successful
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful
- add VLCluster logs (2 vlinsert, 2 vlselect, 3 vlstorage, 180d)
- repoint the logs-ingest HTTPRoute at vlinsert-logs:9481
- add a VictoriaLogs Grafana datasource and its plugin
2026-09-26 18:37:47 +10:00
unkin-agent 4190785389 Move the au-syd1 haproxy edge into Kubernetes (#485)
The au-syd1 edge proxy runs on a hand-managed LXD container outside the cluster, with no HA and no shared config source.

- Add `apps/base/haproxy/`: 3 replicas behind the DMZ LoadBalancer 198.18.199.1, config from a ConfigMap, wildcard certs from reflected secrets.
- Keep source IPs via `externalTrafficPolicy: Local`; `sessionAffinity: ClientIP` stands in for the stick-table peers a Deployment cannot name.
- Drain on shutdown: `hard-stop-after 2m`, a preStop SIGUSR1 soft-stop, 150s grace.
- Bind the stats listener to 127.0.0.1 so it is port-forward only.
- Register the app in the platform project and ApplicationSet.

Reviewed-on: #485
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-26 18:22:57 +10:00
unkin-agent 812a9a2f2b Publish real delegation records for acme.unkin.net (#486)
The acme.unkin.net zone still serves only the bind-operator seed apex: NS ns1.acme.unkin.net glued to A 10.42.6.38, a pod IP no pod holds. The parent delegates to acme-ns1.unkin.net, but public resolvers have already promoted the child NS RRset, so when the cached address expires DNS-01 fails for every unkin.net wildcard at once.

- Add apex NS acme-ns1.unkin.net., matching the parent delegation (out of zone, no glue needed).
- Point ns1.acme.unkin.net at 103.216.191.185 so resolvers holding the seeded NS name still reach the zone.
- The operator seed placeholder itself is tracked separately in bind-operator.

---------

Co-authored-by: unkin-agent <agent@unkin.net>
Reviewed-on: #486
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
2026-09-26 16:42:02 +10:00
unkin-agent f37749523d Add *.main and *.ceph wildcard certificates for haproxy (#484)
The haproxy edge terminates TLS for hosts under `main.unkin.net` and `ceph.unkin.net`, which the single `*.unkin.net` wildcard does not cover.

- Add cert-manager Certificates for both wildcards from the `letsencrypt` ClusterIssuer.
- Reflect the minted secrets into the `haproxy` namespace.

Needs these records in the public unkin.net zone first:
`_acme-challenge.main.unkin.net. CNAME _acme-challenge.main.acme.unkin.net.`
`_acme-challenge.ceph.unkin.net. CNAME _acme-challenge.ceph.acme.unkin.net.`

Reviewed-on: #484
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-26 16:38:31 +10:00
unkin-agent fe51aa07be Give the puppetserver compilers the Vault cert helpers (#482)
profiles::pki::vault and profiles::ssh::sign shell out to
/usr/local/bin/certmanager and /usr/local/bin/sshsignhost from generate()
during catalog compilation. Neither binary exists in the compiler image, so
every node using them fails to compile.

- install certmanager v0.2.0 and sshsignhost v0.1.0 onto the shared bin volume with sha256 verification
- wrap both at /usr/local/bin from a pre-default entrypoint hook, failing startup loudly if either is missing
- mount read-only Vault configs for both: kubernetes auth on k8s/au/syd1, internal CA verified rather than skipped

Reviewed-on: #482
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-24 21:15:09 +10:00
unkin-agent cdaab736b5 Bump jellyfin-ha to v0.4.0 (#483)
The deployed v0.3.3 build returns 500 from /Shows/NextUp on PostgreSQL, breaking the home screen, and lets replicas diverge: library-visibility and shared-config changes never propagate, user data (resume, played state, favourites, ratings) is overwritten between pods, and eight scheduled tasks run on every replica instead of only the scan leader. v0.4.0 carries the fixes.

- Pin cheeztv and fafflix to jellyfin-ha:v0.4.0.

No config change needed: cross-pod invalidation reuses the transcode-store Redis connection string both apps already set.

Reviewed-on: #483
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-22 22:57:06 +10:00
unkin-agent b31517e6d9 Merge pull request #481 from benvin/jellyfin-sso-valkey-state
Roll jellyfin-ha to v0.3.3 and drop Service session affinity

Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-20 14:02:48 +10:00
29 changed files with 1008 additions and 14 deletions
@@ -7,4 +7,5 @@ resources:
- cluster.yaml
- tsigkey.yaml
- zones.yaml
- records.yaml
- agent-dns-rolebinding.yaml
+36
View File
@@ -0,0 +1,36 @@
# Authoritative delegation records for acme.unkin.net. Without these the zone
# only holds the operator's seed apex (NS ns1.acme.unkin.net glued to the
# primary pod IP), which is unroutable off-cluster and goes stale on
# reschedule. DNSRecords must live in the same namespace as their BindZone.
---
apiVersion: bind.unkin.net/v1alpha1
kind: DNSRecord
metadata:
name: acme-apex-ns
namespace: bind-external
spec:
zoneRef: acme-unkin-net
# "@" is the zone apex.
name: "@"
type: NS
ttl: 3600
values:
# Matches the parent delegation in Google Cloud DNS. Out of zone, so the
# child needs no glue of its own.
- acme-ns1.unkin.net.
---
apiVersion: bind.unkin.net/v1alpha1
kind: DNSRecord
metadata:
name: acme-ns1-a
namespace: bind-external
spec:
zoneRef: acme-unkin-net
name: ns1
type: A
ttl: 3600
values:
# Public address of this cluster's external BIND, same target as
# acme-ns1.unkin.net. Resolvers that cached the seeded ns1.acme.unkin.net
# NS name must still reach the zone.
- 103.216.191.185
@@ -0,0 +1,26 @@
---
# Let's Encrypt *.ceph.unkin.net wildcard for the haproxy edge (ceph dashboard).
# DNS-01 needs the delegated _acme-challenge.ceph.unkin.net CNAME in the public
# unkin.net zone.
# _acme-challenge.ceph.unkin.net. CNAME _acme-challenge.ceph.acme.unkin.net.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard-ceph-unkin-net
namespace: cert-manager
spec:
secretName: wildcard-ceph-unkin-net-tls
secretTemplate:
annotations:
reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "haproxy"
reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "haproxy"
privateKey:
size: 4096
dnsNames:
- "*.ceph.unkin.net"
issuerRef:
name: letsencrypt
kind: ClusterIssuer
group: cert-manager.io
@@ -0,0 +1,26 @@
---
# Let's Encrypt *.main.unkin.net wildcard for the haproxy edge (pve, arr stack,
# jellyfin, stalwart webadmin/autoconfig). DNS-01 needs the delegated
# _acme-challenge.main.unkin.net CNAME in the public unkin.net zone.
# _acme-challenge.main.unkin.net. CNAME _acme-challenge.main.acme.unkin.net.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: wildcard-main-unkin-net
namespace: cert-manager
spec:
secretName: wildcard-main-unkin-net-tls
secretTemplate:
annotations:
reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "haproxy"
reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "haproxy"
privateKey:
size: 4096
dnsNames:
- "*.main.unkin.net"
issuerRef:
name: letsencrypt
kind: ClusterIssuer
group: cert-manager.io
@@ -14,9 +14,9 @@ spec:
secretTemplate:
annotations:
reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "cheeztv,arrstack,authentik,gitea,watchstate,mediamark,repospawner"
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "cheeztv,arrstack,authentik,gitea,watchstate,mediamark,repospawner,haproxy"
reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "cheeztv,arrstack,authentik,gitea,watchstate,mediamark,repospawner"
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "cheeztv,arrstack,authentik,gitea,watchstate,mediamark,repospawner,haproxy"
privateKey:
size: 4096
dnsNames:
@@ -12,3 +12,5 @@ resources:
- clusterissuer_letsencrypt.yaml
- clusterissuer_letsencrypt-staging.yaml
- certificate_wildcard-unkin-net.yaml
- certificate_wildcard-main-unkin-net.yaml
- certificate_wildcard-ceph-unkin-net.yaml
+1 -1
View File
@@ -164,7 +164,7 @@ spec:
readOnly: true
containers:
- name: cheeztv
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.3.3
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.4.0
imagePullPolicy: IfNotPresent
ports:
- name: http
+1 -1
View File
@@ -164,7 +164,7 @@ spec:
readOnly: true
containers:
- name: fafflix
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.3.3
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.4.0
imagePullPolicy: IfNotPresent
ports:
- name: http
+19
View File
@@ -20,3 +20,22 @@ spec:
jsonData:
timeInterval: "15s"
httpMethod: "POST"
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
name: victorialogs
namespace: grafana
spec:
instanceSelector:
matchLabels:
dashboards: "grafana"
plugins:
- name: victoriametrics-logs-datasource
version: 0.32.0
datasource:
name: "VictoriaLogs"
type: "victoriametrics-logs-datasource"
uid: "victorialogs"
access: "proxy"
url: "http://vlselect-logs.logging.svc.cluster.local:9471"
+452
View File
@@ -0,0 +1,452 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: haproxy-config
namespace: haproxy
data:
certificate.list: |
# First entry is the default cert for non-matching SNI.
/etc/haproxy/certs/unkin-net/tls.crt
/etc/haproxy/certs/main-unkin-net/tls.crt
/etc/haproxy/certs/ceph-unkin-net/tls.crt
fe_http.map: |
sonarr.main.unkin.net be_sonarr
radarr.main.unkin.net be_radarr
lidarr.main.unkin.net be_lidarr
readarr.main.unkin.net be_readarr
prowlarr.main.unkin.net be_prowlarr
nzbget.main.unkin.net be_nzbget
jellyfin.main.unkin.net be_jellyfin
fafflix.unkin.net be_jellyfin
git.unkin.net be_gitea
grafana.unkin.net be_grafana
dashboard.ceph.unkin.net be_ceph_dashboard
mail-webadmin.main.unkin.net be_stalwart_webadmin
autoconfig.main.unkin.net be_stalwart_webadmin
autodiscovery.main.unkin.net be_stalwart_webadmin
auth.unkin.net be_k8s_kanidm
fe_https.map: |
sonarr.main.unkin.net be_sonarr
radarr.main.unkin.net be_radarr
lidarr.main.unkin.net be_lidarr
readarr.main.unkin.net be_readarr
prowlarr.main.unkin.net be_prowlarr
nzbget.main.unkin.net be_nzbget
jellyfin.main.unkin.net be_jellyfin
fafflix.unkin.net be_jellyfin
git.unkin.net be_gitea
grafana.unkin.net be_grafana
dashboard.ceph.unkin.net be_ceph_dashboard
mail-webadmin.main.unkin.net be_stalwart_webadmin
autoconfig.main.unkin.net be_stalwart_webadmin
autodiscovery.main.unkin.net be_stalwart_webadmin
auth.unkin.net be_k8s_kanidm
haproxy.cfg: |
global
log stdout format raw local0
log stdout format raw local1 notice
maxconn 4000
hard-stop-after 2m
ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3
ssl-default-server-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
ssl-default-server-options no-sslv3
stats timeout 30s
stats socket /var/lib/haproxy/stats
stats socket /var/lib/haproxy/admin.sock mode 660 level admin
tune.ssl.default-dh-param 2048
defaults
log global
maxconn 5000
mode http
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
stats enable
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 5m
timeout server 5m
timeout http-keep-alive 10s
timeout check 10s
frontend fe_http
bind 0.0.0.0:80
mode http
description Global HTTP Frontend
acl acl-letsencrypt path_beg /.well-known/acme-challenge/
http-request set-header X-Forwarded-Proto https
http-request set-header X-Real-IP %[src]
use_backend be_letsencrypt if acl-letsencrypt
use_backend %[req.hdr(host),lower,map(/usr/local/etc/haproxy/fe_http.map,be_default)]
frontend fe_https
bind 0.0.0.0:443 ssl crt-list /usr/local/etc/haproxy/certificate.list ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH force-tlsv12
mode http
description Global HTTPS Frontend
acl acl-letsencrypt path_beg /.well-known/acme-challenge/
acl acl_sonarr req.hdr(host) -i sonarr.main.unkin.net
acl acl_radarr req.hdr(host) -i radarr.main.unkin.net
acl acl_lidarr req.hdr(host) -i lidarr.main.unkin.net
acl acl_readarr req.hdr(host) -i readarr.main.unkin.net
acl acl_prowlarr req.hdr(host) -i prowlarr.main.unkin.net
acl acl_nzbget req.hdr(host) -i nzbget.main.unkin.net
acl acl_jellyfin req.hdr(host) -i jellyfin.main.unkin.net
acl acl_fafflix req.hdr(host) -i fafflix.unkin.net
acl acl_gitea req.hdr(host) -i git.unkin.net
acl acl_grafana req.hdr(host) -i grafana.unkin.net
acl acl_ceph_dashboard req.hdr(host) -i dashboard.ceph.unkin.net
acl acl_stalwart_webadmin req.hdr(host) -i mail-webadmin.main.unkin.net
acl acl_stalwart_webadmin req.hdr(host) -i autoconfig.main.unkin.net
acl acl_stalwart_webadmin req.hdr(host) -i autodiscovery.main.unkin.net
acl acl_kanidm req.hdr(host) -i auth.unkin.net
http-request set-header X-Forwarded-Proto https
http-request set-header X-Real-IP %[src]
http-response set-header X-Frame-Options DENY if acl_sonarr
http-response set-header X-Frame-Options DENY if acl_radarr
http-response set-header X-Frame-Options DENY if acl_lidarr
http-response set-header X-Frame-Options DENY if acl_readarr
http-response set-header X-Frame-Options DENY if acl_prowlarr
http-response set-header X-Frame-Options DENY if acl_nzbget
http-response set-header X-Frame-Options DENY if acl_jellyfin
http-response set-header X-Frame-Options DENY if acl_fafflix
http-response set-header X-Frame-Options DENY if acl_gitea
http-response set-header X-Frame-Options DENY if acl_grafana
http-response set-header X-Frame-Options DENY if acl_ceph_dashboard
http-response set-header X-Frame-Options DENY if acl_stalwart_webadmin
http-response set-header X-Frame-Options DENY if acl_kanidm
http-response set-header X-Content-Type-Options nosniff
http-response set-header X-XSS-Protection 1;mode=block
use_backend be_letsencrypt if acl-letsencrypt
use_backend %[req.hdr(host),lower,map(/usr/local/etc/haproxy/fe_https.map,be_default)]
frontend fe_imap
bind 0.0.0.0:143
mode tcp
description Frontend for Stalwart IMAP (STARTTLS)
default_backend be_stalwart_imap
log global
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_len 0 }
frontend fe_imaps
bind 0.0.0.0:993
mode tcp
description Frontend for Stalwart IMAPS (implicit TLS)
default_backend be_stalwart_imaps
log global
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_len 0 }
frontend fe_metrics
bind 0.0.0.0:8405
mode http
description Metrics Frontend
http-request set-header X-Forwarded-Proto https
http-request set-header X-Real-IP %[src]
http-request use-service prometheus-exporter if { path /metrics }
frontend fe_smtp
bind 0.0.0.0:25
mode tcp
description Frontend for Stalwart SMTP
default_backend be_stalwart_smtp
log global
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_len 0 }
frontend fe_submission
bind 0.0.0.0:587
mode tcp
description Frontend for Stalwart SMTP Submission
default_backend be_stalwart_submission
log global
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_len 0 }
backend be_ceph_dashboard
description Backend for Ceph Dashboard from Mgr instances
balance roundrobin
cookie SRVNAME insert indirect nocache
http-check expect status 200
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 9443 }
http-reuse always
option httpchk GET /
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
stick-table type ip size 200k expire 30m
server prodnxsr0009 198.18.23.9:9443 check cookie prodnxsr0009 fall 2 inter 2s rise 3 ssl verify none
server prodnxsr0010 198.18.23.10:9443 check cookie prodnxsr0010 fall 2 inter 2s rise 3 ssl verify none
server prodnxsr0011 198.18.23.11:9443 check cookie prodnxsr0011 fall 2 inter 2s rise 3 ssl verify none
server prodnxsr0012 198.18.23.12:9443 check cookie prodnxsr0012 fall 2 inter 2s rise 3 ssl verify none
server prodnxsr0013 198.18.23.13:9443 check cookie prodnxsr0013 fall 2 inter 2s rise 3 ssl verify none
backend be_default
description Backend for unmatched HTTP traffic
balance roundrobin
cookie SRVNAME insert
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
option httpchk GET /
option forwardfor
backend be_gitea
description Backend for gitea cluster
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
stick on src
stick-table type ip size 200k expire 30m
server ausyd1nxvm2080 198.18.26.18:443 check cookie ausyd1nxvm2080 fall 2 inter 2s rise 3 ssl verify none
server ausyd1nxvm2081 198.18.27.117:443 check cookie ausyd1nxvm2081 fall 2 inter 2s rise 3 ssl verify none
server ausyd1nxvm2082 198.18.28.71:443 check cookie ausyd1nxvm2082 fall 2 inter 2s rise 3 ssl verify none
backend be_grafana
description Backend for grafana nodes
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
stick on src
stick-table type ip size 200k expire 30m
server ausyd1nxvm2015 198.18.27.2:443 check cookie ausyd1nxvm2015 fall 2 inter 2s rise 3 ssl verify none
server ausyd1nxvm2016 198.18.28.189:443 check cookie ausyd1nxvm2016 fall 2 inter 2s rise 3 ssl verify none
backend be_jellyfin
description Backend for au-syd1 jellyfin
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
server ausyd1nxvm2051 198.18.25.164:443 check cookie ausyd1nxvm2051 fall 2 inter 2s rise 3 ssl verify none
backend be_k8s_kanidm
description Backend for Kanidm (auth.unkin.net via Kubernetes internal Traefik)
balance roundrobin
http-reuse always
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
redirect scheme https if !{ ssl_fc }
option httpchk
option forwardfor
option http-keep-alive
option prefer-last-server
http-check connect ssl sni auth.unkin.net
http-check send meth GET uri /status ver HTTP/1.1 hdr Host auth.unkin.net
http-check expect status 200
server k8s-traefik-internal 198.18.200.4:443 ssl verify none check inter 2s rise 3 fall 2 sni str(auth.unkin.net)
backend be_letsencrypt
description Backend for LetsEncrypt Verifications
balance roundrobin
server ausyd1nxvm2057 198.18.25.3:8888
backend be_lidarr
description Backend for au-syd1 lidarr
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /consul/health
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
server ausyd1nxvm2048 198.18.28.165:443 check cookie ausyd1nxvm2048 fall 2 inter 2s rise 3 ssl verify none
backend be_nzbget
description Backend for au-syd1 nzbget
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /consul/health
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
server ausyd1nxvm2045 198.18.25.44:443 check cookie ausyd1nxvm2045 fall 2 inter 2s rise 3 ssl verify none
backend be_prowlarr
description Backend for au-syd1 prowlarr
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /consul/health
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
server ausyd1nxvm2050 198.18.25.66:443 check cookie ausyd1nxvm2050 fall 2 inter 2s rise 3 ssl verify none
backend be_radarr
description Backend for au-syd1 radarr
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /consul/health
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
server ausyd1nxvm2047 198.18.27.131:443 check cookie ausyd1nxvm2047 fall 2 inter 2s rise 3 ssl verify none
backend be_readarr
description Backend for au-syd1 readarr
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /consul/health
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
server ausyd1nxvm2049 198.18.29.32:443 check cookie ausyd1nxvm2049 fall 2 inter 2s rise 3 ssl verify none
backend be_sonarr
description Backend for au-syd1 sonarr
balance roundrobin
cookie SRVNAME insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 443 }
http-reuse always
option httpchk GET /consul/health
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
server ausyd1nxvm2046 198.18.26.161:443 check cookie ausyd1nxvm2046 fall 2 inter 2s rise 3 ssl verify none
backend be_stalwart_imap
description Backend for Stalwart IMAP (STARTTLS)
balance roundrobin
mode tcp
option tcp-check
option prefer-last-server
stick on src
stick-table type ip size 200k expire 30m
tcp-check connect port 143 send-proxy
tcp-check expect string "* OK"
tcp-check send "A001 STARTTLS\r\n"
tcp-check expect rstring "A001 (OK|2.0.0)"
server ausyd1nxvm2124 198.18.28.76:143 check fall 3 inter 3s rise 2 send-proxy-v2
server ausyd1nxvm2125 198.18.29.44:143 check fall 3 inter 3s rise 2 send-proxy-v2
server ausyd1nxvm2126 198.18.25.160:143 check fall 3 inter 3s rise 2 send-proxy-v2
backend be_stalwart_imaps
description Backend for Stalwart IMAPS (implicit TLS)
balance roundrobin
mode tcp
option tcp-check
option prefer-last-server
stick on src
stick-table type ip size 200k expire 30m
tcp-check connect ssl send-proxy
tcp-check expect string "* OK"
server ausyd1nxvm2124 198.18.28.76:993 check fall 3 inter 3s rise 2 send-proxy-v2 ssl verify none
server ausyd1nxvm2125 198.18.29.44:993 check fall 3 inter 3s rise 2 send-proxy-v2 ssl verify none
server ausyd1nxvm2126 198.18.25.160:993 check fall 3 inter 3s rise 2 send-proxy-v2 ssl verify none
backend be_stalwart_smtp
description Backend for Stalwart SMTP
balance roundrobin
mode tcp
option tcp-check
option prefer-last-server
stick on src
stick-table type ip size 200k expire 30m
tcp-check connect port 25 send-proxy
tcp-check expect string "220 "
server ausyd1nxvm2124 198.18.28.76:25 check fall 3 inter 3s rise 2 send-proxy-v2
server ausyd1nxvm2125 198.18.29.44:25 check fall 3 inter 3s rise 2 send-proxy-v2
server ausyd1nxvm2126 198.18.25.160:25 check fall 3 inter 3s rise 2 send-proxy-v2
backend be_stalwart_submission
description Backend for Stalwart SMTP Submission
balance roundrobin
mode tcp
option tcp-check
option prefer-last-server
stick on src
stick-table type ip size 200k expire 30m
tcp-check connect port 587 send-proxy
tcp-check expect string "220 "
server ausyd1nxvm2124 198.18.28.76:587 check fall 3 inter 3s rise 2 send-proxy-v2
server ausyd1nxvm2125 198.18.29.44:587 check fall 3 inter 3s rise 2 send-proxy-v2
server ausyd1nxvm2126 198.18.25.160:587 check fall 3 inter 3s rise 2 send-proxy-v2
backend be_stalwart_webadmin
description Backend for Stalwart Webadmin
balance roundrobin
cookie SRVNAME insert indirect nocache
http-check expect status 200
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { dst_port 9443 }
http-reuse always
option httpchk GET /
option forwardfor
option http-keep-alive
option prefer-last-server
redirect scheme https if !{ ssl_fc }
stick-table type ip size 200k expire 30m
server ausyd1nxvm2124 198.18.28.76:443 check cookie ausyd1nxvm2124 fall 2 inter 2s rise 3 send-proxy-v2 ssl verify none
server ausyd1nxvm2125 198.18.29.44:443 check cookie ausyd1nxvm2125 fall 2 inter 2s rise 3 send-proxy-v2 ssl verify none
server ausyd1nxvm2126 198.18.25.160:443 check cookie ausyd1nxvm2126 fall 2 inter 2s rise 3 send-proxy-v2 ssl verify none
# The `peers au-syd1-prod` section is dropped: peer names must be static and a
# Deployment cannot provide them. Service sessionAffinity: ClientIP pins a
# client to one replica so the per-replica stick-tables behave as before.
listen health
bind 0.0.0.0:8404
mode http
monitor-uri /healthz
listen stats
bind 127.0.0.1:9090
mode http
stats uri /
stats auth admin:admin
+162
View File
@@ -0,0 +1,162 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: haproxy
namespace: haproxy
annotations:
reloader.stakater.com/auto: "true"
spec:
replicas: 3
selector:
matchLabels:
app: haproxy
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app: haproxy
spec:
automountServiceAccountToken: false
terminationGracePeriodSeconds: 150
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: haproxy
topologyKey: kubernetes.io/hostname
securityContext:
runAsNonRoot: true
runAsUser: 99
runAsGroup: 99
seccompProfile:
type: RuntimeDefault
containers:
- name: haproxy
image: haproxy:3.2.24-alpine
imagePullPolicy: IfNotPresent
command:
- haproxy
- -W
- -db
- -f
- /usr/local/etc/haproxy/haproxy.cfg
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
# Frontends bind 25/80/143/443/587; the dst_port ACLs need the real ports.
add: [NET_BIND_SERVICE]
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
- name: smtp
containerPort: 25
protocol: TCP
- name: imap
containerPort: 143
protocol: TCP
- name: submission
containerPort: 587
protocol: TCP
- name: imaps
containerPort: 993
protocol: TCP
- name: health
containerPort: 8404
protocol: TCP
- name: metrics
containerPort: 8405
protocol: TCP
- name: stats
containerPort: 9090
protocol: TCP
lifecycle:
preStop:
exec:
# SIGUSR1 to the master soft-stops the workers; hard-stop-after
# caps the drain. Wait so kubelet holds SIGTERM until it is done.
command:
- /bin/sh
- -c
- kill -s USR1 1; while kill -0 1 2>/dev/null; do sleep 1; done
livenessProbe:
httpGet:
path: /healthz
port: health
initialDelaySeconds: 15
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz
port: health
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 3
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 2
memory: 1Gi
volumeMounts:
- name: config
mountPath: /usr/local/etc/haproxy
readOnly: true
- name: cert-unkin-net
mountPath: /etc/haproxy/certs/unkin-net
readOnly: true
- name: cert-main-unkin-net
mountPath: /etc/haproxy/certs/main-unkin-net
readOnly: true
- name: cert-ceph-unkin-net
mountPath: /etc/haproxy/certs/ceph-unkin-net
readOnly: true
- name: run
mountPath: /var/lib/haproxy
volumes:
- name: config
configMap:
name: haproxy-config
# ssl-load-extra-files loads <crtfile>.key by default, so the key is
# projected next to the cert as tls.crt.key.
- name: cert-unkin-net
secret:
secretName: wildcard-unkin-net-tls
items:
- key: tls.crt
path: tls.crt
- key: tls.key
path: tls.crt.key
- name: cert-main-unkin-net
secret:
secretName: wildcard-main-unkin-net-tls
items:
- key: tls.crt
path: tls.crt
- key: tls.key
path: tls.crt.key
- name: cert-ceph-unkin-net
secret:
secretName: wildcard-ceph-unkin-net-tls
items:
- key: tls.crt
path: tls.crt
- key: tls.key
path: tls.crt.key
- name: run
emptyDir: {}
restartPolicy: Always
+12
View File
@@ -0,0 +1,12 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- configmap.yaml
- deployment.yaml
- service.yaml
- pdb.yaml
- vpa.yaml
- vmpodscrape.yaml
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: haproxy
+11
View File
@@ -0,0 +1,11 @@
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: haproxy
namespace: haproxy
spec:
maxUnavailable: 1
selector:
matchLabels:
app: haproxy
+43
View File
@@ -0,0 +1,43 @@
---
apiVersion: v1
kind: Service
metadata:
name: haproxy
namespace: haproxy
annotations:
purelb.io/service-group: dmz
purelb.io/addresses: 198.18.199.1
spec:
type: LoadBalancer
loadBalancerIP: "198.18.199.1"
# Source IP must survive for acl_internalsubnets, X-Real-IP and SMTP.
externalTrafficPolicy: Local
# Pins a client to one replica, standing in for the dropped stick-table peers.
sessionAffinity: ClientIP
selector:
app: haproxy
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
- name: smtp
port: 25
protocol: TCP
targetPort: smtp
- name: imap
port: 143
protocol: TCP
targetPort: imap
- name: submission
port: 587
protocol: TCP
targetPort: submission
- name: imaps
port: 993
protocol: TCP
targetPort: imaps
+13
View File
@@ -0,0 +1,13 @@
---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: haproxy
namespace: haproxy
spec:
selector:
matchLabels:
app: haproxy
podMetricsEndpoints:
- port: metrics
path: /metrics
+13
View File
@@ -0,0 +1,13 @@
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: haproxy-vpa
namespace: haproxy
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: haproxy
updatePolicy:
updateMode: "Off"
+3 -6
View File
@@ -1,16 +1,13 @@
---
# Log ingestion endpoint for puppet-managed VMs (and any non-k8s client).
# Reuses the internal Traefik gateway + cert-manager + external-dns pattern so
# VMs reach the Vector aggregator's HTTP source over TLS at a DNS name they can
# resolve. The puppet-side Vector rollout ships NDJSON to
# https://logs-ingest.k8s.syd1.au.unkin.net/ (a later task).
# Log ingestion endpoint for puppet-managed VMs (and any non-k8s client):
# fronts the VLCluster vlinsert service over TLS at a name VMs can resolve.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: logs-ingest
namespace: logging
labels:
app.kubernetes.io/name: vector-aggregator
app.kubernetes.io/name: victorialogs
app.kubernetes.io/component: ingest
traefik.io/instance: internal
annotations:
+4 -4
View File
@@ -5,7 +5,7 @@ metadata:
name: logs-ingest-http-redirect
namespace: logging
labels:
app.kubernetes.io/name: vector-aggregator
app.kubernetes.io/name: victorialogs
app.kubernetes.io/component: ingest
spec:
hostnames:
@@ -32,7 +32,7 @@ metadata:
name: logs-ingest
namespace: logging
labels:
app.kubernetes.io/name: vector-aggregator
app.kubernetes.io/name: victorialogs
app.kubernetes.io/component: ingest
spec:
hostnames:
@@ -46,8 +46,8 @@ spec:
- backendRefs:
- group: ""
kind: Service
name: vector-vm-ingest
port: 8080
name: vlinsert-logs
port: 9481
weight: 1
matches:
- path:
+1
View File
@@ -10,6 +10,7 @@ resources:
- job_clickhouse-schema.yaml
- nats-bootstrap-job.yaml
- cephrgw.yaml
- vlcluster.yaml
- gateway.yaml
- httproute.yaml
- serviceaccount_logarchiver.yaml
+47
View File
@@ -0,0 +1,47 @@
---
apiVersion: operator.victoriametrics.com/v1
kind: VLCluster
metadata:
name: logs
namespace: logging
spec:
clusterVersion: v1.52.0
vlinsert:
replicaCount: 2
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "2"
memory: 4Gi
vlselect:
replicaCount: 2
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "2"
memory: 4Gi
vlstorage:
replicaCount: 3
retentionPeriod: 180d
# ~3 GiB/day measured; 220GiB/node cap keeps 180d time-based, not disk-bound
retentionMaxDiskSpaceUsageBytes: 220GiB
storage:
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
storageClassName: cephrbd-fast-delete
resources:
requests:
storage: 250Gi
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "4"
memory: 8Gi
@@ -105,6 +105,17 @@ spec:
- mountPath: /docker-custom-entrypoint.d/pre-default/10-auth-conf.sh
name: compiler-auth-conf-seed
subPath: 10-auth-conf.sh
- mountPath: /docker-custom-entrypoint.d/pre-default/20-vault-helpers.sh
name: compiler-vault-helpers-seed
subPath: 20-vault-helpers.sh
- mountPath: /opt/certmanager/config.yaml
name: certmanager-config
subPath: certmanager.yaml
readOnly: true
- mountPath: /opt/sshsignhost/config.yaml
name: sshsignhost-config
subPath: sshsignhost.yaml
readOnly: true
initContainers:
- name: copy-configmaps
image: busybox:1.35
@@ -202,7 +213,38 @@ spec:
echo "$EXPECTED encapic" | sha256sum -c -
install -m 0755 encapic /opt/bin/encapic
# Puppet shells out to these two from generate() during catalog
# compilation: profiles::pki::vault runs certmanager and
# profiles::ssh::sign runs sshsignhost.
install_release() {
name=$1
version=$2
asset="$name-linux-amd64"
base="https://git.unkin.net/unkin/$name/releases/download/$version"
curl -fsSL -o "$name" "$base/$asset"
curl -fsSL -o "$name.checksums" "$base/checksums.txt"
# checksums.txt covers every release asset; pick the line for the
# one we downloaded and verify it under our local filename.
expected=$(awk -v a="$asset" '$NF == a || $NF == "*"a {print $1}' "$name.checksums")
if [ -z "$expected" ]; then
echo "no checksum for $asset in $version checksums.txt" >&2
exit 1
fi
echo "$expected $name" | sha256sum -c -
install -m 0755 "$name" "/opt/bin/$name"
}
install_release certmanager v0.2.0
install_release sshsignhost v0.1.0
echo "Shared binaries setup completed"
resources:
limits:
cpu: 300m
memory: 256Mi
requests:
cpu: 100m
memory: 64Mi
volumeMounts:
- mountPath: /opt/bin/
name: puppet-shared-bins
@@ -247,5 +289,15 @@ spec:
configMap:
name: compiler-auth-conf-seed
defaultMode: 0755
- name: compiler-vault-helpers-seed
configMap:
name: compiler-vault-helpers-seed
defaultMode: 0755
- name: certmanager-config
configMap:
name: certmanager-config
- name: sshsignhost-config
configMap:
name: sshsignhost-config
strategy:
type: RollingUpdate
+15
View File
@@ -64,6 +64,21 @@ configMapGenerator:
- resources/compiler/10-auth-conf.sh
options:
disableNameSuffixHash: true
- name: compiler-vault-helpers-seed
files:
- resources/compiler/20-vault-helpers.sh
options:
disableNameSuffixHash: true
- name: certmanager-config
files:
- resources/compiler/certmanager.yaml
options:
disableNameSuffixHash: true
- name: sshsignhost-config
files:
- resources/compiler/sshsignhost.yaml
options:
disableNameSuffixHash: true
- name: additional-ruby-gems
files:
- resources/additional-ruby-gems.sh
+29
View File
@@ -0,0 +1,29 @@
#!/bin/bash
set -euo pipefail
BIN_DIR=/opt/bin
CA=/opt/vault-ca-cert.crt
if [ ! -s "$CA" ]; then
echo "FATAL: $CA missing or empty; certmanager and sshsignhost cannot verify Vault" >&2
exit 1
fi
# profiles::pki::vault and profiles::ssh::sign shell out to fixed /usr/local/bin
# paths from generate(); the binaries ship on the shared PVC, and /usr/local/bin
# lives in the image. Wrappers rather than symlinks because neither binary reads
# a CA path from its config: SSL_CERT_FILE scopes the internal CA to these two
# processes instead of the puppetserver JVM's own trust store.
for bin in certmanager sshsignhost; do
if [ ! -x "$BIN_DIR/$bin" ]; then
echo "FATAL: $BIN_DIR/$bin missing; generate() would abort every catalog compile" >&2
exit 1
fi
cat > "/usr/local/bin/$bin" <<WRAPPER
#!/bin/sh
SSL_CERT_FILE=$CA
export SSL_CERT_FILE
exec $BIN_DIR/$bin "\$@"
WRAPPER
chmod 0755 "/usr/local/bin/$bin"
done
@@ -0,0 +1,12 @@
---
vault:
addr: https://vault.service.consul:8200
auth_method: kubernetes
k8s_mount: k8s/au/syd1
k8s_role: puppet_certmanager
jwt_path: /var/run/secrets/kubernetes.io/serviceaccount/token
mount_point: pki_int
role_name: servers_default
output_path: /tmp/certmanager
tls_skip_verify: false
timeout: 30s
@@ -0,0 +1,11 @@
---
vault:
addr: https://vault.service.consul:8200
auth_method: kubernetes
k8s_mount: k8s/au/syd1
k8s_role: puppet_sshsigner
jwt_path: /var/run/secrets/kubernetes.io/serviceaccount/token
mount_point: sshca
role_name: signhost
tls_skip_verify: false
timeout: 30s
@@ -0,0 +1,6 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../base/haproxy
+1
View File
@@ -29,6 +29,7 @@ spec:
- path: apps/overlays/*/ghp
- path: apps/overlays/*/gitea
- path: apps/overlays/*/grafana-system
- path: apps/overlays/*/haproxy
- path: apps/overlays/*/inteldeviceplugins-system
- path: apps/overlays/*/jfrog
- path: apps/overlays/*/k8up-system
+2
View File
@@ -43,6 +43,8 @@ spec:
server: https://kubernetes.default.svc
- namespace: 'gitea'
server: https://kubernetes.default.svc
- namespace: 'haproxy'
server: https://kubernetes.default.svc
- namespace: 'jfrog'
server: https://kubernetes.default.svc
- namespace: 'kanidm'