The v3.13.0 images were never published (registry push failed), so the current pin cannot be pulled.
- pin api and ui images back to v3.12.0
Reviewed-on: #532
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Bump artifactapi API and UI images from v3.12.0 to v3.13.0.
- Update git.unkin.net/unkin/artifactapi to v3.13.0
- Update git.unkin.net/unkin/artifactapi-ui to v3.13.0
Reviewed-on: #529
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The standalone k8s Consul runs its own raft under DC `au-syd1`, the same DC name as the VM cluster that k8s servers are about to join. Its leftover state risks a split brain, so it goes before the replacement lands.
- remove `apps/overlays/au-syd1/consul`, dropping `platform-consul` from the platform ApplicationSet and cascading deletion of the `consul` namespace and PVCs
- keep `apps/base/consul` for reuse by the replacement
Reviewed-on: #530
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v0.1.7 stops the operator reconcile hot-loop, which was hot-reloading Kea about 9 times a second and flooding the dhcp-system logs. Images now publish to docker-internal.
- bump kea-operator, kea and kea-api to v0.1.7
- pull all three from artifactapi docker-internal
Reviewed-on: #527
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The consul overlay fetches its chart directly from helm.releases.hashicorp.com, while the other 15 helm overlays on the estate mirror use the artifactapi virtual helm repo, which already proxies hashicorp-helm.
- point consul helmCharts repo at the artifactapi virtual helm repo
Reviewed-on: #528
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Querying the VLCluster from outside the cluster (CLI tools, puppet VMs) needs a resolvable TLS endpoint for vlselect, mirroring the existing logs-ingest endpoint for vlinsert.
- add `vlselect` Gateway on `traefik-internal` with vault-issuer TLS and external-dns for `vlselect.k8s.syd1.au.unkin.net`
- add HTTP->HTTPS redirect route and HTTPS route to `vlselect-logs:9471`, unauthenticated
Reviewed-on: #525
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
authentik-server requests only 50m CPU, so on busy nodes it is starved, `/-/health/live/` exceeds the 3s liveness timeout and kubelet restarts it (~100 restarts per pod).
- raise `server.resources.requests.cpu` from 50m to 500m
Reviewed-on: #526
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Compiler pods average ~2Gi memory but request 1Gi, so the scheduler overpacks nodes and the HPA's utilisation target is skewed.
- raise compiler requests to 1 cpu / 2Gi and limits to 4 cpu / 4Gi
- scale up faster (30s window, 75%) and down slower (600s window)
Reviewed-on: #524
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The VM haproxies are being decommissioned; the sites they published via puppet CNAMEs are already served by the k8s haproxy edge, so k8s DNS publishes them instead.
- add main.unkin.net A records for sonarr, radarr, lidarr, readarr, prowlarr, nzbget, jellyfin -> 198.18.199.0
- add unkin.net A records for fafflix and git (git-edge.yaml) -> 198.18.199.0, auth -> 198.18.200.4
- point the commented git.yaml cutover at replacing git-edge.yaml
Merge alongside puppet-prod halb DNS removal; A records only take effect once the puppet CNAMEs are gone.
Reviewed-on: #523
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
grafana.unkin.net still routes through the puppet haproxy edge to the old grafana VMs; the k8s grafana should serve it directly like identity and vlogs.
- add grafana-external Gateway (traefik-external, *.unkin.net wildcard) with redirect + main HTTPRoutes
- reflect wildcard-unkin-net-tls into grafana
- set grafana root_url to https://grafana.unkin.net
- add grafana A record -> 198.18.199.0 in the bind-operator unkin.net zone
- drop grafana.unkin.net from the k8s haproxy routes and config
Requires terraform-authentik grafana redirect URI PR applied first, and the puppet halb vrrp_cnames grafana.unkin.net CNAME removed.
Reviewed-on: #522
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
artifactapi v3.12.0 makes DELETE eviction clear every cache layer, so stale mutable indexes (e.g. epel repomd) no longer survive eviction. It also adds `?prefix=` filtering to object listings.
- bump artifactapi and artifactapi-ui images to v3.12.0
Reviewed-on: #521
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Both traefik HPAs report `ScalingActive False: missing request for cpu` because the chart values set no container resources, so the CPU-utilisation metric can't be computed and neither Deployment ever scales.
- traefik-external requests 200m CPU / 128Mi, memory limit 512Mi
- traefik-internal requests 500m CPU / 256Mi, memory limit 1Gi
- No CPU limits, to avoid throttling the proxy
- The chart derives GOMEMLIMIT (90% of the memory limit) from the new limits
Reviewed-on: #520
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
With the default Cluster external policy, kube-proxy SNATs inbound traffic to the traefik-external LoadBalancer, hiding real client IPs from traefik and adding a cross-node hop. Local preserves source IPs. Internal policy stays Cluster so in-cluster callers on nodes without a traefik-external pod still reach it.
- Set `externalTrafficPolicy: Local` on the traefik-external Service
- Set `internalTrafficPolicy: Cluster` explicitly
Reviewed-on: #519
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
git.unkin.net is served by the VM haproxy during the forge migration. Claiming it on the k8s gitea Gateway/HTTPRoutes competes with that path, so the k8s gitea serves only its admin name until cutover.
- remove git.unkin.net from the gitea HTTPRoute and redirect route hostnames
- remove the http-primary/https-primary listeners and their parentRefs
- set the gateway cert common-name to git.k8s.syd1.au.unkin.net
Reviewed-on: #518
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The kubernetes-apiservers job reports up=0 on every target because the apiserver /metrics response exceeds vmagent's default 16MiB scrape size limit.
- set max_scrape_size: 64MiB on the kubernetes-apiservers job only
Reviewed-on: #517
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The consul job joins node and port with ":" but then sets `replacement: "${1}:${2}"` with the default `(.*)` regex, so `__address__` becomes `node:port:`. vmagent rejects every consul target as an invalid port, including node_exporter.
- drop the replacement so the default `$1` yields `node:port`
Reviewed-on: #516
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
nginx in front of Consul closes requests at 90s. vmagent's consul SD sends blocking queries with a 525s wait, so every watch fails with a 504 and the consul job finds no targets. That leaves no puppet-host metrics (node_exporter etc.) in VictoriaMetrics.
- set `promscrape.consul.waitTime: 50s` on the main VMAgent
Reviewed-on: #515
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
terraform-infra is replaced by terraform-netbox, whose Vault kubernetes auth role binds the terraform-netbox service account in the woodpecker namespace.
- rename serviceaccount_terraform_infra to serviceaccount_terraform_netbox
Pairs with terraform-vault #161 (the auth role) and terraform-git #105 (the repo).
Reviewed-on: #514
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
With k8up operator metrics now scraped into VictoriaMetrics, backup job and schedule state still has no dashboard to view it.
- add the k8up dashboard from M0NsTeRRR/grafana-dashboards as a gzipped GrafanaDashboard CR
- resolve its datasource to the in-cluster VictoriaMetrics uid
- register it in the grafana kustomization
Reviewed-on: #513
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The k8up operator already serves Prometheus metrics (backup/check/prune job counts, controller-runtime) on its k8up-metrics Service, but nothing scrapes them, so backup health is invisible in VictoriaMetrics.
- add a VMServiceScrape for the k8up-metrics Service (port http, /metrics)
Reviewed-on: #512
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Grafana lacks Kubernetes cluster/workload dashboards; the victoria-metrics-k8s-stack sync-job ships a maintained set as GrafanaDashboard CRs.
- render victoria-metrics-k8s-stack 0.95.0 in the grafana overlay with only the dashboard sync-job enabled
- sync dotdc kubernetes-views/api-server and kube-prometheus dashboards via the artifactapi github_user remote
- skip dashboards already in apps/base/grafana and those without scraped metrics
- trust vault-ca-cert for artifactapi TLS
Requires terraform-artifactapi #49 applied first.
Reviewed-on: #511
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
grafana-operator already creates the headless `grafana-alerting` Service (port 9094) and injects `POD_IP` into the grafana container, so the copies added in #509 make Argo CD and the operator fight over the same objects.
- remove `service-alerting.yaml` from the grafana base
- remove the `POD_IP` env entry from the Grafana CR
- keep the `unified_alerting` `ha_*` config, which targets the operator's Service
Reviewed-on: #510
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Grafana runs 3 replicas with no alerting HA config, so every replica sends every notification.
- add headless `grafana-alerting` Service exposing gossip on 9094 TCP/UDP
- inject `POD_IP` from `status.podIP` into the grafana container
- set `unified_alerting` `ha_peers`, `ha_listen_address` and `ha_advertise_address` so replicas form one Alertmanager cluster
Reviewed-on: #509
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A single Grafana pod with a 1 cpu/1Gi limit is a single point of failure and gets throttled under dashboard load. State lives in Postgres, so extra replicas are safe; spreading them keeps a node loss from taking out Grafana.
- set `replicas: 3` on the grafana deployment
- raise grafana container limits to 2 cpu / 4Gi
- spread grafana pods across nodes with a soft hostname topology constraint
Reviewed-on: #508
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The Vault KV layout is kubernetes/namespace/<ns>/<sa>/<secret>, so vlogs and logviewer both running as SA default in the shared logging namespace would collide on one oauth-credentials entry. Splitting vlogs out resolves it without widening any Vault policy.
- Move apps/base/logging/vlogs to apps/base/vlogs, namespace vlogs
- Add namespace.yaml and a vlogs-scoped VaultAuth (role default)
- Point the VaultStaticSecret at kubernetes/namespace/vlogs/default/oauth-credentials
- Add the au-syd1 overlay, platform ApplicationSet path and project destination
- Move the wildcard-unkin-net-tls reflection from logging to vlogs; vlogs was its only consumer there
Secret is already seeded at the new Vault path.
Reviewed-on: #507
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The vector -> JetStream -> ClickHouse chain duplicated storage the in-cluster VictoriaLogs already provides, and its NATS StatefulSet keeps the logging Application from syncing.
- collect pod logs with a VLAgent DaemonSet remote-writing to vlinsert-logs
- delete vector, NATS, ClickHouse, logarchiver and logviewer
- drop clickhouse-system from the logging ApplicationSet
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #506
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Vector is being removed from the logging stack in favour of a VictoriaLogs VLAgent, so every path this pipeline tests is going away and the pipeline becomes dead weight.
- Delete `.woodpecker/vector-test.yaml`.
Companion PR removes the vector manifests under `apps/base/logging/vector/`.
Reviewed-on: #505
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
bind-operator v0.3.1 fixes the apex NS bug two ways: the operator no longer glues apex NS records to pod IPs, and a new CEL rule on `DNSRecordSpec` rejects apex NS DNSRecords at admission. The CRDs are pinned by tag URL, so bumping only the image would leave the admission half inert.
- Bump the bind-operator image to `v0.3.1`
- Bump the CRD install URL to the matching `v0.3.1` tag
---------
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Reviewed-on: #503
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Visiting https://vlogs.unkin.net/ currently lands on the bare VictoriaLogs index page with relative links, not the query UI.
- Add an Exact `/` rule to the `vlogs` HTTPRoute that 302s to `/select/vmui/`
- Keep the PathPrefix `/` catch-all pointed at `vlogs-oauth2:80`, so auth is unchanged
Reviewed-on: #504
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
records.yaml had grown to ten DNSRecord documents across two zones, so finding or reviewing a single record meant scanning the whole file and every change touched it.
- move each record to authoritative/<zone>/<type>/<record>.yaml
- add a kustomization.yaml per zone and per type directory
- keep the git.unkin.net cutover record commented out, alongside a commented kustomization entry
- move the DNSRecord-namespace rationale onto the authoritative kustomization
- delete records.yaml
Rendered output is unchanged.
Reviewed-on: #501
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A single records.yaml holding every DNSRecord means any record change touches a shared file, and deleting one record is a hunk edit rather than a file removal. One file per record under <zone>/<type>/<record>.yaml makes each record independently editable and removable.
- move acme-apex-ns to acme-unkin-net/ns/apex.yaml and acme-ns1-a to acme-unkin-net/a/ns1.yaml
- add kustomization.yaml per zone and type directory, referencing directories from the parent
- carry the delegation rationale onto the zone kustomization
- drop records.yaml and reference the zone directory from the app kustomization
Rendered output is unchanged.
Reviewed-on: #500
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The vlselect query UI is only reachable in-cluster, so every log search needs a port-forward. Publish it at vlogs.unkin.net behind oauth2-proxy on the external (DMZ) Traefik.
- Add `apps/base/logging/vlogs`: external Gateway, http->https redirect and main HTTPRoute
- Terminate TLS with the reflected Let's Encrypt `*.unkin.net` wildcard; reflect it into `logging`
- Publish the `vlogs` A record at the DMZ gateway VIP from the bind-operator `unkin.net` zone
- Route all traffic through the `vlogs-oauth2` Service, upstreaming to `vlselect-logs:9471`
- Gate on the Authentik `vlogs` application, group `akP-vlogs-admin`
- Read OIDC credentials from `kv/kubernetes/namespace/logging/default/vlogs-oauth-credentials`
Depends on unkin/terraform-authentik#40.
Reviewed-on: #498
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
`connect` mode gives the client no local cache, so every cache operation is a network round trip -- about 2.7x slower than pointing the plugin straight at S3. Developers and CI run the plugin in direct mode instead.
- Remove the `go-cache-plugin serve` Deployment and its nginx stream-proxy ConfigMap
- Remove the PureLB Service, freeing `198.18.200.11`
- Keep the `gocache` bucket and `gocache-s3` Secret for direct-mode clients
Reviewed-on: #499
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The k8s log pipeline stores to ClickHouse via NATS+vector, while the VM estate ships journald to a separate puppet-managed VictoriaLogs cluster. Consolidating on VictoriaLogs in-cluster collapses the two paths, and the logs-ingest gateway has no clients yet so it can be repointed now, ahead of the puppet change.
- add VLCluster `logs` at v1.52.0 (2 vlinsert, 2 vlselect, 3 vlstorage, 180d retention, 250Gi each on cephrbd-fast-delete)
- cap vlstorage disk use at 220GiB per node so 180d stays time-based rather than disk-bound
- repoint the logs-ingest HTTPRoute at `vlinsert-logs:9481`
- add a VictoriaLogs Grafana datasource and install its plugin
Nothing is removed here; NATS, ClickHouse, vector, logarchiver and logviewer keep running until a follow-up drops them.
Reviewed-on: #488
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
`go-cache-plugin serve` binds `127.0.0.1` only, so nothing outside the pod can reach it and laptops have no way to use the S3-backed Go cache without holding RGW credentials.
- Run `go-cache-plugin serve` against the `gocache` bucket, path-style, explicit region to skip the GetBucketLocation probe
- Add an nginx sidecar stream-proxying `:9090` to the loopback plugin port, `proxy_timeout 2h`
- Publish it on PureLB `198.18.200.11`, `externalTrafficPolicy: Local` so the client IP reaches the allow rules
- Restrict to workstation + pod CIDRs: GOCACHEPROG is unauthenticated and a poisoned entry runs in every consuming build
Merge only after `docker-internal/go-cache-plugin:v0.1.0` is published.
Reviewed-on: #496
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The zone was seeded with an apex `NS ns1.acme.unkin.net` glued to the primary pod IP. Both were later corrected by hand, so the live RRset and the ns1 address exist only in the zone journal -- a reseed republishes the pod IP and breaks DNS-01 for every `*.unkin.net` cert. Declaring them makes git the source of truth.
- declare the two published apex NS names
- declare the in-zone ns1 address, which a seed would otherwise glue to the pod IP
Matches what the zone serves today, so applying it changes no records. Requires bind-operator v0.3.0.
Reviewed-on: #495
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v0.3.0 converges a zone apex NS onto its declared nameservers instead of leaving the seed placeholder, which publishes a primary pod IP. The CRD moves with the image because the operator reads the new `spec.nameservers` field.
- pin the bind-operator image to v0.3.0
- pull the CRDs from the v0.3.0 tag
Reviewed-on: #494
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The haproxy edge holds its own DMZ VIP, a second public entry point alongside
traefik-external that must be firewalled and DNS'd separately. Traefik can
front it with TLS passthrough, leaving haproxy's certs and backends untouched.
- Add a `traefik-external` Gateway: HTTP :80 plus Passthrough TLS :443.
- TLSRoute the 12 `fe_https.map` hostnames to haproxy:443; HTTPRoute 301s :80.
- Make the Service ClusterIP on 443 only, releasing 198.18.199.1.
- Drop `fe_http`, `be_letsencrypt` and `fe_http.map`; certs are DNS-01 only.
Client IP now reads as a Traefik pod — the Gateway provider cannot emit PROXY protocol to a TLSRoute backend. `sessionAffinity` goes too (it would pin Traefik pods, not clients); SRVNAME cookies keep persistence.
Reviewed-on: #492
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Stalwart was only ever a test deployment. The daemon is dead on all three
backend VMs and nothing public depends on it — `unkin.net` MX points at Google —
so the edge is proxying mail to nowhere and the tcp frontends make `defaults`
emit 20 spurious HTTP-mode warnings.
- Drop the `fe_smtp`, `fe_submission`, `fe_imap` and `fe_imaps` frontends.
- Drop the five `be_stalwart_*` backends and their map entries in `fe_http.map`/`fe_https.map`.
- Drop the now-unused 25/143/587/993 Service and container ports.
`haproxy -c` on the rendered config: exit 0, 0 warnings (was 20), 0 alerts.
Reviewed-on: #491
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Go builds on CI and laptops each rebuild the same packages from scratch. A
GOCACHEPROG backend needs an S3 bucket plus credentials before anything can
point at it, so provision those first. The bucket lives in the woodpecker
namespace because CI is the primary consumer and reads the Secret there.
- add Bucket and ObjectStoreUser for the shared Go build cache
- use default (replicated) placement rather than the ec target, since a build
cache is millions of small objects
- purge and drop the bucket and user on delete; the cache is disposable
Nothing consumes the bucket yet.
Reviewed-on: #489
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
A g10k-code job wedged in ContainerCreating on a failed CephFS mount and never reached a terminal condition, so it stayed in the CronJob active list and `concurrencyPolicy: Forbid` skipped every following minute. No Puppet code reached the estate for 6 days, and the piled-up missed slots crossed the controller 100-slot cap into `TooManyMissedTimes`. The CronJob carried no deadlines at all.
- Cap a job at `activeDeadlineSeconds: 300` on the Job spec, so a hang is failed as `DeadlineExceeded` and drops out of the active list (healthy runs take 16-18s).
- Set `startingDeadlineSeconds: 200`, bounding missed-schedule look-back to ~3 slots so the count cannot reach 100.
Reviewed-on: #490
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The 13 `X-Frame-Options DENY if acl_<host>` rules in `fe_https` have never fired:
their ACLs use `req.hdr(host)`, a request-direction fetch that is invalid in a
response ruleset, so HAProxy rejects them at config-check time. Carried over
verbatim from the Puppet LXD config during the k8s move.
- Remove the 13 dead `http-response set-header X-Frame-Options` rules.
- Remove the 15 now-orphaned `acl acl_*` definition lines.
Not switching the header on: it has never been live, and Grafana/Gitea send their
own. `haproxy -c` warnings drop 33 -> 20; the two working response headers stay.
Reviewed-on: #487
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The au-syd1 edge proxy runs on a hand-managed LXD container outside the cluster, with no HA and no shared config source.
- Add `apps/base/haproxy/`: 3 replicas behind the DMZ LoadBalancer 198.18.199.1, config from a ConfigMap, wildcard certs from reflected secrets.
- Keep source IPs via `externalTrafficPolicy: Local`; `sessionAffinity: ClientIP` stands in for the stick-table peers a Deployment cannot name.
- Drain on shutdown: `hard-stop-after 2m`, a preStop SIGUSR1 soft-stop, 150s grace.
- Bind the stats listener to 127.0.0.1 so it is port-forward only.
- Register the app in the platform project and ApplicationSet.
Reviewed-on: #485
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The acme.unkin.net zone still serves only the bind-operator seed apex: NS ns1.acme.unkin.net glued to A 10.42.6.38, a pod IP no pod holds. The parent delegates to acme-ns1.unkin.net, but public resolvers have already promoted the child NS RRset, so when the cached address expires DNS-01 fails for every unkin.net wildcard at once.
- Add apex NS acme-ns1.unkin.net., matching the parent delegation (out of zone, no glue needed).
- Point ns1.acme.unkin.net at 103.216.191.185 so resolvers holding the seeded NS name still reach the zone.
- The operator seed placeholder itself is tracked separately in bind-operator.
---------
Co-authored-by: unkin-agent <agent@unkin.net>
Reviewed-on: #486
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
The haproxy edge terminates TLS for hosts under `main.unkin.net` and `ceph.unkin.net`, which the single `*.unkin.net` wildcard does not cover.
- Add cert-manager Certificates for both wildcards from the `letsencrypt` ClusterIssuer.
- Reflect the minted secrets into the `haproxy` namespace.
Needs these records in the public unkin.net zone first:
`_acme-challenge.main.unkin.net. CNAME _acme-challenge.main.acme.unkin.net.`
`_acme-challenge.ceph.unkin.net. CNAME _acme-challenge.ceph.acme.unkin.net.`
Reviewed-on: #484
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
profiles::pki::vault and profiles::ssh::sign shell out to
/usr/local/bin/certmanager and /usr/local/bin/sshsignhost from generate()
during catalog compilation. Neither binary exists in the compiler image, so
every node using them fails to compile.
- install certmanager v0.2.0 and sshsignhost v0.1.0 onto the shared bin volume with sha256 verification
- wrap both at /usr/local/bin from a pre-default entrypoint hook, failing startup loudly if either is missing
- mount read-only Vault configs for both: kubernetes auth on k8s/au/syd1, internal CA verified rather than skipped
Reviewed-on: #482
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The deployed v0.3.3 build returns 500 from /Shows/NextUp on PostgreSQL, breaking the home screen, and lets replicas diverge: library-visibility and shared-config changes never propagate, user data (resume, played state, favourites, ratings) is overwritten between pods, and eight scheduled tasks run on every replica instead of only the scan leader. v0.4.0 carries the fixes.
- Pin cheeztv and fafflix to jellyfin-ha:v0.4.0.
No config change needed: cross-pod invalidation reuses the transcode-store Redis connection string both apps already set.
Reviewed-on: #483
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
unkin-agent
unkinben