Compare commits

...

10 Commits

Author SHA1 Message Date
unkin-agent b31517e6d9 Merge pull request #481 from benvin/jellyfin-sso-valkey-state
Roll jellyfin-ha to v0.3.3 and drop Service session affinity

Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-20 14:02:48 +10:00
unkin-agent 5a74b2cec6 Roll bind-operator to v0.2.7 (journal-aware zone seeding) (#480)
Deploy bind-operator v0.2.7. The operator seeded a fresh skeleton zone file at serial 1 over zones whose BIND journal was still on disk at a higher serial; BIND rejected the inconsistent pair (`addzone failed: out of range`) and, with a PVC per replica, the stale journal outlived restarts while every reconcile rewrote the skeleton, so it never converged. That SERVFAILed roughly 1 in 3 authoritative answers for k8s.syd1.au.unkin.net and resolvers cached the failures.

- Bumps the operator image to v0.2.7
- Bumps the CRD install pin to the v0.2.7 tag, which changes the CRDs

Expect one rolling restart of the operator Deployment as the new image lands.

Reviewed-on: #480
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-20 00:53:47 +10:00
unkin-agent f14bcc4d2a Add woodpecker ServiceAccount for jellyfin-plugin-sso CI (#479)
The new `unkin/jellyfin-plugin-sso` fork is getting a Woodpecker pipeline, and its build step will set `serviceAccountName: jellyfin-plugin-sso`. Without the SA declared here the pipeline pods fail to schedule.

- add a bare ServiceAccount `jellyfin-plugin-sso` in the `woodpecker` namespace
- register it in the woodpecker base kustomization

The step only builds .NET code, so no Vault kube-auth role or RBAC is needed.

Reviewed-on: #479
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 17:32:31 +10:00
unkin-agent b01e4c3241 Rewrite slash-less Authentik token endpoint to the canonical path (#478)
Authentik advertises the token endpoint with a trailing slash, but some OIDC clients (the ArgoCD iOS app) POST to /application/o/token without one; Django's APPEND_SLASH will not redirect a POST, so the token exchange gets 405 and login fails.

- Add an exact-match rule on /application/o/token to the authentik and authentik-internal HTTPRoutes.
- Rewrite it to /application/o/token/ with a URLRewrite ReplaceFullPath filter, preserving the method and the authentik-server backend.
- Leave the catch-all PathPrefix rule untouched; exact matches outrank it in Gateway API precedence.

Reviewed-on: #478
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 16:10:30 +10:00
unkin-agent bbd5bdaa95 Enable PKCE for ArgoCD OIDC login (#477)
The Authentik client for ArgoCD is now public (the iOS app can't hold
a secret), so Authentik no longer enforces client_secret on token
exchange. PKCE replaces that as the protection against
authorization-code interception.

- Add `enablePKCEAuthentication: true` to the `oidc.config` block in
  `argocd-cm-patch.yaml`
- Note why PKCE is needed now that the client is public

Reviewed-on: #477
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 16:10:05 +10:00
unkin-agent 4762cf9e03 Add VMPodScrape for authentik-server metrics (#476)
Authentik server pods expose django_prometheus metrics on :9300, but only ldap-outpost and redis-exporter are scraped in this namespace. Add the missing per-app scrape.

- add apps/base/authentik/server-vmpodscrape.yaml selecting app.kubernetes.io/name=authentik, component=server on the metrics port
- wire it into apps/base/authentik/kustomization.yaml

Reviewed-on: #476
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 15:34:26 +10:00
unkin-agent c83a886e74 Enable pg_stat_statements on the authentik postgres cluster (#475)
The cluster preloads no statement-statistics library, so there is no per-query cost attribution in postgres and slow query paths have to be inferred from application-side metrics instead of read straight out of the database.

- preload `pg_stat_statements`
- set `pg_stat_statements.max` and `.track`, which is what makes CNPG manage the extension and create it in every database

Requires a postgres restart. Stacked on `benvin/authentik-cnpg-resources`.

Reviewed-on: #475
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 14:48:45 +10:00
unkin-agent 9a7200636c Raise authentik postgres CPU, memory and buffer sizing (#474)
The 500m CPU limit is a 50ms CFS quota per 100ms period, so the postgres pods are throttled on every burst even at ~0.01 cores average and each query pays that latency. 128MB of `shared_buffers` and a 256MB cache estimate also under-serve the planner on the joins authentik issues on its hot read paths.

- raise resources to requests `500m`/`1Gi`, limits `2`/`2Gi`
- raise `shared_buffers` to 512MB and `effective_cache_size` to 1536MB
- hold the post-incident memory headroom multiple over `shared_buffers`

Rolling restart with switchover. Stacked on `benvin/authentik-hot-standby-feedback`.

Reviewed-on: #474
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 14:35:16 +10:00
unkin-agent 9535bad9bc Enable hot_standby_feedback on the authentik postgres cluster (#473)
Authentik serves multi-second API reads from the CNPG hot standbys. Those reads outlive `max_standby_streaming_delay`, so recovery cancels them with `canceling statement due to conflict with recovery`, which authentik surfaces as HTTP 500 — enough to break a terraform apply mid-run.

- set `hot_standby_feedback` on so replicas report their oldest xmin to the primary and long reads stop being cancelled
- SIGHUP reload only, no restart or switchover
- retained-dead-tuple cost is negligible on a ~155MB database

Reviewed-on: #473
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 14:34:33 +10:00
unkin-agent 34dd70435e Auto-reload cheeztv and fafflix on plugin ConfigMap change (#471)
Edits to the cheeztv/fafflix plugin ConfigMaps only reach the pods via the inject-plugin-config initContainer, so a config change sat inert until someone manually rolled the StatefulSet. Reloader is deployed cluster-wide with autoReloadAll disabled, so each workload has to opt in.

- annotate both StatefulSets with configmap.reloader.stakater.com/auto: "true"

Reviewed-on: #471
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
2026-09-19 12:25:08 +10:00
13 changed files with 89 additions and 14 deletions
+21 -6
View File
@@ -64,8 +64,12 @@ spec:
archive_mode: "on"
archive_timeout: 5min
dynamic_shared_memory_type: posix
effective_cache_size: 256MB
effective_cache_size: 1536MB
full_page_writes: "on"
# Replicas report their oldest xmin to the primary, so multi-second reads on
# a hot standby stop exhausting max_standby_streaming_delay and being
# cancelled. Retained-dead-tuple cost is negligible on a ~155MB database.
hot_standby_feedback: "on"
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
@@ -77,7 +81,12 @@ spec:
max_parallel_workers: "16"
max_replication_slots: "16"
max_worker_processes: "16"
shared_buffers: 128MB
# A pg_stat_statements.* parameter is what makes CNPG treat the extension as
# managed and run CREATE EXTENSION in every database; preloading alone does
# not create it.
pg_stat_statements.max: "10000"
pg_stat_statements.track: top
shared_buffers: 512MB
shared_memory_type: mmap
ssl_max_protocol_version: TLSv1.3
ssl_min_protocol_version: TLSv1.3
@@ -86,6 +95,9 @@ spec:
wal_log_hints: "on"
wal_receiver_timeout: 5s
wal_sender_timeout: 5s
# CNPG merges this with the libraries it manages itself.
shared_preload_libraries:
- pg_stat_statements
syncReplicaElectionConstraint:
enabled: false
primaryUpdateMethod: restart
@@ -105,13 +117,16 @@ spec:
updateInterval: 30
resources:
limits:
cpu: 500m
# 500m is a 50ms CFS quota per 100ms period, exhausted by bursts even at
# ~0.01 cores average, so every query pays throttle latency.
cpu: "2"
# 512Mi OOMKilled replicas under load (shared_buffers 128MB +
# max_connections 200 leave no headroom) — see incident 2026-07-28.
memory: 1Gi
# shared_buffers 512MB needs the same headroom multiple, hence 2Gi.
memory: 2Gi
requests:
cpu: 50m
memory: 512Mi
cpu: 500m
memory: 1Gi
smartShutdownTimeout: 180
startDelay: 3600
stopDelay: 1800
+32
View File
@@ -37,6 +37,22 @@ spec:
name: authentik
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: authentik-server
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplaceFullPath
replaceFullPath: /application/o/token/
matches:
- path:
type: Exact
value: /application/o/token
- backendRefs:
- group: ""
kind: Service
@@ -86,6 +102,22 @@ spec:
name: authentik-internal
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: authentik-server
port: 80
weight: 1
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplaceFullPath
replaceFullPath: /application/o/token/
matches:
- path:
type: Exact
value: /application/o/token
- backendRefs:
- group: ""
kind: Service
+1
View File
@@ -19,6 +19,7 @@ resources:
- redis-deployment.yaml
- redis-pvc.yaml
- redis-service.yaml
- server-vmpodscrape.yaml
- vaultauth.yaml
- vaultstaticsecret.yaml
- vmpodscrape.yaml
@@ -0,0 +1,16 @@
---
# Scrape the authentik server's django_prometheus endpoint (:9300). Picked up
# by the observability VMAgent (selectAllByDefault).
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: authentik-server
namespace: authentik
spec:
selector:
matchLabels:
app.kubernetes.io/name: authentik
app.kubernetes.io/component: server
podMetricsEndpoints:
- port: metrics
path: /metrics
+1 -1
View File
@@ -21,7 +21,7 @@ spec:
runAsNonRoot: true
containers:
- name: operator
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/bind-operator:v0.2.6
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/bind-operator:v0.2.7
args:
- --metrics-bind-address=:8080
- --health-probe-bind-address=:8081
+1 -1
View File
@@ -6,7 +6,7 @@ resources:
- namespace.yaml
# CRDs are pulled from the bind-operator repo at the matching tag rather than
# vendored here, so they never drift from the operator.
- https://git.unkin.net/unkin/bind-operator/raw/tag/v0.2.6/config/crd/install.yaml
- https://git.unkin.net/unkin/bind-operator/raw/tag/v0.2.7/config/crd/install.yaml
- rbac.yaml
- agent-dns-rbac.yaml
- deployment.yaml
-2
View File
@@ -13,6 +13,4 @@ spec:
targetPort: http
selector:
app: cheeztv
# Pin each client to one replica to reduce transcode-session churn/takeover.
sessionAffinity: ClientIP
type: ClusterIP
+3 -1
View File
@@ -4,6 +4,8 @@ kind: StatefulSet
metadata:
name: cheeztv
namespace: cheeztv
annotations:
configmap.reloader.stakater.com/auto: "true"
spec:
# HA: two replicas coordinate transcode session ownership through Valkey and
# resume each other's HLS segments off the shared RWX transcode PVC. Stable
@@ -162,7 +164,7 @@ spec:
readOnly: true
containers:
- name: cheeztv
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.3.2
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.3.3
imagePullPolicy: IfNotPresent
ports:
- name: http
-2
View File
@@ -13,6 +13,4 @@ spec:
targetPort: http
selector:
app: fafflix
# Pin each client to one replica to reduce transcode-session churn/takeover.
sessionAffinity: ClientIP
type: ClusterIP
+3 -1
View File
@@ -4,6 +4,8 @@ kind: StatefulSet
metadata:
name: fafflix
namespace: fafflix
annotations:
configmap.reloader.stakater.com/auto: "true"
spec:
# HA: two replicas coordinate transcode session ownership through Valkey and
# resume each other's HLS segments off the shared RWX transcode PVC. Stable
@@ -162,7 +164,7 @@ spec:
readOnly: true
containers:
- name: fafflix
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.3.2
image: artifactapi.k8s.syd1.au.unkin.net/docker-internal/jellyfin-ha:v0.3.3
imagePullPolicy: IfNotPresent
ports:
- name: http
+1
View File
@@ -15,6 +15,7 @@ resources:
- serviceaccount_mediamark_ci.yaml
- serviceaccount_plugin_docker_buildx.yaml
- serviceaccount_jellyfin_ha_src.yaml
- serviceaccount_jellyfin_plugin_sso.yaml
- serviceaccount_repospawner_ci.yaml
- serviceaccount_terraform_artifactapi.yaml
- serviceaccount_terraform_authentik.yaml
@@ -0,0 +1,6 @@
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: jellyfin-plugin-sso
namespace: woodpecker
@@ -26,6 +26,10 @@ data:
issuer: https://identity.unkin.net/application/o/argocd/
clientID: argocd
clientSecret: $argocd-oidc:client_secret
# The Authentik client is public (the iOS app can't hold a secret), so
# Authentik no longer enforces clientSecret; PKCE replaces it as the
# protection against authorization-code interception.
enablePKCEAuthentication: true
# identity.unkin.net now serves the LetsEncrypt *.unkin.net wildcard, so the
# stock image trust store validates it; no rootCA pin.
requestedScopes: