watchstate: deploy admin-gated jellyfin watch-state sync tool (#419)

## Why

Deploy WatchState (arabcoders/watchstate), the Jellyfin/Plex/Emby watch-state
sync tool, as an internal admin tool. It gets an admin-only web UI/API gated the
same way as logviewer: an oauth2-proxy front backed by Authentik OIDC.

## What

- New `watchstate` namespace + media-project app; base at `apps/base/watchstate`,
  overlay at `apps/overlays/au-syd1/watchstate`.
- Image `ghcr.io/arabcoders/watchstate:v1.10.3` (current release; canonical
  upstream name, containerd mirrors route ghcr via artifactapi).
- `replicas: 1`, `strategy: Recreate`, single `5Gi` `cephrbd-fast-delete` RWO PVC
  at `/config` — sqlite + the in-container cron/redis are single-writer.
- `runAsUser/runAsGroup/fsGroup: 1000` (image's rootless user); liveness/readiness
  `GET /v1/api/system/healthcheck` on 8080 (route confirmed in upstream
  `src/API/System/HealthCheck.php`, no auth guard).
- oauth2-proxy (mirrors logviewer 1:1) fronts every path. Authentik issuer
  `identity.k8s.syd1.au.unkin.net`, redirect
  `https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback`. Authorization is
  enforced Authentik-side (akR-global-admin only), so no oauth2-proxy group
  allowlist is configured.
- Internal-only Gateway (`traefik-internal`) for
  `watchstate.k8s.syd1.au.unkin.net`, `vault-issuer` TLS leaf, external-dns to
  `198.18.200.4`. HTTP -> HTTPS redirect.
- `VaultStaticSecret` pulls OIDC creds from
  `kv/kubernetes/namespace/watchstate/default/oauth-credentials`; `vault-ca-cert`
  auto-reflects into the namespace.
- Registered in the media `ApplicationSet` + `AppProject`.

## Scrape decision

No `VMPodScrape`: WatchState exposes no Prometheus/`/metrics` endpoint.

## Follow-ups

- **Seed check:** the VaultStaticSecret expects `client_id`, `client_secret`, and
  `cookie_secret` keys at the kv path. Only `client_secret` was confirmed seeded;
  `client_id` and a generated `cookie_secret` must also be present or the
  oauth2-proxy pod will not start.
- **Webhook ingestion:** the Jellyfin webhook endpoint (`/v1/api/webhook`)
  currently sits behind oauth2-proxy like everything else. When sync is wired up,
  the jellyfins pushing webhooks will need an auth-bypass or an apikey route for
  that path.

Reviewed-on: #419
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>

apps/base/watchstate/deployment.yaml

@@ -0,0 +1,81 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: watchstate
+  namespace: watchstate
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: watchstate
+  strategy:
+    # sqlite + the in-container cron/redis single-writer; never run two pods.
+    type: Recreate
+  template:
+    metadata:
+      labels:
+        app: watchstate
+    spec:
+      serviceAccountName: default
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        fsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: watchstate
+          # Canonical upstream image; containerd mirrors route ghcr.io via
+          # artifactapi (do NOT prefix with the artifactapi host).
+          image: ghcr.io/arabcoders/watchstate:v1.10.3
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 8080
+              name: http
+              protocol: TCP
+          env:
+            - name: WS_UID
+              value: "1000"
+            - name: WS_GID
+              value: "1000"
+            - name: WS_TZ
+              value: Australia/Sydney
+          volumeMounts:
+            - name: config
+              mountPath: /config
+          livenessProbe:
+            httpGet:
+              path: /v1/api/system/healthcheck
+              port: http
+            initialDelaySeconds: 20
+            periodSeconds: 30
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            httpGet:
+              path: /v1/api/system/healthcheck
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: "1"
+              memory: 1Gi
+      volumes:
+        - name: config
+          persistentVolumeClaim:
+            claimName: watchstate-config
+      restartPolicy: Always

apps/base/watchstate/gateway.yaml

@@ -0,0 +1,38 @@
+---
+# Internal-only front for the WatchState admin UI (cf. pdbmux/logviewer).
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  labels:
+    traefik.io/instance: internal
+  annotations:
+    cert-manager.io/cluster-issuer: vault-issuer
+    cert-manager.io/common-name: watchstate.k8s.syd1.au.unkin.net
+    cert-manager.io/private-key-size: "4096"
+    external-dns.alpha.kubernetes.io/hostname: watchstate.k8s.syd1.au.unkin.net
+    external-dns.alpha.kubernetes.io/target: 198.18.200.4
+  name: watchstate
+  namespace: watchstate
+spec:
+  gatewayClassName: traefik-internal
+  listeners:
+    - allowedRoutes:
+        namespaces:
+          from: Same
+      hostname: watchstate.k8s.syd1.au.unkin.net
+      name: http
+      port: 80
+      protocol: HTTP
+    - allowedRoutes:
+        namespaces:
+          from: Same
+      hostname: watchstate.k8s.syd1.au.unkin.net
+      name: https
+      port: 443
+      protocol: HTTPS
+      tls:
+        certificateRefs:
+          - group: ""
+            kind: Secret
+            name: watchstate-tls
+        mode: Terminate

apps/base/watchstate/httproute.yaml

@@ -0,0 +1,49 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: watchstate-http-redirect
+  namespace: watchstate
+spec:
+  hostnames:
+    - watchstate.k8s.syd1.au.unkin.net
+  parentRefs:
+    - group: gateway.networking.k8s.io
+      kind: Gateway
+      name: watchstate
+      sectionName: http
+  rules:
+    - filters:
+        - type: RequestRedirect
+          requestRedirect:
+            scheme: https
+            statusCode: 301
+      matches:
+        - path:
+            type: PathPrefix
+            value: /
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: watchstate
+  namespace: watchstate
+spec:
+  hostnames:
+    - watchstate.k8s.syd1.au.unkin.net
+  parentRefs:
+    - group: gateway.networking.k8s.io
+      kind: Gateway
+      name: watchstate
+      sectionName: https
+  rules:
+    - backendRefs:
+        - group: ""
+          kind: Service
+          name: watchstate-oauth2
+          port: 80
+          weight: 1
+      matches:
+        - path:
+            type: PathPrefix
+            value: /

apps/base/watchstate/kustomization.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - namespace.yaml
+  - vaultauth.yaml
+  - vaultstaticsecret.yaml
+  - pvc-config.yaml
+  - deployment.yaml
+  - oauth2-proxy-configmap.yaml
+  - oauth2-proxy-deployment.yaml
+  - service.yaml
+  - gateway.yaml
+  - httproute.yaml

apps/base/watchstate/namespace.yaml

@@ -0,0 +1,7 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  labels:
+    app.kubernetes.io/name: watchstate
+  name: watchstate

apps/base/watchstate/oauth2-proxy-configmap.yaml

@@ -0,0 +1,31 @@
+---
+# Non-secret oauth2-proxy configuration (client_id/secret/cookie_secret come
+# from the watchstate-oauth-credentials Secret). Single auth front for the
+# WatchState UI + API: every path requires a valid Authentik session. Access is
+# authorized Authentik-side (the watchstate application binds akR-global-admin
+# only), so no oauth2-proxy group allowlist is set here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: watchstate-oauth2-env
+  namespace: watchstate
+data:
+  OAUTH2_PROXY_HTTP_ADDRESS: "0.0.0.0:4180"
+  OAUTH2_PROXY_PROVIDER: "oidc"
+  OAUTH2_PROXY_OIDC_ISSUER_URL: "https://identity.k8s.syd1.au.unkin.net/application/o/watchstate/"
+  OAUTH2_PROXY_REDIRECT_URL: "https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback"
+  OAUTH2_PROXY_UPSTREAMS: "http://watchstate.watchstate.svc.cluster.local:8080/"
+  OAUTH2_PROXY_SCOPE: "openid email profile ak_groups"
+  OAUTH2_PROXY_OIDC_GROUPS_CLAIM: "ak_groups"
+  OAUTH2_PROXY_PASS_USER_HEADERS: "true"
+  OAUTH2_PROXY_EMAIL_DOMAINS: "*"
+  # Authentik hardcodes email_verified=false in the id_token; authorization is
+  # enforced Authentik-side, so accepting the unverified email is safe.
+  OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL: "true"
+  OAUTH2_PROXY_COOKIE_SECURE: "true"
+  OAUTH2_PROXY_COOKIE_DOMAINS: "watchstate.k8s.syd1.au.unkin.net"
+  OAUTH2_PROXY_WHITELIST_DOMAINS: "watchstate.k8s.syd1.au.unkin.net"
+  OAUTH2_PROXY_REVERSE_PROXY: "true"
+  OAUTH2_PROXY_PROVIDER_CA_FILES: "/etc/ssl/combined/ca-certificates.crt"
+  OAUTH2_PROXY_CODE_CHALLENGE_METHOD: "S256"
+  OAUTH2_PROXY_SKIP_PROVIDER_BUTTON: "true"

apps/base/watchstate/oauth2-proxy-deployment.yaml

@@ -0,0 +1,133 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: watchstate-oauth2
+  namespace: watchstate
+  annotations:
+    configmap.reloader.stakater.com/auto: "true"
+    secret.reloader.stakater.com/reload: "watchstate-oauth-credentials,vault-ca-cert"
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: watchstate-oauth2
+  strategy:
+    rollingUpdate:
+      maxUnavailable: 1
+    type: RollingUpdate
+  template:
+    metadata:
+      labels:
+        app: watchstate-oauth2
+    spec:
+      serviceAccountName: default
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+        # identity.k8s.syd1.au.unkin.net serves a Vault-PKI cert; combine the
+        # system roots with the internal CA so oauth2-proxy's OIDC HTTP client
+        # trusts it.
+        - name: combine-certs
+          image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/library/alpine:3
+          imagePullPolicy: IfNotPresent
+          command:
+            - sh
+            - -c
+            - cat /etc/ssl/certs/ca-certificates.crt /custom-ca/ca.crt > /combined-certs/ca-certificates.crt
+          volumeMounts:
+            - name: vault-ca-cert
+              mountPath: /custom-ca
+              readOnly: true
+            - name: combined-certs
+              mountPath: /combined-certs
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 50m
+              memory: 32Mi
+            limits:
+              cpu: 200m
+              memory: 64Mi
+      containers:
+        - name: oauth2-proxy
+          image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.3
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 4180
+              name: http
+              protocol: TCP
+          envFrom:
+            - configMapRef:
+                name: watchstate-oauth2-env
+                optional: false
+          env:
+            - name: OAUTH2_PROXY_CLIENT_ID
+              valueFrom:
+                secretKeyRef:
+                  name: watchstate-oauth-credentials
+                  key: client_id
+            - name: OAUTH2_PROXY_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: watchstate-oauth-credentials
+                  key: client_secret
+            - name: OAUTH2_PROXY_COOKIE_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: watchstate-oauth-credentials
+                  key: cookie_secret
+          volumeMounts:
+            - name: combined-certs
+              mountPath: /etc/ssl/combined
+              readOnly: true
+          livenessProbe:
+            httpGet:
+              path: /ping
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 30
+            timeoutSeconds: 5
+            failureThreshold: 3
+          readinessProbe:
+            httpGet:
+              path: /ready
+              port: http
+            initialDelaySeconds: 5
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 3
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 500m
+              memory: 256Mi
+      volumes:
+        - name: vault-ca-cert
+          secret:
+            secretName: vault-ca-cert
+            items:
+              - key: ca.crt
+                path: ca.crt
+        - name: combined-certs
+          emptyDir: {}
+      restartPolicy: Always

apps/base/watchstate/pvc-config.yaml

@@ -0,0 +1,18 @@
+---
+# Single WatchState state volume: the sqlite DB, config, and per-backend cache
+# all live under /config. RWO because the Deployment is pinned to replicas: 1
+# with a Recreate strategy (sqlite + the in-container cron/redis are not
+# multi-writer safe).
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: watchstate-config
+  namespace: watchstate
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 5Gi
+  storageClassName: cephrbd-fast-delete
+  volumeMode: Filesystem

apps/base/watchstate/service.yaml

@@ -0,0 +1,36 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: watchstate
+  namespace: watchstate
+spec:
+  internalTrafficPolicy: Cluster
+  ports:
+    - name: http
+      port: 8080
+      protocol: TCP
+      targetPort: http
+  selector:
+    app: watchstate
+  sessionAffinity: None
+  type: ClusterIP
+---
+# Front-door entry Service: the HTTPRoute for watchstate.k8s.syd1.au.unkin.net
+# targets this; all traffic enters via oauth2-proxy.
+apiVersion: v1
+kind: Service
+metadata:
+  name: watchstate-oauth2
+  namespace: watchstate
+spec:
+  internalTrafficPolicy: Cluster
+  ports:
+    - name: http
+      port: 80
+      protocol: TCP
+      targetPort: http
+  selector:
+    app: watchstate-oauth2
+  sessionAffinity: None
+  type: ClusterIP

apps/base/watchstate/vaultauth.yaml

@@ -0,0 +1,20 @@
+---
+apiVersion: secrets.hashicorp.com/v1beta1
+kind: VaultAuth
+metadata:
+  name: default
+  namespace: watchstate
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  allowedNamespaces:
+    - watchstate
+  kubernetes:
+    audiences:
+      - vault
+    role: default
+    serviceAccount: default
+    tokenExpirationSeconds: 600
+  method: kubernetes
+  mount: k8s/au/syd1
+  vaultConnectionRef: vso-system/default

apps/base/watchstate/vaultstaticsecret.yaml

@@ -0,0 +1,21 @@
+---
+# Authentik OIDC client for watchstate (client_id, client_secret, cookie_secret)
+# seeded at kv/kubernetes/namespace/watchstate/default/oauth-credentials; the
+# watchstate/default templated policy grants read, so no terraform-vault change
+# is needed.
+apiVersion: secrets.hashicorp.com/v1beta1
+kind: VaultStaticSecret
+metadata:
+  name: watchstate-oauth-credentials
+  namespace: watchstate
+spec:
+  destination:
+    create: true
+    name: watchstate-oauth-credentials
+    overwrite: true
+  hmacSecretData: true
+  mount: kv
+  path: kubernetes/namespace/watchstate/default/oauth-credentials
+  refreshAfter: 5m
+  type: kv-v2
+  vaultAuthRef: default

apps/overlays/au-syd1/watchstate/kustomization.yaml

@@ -0,0 +1,6 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+resources:
+  - ../../../base/watchstate

argocd/applicationsets/media.yaml

@@ -13,6 +13,7 @@ spec:
       - path: apps/overlays/*/fafflix
       - path: apps/overlays/*/cheeztv
       - path: apps/overlays/*/arrstack
+      - path: apps/overlays/*/watchstate
   template:
     metadata:
       name: 'media-{{path[3]}}'

argocd/projects/media.yaml

@@ -15,6 +15,8 @@ spec:
       server: https://kubernetes.default.svc
     - namespace: 'arrstack'
       server: https://kubernetes.default.svc
+    - namespace: 'watchstate'
+      server: https://kubernetes.default.svc
   clusterResourceWhitelist:
     - group: ''
       kind: Namespace