cheeztv is a second, kids-only Jellyfin instance in the media project. Kids
titles need a separate, safe library front (cheeztv.unkin.net) while staying
resumable in the existing adult instance (fafflix).
- apps/base/cheeztv: mirror the jellyfin (fafflix) HA stack 1:1 - same
jellyfin-ha:v0.1.3 fork image, 2-replica StatefulSet, RWX transcode PVC,
Intel iGPU transcode, CNPG Postgres + PgBouncer, Valkey transcode store,
k8up config backup, VMPodScrape. Media mounts subPath kids on the shared
movies/tv CephFS subvolumes so cheeztv sees only the kids trees.
- Separate state: own namespace, config PVC, cheeztv-postgres cluster,
cheeztv-valkey, and cnpg-cheeztv / cheeztv-config-backup buckets - nothing
shared with fafflix.
- Ingress/DNS: cheeztv.unkin.net (internal Traefik + external-dns at
198.18.200.4, cert-manager cheeztv-tls) plus the cluster hostname variant
cheeztv.k8s.syd1.au.unkin.net, matching how fafflix and logviewer.unkin.net
are wired.
- fafflix: add movies/kids and tvshows/kids subPath mounts alongside its
existing media mounts so kids libraries are browsable/resumable there; its
existing mounts, hostname and ingress are untouched.
- Register cheeztv in the media ApplicationSet generator and AppProject
destinations.
Deploy the new space-pirate UI background released in arrproxy v0.3.1.
- Bump arrproxy-api image to v0.3.1
- Bump arrproxy-ui image to v0.3.1
Reviewed-on: #411
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Tofu applies against authentik hit a cross-replica read-back race: a create lands on one of the 3 server pods and the provider's immediate read-back hits another pod that returns stale data, failing the run with "inconsistent result after apply" (upstream bug goauthentik/terraform-provider-authentik#954, still open). ClientIP session affinity keeps a whole apply run on a single pod, sidestepping the race until the provider is fixed.
## Changes
- Sets `server.service.sessionAffinity: ClientIP` with `sessionAffinityConfig.clientIP.timeoutSeconds: 60` in the au-syd1 authentik overlay values (rendered by the vendored chart's `templates/server/service.yaml`)
Reviewed-on: #410
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The `readonly` profile used by `logreader` sets `readonly=1`, which rejects any per-query settings change. HTTP clients (logviewer, chlog) send settings params like `max_execution_time` with their queries, so every query fails with `Code: 164 DB::Exception: Cannot modify 'max_execution_time' setting in readonly mode` — logviewer healthz returns 503 and CrashLoops, chlog queries fail. `readonly=2` keeps data access read-only while permitting settings changes, which is what HTTP clients need.
## Changes
- Changes the `readonly` profile in `apps/base/logging/clickhouseinstallation.yaml` from `readonly: "1"` to `readonly: "2"`
Reviewed-on: #409
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Both `arrstack-postgres` replicas were OOMKilled at the 1Gi memory limit on 2026-08-23 07:19, causing a failover to timeline 14, leaving replica-3 unrecoverable via pg_rewind, and blocking the argocd sync wave on Cluster Ready=False.
## Changes
- Raises the CNPG Cluster resources to requests 1Gi memory / 250m CPU and limits 2Gi memory / 1 CPU
**NOTE:** applying this restarts the postgres instances (CNPG rolling restart).
Reviewed-on: #408
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Traefik metrics (traefik_entrypoint_requests_total, traefik_service_*, etc.) are now flowing into VictoriaMetrics with the relabeled `traefik_instance` label, but there is no dashboard to view them.
## Changes
- Add `apps/base/grafana/dashboards/traefik.yaml`: GrafanaDashboard CR mirroring the redis.yaml sibling (gzipJson embed, `instanceSelector dashboards: grafana`, `resyncPeriod: 5m`, `allowCrossNamespaceImport: false`)
- Embed the official Traefik Standalone dashboard (grafana.com ID 17346, revision 9), adapted:
- strip `__inputs`/`__requires`/`__elements` and the `DS_PROMETHEUS` datasource variable
- rewrite all prometheus datasource uids to the estate VictoriaMetrics uid `det2y55dac4jkc`
- add an `instance` template variable over `traefik_instance` and filter every query with `traefik_instance=~"$instance"` so traefik-internal / traefik-external are switchable
- chain the entrypoint/service variable queries through the instance filter
- Register the dashboard in `apps/base/grafana/kustomization.yaml`
Validated: gzipJson decode roundtrip, `kustomize build`, kubeconform.
Reviewed-on: #407
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Ship icon grid + jellyfin tile features by upgrading arrproxy UI and API to v0.3.0.
- Bump arrproxy-api image from v0.2.0 to v0.3.0
- Bump arrproxy-ui image from v0.2.0 to v0.3.0
- Verified images built and pushed to artifactapi
Reviewed-on: #406
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
logviewer.unkin.net does not resolve, so the log viewer UI is unreachable even though its Gateway and certificate already exist (from #404).
## Changes
- Adds DNSRecord `logviewer-dns-internal` (A, ttl 600 → 198.18.200.4, the internal traefik gateway VIP) to bind-internal authoritative records, mirroring the existing ghp/identity records
Reviewed-on: #405
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer.
## Changes
- replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel
- renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel
- adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote
- adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules)
- adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules)
## Prerequisite
The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff.
## Notes
- validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
- the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
Reviewed-on: #399
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
No redis/valkey instance managed in this repo exports metrics today — the six caches (artifactapi, authentik, litellm, gitea, netbox, jellyfin) are blind spots in VictoriaMetrics.
## What
- Enables the valkey-operator exporter sidecar on the jellyfin ValkeyCluster (`exporter.enabled: true`); overrides the operator's default bare-dockerhub image (`oliver006/redis_exporter:v1.80.0`) with the artifactapi-proxied pin. The operator manages a dedicated `_exporter` ACL user for it.
- Adds a `metrics-exporter` sidecar (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/oliver006/redis_exporter:v1.89.0`, :9121, port name `metrics`, req 32Mi/25m lim 64Mi/100m) to the five raw Deployments: artifactapi/authentik/litellm redis and gitea/netbox valkey. gitea/netbox sidecars get the same restrictive securityContext as their valkey container.
- Adds the `monitoring.unkin.net/redis-exporter: "true"` opt-in pod label to all five pod templates.
- Adds `apps/base/observability/vmpodscrape-redis.yaml` with two VMPodScrapes (any namespace, port `metrics`): `redis-exporters` selecting the opt-in label, and `valkey-operator-clusters` selecting `app.kubernetes.io/managed-by: valkey-operator` + `app.kubernetes.io/component: valkey-node` — the ValkeyCluster CR has no pod-label passthrough (verified against the v0.5.0 CRD/source), so operator pods cannot carry the opt-in label.
## Notes
- No instance runs with `requirepass`/auth, so no REDIS_PASSWORD wiring is needed on the raw Deployments; the jellyfin exporter auth is operator-managed.
- Validated: `kustomize build` on every touched base + au-syd1 overlay (helm overlays with `--enable-helm`) and `kubeconform -strict -ignore-missing-schemas` (68 valid, 0 invalid).
Reviewed-on: #398
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The logviewer UI gives browser access to the ClickHouse log store (`logreader` user). It must only be reachable by authorized users, so it sits behind an Authentik-gated oauth2-proxy front at https://logviewer.unkin.net, mirroring the arrstack front-door and mediamover/pdbmux gateway patterns.
## Changes
- Adds the logviewer Deployment (1 replica, `docker-internal/logviewer:v0.1.0`) reading ClickHouse at `clickhouse-logs` via the existing `clickhouse-logreader` Secret, `/healthz` probes, nonroot securityContext
- Adds an oauth2-proxy (quay.io/oauth2-proxy/oauth2-proxy:v7.15.3) front with issuer `https://identity.unkin.net/application/o/logviewer/`, `ak_groups` claim, allowed group `akP-logviewer-admin`, internal-CA trust via the combine-certs initContainer
- Syncs kv `kubernetes/namespace/logging/default/oauth-credentials` into the `logviewer-oauth-credentials` Secret via the existing `default` VaultAuth (kv path already seeded; no terraform-vault change needed)
- Adds ClusterIP Services for the app and the oauth2-proxy front door
- Adds internal Gateway + HTTPRoutes for `logviewer.unkin.net` (vault-issuer cert, external-dns -> 198.18.200.4, http->https redirect)
- Registers `logviewer/` in the logging base kustomization
## Dependency
Login requires the Authentik `logviewer` provider/application from terraform-authentik PR #21. This PR is safe to merge first: the VSO secret sync and the deployment come up regardless; OIDC login only works once #21 is applied.
Validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (49/49 valid).
Reviewed-on: #404
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
CLI tools and the logviewer UI need ClickHouse access without write or DDL rights. The only existing app user (`vector`) has `access_management` and write access; this adds a properly scoped read-only login using the same secret-sourcing mechanism.
## Changes
- Adds a `logreader` user to the `logs` ClickHouseInstallation: password hash via `secretKeyRef` to the Vault-synced `clickhouse-logreader` Secret (same mechanism as `vector`), networks `::/0`, `readonly` profile, scoped to the `logs` database only
- Adds a `readonly` profile (`readonly: 1`, same memory/time limits as default)
- Adds a `clickhouse-logreader` VaultStaticSecret mirroring the existing `clickhouse-credentials` object (kv-v2, 5m refresh, VSO-created destination Secret)
## Vault kv
Seeded at `kv/kubernetes/namespace/logging/default/clickhouse-logreader` (version 1) with keys: `username`, `password`, `password_sha256_hex`. No secret material lands in git; the CHI only ever sees the sha256 hash via the synced Secret.
Reviewed-on: #400
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Both traefik instances (traefik-internal/traefik-external, chart 40.0.0, image v3.7.0) already expose prometheus metrics on container port `metrics` (9100), but nothing scrapes them — VictoriaMetrics has no traefik series (verified empty). The observability VMAgent runs with selectAllByDefault and consumes VM*Scrape CRs only, so a VMPodScrape is all that is needed.
## Changes
- Adds `apps/base/observability/vmpodscrape-traefik.yaml`: scrapes port `metrics` /metrics on `app.kubernetes.io/name: traefik` pods in traefik-system, relabeling `app.kubernetes.io/instance` to a stable `traefik_instance` label to split internal vs external series
- Registers the new CR in the observability base kustomization
- Enables `metrics.prometheus.addEntryPointsLabels` and `addRoutersLabels` in both au-syd1 traefik values files for per-entrypoint/per-route dashboards (pods restart on rollout — acceptable)
Validated with `kustomize build` (observability base + overlay, traefik-system overlay with --enable-helm) and kubeconform (unknown CRDs skipped); rendered deployments carry the new metric flags and the VMPodScrape renders in the overlay.
Reviewed-on: #397
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The grafana-operator estate has no dashboard for Redis/Valkey; redis_exporter instances are being added in a parallel PR and need visibility.
## Changes
- Adds `apps/base/grafana/dashboards/redis.yaml`, a GrafanaDashboard CR embedding grafana.com dashboard 763 ("Redis Dashboard for Prometheus Redis Exporter 1.x") as gzipJson like the sibling dashboards, with namespace/instance template variables over `redis_up` for multi-instance use and all datasource refs pinned to the VictoriaMetrics datasource uid (`det2y55dac4jkc`), registered in the grafana kustomization
Reviewed-on: #396
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Activates the arr forks (sonarr/radarr/prowlarr) #14 Redis features — SignalR backplane, cross-replica cache-invalidation bus, and distributed rate limiter — which ship deployed but inert because no Valkey exists in arrstack and nothing is wired to it. Ben wants ONE Valkey shared by all three apps.
## Changes
- Adds a single shared `ValkeyCluster` (`arrstack-valkey`) under `apps/base/arrstack/valkey/`, modeled on `jellyfin-valkey`: `shards:1`/`replicas:2` HA (one primary + two replicas, automatic failover), image via artifactapi (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/valkey/valkey:9.0.0`), operator-default passwordless `default` user, node spread across hosts, cluster-aware PDB, ephemeral `/data`.
- Registers the `valkey` component in the arrstack base kustomization.
- Wires `<App>__Redis__Host` + `<App>__Redis__Port` into the `sonarr-env` / `radarr-env` / `prowlarr-env` ConfigMaps, all pointing at the shared service `valkey-arrstack-valkey.arrstack.svc.cluster.local:6379`.
## Notes
- Setting `Host` is the activation switch: the fork's `RedisOptions.IsConfigured` gates purely on a non-empty `Host`, so there is no separate Enabled flag.
- Passwordless (jellyfin parity — the operator leaves the `default` user without auth), so no `Password`/`Ssl` is wired and there is no secret to reference.
- One cluster is safe for all three: each fork namespaces its keys and pub/sub channels by a per-app prefix (`sonarr:ratelimit:` / `radarr:ratelimit:` / `prowlarr:ratelimit:`), so their state never collides.
- App `image:` (-unkin5), the waitfordb initContainer, and S3 buckets are untouched.
- Validated: `kubectl kustomize` renders clean for both `apps/base/arrstack` and `apps/overlays/au-syd1/arrstack`.
Follow-up: after merge, Valkey must come up and the three apps must roll (pick up the new env) before the #14 features can be validated live.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #394
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Sonarr and radarr currently expose no Prometheus metrics, so queue depth, indexer health, and library stats are invisible to the observability stack. This adds the standard exportarr exporter as a sidecar to both Deployments.
- Adds an exportarr sidecar (ghcr.io/onedr0p/exportarr v2.3.0, pulled through the artifactapi `ghcr` remote, which already whitelists `^onedr0p/`) to the sonarr and radarr Deployments
- Points each exporter at its local replica via env (`URL=http://localhost:8989/sonarr` / `:7878/radarr`, matching the apps' UrlBase) and reuses the existing VSO-synced API-key Secrets (`sonarr-apikey`/`radarr-apikey`, key `apitoken`) via `APIKEY` secretKeyRef — no key in argv
- Serves metrics on 9707 (sonarr) / 9708 (radarr) with `/healthz` liveness+readiness probes and small resources (25m/32Mi requests, 100m/128Mi limits)
- Adds per-app VMPodScrape objects (picked up by the observability VMAgent, selectAllByDefault); pod-level rather than VMServiceScrape because the app Services don't expose the metrics port
Validated with `kustomize build apps/base/arrstack` + kubeconform (0 invalid).
Note: with 3 replicas per app, each pod's exporter reports the same shared-Postgres-backed stats, so series are duplicated across the `pod` label; dashboards should aggregate with `max` or filter to one pod.
Reviewed-on: #395
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Deploys the released mediamover v0.1.0 into the arrstack namespace: a single-replica server (API + UI on :8080) that browses the mediafs source, queues per-file copy/move operations into the media PVCs, and spawns one worker Job per file using its own image — its ServiceAccount/RBAC grants the Job and Pod access that requires. All PVs/PVCs it mounts are already live.
- Adds `apps/base/arrstack/mediamover/` with SA, Role/RoleBinding (batch jobs create/get/list/watch/delete; pods get/list/watch; pods/log get/list), Deployment, Service, Gateway, HTTPRoute
- Runs `mediamover server` with `--src-root /srv/src --src-pvc mediafs`, `--dst-roots movies=/srv/dst/movies,tv=/srv/dst/tv`, `--dst-pvc movies=media-movies,tv=media-tv`, `--namespace arrstack`, `--image ...mediamover:v0.1.0`
- Mounts mediafs RW at /srv/src (move deletes the source) plus media-movies and media-tv destinations; keeps 1 replica with Recreate strategy since the queue is in-memory
- Names the Service `mediamover` on 8080 to match the worker callback default `http://mediamover.arrstack.svc:8080`
- Exposes the UI at https://mediamover.k8s.syd1.au.unkin.net via a dedicated internal Gateway (pdbmux pattern); the existing arrproxy Gateway is external and hostname-locked to arrstack.unkin.net so it cannot carry this route
- Probes hit `GET /api/limit` (the server has no dedicated health endpoint); registers `mediamover` in the arrstack base kustomization
Reviewed-on: #393
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Ben merged the S3 TLS-trust fix into each Servarr fork's `main` (sonarr#10, radarr#11, prowlarr#10, branch `benvin/s3-tls-trust-fix`). The fix corrects TLS trust for the leaf-only certificate presentation of the Ceph RGW endpoint, so backups and MediaCover uploads to S3 succeed across all replicas instead of failing the TLS handshake.
The `-unkin5` images carrying the fix are built and pushed to `artifactapi.k8s.syd1.au.unkin.net/docker-internal/<app>`. This PR rolls them out.
## Changes
- Bump `apps/base/arrstack/sonarr/deployment.yaml` app image `v5.0.0-unkin4` -> `v5.0.0-unkin5`
- Bump `apps/base/arrstack/radarr/deployment.yaml` app image `v6.4.2-unkin4` -> `v6.4.2-unkin5`
- Bump `apps/base/arrstack/prowlarr/deployment.yaml` app image `v2.6.2-unkin4` -> `v2.6.2-unkin5`
Only the application container `image:` line changes in each file; the `waitfordb` initContainer and bucket YAML are untouched.
Reviewed-on: #392
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Exposes the legacy mediafs CephFS filesystem to the arrstack namespace so the upcoming mediamover tool can migrate media and run bandwidth tests against the old filesystem; the ceph-mediafs client secret is already seeded in Vault. The mover deployment itself lands in a follow-up PR.
- Adds a `ceph-mediafs-secret` VaultStaticSecret in csi-cephfs syncing `kv/service/kubernetes/au/syd1/csi/ceph-mediafs-secret`
- Adds a 10Ti static RWX PV `arrstack-mediafs` (fsName `mediafs`, rootPath `/`, staged with the mediafs secret) pinned by claimRef
- Adds the matching `mediafs` PVC in arrstack (storageClassName "", volumeName-bound, backups excluded)
- Registers all three in their base kustomizations
Reviewed-on: #391
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The cluster runs the arrstack forks at `-unkin3`. Ben merged two workstreams into each fork's `main`:
- **Redis SignalR backplane + cross-replica cache invalidation** — SignalR messages and cache-invalidation events fan out across replicas via Redis, so any replica serves consistent live UI/state.
- **MediaCover -> S3 + backups -> S3** — MediaCover (sonarr/radarr) and backups (all three) use an optional S3 store, so any stateless replica can serve covers and write/restore backups.
Fresh `-unkin4` tags were cut on each fork's `main` HEAD, building `docker-internal/<app>:<base>-unkin4` images. This bumps the deployment images so ArgoCD rolls the merged work into the cluster.
## Changes
- Bump sonarr app image `v5.0.0-unkin3` -> `v5.0.0-unkin4`
- Bump radarr app image `v6.4.2-unkin3` -> `v6.4.2-unkin4`
- Bump prowlarr app image `v2.6.2-unkin3` -> `v2.6.2-unkin4`
The waitfordb initContainer and bucket resources are unchanged; only the app-container `image:` tags move.
Reviewed-on: #390
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Servarr backups (periodic config+DB zips) and sonarr MediaCover posters still live on each replica's local `/config` emptyDir: backups are leader-local and lost on reschedule, and sonarr covers render inconsistently across the 3 active-active replicas. #387 moved radarr covers to a shared Ceph RGW bucket; this extends that pattern to backups for all three apps and to sonarr's covers.
## Changes
- Add an `arrstack-backups` ObjectStoreUser + Bucket (cephrgw-operator); the operator mints the `arrstack-backups-s3` credential Secret in-namespace (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / S3_ENDPOINT), so no Vault KV seeding is required. Backups get their own bucket (not `arrstack-media`) because they carry secrets (Config.xml API keys, the DB) and warrant separate credentials/lifecycle from poster art; the three apps share it, isolated by a per-app key prefix.
- Wire sonarr to the shared `arrstack-media` bucket via `Sonarr__MediaCoverS3__*` (prefix `sonarr`, partitioned from radarr's `radarr` prefix to avoid movie/series id collisions), plus the estate CA mount at `/etc/ssl/vault-ca/ca.crt` for RGW TLS validation.
- Wire radarr/sonarr/prowlarr backups via `<App>__BackupS3__*` to `arrstack-backups` (per-app prefix); add the CA mount to sonarr and prowlarr.
## Notes
- Credentials come only from the operator-minted Secret via env; no secrets in manifests.
- The env keys are consumed by the -unkin3+ images (separate app PRs) and ignored by older images, so this is safe to apply ahead of the image bump.
- `kubectl kustomize apps/base/arrstack` builds clean.
Reviewed-on: #389
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Deploy the merged active-active work now on each Servarr fork's main. The cluster still runs the -unkin2 images, so the merged code (K8s process lifecycle, Redis-backed distributed rate limiting, and radarr MediaCover->S3) is not yet deployed. The -unkin3 tag pipelines have built and pushed new images to docker-internal.
Changes:
- Bump sonarr app image v5.0.0-unkin2 -> v5.0.0-unkin3
- Bump radarr app image v6.4.2-unkin2 -> v6.4.2-unkin3
- Bump prowlarr app image v2.6.2-unkin2 -> v2.6.2-unkin3
Touches only the app container image line in each apps/base/arrstack/<app>/deployment.yaml; the waitfordb initContainer is unchanged.
Reviewed-on: #388
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Servarr MediaCover assets (movie/series posters and fanart) and app backups currently live on each replica's local `/config`, which is an `emptyDir`. A poster the leader downloads is invisible to the other replicas behind the Service, so covers render inconsistently and backups are leader-local. This provisions a shared Ceph RGW (S3) bucket to hold those assets instead.
## Changes
- Add an `arrstack-media` `ObjectStoreUser` + `Bucket` (cephrgw-operator), mirroring the CNPG backup pattern. The operator mints the `arrstack-media-s3` credential Secret in-namespace (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `S3_ENDPOINT`), so **no Vault KV seeding is required**.
- Wire the radarr Deployment to the bucket via `Radarr__MediaCover__S3__*` env (creds from the operator Secret; estate CA mounted at `/etc/ssl/vault-ca/ca.crt` for RGW TLS validation).
## Notes
- The env keys are unknown to the current `-unkin2` image and are ignored by it; they activate once the `-unkin3+` image (radarr MediaCover-S3 support, separate radarr PR) ships. sonarr/prowlarr wiring and backups-to-S3 follow in later PRs.
- `kubectl kustomize apps/base/arrstack` builds clean.
Reviewed-on: #387
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The sonarr/radarr/prowlarr deployments gated app startup on their Postgres
database being reachable using a hand-written `psql`-in-a-shell loop running
the `postgres:17-alpine` image. Now that `unkin/waitfordb` is released
(v0.1.0), replace that inline probe with the purpose-built tool so the wait
logic lives in one tested place instead of being copy-pasted per workload.
`waitfordb` reads the same `PG*` env as a libpq fallback and exits `0` once
`SELECT 1` succeeds under those credentials/database, so the retrofit is a
drop-in — the password still never lands in argv.
## Changes
- Swap the `wait-for-db` initContainer image from `dockerhub/library/postgres:17-alpine`
to `docker-internal/waitfordb:v0.1.0` (pinned, not `latest`) in all three
arrstack deployments.
- Drop the `sh -c` psql loop `command` — `waitfordb` is the image entrypoint.
- Keep the existing `PGHOST/PGPORT/PGDATABASE/PGUSER/PGPASSWORD` env, the
`wait-for-db` name, and the resource requests/limits unchanged.
- Add `WAITFORDB_TIMEOUT=5m` so a stuck database fails the pod fast, and
`WAITFORDB_SSLMODE=disable` to match the in-cluster Postgres.
Validated with `kubectl kustomize apps/overlays/au-syd1/arrstack` (builds clean;
3 waitfordb refs, no remaining postgres init image).
Reviewed-on: #386
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The active-active arrstack deploy (#383) is stuck in-cluster: the app Deployments (sync-wave 0) can never become Healthy without their Postgres DB, so ArgoCD never advances to create the shared `arrstack-postgres` cluster (wave 1). Separately, `/config` is an immutable RWO cephrbd PVC, so 3 replicas fail with Multi-Attach.
## How
- Order DB infra before the apps: `<app>-db` VaultStaticSecrets + backup ObjectStoreUser/Bucket at sync-wave -3, CNPG `Cluster` -2, `Database` CRDs -1; app Deployments stay at default 0.
- Switch each app's `/config` to an emptyDir per-pod volume (library DB is in Postgres, config is env-driven; MediaCover regenerates). Old RWO config PVCs orphan.
Validated with kustomize build (62 resources). arrproxy/nzbget untouched.
Reviewed-on: #385
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The future Vault engine needs to machine-mint arrproxy tokens without an interactive Authentik session. arrproxy v0.2.0 adds an admin route (`POST /api/admin/...`) protected by its own bearer token; OpenBao running on the VMs calls it through the arrstack ingress. This deploys that route.
## Changes
- Add an `arrproxy-admin-token` VaultStaticSecret (mirrors the `arrproxy-pepper` VSO: same `default` VaultAuth, k8s-auth default-SA pattern) syncing `kv/kubernetes/namespace/arrstack/default/arrproxy-admin-token` into the `arrproxy-admin-token` Secret. The token is seeded in Vault KV as the shared source of truth for the future Vault engine.
- Set `ARRPROXY_ADMIN_TOKEN` on the arrproxy-api Deployment via `secretKeyRef` (key `token`), and add `arrproxy-admin-token` to the Reloader `secret.reloader.stakater.com/reload` annotation so the pod rolls on rotation.
- Skip-auth the `/api/admin/` route in oauth2-proxy: `OAUTH2_PROXY_SKIP_AUTH_REGEX` becomes `^/[^/]+/api,^/api/admin/`. The admin route is intentionally oauth-skipped because it is protected by arrproxy's OWN bearer token so OpenBao can reach it machine-to-machine. `/api/tokens` and `/api/me` are NOT matched and stay oauth-gated. `/api/admin/*` already routes to the arrproxy-api upstream via the existing catch-all `/api/` upstream, so no upstream change is needed.
- Bump arrproxy-api and arrproxy-ui images to `v0.2.0` (kept in lockstep).
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` succeeds.
- pre-commit passes, including the plain-Secret guard (only the VSO CRD is used; no plain Secret objects added).
Reviewed-on: #384
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The arrstack `sonarr`/`radarr`/`prowlarr` apps currently run as single-replica upstream LinuxServer images on SQLite (RWO config PVC, `Recreate` strategy) — no HA, no horizontal scale, and a hard one-pod ceiling because SQLite can't be shared. Our `-unkin2` forks move all app state into PostgreSQL and use Postgres advisory locks for cross-replica coordination (migrations, scheduled tasks, command execution), which makes an **active-active multi-replica** deployment safe.
This is a fresh Postgres backend — **no SQLite data migration**. The current single-replica pods are throwaway baselines; the new pods start on an empty per-app database and the fork's provider migrates the schema on first start. `arrproxy` (its own `arrproxy-db`) and `nzbget` are untouched.
## What
New `apps/base/arrstack/postgres/`:
- `cnpg_cluster.yaml` — CNPG `Cluster` `arrstack-postgres` (3 instances, `cephrbd-fast-delete`), throwaway `app` initdb owner, and 3 `managed.roles` (`sonarr`/`radarr`/`prowlarr`, login non-superuser) each with `passwordSecret: <app>-db`.
- `database-{sonarr,radarr,prowlarr}.yaml` — CNPG `Database` `<app>-main` owned by `<app>`, `databaseReclaimPolicy: retain`.
- `cnpg_backup.yaml` — cephrgw `ObjectStoreUser` + `Bucket cnpg-arrstack` + nightly `ScheduledBackup` (mirrors jellyfin).
- `vaultstaticsecret.yaml` — 3 VSO `VaultStaticSecret`s syncing `<app>-db` (keys `username`/`password`) from `kv/kubernetes/namespace/arrstack/default/<app>-db` (already seeded); these are both the CNPG role `passwordSecret` and the Deployment's Postgres creds.
- sync-waves: `<app>-db` Secrets (0) → Cluster (1) → Databases (2).
- added `postgres` to `apps/base/arrstack/kustomization.yaml`.
Per app (`sonarr`/`radarr`/`prowlarr`):
- `deployment.yaml`: image → `docker-internal/<app>:v<base>-unkin2` (sonarr `v5.0.0`, radarr `v6.4.2`, prowlarr `v2.6.2`); `replicas: 3`; `RollingUpdate`. Removed the `apikey-init` initContainer and `PUID`/`PGID`/`TZ`. Runs the binary directly (`command: [/app/<Bin>]`, `args: [-nobrowser, -data=/config, -nosingleinstancecheck]`). Env from the new ConfigMap + `secretKeyRef`: `<App>__Postgres__User`/`__Password` (Secret `<app>-db`) and `<App>__Auth__ApiKey` (existing `<app>-apikey`, key `apitoken`). `/<app>/ping` probes kept. `securityContext` runAsUser/runAsGroup/fsGroup 1000 for CephFS group-write.
- new `configmap.yaml`: non-secret `<App>__…` env (Postgres Host/Port/MainDb, `Log__DbEnabled=false`, `Auth__Method=External`, `Auth__Required=DisabledForLocalAddresses`, `App__InstanceName`, `Server__Port`, `Server__UrlBase=/<app>`, `Update__Mechanism=External`).
- `pvc-config.yaml` → RWX `cephfs-raid5-retain` (shared `/config` for `config.xml`/MediaCover).
- unchanged: `service.yaml`, `gateway.yaml`, `httproute.yaml`, apikey `vaultstaticsecret.yaml` — arrproxy still addresses the Service VIP, so N replicas are transparent.
## Validation
`kustomize build apps/overlays/au-syd1/arrstack` clean; `kubeconform` **64/64 valid**; `yamllint` and the repo pre-commit (incl. no-plain-secrets) pass. Runtime validation (pods Ready, single migrate under advisory lock, scheduled-task/command single-owner, UI via `https://arrstack.unkin.net/<app>/`, 3→1→3 rescale) is post-merge once ArgoCD syncs.
## Decisions / unknowns
- **Fork image UID/fsGroup**: the `-unkin2` images have no `USER` (run as root); pinned `runAsUser`/`runAsGroup`/`fsGroup: 1000` with `fsGroupChangePolicy: OnRootMismatch`. Verify group-write on the shared CephFS `/config` on first deploy.
- **Config storage**: chose RWX `cephfs-raid5-retain` shared `/config` (per spec preference) over per-pod `emptyDir`.
- **Postgres storage/version/resources**: mirrored jellyfin/arrproxy CNPG conventions — `cephrbd-fast-delete` 10Gi, PG17, 3 instances.
Reviewed-on: #383
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The *arr UIs reached through **arrproxy** were prompting **"Authentication Required"**. Sonarr v4 (and Radarr/Prowlarr on the same Servarr auth code) refuses remote access when `AuthenticationMethod=None`, so the web UI kept demanding a login even though **arrproxy + oauth2-proxy already authenticate every user at the front door**. That is a double prompt with no purpose.
## Change
Extend the existing `apikey-init` container for **sonarr / radarr / prowlarr** to also idempotently enforce, in `/config/config.xml`:
- `<AuthenticationMethod>External</AuthenticationMethod>`
- `<AuthenticationRequired>Enabled</AuthenticationRequired>`
Create-or-replace both elements (same idempotent sed/printf pattern already used for `<ApiKey>`/`<UrlBase>`), without disturbing ApiKey or UrlBase. Pods use `Recreate`, so they roll and re-run the init container on apply.
## Why this is the correct, header-less fix
In the Servarr v4 source, `AddExternal()` registers the **identical `NoAuthenticationHandler` as `AddNone()`** — `External` requires **no** username header (no `X-Forwarded-User` / `Remote-User`). It differs from `None` only in that it is **exempt from the None remote-access block**, so the UI stops prompting while remote access is permitted.
This matters because arrproxy's `trustBoundary` deliberately **strips all inbound identity headers** and forwards only the real `X-Api-Key` to the upstream *arr — it forwards no username header. Because `External` needs none, that stripping is irrelevant and **no arrproxy change is required**. The API path is unaffected (arrproxy injects the real key; *arr API auth is key-based regardless of AuthenticationMethod).
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` → OK (rendered init carries the auth logic for all 3 apps)
- pre-commit (yamllint + all hooks) → Passed
Versions in scope: sonarr 4.0.19, radarr 6.3.0, prowlarr 2.5.2 (all share the Servarr v4 auth handler).
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #382
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Authentik hardcodes email_verified=false in the id_token it issues for the arrstack application. oauth2-proxy rejects such sessions by default ("email ... isnt verified"), so the /oauth2/callback returns a 500 and login fails.
Setting OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true tells oauth2-proxy to accept the id_token and complete the callback. This is safe: authorization is not derived from email verification here -- access is enforced downstream via the ak_groups claim (X-Forwarded-Groups), so accepting the unverified email only lets the OIDC session establish.
Restart mechanism: the arrproxy-oauth2 Deployment already carries configmap.reloader.stakater.com/auto=true and Reloader is deployed/running, so the ConfigMap change rolls the pods automatically after ArgoCD syncs.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #381
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Repoints the arrproxy `oauth2-proxy` sidecar image to the real upstream address `quay.io/oauth2-proxy/oauth2-proxy:v7.15.3`, replacing the one-off artifactapi copy (`artifactapi.k8s.syd1.au.unkin.net/ghcr/oauth2-proxy/oauth2-proxy:v7.15.3`).
## Why
Pull the image from its canonical upstream location instead of a hand-copied artifactapi path. containerd on rke2 transparently mirrors `quay.io` through artifactapi, so the real address resolves without hardcoding the proxy host.
## Change
- `apps/base/arrstack/arrproxy/oauth2-proxy-deployment.yaml`: image only -> `quay.io/oauth2-proxy/oauth2-proxy:v7.15.3`.
Validated with `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` and pre-commit.
## Dependencies (must be in place before merge)
1. **rke2 `registries.yaml` mirrors quay.io** — already configured in puppet (`hieradata/roles/infra/k8s.yaml`, `rke2::registries` rewrites `quay.io` -> `artifactapi.k8s.syd1.au.unkin.net/quay/$1`). No change needed.
2. **artifactapi quay remote allowlists oauth2-proxy** — terraform-artifactapi PR unkin/terraform-artifactapi#35 adds `^oauth2-proxy/` to the quay remote patterns. **This PR must NOT be merged until #35 is applied**, otherwise the caching proxy rejects `oauth2-proxy/*` and the sidecar fails to pull.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #380
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
## Why
arrproxy forwards `arrstack.unkin.net/<app>/...` to each *arr Service **preserving** the `/<app>` prefix. Without a matching `<UrlBase>` the apps 307-redirect to `/`, breaking path-based hosting. Prerequisite for arrproxy path routing (PR #377... arrproxy service account added in #376).
## Change
Extend the existing idempotent `apikey-init` init container in sonarr/radarr/prowlarr to also enforce `<UrlBase>/<app></UrlBase>` in `/config/config.xml` (sonarr=/sonarr, radarr=/radarr, prowlarr=/prowlarr):
- Set/replace `<UrlBase>` if present, insert after `<Config>` if absent, or write both `<ApiKey>` + `<UrlBase>` when creating a fresh config. `<ApiKey>` handling is unchanged.
- Because Servarr moves all routes (including `/ping`) under the URL base, update liveness/readiness probes from `/ping` to `/<app>/ping` so they don't 404 once UrlBase is set.
nzbget is not fronted by arrproxy and is left untouched.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` succeeds
- `pre-commit` (yamllint etc.) passes
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #378
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Deploys arrproxy v0.1.0 (task 9 final integration) into the existing `arrstack` app: the external, Authentik-gated front door for sonarr/radarr/prowlarr with per-user API-key brokering. Manifests are wired to match the arrproxy v0.1.0 code (ports, routes, identity headers, keys dir), mirroring the ghp/artifactapi estate patterns.
## Topology (`apps/base/arrstack/arrproxy/`)
- **oauth2-proxy** Deployment + Service `arrproxy` is the single front (OIDC against the Authentik `arrstack` app). It path-routes via `--upstreams` to:
- `arrproxy-ui` for `/` and static SPA assets,
- `arrproxy-api` for `/api/*` (token API) and `/sonarr|/radarr|/prowlarr` (the *arr proxy).
- `--skip-auth-regex=^/[^/]+/api` bypasses oauth **only** for the *arr proxy API (`/sonarr/api...`), while `/api/tokens`, `/api/me` and the UI stay authenticated.
- Identity+groups reach the api via `--pass-user-headers` (`X-Forwarded-{User,Email,Groups}`). The api reads `ARRPROXY_GROUPS_HEADER=X-Forwarded-Groups` — `--set-xauthrequest` only populates auth_request *response* headers and never reaches an `--upstreams` backend, so it is intentionally not used.
- oauth2-proxy trusts `identity.unkin.net` (internal Vault-PKI CA) via a `combine-certs` initContainer (system roots + `vault-ca-cert`), same pattern as artifactapi.
## Data + secrets
- **CNPG** token store `arrproxy-db` (2 instances, cephrgw S3 backups + nightly ScheduledBackup). arrproxy-api does **not** self-migrate, so a wave-1 ArgoCD `Sync` hook Job applies `migrations/0001_init.sql` (mirrored into a ConfigMap) as the CNPG app user. DSN built from the generated `arrproxy-db-app` Secret.
- **VaultStaticSecrets** (arrstack `default` SA / templated role — no terraform-vault change): `arrproxy-pepper` (seeded) and `oauth-credentials`. The three real *arr keys are **reused** from the existing `<app>-apikey` Secrets, projected one file per app into `/etc/arrproxy/keys`.
## Exposure
- **Gateway** `traefik-external`, hostname `arrstack.unkin.net`, cert-manager `vault-issuer` TLS (`arrproxy-gateway-tls`); HTTPRoute → the oauth2-proxy entry Service (+ HTTP→HTTPS redirect).
- Adds the `arrstack.unkin.net` apex A record → external DMZ VIP `198.18.199.0` to the bind-operator `unkin.net` zone (`apps/base/bind-internal/authoritative/records.yaml`), mirroring the ghp/identity apex records.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly; `kubeconform` = 51/51 valid; `pre-commit` passes (no plain Secrets).
## Prerequisites / flags for end-to-end function (NOT in this PR)
- **(a)** Each *arr must run with `UrlBase=/<app>` (path-based hosting) — arrproxy preserves the `/<app>` prefix upstream.
- **(b)** The Authentik `arrstack` OIDC app (terraform-authentik PR #18) must be applied, with redirect URI `https://arrstack.unkin.net/oauth2/callback` registered and the `ak_groups` scope emitted. Confirm the discovery slug matches `OAUTH2_PROXY_OIDC_ISSUER_URL` (`https://identity.unkin.net/application/o/arrstack/`).
- **(c)** Browser XHR to `/<app>/api` is oauth-**bypassed** (skip-auth-regex), so the SPA-driven proxy calls that rely on the session (no per-user token) will not carry identity on that path — expected per the arrproxy trust model; token-based *arr clients are unaffected.
- **NetworkPolicy:** the arrproxy-api Service is not exposed via any HTTPRoute and the api strips forged identity headers in-code, but the README recommends a NetworkPolicy restricting api ingress to oauth2-proxy. The estate currently has **zero** NetworkPolicies, so one was not added here to avoid being the first (and risking probe breakage) before the CNI’s policy/probe behavior is confirmed — flagged as follow-up hardening.
- Please confirm: CNPG size (2 instances / 10Gi), backups on, hostname/VIP, and UI theme.
ARRPROXY_PEPPER was seeded via the agents AppRole at `kv/kubernetes/namespace/arrstack/default/arrproxy-pepper` (key `pepper`).
Reviewed-on: #377
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Jellyfin runs 2 replicas that coordinate distributed transcoding through the operator-managed `jellyfin-valkey` ValkeyCluster, but it was single-instance (shards:1, replicas:0) with no failover, so a node/pod loss would drop the shared transcode-lease state. This promotes that store to an HA topology.
Changes:
- Set replicas:2 (shards:1 unchanged) so the shard group is one primary + two replicas; losing the primary triggers an automatic in-cluster failover to a replica.
- Add scheduling.node.spread.shard.mode:Required so the three ValkeyNodes land on distinct hosts (kubernetes.io/hostname anti-affinity) and one host loss removes at most one node.
- Set podDisruptionBudget.mode:Cluster so the operator manages a quorum-aware PDB.
- Keep persistence omitted (emptyDir): lease state is ephemeral (30s TTLs), replication+failover already provide redundancy and a replaced node re-syncs from the primary, and an operator-managed PVC cannot carry the k8up.io/backup:"false" annotation the namespace k8up Schedule needs to skip in-use RWO volumes.
- Bump per-node memory limit to 512Mi (unchanged) / requests unchanged; resources are per ValkeyNode.
Connection string is unchanged: still cluster-mode via the single headless service `valkey-jellyfin-valkey:6379`; StackExchange.Redis seeds off it and auto-discovers topology and failovers.
kustomize build --enable-helm apps/overlays/au-syd1/jellyfin renders cleanly; pre-commit passes.
Reviewed-on: #375
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
jellyfin-ha v0.1.2 crashed on startup: the framework-dependent .NET 9 build could not find its runtime (`Microsoft.NETCore.App 9.0.0` required, `10.0.11` found), because the runtime image was pinned to a .NET 10 base. jellyfin-ha PR #7 realigned the runtime image to `mcr.microsoft.com/dotnet/aspnet:9.0` and v0.1.3 was released.
Verified before this bump: `docker run --entrypoint dotnet ...:v0.1.3 --list-runtimes` lists `Microsoft.NETCore.App 9.0.19`, and `/jellyfin/jellyfin --version` prints `Jellyfin.Server 10.11.7.0` with no framework-not-found error.
## Change
- Bump jellyfin image tag from v0.1.2 to v0.1.3 in `apps/base/jellyfin/statefulset.yaml`.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #374
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Bumps the jellyfin StatefulSet image tag to v0.1.2.
v0.1.2 is the first jellyfin-ha image successfully pushed to the artifactapi docker-internal registry: pipeline #7 (tag v0.1.2) is green, the docker build+push step exited 0, and the image is pullable (tags/list shows v0.1.2, manifest GET returns 200). v0.1.0 and v0.1.1 tag pipelines failed the CA/TLS check before the plugin-image + buildkit_config fix landed (PR #6).
Changes:
- Set apps/base/jellyfin/statefulset.yaml jellyfin container image tag from v0.1.0 to v0.1.2 (tag only).
Validated with kustomize build --enable-helm apps/overlays/au-syd1/jellyfin and pre-commit.
---------
Co-authored-by: unkin-agent <unkin-agent@users.noreply.git.unkin.net>
Reviewed-on: #373
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Why: P4 step 3. Now that the CA-baked buildx plugin lets `docker-internal/ghp` push succeed, deploy the in-house ghp fork (`v0.20.0-unkin4`) which adds static admin service tokens — the mechanism the forthcoming Vault ghp secrets engine uses to authenticate as admin and mint scoped tokens.
How:
- image: ghcr.io/goodtune/ghp:0.20.0 -> artifactapi.k8s.syd1.au.unkin.net/docker-internal/ghp:v0.20.0-unkin4 (a LOCAL registry image, not the ghcr pull-through that previously ImagePullBackOff-ed).
- add env GHP_AUTH_SERVICE_TOKENS from Secret ghp-app key `service_token`, `optional: true` — ghp runs unchanged until the token is seeded.
Requires (Ben, runbook): seed `service_token` into kv/kubernetes/namespace/ghp/default/app (same pattern as encryption_key/github-app). The default role's templated policy already grants read; no terraform-vault change. The existing ghp-app reloader annotation rolls the Deployment once the key appears. This same token value later becomes the Vault ghp engine admin_token (P4 steps 5-6).
Reviewed-on: #372
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: pushing in-house images to the internal-CA artifactapi registry fails `x509: certificate signed by unknown authority` with the upstream buildx plugin (and `:latest-insecure` does not fix it — buildkit still verifies the registry cert). The fix is the CA-baked plugin image now published at `artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx` (unkin/plugin-docker-buildx v0.1.0).
How: add that image (bare + `:latest`) to `WOODPECKER_PLUGINS_PRIVILEGED` in the au-syd1 overlay so it may run privileged. Existing upstream entries kept so current repos are unaffected until they switch over.
Next: ghp `.woodpecker` switches its buildx step to this image, then ghp re-tags and its push succeeds.
Reviewed-on: #371
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The nzbget pod in namespace `arrstack` is stuck in `Init:ImagePullBackOff`.
## Why
The deployment pins `artifactapi.k8s.syd1.au.unkin.net/dockerhub/linuxserver/nzbget:v26.2` on both the `seed-config` initContainer and the main `nzbget` container. linuxserver does not publish a bare `v26.2` tag - the pull fails with:
```
failed to resolve reference ".../linuxserver/nzbget:v26.2": ... not found
Error: ImagePullBackOff
```
Confirmed against the artifactapi dockerhub mirror: `manifests/v26.2` -> HTTP 404. linuxserver version-pins nzbget under the `version-v<MAJOR>.<MINOR>` scheme, and the immutable date tags `26.2.<date>`. `manifests/version-v26.2` -> HTTP 200.
## Change
- Set both nzbget images (initContainer + main container) to `version-v26.2`, keeping nzbget on the intended 26.2 line and matching the version-pin convention used by the sibling sonarr (`4.0.19`) / radarr (`6.3.0`) deployments.
No other change. The `seed-config` init logic is unaffected - it already mirrors linuxserver's own init (`cp /app/nzbget/share/nzbget/nzbget.conf /config/nzbget.conf`), and since it seeds `/config/nzbget.conf` before the main container, linuxserver's init copy is skipped (no conflict). Instant-move download layout on the media PVCs is preserved.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly (both images -> `version-v26.2`, no stale `v26.2`).
- `pre-commit run --files apps/base/arrstack/nzbget/deployment.yaml` passes (yamllint et al).
Scoped to `apps/base/arrstack/nzbget/deployment.yaml` only; the top-level `apps/base/arrstack/kustomization.yaml` was not touched (another agent is editing sonarr/radarr/prowlarr on a separate branch).
---------
Co-authored-by: unkin-agent <unkin-agent@users.noreply.git.unkin.net>
Reviewed-on: #370
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
## Why
The sonarr/radarr/prowlarr images self-generate an API key into
/config/config.xml on first boot, so the key is unmanaged and differs per
volume reset. This makes Vault the source of truth for those keys (override
bootstrap, chosen by Ben): the key is minted in Vault and enforced into
config.xml before each app starts.
## Changes
- Add a `VaultAuth` `default` in the `arrstack` namespace (kubernetes auth,
mount `k8s/au/syd1`, role `default`, SA `default`), mirroring jellyfin.
- Add a per-app `VaultStaticSecret` that syncs
`kv/kubernetes/namespace/arrstack/default/<app>` (key `apitoken`) into the
`<app>-apikey` Secret. The `default` k8s role's templated policy already
grants read on that path for the `arrstack/default` SA, so no
terraform-vault change is needed.
- Add an `apikey-init` initContainer to each of the three deployments that
reads `API_KEY` from the VSO-created Secret, fails closed on a missing or
non-hex value, and writes/updates only the `<ApiKey>` element in
`/config/config.xml` (then fixes ownership 1000:1000, mode 600). Image is a
pinned busybox via artifactapi to keep this PR atomic (no new image
dependency).
- Wire the new manifests into the base and per-app kustomizations.
## Notes
- Keys already seeded in Vault at `kv/kubernetes/namespace/arrstack/default/<app>`.
- nzbget is out of scope: it has no config.xml `<ApiKey>` (uses ControlPassword),
a separate follow-up.
- Downstream consumers (proxy, terraform) currently read
`kv/service/media-apps/<app>`; the authoritative key now lives at the path
above. Reconciliation is deferred.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #369
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Why: the new unkin/plugin-docker-buildx repo's pipelines run their buildah build/push step under `serviceAccountName: plugin-docker-buildx`, which must exist in the woodpecker namespace.
How: add serviceaccount_plugin_docker_buildx.yaml + wire it into kustomization.yaml, mirroring the existing per-repo woodpecker SAs.
Pairs with unkin/plugin-docker-buildx#1.
Reviewed-on: #368
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Stands up the download-automation stack (sonarr, radarr, prowlarr, nzbget) in a new `arrstack` namespace, sharing the same CephFS TV/movies subvolumes that jellyfin serves read-only (jellyfin static-PV rework is on the base branch, PR #237). Downloads and libraries live on one filesystem per media type so the arrs import with atomic hardlink moves.
## Changes
- Add `apps/base/arrstack/` namespace with its own static CephFS PVs (`arrstack-media-tv`, `arrstack-media-movies`) + RWX PVCs (`media-tv`, `media-movies`) bound to the same rootPaths jellyfin uses; each namespace gets a unique PV name/volumeHandle pinned by claimRef.
- Deploy sonarr (mounts media-tv RW), radarr (media-movies RW), prowlarr (config only), nzbget (both media PVCs RW); single replica, Recreate strategy, LinuxServer images via artifactapi dockerhub remote, PUID/PGID/TZ, probes, fsGroup for CephFS writability.
- Per-app config PVCs on `cephrbd-fast-retain` (RWO, 5Gi) for the SQLite state; internal-only Gateway + HTTPRoute per app (traefik-internal, vault-issuer, external-dns `<app>.k8s.syd1.au.unkin.net`), no oauth yet.
- nzbget: initContainer seeds `/config/nzbget.conf` (copy image template if absent, append managed path/category block once) and creates the media dirs; category `tv` -> `/media/tv/downloads`, `movies` -> `/media/movies/downloads`.
- Wire ArgoCD: add `apps/overlays/*/arrstack` to the media ApplicationSet. (media AppProject already gains the arrstack destination + PersistentVolume whitelist on the base branch.)
## Depends on
terraform-artifactapi PR #31 (allowlist `^linuxserver/` on the dockerhub remote) must be merged + applied before these pods can pull.
## Follow-up (terraform)
Root folders (/media/tv/series, /media/movies/films), download-client wiring (host `nzbget`, port 6789, categories tv/movies) and API keys are configured later via terraform-arr.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Co-authored-by: Ben Vin <neotheo@gmail.com>
Reviewed-on: #366
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Jellyfin depended on a self-managed `redis` Deployment for its distributed transcode-session store. Now that the valkey-operator is deployed cluster-wide, this hands that store to the operator so it is declaratively managed (rolling upgrades, health, future HA) instead of a hand-rolled Deployment.
## Changes
- Add `apps/base/jellyfin/valkey.yaml`: a `ValkeyCluster` (`valkey.io/v1alpha1`) named `jellyfin-valkey`, single instance (`shards: 1`, `replicas: 0`), image via the artifactapi dockerhub mirror (`valkey/valkey:9.0.0`), metrics exporter disabled, redis-parity resources.
- Omit persistence so `/data` is an `emptyDir`: the transcode-lease state is ephemeral, and the operator-managed PVC cannot carry the `k8up.io/backup: "false"` annotation the namespace-wide k8up Schedule needs to skip an in-use RWO volume (the old redis PVC carried it).
- Remove `redis-deployment.yaml`, `redis-service.yaml`, `redis-pvc.yaml` and drop them from `kustomization.yaml`; add `valkey.yaml`.
- Repoint `Jellyfin__TranscodeStore__RedisConnectionString` at the operator client Service `valkey-jellyfin-valkey:6379`, preserving `abortConnect=false` and `LeaseDurationSeconds`.
## Auth / TLS
None. This operator runs Valkey cluster-mode-enabled with `protected-mode no` and leaves the built-in `default` user passwordless (no `requirepass`), so clients connect unauthenticated over plaintext — parity with the previous open redis. StackExchange.Redis auto-discovers the single node via the headless Service. No `VaultStaticSecret` / plain Secret is introduced.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/jellyfin` renders cleanly: `ValkeyCluster` present, no redis Deployment/Service/PVC, connection string updated.
- pre-commit passes (incl. the plain-secrets guard).
- kubeconform: 10 valid / 0 invalid; the `ValkeyCluster` is skipped (CRD schema not vendored).
## Follow-up
HA (task 6) is intentionally out of scope: it will raise `replicas` (and/or `shards`) and add anti-affinity/topology spread. Enabling auth (ACL `default` user + password via `VaultStaticSecret`) and/or TLS can also be layered on later.
Reviewed-on: #367
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
unkin-agent