## Why
The new `golib` repo's Woodpecker pipeline needs a dedicated ServiceAccount to run its CI steps under.
## How
- Add bare ServiceAccount `golib-ci` in the `woodpecker` namespace and wire it into the base kustomization.
Reviewed-on: #447
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: repospawner v0.1.1 stops spawned job pods from automounting the API token.
- Bump the deployment image and the matching REPOSPAWNER_IMAGE env value to v0.1.1
Reviewed-on: #446
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
repospawner v0.1.0 is built and its Vault kubernetes auth role is applied, but nothing deploys it. It turns a "I want a new repository" request into a terraform-git pull request, follows that PR to merge, and optionally activates the repo in Woodpecker, so the review gate stays where it is instead of moving into an agent's hands.
## How
- Add `apps/base/repospawner/`: namespace, ServiceAccount `repospawner`, `default` VaultAuth for VSO, and a namespaced Role/RoleBinding granting jobs create/get/list/watch/delete plus pods and pods/log reads (mirrors mediamover).
- Deployment pinned to `artifactapi.k8s.syd1.au.unkin.net/docker-internal/repospawner:v0.1.0`, one replica with the `Recreate` strategy because request state is in memory and rebuilt from Job labels; the same image reference is passed down as `REPOSPAWNER_IMAGE` so the spawned Jobs stay in step.
- Mount a projected `audience: vault` service account token at `/var/run/secrets/vault` — the app logs into Vault natively rather than through VSO — and the `repospawner-woodpecker` Secret at `/etc/repospawner/woodpecker`, optional so the server still starts and refuses `woodpecker: true` with 503 when it is absent.
- Two VaultStaticSecrets: `oauth-credentials` from `kv/kubernetes/namespace/repospawner/default/oauth-credentials` and `repospawner-woodpecker` (key `token`) from `.../default/woodpecker`, with reloader annotations on both consumers.
- oauth2-proxy front door on the watchstate/mediamark pattern, gated on `akP-repospawner-admin` via the `ak_groups` claim and re-checked by the app from `X-Forwarded-Groups`; public `repospawner.unkin.net` on the reflected wildcard and internal `repospawner.k8s.syd1.au.unkin.net` on `vault-issuer`, both routed to the oauth2 Service.
- Register the overlay in the platform ApplicationSet and AppProject, and append `repospawner` to the wildcard Certificate's two reflector namespace lists.
Depends on the terraform-authentik `repospawner` client being applied and `kv/kubernetes/namespace/repospawner/default/oauth-credentials` + `.../woodpecker` being seeded.
Reviewed-on: #445
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
arrproxy v0.6.0 applies its own schema at startup under a Postgres advisory lock and holds `/readyz` until the schema is current, so every replica is safe to roll without an external gate. The wave-1 psql `arrproxy-migrate` Job and its SQL ConfigMap now only re-run idempotent statements the app already owns — dead weight, a second source of truth for the schema, and a standing drift trap whenever the app's embedded migrations move ahead of the manifests.
## How
- Bump `arrproxy-api` and `arrproxy-ui` to `v0.6.0`.
- Delete `migrate-job.yaml` and `migrations-configmap.yaml` and drop both from the arrproxy kustomization.
- Keep the wave-0/wave-2 split: wave 2 still orders the api behind the wave-0 CNPG Cluster and VSO-synced Secrets, which is independent of the migrate Job; the stale "serve only after the wave-1 migrate Job" comment is corrected.
- Rendered diff vs `main` is exactly the two image bumps plus the `arrproxy-migrate` Job and `arrproxy-migrations` ConfigMap disappearing; `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` and pre-commit both clean.
Reviewed-on: #444
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
mediamark needs a home in the cluster: it marks/organises media on the shared mediastore tree and drives the adult-tier sonarr/radarr through arrproxy's hash routes. This adds the full app to the `media` project, mirroring the watchstate two-host oauth2-proxy pattern and the arrstack static-CephFS + projected-API-keys patterns.
## How
- Adds `apps/base/mediamark/`: namespace, VaultAuth (`k8s/au/syd1`, role `default`), three VaultStaticSecrets, static mediastore PV/PVC, the app Deployment, oauth2-proxy ConfigMap/Deployment, two Services, and internal + external Gateway/HTTPRoute pairs.
- Binds a dedicated static PV (`mediamark-mediastore`, own `volumeHandle`, `claimRef`-pinned) to the same CephFS mediastore subvolume arrstack/fafflix/cheeztv use, RWX 10Ti Retain, mounted at `/media`.
- Runs the app as 1000:1000 (deliberately not 65532) so it owns files on the shared media tree and hardlink/rename moves stay valid; read-only root filesystem, all caps dropped, no service-account token, `/livez` + `/readyz` probes.
- Projects the sonarr/radarr API keys as one file per app under `/etc/mediamark/keys`, mirroring arrproxy's keys projection, with reloader annotations on both secrets.
- Fronts both `mediamark.unkin.net` (traefik-external, reflected Let's Encrypt wildcard, no cert-manager annotations) and `mediamark.k8s.syd1.au.unkin.net` (traefik-internal, vault-issuer) with a single oauth2-proxy using a relative `/oauth2/callback` redirect; gated on `akP-mediamark-user` and passing identity to the app as `X-Forwarded-Groups` via `PASS_USER_HEADERS`.
- Appends `mediamark` to the `wildcard-unkin-net` Certificate's two reflector namespace lists, and registers the app in `argocd/applicationsets/media.yaml` + `argocd/projects/media.yaml` with a passthrough `apps/overlays/au-syd1/mediamark` overlay.
## Prerequisite seeds (Ben, before pods go Ready)
These KV paths must exist under `kv/kubernetes/namespace/mediamark/default/` — the `mediamark/default` templated policy already grants read, so no terraform-vault change is needed:
- `oauth-credentials` — needs `client_id` and `cookie_secret` added alongside the existing `client_secret` (Authentik mediamark provider; both absolute callback URIs registered there).
- `sonarr` — key `apitoken`.
- `radarr` — key `apitoken`.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/mediamark` (18 resources) and `.../cert-manager` both build.
- kubeconform clean on both touched overlays.
- `pre-commit run --all-files` passes.
Reviewed-on: #441
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
arrproxy v0.5.0 ships per-token HTTP method scoping for machine tokens, so a minted token can be limited to e.g. `GET` only. Zero-downtime: the mint-API field is additive and existing tokens get an empty methods list, which means unrestricted — they behave exactly as before.
## How
- Bump `arrproxy-api` and `arrproxy-ui` pins from v0.4.0 to v0.5.0.
- Mirror repo migrations `0002_tier_tokens.sql` and `0003_token_methods.sql` into the migrations ConfigMap. It had drifted at 0001 while v0.4.0 already queried `tier`/`read_only`, and every v0.5.0 token query selects `methods` — without this the new API errors on every token read.
- Have the wave-1 migrate Job apply all three files in order. Every statement is `IF NOT EXISTS`, so a resync over an already-migrated database is a no-op.
Rendered `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` diff vs main is exactly the two image tags, the two added ConfigMap keys, and the two added `-f` args.
Reviewed-on: #443
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The new repospawner repo needs a Woodpecker CI pipeline, and every pipeline step must run under its own ServiceAccount in the woodpecker namespace.
- Add `apps/base/woodpecker/serviceaccount_repospawner_ci.yaml` (bare SA `repospawner-ci`, ns woodpecker), mirroring the existing mediamark-ci SA.
- Register it in the woodpecker kustomization resources list.
Reviewed-on: #442
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit (logarchiver/logviewer are untouched).
Changes:
- Point the ClickHouseInstallation and the clickhouse-schema job at `docker.io/clickhouse/clickhouse-server:24.8`.
- Point the logviewer oauth2-proxy cert-combine init container at `docker.io/library/alpine:3`.
- Point the NATS bootstrap job at `docker.io/natsio/nats-box:0.18.0`.
- Point the NATS chart values at `docker.io/library/nats` and `docker.io/natsio/nats-server-config-reloader`.
- Point all three Vector values files (agent, aggregator, vm-ingest) at `docker.io/timberio/vector`.
- Drop the now-wrong "pulled through the artifactapi dockerhub remote" comments in the NATS and vector-agent values.
Tags/digests unchanged and the `repository`/`tag` split is preserved. `kustomize build --enable-helm apps/overlays/au-syd1/logging` differs from main only in those nine image strings.
Extra found, not changed here: `.woodpecker/vector-test.yaml` still pins its CI step image to `artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector:0.57.0-debian`. That is a Woodpecker step image rather than a namespace manifest, so it is left out to keep this PR to the logging namespace — say the word and I will fix it separately.
Reviewed-on: #433
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
`identity.unkin.net` moved from an internal `unkin.net` CA-issued cert to the LetsEncrypt `*.unkin.net` wildcard. `auth.generic_oauth`'s `tls_client_ca` pointed Grafana at the internal root only, so the OAuth handshake to the LE-issued cert now fails. Grafana's image trust store already contains the public roots.
## Changes
- Remove `tls_client_ca: /etc/grafana/vault-ca/ca.crt` (and its stale comment) from `auth.generic_oauth`.
- Remove the now-unused `vault-ca-cert` volume and volumeMount from the Grafana pod spec — nothing else in the pod referenced it (the CNPG `endpointCA` reference to `vault-ca-cert` for `s3.ceph.unkin.net` is a separate resource and stays).
- Leave the auth/token/api URLs, scopes and `role_attribute_path` untouched.
Reviewed-on: #440
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The mediamark Woodpecker docker step needs a dedicated ServiceAccount so it can push to the trusted in-cluster registry, mirroring the existing arrproxy-ci setup.
- Add bare ServiceAccount `mediamark-ci` in namespace `woodpecker` and register it in the woodpecker base kustomization.
Reviewed-on: #438
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the watchstate oauth2-proxy cert-combine init container at `docker.io/library/alpine:3`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/watchstate` differs from main only in that image string. No extra proxied refs in the file (the oauth2-proxy image itself is already canonical `quay.io/...`).
Reviewed-on: #435
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the netbox valkey metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/netbox` differs from main only in that image string. No extra proxied refs in the file (the `valkey/valkey:8-alpine` container is already a bare upstream name).
Reviewed-on: #434
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the litellm redis metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/litellm` differs from main only in that image string. No extra proxied refs in the file (the `redis:7-alpine` container is already a bare upstream name).
Reviewed-on: #432
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the gitea valkey metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/gitea` differs from main only in that image string. No extra proxied refs in the file (the `valkey/valkey:8-alpine` container is already a bare upstream name).
Reviewed-on: #431
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the authentik redis metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/authentik` differs from main only in that image string. No extra proxied refs in the file (the `redis:7-alpine` container is already a bare upstream name).
Reviewed-on: #430
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the artifactapi redis metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/artifactapi` differs from main only in that image string. No extra proxied refs in the file (the `redis:7-alpine` container is already a bare upstream name).
Reviewed-on: #429
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The media estate is currently split across two 1Ti CephFS subvolumes (`media-tv`
and `media-movies`), so a download that lands in one and imports to the other is
a cross-filesystem copy, not a hardlink move. A single 10Ti `mediastore`
subvolume (already created on `cephfs_csi_ssd_ec_4_1`) collapses that: one
filesystem holding `/{fafflix,cheeztv}/{tvseries,movies}` plus
`/nzbget/downloads/complete`, so every arr import is an atomic same-fs hardlink.
This PR only provisions the volume and seeds its tree. No existing PV, PVC or
workload is touched, and nothing mounts the new claims yet — the cutover of the
arrs and the two jellyfins is a separate change.
## How
- Add static PVs `arrstack-mediastore`, `fafflix-mediastore` and
`cheeztv-mediastore`, all pointing at the same rootPath
(`/volumes/csi_ssd_ec_4_1/mediastore/a0152dac-…`) with unique names and
volumeHandles pinned by `claimRef` — the established pattern for the shared
media subvolumes.
- Add the matching RWX 10Ti PVCs (`mediastore` in arrstack,
`fafflix-mediastore`, `cheeztv-mediastore`), annotated
`k8up.io/backup: "false"` and statically bound via `volumeName` +
`storageClassName: ""`.
- Add `mediastore-bootstrap`, a one-shot ArgoCD Sync-hook Job
(`hook-delete-policy: BeforeHookCreation`, no sync-wave needed) that mounts
the arrstack claim and `mkdir -p`s the directory tree as uid/gid 1000 —
the uid the arrstack media pods run as. Idempotent, so it self-heals on
every sync.
- Wire the new manifests into the arrstack, fafflix and cheeztv bases.
## Validation
- `kustomize build` clean on `apps/overlays/au-syd1/{arrstack,fafflix,cheeztv}`
- `kubeconform` clean on all three overlays (91 / 29 / 32 resources valid)
- pre-commit (yamllint, check-yaml, no-plain-secrets) passed
Reviewed-on: #428
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the arrproxy migrate job at `docker.io/library/postgres:18-alpine`.
- Point the arrproxy oauth2-proxy cert-combine init container at `docker.io/library/alpine:3`.
- Point the arrstack ValkeyCluster at `docker.io/valkey/valkey:9.0.0`.
Tags are unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` differs from main only in those three image strings. No extra proxied refs found in these files (the oauth2-proxy image itself is already canonical `quay.io/...`).
Reviewed-on: #427
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Summary
- Adds SSO-Auth.xml, LDAP-Auth.xml, and branding.xml plugin config (rendered by an initContainer into /config) to fafflix and cheeztv
- Adds a per-namespace oauth-credentials VaultStaticSecret to each instance
- Adds a "Sign in with SSO" login link via jellyfin branding config
## Why
Phase-1 jellyfin auth: browser SSO via the 9p4 SSO plugin (Authentik OIDC, ak_groups claim) and phone/TV app-passwords via the LDAP plugin against the Authentik LDAP outpost. Config is declarative — restarts overwrite UI edits. Activates when the plugin-baked jellyfin-ha image lands (separate PR).
Reviewed-on: #424
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Summary
- Bumps the jellyfin-ha image to v0.2.0 on fafflix and cheeztv
## Why
v0.2.0 bakes the LDAP and SSO auth plugins into the image; with the plugin config PR this enables browser SSO and app-password logins.
Reviewed-on: #426
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Summary
- Bumps kea-operator image pin to v0.1.5
## Why
v0.1.4 fixed HA peer URLs but the HA hook then bound its dedicated listener to the ClusterIP and crashed; v0.1.5 disables the dedicated listener so HA flows via the ctrl-agent.
Reviewed-on: #425
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
why: production kea-dhcp4 is in a permanent crashloop — Kea's HA hook rejects DNS hostnames in peer URLs and v0.1.3 renders headless-DNS peer URLs; v0.1.4 ships the merged fix (per-peer ClusterIP Services + IP-literal URLs).
- bump kea-operator image pin to v0.1.4
Reviewed-on: #423
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Jellyfin authenticates users against Authentik over LDAP (app-password binds). The `authentik-ldap` Service, Gateway, and TLSRoute already exist in the authentik base, but nothing backed the Service — no LDAP outpost was ever deployed. This adds the missing Deployment so in-cluster clients (Jellyfin) can bind.
Deployed as a separate Deployment inside the existing `authentik` namespace rather than a new app/namespace: the authentik overlay is already wired into the `platform` ApplicationSet and the LDAP Service/Gateway/TLSRoute already live here, so this needs no new AppProject/ApplicationSet wiring and keeps the outpost next to the core it serves.
## How
- `ldap-outpost-deployment.yaml`: 2-replica (stateless) Deployment, image `ghcr.io/goauthentik/ldap:2026.5.3` (canonical upstream name; matches the deployed authentik chart version; containerd mirrors route via artifactapi). Pod labels match the existing `authentik-ldap` Service selector.
- `AUTHENTIK_HOST=https://identity.k8s.syd1.au.unkin.net`, `AUTHENTIK_INSECURE=false`.
- Trusts the internal CA via the established combine-certs initContainer pattern (concatenates the base roots with the reflected `vault-ca-cert`) + `SSL_CERT_FILE`.
- `AUTHENTIK_TOKEN` sourced from secret `authentik-ldap-outpost-token`.
- Resources set; TCP probes on the LDAP port; reloader annotation for token + CA rotation.
- `ldap-outpost-vaultstaticsecret.yaml`: VSS (reuses the namespace `default` VaultAuth) materialising the token from KV `kubernetes/namespace/authentik/default/outpost-token` (key `token`).
- `ldap-service.yaml`: adds the plaintext `ldap` port 3389 alongside the existing `ldaps` 6636 (ClusterIP only, no public exposure — external reach is via the pre-existing internal Gateway/TLSRoute).
- `ldap-outpost-vmpodscrape.yaml`: VMPodScrape for the outpost metrics endpoint (:9300).
- No sync-wave annotations.
## Dependency — token seed (blocking)
The outpost pods CrashLoop until the token exists. After merge, the Authentik LDAP outpost token must be seeded into Vault KV at `kv/kubernetes/namespace/authentik/default/outpost-token` with key `token`. This path is provided by the corresponding terraform-authentik PR; the value must match the outpost token authentik issues.
## Validation
`kustomize build --enable-helm apps/overlays/au-syd1/authentik` + repo kubeconform: 38 resources, Valid: 38, Invalid: 0, Errors: 0.
Reviewed-on: #422
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
WatchState currently only has an internal front door (watchstate.k8s.syd1.au.unkin.net, PR #419, merged). This adds a public front door at **watchstate.unkin.net** via the external (DMZ) Traefik so the admin UI is reachable off-cluster, still fully gated by oauth2-proxy/Authentik. The internal gateway and routes are untouched; oauth2-proxy now fronts BOTH hostnames.
Stacked base note: the intended base (benvin/watchstate-deploy, PR #419) merged into main before this PR opened, so this targets `main`.
## How
- **DNS**: `watchstate-dns-internal` DNSRecord in `apps/base/bind-internal/authoritative/records.yaml` (unkin.net zone, bind-internal) -> `A 198.18.199.0`, the traefik-external VIP. Mirrors the `arrstack-dns-internal` precedent (external front door via bind, not external-dns). Kept in the central bind-internal location, so no AppProject destination widening is needed.
- **Gateway**: new `watchstate-external` Gateway (`gatewayClassName: traefik-external`), http + https listeners on hostname `watchstate.unkin.net`. TLS terminated with the Let's Encrypt `*.unkin.net` wildcard secret `wildcard-unkin-net-tls` — so **no cert-manager and no external-dns annotations**.
- **Routes**: `watchstate-external-http-redirect` (http->https 301) and `watchstate-external` (https -> `watchstate-oauth2:80`), mirroring the arrproxy/cheeztv external patterns.
- **oauth2-proxy dual-host**: `OAUTH2_PROXY_REDIRECT_URL` changed from the pinned cluster callback to the **relative** `/oauth2/callback`. With `OAUTH2_PROXY_REVERSE_PROXY=true` (already set), oauth2-proxy derives scheme+host per request from `X-Forwarded-Proto/Host` (verified in oauth2-proxy v7.15.3 `getOAuthRedirectURI`: host-less redirectURL falls through to request-derived), so the callback works on both `watchstate.unkin.net` and `watchstate.k8s.syd1.au.unkin.net`. `COOKIE_DOMAINS` and `WHITELIST_DOMAINS` now list both hosts (one cookie per host; a single parent-domain cookie can't cleanly span unkin.net vs k8s.syd1.au.unkin.net).
- Dropped the no-op `argocd.argoproj.io/sync-wave: "0"` annotation on the `default` VaultAuth. New resources intentionally carry no sync-wave annotations.
## Validation
- `kustomize build --enable-helm` on `apps/overlays/au-syd1/watchstate` and `.../bind-internal`: OK.
- kubeconform (repo CI config, k8s 1.33.7): watchstate overlay 15/15 valid (2 Gateways, 4 HTTPRoutes); bind-internal overlay 67/67 valid (incl. `watchstate-dns-internal`).
## Dependencies
- **Reflector allow-list (argocd-apps PR #418)**: reflects `wildcard-unkin-net-tls` into the `watchstate` namespace. This PR references that secret as if present.
- **Authentik (terraform-authentik, separate PR)**: register both `https://watchstate.unkin.net/oauth2/callback` and `https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback` as redirect URIs on the watchstate provider.
Reviewed-on: #420
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
#417 stood up the central Let's Encrypt `*.unkin.net` wildcard (Certificate `wildcard-unkin-net` in `cert-manager`, secret `wildcard-unkin-net-tls`, emberstack-reflected). This wires the remaining single-label public hosts onto that publicly-trusted leaf instead of the internal Vault-PKI CA.
A `*.unkin.net` wildcard matches **single-label** subdomains only, so cluster hostnames under `*.k8s.syd1.au.unkin.net` are deliberately left on `vault-issuer`.
> Note: PR #417 already merged to `main`, so this is no longer stacked — it targets `main` directly.
## How
- **cert-manager**: reflect `wildcard-unkin-net-tls` into `arrstack`, `authentik`, `gitea` (appended to both reflector namespace lists → `cheeztv,arrstack,authentik,gitea`).
- **arrstack**: repoint the public `arrproxy` gateway listener (`arrstack.unkin.net`) `certificateRefs` → `wildcard-unkin-net-tls`; drop its `cert-manager.io/*` vault-issuer annotations.
- **authentik**: repoint the `identity.unkin.net` https listener → `wildcard-unkin-net-tls`. The `identity.k8s.syd1.au.unkin.net` listeners keep `authentik-tls` (vault-issuer), so the gateway `common-name` is retargeted to that cluster host. `authentik-ldap` gateway untouched.
- **gitea**: secret **reflected only**; `git.unkin.net` cutover deferred (no manifest change), same deferred pattern as fafflix.
## Validation
- `kustomize build` OK for cert-manager / arrstack / authentik / gitea overlays.
- `kubeconform` all valid (0 invalid, 0 errors) across the four overlays.
- Verified no `*.k8s.syd1.au.unkin.net` listener lost its vault-issuer wiring.
Reviewed-on: #418
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Deploy WatchState (arabcoders/watchstate), the Jellyfin/Plex/Emby watch-state
sync tool, as an internal admin tool. It gets an admin-only web UI/API gated the
same way as logviewer: an oauth2-proxy front backed by Authentik OIDC.
## What
- New `watchstate` namespace + media-project app; base at `apps/base/watchstate`,
overlay at `apps/overlays/au-syd1/watchstate`.
- Image `ghcr.io/arabcoders/watchstate:v1.10.3` (current release; canonical
upstream name, containerd mirrors route ghcr via artifactapi).
- `replicas: 1`, `strategy: Recreate`, single `5Gi` `cephrbd-fast-delete` RWO PVC
at `/config` — sqlite + the in-container cron/redis are single-writer.
- `runAsUser/runAsGroup/fsGroup: 1000` (image's rootless user); liveness/readiness
`GET /v1/api/system/healthcheck` on 8080 (route confirmed in upstream
`src/API/System/HealthCheck.php`, no auth guard).
- oauth2-proxy (mirrors logviewer 1:1) fronts every path. Authentik issuer
`identity.k8s.syd1.au.unkin.net`, redirect
`https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback`. Authorization is
enforced Authentik-side (akR-global-admin only), so no oauth2-proxy group
allowlist is configured.
- Internal-only Gateway (`traefik-internal`) for
`watchstate.k8s.syd1.au.unkin.net`, `vault-issuer` TLS leaf, external-dns to
`198.18.200.4`. HTTP -> HTTPS redirect.
- `VaultStaticSecret` pulls OIDC creds from
`kv/kubernetes/namespace/watchstate/default/oauth-credentials`; `vault-ca-cert`
auto-reflects into the namespace.
- Registered in the media `ApplicationSet` + `AppProject`.
## Scrape decision
No `VMPodScrape`: WatchState exposes no Prometheus/`/metrics` endpoint.
## Follow-ups
- **Seed check:** the VaultStaticSecret expects `client_id`, `client_secret`, and
`cookie_secret` keys at the kv path. Only `client_secret` was confirmed seeded;
`client_id` and a generated `cookie_secret` must also be present or the
oauth2-proxy pod will not start.
- **Webhook ingestion:** the Jellyfin webhook endpoint (`/v1/api/webhook`)
currently sits behind oauth2-proxy like everything else. When sync is wired up,
the jellyfins pushing webhooks will need an auth-bypass or an apikey route for
that path.
Reviewed-on: #419
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
why: cheeztv.unkin.net needs a publicly-trusted cert; DNS-01 via the delegated acme.unkin.net zone is now working (staging wildcard verified end-to-end in ~70s). A single centrally-managed wildcard avoids per-host CNAMEs and LE duplicate-order rate limits.
- add Certificate wildcard-unkin-net (cert-manager ns, ClusterIssuer letsencrypt prod, secret wildcard-unkin-net-tls) with emberstack reflector secretTemplate annotations
- reflect the secret into the cheeztv namespace (append future consumers to the two reflector namespace lists)
- point the public cheeztv gateway listener at wildcard-unkin-net-tls and drop its vault-issuer annotations (cheeztv-internal stays on vault-issuer)
Reviewed-on: #417
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The old bare `sonarr`/`radarr` Services are gone, but arrproxy still ran the v0.3.x built-in default topology pointing at non-existent `sonarr`/`radarr` upstreams and mounted dead `sonarr-apikey`/`radarr-apikey` Secrets. This cuts the front door over to arrproxy v0.4.0 tier routing so the adult (fafflix, `/3aa168`) and kids (cheeztv, `/3df803`) *arr instances are reachable again, with per-tier group authorization.
## Changes
- Bump `arrproxy-api` and `arrproxy-ui` images to `v0.4.0`.
- Add `arrproxy-tiers` ConfigMap (`ARRPROXY_TIERS_FILE`): `fafflix` + `cheeztv` tiers, each with sonarr/radarr upstreams, route hash, key subdir and group grants. Adults (`akP-media-fafflix`) reach both tiers all methods; kids (`akP-media-cheeztv`) reach only cheeztv, GET/HEAD. `legacyRoutes:false` retires the unprefixed routes.
- Because arrproxy strips `/<hash>` and proxies `/<app>/...`, each upstream URL carries the `/<hash>` path segment so the joined upstream path (`/3aa168/sonarr/...`) matches the *arr's own UrlBase.
- Mount the tiers file and rework the projected keys volume to the real per-instance `{sonarr,radarr}-{adult,kids}-apikey` Secrets under `<keyDir>/<app>` (`adult/*`, `kids/*`); refresh reloader annotations.
- oauth2-proxy: route `/3aa168` and `/3df803` to arrproxy-api (drop dead `/sonarr` `/radarr` `/prowlarr`; prowlarr is served directly), and widen `SKIP_AUTH_REGEX` to `^/[^/]+/[^/]+/api` for the hash-prefixed tier api paths.
Group headers keep flowing via the existing Authentik `ak_groups` claim -> oauth2-proxy `X-Forwarded-Groups` mechanism. Validated with `kustomize build` + repo kubeconform (88/88 valid). No new Vault seeds required — the per-instance apikey Secrets already exist.
Reviewed-on: #416
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Rebuild the adult media instance as `fafflix`, mirroring the kids instance (`cheeztv`) HA pattern (Postgres-backed jellyfin-ha fork, Valkey transcode-lease store, shared-RWX transcode, CNPG + k8up/restic backups, static CephFS media PVs). Ben: "replace the adult jellyfin with the same pattern as kids." The adult instance holds no data currently, so the wipe/replace is sanctioned.
Stacked on top of `benvin/cheeztv` (#413) — base is that branch, not main.
## How
- Add `apps/base/fafflix/` — cheeztv pattern with `s/cheeztv/fafflix` on names/namespace/labels/secrets/buckets and the Vault kv path `kubernetes/namespace/fafflix/default/k8up-restic`.
- Media: mount the shared movies/tv CephFS subvolumes' `adult` subtree at `/media/movies` and `/media/tv`, plus the `kids` subtree at `/media/movies-kids` and `/media/tv-kids` (same two static PVs, new unique volumeHandles `fafflix-media-{movies,tv}-static`) so fafflix can resume kids content started on cheeztv. All media mounts readOnly, matching cheeztv.
- **Hostname preserved:** fafflix keeps serving the legacy jellyfin host `jellyfin.k8s.syd1.au.unkin.net` (TLS secret `fafflix-tls`, cert-manager re-mints). The dedicated fafflix domain switch is explicitly deferred — no `fafflix.unkin.net` created.
- config PVC on `cephfs-raid5-delete` (delete variant).
- Remove `apps/base/jellyfin/` + its overlay; swap `jellyfin`->`fafflix` in the `media-apps` ApplicationSet directory glob and the `media` AppProject namespace destination.
- Third-party images stay canonical upstream; the in-house `jellyfin-ha` image ref is unchanged.
## Verification
- `kustomize build` + `kubeconform -strict` clean on the fafflix overlay and the whole media appset scope (fafflix/cheeztv/arrstack): 0 invalid, 0 errors.
- No dangling `apps/base/jellyfin` references; `jellyfin.k8s.syd1.au.unkin.net` now served by exactly one app (fafflix); no `fafflix.unkin.net` anywhere.
Reviewed-on: #415
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Kids need their own safe Jellyfin front (cheeztv.unkin.net) with a library that only ever exposes kids titles, while those same titles stay resumable in the existing adult instance (fafflix). The fafflix.unkin.net domain switch is deliberately out of scope.
## How
- **New `cheeztv` instance** (`apps/base/cheeztv`): mirrors the fafflix HA Jellyfin stack 1:1 - same `jellyfin-ha:v0.1.3` fork image, 2-replica StatefulSet, RWX transcode PVC + HA takeover, Intel iGPU transcode, CNPG Postgres + PgBouncer, Valkey transcode store, k8up config backup, VMPodScrape.
- **Kids-only media**: its own static PVs on the *same* shared movies/tv CephFS subvolumes, mounted with `subPath: kids` so it sees only `movies/kids` and `tvshows/kids`.
- **Fully separate state**: own namespace, config PVC, `cheeztv-postgres` cluster, `cheeztv-valkey`, and `cnpg-cheeztv` / `cheeztv-config-backup` RGW buckets - nothing shared with fafflix.
- **DNS/cert**: `cheeztv.unkin.net` via internal Traefik + external-dns (198.18.200.4) + cert-manager `cheeztv-tls`, matching logviewer.unkin.net; plus the cluster hostname `cheeztv.k8s.syd1.au.unkin.net` mirroring fafflix.
- **fafflix change**: ADDS `movies/kids` + `tvshows/kids` subPath mounts alongside its current media mounts so kids libraries are resumable there. Existing mounts, hostname and ingress untouched.
- Registers cheeztv in the media ApplicationSet generator and AppProject destinations.
## Validation
kubeconform clean: cheeztv 28/28, jellyfin 25/25 valid. No arr/watchstate/domain changes.
Note: the k8up restic repo password must be seeded at `kv/kubernetes/namespace/cheeztv/default/k8up-restic` (same manual step as fafflix); templated Vault policy already grants read, no terraform-vault change needed.
Reviewed-on: #413
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Splits the arrstack media pipeline into an adult and a kids tier so kids content lives in its own libraries, databases, and downloader while reusing the shared prowlarr, CNPG cluster, Valkey, media PVCs, and S3 buckets. Backend workloads only — no routing/ingress, no arrproxy, no adult-instance changes in this PR.
How:
- sonarr-kids / radarr-kids: mirror the adult -unkin fork Deployments (same images/tags, 3 active-active replicas) on their own Postgres DBs (sonarr-kids-main / radarr-kids-main), UrlBase /3df803/sonarr and /3df803/radarr, and kids MediaCover/Backup S3 prefixes.
- Media mounts use the tvshows/kids and movies/kids subPaths of the shared media-tv / media-movies PVCs, so kids libraries are isolated on the same CephFS subvolumes.
- nzbget-kids: second downloader on its own RWO config PVC, completed downloads land in the kids media subtrees for same-filesystem hardlink imports.
- postgres: two managed roles + Database CRDs and their VSO-synced <app>-kids-db credential secrets.
- per-app VMPodScrape for each kids exportarr sidecar; new apps registered in the base kustomization.
Validated: kustomize build + kubeconform (96/96 valid), yamllint + pre-commit clean.
Reviewed-on: #412
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Deploy the new space-pirate UI background released in arrproxy v0.3.1.
- Bump arrproxy-api image to v0.3.1
- Bump arrproxy-ui image to v0.3.1
Reviewed-on: #411
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The `readonly` profile used by `logreader` sets `readonly=1`, which rejects any per-query settings change. HTTP clients (logviewer, chlog) send settings params like `max_execution_time` with their queries, so every query fails with `Code: 164 DB::Exception: Cannot modify 'max_execution_time' setting in readonly mode` — logviewer healthz returns 503 and CrashLoops, chlog queries fail. `readonly=2` keeps data access read-only while permitting settings changes, which is what HTTP clients need.
## Changes
- Changes the `readonly` profile in `apps/base/logging/clickhouseinstallation.yaml` from `readonly: "1"` to `readonly: "2"`
Reviewed-on: #409
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Both `arrstack-postgres` replicas were OOMKilled at the 1Gi memory limit on 2026-08-23 07:19, causing a failover to timeline 14, leaving replica-3 unrecoverable via pg_rewind, and blocking the argocd sync wave on Cluster Ready=False.
## Changes
- Raises the CNPG Cluster resources to requests 1Gi memory / 250m CPU and limits 2Gi memory / 1 CPU
**NOTE:** applying this restarts the postgres instances (CNPG rolling restart).
Reviewed-on: #408
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Traefik metrics (traefik_entrypoint_requests_total, traefik_service_*, etc.) are now flowing into VictoriaMetrics with the relabeled `traefik_instance` label, but there is no dashboard to view them.
## Changes
- Add `apps/base/grafana/dashboards/traefik.yaml`: GrafanaDashboard CR mirroring the redis.yaml sibling (gzipJson embed, `instanceSelector dashboards: grafana`, `resyncPeriod: 5m`, `allowCrossNamespaceImport: false`)
- Embed the official Traefik Standalone dashboard (grafana.com ID 17346, revision 9), adapted:
- strip `__inputs`/`__requires`/`__elements` and the `DS_PROMETHEUS` datasource variable
- rewrite all prometheus datasource uids to the estate VictoriaMetrics uid `det2y55dac4jkc`
- add an `instance` template variable over `traefik_instance` and filter every query with `traefik_instance=~"$instance"` so traefik-internal / traefik-external are switchable
- chain the entrypoint/service variable queries through the instance filter
- Register the dashboard in `apps/base/grafana/kustomization.yaml`
Validated: gzipJson decode roundtrip, `kustomize build`, kubeconform.
Reviewed-on: #407
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Ship icon grid + jellyfin tile features by upgrading arrproxy UI and API to v0.3.0.
- Bump arrproxy-api image from v0.2.0 to v0.3.0
- Bump arrproxy-ui image from v0.2.0 to v0.3.0
- Verified images built and pushed to artifactapi
Reviewed-on: #406
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
logviewer.unkin.net does not resolve, so the log viewer UI is unreachable even though its Gateway and certificate already exist (from #404).
## Changes
- Adds DNSRecord `logviewer-dns-internal` (A, ttl 600 → 198.18.200.4, the internal traefik gateway VIP) to bind-internal authoritative records, mirroring the existing ghp/identity records
Reviewed-on: #405
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer.
## Changes
- replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel
- renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel
- adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote
- adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules)
- adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules)
## Prerequisite
The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff.
## Notes
- validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
- the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
Reviewed-on: #399
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
No redis/valkey instance managed in this repo exports metrics today — the six caches (artifactapi, authentik, litellm, gitea, netbox, jellyfin) are blind spots in VictoriaMetrics.
## What
- Enables the valkey-operator exporter sidecar on the jellyfin ValkeyCluster (`exporter.enabled: true`); overrides the operator's default bare-dockerhub image (`oliver006/redis_exporter:v1.80.0`) with the artifactapi-proxied pin. The operator manages a dedicated `_exporter` ACL user for it.
- Adds a `metrics-exporter` sidecar (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/oliver006/redis_exporter:v1.89.0`, :9121, port name `metrics`, req 32Mi/25m lim 64Mi/100m) to the five raw Deployments: artifactapi/authentik/litellm redis and gitea/netbox valkey. gitea/netbox sidecars get the same restrictive securityContext as their valkey container.
- Adds the `monitoring.unkin.net/redis-exporter: "true"` opt-in pod label to all five pod templates.
- Adds `apps/base/observability/vmpodscrape-redis.yaml` with two VMPodScrapes (any namespace, port `metrics`): `redis-exporters` selecting the opt-in label, and `valkey-operator-clusters` selecting `app.kubernetes.io/managed-by: valkey-operator` + `app.kubernetes.io/component: valkey-node` — the ValkeyCluster CR has no pod-label passthrough (verified against the v0.5.0 CRD/source), so operator pods cannot carry the opt-in label.
## Notes
- No instance runs with `requirepass`/auth, so no REDIS_PASSWORD wiring is needed on the raw Deployments; the jellyfin exporter auth is operator-managed.
- Validated: `kustomize build` on every touched base + au-syd1 overlay (helm overlays with `--enable-helm`) and `kubeconform -strict -ignore-missing-schemas` (68 valid, 0 invalid).
Reviewed-on: #398
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The logviewer UI gives browser access to the ClickHouse log store (`logreader` user). It must only be reachable by authorized users, so it sits behind an Authentik-gated oauth2-proxy front at https://logviewer.unkin.net, mirroring the arrstack front-door and mediamover/pdbmux gateway patterns.
## Changes
- Adds the logviewer Deployment (1 replica, `docker-internal/logviewer:v0.1.0`) reading ClickHouse at `clickhouse-logs` via the existing `clickhouse-logreader` Secret, `/healthz` probes, nonroot securityContext
- Adds an oauth2-proxy (quay.io/oauth2-proxy/oauth2-proxy:v7.15.3) front with issuer `https://identity.unkin.net/application/o/logviewer/`, `ak_groups` claim, allowed group `akP-logviewer-admin`, internal-CA trust via the combine-certs initContainer
- Syncs kv `kubernetes/namespace/logging/default/oauth-credentials` into the `logviewer-oauth-credentials` Secret via the existing `default` VaultAuth (kv path already seeded; no terraform-vault change needed)
- Adds ClusterIP Services for the app and the oauth2-proxy front door
- Adds internal Gateway + HTTPRoutes for `logviewer.unkin.net` (vault-issuer cert, external-dns -> 198.18.200.4, http->https redirect)
- Registers `logviewer/` in the logging base kustomization
## Dependency
Login requires the Authentik `logviewer` provider/application from terraform-authentik PR #21. This PR is safe to merge first: the VSO secret sync and the deployment come up regardless; OIDC login only works once #21 is applied.
Validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (49/49 valid).
Reviewed-on: #404
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
CLI tools and the logviewer UI need ClickHouse access without write or DDL rights. The only existing app user (`vector`) has `access_management` and write access; this adds a properly scoped read-only login using the same secret-sourcing mechanism.
## Changes
- Adds a `logreader` user to the `logs` ClickHouseInstallation: password hash via `secretKeyRef` to the Vault-synced `clickhouse-logreader` Secret (same mechanism as `vector`), networks `::/0`, `readonly` profile, scoped to the `logs` database only
- Adds a `readonly` profile (`readonly: 1`, same memory/time limits as default)
- Adds a `clickhouse-logreader` VaultStaticSecret mirroring the existing `clickhouse-credentials` object (kv-v2, 5m refresh, VSO-created destination Secret)
## Vault kv
Seeded at `kv/kubernetes/namespace/logging/default/clickhouse-logreader` (version 1) with keys: `username`, `password`, `password_sha256_hex`. No secret material lands in git; the CHI only ever sees the sha256 hash via the synced Secret.
Reviewed-on: #400
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Both traefik instances (traefik-internal/traefik-external, chart 40.0.0, image v3.7.0) already expose prometheus metrics on container port `metrics` (9100), but nothing scrapes them — VictoriaMetrics has no traefik series (verified empty). The observability VMAgent runs with selectAllByDefault and consumes VM*Scrape CRs only, so a VMPodScrape is all that is needed.
## Changes
- Adds `apps/base/observability/vmpodscrape-traefik.yaml`: scrapes port `metrics` /metrics on `app.kubernetes.io/name: traefik` pods in traefik-system, relabeling `app.kubernetes.io/instance` to a stable `traefik_instance` label to split internal vs external series
- Registers the new CR in the observability base kustomization
- Enables `metrics.prometheus.addEntryPointsLabels` and `addRoutersLabels` in both au-syd1 traefik values files for per-entrypoint/per-route dashboards (pods restart on rollout — acceptable)
Validated with `kustomize build` (observability base + overlay, traefik-system overlay with --enable-helm) and kubeconform (unknown CRDs skipped); rendered deployments carry the new metric flags and the VMPodScrape renders in the overlay.
Reviewed-on: #397
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The grafana-operator estate has no dashboard for Redis/Valkey; redis_exporter instances are being added in a parallel PR and need visibility.
## Changes
- Adds `apps/base/grafana/dashboards/redis.yaml`, a GrafanaDashboard CR embedding grafana.com dashboard 763 ("Redis Dashboard for Prometheus Redis Exporter 1.x") as gzipJson like the sibling dashboards, with namespace/instance template variables over `redis_up` for multi-instance use and all datasource refs pinned to the VictoriaMetrics datasource uid (`det2y55dac4jkc`), registered in the grafana kustomization
Reviewed-on: #396
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Activates the arr forks (sonarr/radarr/prowlarr) #14 Redis features — SignalR backplane, cross-replica cache-invalidation bus, and distributed rate limiter — which ship deployed but inert because no Valkey exists in arrstack and nothing is wired to it. Ben wants ONE Valkey shared by all three apps.
## Changes
- Adds a single shared `ValkeyCluster` (`arrstack-valkey`) under `apps/base/arrstack/valkey/`, modeled on `jellyfin-valkey`: `shards:1`/`replicas:2` HA (one primary + two replicas, automatic failover), image via artifactapi (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/valkey/valkey:9.0.0`), operator-default passwordless `default` user, node spread across hosts, cluster-aware PDB, ephemeral `/data`.
- Registers the `valkey` component in the arrstack base kustomization.
- Wires `<App>__Redis__Host` + `<App>__Redis__Port` into the `sonarr-env` / `radarr-env` / `prowlarr-env` ConfigMaps, all pointing at the shared service `valkey-arrstack-valkey.arrstack.svc.cluster.local:6379`.
## Notes
- Setting `Host` is the activation switch: the fork's `RedisOptions.IsConfigured` gates purely on a non-empty `Host`, so there is no separate Enabled flag.
- Passwordless (jellyfin parity — the operator leaves the `default` user without auth), so no `Password`/`Ssl` is wired and there is no secret to reference.
- One cluster is safe for all three: each fork namespaces its keys and pub/sub channels by a per-app prefix (`sonarr:ratelimit:` / `radarr:ratelimit:` / `prowlarr:ratelimit:`), so their state never collides.
- App `image:` (-unkin5), the waitfordb initContainer, and S3 buckets are untouched.
- Validated: `kubectl kustomize` renders clean for both `apps/base/arrstack` and `apps/overlays/au-syd1/arrstack`.
Follow-up: after merge, Valkey must come up and the three apps must roll (pick up the new env) before the #14 features can be validated live.
---------
Co-authored-by: Ben Vincent <ben@unkin.net>
Reviewed-on: #394
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Sonarr and radarr currently expose no Prometheus metrics, so queue depth, indexer health, and library stats are invisible to the observability stack. This adds the standard exportarr exporter as a sidecar to both Deployments.
- Adds an exportarr sidecar (ghcr.io/onedr0p/exportarr v2.3.0, pulled through the artifactapi `ghcr` remote, which already whitelists `^onedr0p/`) to the sonarr and radarr Deployments
- Points each exporter at its local replica via env (`URL=http://localhost:8989/sonarr` / `:7878/radarr`, matching the apps' UrlBase) and reuses the existing VSO-synced API-key Secrets (`sonarr-apikey`/`radarr-apikey`, key `apitoken`) via `APIKEY` secretKeyRef — no key in argv
- Serves metrics on 9707 (sonarr) / 9708 (radarr) with `/healthz` liveness+readiness probes and small resources (25m/32Mi requests, 100m/128Mi limits)
- Adds per-app VMPodScrape objects (picked up by the observability VMAgent, selectAllByDefault); pod-level rather than VMServiceScrape because the app Services don't expose the metrics port
Validated with `kustomize build apps/base/arrstack` + kubeconform (0 invalid).
Note: with 3 replicas per app, each pod's exporter reports the same shared-Postgres-backed stats, so series are duplicated across the `pod` label; dashboards should aggregate with `max` or filter to one pod.
Reviewed-on: #395
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
unkin-agent