74eb09b638819ce665681e056d46795cfdba6750
407 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 unkin-agent
|
74eb09b638 |
kea-operator: bump to v0.1.4 (#423)
why: production kea-dhcp4 is in a permanent crashloop — Kea's HA hook rejects DNS hostnames in peer URLs and v0.1.3 renders headless-DNS peer URLs; v0.1.4 ships the merged fix (per-peer ClusterIP Services + IP-literal URLs). - bump kea-operator image pin to v0.1.4 Reviewed-on: #423 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
8eb57f9c19 |
authentik: deploy LDAP outpost for in-cluster app-password binds (#422)
## Why Jellyfin authenticates users against Authentik over LDAP (app-password binds). The `authentik-ldap` Service, Gateway, and TLSRoute already exist in the authentik base, but nothing backed the Service — no LDAP outpost was ever deployed. This adds the missing Deployment so in-cluster clients (Jellyfin) can bind. Deployed as a separate Deployment inside the existing `authentik` namespace rather than a new app/namespace: the authentik overlay is already wired into the `platform` ApplicationSet and the LDAP Service/Gateway/TLSRoute already live here, so this needs no new AppProject/ApplicationSet wiring and keeps the outpost next to the core it serves. ## How - `ldap-outpost-deployment.yaml`: 2-replica (stateless) Deployment, image `ghcr.io/goauthentik/ldap:2026.5.3` (canonical upstream name; matches the deployed authentik chart version; containerd mirrors route via artifactapi). Pod labels match the existing `authentik-ldap` Service selector. - `AUTHENTIK_HOST=https://identity.k8s.syd1.au.unkin.net`, `AUTHENTIK_INSECURE=false`. - Trusts the internal CA via the established combine-certs initContainer pattern (concatenates the base roots with the reflected `vault-ca-cert`) + `SSL_CERT_FILE`. - `AUTHENTIK_TOKEN` sourced from secret `authentik-ldap-outpost-token`. - Resources set; TCP probes on the LDAP port; reloader annotation for token + CA rotation. - `ldap-outpost-vaultstaticsecret.yaml`: VSS (reuses the namespace `default` VaultAuth) materialising the token from KV `kubernetes/namespace/authentik/default/outpost-token` (key `token`). - `ldap-service.yaml`: adds the plaintext `ldap` port 3389 alongside the existing `ldaps` 6636 (ClusterIP only, no public exposure — external reach is via the pre-existing internal Gateway/TLSRoute). - `ldap-outpost-vmpodscrape.yaml`: VMPodScrape for the outpost metrics endpoint (:9300). - No sync-wave annotations. ## Dependency — token seed (blocking) The outpost pods CrashLoop until the token exists. After merge, the Authentik LDAP outpost token must be seeded into Vault KV at `kv/kubernetes/namespace/authentik/default/outpost-token` with key `token`. This path is provided by the corresponding terraform-authentik PR; the value must match the outpost token authentik issues. ## Validation `kustomize build --enable-helm apps/overlays/au-syd1/authentik` + repo kubeconform: 38 resources, Valid: 38, Invalid: 0, Errors: 0. Reviewed-on: #422 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
b9f727bc97 |
watchstate: expose externally at watchstate.unkin.net (#420)
## Why WatchState currently only has an internal front door (watchstate.k8s.syd1.au.unkin.net, PR #419, merged). This adds a public front door at **watchstate.unkin.net** via the external (DMZ) Traefik so the admin UI is reachable off-cluster, still fully gated by oauth2-proxy/Authentik. The internal gateway and routes are untouched; oauth2-proxy now fronts BOTH hostnames. Stacked base note: the intended base (benvin/watchstate-deploy, PR #419) merged into main before this PR opened, so this targets `main`. ## How - **DNS**: `watchstate-dns-internal` DNSRecord in `apps/base/bind-internal/authoritative/records.yaml` (unkin.net zone, bind-internal) -> `A 198.18.199.0`, the traefik-external VIP. Mirrors the `arrstack-dns-internal` precedent (external front door via bind, not external-dns). Kept in the central bind-internal location, so no AppProject destination widening is needed. - **Gateway**: new `watchstate-external` Gateway (`gatewayClassName: traefik-external`), http + https listeners on hostname `watchstate.unkin.net`. TLS terminated with the Let's Encrypt `*.unkin.net` wildcard secret `wildcard-unkin-net-tls` — so **no cert-manager and no external-dns annotations**. - **Routes**: `watchstate-external-http-redirect` (http->https 301) and `watchstate-external` (https -> `watchstate-oauth2:80`), mirroring the arrproxy/cheeztv external patterns. - **oauth2-proxy dual-host**: `OAUTH2_PROXY_REDIRECT_URL` changed from the pinned cluster callback to the **relative** `/oauth2/callback`. With `OAUTH2_PROXY_REVERSE_PROXY=true` (already set), oauth2-proxy derives scheme+host per request from `X-Forwarded-Proto/Host` (verified in oauth2-proxy v7.15.3 `getOAuthRedirectURI`: host-less redirectURL falls through to request-derived), so the callback works on both `watchstate.unkin.net` and `watchstate.k8s.syd1.au.unkin.net`. `COOKIE_DOMAINS` and `WHITELIST_DOMAINS` now list both hosts (one cookie per host; a single parent-domain cookie can't cleanly span unkin.net vs k8s.syd1.au.unkin.net). - Dropped the no-op `argocd.argoproj.io/sync-wave: "0"` annotation on the `default` VaultAuth. New resources intentionally carry no sync-wave annotations. ## Validation - `kustomize build --enable-helm` on `apps/overlays/au-syd1/watchstate` and `.../bind-internal`: OK. - kubeconform (repo CI config, k8s 1.33.7): watchstate overlay 15/15 valid (2 Gateways, 4 HTTPRoutes); bind-internal overlay 67/67 valid (incl. `watchstate-dns-internal`). ## Dependencies - **Reflector allow-list (argocd-apps PR #418)**: reflects `wildcard-unkin-net-tls` into the `watchstate` namespace. This PR references that secret as if present. - **Authentik (terraform-authentik, separate PR)**: register both `https://watchstate.unkin.net/oauth2/callback` and `https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback` as redirect URIs on the watchstate provider. Reviewed-on: #420 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
062d217ea8 |
bind-internal: add cheeztv.unkin.net record (#421)
why: cheeztv.unkin.net never resolved — external-dns ignores unkin.net (domain-filter is k8s.syd1.au.unkin.net only); *.unkin.net names are DNSRecord CRs. - add cheeztv-dns-internal A record → 198.18.200.4 (internal traefik VIP serving the cheeztv gateway) Reviewed-on: #421 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
5da12e398d |
Extend LE *.unkin.net wildcard to arrstack + authentik (reflect into gitea) (#418)
## Why #417 stood up the central Let's Encrypt `*.unkin.net` wildcard (Certificate `wildcard-unkin-net` in `cert-manager`, secret `wildcard-unkin-net-tls`, emberstack-reflected). This wires the remaining single-label public hosts onto that publicly-trusted leaf instead of the internal Vault-PKI CA. A `*.unkin.net` wildcard matches **single-label** subdomains only, so cluster hostnames under `*.k8s.syd1.au.unkin.net` are deliberately left on `vault-issuer`. > Note: PR #417 already merged to `main`, so this is no longer stacked — it targets `main` directly. ## How - **cert-manager**: reflect `wildcard-unkin-net-tls` into `arrstack`, `authentik`, `gitea` (appended to both reflector namespace lists → `cheeztv,arrstack,authentik,gitea`). - **arrstack**: repoint the public `arrproxy` gateway listener (`arrstack.unkin.net`) `certificateRefs` → `wildcard-unkin-net-tls`; drop its `cert-manager.io/*` vault-issuer annotations. - **authentik**: repoint the `identity.unkin.net` https listener → `wildcard-unkin-net-tls`. The `identity.k8s.syd1.au.unkin.net` listeners keep `authentik-tls` (vault-issuer), so the gateway `common-name` is retargeted to that cluster host. `authentik-ldap` gateway untouched. - **gitea**: secret **reflected only**; `git.unkin.net` cutover deferred (no manifest change), same deferred pattern as fafflix. ## Validation - `kustomize build` OK for cert-manager / arrstack / authentik / gitea overlays. - `kubeconform` all valid (0 invalid, 0 errors) across the four overlays. - Verified no `*.k8s.syd1.au.unkin.net` listener lost its vault-issuer wiring. Reviewed-on: #418 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
48632eb9f9 |
watchstate: deploy admin-gated jellyfin watch-state sync tool (#419)
## Why Deploy WatchState (arabcoders/watchstate), the Jellyfin/Plex/Emby watch-state sync tool, as an internal admin tool. It gets an admin-only web UI/API gated the same way as logviewer: an oauth2-proxy front backed by Authentik OIDC. ## What - New `watchstate` namespace + media-project app; base at `apps/base/watchstate`, overlay at `apps/overlays/au-syd1/watchstate`. - Image `ghcr.io/arabcoders/watchstate:v1.10.3` (current release; canonical upstream name, containerd mirrors route ghcr via artifactapi). - `replicas: 1`, `strategy: Recreate`, single `5Gi` `cephrbd-fast-delete` RWO PVC at `/config` — sqlite + the in-container cron/redis are single-writer. - `runAsUser/runAsGroup/fsGroup: 1000` (image's rootless user); liveness/readiness `GET /v1/api/system/healthcheck` on 8080 (route confirmed in upstream `src/API/System/HealthCheck.php`, no auth guard). - oauth2-proxy (mirrors logviewer 1:1) fronts every path. Authentik issuer `identity.k8s.syd1.au.unkin.net`, redirect `https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback`. Authorization is enforced Authentik-side (akR-global-admin only), so no oauth2-proxy group allowlist is configured. - Internal-only Gateway (`traefik-internal`) for `watchstate.k8s.syd1.au.unkin.net`, `vault-issuer` TLS leaf, external-dns to `198.18.200.4`. HTTP -> HTTPS redirect. - `VaultStaticSecret` pulls OIDC creds from `kv/kubernetes/namespace/watchstate/default/oauth-credentials`; `vault-ca-cert` auto-reflects into the namespace. - Registered in the media `ApplicationSet` + `AppProject`. ## Scrape decision No `VMPodScrape`: WatchState exposes no Prometheus/`/metrics` endpoint. ## Follow-ups - **Seed check:** the VaultStaticSecret expects `client_id`, `client_secret`, and `cookie_secret` keys at the kv path. Only `client_secret` was confirmed seeded; `client_id` and a generated `cookie_secret` must also be present or the oauth2-proxy pod will not start. - **Webhook ingestion:** the Jellyfin webhook endpoint (`/v1/api/webhook`) currently sits behind oauth2-proxy like everything else. When sync is wired up, the jellyfins pushing webhooks will need an auth-bypass or an apikey route for that path. Reviewed-on: #419 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
d42d013541 |
cheeztv: terminate public cheeztv.unkin.net TLS with the LE *.unkin.net wildcard (#417)
why: cheeztv.unkin.net needs a publicly-trusted cert; DNS-01 via the delegated acme.unkin.net zone is now working (staging wildcard verified end-to-end in ~70s). A single centrally-managed wildcard avoids per-host CNAMEs and LE duplicate-order rate limits. - add Certificate wildcard-unkin-net (cert-manager ns, ClusterIssuer letsencrypt prod, secret wildcard-unkin-net-tls) with emberstack reflector secretTemplate annotations - reflect the secret into the cheeztv namespace (append future consumers to the two reflector namespace lists) - point the public cheeztv gateway listener at wildcard-unkin-net-tls and drop its vault-issuer annotations (cheeztv-internal stays on vault-issuer) Reviewed-on: #417 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
1ecba543aa |
arrstack: wire arrproxy v0.4.0 tier routing (fafflix/cheeztv) (#416)
## Why
The old bare `sonarr`/`radarr` Services are gone, but arrproxy still ran the v0.3.x built-in default topology pointing at non-existent `sonarr`/`radarr` upstreams and mounted dead `sonarr-apikey`/`radarr-apikey` Secrets. This cuts the front door over to arrproxy v0.4.0 tier routing so the adult (fafflix, `/3aa168`) and kids (cheeztv, `/3df803`) *arr instances are reachable again, with per-tier group authorization.
## Changes
- Bump `arrproxy-api` and `arrproxy-ui` images to `v0.4.0`.
- Add `arrproxy-tiers` ConfigMap (`ARRPROXY_TIERS_FILE`): `fafflix` + `cheeztv` tiers, each with sonarr/radarr upstreams, route hash, key subdir and group grants. Adults (`akP-media-fafflix`) reach both tiers all methods; kids (`akP-media-cheeztv`) reach only cheeztv, GET/HEAD. `legacyRoutes:false` retires the unprefixed routes.
- Because arrproxy strips `/<hash>` and proxies `/<app>/...`, each upstream URL carries the `/<hash>` path segment so the joined upstream path (`/3aa168/sonarr/...`) matches the *arr's own UrlBase.
- Mount the tiers file and rework the projected keys volume to the real per-instance `{sonarr,radarr}-{adult,kids}-apikey` Secrets under `<keyDir>/<app>` (`adult/*`, `kids/*`); refresh reloader annotations.
- oauth2-proxy: route `/3aa168` and `/3df803` to arrproxy-api (drop dead `/sonarr` `/radarr` `/prowlarr`; prowlarr is served directly), and widen `SKIP_AUTH_REGEX` to `^/[^/]+/[^/]+/api` for the hash-prefixed tier api paths.
Group headers keep flowing via the existing Authentik `ak_groups` claim -> oauth2-proxy `X-Forwarded-Groups` mechanism. Validated with `kustomize build` + repo kubeconform (88/88 valid). No new Vault seeds required — the per-instance apikey Secrets already exist.
Reviewed-on: #416
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
fe689dc08e |
Replace legacy jellyfin app with fafflix (adult, cheeztv pattern) (#415)
## Why Rebuild the adult media instance as `fafflix`, mirroring the kids instance (`cheeztv`) HA pattern (Postgres-backed jellyfin-ha fork, Valkey transcode-lease store, shared-RWX transcode, CNPG + k8up/restic backups, static CephFS media PVs). Ben: "replace the adult jellyfin with the same pattern as kids." The adult instance holds no data currently, so the wipe/replace is sanctioned. Stacked on top of `benvin/cheeztv` (#413) — base is that branch, not main. ## How - Add `apps/base/fafflix/` — cheeztv pattern with `s/cheeztv/fafflix` on names/namespace/labels/secrets/buckets and the Vault kv path `kubernetes/namespace/fafflix/default/k8up-restic`. - Media: mount the shared movies/tv CephFS subvolumes' `adult` subtree at `/media/movies` and `/media/tv`, plus the `kids` subtree at `/media/movies-kids` and `/media/tv-kids` (same two static PVs, new unique volumeHandles `fafflix-media-{movies,tv}-static`) so fafflix can resume kids content started on cheeztv. All media mounts readOnly, matching cheeztv. - **Hostname preserved:** fafflix keeps serving the legacy jellyfin host `jellyfin.k8s.syd1.au.unkin.net` (TLS secret `fafflix-tls`, cert-manager re-mints). The dedicated fafflix domain switch is explicitly deferred — no `fafflix.unkin.net` created. - config PVC on `cephfs-raid5-delete` (delete variant). - Remove `apps/base/jellyfin/` + its overlay; swap `jellyfin`->`fafflix` in the `media-apps` ApplicationSet directory glob and the `media` AppProject namespace destination. - Third-party images stay canonical upstream; the in-house `jellyfin-ha` image ref is unchanged. ## Verification - `kustomize build` + `kubeconform -strict` clean on the fafflix overlay and the whole media appset scope (fafflix/cheeztv/arrstack): 0 invalid, 0 errors. - No dangling `apps/base/jellyfin` references; `jellyfin.k8s.syd1.au.unkin.net` now served by exactly one app (fafflix); no `fafflix.unkin.net` anywhere. Reviewed-on: #415 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
9db52c5e26 |
arrstack: rebuild adult tier on the shared-Postgres pattern (#414)
## Why Rebuild the ADULT arr tier on the same shared-Postgres, active-active pattern the kids tier now uses, replacing the legacy adult sonarr/radarr/nzbget. The old adult apps hold **no data**, so the wipe is sanctioned. **Stacked on** `benvin/arrstack-kids` (#412) — review/merge that first. ## How - **Add** `sonarr-adult` / `radarr-adult` / `nzbget-adult` (copies of the `*-kids` dirs, `s/kids/adult`): - own DBs `sonarr-adult-main` / `radarr-adult-main`, Vault paths `.../default/{sonarr,radarr}-adult(+-db)` - S3 prefixes `sonarr-adult` / `radarr-adult`; UrlBase + probe + exportarr URLs `/3aa168/{sonarr,radarr}` (adult hash) - media subPaths `tvshows/adult` + `movies/adult`; nzbget config PVC on `cephrbd-fast-delete` - **postgres**: add managed roles `sonarr-adult`/`radarr-adult`, Database CRDs `{sonarr,radarr}-adult-main`, VaultStaticSecrets `{sonarr,radarr}-adult-db` - **Remove** legacy `sonarr`/`radarr`/`nzbget` dirs (incl. their gateway/httproute), managed roles `sonarr`/`radarr`, databases `{sonarr,radarr}-main`, VSS `{sonarr,radarr}-db`, and all kustomization entries - Third-party images stay canonical upstream; in-house images stay explicit `docker-internal` `prowlarr`, `arrproxy`, and everything else untouched. arrproxy path re-wiring is a later PR. Verified: `kustomize build` + `kubeconform` (87/87 valid), no dangling refs to removed apps, no `3df803` in adult files. Reviewed-on: #414 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
94da740a9d |
Add cheeztv kids Jellyfin instance + fafflix kids mounts (#413)
## Why Kids need their own safe Jellyfin front (cheeztv.unkin.net) with a library that only ever exposes kids titles, while those same titles stay resumable in the existing adult instance (fafflix). The fafflix.unkin.net domain switch is deliberately out of scope. ## How - **New `cheeztv` instance** (`apps/base/cheeztv`): mirrors the fafflix HA Jellyfin stack 1:1 - same `jellyfin-ha:v0.1.3` fork image, 2-replica StatefulSet, RWX transcode PVC + HA takeover, Intel iGPU transcode, CNPG Postgres + PgBouncer, Valkey transcode store, k8up config backup, VMPodScrape. - **Kids-only media**: its own static PVs on the *same* shared movies/tv CephFS subvolumes, mounted with `subPath: kids` so it sees only `movies/kids` and `tvshows/kids`. - **Fully separate state**: own namespace, config PVC, `cheeztv-postgres` cluster, `cheeztv-valkey`, and `cnpg-cheeztv` / `cheeztv-config-backup` RGW buckets - nothing shared with fafflix. - **DNS/cert**: `cheeztv.unkin.net` via internal Traefik + external-dns (198.18.200.4) + cert-manager `cheeztv-tls`, matching logviewer.unkin.net; plus the cluster hostname `cheeztv.k8s.syd1.au.unkin.net` mirroring fafflix. - **fafflix change**: ADDS `movies/kids` + `tvshows/kids` subPath mounts alongside its current media mounts so kids libraries are resumable there. Existing mounts, hostname and ingress untouched. - Registers cheeztv in the media ApplicationSet generator and AppProject destinations. ## Validation kubeconform clean: cheeztv 28/28, jellyfin 25/25 valid. No arr/watchstate/domain changes. Note: the k8up restic repo password must be seeded at `kv/kubernetes/namespace/cheeztv/default/k8up-restic` (same manual step as fafflix); templated Vault policy already grants read, no terraform-vault change needed. Reviewed-on: #413 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
b419635fc2 |
arrstack: add kids backend tier (sonarr/radarr/nzbget-kids) (#412)
Splits the arrstack media pipeline into an adult and a kids tier so kids content lives in its own libraries, databases, and downloader while reusing the shared prowlarr, CNPG cluster, Valkey, media PVCs, and S3 buckets. Backend workloads only — no routing/ingress, no arrproxy, no adult-instance changes in this PR. How: - sonarr-kids / radarr-kids: mirror the adult -unkin fork Deployments (same images/tags, 3 active-active replicas) on their own Postgres DBs (sonarr-kids-main / radarr-kids-main), UrlBase /3df803/sonarr and /3df803/radarr, and kids MediaCover/Backup S3 prefixes. - Media mounts use the tvshows/kids and movies/kids subPaths of the shared media-tv / media-movies PVCs, so kids libraries are isolated on the same CephFS subvolumes. - nzbget-kids: second downloader on its own RWO config PVC, completed downloads land in the kids media subtrees for same-filesystem hardlink imports. - postgres: two managed roles + Database CRDs and their VSO-synced <app>-kids-db credential secrets. - per-app VMPodScrape for each kids exportarr sidecar; new apps registered in the base kustomization. Validated: kustomize build + kubeconform (96/96 valid), yamllint + pre-commit clean. Reviewed-on: #412 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
a064d9a3c4 |
arrproxy: bump images to v0.3.1 (#411)
Deploy the new space-pirate UI background released in arrproxy v0.3.1. - Bump arrproxy-api image to v0.3.1 - Bump arrproxy-ui image to v0.3.1 Reviewed-on: #411 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
aea319cec0 |
authentik: ClientIP session affinity on server service (#410)
## Why Tofu applies against authentik hit a cross-replica read-back race: a create lands on one of the 3 server pods and the provider's immediate read-back hits another pod that returns stale data, failing the run with "inconsistent result after apply" (upstream bug goauthentik/terraform-provider-authentik#954, still open). ClientIP session affinity keeps a whole apply run on a single pod, sidestepping the race until the provider is fixed. ## Changes - Sets `server.service.sessionAffinity: ClientIP` with `sessionAffinityConfig.clientIP.timeoutSeconds: 60` in the au-syd1 authentik overlay values (rendered by the vendored chart's `templates/server/service.yaml`) Reviewed-on: #410 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
5a23146513 |
logging: set clickhouse readonly profile to readonly=2 (#409)
## Why The `readonly` profile used by `logreader` sets `readonly=1`, which rejects any per-query settings change. HTTP clients (logviewer, chlog) send settings params like `max_execution_time` with their queries, so every query fails with `Code: 164 DB::Exception: Cannot modify 'max_execution_time' setting in readonly mode` — logviewer healthz returns 503 and CrashLoops, chlog queries fail. `readonly=2` keeps data access read-only while permitting settings changes, which is what HTTP clients need. ## Changes - Changes the `readonly` profile in `apps/base/logging/clickhouseinstallation.yaml` from `readonly: "1"` to `readonly: "2"` Reviewed-on: #409 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
03c56c0ff0 |
arrstack: raise arrstack-postgres resources after OOMKills (#408)
## Why Both `arrstack-postgres` replicas were OOMKilled at the 1Gi memory limit on 2026-08-23 07:19, causing a failover to timeline 14, leaving replica-3 unrecoverable via pg_rewind, and blocking the argocd sync wave on Cluster Ready=False. ## Changes - Raises the CNPG Cluster resources to requests 1Gi memory / 250m CPU and limits 2Gi memory / 1 CPU **NOTE:** applying this restarts the postgres instances (CNPG rolling restart). Reviewed-on: #408 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
61b5a7f695 |
Add Traefik Grafana dashboard (#407)
## Why Traefik metrics (traefik_entrypoint_requests_total, traefik_service_*, etc.) are now flowing into VictoriaMetrics with the relabeled `traefik_instance` label, but there is no dashboard to view them. ## Changes - Add `apps/base/grafana/dashboards/traefik.yaml`: GrafanaDashboard CR mirroring the redis.yaml sibling (gzipJson embed, `instanceSelector dashboards: grafana`, `resyncPeriod: 5m`, `allowCrossNamespaceImport: false`) - Embed the official Traefik Standalone dashboard (grafana.com ID 17346, revision 9), adapted: - strip `__inputs`/`__requires`/`__elements` and the `DS_PROMETHEUS` datasource variable - rewrite all prometheus datasource uids to the estate VictoriaMetrics uid `det2y55dac4jkc` - add an `instance` template variable over `traefik_instance` and filter every query with `traefik_instance=~"$instance"` so traefik-internal / traefik-external are switchable - chain the entrypoint/service variable queries through the instance filter - Register the dashboard in `apps/base/grafana/kustomization.yaml` Validated: gzipJson decode roundtrip, `kustomize build`, kubeconform. Reviewed-on: #407 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
8da505ceb4 |
arrproxy: bump image versions to v0.3.0 (#406)
Ship icon grid + jellyfin tile features by upgrading arrproxy UI and API to v0.3.0. - Bump arrproxy-api image from v0.2.0 to v0.3.0 - Bump arrproxy-ui image from v0.2.0 to v0.3.0 - Verified images built and pushed to artifactapi Reviewed-on: #406 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
f6c297dcb5 |
Add logviewer.unkin.net internal DNS record (#405)
## Why logviewer.unkin.net does not resolve, so the log viewer UI is unreachable even though its Gateway and certificate already exist (from #404). ## Changes - Adds DNSRecord `logviewer-dns-internal` (A, ttl 600 → 198.18.200.4, the internal traefik gateway VIP) to bind-internal authoritative records, mirroring the existing ghp/identity records Reviewed-on: #405 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
e608dadae7 |
observability: fix dropped kubelet/cadvisor metrics, add kube-state-metrics + VMAlert k8s.rules (#399)
## Why
Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer.
## Changes
- replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel
- renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel
- adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote
- adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules)
- adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules)
## Prerequisite
The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff.
## Notes
- validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
- the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
Reviewed-on: #399
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
13a04a6ebc |
Export and scrape metrics from all redis/valkey instances (#398)
## Why No redis/valkey instance managed in this repo exports metrics today — the six caches (artifactapi, authentik, litellm, gitea, netbox, jellyfin) are blind spots in VictoriaMetrics. ## What - Enables the valkey-operator exporter sidecar on the jellyfin ValkeyCluster (`exporter.enabled: true`); overrides the operator's default bare-dockerhub image (`oliver006/redis_exporter:v1.80.0`) with the artifactapi-proxied pin. The operator manages a dedicated `_exporter` ACL user for it. - Adds a `metrics-exporter` sidecar (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/oliver006/redis_exporter:v1.89.0`, :9121, port name `metrics`, req 32Mi/25m lim 64Mi/100m) to the five raw Deployments: artifactapi/authentik/litellm redis and gitea/netbox valkey. gitea/netbox sidecars get the same restrictive securityContext as their valkey container. - Adds the `monitoring.unkin.net/redis-exporter: "true"` opt-in pod label to all five pod templates. - Adds `apps/base/observability/vmpodscrape-redis.yaml` with two VMPodScrapes (any namespace, port `metrics`): `redis-exporters` selecting the opt-in label, and `valkey-operator-clusters` selecting `app.kubernetes.io/managed-by: valkey-operator` + `app.kubernetes.io/component: valkey-node` — the ValkeyCluster CR has no pod-label passthrough (verified against the v0.5.0 CRD/source), so operator pods cannot carry the opt-in label. ## Notes - No instance runs with `requirepass`/auth, so no REDIS_PASSWORD wiring is needed on the raw Deployments; the jellyfin exporter auth is operator-managed. - Validated: `kustomize build` on every touched base + au-syd1 overlay (helm overlays with `--enable-helm`) and `kubeconform -strict -ignore-missing-schemas` (68 valid, 0 invalid). Reviewed-on: #398 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
08a3493097 |
logging: deploy logviewer v0.1.0 behind oauth2-proxy (#404)
## Why The logviewer UI gives browser access to the ClickHouse log store (`logreader` user). It must only be reachable by authorized users, so it sits behind an Authentik-gated oauth2-proxy front at https://logviewer.unkin.net, mirroring the arrstack front-door and mediamover/pdbmux gateway patterns. ## Changes - Adds the logviewer Deployment (1 replica, `docker-internal/logviewer:v0.1.0`) reading ClickHouse at `clickhouse-logs` via the existing `clickhouse-logreader` Secret, `/healthz` probes, nonroot securityContext - Adds an oauth2-proxy (quay.io/oauth2-proxy/oauth2-proxy:v7.15.3) front with issuer `https://identity.unkin.net/application/o/logviewer/`, `ak_groups` claim, allowed group `akP-logviewer-admin`, internal-CA trust via the combine-certs initContainer - Syncs kv `kubernetes/namespace/logging/default/oauth-credentials` into the `logviewer-oauth-credentials` Secret via the existing `default` VaultAuth (kv path already seeded; no terraform-vault change needed) - Adds ClusterIP Services for the app and the oauth2-proxy front door - Adds internal Gateway + HTTPRoutes for `logviewer.unkin.net` (vault-issuer cert, external-dns -> 198.18.200.4, http->https redirect) - Registers `logviewer/` in the logging base kustomization ## Dependency Login requires the Authentik `logviewer` provider/application from terraform-authentik PR #21. This PR is safe to merge first: the VSO secret sync and the deployment come up regardless; OIDC login only works once #21 is applied. Validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (49/49 valid). Reviewed-on: #404 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
4e3c8288e2 |
Bump radarr to v6.4.2-unkin7 (#403)
Ship Commands-index migration (PR #13) which adds partial indexes on Commands for queue claim polling. - Bump radarr image to v6.4.2-unkin7 in apps/base/arrstack/radarr/deployment.yaml Reviewed-on: #403 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
c649cc7ee3 |
Bump prowlarr to v2.6.2-unkin6 (#402)
Ship Commands-index migration. - Update prowlarr image tag to v2.6.2-unkin6 in apps/base/arrstack/prowlarr/deployment.yaml Reviewed-on: #402 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
cd51078d79 |
logging: add read-only clickhouse logreader user + Vault-synced secret (#400)
## Why CLI tools and the logviewer UI need ClickHouse access without write or DDL rights. The only existing app user (`vector`) has `access_management` and write access; this adds a properly scoped read-only login using the same secret-sourcing mechanism. ## Changes - Adds a `logreader` user to the `logs` ClickHouseInstallation: password hash via `secretKeyRef` to the Vault-synced `clickhouse-logreader` Secret (same mechanism as `vector`), networks `::/0`, `readonly` profile, scoped to the `logs` database only - Adds a `readonly` profile (`readonly: 1`, same memory/time limits as default) - Adds a `clickhouse-logreader` VaultStaticSecret mirroring the existing `clickhouse-credentials` object (kv-v2, 5m refresh, VSO-created destination Secret) ## Vault kv Seeded at `kv/kubernetes/namespace/logging/default/clickhouse-logreader` (version 1) with keys: `username`, `password`, `password_sha256_hex`. No secret material lands in git; the CHI only ever sees the sha256 hash via the synced Secret. Reviewed-on: #400 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
b5acb19ac0 |
Bump arrstack image pins: radarr v6.4.2-unkin6, sonarr v5.0.0-unkin6 (#401)
Ship ffprobe exec bit fix (radarr) and Commands-index migration (sonarr). - Update radarr image tag v6.4.2-unkin5 → v6.4.2-unkin6 (ffprobe fix) - Update sonarr image tag v5.0.0-unkin5 → v5.0.0-unkin6 (Commands-index) Reviewed-on: #401 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
ba71bd1a35 |
Scrape traefik gateway metrics into VictoriaMetrics (#397)
## Why Both traefik instances (traefik-internal/traefik-external, chart 40.0.0, image v3.7.0) already expose prometheus metrics on container port `metrics` (9100), but nothing scrapes them — VictoriaMetrics has no traefik series (verified empty). The observability VMAgent runs with selectAllByDefault and consumes VM*Scrape CRs only, so a VMPodScrape is all that is needed. ## Changes - Adds `apps/base/observability/vmpodscrape-traefik.yaml`: scrapes port `metrics` /metrics on `app.kubernetes.io/name: traefik` pods in traefik-system, relabeling `app.kubernetes.io/instance` to a stable `traefik_instance` label to split internal vs external series - Registers the new CR in the observability base kustomization - Enables `metrics.prometheus.addEntryPointsLabels` and `addRoutersLabels` in both au-syd1 traefik values files for per-entrypoint/per-route dashboards (pods restart on rollout — acceptable) Validated with `kustomize build` (observability base + overlay, traefik-system overlay with --enable-helm) and kubeconform (unknown CRDs skipped); rendered deployments carry the new metric flags and the VMPodScrape renders in the overlay. Reviewed-on: #397 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
8fd1d83b1b |
Add Redis/Valkey Grafana dashboard (#396)
## Why
The grafana-operator estate has no dashboard for Redis/Valkey; redis_exporter instances are being added in a parallel PR and need visibility.
## Changes
- Adds `apps/base/grafana/dashboards/redis.yaml`, a GrafanaDashboard CR embedding grafana.com dashboard 763 ("Redis Dashboard for Prometheus Redis Exporter 1.x") as gzipJson like the sibling dashboards, with namespace/instance template variables over `redis_up` for multi-instance use and all datasource refs pinned to the VictoriaMetrics datasource uid (`det2y55dac4jkc`), registered in the grafana kustomization
Reviewed-on: #396
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
19f7afac92 |
Add shared arrstack Valkey and wire arr apps' Redis features (#394)
Activates the arr forks (sonarr/radarr/prowlarr) #14 Redis features — SignalR backplane, cross-replica cache-invalidation bus, and distributed rate limiter — which ship deployed but inert because no Valkey exists in arrstack and nothing is wired to it. Ben wants ONE Valkey shared by all three apps. ## Changes - Adds a single shared `ValkeyCluster` (`arrstack-valkey`) under `apps/base/arrstack/valkey/`, modeled on `jellyfin-valkey`: `shards:1`/`replicas:2` HA (one primary + two replicas, automatic failover), image via artifactapi (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/valkey/valkey:9.0.0`), operator-default passwordless `default` user, node spread across hosts, cluster-aware PDB, ephemeral `/data`. - Registers the `valkey` component in the arrstack base kustomization. - Wires `<App>__Redis__Host` + `<App>__Redis__Port` into the `sonarr-env` / `radarr-env` / `prowlarr-env` ConfigMaps, all pointing at the shared service `valkey-arrstack-valkey.arrstack.svc.cluster.local:6379`. ## Notes - Setting `Host` is the activation switch: the fork's `RedisOptions.IsConfigured` gates purely on a non-empty `Host`, so there is no separate Enabled flag. - Passwordless (jellyfin parity — the operator leaves the `default` user without auth), so no `Password`/`Ssl` is wired and there is no secret to reference. - One cluster is safe for all three: each fork namespaces its keys and pub/sub channels by a per-app prefix (`sonarr:ratelimit:` / `radarr:ratelimit:` / `prowlarr:ratelimit:`), so their state never collides. - App `image:` (-unkin5), the waitfordb initContainer, and S3 buckets are untouched. - Validated: `kubectl kustomize` renders clean for both `apps/base/arrstack` and `apps/overlays/au-syd1/arrstack`. Follow-up: after merge, Valkey must come up and the three apps must roll (pick up the new env) before the #14 features can be validated live. --------- Co-authored-by: Ben Vincent <ben@unkin.net> Reviewed-on: #394 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
b3d202b8fa |
arrstack: add exportarr metrics sidecars to sonarr and radarr (#395)
Sonarr and radarr currently expose no Prometheus metrics, so queue depth, indexer health, and library stats are invisible to the observability stack. This adds the standard exportarr exporter as a sidecar to both Deployments. - Adds an exportarr sidecar (ghcr.io/onedr0p/exportarr v2.3.0, pulled through the artifactapi `ghcr` remote, which already whitelists `^onedr0p/`) to the sonarr and radarr Deployments - Points each exporter at its local replica via env (`URL=http://localhost:8989/sonarr` / `:7878/radarr`, matching the apps' UrlBase) and reuses the existing VSO-synced API-key Secrets (`sonarr-apikey`/`radarr-apikey`, key `apitoken`) via `APIKEY` secretKeyRef — no key in argv - Serves metrics on 9707 (sonarr) / 9708 (radarr) with `/healthz` liveness+readiness probes and small resources (25m/32Mi requests, 100m/128Mi limits) - Adds per-app VMPodScrape objects (picked up by the observability VMAgent, selectAllByDefault); pod-level rather than VMServiceScrape because the app Services don't expose the metrics port Validated with `kustomize build apps/base/arrstack` + kubeconform (0 invalid). Note: with 3 replicas per app, each pod's exporter reports the same shared-Postgres-backed stats, so series are duplicated across the `pod` label; dashboards should aggregate with `max` or filter to one pod. Reviewed-on: #395 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
32611bfd16 |
Deploy mediamover v0.1.0 into arrstack (#393)
Deploys the released mediamover v0.1.0 into the arrstack namespace: a single-replica server (API + UI on :8080) that browses the mediafs source, queues per-file copy/move operations into the media PVCs, and spawns one worker Job per file using its own image — its ServiceAccount/RBAC grants the Job and Pod access that requires. All PVs/PVCs it mounts are already live. - Adds `apps/base/arrstack/mediamover/` with SA, Role/RoleBinding (batch jobs create/get/list/watch/delete; pods get/list/watch; pods/log get/list), Deployment, Service, Gateway, HTTPRoute - Runs `mediamover server` with `--src-root /srv/src --src-pvc mediafs`, `--dst-roots movies=/srv/dst/movies,tv=/srv/dst/tv`, `--dst-pvc movies=media-movies,tv=media-tv`, `--namespace arrstack`, `--image ...mediamover:v0.1.0` - Mounts mediafs RW at /srv/src (move deletes the source) plus media-movies and media-tv destinations; keeps 1 replica with Recreate strategy since the queue is in-memory - Names the Service `mediamover` on 8080 to match the worker callback default `http://mediamover.arrstack.svc:8080` - Exposes the UI at https://mediamover.k8s.syd1.au.unkin.net via a dedicated internal Gateway (pdbmux pattern); the existing arrproxy Gateway is external and hostname-locked to arrstack.unkin.net so it cannot carry this route - Probes hit `GET /api/limit` (the server has no dedicated health endpoint); registers `mediamover` in the arrstack base kustomization Reviewed-on: #393 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
7c49a9f3ad |
Deploy arrstack S3 TLS-trust fix (arrstack images -unkin4 -> -unkin5) (#392)
## Why Ben merged the S3 TLS-trust fix into each Servarr fork's `main` (sonarr#10, radarr#11, prowlarr#10, branch `benvin/s3-tls-trust-fix`). The fix corrects TLS trust for the leaf-only certificate presentation of the Ceph RGW endpoint, so backups and MediaCover uploads to S3 succeed across all replicas instead of failing the TLS handshake. The `-unkin5` images carrying the fix are built and pushed to `artifactapi.k8s.syd1.au.unkin.net/docker-internal/<app>`. This PR rolls them out. ## Changes - Bump `apps/base/arrstack/sonarr/deployment.yaml` app image `v5.0.0-unkin4` -> `v5.0.0-unkin5` - Bump `apps/base/arrstack/radarr/deployment.yaml` app image `v6.4.2-unkin4` -> `v6.4.2-unkin5` - Bump `apps/base/arrstack/prowlarr/deployment.yaml` app image `v2.6.2-unkin4` -> `v2.6.2-unkin5` Only the application container `image:` line changes in each file; the `waitfordb` initContainer and bucket YAML are untouched. Reviewed-on: #392 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
671c075a08 |
Mount mediafs CephFS into arrstack (#391)
Exposes the legacy mediafs CephFS filesystem to the arrstack namespace so the upcoming mediamover tool can migrate media and run bandwidth tests against the old filesystem; the ceph-mediafs client secret is already seeded in Vault. The mover deployment itself lands in a follow-up PR. - Adds a `ceph-mediafs-secret` VaultStaticSecret in csi-cephfs syncing `kv/service/kubernetes/au/syd1/csi/ceph-mediafs-secret` - Adds a 10Ti static RWX PV `arrstack-mediafs` (fsName `mediafs`, rootPath `/`, staged with the mediafs secret) pinned by claimRef - Adds the matching `mediafs` PVC in arrstack (storageClassName "", volumeName-bound, backups excluded) - Registers all three in their base kustomizations Reviewed-on: #391 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
06fe261661 |
arrstack: deploy -unkin4 (Redis SignalR backplane + S3 storage) (#390)
## Why The cluster runs the arrstack forks at `-unkin3`. Ben merged two workstreams into each fork's `main`: - **Redis SignalR backplane + cross-replica cache invalidation** — SignalR messages and cache-invalidation events fan out across replicas via Redis, so any replica serves consistent live UI/state. - **MediaCover -> S3 + backups -> S3** — MediaCover (sonarr/radarr) and backups (all three) use an optional S3 store, so any stateless replica can serve covers and write/restore backups. Fresh `-unkin4` tags were cut on each fork's `main` HEAD, building `docker-internal/<app>:<base>-unkin4` images. This bumps the deployment images so ArgoCD rolls the merged work into the cluster. ## Changes - Bump sonarr app image `v5.0.0-unkin3` -> `v5.0.0-unkin4` - Bump radarr app image `v6.4.2-unkin3` -> `v6.4.2-unkin4` - Bump prowlarr app image `v2.6.2-unkin3` -> `v2.6.2-unkin4` The waitfordb initContainer and bucket resources are unchanged; only the app-container `image:` tags move. Reviewed-on: #390 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
bdf6114794 |
arrstack: S3 buckets + wiring for app backups and sonarr MediaCover (#389)
## Why Servarr backups (periodic config+DB zips) and sonarr MediaCover posters still live on each replica's local `/config` emptyDir: backups are leader-local and lost on reschedule, and sonarr covers render inconsistently across the 3 active-active replicas. #387 moved radarr covers to a shared Ceph RGW bucket; this extends that pattern to backups for all three apps and to sonarr's covers. ## Changes - Add an `arrstack-backups` ObjectStoreUser + Bucket (cephrgw-operator); the operator mints the `arrstack-backups-s3` credential Secret in-namespace (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / S3_ENDPOINT), so no Vault KV seeding is required. Backups get their own bucket (not `arrstack-media`) because they carry secrets (Config.xml API keys, the DB) and warrant separate credentials/lifecycle from poster art; the three apps share it, isolated by a per-app key prefix. - Wire sonarr to the shared `arrstack-media` bucket via `Sonarr__MediaCoverS3__*` (prefix `sonarr`, partitioned from radarr's `radarr` prefix to avoid movie/series id collisions), plus the estate CA mount at `/etc/ssl/vault-ca/ca.crt` for RGW TLS validation. - Wire radarr/sonarr/prowlarr backups via `<App>__BackupS3__*` to `arrstack-backups` (per-app prefix); add the CA mount to sonarr and prowlarr. ## Notes - Credentials come only from the operator-minted Secret via env; no secrets in manifests. - The env keys are consumed by the -unkin3+ images (separate app PRs) and ignored by older images, so this is safe to apply ahead of the image bump. - `kubectl kustomize apps/base/arrstack` builds clean. Reviewed-on: #389 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
eb37beb7e4 |
Deploy arrstack -unkin3 (active-active code) (#388)
Deploy the merged active-active work now on each Servarr fork's main. The cluster still runs the -unkin2 images, so the merged code (K8s process lifecycle, Redis-backed distributed rate limiting, and radarr MediaCover->S3) is not yet deployed. The -unkin3 tag pipelines have built and pushed new images to docker-internal. Changes: - Bump sonarr app image v5.0.0-unkin2 -> v5.0.0-unkin3 - Bump radarr app image v6.4.2-unkin2 -> v6.4.2-unkin3 - Bump prowlarr app image v2.6.2-unkin2 -> v2.6.2-unkin3 Touches only the app container image line in each apps/base/arrstack/<app>/deployment.yaml; the waitfordb initContainer is unchanged. Reviewed-on: #388 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
9cdcdf62bc |
arrstack: provision shared Ceph RGW bucket for media assets (#387)
## Why Servarr MediaCover assets (movie/series posters and fanart) and app backups currently live on each replica's local `/config`, which is an `emptyDir`. A poster the leader downloads is invisible to the other replicas behind the Service, so covers render inconsistently and backups are leader-local. This provisions a shared Ceph RGW (S3) bucket to hold those assets instead. ## Changes - Add an `arrstack-media` `ObjectStoreUser` + `Bucket` (cephrgw-operator), mirroring the CNPG backup pattern. The operator mints the `arrstack-media-s3` credential Secret in-namespace (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `S3_ENDPOINT`), so **no Vault KV seeding is required**. - Wire the radarr Deployment to the bucket via `Radarr__MediaCover__S3__*` env (creds from the operator Secret; estate CA mounted at `/etc/ssl/vault-ca/ca.crt` for RGW TLS validation). ## Notes - The env keys are unknown to the current `-unkin2` image and are ignored by it; they activate once the `-unkin3+` image (radarr MediaCover-S3 support, separate radarr PR) ships. sonarr/prowlarr wiring and backups-to-S3 follow in later PRs. - `kubectl kustomize apps/base/arrstack` builds clean. Reviewed-on: #387 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
5155cee5e7 |
arrstack: use waitfordb initContainer instead of inline psql loop (#386)
## Why The sonarr/radarr/prowlarr deployments gated app startup on their Postgres database being reachable using a hand-written `psql`-in-a-shell loop running the `postgres:17-alpine` image. Now that `unkin/waitfordb` is released (v0.1.0), replace that inline probe with the purpose-built tool so the wait logic lives in one tested place instead of being copy-pasted per workload. `waitfordb` reads the same `PG*` env as a libpq fallback and exits `0` once `SELECT 1` succeeds under those credentials/database, so the retrofit is a drop-in — the password still never lands in argv. ## Changes - Swap the `wait-for-db` initContainer image from `dockerhub/library/postgres:17-alpine` to `docker-internal/waitfordb:v0.1.0` (pinned, not `latest`) in all three arrstack deployments. - Drop the `sh -c` psql loop `command` — `waitfordb` is the image entrypoint. - Keep the existing `PGHOST/PGPORT/PGDATABASE/PGUSER/PGPASSWORD` env, the `wait-for-db` name, and the resource requests/limits unchanged. - Add `WAITFORDB_TIMEOUT=5m` so a stuck database fails the pod fast, and `WAITFORDB_SSLMODE=disable` to match the in-cluster Postgres. Validated with `kubectl kustomize apps/overlays/au-syd1/arrstack` (builds clean; 3 waitfordb refs, no remaining postgres init image). Reviewed-on: #386 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
8446c31414 |
arrstack: fix sync-wave deadlock + config multi-attach (#385)
## Why The active-active arrstack deploy (#383) is stuck in-cluster: the app Deployments (sync-wave 0) can never become Healthy without their Postgres DB, so ArgoCD never advances to create the shared `arrstack-postgres` cluster (wave 1). Separately, `/config` is an immutable RWO cephrbd PVC, so 3 replicas fail with Multi-Attach. ## How - Order DB infra before the apps: `<app>-db` VaultStaticSecrets + backup ObjectStoreUser/Bucket at sync-wave -3, CNPG `Cluster` -2, `Database` CRDs -1; app Deployments stay at default 0. - Switch each app's `/config` to an emptyDir per-pod volume (library DB is in Postgres, config is env-driven; MediaCover regenerates). Old RWO config PVCs orphan. Validated with kustomize build (62 resources). arrproxy/nzbget untouched. Reviewed-on: #385 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
e0d47295f2 |
Deploy arrproxy v0.2.0 machine-mint admin route (#384)
## Why The future Vault engine needs to machine-mint arrproxy tokens without an interactive Authentik session. arrproxy v0.2.0 adds an admin route (`POST /api/admin/...`) protected by its own bearer token; OpenBao running on the VMs calls it through the arrstack ingress. This deploys that route. ## Changes - Add an `arrproxy-admin-token` VaultStaticSecret (mirrors the `arrproxy-pepper` VSO: same `default` VaultAuth, k8s-auth default-SA pattern) syncing `kv/kubernetes/namespace/arrstack/default/arrproxy-admin-token` into the `arrproxy-admin-token` Secret. The token is seeded in Vault KV as the shared source of truth for the future Vault engine. - Set `ARRPROXY_ADMIN_TOKEN` on the arrproxy-api Deployment via `secretKeyRef` (key `token`), and add `arrproxy-admin-token` to the Reloader `secret.reloader.stakater.com/reload` annotation so the pod rolls on rotation. - Skip-auth the `/api/admin/` route in oauth2-proxy: `OAUTH2_PROXY_SKIP_AUTH_REGEX` becomes `^/[^/]+/api,^/api/admin/`. The admin route is intentionally oauth-skipped because it is protected by arrproxy's OWN bearer token so OpenBao can reach it machine-to-machine. `/api/tokens` and `/api/me` are NOT matched and stay oauth-gated. `/api/admin/*` already routes to the arrproxy-api upstream via the existing catch-all `/api/` upstream, so no upstream change is needed. - Bump arrproxy-api and arrproxy-ui images to `v0.2.0` (kept in lockstep). ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` succeeds. - pre-commit passes, including the plain-Secret guard (only the VSO CRD is used; no plain Secret objects added). Reviewed-on: #384 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
187fabb72b |
arrstack: run sonarr/radarr/prowlarr active-active on shared Postgres (#383)
## Why
The arrstack `sonarr`/`radarr`/`prowlarr` apps currently run as single-replica upstream LinuxServer images on SQLite (RWO config PVC, `Recreate` strategy) — no HA, no horizontal scale, and a hard one-pod ceiling because SQLite can't be shared. Our `-unkin2` forks move all app state into PostgreSQL and use Postgres advisory locks for cross-replica coordination (migrations, scheduled tasks, command execution), which makes an **active-active multi-replica** deployment safe.
This is a fresh Postgres backend — **no SQLite data migration**. The current single-replica pods are throwaway baselines; the new pods start on an empty per-app database and the fork's provider migrates the schema on first start. `arrproxy` (its own `arrproxy-db`) and `nzbget` are untouched.
## What
New `apps/base/arrstack/postgres/`:
- `cnpg_cluster.yaml` — CNPG `Cluster` `arrstack-postgres` (3 instances, `cephrbd-fast-delete`), throwaway `app` initdb owner, and 3 `managed.roles` (`sonarr`/`radarr`/`prowlarr`, login non-superuser) each with `passwordSecret: <app>-db`.
- `database-{sonarr,radarr,prowlarr}.yaml` — CNPG `Database` `<app>-main` owned by `<app>`, `databaseReclaimPolicy: retain`.
- `cnpg_backup.yaml` — cephrgw `ObjectStoreUser` + `Bucket cnpg-arrstack` + nightly `ScheduledBackup` (mirrors jellyfin).
- `vaultstaticsecret.yaml` — 3 VSO `VaultStaticSecret`s syncing `<app>-db` (keys `username`/`password`) from `kv/kubernetes/namespace/arrstack/default/<app>-db` (already seeded); these are both the CNPG role `passwordSecret` and the Deployment's Postgres creds.
- sync-waves: `<app>-db` Secrets (0) → Cluster (1) → Databases (2).
- added `postgres` to `apps/base/arrstack/kustomization.yaml`.
Per app (`sonarr`/`radarr`/`prowlarr`):
- `deployment.yaml`: image → `docker-internal/<app>:v<base>-unkin2` (sonarr `v5.0.0`, radarr `v6.4.2`, prowlarr `v2.6.2`); `replicas: 3`; `RollingUpdate`. Removed the `apikey-init` initContainer and `PUID`/`PGID`/`TZ`. Runs the binary directly (`command: [/app/<Bin>]`, `args: [-nobrowser, -data=/config, -nosingleinstancecheck]`). Env from the new ConfigMap + `secretKeyRef`: `<App>__Postgres__User`/`__Password` (Secret `<app>-db`) and `<App>__Auth__ApiKey` (existing `<app>-apikey`, key `apitoken`). `/<app>/ping` probes kept. `securityContext` runAsUser/runAsGroup/fsGroup 1000 for CephFS group-write.
- new `configmap.yaml`: non-secret `<App>__…` env (Postgres Host/Port/MainDb, `Log__DbEnabled=false`, `Auth__Method=External`, `Auth__Required=DisabledForLocalAddresses`, `App__InstanceName`, `Server__Port`, `Server__UrlBase=/<app>`, `Update__Mechanism=External`).
- `pvc-config.yaml` → RWX `cephfs-raid5-retain` (shared `/config` for `config.xml`/MediaCover).
- unchanged: `service.yaml`, `gateway.yaml`, `httproute.yaml`, apikey `vaultstaticsecret.yaml` — arrproxy still addresses the Service VIP, so N replicas are transparent.
## Validation
`kustomize build apps/overlays/au-syd1/arrstack` clean; `kubeconform` **64/64 valid**; `yamllint` and the repo pre-commit (incl. no-plain-secrets) pass. Runtime validation (pods Ready, single migrate under advisory lock, scheduled-task/command single-owner, UI via `https://arrstack.unkin.net/<app>/`, 3→1→3 rescale) is post-merge once ArgoCD syncs.
## Decisions / unknowns
- **Fork image UID/fsGroup**: the `-unkin2` images have no `USER` (run as root); pinned `runAsUser`/`runAsGroup`/`fsGroup: 1000` with `fsGroupChangePolicy: OnRootMismatch`. Verify group-write on the shared CephFS `/config` on first deploy.
- **Config storage**: chose RWX `cephfs-raid5-retain` shared `/config` (per spec preference) over per-pod `emptyDir`.
- **Postgres storage/version/resources**: mirrored jellyfin/arrproxy CNPG conventions — `cephrbd-fast-delete` 10Gi, PG17, 3 instances.
Reviewed-on: #383
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
108d1cb213 |
arrstack: set *arr AuthenticationMethod=External (proxy-trusted) (#382)
## Why The *arr UIs reached through **arrproxy** were prompting **"Authentication Required"**. Sonarr v4 (and Radarr/Prowlarr on the same Servarr auth code) refuses remote access when `AuthenticationMethod=None`, so the web UI kept demanding a login even though **arrproxy + oauth2-proxy already authenticate every user at the front door**. That is a double prompt with no purpose. ## Change Extend the existing `apikey-init` container for **sonarr / radarr / prowlarr** to also idempotently enforce, in `/config/config.xml`: - `<AuthenticationMethod>External</AuthenticationMethod>` - `<AuthenticationRequired>Enabled</AuthenticationRequired>` Create-or-replace both elements (same idempotent sed/printf pattern already used for `<ApiKey>`/`<UrlBase>`), without disturbing ApiKey or UrlBase. Pods use `Recreate`, so they roll and re-run the init container on apply. ## Why this is the correct, header-less fix In the Servarr v4 source, `AddExternal()` registers the **identical `NoAuthenticationHandler` as `AddNone()`** — `External` requires **no** username header (no `X-Forwarded-User` / `Remote-User`). It differs from `None` only in that it is **exempt from the None remote-access block**, so the UI stops prompting while remote access is permitted. This matters because arrproxy's `trustBoundary` deliberately **strips all inbound identity headers** and forwards only the real `X-Api-Key` to the upstream *arr — it forwards no username header. Because `External` needs none, that stripping is irrelevant and **no arrproxy change is required**. The API path is unaffected (arrproxy injects the real key; *arr API auth is key-based regardless of AuthenticationMethod). ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` → OK (rendered init carries the auth logic for all 3 apps) - pre-commit (yamllint + all hooks) → Passed Versions in scope: sonarr 4.0.19, radarr 6.3.0, prowlarr 2.5.2 (all share the Servarr v4 auth handler). --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #382 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
c22fbe435a |
arrproxy: allow unverified email in oauth2-proxy (Authentik sets email_verified=false) (#381)
Authentik hardcodes email_verified=false in the id_token it issues for the arrstack application. oauth2-proxy rejects such sessions by default ("email ... isnt verified"), so the /oauth2/callback returns a 500 and login fails.
Setting OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true tells oauth2-proxy to accept the id_token and complete the callback. This is safe: authorization is not derived from email verification here -- access is enforced downstream via the ak_groups claim (X-Forwarded-Groups), so accepting the unverified email only lets the OIDC session establish.
Restart mechanism: the arrproxy-oauth2 Deployment already carries configmap.reloader.stakater.com/auto=true and Reloader is deployed/running, so the ConfigMap change rolls the pods automatically after ArgoCD syncs.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #381
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
5466562bc1 |
arrproxy: use real quay.io oauth2-proxy address (rke2 mirror) (#380)
Repoints the arrproxy `oauth2-proxy` sidecar image to the real upstream address `quay.io/oauth2-proxy/oauth2-proxy:v7.15.3`, replacing the one-off artifactapi copy (`artifactapi.k8s.syd1.au.unkin.net/ghcr/oauth2-proxy/oauth2-proxy:v7.15.3`). ## Why Pull the image from its canonical upstream location instead of a hand-copied artifactapi path. containerd on rke2 transparently mirrors `quay.io` through artifactapi, so the real address resolves without hardcoding the proxy host. ## Change - `apps/base/arrstack/arrproxy/oauth2-proxy-deployment.yaml`: image only -> `quay.io/oauth2-proxy/oauth2-proxy:v7.15.3`. Validated with `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` and pre-commit. ## Dependencies (must be in place before merge) 1. **rke2 `registries.yaml` mirrors quay.io** — already configured in puppet (`hieradata/roles/infra/k8s.yaml`, `rke2::registries` rewrites `quay.io` -> `artifactapi.k8s.syd1.au.unkin.net/quay/$1`). No change needed. 2. **artifactapi quay remote allowlists oauth2-proxy** — terraform-artifactapi PR unkin/terraform-artifactapi#35 adds `^oauth2-proxy/` to the quay remote patterns. **This PR must NOT be merged until #35 is applied**, otherwise the caching proxy rejects `oauth2-proxy/*` and the sidecar fails to pull. --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #380 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
76da30d2f7 |
arrstack: set *arr UrlBase=/<app> for path-based proxy hosting (#378)
## Why arrproxy forwards `arrstack.unkin.net/<app>/...` to each *arr Service **preserving** the `/<app>` prefix. Without a matching `<UrlBase>` the apps 307-redirect to `/`, breaking path-based hosting. Prerequisite for arrproxy path routing (PR #377... arrproxy service account added in #376). ## Change Extend the existing idempotent `apikey-init` init container in sonarr/radarr/prowlarr to also enforce `<UrlBase>/<app></UrlBase>` in `/config/config.xml` (sonarr=/sonarr, radarr=/radarr, prowlarr=/prowlarr): - Set/replace `<UrlBase>` if present, insert after `<Config>` if absent, or write both `<ApiKey>` + `<UrlBase>` when creating a fresh config. `<ApiKey>` handling is unchanged. - Because Servarr moves all routes (including `/ping`) under the URL base, update liveness/readiness probes from `/ping` to `/<app>/ping` so they don't 404 once UrlBase is set. nzbget is not fronted by arrproxy and is left untouched. ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` succeeds - `pre-commit` (yamllint etc.) passes --------- Co-authored-by: BenVincent <benvin@main.unkin.net> Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #378 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
cd8e2c262f |
deploy arrproxy (oauth2-gated *arr proxy + per-user API keys) (#377)
Deploys arrproxy v0.1.0 (task 9 final integration) into the existing `arrstack` app: the external, Authentik-gated front door for sonarr/radarr/prowlarr with per-user API-key brokering. Manifests are wired to match the arrproxy v0.1.0 code (ports, routes, identity headers, keys dir), mirroring the ghp/artifactapi estate patterns.
## Topology (`apps/base/arrstack/arrproxy/`)
- **oauth2-proxy** Deployment + Service `arrproxy` is the single front (OIDC against the Authentik `arrstack` app). It path-routes via `--upstreams` to:
- `arrproxy-ui` for `/` and static SPA assets,
- `arrproxy-api` for `/api/*` (token API) and `/sonarr|/radarr|/prowlarr` (the *arr proxy).
- `--skip-auth-regex=^/[^/]+/api` bypasses oauth **only** for the *arr proxy API (`/sonarr/api...`), while `/api/tokens`, `/api/me` and the UI stay authenticated.
- Identity+groups reach the api via `--pass-user-headers` (`X-Forwarded-{User,Email,Groups}`). The api reads `ARRPROXY_GROUPS_HEADER=X-Forwarded-Groups` — `--set-xauthrequest` only populates auth_request *response* headers and never reaches an `--upstreams` backend, so it is intentionally not used.
- oauth2-proxy trusts `identity.unkin.net` (internal Vault-PKI CA) via a `combine-certs` initContainer (system roots + `vault-ca-cert`), same pattern as artifactapi.
## Data + secrets
- **CNPG** token store `arrproxy-db` (2 instances, cephrgw S3 backups + nightly ScheduledBackup). arrproxy-api does **not** self-migrate, so a wave-1 ArgoCD `Sync` hook Job applies `migrations/0001_init.sql` (mirrored into a ConfigMap) as the CNPG app user. DSN built from the generated `arrproxy-db-app` Secret.
- **VaultStaticSecrets** (arrstack `default` SA / templated role — no terraform-vault change): `arrproxy-pepper` (seeded) and `oauth-credentials`. The three real *arr keys are **reused** from the existing `<app>-apikey` Secrets, projected one file per app into `/etc/arrproxy/keys`.
## Exposure
- **Gateway** `traefik-external`, hostname `arrstack.unkin.net`, cert-manager `vault-issuer` TLS (`arrproxy-gateway-tls`); HTTPRoute → the oauth2-proxy entry Service (+ HTTP→HTTPS redirect).
- Adds the `arrstack.unkin.net` apex A record → external DMZ VIP `198.18.199.0` to the bind-operator `unkin.net` zone (`apps/base/bind-internal/authoritative/records.yaml`), mirroring the ghp/identity apex records.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly; `kubeconform` = 51/51 valid; `pre-commit` passes (no plain Secrets).
## Prerequisites / flags for end-to-end function (NOT in this PR)
- **(a)** Each *arr must run with `UrlBase=/<app>` (path-based hosting) — arrproxy preserves the `/<app>` prefix upstream.
- **(b)** The Authentik `arrstack` OIDC app (terraform-authentik PR #18) must be applied, with redirect URI `https://arrstack.unkin.net/oauth2/callback` registered and the `ak_groups` scope emitted. Confirm the discovery slug matches `OAUTH2_PROXY_OIDC_ISSUER_URL` (`https://identity.unkin.net/application/o/arrstack/`).
- **(c)** Browser XHR to `/<app>/api` is oauth-**bypassed** (skip-auth-regex), so the SPA-driven proxy calls that rely on the session (no per-user token) will not carry identity on that path — expected per the arrproxy trust model; token-based *arr clients are unaffected.
- **NetworkPolicy:** the arrproxy-api Service is not exposed via any HTTPRoute and the api strips forged identity headers in-code, but the README recommends a NetworkPolicy restricting api ingress to oauth2-proxy. The estate currently has **zero** NetworkPolicies, so one was not added here to avoid being the first (and risking probe breakage) before the CNI’s policy/probe behavior is confirmed — flagged as follow-up hardening.
- Please confirm: CNPG size (2 instances / 10Gi), backups on, hostname/VIP, and UI theme.
ARRPROXY_PEPPER was seeded via the agents AppRole at `kv/kubernetes/namespace/arrstack/default/arrproxy-pepper` (key `pepper`).
Reviewed-on: #377
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
d703e67090 |
woodpecker: add arrproxy-ci service account (#376)
Adds the arrproxy-ci Woodpecker ServiceAccount in the woodpecker namespace so unkin/arrproxy CI pipeline pods (which set serviceAccountName: arrproxy-ci) can run. Change: - Add apps/base/woodpecker/serviceaccount_arrproxy_ci.yaml (ServiceAccount arrproxy-ci, namespace woodpecker), mirroring the existing per-repo CI SA pattern (e.g. autobackup-operator-ci, kea-operator-ci). - Wire it into apps/base/woodpecker/kustomization.yaml resources. Validation: kustomize build --enable-helm apps/overlays/au-syd1/woodpecker renders cleanly; pre-commit passes. Unblocks unkin/arrproxy CI. --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #376 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
9450a70652 |
jellyfin: make Valkey HA (#375)
Jellyfin runs 2 replicas that coordinate distributed transcoding through the operator-managed `jellyfin-valkey` ValkeyCluster, but it was single-instance (shards:1, replicas:0) with no failover, so a node/pod loss would drop the shared transcode-lease state. This promotes that store to an HA topology. Changes: - Set replicas:2 (shards:1 unchanged) so the shard group is one primary + two replicas; losing the primary triggers an automatic in-cluster failover to a replica. - Add scheduling.node.spread.shard.mode:Required so the three ValkeyNodes land on distinct hosts (kubernetes.io/hostname anti-affinity) and one host loss removes at most one node. - Set podDisruptionBudget.mode:Cluster so the operator manages a quorum-aware PDB. - Keep persistence omitted (emptyDir): lease state is ephemeral (30s TTLs), replication+failover already provide redundancy and a replaced node re-syncs from the primary, and an operator-managed PVC cannot carry the k8up.io/backup:"false" annotation the namespace k8up Schedule needs to skip in-use RWO volumes. - Bump per-node memory limit to 512Mi (unchanged) / requests unchanged; resources are per ValkeyNode. Connection string is unchanged: still cluster-mode via the single headless service `valkey-jellyfin-valkey:6379`; StackExchange.Redis seeds off it and auto-discovers topology and failovers. kustomize build --enable-helm apps/overlays/au-syd1/jellyfin renders cleanly; pre-commit passes. Reviewed-on: #375 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
9e1ccdb0a0 |
jellyfin: bump image to v0.1.3 (.NET 9 runtime fix) (#374)
## Why jellyfin-ha v0.1.2 crashed on startup: the framework-dependent .NET 9 build could not find its runtime (`Microsoft.NETCore.App 9.0.0` required, `10.0.11` found), because the runtime image was pinned to a .NET 10 base. jellyfin-ha PR #7 realigned the runtime image to `mcr.microsoft.com/dotnet/aspnet:9.0` and v0.1.3 was released. Verified before this bump: `docker run --entrypoint dotnet ...:v0.1.3 --list-runtimes` lists `Microsoft.NETCore.App 9.0.19`, and `/jellyfin/jellyfin --version` prints `Jellyfin.Server 10.11.7.0` with no framework-not-found error. ## Change - Bump jellyfin image tag from v0.1.2 to v0.1.3 in `apps/base/jellyfin/statefulset.yaml`. --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #374 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
bbfa4db652 |
jellyfin: bump image to v0.1.2 (#373)
Bumps the jellyfin StatefulSet image tag to v0.1.2. v0.1.2 is the first jellyfin-ha image successfully pushed to the artifactapi docker-internal registry: pipeline #7 (tag v0.1.2) is green, the docker build+push step exited 0, and the image is pullable (tags/list shows v0.1.2, manifest GET returns 200). v0.1.0 and v0.1.1 tag pipelines failed the CA/TLS check before the plugin-image + buildkit_config fix landed (PR #6). Changes: - Set apps/base/jellyfin/statefulset.yaml jellyfin container image tag from v0.1.0 to v0.1.2 (tag only). Validated with kustomize build --enable-helm apps/overlays/au-syd1/jellyfin and pre-commit. --------- Co-authored-by: unkin-agent <unkin-agent@users.noreply.git.unkin.net> Reviewed-on: #373 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |