13a04a6ebc084f1c00d14d3e6809abb9391246c3
354 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 unkin-agent
|
13a04a6ebc |
Export and scrape metrics from all redis/valkey instances (#398)
## Why No redis/valkey instance managed in this repo exports metrics today — the six caches (artifactapi, authentik, litellm, gitea, netbox, jellyfin) are blind spots in VictoriaMetrics. ## What - Enables the valkey-operator exporter sidecar on the jellyfin ValkeyCluster (`exporter.enabled: true`); overrides the operator's default bare-dockerhub image (`oliver006/redis_exporter:v1.80.0`) with the artifactapi-proxied pin. The operator manages a dedicated `_exporter` ACL user for it. - Adds a `metrics-exporter` sidecar (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/oliver006/redis_exporter:v1.89.0`, :9121, port name `metrics`, req 32Mi/25m lim 64Mi/100m) to the five raw Deployments: artifactapi/authentik/litellm redis and gitea/netbox valkey. gitea/netbox sidecars get the same restrictive securityContext as their valkey container. - Adds the `monitoring.unkin.net/redis-exporter: "true"` opt-in pod label to all five pod templates. - Adds `apps/base/observability/vmpodscrape-redis.yaml` with two VMPodScrapes (any namespace, port `metrics`): `redis-exporters` selecting the opt-in label, and `valkey-operator-clusters` selecting `app.kubernetes.io/managed-by: valkey-operator` + `app.kubernetes.io/component: valkey-node` — the ValkeyCluster CR has no pod-label passthrough (verified against the v0.5.0 CRD/source), so operator pods cannot carry the opt-in label. ## Notes - No instance runs with `requirepass`/auth, so no REDIS_PASSWORD wiring is needed on the raw Deployments; the jellyfin exporter auth is operator-managed. - Validated: `kustomize build` on every touched base + au-syd1 overlay (helm overlays with `--enable-helm`) and `kubeconform -strict -ignore-missing-schemas` (68 valid, 0 invalid). Reviewed-on: #398 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
08a3493097 |
logging: deploy logviewer v0.1.0 behind oauth2-proxy (#404)
## Why The logviewer UI gives browser access to the ClickHouse log store (`logreader` user). It must only be reachable by authorized users, so it sits behind an Authentik-gated oauth2-proxy front at https://logviewer.unkin.net, mirroring the arrstack front-door and mediamover/pdbmux gateway patterns. ## Changes - Adds the logviewer Deployment (1 replica, `docker-internal/logviewer:v0.1.0`) reading ClickHouse at `clickhouse-logs` via the existing `clickhouse-logreader` Secret, `/healthz` probes, nonroot securityContext - Adds an oauth2-proxy (quay.io/oauth2-proxy/oauth2-proxy:v7.15.3) front with issuer `https://identity.unkin.net/application/o/logviewer/`, `ak_groups` claim, allowed group `akP-logviewer-admin`, internal-CA trust via the combine-certs initContainer - Syncs kv `kubernetes/namespace/logging/default/oauth-credentials` into the `logviewer-oauth-credentials` Secret via the existing `default` VaultAuth (kv path already seeded; no terraform-vault change needed) - Adds ClusterIP Services for the app and the oauth2-proxy front door - Adds internal Gateway + HTTPRoutes for `logviewer.unkin.net` (vault-issuer cert, external-dns -> 198.18.200.4, http->https redirect) - Registers `logviewer/` in the logging base kustomization ## Dependency Login requires the Authentik `logviewer` provider/application from terraform-authentik PR #21. This PR is safe to merge first: the VSO secret sync and the deployment come up regardless; OIDC login only works once #21 is applied. Validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (49/49 valid). Reviewed-on: #404 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
4e3c8288e2 |
Bump radarr to v6.4.2-unkin7 (#403)
Ship Commands-index migration (PR #13) which adds partial indexes on Commands for queue claim polling. - Bump radarr image to v6.4.2-unkin7 in apps/base/arrstack/radarr/deployment.yaml Reviewed-on: #403 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
c649cc7ee3 |
Bump prowlarr to v2.6.2-unkin6 (#402)
Ship Commands-index migration. - Update prowlarr image tag to v2.6.2-unkin6 in apps/base/arrstack/prowlarr/deployment.yaml Reviewed-on: #402 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
cd51078d79 |
logging: add read-only clickhouse logreader user + Vault-synced secret (#400)
## Why CLI tools and the logviewer UI need ClickHouse access without write or DDL rights. The only existing app user (`vector`) has `access_management` and write access; this adds a properly scoped read-only login using the same secret-sourcing mechanism. ## Changes - Adds a `logreader` user to the `logs` ClickHouseInstallation: password hash via `secretKeyRef` to the Vault-synced `clickhouse-logreader` Secret (same mechanism as `vector`), networks `::/0`, `readonly` profile, scoped to the `logs` database only - Adds a `readonly` profile (`readonly: 1`, same memory/time limits as default) - Adds a `clickhouse-logreader` VaultStaticSecret mirroring the existing `clickhouse-credentials` object (kv-v2, 5m refresh, VSO-created destination Secret) ## Vault kv Seeded at `kv/kubernetes/namespace/logging/default/clickhouse-logreader` (version 1) with keys: `username`, `password`, `password_sha256_hex`. No secret material lands in git; the CHI only ever sees the sha256 hash via the synced Secret. Reviewed-on: #400 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
b5acb19ac0 |
Bump arrstack image pins: radarr v6.4.2-unkin6, sonarr v5.0.0-unkin6 (#401)
Ship ffprobe exec bit fix (radarr) and Commands-index migration (sonarr). - Update radarr image tag v6.4.2-unkin5 → v6.4.2-unkin6 (ffprobe fix) - Update sonarr image tag v5.0.0-unkin5 → v5.0.0-unkin6 (Commands-index) Reviewed-on: #401 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
ba71bd1a35 |
Scrape traefik gateway metrics into VictoriaMetrics (#397)
## Why Both traefik instances (traefik-internal/traefik-external, chart 40.0.0, image v3.7.0) already expose prometheus metrics on container port `metrics` (9100), but nothing scrapes them — VictoriaMetrics has no traefik series (verified empty). The observability VMAgent runs with selectAllByDefault and consumes VM*Scrape CRs only, so a VMPodScrape is all that is needed. ## Changes - Adds `apps/base/observability/vmpodscrape-traefik.yaml`: scrapes port `metrics` /metrics on `app.kubernetes.io/name: traefik` pods in traefik-system, relabeling `app.kubernetes.io/instance` to a stable `traefik_instance` label to split internal vs external series - Registers the new CR in the observability base kustomization - Enables `metrics.prometheus.addEntryPointsLabels` and `addRoutersLabels` in both au-syd1 traefik values files for per-entrypoint/per-route dashboards (pods restart on rollout — acceptable) Validated with `kustomize build` (observability base + overlay, traefik-system overlay with --enable-helm) and kubeconform (unknown CRDs skipped); rendered deployments carry the new metric flags and the VMPodScrape renders in the overlay. Reviewed-on: #397 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
8fd1d83b1b |
Add Redis/Valkey Grafana dashboard (#396)
## Why
The grafana-operator estate has no dashboard for Redis/Valkey; redis_exporter instances are being added in a parallel PR and need visibility.
## Changes
- Adds `apps/base/grafana/dashboards/redis.yaml`, a GrafanaDashboard CR embedding grafana.com dashboard 763 ("Redis Dashboard for Prometheus Redis Exporter 1.x") as gzipJson like the sibling dashboards, with namespace/instance template variables over `redis_up` for multi-instance use and all datasource refs pinned to the VictoriaMetrics datasource uid (`det2y55dac4jkc`), registered in the grafana kustomization
Reviewed-on: #396
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
19f7afac92 |
Add shared arrstack Valkey and wire arr apps' Redis features (#394)
Activates the arr forks (sonarr/radarr/prowlarr) #14 Redis features — SignalR backplane, cross-replica cache-invalidation bus, and distributed rate limiter — which ship deployed but inert because no Valkey exists in arrstack and nothing is wired to it. Ben wants ONE Valkey shared by all three apps. ## Changes - Adds a single shared `ValkeyCluster` (`arrstack-valkey`) under `apps/base/arrstack/valkey/`, modeled on `jellyfin-valkey`: `shards:1`/`replicas:2` HA (one primary + two replicas, automatic failover), image via artifactapi (`artifactapi.k8s.syd1.au.unkin.net/dockerhub/valkey/valkey:9.0.0`), operator-default passwordless `default` user, node spread across hosts, cluster-aware PDB, ephemeral `/data`. - Registers the `valkey` component in the arrstack base kustomization. - Wires `<App>__Redis__Host` + `<App>__Redis__Port` into the `sonarr-env` / `radarr-env` / `prowlarr-env` ConfigMaps, all pointing at the shared service `valkey-arrstack-valkey.arrstack.svc.cluster.local:6379`. ## Notes - Setting `Host` is the activation switch: the fork's `RedisOptions.IsConfigured` gates purely on a non-empty `Host`, so there is no separate Enabled flag. - Passwordless (jellyfin parity — the operator leaves the `default` user without auth), so no `Password`/`Ssl` is wired and there is no secret to reference. - One cluster is safe for all three: each fork namespaces its keys and pub/sub channels by a per-app prefix (`sonarr:ratelimit:` / `radarr:ratelimit:` / `prowlarr:ratelimit:`), so their state never collides. - App `image:` (-unkin5), the waitfordb initContainer, and S3 buckets are untouched. - Validated: `kubectl kustomize` renders clean for both `apps/base/arrstack` and `apps/overlays/au-syd1/arrstack`. Follow-up: after merge, Valkey must come up and the three apps must roll (pick up the new env) before the #14 features can be validated live. --------- Co-authored-by: Ben Vincent <ben@unkin.net> Reviewed-on: #394 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
b3d202b8fa |
arrstack: add exportarr metrics sidecars to sonarr and radarr (#395)
Sonarr and radarr currently expose no Prometheus metrics, so queue depth, indexer health, and library stats are invisible to the observability stack. This adds the standard exportarr exporter as a sidecar to both Deployments. - Adds an exportarr sidecar (ghcr.io/onedr0p/exportarr v2.3.0, pulled through the artifactapi `ghcr` remote, which already whitelists `^onedr0p/`) to the sonarr and radarr Deployments - Points each exporter at its local replica via env (`URL=http://localhost:8989/sonarr` / `:7878/radarr`, matching the apps' UrlBase) and reuses the existing VSO-synced API-key Secrets (`sonarr-apikey`/`radarr-apikey`, key `apitoken`) via `APIKEY` secretKeyRef — no key in argv - Serves metrics on 9707 (sonarr) / 9708 (radarr) with `/healthz` liveness+readiness probes and small resources (25m/32Mi requests, 100m/128Mi limits) - Adds per-app VMPodScrape objects (picked up by the observability VMAgent, selectAllByDefault); pod-level rather than VMServiceScrape because the app Services don't expose the metrics port Validated with `kustomize build apps/base/arrstack` + kubeconform (0 invalid). Note: with 3 replicas per app, each pod's exporter reports the same shared-Postgres-backed stats, so series are duplicated across the `pod` label; dashboards should aggregate with `max` or filter to one pod. Reviewed-on: #395 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
32611bfd16 |
Deploy mediamover v0.1.0 into arrstack (#393)
Deploys the released mediamover v0.1.0 into the arrstack namespace: a single-replica server (API + UI on :8080) that browses the mediafs source, queues per-file copy/move operations into the media PVCs, and spawns one worker Job per file using its own image — its ServiceAccount/RBAC grants the Job and Pod access that requires. All PVs/PVCs it mounts are already live. - Adds `apps/base/arrstack/mediamover/` with SA, Role/RoleBinding (batch jobs create/get/list/watch/delete; pods get/list/watch; pods/log get/list), Deployment, Service, Gateway, HTTPRoute - Runs `mediamover server` with `--src-root /srv/src --src-pvc mediafs`, `--dst-roots movies=/srv/dst/movies,tv=/srv/dst/tv`, `--dst-pvc movies=media-movies,tv=media-tv`, `--namespace arrstack`, `--image ...mediamover:v0.1.0` - Mounts mediafs RW at /srv/src (move deletes the source) plus media-movies and media-tv destinations; keeps 1 replica with Recreate strategy since the queue is in-memory - Names the Service `mediamover` on 8080 to match the worker callback default `http://mediamover.arrstack.svc:8080` - Exposes the UI at https://mediamover.k8s.syd1.au.unkin.net via a dedicated internal Gateway (pdbmux pattern); the existing arrproxy Gateway is external and hostname-locked to arrstack.unkin.net so it cannot carry this route - Probes hit `GET /api/limit` (the server has no dedicated health endpoint); registers `mediamover` in the arrstack base kustomization Reviewed-on: #393 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
7c49a9f3ad |
Deploy arrstack S3 TLS-trust fix (arrstack images -unkin4 -> -unkin5) (#392)
## Why Ben merged the S3 TLS-trust fix into each Servarr fork's `main` (sonarr#10, radarr#11, prowlarr#10, branch `benvin/s3-tls-trust-fix`). The fix corrects TLS trust for the leaf-only certificate presentation of the Ceph RGW endpoint, so backups and MediaCover uploads to S3 succeed across all replicas instead of failing the TLS handshake. The `-unkin5` images carrying the fix are built and pushed to `artifactapi.k8s.syd1.au.unkin.net/docker-internal/<app>`. This PR rolls them out. ## Changes - Bump `apps/base/arrstack/sonarr/deployment.yaml` app image `v5.0.0-unkin4` -> `v5.0.0-unkin5` - Bump `apps/base/arrstack/radarr/deployment.yaml` app image `v6.4.2-unkin4` -> `v6.4.2-unkin5` - Bump `apps/base/arrstack/prowlarr/deployment.yaml` app image `v2.6.2-unkin4` -> `v2.6.2-unkin5` Only the application container `image:` line changes in each file; the `waitfordb` initContainer and bucket YAML are untouched. Reviewed-on: #392 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
671c075a08 |
Mount mediafs CephFS into arrstack (#391)
Exposes the legacy mediafs CephFS filesystem to the arrstack namespace so the upcoming mediamover tool can migrate media and run bandwidth tests against the old filesystem; the ceph-mediafs client secret is already seeded in Vault. The mover deployment itself lands in a follow-up PR. - Adds a `ceph-mediafs-secret` VaultStaticSecret in csi-cephfs syncing `kv/service/kubernetes/au/syd1/csi/ceph-mediafs-secret` - Adds a 10Ti static RWX PV `arrstack-mediafs` (fsName `mediafs`, rootPath `/`, staged with the mediafs secret) pinned by claimRef - Adds the matching `mediafs` PVC in arrstack (storageClassName "", volumeName-bound, backups excluded) - Registers all three in their base kustomizations Reviewed-on: #391 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
06fe261661 |
arrstack: deploy -unkin4 (Redis SignalR backplane + S3 storage) (#390)
## Why The cluster runs the arrstack forks at `-unkin3`. Ben merged two workstreams into each fork's `main`: - **Redis SignalR backplane + cross-replica cache invalidation** — SignalR messages and cache-invalidation events fan out across replicas via Redis, so any replica serves consistent live UI/state. - **MediaCover -> S3 + backups -> S3** — MediaCover (sonarr/radarr) and backups (all three) use an optional S3 store, so any stateless replica can serve covers and write/restore backups. Fresh `-unkin4` tags were cut on each fork's `main` HEAD, building `docker-internal/<app>:<base>-unkin4` images. This bumps the deployment images so ArgoCD rolls the merged work into the cluster. ## Changes - Bump sonarr app image `v5.0.0-unkin3` -> `v5.0.0-unkin4` - Bump radarr app image `v6.4.2-unkin3` -> `v6.4.2-unkin4` - Bump prowlarr app image `v2.6.2-unkin3` -> `v2.6.2-unkin4` The waitfordb initContainer and bucket resources are unchanged; only the app-container `image:` tags move. Reviewed-on: #390 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
bdf6114794 |
arrstack: S3 buckets + wiring for app backups and sonarr MediaCover (#389)
## Why Servarr backups (periodic config+DB zips) and sonarr MediaCover posters still live on each replica's local `/config` emptyDir: backups are leader-local and lost on reschedule, and sonarr covers render inconsistently across the 3 active-active replicas. #387 moved radarr covers to a shared Ceph RGW bucket; this extends that pattern to backups for all three apps and to sonarr's covers. ## Changes - Add an `arrstack-backups` ObjectStoreUser + Bucket (cephrgw-operator); the operator mints the `arrstack-backups-s3` credential Secret in-namespace (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / S3_ENDPOINT), so no Vault KV seeding is required. Backups get their own bucket (not `arrstack-media`) because they carry secrets (Config.xml API keys, the DB) and warrant separate credentials/lifecycle from poster art; the three apps share it, isolated by a per-app key prefix. - Wire sonarr to the shared `arrstack-media` bucket via `Sonarr__MediaCoverS3__*` (prefix `sonarr`, partitioned from radarr's `radarr` prefix to avoid movie/series id collisions), plus the estate CA mount at `/etc/ssl/vault-ca/ca.crt` for RGW TLS validation. - Wire radarr/sonarr/prowlarr backups via `<App>__BackupS3__*` to `arrstack-backups` (per-app prefix); add the CA mount to sonarr and prowlarr. ## Notes - Credentials come only from the operator-minted Secret via env; no secrets in manifests. - The env keys are consumed by the -unkin3+ images (separate app PRs) and ignored by older images, so this is safe to apply ahead of the image bump. - `kubectl kustomize apps/base/arrstack` builds clean. Reviewed-on: #389 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
eb37beb7e4 |
Deploy arrstack -unkin3 (active-active code) (#388)
Deploy the merged active-active work now on each Servarr fork's main. The cluster still runs the -unkin2 images, so the merged code (K8s process lifecycle, Redis-backed distributed rate limiting, and radarr MediaCover->S3) is not yet deployed. The -unkin3 tag pipelines have built and pushed new images to docker-internal. Changes: - Bump sonarr app image v5.0.0-unkin2 -> v5.0.0-unkin3 - Bump radarr app image v6.4.2-unkin2 -> v6.4.2-unkin3 - Bump prowlarr app image v2.6.2-unkin2 -> v2.6.2-unkin3 Touches only the app container image line in each apps/base/arrstack/<app>/deployment.yaml; the waitfordb initContainer is unchanged. Reviewed-on: #388 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
9cdcdf62bc |
arrstack: provision shared Ceph RGW bucket for media assets (#387)
## Why Servarr MediaCover assets (movie/series posters and fanart) and app backups currently live on each replica's local `/config`, which is an `emptyDir`. A poster the leader downloads is invisible to the other replicas behind the Service, so covers render inconsistently and backups are leader-local. This provisions a shared Ceph RGW (S3) bucket to hold those assets instead. ## Changes - Add an `arrstack-media` `ObjectStoreUser` + `Bucket` (cephrgw-operator), mirroring the CNPG backup pattern. The operator mints the `arrstack-media-s3` credential Secret in-namespace (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `S3_ENDPOINT`), so **no Vault KV seeding is required**. - Wire the radarr Deployment to the bucket via `Radarr__MediaCover__S3__*` env (creds from the operator Secret; estate CA mounted at `/etc/ssl/vault-ca/ca.crt` for RGW TLS validation). ## Notes - The env keys are unknown to the current `-unkin2` image and are ignored by it; they activate once the `-unkin3+` image (radarr MediaCover-S3 support, separate radarr PR) ships. sonarr/prowlarr wiring and backups-to-S3 follow in later PRs. - `kubectl kustomize apps/base/arrstack` builds clean. Reviewed-on: #387 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
5155cee5e7 |
arrstack: use waitfordb initContainer instead of inline psql loop (#386)
## Why The sonarr/radarr/prowlarr deployments gated app startup on their Postgres database being reachable using a hand-written `psql`-in-a-shell loop running the `postgres:17-alpine` image. Now that `unkin/waitfordb` is released (v0.1.0), replace that inline probe with the purpose-built tool so the wait logic lives in one tested place instead of being copy-pasted per workload. `waitfordb` reads the same `PG*` env as a libpq fallback and exits `0` once `SELECT 1` succeeds under those credentials/database, so the retrofit is a drop-in — the password still never lands in argv. ## Changes - Swap the `wait-for-db` initContainer image from `dockerhub/library/postgres:17-alpine` to `docker-internal/waitfordb:v0.1.0` (pinned, not `latest`) in all three arrstack deployments. - Drop the `sh -c` psql loop `command` — `waitfordb` is the image entrypoint. - Keep the existing `PGHOST/PGPORT/PGDATABASE/PGUSER/PGPASSWORD` env, the `wait-for-db` name, and the resource requests/limits unchanged. - Add `WAITFORDB_TIMEOUT=5m` so a stuck database fails the pod fast, and `WAITFORDB_SSLMODE=disable` to match the in-cluster Postgres. Validated with `kubectl kustomize apps/overlays/au-syd1/arrstack` (builds clean; 3 waitfordb refs, no remaining postgres init image). Reviewed-on: #386 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
8446c31414 |
arrstack: fix sync-wave deadlock + config multi-attach (#385)
## Why The active-active arrstack deploy (#383) is stuck in-cluster: the app Deployments (sync-wave 0) can never become Healthy without their Postgres DB, so ArgoCD never advances to create the shared `arrstack-postgres` cluster (wave 1). Separately, `/config` is an immutable RWO cephrbd PVC, so 3 replicas fail with Multi-Attach. ## How - Order DB infra before the apps: `<app>-db` VaultStaticSecrets + backup ObjectStoreUser/Bucket at sync-wave -3, CNPG `Cluster` -2, `Database` CRDs -1; app Deployments stay at default 0. - Switch each app's `/config` to an emptyDir per-pod volume (library DB is in Postgres, config is env-driven; MediaCover regenerates). Old RWO config PVCs orphan. Validated with kustomize build (62 resources). arrproxy/nzbget untouched. Reviewed-on: #385 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
e0d47295f2 |
Deploy arrproxy v0.2.0 machine-mint admin route (#384)
## Why The future Vault engine needs to machine-mint arrproxy tokens without an interactive Authentik session. arrproxy v0.2.0 adds an admin route (`POST /api/admin/...`) protected by its own bearer token; OpenBao running on the VMs calls it through the arrstack ingress. This deploys that route. ## Changes - Add an `arrproxy-admin-token` VaultStaticSecret (mirrors the `arrproxy-pepper` VSO: same `default` VaultAuth, k8s-auth default-SA pattern) syncing `kv/kubernetes/namespace/arrstack/default/arrproxy-admin-token` into the `arrproxy-admin-token` Secret. The token is seeded in Vault KV as the shared source of truth for the future Vault engine. - Set `ARRPROXY_ADMIN_TOKEN` on the arrproxy-api Deployment via `secretKeyRef` (key `token`), and add `arrproxy-admin-token` to the Reloader `secret.reloader.stakater.com/reload` annotation so the pod rolls on rotation. - Skip-auth the `/api/admin/` route in oauth2-proxy: `OAUTH2_PROXY_SKIP_AUTH_REGEX` becomes `^/[^/]+/api,^/api/admin/`. The admin route is intentionally oauth-skipped because it is protected by arrproxy's OWN bearer token so OpenBao can reach it machine-to-machine. `/api/tokens` and `/api/me` are NOT matched and stay oauth-gated. `/api/admin/*` already routes to the arrproxy-api upstream via the existing catch-all `/api/` upstream, so no upstream change is needed. - Bump arrproxy-api and arrproxy-ui images to `v0.2.0` (kept in lockstep). ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` succeeds. - pre-commit passes, including the plain-Secret guard (only the VSO CRD is used; no plain Secret objects added). Reviewed-on: #384 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
187fabb72b |
arrstack: run sonarr/radarr/prowlarr active-active on shared Postgres (#383)
## Why
The arrstack `sonarr`/`radarr`/`prowlarr` apps currently run as single-replica upstream LinuxServer images on SQLite (RWO config PVC, `Recreate` strategy) — no HA, no horizontal scale, and a hard one-pod ceiling because SQLite can't be shared. Our `-unkin2` forks move all app state into PostgreSQL and use Postgres advisory locks for cross-replica coordination (migrations, scheduled tasks, command execution), which makes an **active-active multi-replica** deployment safe.
This is a fresh Postgres backend — **no SQLite data migration**. The current single-replica pods are throwaway baselines; the new pods start on an empty per-app database and the fork's provider migrates the schema on first start. `arrproxy` (its own `arrproxy-db`) and `nzbget` are untouched.
## What
New `apps/base/arrstack/postgres/`:
- `cnpg_cluster.yaml` — CNPG `Cluster` `arrstack-postgres` (3 instances, `cephrbd-fast-delete`), throwaway `app` initdb owner, and 3 `managed.roles` (`sonarr`/`radarr`/`prowlarr`, login non-superuser) each with `passwordSecret: <app>-db`.
- `database-{sonarr,radarr,prowlarr}.yaml` — CNPG `Database` `<app>-main` owned by `<app>`, `databaseReclaimPolicy: retain`.
- `cnpg_backup.yaml` — cephrgw `ObjectStoreUser` + `Bucket cnpg-arrstack` + nightly `ScheduledBackup` (mirrors jellyfin).
- `vaultstaticsecret.yaml` — 3 VSO `VaultStaticSecret`s syncing `<app>-db` (keys `username`/`password`) from `kv/kubernetes/namespace/arrstack/default/<app>-db` (already seeded); these are both the CNPG role `passwordSecret` and the Deployment's Postgres creds.
- sync-waves: `<app>-db` Secrets (0) → Cluster (1) → Databases (2).
- added `postgres` to `apps/base/arrstack/kustomization.yaml`.
Per app (`sonarr`/`radarr`/`prowlarr`):
- `deployment.yaml`: image → `docker-internal/<app>:v<base>-unkin2` (sonarr `v5.0.0`, radarr `v6.4.2`, prowlarr `v2.6.2`); `replicas: 3`; `RollingUpdate`. Removed the `apikey-init` initContainer and `PUID`/`PGID`/`TZ`. Runs the binary directly (`command: [/app/<Bin>]`, `args: [-nobrowser, -data=/config, -nosingleinstancecheck]`). Env from the new ConfigMap + `secretKeyRef`: `<App>__Postgres__User`/`__Password` (Secret `<app>-db`) and `<App>__Auth__ApiKey` (existing `<app>-apikey`, key `apitoken`). `/<app>/ping` probes kept. `securityContext` runAsUser/runAsGroup/fsGroup 1000 for CephFS group-write.
- new `configmap.yaml`: non-secret `<App>__…` env (Postgres Host/Port/MainDb, `Log__DbEnabled=false`, `Auth__Method=External`, `Auth__Required=DisabledForLocalAddresses`, `App__InstanceName`, `Server__Port`, `Server__UrlBase=/<app>`, `Update__Mechanism=External`).
- `pvc-config.yaml` → RWX `cephfs-raid5-retain` (shared `/config` for `config.xml`/MediaCover).
- unchanged: `service.yaml`, `gateway.yaml`, `httproute.yaml`, apikey `vaultstaticsecret.yaml` — arrproxy still addresses the Service VIP, so N replicas are transparent.
## Validation
`kustomize build apps/overlays/au-syd1/arrstack` clean; `kubeconform` **64/64 valid**; `yamllint` and the repo pre-commit (incl. no-plain-secrets) pass. Runtime validation (pods Ready, single migrate under advisory lock, scheduled-task/command single-owner, UI via `https://arrstack.unkin.net/<app>/`, 3→1→3 rescale) is post-merge once ArgoCD syncs.
## Decisions / unknowns
- **Fork image UID/fsGroup**: the `-unkin2` images have no `USER` (run as root); pinned `runAsUser`/`runAsGroup`/`fsGroup: 1000` with `fsGroupChangePolicy: OnRootMismatch`. Verify group-write on the shared CephFS `/config` on first deploy.
- **Config storage**: chose RWX `cephfs-raid5-retain` shared `/config` (per spec preference) over per-pod `emptyDir`.
- **Postgres storage/version/resources**: mirrored jellyfin/arrproxy CNPG conventions — `cephrbd-fast-delete` 10Gi, PG17, 3 instances.
Reviewed-on: #383
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
108d1cb213 |
arrstack: set *arr AuthenticationMethod=External (proxy-trusted) (#382)
## Why The *arr UIs reached through **arrproxy** were prompting **"Authentication Required"**. Sonarr v4 (and Radarr/Prowlarr on the same Servarr auth code) refuses remote access when `AuthenticationMethod=None`, so the web UI kept demanding a login even though **arrproxy + oauth2-proxy already authenticate every user at the front door**. That is a double prompt with no purpose. ## Change Extend the existing `apikey-init` container for **sonarr / radarr / prowlarr** to also idempotently enforce, in `/config/config.xml`: - `<AuthenticationMethod>External</AuthenticationMethod>` - `<AuthenticationRequired>Enabled</AuthenticationRequired>` Create-or-replace both elements (same idempotent sed/printf pattern already used for `<ApiKey>`/`<UrlBase>`), without disturbing ApiKey or UrlBase. Pods use `Recreate`, so they roll and re-run the init container on apply. ## Why this is the correct, header-less fix In the Servarr v4 source, `AddExternal()` registers the **identical `NoAuthenticationHandler` as `AddNone()`** — `External` requires **no** username header (no `X-Forwarded-User` / `Remote-User`). It differs from `None` only in that it is **exempt from the None remote-access block**, so the UI stops prompting while remote access is permitted. This matters because arrproxy's `trustBoundary` deliberately **strips all inbound identity headers** and forwards only the real `X-Api-Key` to the upstream *arr — it forwards no username header. Because `External` needs none, that stripping is irrelevant and **no arrproxy change is required**. The API path is unaffected (arrproxy injects the real key; *arr API auth is key-based regardless of AuthenticationMethod). ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` → OK (rendered init carries the auth logic for all 3 apps) - pre-commit (yamllint + all hooks) → Passed Versions in scope: sonarr 4.0.19, radarr 6.3.0, prowlarr 2.5.2 (all share the Servarr v4 auth handler). --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #382 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
c22fbe435a |
arrproxy: allow unverified email in oauth2-proxy (Authentik sets email_verified=false) (#381)
Authentik hardcodes email_verified=false in the id_token it issues for the arrstack application. oauth2-proxy rejects such sessions by default ("email ... isnt verified"), so the /oauth2/callback returns a 500 and login fails.
Setting OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true tells oauth2-proxy to accept the id_token and complete the callback. This is safe: authorization is not derived from email verification here -- access is enforced downstream via the ak_groups claim (X-Forwarded-Groups), so accepting the unverified email only lets the OIDC session establish.
Restart mechanism: the arrproxy-oauth2 Deployment already carries configmap.reloader.stakater.com/auto=true and Reloader is deployed/running, so the ConfigMap change rolls the pods automatically after ArgoCD syncs.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #381
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
5466562bc1 |
arrproxy: use real quay.io oauth2-proxy address (rke2 mirror) (#380)
Repoints the arrproxy `oauth2-proxy` sidecar image to the real upstream address `quay.io/oauth2-proxy/oauth2-proxy:v7.15.3`, replacing the one-off artifactapi copy (`artifactapi.k8s.syd1.au.unkin.net/ghcr/oauth2-proxy/oauth2-proxy:v7.15.3`). ## Why Pull the image from its canonical upstream location instead of a hand-copied artifactapi path. containerd on rke2 transparently mirrors `quay.io` through artifactapi, so the real address resolves without hardcoding the proxy host. ## Change - `apps/base/arrstack/arrproxy/oauth2-proxy-deployment.yaml`: image only -> `quay.io/oauth2-proxy/oauth2-proxy:v7.15.3`. Validated with `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` and pre-commit. ## Dependencies (must be in place before merge) 1. **rke2 `registries.yaml` mirrors quay.io** — already configured in puppet (`hieradata/roles/infra/k8s.yaml`, `rke2::registries` rewrites `quay.io` -> `artifactapi.k8s.syd1.au.unkin.net/quay/$1`). No change needed. 2. **artifactapi quay remote allowlists oauth2-proxy** — terraform-artifactapi PR unkin/terraform-artifactapi#35 adds `^oauth2-proxy/` to the quay remote patterns. **This PR must NOT be merged until #35 is applied**, otherwise the caching proxy rejects `oauth2-proxy/*` and the sidecar fails to pull. --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #380 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
76da30d2f7 |
arrstack: set *arr UrlBase=/<app> for path-based proxy hosting (#378)
## Why arrproxy forwards `arrstack.unkin.net/<app>/...` to each *arr Service **preserving** the `/<app>` prefix. Without a matching `<UrlBase>` the apps 307-redirect to `/`, breaking path-based hosting. Prerequisite for arrproxy path routing (PR #377... arrproxy service account added in #376). ## Change Extend the existing idempotent `apikey-init` init container in sonarr/radarr/prowlarr to also enforce `<UrlBase>/<app></UrlBase>` in `/config/config.xml` (sonarr=/sonarr, radarr=/radarr, prowlarr=/prowlarr): - Set/replace `<UrlBase>` if present, insert after `<Config>` if absent, or write both `<ApiKey>` + `<UrlBase>` when creating a fresh config. `<ApiKey>` handling is unchanged. - Because Servarr moves all routes (including `/ping`) under the URL base, update liveness/readiness probes from `/ping` to `/<app>/ping` so they don't 404 once UrlBase is set. nzbget is not fronted by arrproxy and is left untouched. ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` succeeds - `pre-commit` (yamllint etc.) passes --------- Co-authored-by: BenVincent <benvin@main.unkin.net> Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #378 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
cd8e2c262f |
deploy arrproxy (oauth2-gated *arr proxy + per-user API keys) (#377)
Deploys arrproxy v0.1.0 (task 9 final integration) into the existing `arrstack` app: the external, Authentik-gated front door for sonarr/radarr/prowlarr with per-user API-key brokering. Manifests are wired to match the arrproxy v0.1.0 code (ports, routes, identity headers, keys dir), mirroring the ghp/artifactapi estate patterns.
## Topology (`apps/base/arrstack/arrproxy/`)
- **oauth2-proxy** Deployment + Service `arrproxy` is the single front (OIDC against the Authentik `arrstack` app). It path-routes via `--upstreams` to:
- `arrproxy-ui` for `/` and static SPA assets,
- `arrproxy-api` for `/api/*` (token API) and `/sonarr|/radarr|/prowlarr` (the *arr proxy).
- `--skip-auth-regex=^/[^/]+/api` bypasses oauth **only** for the *arr proxy API (`/sonarr/api...`), while `/api/tokens`, `/api/me` and the UI stay authenticated.
- Identity+groups reach the api via `--pass-user-headers` (`X-Forwarded-{User,Email,Groups}`). The api reads `ARRPROXY_GROUPS_HEADER=X-Forwarded-Groups` — `--set-xauthrequest` only populates auth_request *response* headers and never reaches an `--upstreams` backend, so it is intentionally not used.
- oauth2-proxy trusts `identity.unkin.net` (internal Vault-PKI CA) via a `combine-certs` initContainer (system roots + `vault-ca-cert`), same pattern as artifactapi.
## Data + secrets
- **CNPG** token store `arrproxy-db` (2 instances, cephrgw S3 backups + nightly ScheduledBackup). arrproxy-api does **not** self-migrate, so a wave-1 ArgoCD `Sync` hook Job applies `migrations/0001_init.sql` (mirrored into a ConfigMap) as the CNPG app user. DSN built from the generated `arrproxy-db-app` Secret.
- **VaultStaticSecrets** (arrstack `default` SA / templated role — no terraform-vault change): `arrproxy-pepper` (seeded) and `oauth-credentials`. The three real *arr keys are **reused** from the existing `<app>-apikey` Secrets, projected one file per app into `/etc/arrproxy/keys`.
## Exposure
- **Gateway** `traefik-external`, hostname `arrstack.unkin.net`, cert-manager `vault-issuer` TLS (`arrproxy-gateway-tls`); HTTPRoute → the oauth2-proxy entry Service (+ HTTP→HTTPS redirect).
- Adds the `arrstack.unkin.net` apex A record → external DMZ VIP `198.18.199.0` to the bind-operator `unkin.net` zone (`apps/base/bind-internal/authoritative/records.yaml`), mirroring the ghp/identity apex records.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly; `kubeconform` = 51/51 valid; `pre-commit` passes (no plain Secrets).
## Prerequisites / flags for end-to-end function (NOT in this PR)
- **(a)** Each *arr must run with `UrlBase=/<app>` (path-based hosting) — arrproxy preserves the `/<app>` prefix upstream.
- **(b)** The Authentik `arrstack` OIDC app (terraform-authentik PR #18) must be applied, with redirect URI `https://arrstack.unkin.net/oauth2/callback` registered and the `ak_groups` scope emitted. Confirm the discovery slug matches `OAUTH2_PROXY_OIDC_ISSUER_URL` (`https://identity.unkin.net/application/o/arrstack/`).
- **(c)** Browser XHR to `/<app>/api` is oauth-**bypassed** (skip-auth-regex), so the SPA-driven proxy calls that rely on the session (no per-user token) will not carry identity on that path — expected per the arrproxy trust model; token-based *arr clients are unaffected.
- **NetworkPolicy:** the arrproxy-api Service is not exposed via any HTTPRoute and the api strips forged identity headers in-code, but the README recommends a NetworkPolicy restricting api ingress to oauth2-proxy. The estate currently has **zero** NetworkPolicies, so one was not added here to avoid being the first (and risking probe breakage) before the CNI’s policy/probe behavior is confirmed — flagged as follow-up hardening.
- Please confirm: CNPG size (2 instances / 10Gi), backups on, hostname/VIP, and UI theme.
ARRPROXY_PEPPER was seeded via the agents AppRole at `kv/kubernetes/namespace/arrstack/default/arrproxy-pepper` (key `pepper`).
Reviewed-on: #377
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
|
||
|
unkin-agent
|
d703e67090 |
woodpecker: add arrproxy-ci service account (#376)
Adds the arrproxy-ci Woodpecker ServiceAccount in the woodpecker namespace so unkin/arrproxy CI pipeline pods (which set serviceAccountName: arrproxy-ci) can run. Change: - Add apps/base/woodpecker/serviceaccount_arrproxy_ci.yaml (ServiceAccount arrproxy-ci, namespace woodpecker), mirroring the existing per-repo CI SA pattern (e.g. autobackup-operator-ci, kea-operator-ci). - Wire it into apps/base/woodpecker/kustomization.yaml resources. Validation: kustomize build --enable-helm apps/overlays/au-syd1/woodpecker renders cleanly; pre-commit passes. Unblocks unkin/arrproxy CI. --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #376 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
9450a70652 |
jellyfin: make Valkey HA (#375)
Jellyfin runs 2 replicas that coordinate distributed transcoding through the operator-managed `jellyfin-valkey` ValkeyCluster, but it was single-instance (shards:1, replicas:0) with no failover, so a node/pod loss would drop the shared transcode-lease state. This promotes that store to an HA topology. Changes: - Set replicas:2 (shards:1 unchanged) so the shard group is one primary + two replicas; losing the primary triggers an automatic in-cluster failover to a replica. - Add scheduling.node.spread.shard.mode:Required so the three ValkeyNodes land on distinct hosts (kubernetes.io/hostname anti-affinity) and one host loss removes at most one node. - Set podDisruptionBudget.mode:Cluster so the operator manages a quorum-aware PDB. - Keep persistence omitted (emptyDir): lease state is ephemeral (30s TTLs), replication+failover already provide redundancy and a replaced node re-syncs from the primary, and an operator-managed PVC cannot carry the k8up.io/backup:"false" annotation the namespace k8up Schedule needs to skip in-use RWO volumes. - Bump per-node memory limit to 512Mi (unchanged) / requests unchanged; resources are per ValkeyNode. Connection string is unchanged: still cluster-mode via the single headless service `valkey-jellyfin-valkey:6379`; StackExchange.Redis seeds off it and auto-discovers topology and failovers. kustomize build --enable-helm apps/overlays/au-syd1/jellyfin renders cleanly; pre-commit passes. Reviewed-on: #375 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
9e1ccdb0a0 |
jellyfin: bump image to v0.1.3 (.NET 9 runtime fix) (#374)
## Why jellyfin-ha v0.1.2 crashed on startup: the framework-dependent .NET 9 build could not find its runtime (`Microsoft.NETCore.App 9.0.0` required, `10.0.11` found), because the runtime image was pinned to a .NET 10 base. jellyfin-ha PR #7 realigned the runtime image to `mcr.microsoft.com/dotnet/aspnet:9.0` and v0.1.3 was released. Verified before this bump: `docker run --entrypoint dotnet ...:v0.1.3 --list-runtimes` lists `Microsoft.NETCore.App 9.0.19`, and `/jellyfin/jellyfin --version` prints `Jellyfin.Server 10.11.7.0` with no framework-not-found error. ## Change - Bump jellyfin image tag from v0.1.2 to v0.1.3 in `apps/base/jellyfin/statefulset.yaml`. --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #374 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
bbfa4db652 |
jellyfin: bump image to v0.1.2 (#373)
Bumps the jellyfin StatefulSet image tag to v0.1.2. v0.1.2 is the first jellyfin-ha image successfully pushed to the artifactapi docker-internal registry: pipeline #7 (tag v0.1.2) is green, the docker build+push step exited 0, and the image is pullable (tags/list shows v0.1.2, manifest GET returns 200). v0.1.0 and v0.1.1 tag pipelines failed the CA/TLS check before the plugin-image + buildkit_config fix landed (PR #6). Changes: - Set apps/base/jellyfin/statefulset.yaml jellyfin container image tag from v0.1.0 to v0.1.2 (tag only). Validated with kustomize build --enable-helm apps/overlays/au-syd1/jellyfin and pre-commit. --------- Co-authored-by: unkin-agent <unkin-agent@users.noreply.git.unkin.net> Reviewed-on: #373 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
a4808f5528 |
ghp: deploy service-token fork image + wire GHP_AUTH_SERVICE_TOKENS (#372)
Why: P4 step 3. Now that the CA-baked buildx plugin lets `docker-internal/ghp` push succeed, deploy the in-house ghp fork (`v0.20.0-unkin4`) which adds static admin service tokens — the mechanism the forthcoming Vault ghp secrets engine uses to authenticate as admin and mint scoped tokens. How: - image: ghcr.io/goodtune/ghp:0.20.0 -> artifactapi.k8s.syd1.au.unkin.net/docker-internal/ghp:v0.20.0-unkin4 (a LOCAL registry image, not the ghcr pull-through that previously ImagePullBackOff-ed). - add env GHP_AUTH_SERVICE_TOKENS from Secret ghp-app key `service_token`, `optional: true` — ghp runs unchanged until the token is seeded. Requires (Ben, runbook): seed `service_token` into kv/kubernetes/namespace/ghp/default/app (same pattern as encryption_key/github-app). The default role's templated policy already grants read; no terraform-vault change. The existing ghp-app reloader annotation rolls the Deployment once the key appears. This same token value later becomes the Vault ghp engine admin_token (P4 steps 5-6). Reviewed-on: #372 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
7556d08d06 |
woodpecker: allow CA-baked docker-buildx plugin as privileged (#371)
Why: pushing in-house images to the internal-CA artifactapi registry fails `x509: certificate signed by unknown authority` with the upstream buildx plugin (and `:latest-insecure` does not fix it — buildkit still verifies the registry cert). The fix is the CA-baked plugin image now published at `artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx` (unkin/plugin-docker-buildx v0.1.0). How: add that image (bare + `:latest`) to `WOODPECKER_PLUGINS_PRIVILEGED` in the au-syd1 overlay so it may run privileged. Existing upstream entries kept so current repos are unaffected until they switch over. Next: ghp `.woodpecker` switches its buildx step to this image, then ghp re-tags and its push succeeds. Reviewed-on: #371 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
31c5ca10e2 |
arrstack/nzbget: fix nonexistent image tag v26.2 -> version-v26.2 (#370)
The nzbget pod in namespace `arrstack` is stuck in `Init:ImagePullBackOff`. ## Why The deployment pins `artifactapi.k8s.syd1.au.unkin.net/dockerhub/linuxserver/nzbget:v26.2` on both the `seed-config` initContainer and the main `nzbget` container. linuxserver does not publish a bare `v26.2` tag - the pull fails with: ``` failed to resolve reference ".../linuxserver/nzbget:v26.2": ... not found Error: ImagePullBackOff ``` Confirmed against the artifactapi dockerhub mirror: `manifests/v26.2` -> HTTP 404. linuxserver version-pins nzbget under the `version-v<MAJOR>.<MINOR>` scheme, and the immutable date tags `26.2.<date>`. `manifests/version-v26.2` -> HTTP 200. ## Change - Set both nzbget images (initContainer + main container) to `version-v26.2`, keeping nzbget on the intended 26.2 line and matching the version-pin convention used by the sibling sonarr (`4.0.19`) / radarr (`6.3.0`) deployments. No other change. The `seed-config` init logic is unaffected - it already mirrors linuxserver's own init (`cp /app/nzbget/share/nzbget/nzbget.conf /config/nzbget.conf`), and since it seeds `/config/nzbget.conf` before the main container, linuxserver's init copy is skipped (no conflict). Instant-move download layout on the media PVCs is preserved. ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly (both images -> `version-v26.2`, no stale `v26.2`). - `pre-commit run --files apps/base/arrstack/nzbget/deployment.yaml` passes (yamllint et al). Scoped to `apps/base/arrstack/nzbget/deployment.yaml` only; the top-level `apps/base/arrstack/kustomization.yaml` was not touched (another agent is editing sonarr/radarr/prowlarr on a separate branch). --------- Co-authored-by: unkin-agent <unkin-agent@users.noreply.git.unkin.net> Reviewed-on: #370 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
d387301d6b |
arrstack: manage sonarr/radarr/prowlarr API keys via Vault (#369)
## Why The sonarr/radarr/prowlarr images self-generate an API key into /config/config.xml on first boot, so the key is unmanaged and differs per volume reset. This makes Vault the source of truth for those keys (override bootstrap, chosen by Ben): the key is minted in Vault and enforced into config.xml before each app starts. ## Changes - Add a `VaultAuth` `default` in the `arrstack` namespace (kubernetes auth, mount `k8s/au/syd1`, role `default`, SA `default`), mirroring jellyfin. - Add a per-app `VaultStaticSecret` that syncs `kv/kubernetes/namespace/arrstack/default/<app>` (key `apitoken`) into the `<app>-apikey` Secret. The `default` k8s role's templated policy already grants read on that path for the `arrstack/default` SA, so no terraform-vault change is needed. - Add an `apikey-init` initContainer to each of the three deployments that reads `API_KEY` from the VSO-created Secret, fails closed on a missing or non-hex value, and writes/updates only the `<ApiKey>` element in `/config/config.xml` (then fixes ownership 1000:1000, mode 600). Image is a pinned busybox via artifactapi to keep this PR atomic (no new image dependency). - Wire the new manifests into the base and per-app kustomizations. ## Notes - Keys already seeded in Vault at `kv/kubernetes/namespace/arrstack/default/<app>`. - nzbget is out of scope: it has no config.xml `<ApiKey>` (uses ControlPassword), a separate follow-up. - Downstream consumers (proxy, terraform) currently read `kv/service/media-apps/<app>`; the authoritative key now lives at the path above. Reconciliation is deferred. --------- Co-authored-by: unkin-agent <unkin-agent@git.unkin.net> Reviewed-on: #369 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
164938f127 |
woodpecker: add plugin-docker-buildx CI ServiceAccount (#368)
Why: the new unkin/plugin-docker-buildx repo's pipelines run their buildah build/push step under `serviceAccountName: plugin-docker-buildx`, which must exist in the woodpecker namespace. How: add serviceaccount_plugin_docker_buildx.yaml + wire it into kustomization.yaml, mirroring the existing per-repo woodpecker SAs. Pairs with unkin/plugin-docker-buildx#1. Reviewed-on: #368 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
fc08f1fecc |
arrstack: sonarr/radarr/prowlarr/nzbget on shared CephFS media (#366)
## Why Stands up the download-automation stack (sonarr, radarr, prowlarr, nzbget) in a new `arrstack` namespace, sharing the same CephFS TV/movies subvolumes that jellyfin serves read-only (jellyfin static-PV rework is on the base branch, PR #237). Downloads and libraries live on one filesystem per media type so the arrs import with atomic hardlink moves. ## Changes - Add `apps/base/arrstack/` namespace with its own static CephFS PVs (`arrstack-media-tv`, `arrstack-media-movies`) + RWX PVCs (`media-tv`, `media-movies`) bound to the same rootPaths jellyfin uses; each namespace gets a unique PV name/volumeHandle pinned by claimRef. - Deploy sonarr (mounts media-tv RW), radarr (media-movies RW), prowlarr (config only), nzbget (both media PVCs RW); single replica, Recreate strategy, LinuxServer images via artifactapi dockerhub remote, PUID/PGID/TZ, probes, fsGroup for CephFS writability. - Per-app config PVCs on `cephrbd-fast-retain` (RWO, 5Gi) for the SQLite state; internal-only Gateway + HTTPRoute per app (traefik-internal, vault-issuer, external-dns `<app>.k8s.syd1.au.unkin.net`), no oauth yet. - nzbget: initContainer seeds `/config/nzbget.conf` (copy image template if absent, append managed path/category block once) and creates the media dirs; category `tv` -> `/media/tv/downloads`, `movies` -> `/media/movies/downloads`. - Wire ArgoCD: add `apps/overlays/*/arrstack` to the media ApplicationSet. (media AppProject already gains the arrstack destination + PersistentVolume whitelist on the base branch.) ## Depends on terraform-artifactapi PR #31 (allowlist `^linuxserver/` on the dockerhub remote) must be merged + applied before these pods can pull. ## Follow-up (terraform) Root folders (/media/tv/series, /media/movies/films), download-client wiring (host `nzbget`, port 6789, categories tv/movies) and API keys are configured later via terraform-arr. --------- Co-authored-by: Ben Vincent <neotheo@gmail.com> Co-authored-by: Ben Vin <neotheo@gmail.com> Reviewed-on: #366 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
40ba1730de |
jellyfin: replace redis Deployment with valkey-operator instance (#367)
## Why Jellyfin depended on a self-managed `redis` Deployment for its distributed transcode-session store. Now that the valkey-operator is deployed cluster-wide, this hands that store to the operator so it is declaratively managed (rolling upgrades, health, future HA) instead of a hand-rolled Deployment. ## Changes - Add `apps/base/jellyfin/valkey.yaml`: a `ValkeyCluster` (`valkey.io/v1alpha1`) named `jellyfin-valkey`, single instance (`shards: 1`, `replicas: 0`), image via the artifactapi dockerhub mirror (`valkey/valkey:9.0.0`), metrics exporter disabled, redis-parity resources. - Omit persistence so `/data` is an `emptyDir`: the transcode-lease state is ephemeral, and the operator-managed PVC cannot carry the `k8up.io/backup: "false"` annotation the namespace-wide k8up Schedule needs to skip an in-use RWO volume (the old redis PVC carried it). - Remove `redis-deployment.yaml`, `redis-service.yaml`, `redis-pvc.yaml` and drop them from `kustomization.yaml`; add `valkey.yaml`. - Repoint `Jellyfin__TranscodeStore__RedisConnectionString` at the operator client Service `valkey-jellyfin-valkey:6379`, preserving `abortConnect=false` and `LeaseDurationSeconds`. ## Auth / TLS None. This operator runs Valkey cluster-mode-enabled with `protected-mode no` and leaves the built-in `default` user passwordless (no `requirepass`), so clients connect unauthenticated over plaintext — parity with the previous open redis. StackExchange.Redis auto-discovers the single node via the headless Service. No `VaultStaticSecret` / plain Secret is introduced. ## Validation - `kustomize build --enable-helm apps/overlays/au-syd1/jellyfin` renders cleanly: `ValkeyCluster` present, no redis Deployment/Service/PVC, connection string updated. - pre-commit passes (incl. the plain-secrets guard). - kubeconform: 10 valid / 0 invalid; the `ValkeyCluster` is skipped (CRD schema not vendored). ## Follow-up HA (task 6) is intentionally out of scope: it will raise `replicas` (and/or `shards`) and add anti-affinity/topology spread. Enabling auth (ACL `default` user + password via `VaultStaticSecret`) and/or TLS can also be layered on later. Reviewed-on: #367 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
 unkinben
|
6c8c0dd9e4 |
Deploy jellyfin-ha as a true-HA StatefulSet under a new media project (#237)
## Why Deploy the jellyfin-ha fork as a genuine high-availability service rather than a single replica, so its two headline capabilities can actually be exercised: the Redis-coordinated distributed transcoding (a surviving pod resumes the HLS segments of a pod that dies mid-stream) and the experimental PostgreSQL main database (which is what lets more than one replica share the same library). It lands in its own `jellyfin` namespace under a new `media` ArgoCD project. ## How **Workload — StatefulSet, 2 replicas.** The Deployment becomes a StatefulSet so each pod has a stable name. The fork's Redis transcode lease keys ownership on `JELLYFIN_INSTANCE_ID`, which is set from the downward-API pod name, giving each replica a unique, stable lease identity. Soft `podAntiAffinity` spreads the two pods across nodes and a `PodDisruptionBudget` keeps `minAvailable: 1` through drains and rollouts. **Main database — CloudNativePG.** A CNPG trio in-namespace mirrors the litellm pattern: a 3-instance `Cluster`, a PgBouncer `Pooler`, and Ceph RGW (barman) S3 backups to a dedicated `cnpg-jellyfin` bucket owned by a `cephrgw` `ObjectStoreUser`. An init container writes `/config/config/database.xml` selecting the fork's `Jellyfin-PostgreSQL` provider, and the connection string is composed from the CNPG-generated `jellyfin-postgres-app` secret (username / password / dbname) pointed at the pooler service — the password is never rendered into the manifest. Library-item metadata therefore moves off SQLite; metadata images, plugins, subtitles and config XML stay on `/config`. **Storage.** - `/config` is now a shared `ReadWriteMany` cephfs PVC (raid5, retain) so every replica reads/writes the same metadata and config. - `/config/transcodes` — the fork's real transcode temp path — is a shared RWX PVC (raid5, delete). This is the load-bearing fix: takeover reads the dead pod's in-flight `.ts`/`.m3u8` segments off shared storage, so per-pod scratch would silently break it. - `/cache` is per-pod via a `volumeClaimTemplate` (RWO). - The media library stays a fresh, empty RWX PVC mounted read-only; populating it is out of scope. **Hardware transcoding.** The container requests the `gpu.intel.com/i915` Intel device-plugin resource (which pins the pod to a GPU-labelled node and injects the DRI render node — no `/dev/dri` hostPath or privileged container) plus the render/video supplemental groups. VA-API hardware acceleration is now on by default: the `inject-config` init container seeds `/config/config/encoding.xml` with `HardwareAccelerationType` `vaapi`, `EnableHardwareEncoding`, the injected render node (`/dev/dri/renderD128`) and h264/hevc hardware decode, so transcodes use the iGPU on first boot with no manual admin-UI step. Both seed files (`database.xml`, `encoding.xml`) are written only when absent, so later admin changes persisted to the shared RWX `/config` are never clobbered on restart. **Networking.** The Gateway/HTTPRoute (traefik-internal, `jellyfin.k8s.syd1.au.unkin.net`) is unchanged; the Service gains `sessionAffinity: ClientIP` to keep a client pinned to one replica and reduce transcode-session churn. **Redis.** The in-namespace single-replica Redis stays as the transcode lease store. ## Follow-up UDP auto-discovery is disabled, but scheduled library scans still run on every replica (redundant scans). Single-scanner leader election is a planned follow-up pending a fork feature, tracked separately. --------- Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-authored-by: Ben Vincent <neotheo@gmail.com> Co-authored-by: Ben Vin <neotheo@gmail.com> Reviewed-on: #237 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net> |
||
|
unkin-agent
|
b759f4beea |
Install valkey-operator (#365)
Makes the valkey-operator available in the cluster so a later change can replace the self-managed redis Deployment with a Valkey CR. This PR only installs the operator; it does not create any Valkey CR or touch existing apps. Changes: - Add `apps/base/valkey-operator-system/` (namespace `valkey-operator-system`, mirroring the cnpg-system pattern). - Add `apps/overlays/au-syd1/valkey-operator-system/` which inflates the `valkey-operator` 0.5.0 Helm chart from the artifactapi virtual helm repo with `includeCRDs: true` (the chart ships CRDs in the Helm-native `crds/` dir, which `helm template` skips otherwise). - Route the operator image through artifactapi: `artifactapi.k8s.syd1.au.unkin.net/ghcr/valkey-io/valkey-operator:v0.5.0`. - Register `apps/overlays/*/valkey-operator-system` in the platform ApplicationSet. No AppProject change needed: the `*-system` destination and the virtual-helm sourceRepo already cover it (same as cnpg-system/cephrgw-system). Depends on unkin/terraform-artifactapi#29 (adds the valkey helm remote + virtual member and allowlists the operator image). That PR must merge and apply before this one syncs, or the chart pull / image pull will 404/403. Validated with `kustomize build --enable-helm` against the upstream chart (identical version/values): renders the Namespace, 2 CRDs, operator Deployment (image = the artifactapi ghcr path), RBAC, SA, and metrics Service, all in `valkey-operator-system`. Reviewed-on: #365 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
8503a4b7a4 |
woodpecker: add ghp CI ServiceAccount (#364)
## Why The ghp fork (`unkin/ghp`) now has `.woodpecker` pipelines whose steps run under `serviceAccountName: ghp`. Without this ServiceAccount in the `woodpecker` namespace, pods fail to schedule (`error looking up service account woodpecker/ghp: serviceaccount "ghp" not found`), which fails the required `ci/woodpecker/pr/build` status. ## Change - Add `apps/base/woodpecker/serviceaccount_ghp.yaml` (ServiceAccount `ghp` in namespace `woodpecker`), mirroring the existing per-project SAs (e.g. `jellyfin-ha-src`). - Register it in `apps/base/woodpecker/kustomization.yaml`. Unblocks the ghp CI onboarding for unkin/ghp#1. Reviewed-on: #364 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
dc02b3eebd |
Deploy k8up backup operator (platform, k8up-system) (#363)
The k8up CRDs already exist in the cluster (owned by a separate `k8up-crds` Helm release) but the k8up operator itself was never deployed, so no k8up backup/restore custom resources are reconciled. This installs the operator following the cnpg-system pattern (Helm inflated via kustomize `helmCharts`) in the `platform` ArgoCD project, namespace `k8up-system`. - Add `apps/base/k8up-system/` (Namespace + kustomization). - Add `apps/overlays/au-syd1/k8up-system/` inflating the **k8up chart `4.10.0`** (pinned) from the artifactapi virtual-helm proxy: `replicaCount: 2` (leader election with hot-standby follower), operator resource requests/limits (20m/128Mi req, 500m/256Mi lim), and ServiceMonitor/PrometheusRule disabled (require Prometheus Operator CRDs). - Add a `PodDisruptionBudget` (`minAvailable: 1`) in the overlay, since the k8up chart exposes no native PDB value. - Register `apps/overlays/*/k8up-system` in the platform ApplicationSet. **CRD management:** The k8up chart ships its CRDs via the Helm `crds/` directory (there is no `installCRDs`/`skipCrds`/`enableCRDCreation` value). The kustomize helm inflator does not emit `crds/`, so `kustomize build` renders only the operator (Deployment, RBAC, SA, Service) and no CRDs. ArgoCD therefore does not fight the existing `k8up-crds` Helm release over CRD ownership. **platform.yaml (AppProject):** no change needed. The virtual-helm sourceRepo, the `*-system` destination, and the ClusterRole/ClusterRoleBinding/CRD/webhook clusterResourceWhitelist entries are already present and cover everything this app renders. **Validation:** `kustomize build --enable-helm apps/overlays/au-syd1/k8up-system` succeeds (Deployment replicas 2, PDB, Namespace, RBAC; 0 CRDs). `make kubeconform` reports k8up-system 14/14 valid; the only failure is the pre-existing, unrelated `cattle-system` rancher chart kubeVersion issue. pre-commit passes. Reviewed-on: #363 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
20438520ab |
woodpecker: add autobackup-operator CI ServiceAccount (#362)
The autobackup-operator Woodpecker pipeline needs a dedicated ServiceAccount to run its CI jobs under, matching the existing per-project operator CI SAs in the woodpecker namespace (e.g. kea-operator-ci). - Adds ServiceAccount autobackup-operator-ci in the woodpecker namespace - Registers serviceaccount_autobackup_operator_ci.yaml in the woodpecker kustomization.yaml resources list Reviewed-on: #362 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
7a1e8351a7 |
ghp: serve plain HTTP behind the gateway (fix redirect loop) (#361)
https://ghp.unkin.net/ 308-loops onto itself. The traefik gateway terminates TLS and forwards cleartext to the ghp Service port 80 -> container :8080, but :8080 was `GHP_SERVER_HTTP_LISTEN` — ghp's http->https **308 redirect** listener. So ghp bounced every request back to https, the gateway re-forwarded it to :8080, and it looped forever. ## Root cause (confirmed against ghp source) `internal/server/server.go` `Run()` picks the serving mode: ```go hasTLS := s.cfg.Server.HTTPSListen != "" || (systemd socket + certs) if hasTLS { return s.serveTLS(...) } // app on HTTPSListen (TLS); HTTPListen = 308 redirect; Listen IGNORED return s.servePlain(...) // app (full handler: mgmt UI + API) on Listen, cleartext ``` It is **strictly either/or**. `serveTLS` serves the app on `HTTPSListen` and gives `HTTPListen` only `httpsRedirectHandler()` (`redirect.go`: `http.StatusPermanentRedirect` = 308). `servePlain` serves the same full handler on `Listen` in cleartext (`createListener()` uses `cfg.Server.Listen`). The mgmt UI is the same `handler` in both modes, so it IS served on the plain `Listen` port. Behind a TLS-terminating gateway that forwards cleartext to :8080, ghp therefore has to run in **plain mode**. Keeping `GHP_SERVER_HTTPS_LISTEN` would keep `hasTLS` true, leave `GHP_SERVER_LISTEN` ignored, and nothing would serve cleartext on :8080. ## Change - **configmap**: drop `GHP_SERVER_HTTPS_LISTEN` and `GHP_SERVER_HTTP_LISTEN`; set `GHP_SERVER_LISTEN: ":8080"` so :8080 SERVES the app; add `GHP_SERVER_TRUST_PROXY_HEADERS: "true"` so ghp trusts the gateway's `X-Forwarded-*`/`Forwarded` for scheme/host (`GHP_SERVER_BASE_URL` already set). - **deployment + vmservicescrape**: the metrics server only wraps TLS when `hasTLS` is true (`Run()` gates `loadTLSConfig` on `hasTLS`); in plain mode it is cleartext, so the `/metrics` liveness/readiness probes and the VMServiceScrape switch from HTTPS/https to HTTP/http. Service, HTTPRoute and Gateway are unchanged. configmap+deployment carry the stakater reloader annotation, so pods roll on the change. ## Deviation from the brief The brief said to keep `GHP_SERVER_HTTPS_LISTEN: ":8443"`. Source shows that is incompatible with serving cleartext on :8080 (the two modes are mutually exclusive), so this drops it. The unused `GHP_TLS_CERT_FILE`/`KEY_FILE`, the `tls` volume, and containerPort 8443 are left in place (harmless) for an easy revert to TLS mode. The alternative — gateway -> Service 443 -> :8443 with a BackendTLSPolicy — is the bigger change flagged in the brief and is NOT taken here. Validated: `kustomize build apps/overlays/au-syd1/ghp` clean, kubeconform 0 invalid/0 errors, pre-commit clean. Not applied. --------- Co-authored-by: unkin-agent <agent@unkin.net> Reviewed-on: #361 Co-authored-by: Unkin Agent <unkin-agent@unkin.net> Co-committed-by: Unkin Agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
5e2eecda6b |
ghp: serve at ghp.unkin.net (gateway + cert + httproute + DNS) (#360)
## Why Make `ghp.unkin.net` the primary URL that the ghp app responds to. Until now ghp has only been reachable at the internal admin name `ghp.k8s.syd1.au.unkin.net`; this promotes the apex `unkin.net` name to canonical (mirroring the gitea apex pattern) while keeping the k8s name as the admin/internal route. ## How - **`apps/base/ghp/gateway.yaml`** — add `ghp.unkin.net` as the PRIMARY http/https listeners (`http-primary`/`https-primary`) and keep `ghp.k8s.syd1.au.unkin.net` as the admin route (`http-admin`/`https-admin`), mirroring gitea. Add the cert-manager gateway annotations (`cluster-issuer: vault-issuer`, `common-name: ghp.unkin.net`, `private-key-size: "4096"`) so cert-manager mints a NEW `ghp-gateway-tls` Secret with CN `ghp.unkin.net` + a SAN per TLS listener hostname (so it also covers the k8s host). Both https listeners' `certificateRefs` point at `ghp-gateway-tls`. This is a separate Secret from `ghp-tls` (ghp's own GitHub-impersonation cert), so cert-manager never fights over it. external-dns annotation stays scoped to the k8s host only — the apex is not published by external-dns. - **`apps/base/ghp/httproute.yaml`** — add `ghp.unkin.net` to `hostnames`; update `parentRefs` to attach to the renamed `http-primary`/`http-admin`/`https-primary`/`https-admin` sections. - **`apps/base/ghp/configmap.yaml`** — set `GHP_SERVER_MANAGEMENT_HOST: ghp.unkin.net` and `GHP_SERVER_BASE_URL: https://ghp.unkin.net`, so ghp serves its mgmt UI on that Host and OAuth callbacks use the new base. - **`apps/base/bind-internal/authoritative/records.yaml`** — add a bind-operator `DNSRecord` `ghp-dns-internal` for `ghp.unkin.net` -> **A 198.18.200.4** (traefik-internal gateway VIP), mirroring the active `identity-dns-internal` record in the same file. ## ⚠️ NOTE — GitHub App OAuth callback must be updated on github.com The GitHub App's OAuth callback URL must be changed on github.com to **`https://ghp.unkin.net/auth/github/callback`**. It was set to the k8s host in the runbook; OAuth login will break until this is updated. ## Note on DNS authority The bind-operator apex `unkin.net` zone comments warn that some apex names may still be served by the puppet DNS master (records from PuppetDB), with the k8s bind cluster not yet confirmed as the live authority for `unkin.net`. This DNSRecord mirrors the already-active `identity`/`lb1` apex records in the same file, so it is consistent with them — but if the k8s bind cluster is not the live authority for `unkin.net`, `ghp.unkin.net` will not resolve from this Record alone and the puppet master record must be added instead. ## Validation `kustomize build` of the ghp and bind-internal overlays render clean; kubeconform (strict) valid; yamllint + pre-commit clean. Not applied. Reviewed-on: #360 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
0130d538f5 |
ghp: use direct ghcr.io image + set GHP_ADMINS (#359)
## Why The `ghp` app is deployed but its pods are stuck 0/1 Ready (and were ImagePullBackOff), for three separate reasons this PR fixes: - **ImagePullBackOff:** kubelet anonymous pulls fail on the artifactapi ghcr pull-through because ghcr.io's per-scope token auth is not proxied. The direct public image pulls anonymously, so switch to it. - **Pods never Ready:** ghp serves its metrics endpoint over **HTTPS** (TLS is configured globally), but the liveness/readiness probes used the default HTTP scheme, so the kubelet probe got an HTTPS-server error and the pods never went Ready. - **Scrape failure:** the VMServiceScrape hits that same HTTPS endpoint and needs a matching scheme/TLS config, or VM scraping of ghp fails. - The `GHP_ADMINS` value was still a placeholder. ## How - `deployment.yaml`: image -> `ghcr.io/goodtune/ghp:0.20.0`; liveness + readiness probe `scheme: HTTP` -> `HTTPS` (kubelet does not verify the probe cert). - `migrate-job.yaml`: image -> `ghcr.io/goodtune/ghp:0.20.0` (shared image). - `vmservicescrape.yaml`: endpoint `scheme: https` + `tlsConfig.insecureSkipVerify: true` (internal-CA cert; pod-IP target not in SANs). - `configmap.yaml`: `GHP_ADMINS` -> `neoloc`. Validated: `kustomize build apps/overlays/au-syd1/ghp` renders clean, kubeconform + pre-commit pass. Not applied. ## Follow-up (not fixed here) The artifactapi ghcr pull-through does not proxy ghcr.io's per-scope token auth for anonymous kubelet pulls — worth closing that gap so estate images can go back through artifactapi. Reviewed-on: #359 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
fa1f3e7756 |
add ghp (GitHub proxy) deployment (#357)
## Why Deploy `goodtune/ghp` (a GitHub proxy) so estate agents have a controlled GitHub access path. Mirrors the artifactapi app structure and the estate's templated `default` Vault convention. ## How (this PR) - Namespace `ghp`. Runs as the namespace's auto-created `default` ServiceAccount (no custom SA). - CNPG `Cluster` (named `postgres`, db/owner `ghp`) + dedicated `cnpg-ghp` Ceph RGW backup bucket + nightly ScheduledBackup. - Deployment `command: ["/ghp","serve"]`; `GHP_DATABASE_DSN` assembled from the CNPG-generated `postgres-app` Secret via `$(VAR)` expansion pointing at `postgres-rw.ghp.svc`. - **Sync-phase migrate hook Job** `ghp-migrate` (`/ghp migrate`) at sync-wave 1 so serve replicas never race migrations. - VSO wiring on the templated `default` role: `VaultAuth` role/SA `default`; `VaultStaticSecret`s source `kv/kubernetes/namespace/ghp/default/github-app` → Secret `ghp-github-app` and `kv/kubernetes/namespace/ghp/default/app` → Secret `ghp-app`. `private_key` mounted as a file; `encryption_key` → `GHP_ENCRYPTION_KEY`. - cert-manager `Certificate` `ghp-tls` off ClusterIssuer `vault-issuer`, SANs github.com/api.github.com/codeload.github.com/*.githubcopilot.com/ghp.k8s.syd1.au.unkin.net; mounted for `GHP_TLS_CERT_FILE`/`GHP_TLS_KEY_FILE`. - Service (443->8443, 80->8080, metrics 9136), PDB, VMServiceScrape, internal Traefik Gateway + HTTPRoute for `ghp.k8s.syd1.au.unkin.net`. - Sync-wave ordering: DB + VSO/cert (wave 0, wait Healthy) -> migrate Job (wave 1) -> serve Deployment + Service/Gateway/HTTPRoute (wave 2). Writable `/tmp` emptyDir on both pods (root FS is read-only). - Registered in the `platform` ApplicationSet + AppProject. - Validation: `kustomize build` clean, kubeconform 17/17 valid (0 skipped), yamllint + pre-commit clean. ## NOTE — prerequisites before ghp goes Healthy - **`GHP_ADMINS` placeholder**: `apps/base/ghp/configmap.yaml` has `GHP_ADMINS: "REPLACE_ME_ben_github_username"` — set it to Ben's GitHub username. - Ben's GitHub-App runbook + kv population at the templated paths `kv/kubernetes/namespace/ghp/default/github-app` (app_id/client_id/client_secret/private_key) and `kv/kubernetes/namespace/ghp/default/app` (encryption_key). - tf-artifactapi #25: the `ghcr` remote pattern for pulling `ghcr/goodtune/ghp`. - **No terraform-vault change needed** — the estate's templated `default` policy already grants read on `kv/kubernetes/namespace/<ns>/default/*` (the bespoke ghp role PR terraform-vault#120 is closed). - DNS overrides (pointing github.com etc. at ghp) are intentionally **out of scope** — separate deferred bind-operator PR. Assumptions flagged: metrics env var `GHP_METRICS_LISTEN`; two Secrets (`ghp-github-app` + `ghp-app`) rather than one merged Secret; DSN assumes the CNPG-generated password is URL-safe. Reviewed-on: #357 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
4c8a773cd8 |
artifactapi: bump image to v3.11.1 (#358)
## Why Ships the least_conn selection optimization released in artifactapi v3.11.1 (image built, pushed to the registry, pipeline green). ## How - Bump api pin: `artifactapi:v3.11.0` → `:v3.11.1` (apps/base/artifactapi/api-deployment.yaml) - Bump ui pin: `artifactapi-ui:v3.11.0` → `:v3.11.1` (apps/base/artifactapi/ui-deployment.yaml) `kustomize build apps/overlays/au-syd1/artifactapi` clean; both images resolve to v3.11.1. Reviewed-on: #358 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
d37e87f35b |
artifactapi: bump image to v3.11.0 (#356)
## Why Ship artifactapi v3.11.0: mirrorlist support, least-connections balancing, and cache-flush. Both `artifactapi` and `artifactapi-ui` images are built and present in the Gitea registry (release pipeline green). ## How Bump the api and ui image pins v3.10.1 -> v3.11.0: - `apps/base/artifactapi/api-deployment.yaml`: `git.unkin.net/unkin/artifactapi:v3.10.1` -> `:v3.11.0` - `apps/base/artifactapi/ui-deployment.yaml`: `git.unkin.net/unkin/artifactapi-ui:v3.10.1` -> `:v3.11.0` Overlay `apps/overlays/au-syd1/artifactapi` renders clean via `kustomize build`. Reviewed-on: #356 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
496b99f222 |
artifactapi: bump image to v3.10.1 (#355)
Ships the #117 local-repodata determinism fix (rpm + deb/apk) now that the v3.10.1 images are built and present in the Gitea registry. Simple one-release patch bump from the deployed v3.10.0 pin. How: - Bump artifactapi api image pin v3.10.0 -> v3.10.1 - Bump artifactapi-ui image pin v3.10.0 -> v3.10.1 Reviewed-on: #355 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |
||
|
unkin-agent
|
51143c56b7 |
artifactapi: bump images to v3.10.0 (apk repo support + UI) (#354)
## Why artifactapi v3.10.0 ships Alpine/apk support — an apk local repository, a `github_alpine` remote, and metadata-only handling — plus the new UI how-to sections. Rolling the deployed images forward from v3.9.1 delivers apk repo support and the UI docs to the cluster. ## What - Bump `apps/base/artifactapi/api-deployment.yaml` API image `git.unkin.net/unkin/artifactapi` from `v3.9.1` to `v3.10.0`. - Bump `apps/base/artifactapi/ui-deployment.yaml` UI image `git.unkin.net/unkin/artifactapi-ui` from `v3.9.1` to `v3.10.0`. The v3.10.0 images are published/building from the `v3.10.0` tag. Only the two image tags change; `kustomize build`, yamllint, and pre-commit all pass. Reviewed-on: #354 Co-authored-by: unkin-agent <unkin-agent@unkin.net> Co-committed-by: unkin-agent <unkin-agent@unkin.net> |