#456 omitted the combine-certs initContainer and
OAUTH2_PROXY_PROVIDER_CA_FILES on the grounds that identity.unkin.net
serves a publicly trusted Let's Encrypt cert and so needs no internal CA.
That reasoning holds for the browser redirect but not for oauth2-proxy's
own back-channel calls: every other oauth2-proxy in the estate needs the
internal bundle, including repospawner, which uses the same public
identity.unkin.net issuer hostname.
artifactapi is the only one of six without it (arrproxy, logviewer,
mediamark, repospawner and watchstate all have it).
This is NOT the current outage. The UI is 503 because the Authentik
application slug artifactapi does not exist -- terraform-authentik's
push/apply on main (4e16401) failed, so discovery 404s and oauth2-proxy
never starts. This change removes the next blocker, which would surface
as an x509 failure once that apply succeeds.
- Add the combine-certs initContainer, byte-identical to repospawner's.
- Mount the combined bundle and set OAUTH2_PROXY_PROVIDER_CA_FILES.
- Reload the Deployment when vault-ca-cert rotates.
Trust-only and strictly additive: it appends the internal CA to the
system roots, so it is harmless if the back channel turns out to reach a
publicly trusted endpoint after all.
The artifactapi web UI is open to anyone who can reach the host. Front it with Authentik SSO gated on akP-artifactapi-admin, while leaving the package-manager surfaces (/api/v1, /api/v2, /v2 docker registry, /terraform, /.well-known) untouched — dnf, containerd mirrors, buildah, terraform and CI publish steps cannot do a browser flow.
- Add the oauth2-proxy ConfigMap, Deployment, Service and VMPodScrape.
- Add the oauth-credentials VaultStaticSecret.
- Point the api-route /ui rule at oauth2-proxy and add a /oauth2 rule; the catch-all / rule still goes straight to the api Service on both listeners.
Requires terraform-authentik #34 applied and the Vault kv seed first.
Reviewed-on: #456
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
v0.2.0 ships the reports/events union, aggregate summing across backends, config-file support, and the removal of the primary/prefer ranking.
## How
- Pin the pdbmux Deployment image to `v0.2.0`.
- Leave `pdbmux-env` unchanged: `PDBMUX_LISTEN`, `PDBMUX_BACKENDS`, `PDBMUX_MERGE` are the only keys v0.2.0 reads from this ConfigMap, and `PDBMUX_PRIMARY`/`PDBMUX_PREFER` are already gone (#452).
Reviewed-on: #455
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: pick up v3.11.2, which moves DB migrations onto golib/pg with no behavior change.
- Bump the artifactapi API and UI image pins from v3.11.1 to v3.11.2.
Reviewed-on: #453
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
pdbmux#9 drops primary/prefer and makes configured backend order the only tie-break, so the current `old`-first list would silently reverse which PuppetDB wins.
## How
- Order `PDBMUX_BACKENDS` with `new=http://puppetdb.puppet.svc.cluster.local:8080` first and `old=http://puppetdbapi.service.consul:8080` second, URLs unchanged.
- Drop `PDBMUX_PRIMARY` and `PDBMUX_PREFER`; both already resolve to `new`, on the deployed v0.1.0 image (estate defaults) and on pdbmux main (first-backend fallback), so the rendered behaviour is unchanged today.
- Refresh the configmap and deployment comments to describe order-based precedence.
Merge this before pdbmux#9.
Reviewed-on: #452
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: the new vimpack repo's woodpecker pipelines set `serviceAccountName: vimpack-ci`, which does not exist in the woodpecker namespace yet.
- Add bare `vimpack-ci` ServiceAccount in `apps/base/woodpecker/` and register it in the kustomization.
Reviewed-on: #454
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The `ghcr.io/openvoxproject/openvoxserver` image reads `OPENVOXSERVER_JAVA_ARGS` (`/etc/default/puppetserver`: `JAVA_ARGS=$OPENVOXSERVER_JAVA_ARGS`) and ships no `PUPPETSERVER_*` alias, so our heap/JMX flags have been inert since the fork switch — live masters and compilers run the image default `-Xms1024m -Xmx1024m` with no JMX.
## How
- Rename `PUPPETSERVER_JAVA_ARGS` to `OPENVOXSERVER_JAVA_ARGS` in `puppetserver-master-config`, `puppetserver-compiler-config` and `puppetserver-init-config`.
- Rename the same var on the `perms-and-dirs` init container in `deployment_puppetserver-compiler.yaml`.
- Flag values are unchanged (`-Xms1024m -Xmx3072m` plus the JMX flags); heap tuning is a separate call.
Reviewed-on: #451
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: pick up encapi v0.1.2, which moves DB migrations onto golib/pg with no behavior change (release pipeline green).
- Bump the encapi deployment container image from v0.1.1 to v0.1.2
Reviewed-on: #450
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
An unlimited environment timeout serves stale catalogs after code changes.
## How
- Set `OPENVOXSERVER_ENVIRONMENT_TIMEOUT: "0"` in the puppetserver master and compiler ConfigMaps.
Reviewed-on: #449
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The new `golib` repo's Woodpecker pipeline needs a dedicated ServiceAccount to run its CI steps under.
## How
- Add bare ServiceAccount `golib-ci` in the `woodpecker` namespace and wire it into the base kustomization.
Reviewed-on: #447
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: repospawner v0.1.1 stops spawned job pods from automounting the API token.
- Bump the deployment image and the matching REPOSPAWNER_IMAGE env value to v0.1.1
Reviewed-on: #446
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
repospawner v0.1.0 is built and its Vault kubernetes auth role is applied, but nothing deploys it. It turns a "I want a new repository" request into a terraform-git pull request, follows that PR to merge, and optionally activates the repo in Woodpecker, so the review gate stays where it is instead of moving into an agent's hands.
## How
- Add `apps/base/repospawner/`: namespace, ServiceAccount `repospawner`, `default` VaultAuth for VSO, and a namespaced Role/RoleBinding granting jobs create/get/list/watch/delete plus pods and pods/log reads (mirrors mediamover).
- Deployment pinned to `artifactapi.k8s.syd1.au.unkin.net/docker-internal/repospawner:v0.1.0`, one replica with the `Recreate` strategy because request state is in memory and rebuilt from Job labels; the same image reference is passed down as `REPOSPAWNER_IMAGE` so the spawned Jobs stay in step.
- Mount a projected `audience: vault` service account token at `/var/run/secrets/vault` — the app logs into Vault natively rather than through VSO — and the `repospawner-woodpecker` Secret at `/etc/repospawner/woodpecker`, optional so the server still starts and refuses `woodpecker: true` with 503 when it is absent.
- Two VaultStaticSecrets: `oauth-credentials` from `kv/kubernetes/namespace/repospawner/default/oauth-credentials` and `repospawner-woodpecker` (key `token`) from `.../default/woodpecker`, with reloader annotations on both consumers.
- oauth2-proxy front door on the watchstate/mediamark pattern, gated on `akP-repospawner-admin` via the `ak_groups` claim and re-checked by the app from `X-Forwarded-Groups`; public `repospawner.unkin.net` on the reflected wildcard and internal `repospawner.k8s.syd1.au.unkin.net` on `vault-issuer`, both routed to the oauth2 Service.
- Register the overlay in the platform ApplicationSet and AppProject, and append `repospawner` to the wildcard Certificate's two reflector namespace lists.
Depends on the terraform-authentik `repospawner` client being applied and `kv/kubernetes/namespace/repospawner/default/oauth-credentials` + `.../woodpecker` being seeded.
Reviewed-on: #445
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
arrproxy v0.6.0 applies its own schema at startup under a Postgres advisory lock and holds `/readyz` until the schema is current, so every replica is safe to roll without an external gate. The wave-1 psql `arrproxy-migrate` Job and its SQL ConfigMap now only re-run idempotent statements the app already owns — dead weight, a second source of truth for the schema, and a standing drift trap whenever the app's embedded migrations move ahead of the manifests.
## How
- Bump `arrproxy-api` and `arrproxy-ui` to `v0.6.0`.
- Delete `migrate-job.yaml` and `migrations-configmap.yaml` and drop both from the arrproxy kustomization.
- Keep the wave-0/wave-2 split: wave 2 still orders the api behind the wave-0 CNPG Cluster and VSO-synced Secrets, which is independent of the migrate Job; the stale "serve only after the wave-1 migrate Job" comment is corrected.
- Rendered diff vs `main` is exactly the two image bumps plus the `arrproxy-migrate` Job and `arrproxy-migrations` ConfigMap disappearing; `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` and pre-commit both clean.
Reviewed-on: #444
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
mediamark needs a home in the cluster: it marks/organises media on the shared mediastore tree and drives the adult-tier sonarr/radarr through arrproxy's hash routes. This adds the full app to the `media` project, mirroring the watchstate two-host oauth2-proxy pattern and the arrstack static-CephFS + projected-API-keys patterns.
## How
- Adds `apps/base/mediamark/`: namespace, VaultAuth (`k8s/au/syd1`, role `default`), three VaultStaticSecrets, static mediastore PV/PVC, the app Deployment, oauth2-proxy ConfigMap/Deployment, two Services, and internal + external Gateway/HTTPRoute pairs.
- Binds a dedicated static PV (`mediamark-mediastore`, own `volumeHandle`, `claimRef`-pinned) to the same CephFS mediastore subvolume arrstack/fafflix/cheeztv use, RWX 10Ti Retain, mounted at `/media`.
- Runs the app as 1000:1000 (deliberately not 65532) so it owns files on the shared media tree and hardlink/rename moves stay valid; read-only root filesystem, all caps dropped, no service-account token, `/livez` + `/readyz` probes.
- Projects the sonarr/radarr API keys as one file per app under `/etc/mediamark/keys`, mirroring arrproxy's keys projection, with reloader annotations on both secrets.
- Fronts both `mediamark.unkin.net` (traefik-external, reflected Let's Encrypt wildcard, no cert-manager annotations) and `mediamark.k8s.syd1.au.unkin.net` (traefik-internal, vault-issuer) with a single oauth2-proxy using a relative `/oauth2/callback` redirect; gated on `akP-mediamark-user` and passing identity to the app as `X-Forwarded-Groups` via `PASS_USER_HEADERS`.
- Appends `mediamark` to the `wildcard-unkin-net` Certificate's two reflector namespace lists, and registers the app in `argocd/applicationsets/media.yaml` + `argocd/projects/media.yaml` with a passthrough `apps/overlays/au-syd1/mediamark` overlay.
## Prerequisite seeds (Ben, before pods go Ready)
These KV paths must exist under `kv/kubernetes/namespace/mediamark/default/` — the `mediamark/default` templated policy already grants read, so no terraform-vault change is needed:
- `oauth-credentials` — needs `client_id` and `cookie_secret` added alongside the existing `client_secret` (Authentik mediamark provider; both absolute callback URIs registered there).
- `sonarr` — key `apitoken`.
- `radarr` — key `apitoken`.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/mediamark` (18 resources) and `.../cert-manager` both build.
- kubeconform clean on both touched overlays.
- `pre-commit run --all-files` passes.
Reviewed-on: #441
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
arrproxy v0.5.0 ships per-token HTTP method scoping for machine tokens, so a minted token can be limited to e.g. `GET` only. Zero-downtime: the mint-API field is additive and existing tokens get an empty methods list, which means unrestricted — they behave exactly as before.
## How
- Bump `arrproxy-api` and `arrproxy-ui` pins from v0.4.0 to v0.5.0.
- Mirror repo migrations `0002_tier_tokens.sql` and `0003_token_methods.sql` into the migrations ConfigMap. It had drifted at 0001 while v0.4.0 already queried `tier`/`read_only`, and every v0.5.0 token query selects `methods` — without this the new API errors on every token read.
- Have the wave-1 migrate Job apply all three files in order. Every statement is `IF NOT EXISTS`, so a resync over an already-migrated database is a no-op.
Rendered `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` diff vs main is exactly the two image tags, the two added ConfigMap keys, and the two added `-f` args.
Reviewed-on: #443
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The new repospawner repo needs a Woodpecker CI pipeline, and every pipeline step must run under its own ServiceAccount in the woodpecker namespace.
- Add `apps/base/woodpecker/serviceaccount_repospawner_ci.yaml` (bare SA `repospawner-ci`, ns woodpecker), mirroring the existing mediamark-ci SA.
- Register it in the woodpecker kustomization resources list.
Reviewed-on: #442
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit (logarchiver/logviewer are untouched).
Changes:
- Point the ClickHouseInstallation and the clickhouse-schema job at `docker.io/clickhouse/clickhouse-server:24.8`.
- Point the logviewer oauth2-proxy cert-combine init container at `docker.io/library/alpine:3`.
- Point the NATS bootstrap job at `docker.io/natsio/nats-box:0.18.0`.
- Point the NATS chart values at `docker.io/library/nats` and `docker.io/natsio/nats-server-config-reloader`.
- Point all three Vector values files (agent, aggregator, vm-ingest) at `docker.io/timberio/vector`.
- Drop the now-wrong "pulled through the artifactapi dockerhub remote" comments in the NATS and vector-agent values.
Tags/digests unchanged and the `repository`/`tag` split is preserved. `kustomize build --enable-helm apps/overlays/au-syd1/logging` differs from main only in those nine image strings.
Extra found, not changed here: `.woodpecker/vector-test.yaml` still pins its CI step image to `artifactapi.k8s.syd1.au.unkin.net/dockerhub/timberio/vector:0.57.0-debian`. That is a Woodpecker step image rather than a namespace manifest, so it is left out to keep this PR to the logging namespace — say the word and I will fix it separately.
Reviewed-on: #433
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Set the valkey-operator chart `image.registry` to `ghcr.io`.
The `registry`/`repository`/`tag` split is untouched otherwise, so the rendered image is `ghcr.io/valkey-io/valkey-operator:v0.5.0`. `kustomize build --enable-helm apps/overlays/au-syd1/valkey-operator-system` differs from main only in that image string. No other proxied image refs in the file (the `helmCharts[].repo` entry in kustomization.yaml is a Helm chart repo, not a container registry, so it stays on artifactapi).
Reviewed-on: #437
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
`identity.unkin.net` moved from an internal `unkin.net` CA-issued cert to the LetsEncrypt `*.unkin.net` wildcard. `auth.generic_oauth`'s `tls_client_ca` pointed Grafana at the internal root only, so the OAuth handshake to the LE-issued cert now fails. Grafana's image trust store already contains the public roots.
## Changes
- Remove `tls_client_ca: /etc/grafana/vault-ca/ca.crt` (and its stale comment) from `auth.generic_oauth`.
- Remove the now-unused `vault-ca-cert` volume and volumeMount from the Grafana pod spec — nothing else in the pod referenced it (the CNPG `endpointCA` reference to `vault-ca-cert` for `s3.ceph.unkin.net` is a separate resource and stays).
- Leave the auth/token/api URLs, scopes and `role_attribute_path` untouched.
Reviewed-on: #440
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
`identity.unkin.net` moved from an internal `unkin.net` CA-issued cert to the LetsEncrypt `*.unkin.net` wildcard. The `oidc.config` block pinned the internal root as the *only* trust anchor, so argocd-server now rejects OIDC discovery with `x509: certificate signed by unknown authority` and SSO login is broken. The stock image trust store already carries the public roots.
## Changes
- Remove the `rootCA:` block from `argocd-cm`'s `oidc.config` for the `https://identity.unkin.net/application/o/argocd/` issuer.
- Replace the now-false internal-CA rationale comment with a one-liner noting the LE-issued cert needs no pin.
- Leave issuer, clientID, clientSecret ref, `requestedScopes` (incl. `ak_groups`) and `requestedIDTokenClaims` untouched.
Reviewed-on: #439
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The mediamark Woodpecker docker step needs a dedicated ServiceAccount so it can push to the trusted in-cluster registry, mirroring the existing arrproxy-ci setup.
- Add bare ServiceAccount `mediamark-ci` in namespace `woodpecker` and register it in the woodpecker base kustomization.
Reviewed-on: #438
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the Altinity chart crdHook at `docker.io/bitnami/kubectl`.
- Point the operator at `docker.io/altinity/clickhouse-operator`.
- Point the metrics exporter at `docker.io/altinity/metrics-exporter`.
- Reword the header comment that claimed all images are pulled through the artifactapi dockerhub remote.
Only the `repository` keys change; the chart still supplies the tags (0.27.2 / latest), so rendered tags are identical. `kustomize build --enable-helm apps/overlays/au-syd1/clickhouse-system` differs from main only in those three image strings. No other proxied refs in the file.
Reviewed-on: #436
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the watchstate oauth2-proxy cert-combine init container at `docker.io/library/alpine:3`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/watchstate` differs from main only in that image string. No extra proxied refs in the file (the oauth2-proxy image itself is already canonical `quay.io/...`).
Reviewed-on: #435
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the netbox valkey metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/netbox` differs from main only in that image string. No extra proxied refs in the file (the `valkey/valkey:8-alpine` container is already a bare upstream name).
Reviewed-on: #434
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the litellm redis metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/litellm` differs from main only in that image string. No extra proxied refs in the file (the `redis:7-alpine` container is already a bare upstream name).
Reviewed-on: #432
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the gitea valkey metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/gitea` differs from main only in that image string. No extra proxied refs in the file (the `valkey/valkey:8-alpine` container is already a bare upstream name).
Reviewed-on: #431
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the authentik redis metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/authentik` differs from main only in that image string. No extra proxied refs in the file (the `redis:7-alpine` container is already a bare upstream name).
Reviewed-on: #430
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the artifactapi redis metrics sidecar at `docker.io/oliver006/redis_exporter:v1.89.0`.
Tag unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/artifactapi` differs from main only in that image string. No extra proxied refs in the file (the `redis:7-alpine` container is already a bare upstream name).
Reviewed-on: #429
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The media estate is currently split across two 1Ti CephFS subvolumes (`media-tv`
and `media-movies`), so a download that lands in one and imports to the other is
a cross-filesystem copy, not a hardlink move. A single 10Ti `mediastore`
subvolume (already created on `cephfs_csi_ssd_ec_4_1`) collapses that: one
filesystem holding `/{fafflix,cheeztv}/{tvseries,movies}` plus
`/nzbget/downloads/complete`, so every arr import is an atomic same-fs hardlink.
This PR only provisions the volume and seeds its tree. No existing PV, PVC or
workload is touched, and nothing mounts the new claims yet — the cutover of the
arrs and the two jellyfins is a separate change.
## How
- Add static PVs `arrstack-mediastore`, `fafflix-mediastore` and
`cheeztv-mediastore`, all pointing at the same rootPath
(`/volumes/csi_ssd_ec_4_1/mediastore/a0152dac-…`) with unique names and
volumeHandles pinned by `claimRef` — the established pattern for the shared
media subvolumes.
- Add the matching RWX 10Ti PVCs (`mediastore` in arrstack,
`fafflix-mediastore`, `cheeztv-mediastore`), annotated
`k8up.io/backup: "false"` and statically bound via `volumeName` +
`storageClassName: ""`.
- Add `mediastore-bootstrap`, a one-shot ArgoCD Sync-hook Job
(`hook-delete-policy: BeforeHookCreation`, no sync-wave needed) that mounts
the arrstack claim and `mkdir -p`s the directory tree as uid/gid 1000 —
the uid the arrstack media pods run as. Idempotent, so it self-heals on
every sync.
- Wire the new manifests into the arrstack, fafflix and cheeztv bases.
## Validation
- `kustomize build` clean on `apps/overlays/au-syd1/{arrstack,fafflix,cheeztv}`
- `kubeconform` clean on all three overlays (91 / 29 / 32 resources valid)
- pre-commit (yamllint, check-yaml, no-plain-secrets) passed
Reviewed-on: #428
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
rke2's `registries.yaml` already rewrites upstream image names to the artifactapi mirror, so manifests must carry canonical upstream names. Only in-house `artifactapi.k8s.syd1.au.unkin.net/docker-internal/...` images stay explicit.
Changes:
- Point the arrproxy migrate job at `docker.io/library/postgres:18-alpine`.
- Point the arrproxy oauth2-proxy cert-combine init container at `docker.io/library/alpine:3`.
- Point the arrstack ValkeyCluster at `docker.io/valkey/valkey:9.0.0`.
Tags are unchanged. `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` differs from main only in those three image strings. No extra proxied refs found in these files (the oauth2-proxy image itself is already canonical `quay.io/...`).
Reviewed-on: #427
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Summary
- Adds SSO-Auth.xml, LDAP-Auth.xml, and branding.xml plugin config (rendered by an initContainer into /config) to fafflix and cheeztv
- Adds a per-namespace oauth-credentials VaultStaticSecret to each instance
- Adds a "Sign in with SSO" login link via jellyfin branding config
## Why
Phase-1 jellyfin auth: browser SSO via the 9p4 SSO plugin (Authentik OIDC, ak_groups claim) and phone/TV app-passwords via the LDAP plugin against the Authentik LDAP outpost. Config is declarative — restarts overwrite UI edits. Activates when the plugin-baked jellyfin-ha image lands (separate PR).
Reviewed-on: #424
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Summary
- Bumps the jellyfin-ha image to v0.2.0 on fafflix and cheeztv
## Why
v0.2.0 bakes the LDAP and SSO auth plugins into the image; with the plugin config PR this enables browser SSO and app-password logins.
Reviewed-on: #426
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Summary
- Bumps kea-operator image pin to v0.1.5
## Why
v0.1.4 fixed HA peer URLs but the HA hook then bound its dedicated listener to the ClusterIP and crashed; v0.1.5 disables the dedicated listener so HA flows via the ctrl-agent.
Reviewed-on: #425
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
why: production kea-dhcp4 is in a permanent crashloop — Kea's HA hook rejects DNS hostnames in peer URLs and v0.1.3 renders headless-DNS peer URLs; v0.1.4 ships the merged fix (per-peer ClusterIP Services + IP-literal URLs).
- bump kea-operator image pin to v0.1.4
Reviewed-on: #423
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Jellyfin authenticates users against Authentik over LDAP (app-password binds). The `authentik-ldap` Service, Gateway, and TLSRoute already exist in the authentik base, but nothing backed the Service — no LDAP outpost was ever deployed. This adds the missing Deployment so in-cluster clients (Jellyfin) can bind.
Deployed as a separate Deployment inside the existing `authentik` namespace rather than a new app/namespace: the authentik overlay is already wired into the `platform` ApplicationSet and the LDAP Service/Gateway/TLSRoute already live here, so this needs no new AppProject/ApplicationSet wiring and keeps the outpost next to the core it serves.
## How
- `ldap-outpost-deployment.yaml`: 2-replica (stateless) Deployment, image `ghcr.io/goauthentik/ldap:2026.5.3` (canonical upstream name; matches the deployed authentik chart version; containerd mirrors route via artifactapi). Pod labels match the existing `authentik-ldap` Service selector.
- `AUTHENTIK_HOST=https://identity.k8s.syd1.au.unkin.net`, `AUTHENTIK_INSECURE=false`.
- Trusts the internal CA via the established combine-certs initContainer pattern (concatenates the base roots with the reflected `vault-ca-cert`) + `SSL_CERT_FILE`.
- `AUTHENTIK_TOKEN` sourced from secret `authentik-ldap-outpost-token`.
- Resources set; TCP probes on the LDAP port; reloader annotation for token + CA rotation.
- `ldap-outpost-vaultstaticsecret.yaml`: VSS (reuses the namespace `default` VaultAuth) materialising the token from KV `kubernetes/namespace/authentik/default/outpost-token` (key `token`).
- `ldap-service.yaml`: adds the plaintext `ldap` port 3389 alongside the existing `ldaps` 6636 (ClusterIP only, no public exposure — external reach is via the pre-existing internal Gateway/TLSRoute).
- `ldap-outpost-vmpodscrape.yaml`: VMPodScrape for the outpost metrics endpoint (:9300).
- No sync-wave annotations.
## Dependency — token seed (blocking)
The outpost pods CrashLoop until the token exists. After merge, the Authentik LDAP outpost token must be seeded into Vault KV at `kv/kubernetes/namespace/authentik/default/outpost-token` with key `token`. This path is provided by the corresponding terraform-authentik PR; the value must match the outpost token authentik issues.
## Validation
`kustomize build --enable-helm apps/overlays/au-syd1/authentik` + repo kubeconform: 38 resources, Valid: 38, Invalid: 0, Errors: 0.
Reviewed-on: #422
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
WatchState currently only has an internal front door (watchstate.k8s.syd1.au.unkin.net, PR #419, merged). This adds a public front door at **watchstate.unkin.net** via the external (DMZ) Traefik so the admin UI is reachable off-cluster, still fully gated by oauth2-proxy/Authentik. The internal gateway and routes are untouched; oauth2-proxy now fronts BOTH hostnames.
Stacked base note: the intended base (benvin/watchstate-deploy, PR #419) merged into main before this PR opened, so this targets `main`.
## How
- **DNS**: `watchstate-dns-internal` DNSRecord in `apps/base/bind-internal/authoritative/records.yaml` (unkin.net zone, bind-internal) -> `A 198.18.199.0`, the traefik-external VIP. Mirrors the `arrstack-dns-internal` precedent (external front door via bind, not external-dns). Kept in the central bind-internal location, so no AppProject destination widening is needed.
- **Gateway**: new `watchstate-external` Gateway (`gatewayClassName: traefik-external`), http + https listeners on hostname `watchstate.unkin.net`. TLS terminated with the Let's Encrypt `*.unkin.net` wildcard secret `wildcard-unkin-net-tls` — so **no cert-manager and no external-dns annotations**.
- **Routes**: `watchstate-external-http-redirect` (http->https 301) and `watchstate-external` (https -> `watchstate-oauth2:80`), mirroring the arrproxy/cheeztv external patterns.
- **oauth2-proxy dual-host**: `OAUTH2_PROXY_REDIRECT_URL` changed from the pinned cluster callback to the **relative** `/oauth2/callback`. With `OAUTH2_PROXY_REVERSE_PROXY=true` (already set), oauth2-proxy derives scheme+host per request from `X-Forwarded-Proto/Host` (verified in oauth2-proxy v7.15.3 `getOAuthRedirectURI`: host-less redirectURL falls through to request-derived), so the callback works on both `watchstate.unkin.net` and `watchstate.k8s.syd1.au.unkin.net`. `COOKIE_DOMAINS` and `WHITELIST_DOMAINS` now list both hosts (one cookie per host; a single parent-domain cookie can't cleanly span unkin.net vs k8s.syd1.au.unkin.net).
- Dropped the no-op `argocd.argoproj.io/sync-wave: "0"` annotation on the `default` VaultAuth. New resources intentionally carry no sync-wave annotations.
## Validation
- `kustomize build --enable-helm` on `apps/overlays/au-syd1/watchstate` and `.../bind-internal`: OK.
- kubeconform (repo CI config, k8s 1.33.7): watchstate overlay 15/15 valid (2 Gateways, 4 HTTPRoutes); bind-internal overlay 67/67 valid (incl. `watchstate-dns-internal`).
## Dependencies
- **Reflector allow-list (argocd-apps PR #418)**: reflects `wildcard-unkin-net-tls` into the `watchstate` namespace. This PR references that secret as if present.
- **Authentik (terraform-authentik, separate PR)**: register both `https://watchstate.unkin.net/oauth2/callback` and `https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback` as redirect URIs on the watchstate provider.
Reviewed-on: #420
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
#417 stood up the central Let's Encrypt `*.unkin.net` wildcard (Certificate `wildcard-unkin-net` in `cert-manager`, secret `wildcard-unkin-net-tls`, emberstack-reflected). This wires the remaining single-label public hosts onto that publicly-trusted leaf instead of the internal Vault-PKI CA.
A `*.unkin.net` wildcard matches **single-label** subdomains only, so cluster hostnames under `*.k8s.syd1.au.unkin.net` are deliberately left on `vault-issuer`.
> Note: PR #417 already merged to `main`, so this is no longer stacked — it targets `main` directly.
## How
- **cert-manager**: reflect `wildcard-unkin-net-tls` into `arrstack`, `authentik`, `gitea` (appended to both reflector namespace lists → `cheeztv,arrstack,authentik,gitea`).
- **arrstack**: repoint the public `arrproxy` gateway listener (`arrstack.unkin.net`) `certificateRefs` → `wildcard-unkin-net-tls`; drop its `cert-manager.io/*` vault-issuer annotations.
- **authentik**: repoint the `identity.unkin.net` https listener → `wildcard-unkin-net-tls`. The `identity.k8s.syd1.au.unkin.net` listeners keep `authentik-tls` (vault-issuer), so the gateway `common-name` is retargeted to that cluster host. `authentik-ldap` gateway untouched.
- **gitea**: secret **reflected only**; `git.unkin.net` cutover deferred (no manifest change), same deferred pattern as fafflix.
## Validation
- `kustomize build` OK for cert-manager / arrstack / authentik / gitea overlays.
- `kubeconform` all valid (0 invalid, 0 errors) across the four overlays.
- Verified no `*.k8s.syd1.au.unkin.net` listener lost its vault-issuer wiring.
Reviewed-on: #418
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Deploy WatchState (arabcoders/watchstate), the Jellyfin/Plex/Emby watch-state
sync tool, as an internal admin tool. It gets an admin-only web UI/API gated the
same way as logviewer: an oauth2-proxy front backed by Authentik OIDC.
## What
- New `watchstate` namespace + media-project app; base at `apps/base/watchstate`,
overlay at `apps/overlays/au-syd1/watchstate`.
- Image `ghcr.io/arabcoders/watchstate:v1.10.3` (current release; canonical
upstream name, containerd mirrors route ghcr via artifactapi).
- `replicas: 1`, `strategy: Recreate`, single `5Gi` `cephrbd-fast-delete` RWO PVC
at `/config` — sqlite + the in-container cron/redis are single-writer.
- `runAsUser/runAsGroup/fsGroup: 1000` (image's rootless user); liveness/readiness
`GET /v1/api/system/healthcheck` on 8080 (route confirmed in upstream
`src/API/System/HealthCheck.php`, no auth guard).
- oauth2-proxy (mirrors logviewer 1:1) fronts every path. Authentik issuer
`identity.k8s.syd1.au.unkin.net`, redirect
`https://watchstate.k8s.syd1.au.unkin.net/oauth2/callback`. Authorization is
enforced Authentik-side (akR-global-admin only), so no oauth2-proxy group
allowlist is configured.
- Internal-only Gateway (`traefik-internal`) for
`watchstate.k8s.syd1.au.unkin.net`, `vault-issuer` TLS leaf, external-dns to
`198.18.200.4`. HTTP -> HTTPS redirect.
- `VaultStaticSecret` pulls OIDC creds from
`kv/kubernetes/namespace/watchstate/default/oauth-credentials`; `vault-ca-cert`
auto-reflects into the namespace.
- Registered in the media `ApplicationSet` + `AppProject`.
## Scrape decision
No `VMPodScrape`: WatchState exposes no Prometheus/`/metrics` endpoint.
## Follow-ups
- **Seed check:** the VaultStaticSecret expects `client_id`, `client_secret`, and
`cookie_secret` keys at the kv path. Only `client_secret` was confirmed seeded;
`client_id` and a generated `cookie_secret` must also be present or the
oauth2-proxy pod will not start.
- **Webhook ingestion:** the Jellyfin webhook endpoint (`/v1/api/webhook`)
currently sits behind oauth2-proxy like everything else. When sync is wired up,
the jellyfins pushing webhooks will need an auth-bypass or an apikey route for
that path.
Reviewed-on: #419
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
why: cheeztv.unkin.net needs a publicly-trusted cert; DNS-01 via the delegated acme.unkin.net zone is now working (staging wildcard verified end-to-end in ~70s). A single centrally-managed wildcard avoids per-host CNAMEs and LE duplicate-order rate limits.
- add Certificate wildcard-unkin-net (cert-manager ns, ClusterIssuer letsencrypt prod, secret wildcard-unkin-net-tls) with emberstack reflector secretTemplate annotations
- reflect the secret into the cheeztv namespace (append future consumers to the two reflector namespace lists)
- point the public cheeztv gateway listener at wildcard-unkin-net-tls and drop its vault-issuer annotations (cheeztv-internal stays on vault-issuer)
Reviewed-on: #417
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The old bare `sonarr`/`radarr` Services are gone, but arrproxy still ran the v0.3.x built-in default topology pointing at non-existent `sonarr`/`radarr` upstreams and mounted dead `sonarr-apikey`/`radarr-apikey` Secrets. This cuts the front door over to arrproxy v0.4.0 tier routing so the adult (fafflix, `/3aa168`) and kids (cheeztv, `/3df803`) *arr instances are reachable again, with per-tier group authorization.
## Changes
- Bump `arrproxy-api` and `arrproxy-ui` images to `v0.4.0`.
- Add `arrproxy-tiers` ConfigMap (`ARRPROXY_TIERS_FILE`): `fafflix` + `cheeztv` tiers, each with sonarr/radarr upstreams, route hash, key subdir and group grants. Adults (`akP-media-fafflix`) reach both tiers all methods; kids (`akP-media-cheeztv`) reach only cheeztv, GET/HEAD. `legacyRoutes:false` retires the unprefixed routes.
- Because arrproxy strips `/<hash>` and proxies `/<app>/...`, each upstream URL carries the `/<hash>` path segment so the joined upstream path (`/3aa168/sonarr/...`) matches the *arr's own UrlBase.
- Mount the tiers file and rework the projected keys volume to the real per-instance `{sonarr,radarr}-{adult,kids}-apikey` Secrets under `<keyDir>/<app>` (`adult/*`, `kids/*`); refresh reloader annotations.
- oauth2-proxy: route `/3aa168` and `/3df803` to arrproxy-api (drop dead `/sonarr` `/radarr` `/prowlarr`; prowlarr is served directly), and widen `SKIP_AUTH_REGEX` to `^/[^/]+/[^/]+/api` for the hash-prefixed tier api paths.
Group headers keep flowing via the existing Authentik `ak_groups` claim -> oauth2-proxy `X-Forwarded-Groups` mechanism. Validated with `kustomize build` + repo kubeconform (88/88 valid). No new Vault seeds required — the per-instance apikey Secrets already exist.
Reviewed-on: #416
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Rebuild the adult media instance as `fafflix`, mirroring the kids instance (`cheeztv`) HA pattern (Postgres-backed jellyfin-ha fork, Valkey transcode-lease store, shared-RWX transcode, CNPG + k8up/restic backups, static CephFS media PVs). Ben: "replace the adult jellyfin with the same pattern as kids." The adult instance holds no data currently, so the wipe/replace is sanctioned.
Stacked on top of `benvin/cheeztv` (#413) — base is that branch, not main.
## How
- Add `apps/base/fafflix/` — cheeztv pattern with `s/cheeztv/fafflix` on names/namespace/labels/secrets/buckets and the Vault kv path `kubernetes/namespace/fafflix/default/k8up-restic`.
- Media: mount the shared movies/tv CephFS subvolumes' `adult` subtree at `/media/movies` and `/media/tv`, plus the `kids` subtree at `/media/movies-kids` and `/media/tv-kids` (same two static PVs, new unique volumeHandles `fafflix-media-{movies,tv}-static`) so fafflix can resume kids content started on cheeztv. All media mounts readOnly, matching cheeztv.
- **Hostname preserved:** fafflix keeps serving the legacy jellyfin host `jellyfin.k8s.syd1.au.unkin.net` (TLS secret `fafflix-tls`, cert-manager re-mints). The dedicated fafflix domain switch is explicitly deferred — no `fafflix.unkin.net` created.
- config PVC on `cephfs-raid5-delete` (delete variant).
- Remove `apps/base/jellyfin/` + its overlay; swap `jellyfin`->`fafflix` in the `media-apps` ApplicationSet directory glob and the `media` AppProject namespace destination.
- Third-party images stay canonical upstream; the in-house `jellyfin-ha` image ref is unchanged.
## Verification
- `kustomize build` + `kubeconform -strict` clean on the fafflix overlay and the whole media appset scope (fafflix/cheeztv/arrstack): 0 invalid, 0 errors.
- No dangling `apps/base/jellyfin` references; `jellyfin.k8s.syd1.au.unkin.net` now served by exactly one app (fafflix); no `fafflix.unkin.net` anywhere.
Reviewed-on: #415
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Kids need their own safe Jellyfin front (cheeztv.unkin.net) with a library that only ever exposes kids titles, while those same titles stay resumable in the existing adult instance (fafflix). The fafflix.unkin.net domain switch is deliberately out of scope.
## How
- **New `cheeztv` instance** (`apps/base/cheeztv`): mirrors the fafflix HA Jellyfin stack 1:1 - same `jellyfin-ha:v0.1.3` fork image, 2-replica StatefulSet, RWX transcode PVC + HA takeover, Intel iGPU transcode, CNPG Postgres + PgBouncer, Valkey transcode store, k8up config backup, VMPodScrape.
- **Kids-only media**: its own static PVs on the *same* shared movies/tv CephFS subvolumes, mounted with `subPath: kids` so it sees only `movies/kids` and `tvshows/kids`.
- **Fully separate state**: own namespace, config PVC, `cheeztv-postgres` cluster, `cheeztv-valkey`, and `cnpg-cheeztv` / `cheeztv-config-backup` RGW buckets - nothing shared with fafflix.
- **DNS/cert**: `cheeztv.unkin.net` via internal Traefik + external-dns (198.18.200.4) + cert-manager `cheeztv-tls`, matching logviewer.unkin.net; plus the cluster hostname `cheeztv.k8s.syd1.au.unkin.net` mirroring fafflix.
- **fafflix change**: ADDS `movies/kids` + `tvshows/kids` subPath mounts alongside its current media mounts so kids libraries are resumable there. Existing mounts, hostname and ingress untouched.
- Registers cheeztv in the media ApplicationSet generator and AppProject destinations.
## Validation
kubeconform clean: cheeztv 28/28, jellyfin 25/25 valid. No arr/watchstate/domain changes.
Note: the k8up restic repo password must be seeded at `kv/kubernetes/namespace/cheeztv/default/k8up-restic` (same manual step as fafflix); templated Vault policy already grants read, no terraform-vault change needed.
Reviewed-on: #413
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Splits the arrstack media pipeline into an adult and a kids tier so kids content lives in its own libraries, databases, and downloader while reusing the shared prowlarr, CNPG cluster, Valkey, media PVCs, and S3 buckets. Backend workloads only — no routing/ingress, no arrproxy, no adult-instance changes in this PR.
How:
- sonarr-kids / radarr-kids: mirror the adult -unkin fork Deployments (same images/tags, 3 active-active replicas) on their own Postgres DBs (sonarr-kids-main / radarr-kids-main), UrlBase /3df803/sonarr and /3df803/radarr, and kids MediaCover/Backup S3 prefixes.
- Media mounts use the tvshows/kids and movies/kids subPaths of the shared media-tv / media-movies PVCs, so kids libraries are isolated on the same CephFS subvolumes.
- nzbget-kids: second downloader on its own RWO config PVC, completed downloads land in the kids media subtrees for same-filesystem hardlink imports.
- postgres: two managed roles + Database CRDs and their VSO-synced <app>-kids-db credential secrets.
- per-app VMPodScrape for each kids exportarr sidecar; new apps registered in the base kustomization.
Validated: kustomize build + kubeconform (96/96 valid), yamllint + pre-commit clean.
Reviewed-on: #412
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Deploy the new space-pirate UI background released in arrproxy v0.3.1.
- Bump arrproxy-api image to v0.3.1
- Bump arrproxy-ui image to v0.3.1
Reviewed-on: #411
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Tofu applies against authentik hit a cross-replica read-back race: a create lands on one of the 3 server pods and the provider's immediate read-back hits another pod that returns stale data, failing the run with "inconsistent result after apply" (upstream bug goauthentik/terraform-provider-authentik#954, still open). ClientIP session affinity keeps a whole apply run on a single pod, sidestepping the race until the provider is fixed.
## Changes
- Sets `server.service.sessionAffinity: ClientIP` with `sessionAffinityConfig.clientIP.timeoutSeconds: 60` in the au-syd1 authentik overlay values (rendered by the vendored chart's `templates/server/service.yaml`)
Reviewed-on: #410
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The `readonly` profile used by `logreader` sets `readonly=1`, which rejects any per-query settings change. HTTP clients (logviewer, chlog) send settings params like `max_execution_time` with their queries, so every query fails with `Code: 164 DB::Exception: Cannot modify 'max_execution_time' setting in readonly mode` — logviewer healthz returns 503 and CrashLoops, chlog queries fail. `readonly=2` keeps data access read-only while permitting settings changes, which is what HTTP clients need.
## Changes
- Changes the `readonly` profile in `apps/base/logging/clickhouseinstallation.yaml` from `readonly: "1"` to `readonly: "2"`
Reviewed-on: #409
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Both `arrstack-postgres` replicas were OOMKilled at the 1Gi memory limit on 2026-08-23 07:19, causing a failover to timeline 14, leaving replica-3 unrecoverable via pg_rewind, and blocking the argocd sync wave on Cluster Ready=False.
## Changes
- Raises the CNPG Cluster resources to requests 1Gi memory / 250m CPU and limits 2Gi memory / 1 CPU
**NOTE:** applying this restarts the postgres instances (CNPG rolling restart).
Reviewed-on: #408
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
unkin-agent