ghcr.io/oauth2-proxy/oauth2-proxy does not permit anonymous pulls (its
ghcr token endpoint returns DENIED), so artifactapi's anon-only ghcr
remote returns 401 and the sidecar is stuck in ImagePullBackOff. valkey-io
and other ghcr images work because they issue anon tokens.
Repoint the oauth2-proxy sidecar to the docker-internal local registry,
where the multi-arch v7.15.3 image (mirrored from quay.io, identical
digest sha256:10a11657...) is anon-pullable, restoring the arrproxy front
door.
Deploys arrproxy v0.1.0 (task 9 final integration) into the existing `arrstack` app: the external, Authentik-gated front door for sonarr/radarr/prowlarr with per-user API-key brokering. Manifests are wired to match the arrproxy v0.1.0 code (ports, routes, identity headers, keys dir), mirroring the ghp/artifactapi estate patterns.
## Topology (`apps/base/arrstack/arrproxy/`)
- **oauth2-proxy** Deployment + Service `arrproxy` is the single front (OIDC against the Authentik `arrstack` app). It path-routes via `--upstreams` to:
- `arrproxy-ui` for `/` and static SPA assets,
- `arrproxy-api` for `/api/*` (token API) and `/sonarr|/radarr|/prowlarr` (the *arr proxy).
- `--skip-auth-regex=^/[^/]+/api` bypasses oauth **only** for the *arr proxy API (`/sonarr/api...`), while `/api/tokens`, `/api/me` and the UI stay authenticated.
- Identity+groups reach the api via `--pass-user-headers` (`X-Forwarded-{User,Email,Groups}`). The api reads `ARRPROXY_GROUPS_HEADER=X-Forwarded-Groups` — `--set-xauthrequest` only populates auth_request *response* headers and never reaches an `--upstreams` backend, so it is intentionally not used.
- oauth2-proxy trusts `identity.unkin.net` (internal Vault-PKI CA) via a `combine-certs` initContainer (system roots + `vault-ca-cert`), same pattern as artifactapi.
## Data + secrets
- **CNPG** token store `arrproxy-db` (2 instances, cephrgw S3 backups + nightly ScheduledBackup). arrproxy-api does **not** self-migrate, so a wave-1 ArgoCD `Sync` hook Job applies `migrations/0001_init.sql` (mirrored into a ConfigMap) as the CNPG app user. DSN built from the generated `arrproxy-db-app` Secret.
- **VaultStaticSecrets** (arrstack `default` SA / templated role — no terraform-vault change): `arrproxy-pepper` (seeded) and `oauth-credentials`. The three real *arr keys are **reused** from the existing `<app>-apikey` Secrets, projected one file per app into `/etc/arrproxy/keys`.
## Exposure
- **Gateway** `traefik-external`, hostname `arrstack.unkin.net`, cert-manager `vault-issuer` TLS (`arrproxy-gateway-tls`); HTTPRoute → the oauth2-proxy entry Service (+ HTTP→HTTPS redirect).
- Adds the `arrstack.unkin.net` apex A record → external DMZ VIP `198.18.199.0` to the bind-operator `unkin.net` zone (`apps/base/bind-internal/authoritative/records.yaml`), mirroring the ghp/identity apex records.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly; `kubeconform` = 51/51 valid; `pre-commit` passes (no plain Secrets).
## Prerequisites / flags for end-to-end function (NOT in this PR)
- **(a)** Each *arr must run with `UrlBase=/<app>` (path-based hosting) — arrproxy preserves the `/<app>` prefix upstream.
- **(b)** The Authentik `arrstack` OIDC app (terraform-authentik PR #18) must be applied, with redirect URI `https://arrstack.unkin.net/oauth2/callback` registered and the `ak_groups` scope emitted. Confirm the discovery slug matches `OAUTH2_PROXY_OIDC_ISSUER_URL` (`https://identity.unkin.net/application/o/arrstack/`).
- **(c)** Browser XHR to `/<app>/api` is oauth-**bypassed** (skip-auth-regex), so the SPA-driven proxy calls that rely on the session (no per-user token) will not carry identity on that path — expected per the arrproxy trust model; token-based *arr clients are unaffected.
- **NetworkPolicy:** the arrproxy-api Service is not exposed via any HTTPRoute and the api strips forged identity headers in-code, but the README recommends a NetworkPolicy restricting api ingress to oauth2-proxy. The estate currently has **zero** NetworkPolicies, so one was not added here to avoid being the first (and risking probe breakage) before the CNI’s policy/probe behavior is confirmed — flagged as follow-up hardening.
- Please confirm: CNPG size (2 instances / 10Gi), backups on, hostname/VIP, and UI theme.
ARRPROXY_PEPPER was seeded via the agents AppRole at `kv/kubernetes/namespace/arrstack/default/arrproxy-pepper` (key `pepper`).
Reviewed-on: #377
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Jellyfin runs 2 replicas that coordinate distributed transcoding through the operator-managed `jellyfin-valkey` ValkeyCluster, but it was single-instance (shards:1, replicas:0) with no failover, so a node/pod loss would drop the shared transcode-lease state. This promotes that store to an HA topology.
Changes:
- Set replicas:2 (shards:1 unchanged) so the shard group is one primary + two replicas; losing the primary triggers an automatic in-cluster failover to a replica.
- Add scheduling.node.spread.shard.mode:Required so the three ValkeyNodes land on distinct hosts (kubernetes.io/hostname anti-affinity) and one host loss removes at most one node.
- Set podDisruptionBudget.mode:Cluster so the operator manages a quorum-aware PDB.
- Keep persistence omitted (emptyDir): lease state is ephemeral (30s TTLs), replication+failover already provide redundancy and a replaced node re-syncs from the primary, and an operator-managed PVC cannot carry the k8up.io/backup:"false" annotation the namespace k8up Schedule needs to skip in-use RWO volumes.
- Bump per-node memory limit to 512Mi (unchanged) / requests unchanged; resources are per ValkeyNode.
Connection string is unchanged: still cluster-mode via the single headless service `valkey-jellyfin-valkey:6379`; StackExchange.Redis seeds off it and auto-discovers topology and failovers.
kustomize build --enable-helm apps/overlays/au-syd1/jellyfin renders cleanly; pre-commit passes.
Reviewed-on: #375
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
jellyfin-ha v0.1.2 crashed on startup: the framework-dependent .NET 9 build could not find its runtime (`Microsoft.NETCore.App 9.0.0` required, `10.0.11` found), because the runtime image was pinned to a .NET 10 base. jellyfin-ha PR #7 realigned the runtime image to `mcr.microsoft.com/dotnet/aspnet:9.0` and v0.1.3 was released.
Verified before this bump: `docker run --entrypoint dotnet ...:v0.1.3 --list-runtimes` lists `Microsoft.NETCore.App 9.0.19`, and `/jellyfin/jellyfin --version` prints `Jellyfin.Server 10.11.7.0` with no framework-not-found error.
## Change
- Bump jellyfin image tag from v0.1.2 to v0.1.3 in `apps/base/jellyfin/statefulset.yaml`.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #374
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Bumps the jellyfin StatefulSet image tag to v0.1.2.
v0.1.2 is the first jellyfin-ha image successfully pushed to the artifactapi docker-internal registry: pipeline #7 (tag v0.1.2) is green, the docker build+push step exited 0, and the image is pullable (tags/list shows v0.1.2, manifest GET returns 200). v0.1.0 and v0.1.1 tag pipelines failed the CA/TLS check before the plugin-image + buildkit_config fix landed (PR #6).
Changes:
- Set apps/base/jellyfin/statefulset.yaml jellyfin container image tag from v0.1.0 to v0.1.2 (tag only).
Validated with kustomize build --enable-helm apps/overlays/au-syd1/jellyfin and pre-commit.
---------
Co-authored-by: unkin-agent <unkin-agent@users.noreply.git.unkin.net>
Reviewed-on: #373
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Why: P4 step 3. Now that the CA-baked buildx plugin lets `docker-internal/ghp` push succeed, deploy the in-house ghp fork (`v0.20.0-unkin4`) which adds static admin service tokens — the mechanism the forthcoming Vault ghp secrets engine uses to authenticate as admin and mint scoped tokens.
How:
- image: ghcr.io/goodtune/ghp:0.20.0 -> artifactapi.k8s.syd1.au.unkin.net/docker-internal/ghp:v0.20.0-unkin4 (a LOCAL registry image, not the ghcr pull-through that previously ImagePullBackOff-ed).
- add env GHP_AUTH_SERVICE_TOKENS from Secret ghp-app key `service_token`, `optional: true` — ghp runs unchanged until the token is seeded.
Requires (Ben, runbook): seed `service_token` into kv/kubernetes/namespace/ghp/default/app (same pattern as encryption_key/github-app). The default role's templated policy already grants read; no terraform-vault change. The existing ghp-app reloader annotation rolls the Deployment once the key appears. This same token value later becomes the Vault ghp engine admin_token (P4 steps 5-6).
Reviewed-on: #372
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Why: pushing in-house images to the internal-CA artifactapi registry fails `x509: certificate signed by unknown authority` with the upstream buildx plugin (and `:latest-insecure` does not fix it — buildkit still verifies the registry cert). The fix is the CA-baked plugin image now published at `artifactapi.k8s.syd1.au.unkin.net/docker-internal/plugin-docker-buildx` (unkin/plugin-docker-buildx v0.1.0).
How: add that image (bare + `:latest`) to `WOODPECKER_PLUGINS_PRIVILEGED` in the au-syd1 overlay so it may run privileged. Existing upstream entries kept so current repos are unaffected until they switch over.
Next: ghp `.woodpecker` switches its buildx step to this image, then ghp re-tags and its push succeeds.
Reviewed-on: #371
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The nzbget pod in namespace `arrstack` is stuck in `Init:ImagePullBackOff`.
## Why
The deployment pins `artifactapi.k8s.syd1.au.unkin.net/dockerhub/linuxserver/nzbget:v26.2` on both the `seed-config` initContainer and the main `nzbget` container. linuxserver does not publish a bare `v26.2` tag - the pull fails with:
```
failed to resolve reference ".../linuxserver/nzbget:v26.2": ... not found
Error: ImagePullBackOff
```
Confirmed against the artifactapi dockerhub mirror: `manifests/v26.2` -> HTTP 404. linuxserver version-pins nzbget under the `version-v<MAJOR>.<MINOR>` scheme, and the immutable date tags `26.2.<date>`. `manifests/version-v26.2` -> HTTP 200.
## Change
- Set both nzbget images (initContainer + main container) to `version-v26.2`, keeping nzbget on the intended 26.2 line and matching the version-pin convention used by the sibling sonarr (`4.0.19`) / radarr (`6.3.0`) deployments.
No other change. The `seed-config` init logic is unaffected - it already mirrors linuxserver's own init (`cp /app/nzbget/share/nzbget/nzbget.conf /config/nzbget.conf`), and since it seeds `/config/nzbget.conf` before the main container, linuxserver's init copy is skipped (no conflict). Instant-move download layout on the media PVCs is preserved.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/arrstack` renders cleanly (both images -> `version-v26.2`, no stale `v26.2`).
- `pre-commit run --files apps/base/arrstack/nzbget/deployment.yaml` passes (yamllint et al).
Scoped to `apps/base/arrstack/nzbget/deployment.yaml` only; the top-level `apps/base/arrstack/kustomization.yaml` was not touched (another agent is editing sonarr/radarr/prowlarr on a separate branch).
---------
Co-authored-by: unkin-agent <unkin-agent@users.noreply.git.unkin.net>
Reviewed-on: #370
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
## Why
The sonarr/radarr/prowlarr images self-generate an API key into
/config/config.xml on first boot, so the key is unmanaged and differs per
volume reset. This makes Vault the source of truth for those keys (override
bootstrap, chosen by Ben): the key is minted in Vault and enforced into
config.xml before each app starts.
## Changes
- Add a `VaultAuth` `default` in the `arrstack` namespace (kubernetes auth,
mount `k8s/au/syd1`, role `default`, SA `default`), mirroring jellyfin.
- Add a per-app `VaultStaticSecret` that syncs
`kv/kubernetes/namespace/arrstack/default/<app>` (key `apitoken`) into the
`<app>-apikey` Secret. The `default` k8s role's templated policy already
grants read on that path for the `arrstack/default` SA, so no
terraform-vault change is needed.
- Add an `apikey-init` initContainer to each of the three deployments that
reads `API_KEY` from the VSO-created Secret, fails closed on a missing or
non-hex value, and writes/updates only the `<ApiKey>` element in
`/config/config.xml` (then fixes ownership 1000:1000, mode 600). Image is a
pinned busybox via artifactapi to keep this PR atomic (no new image
dependency).
- Wire the new manifests into the base and per-app kustomizations.
## Notes
- Keys already seeded in Vault at `kv/kubernetes/namespace/arrstack/default/<app>`.
- nzbget is out of scope: it has no config.xml `<ApiKey>` (uses ControlPassword),
a separate follow-up.
- Downstream consumers (proxy, terraform) currently read
`kv/service/media-apps/<app>`; the authoritative key now lives at the path
above. Reconciliation is deferred.
---------
Co-authored-by: unkin-agent <unkin-agent@git.unkin.net>
Reviewed-on: #369
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
Why: the new unkin/plugin-docker-buildx repo's pipelines run their buildah build/push step under `serviceAccountName: plugin-docker-buildx`, which must exist in the woodpecker namespace.
How: add serviceaccount_plugin_docker_buildx.yaml + wire it into kustomization.yaml, mirroring the existing per-repo woodpecker SAs.
Pairs with unkin/plugin-docker-buildx#1.
Reviewed-on: #368
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Stands up the download-automation stack (sonarr, radarr, prowlarr, nzbget) in a new `arrstack` namespace, sharing the same CephFS TV/movies subvolumes that jellyfin serves read-only (jellyfin static-PV rework is on the base branch, PR #237). Downloads and libraries live on one filesystem per media type so the arrs import with atomic hardlink moves.
## Changes
- Add `apps/base/arrstack/` namespace with its own static CephFS PVs (`arrstack-media-tv`, `arrstack-media-movies`) + RWX PVCs (`media-tv`, `media-movies`) bound to the same rootPaths jellyfin uses; each namespace gets a unique PV name/volumeHandle pinned by claimRef.
- Deploy sonarr (mounts media-tv RW), radarr (media-movies RW), prowlarr (config only), nzbget (both media PVCs RW); single replica, Recreate strategy, LinuxServer images via artifactapi dockerhub remote, PUID/PGID/TZ, probes, fsGroup for CephFS writability.
- Per-app config PVCs on `cephrbd-fast-retain` (RWO, 5Gi) for the SQLite state; internal-only Gateway + HTTPRoute per app (traefik-internal, vault-issuer, external-dns `<app>.k8s.syd1.au.unkin.net`), no oauth yet.
- nzbget: initContainer seeds `/config/nzbget.conf` (copy image template if absent, append managed path/category block once) and creates the media dirs; category `tv` -> `/media/tv/downloads`, `movies` -> `/media/movies/downloads`.
- Wire ArgoCD: add `apps/overlays/*/arrstack` to the media ApplicationSet. (media AppProject already gains the arrstack destination + PersistentVolume whitelist on the base branch.)
## Depends on
terraform-artifactapi PR #31 (allowlist `^linuxserver/` on the dockerhub remote) must be merged + applied before these pods can pull.
## Follow-up (terraform)
Root folders (/media/tv/series, /media/movies/films), download-client wiring (host `nzbget`, port 6789, categories tv/movies) and API keys are configured later via terraform-arr.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Co-authored-by: Ben Vin <neotheo@gmail.com>
Reviewed-on: #366
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Jellyfin depended on a self-managed `redis` Deployment for its distributed transcode-session store. Now that the valkey-operator is deployed cluster-wide, this hands that store to the operator so it is declaratively managed (rolling upgrades, health, future HA) instead of a hand-rolled Deployment.
## Changes
- Add `apps/base/jellyfin/valkey.yaml`: a `ValkeyCluster` (`valkey.io/v1alpha1`) named `jellyfin-valkey`, single instance (`shards: 1`, `replicas: 0`), image via the artifactapi dockerhub mirror (`valkey/valkey:9.0.0`), metrics exporter disabled, redis-parity resources.
- Omit persistence so `/data` is an `emptyDir`: the transcode-lease state is ephemeral, and the operator-managed PVC cannot carry the `k8up.io/backup: "false"` annotation the namespace-wide k8up Schedule needs to skip an in-use RWO volume (the old redis PVC carried it).
- Remove `redis-deployment.yaml`, `redis-service.yaml`, `redis-pvc.yaml` and drop them from `kustomization.yaml`; add `valkey.yaml`.
- Repoint `Jellyfin__TranscodeStore__RedisConnectionString` at the operator client Service `valkey-jellyfin-valkey:6379`, preserving `abortConnect=false` and `LeaseDurationSeconds`.
## Auth / TLS
None. This operator runs Valkey cluster-mode-enabled with `protected-mode no` and leaves the built-in `default` user passwordless (no `requirepass`), so clients connect unauthenticated over plaintext — parity with the previous open redis. StackExchange.Redis auto-discovers the single node via the headless Service. No `VaultStaticSecret` / plain Secret is introduced.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/jellyfin` renders cleanly: `ValkeyCluster` present, no redis Deployment/Service/PVC, connection string updated.
- pre-commit passes (incl. the plain-secrets guard).
- kubeconform: 10 valid / 0 invalid; the `ValkeyCluster` is skipped (CRD schema not vendored).
## Follow-up
HA (task 6) is intentionally out of scope: it will raise `replicas` (and/or `shards`) and add anti-affinity/topology spread. Enabling auth (ACL `default` user + password via `VaultStaticSecret`) and/or TLS can also be layered on later.
Reviewed-on: #367
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Deploy the jellyfin-ha fork as a genuine high-availability service rather than a
single replica, so its two headline capabilities can actually be exercised: the
Redis-coordinated distributed transcoding (a surviving pod resumes the HLS
segments of a pod that dies mid-stream) and the experimental PostgreSQL main
database (which is what lets more than one replica share the same library). It
lands in its own `jellyfin` namespace under a new `media` ArgoCD project.
## How
**Workload — StatefulSet, 2 replicas.** The Deployment becomes a StatefulSet so
each pod has a stable name. The fork's Redis transcode lease keys ownership on
`JELLYFIN_INSTANCE_ID`, which is set from the downward-API pod name, giving each
replica a unique, stable lease identity. Soft `podAntiAffinity` spreads the two
pods across nodes and a `PodDisruptionBudget` keeps `minAvailable: 1` through
drains and rollouts.
**Main database — CloudNativePG.** A CNPG trio in-namespace mirrors the litellm
pattern: a 3-instance `Cluster`, a PgBouncer `Pooler`, and Ceph RGW (barman) S3
backups to a dedicated `cnpg-jellyfin` bucket owned by a `cephrgw`
`ObjectStoreUser`. An init container writes `/config/config/database.xml`
selecting the fork's `Jellyfin-PostgreSQL` provider, and the connection string is
composed from the CNPG-generated `jellyfin-postgres-app` secret (username /
password / dbname) pointed at the pooler service — the password is never
rendered into the manifest. Library-item metadata therefore moves off SQLite;
metadata images, plugins, subtitles and config XML stay on `/config`.
**Storage.**
- `/config` is now a shared `ReadWriteMany` cephfs PVC (raid5, retain) so every
replica reads/writes the same metadata and config.
- `/config/transcodes` — the fork's real transcode temp path — is a shared RWX
PVC (raid5, delete). This is the load-bearing fix: takeover reads the dead
pod's in-flight `.ts`/`.m3u8` segments off shared storage, so per-pod scratch
would silently break it.
- `/cache` is per-pod via a `volumeClaimTemplate` (RWO).
- The media library stays a fresh, empty RWX PVC mounted read-only; populating
it is out of scope.
**Hardware transcoding.** The container requests the `gpu.intel.com/i915` Intel
device-plugin resource (which pins the pod to a GPU-labelled node and injects the
DRI render node — no `/dev/dri` hostPath or privileged container) plus the
render/video supplemental groups. VA-API hardware acceleration is now on by
default: the `inject-config` init container seeds `/config/config/encoding.xml`
with `HardwareAccelerationType` `vaapi`, `EnableHardwareEncoding`, the injected
render node (`/dev/dri/renderD128`) and h264/hevc hardware decode, so transcodes
use the iGPU on first boot with no manual admin-UI step. Both seed files
(`database.xml`, `encoding.xml`) are written only when absent, so later admin
changes persisted to the shared RWX `/config` are never clobbered on restart.
**Networking.** The Gateway/HTTPRoute (traefik-internal,
`jellyfin.k8s.syd1.au.unkin.net`) is unchanged; the Service gains
`sessionAffinity: ClientIP` to keep a client pinned to one replica and reduce
transcode-session churn.
**Redis.** The in-namespace single-replica Redis stays as the transcode lease
store.
## Follow-up
UDP auto-discovery is disabled, but scheduled library scans still run on every
replica (redundant scans). Single-scanner leader election is a planned follow-up
pending a fork feature, tracked separately.
---------
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Co-authored-by: Ben Vin <neotheo@gmail.com>
Reviewed-on: #237
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Makes the valkey-operator available in the cluster so a later change can replace the self-managed redis Deployment with a Valkey CR. This PR only installs the operator; it does not create any Valkey CR or touch existing apps.
Changes:
- Add `apps/base/valkey-operator-system/` (namespace `valkey-operator-system`, mirroring the cnpg-system pattern).
- Add `apps/overlays/au-syd1/valkey-operator-system/` which inflates the `valkey-operator` 0.5.0 Helm chart from the artifactapi virtual helm repo with `includeCRDs: true` (the chart ships CRDs in the Helm-native `crds/` dir, which `helm template` skips otherwise).
- Route the operator image through artifactapi: `artifactapi.k8s.syd1.au.unkin.net/ghcr/valkey-io/valkey-operator:v0.5.0`.
- Register `apps/overlays/*/valkey-operator-system` in the platform ApplicationSet. No AppProject change needed: the `*-system` destination and the virtual-helm sourceRepo already cover it (same as cnpg-system/cephrgw-system).
Depends on unkin/terraform-artifactapi#29 (adds the valkey helm remote + virtual member and allowlists the operator image). That PR must merge and apply before this one syncs, or the chart pull / image pull will 404/403.
Validated with `kustomize build --enable-helm` against the upstream chart (identical version/values): renders the Namespace, 2 CRDs, operator Deployment (image = the artifactapi ghcr path), RBAC, SA, and metrics Service, all in `valkey-operator-system`.
Reviewed-on: #365
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The ghp fork (`unkin/ghp`) now has `.woodpecker` pipelines whose steps run under `serviceAccountName: ghp`. Without this ServiceAccount in the `woodpecker` namespace, pods fail to schedule (`error looking up service account woodpecker/ghp: serviceaccount "ghp" not found`), which fails the required `ci/woodpecker/pr/build` status.
## Change
- Add `apps/base/woodpecker/serviceaccount_ghp.yaml` (ServiceAccount `ghp` in namespace `woodpecker`), mirroring the existing per-project SAs (e.g. `jellyfin-ha-src`).
- Register it in `apps/base/woodpecker/kustomization.yaml`.
Unblocks the ghp CI onboarding for unkin/ghp#1.
Reviewed-on: #364
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The k8up CRDs already exist in the cluster (owned by a separate `k8up-crds` Helm release) but the k8up operator itself was never deployed, so no k8up backup/restore custom resources are reconciled. This installs the operator following the cnpg-system pattern (Helm inflated via kustomize `helmCharts`) in the `platform` ArgoCD project, namespace `k8up-system`.
- Add `apps/base/k8up-system/` (Namespace + kustomization).
- Add `apps/overlays/au-syd1/k8up-system/` inflating the **k8up chart `4.10.0`** (pinned) from the artifactapi virtual-helm proxy: `replicaCount: 2` (leader election with hot-standby follower), operator resource requests/limits (20m/128Mi req, 500m/256Mi lim), and ServiceMonitor/PrometheusRule disabled (require Prometheus Operator CRDs).
- Add a `PodDisruptionBudget` (`minAvailable: 1`) in the overlay, since the k8up chart exposes no native PDB value.
- Register `apps/overlays/*/k8up-system` in the platform ApplicationSet.
**CRD management:** The k8up chart ships its CRDs via the Helm `crds/` directory (there is no `installCRDs`/`skipCrds`/`enableCRDCreation` value). The kustomize helm inflator does not emit `crds/`, so `kustomize build` renders only the operator (Deployment, RBAC, SA, Service) and no CRDs. ArgoCD therefore does not fight the existing `k8up-crds` Helm release over CRD ownership.
**platform.yaml (AppProject):** no change needed. The virtual-helm sourceRepo, the `*-system` destination, and the ClusterRole/ClusterRoleBinding/CRD/webhook clusterResourceWhitelist entries are already present and cover everything this app renders.
**Validation:** `kustomize build --enable-helm apps/overlays/au-syd1/k8up-system` succeeds (Deployment replicas 2, PDB, Namespace, RBAC; 0 CRDs). `make kubeconform` reports k8up-system 14/14 valid; the only failure is the pre-existing, unrelated `cattle-system` rancher chart kubeVersion issue. pre-commit passes.
Reviewed-on: #363
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The autobackup-operator Woodpecker pipeline needs a dedicated ServiceAccount to run its CI jobs under, matching the existing per-project operator CI SAs in the woodpecker namespace (e.g. kea-operator-ci).
- Adds ServiceAccount autobackup-operator-ci in the woodpecker namespace
- Registers serviceaccount_autobackup_operator_ci.yaml in the woodpecker kustomization.yaml resources list
Reviewed-on: #362
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
https://ghp.unkin.net/ 308-loops onto itself. The traefik gateway terminates TLS and forwards cleartext to the ghp Service port 80 -> container :8080, but :8080 was `GHP_SERVER_HTTP_LISTEN` — ghp's http->https **308 redirect** listener. So ghp bounced every request back to https, the gateway re-forwarded it to :8080, and it looped forever.
## Root cause (confirmed against ghp source)
`internal/server/server.go` `Run()` picks the serving mode:
```go
hasTLS := s.cfg.Server.HTTPSListen != "" || (systemd socket + certs)
if hasTLS { return s.serveTLS(...) } // app on HTTPSListen (TLS); HTTPListen = 308 redirect; Listen IGNORED
return s.servePlain(...) // app (full handler: mgmt UI + API) on Listen, cleartext
```
It is **strictly either/or**. `serveTLS` serves the app on `HTTPSListen` and gives `HTTPListen` only `httpsRedirectHandler()` (`redirect.go`: `http.StatusPermanentRedirect` = 308). `servePlain` serves the same full handler on `Listen` in cleartext (`createListener()` uses `cfg.Server.Listen`). The mgmt UI is the same `handler` in both modes, so it IS served on the plain `Listen` port.
Behind a TLS-terminating gateway that forwards cleartext to :8080, ghp therefore has to run in **plain mode**. Keeping `GHP_SERVER_HTTPS_LISTEN` would keep `hasTLS` true, leave `GHP_SERVER_LISTEN` ignored, and nothing would serve cleartext on :8080.
## Change
- **configmap**: drop `GHP_SERVER_HTTPS_LISTEN` and `GHP_SERVER_HTTP_LISTEN`; set `GHP_SERVER_LISTEN: ":8080"` so :8080 SERVES the app; add `GHP_SERVER_TRUST_PROXY_HEADERS: "true"` so ghp trusts the gateway's `X-Forwarded-*`/`Forwarded` for scheme/host (`GHP_SERVER_BASE_URL` already set).
- **deployment + vmservicescrape**: the metrics server only wraps TLS when `hasTLS` is true (`Run()` gates `loadTLSConfig` on `hasTLS`); in plain mode it is cleartext, so the `/metrics` liveness/readiness probes and the VMServiceScrape switch from HTTPS/https to HTTP/http.
Service, HTTPRoute and Gateway are unchanged. configmap+deployment carry the stakater reloader annotation, so pods roll on the change.
## Deviation from the brief
The brief said to keep `GHP_SERVER_HTTPS_LISTEN: ":8443"`. Source shows that is incompatible with serving cleartext on :8080 (the two modes are mutually exclusive), so this drops it. The unused `GHP_TLS_CERT_FILE`/`KEY_FILE`, the `tls` volume, and containerPort 8443 are left in place (harmless) for an easy revert to TLS mode. The alternative — gateway -> Service 443 -> :8443 with a BackendTLSPolicy — is the bigger change flagged in the brief and is NOT taken here.
Validated: `kustomize build apps/overlays/au-syd1/ghp` clean, kubeconform 0 invalid/0 errors, pre-commit clean. Not applied.
---------
Co-authored-by: unkin-agent <agent@unkin.net>
Reviewed-on: #361
Co-authored-by: Unkin Agent <unkin-agent@unkin.net>
Co-committed-by: Unkin Agent <unkin-agent@unkin.net>
## Why
Make `ghp.unkin.net` the primary URL that the ghp app responds to. Until now ghp has only been reachable at the internal admin name `ghp.k8s.syd1.au.unkin.net`; this promotes the apex `unkin.net` name to canonical (mirroring the gitea apex pattern) while keeping the k8s name as the admin/internal route.
## How
- **`apps/base/ghp/gateway.yaml`** — add `ghp.unkin.net` as the PRIMARY http/https listeners (`http-primary`/`https-primary`) and keep `ghp.k8s.syd1.au.unkin.net` as the admin route (`http-admin`/`https-admin`), mirroring gitea. Add the cert-manager gateway annotations (`cluster-issuer: vault-issuer`, `common-name: ghp.unkin.net`, `private-key-size: "4096"`) so cert-manager mints a NEW `ghp-gateway-tls` Secret with CN `ghp.unkin.net` + a SAN per TLS listener hostname (so it also covers the k8s host). Both https listeners' `certificateRefs` point at `ghp-gateway-tls`. This is a separate Secret from `ghp-tls` (ghp's own GitHub-impersonation cert), so cert-manager never fights over it. external-dns annotation stays scoped to the k8s host only — the apex is not published by external-dns.
- **`apps/base/ghp/httproute.yaml`** — add `ghp.unkin.net` to `hostnames`; update `parentRefs` to attach to the renamed `http-primary`/`http-admin`/`https-primary`/`https-admin` sections.
- **`apps/base/ghp/configmap.yaml`** — set `GHP_SERVER_MANAGEMENT_HOST: ghp.unkin.net` and `GHP_SERVER_BASE_URL: https://ghp.unkin.net`, so ghp serves its mgmt UI on that Host and OAuth callbacks use the new base.
- **`apps/base/bind-internal/authoritative/records.yaml`** — add a bind-operator `DNSRecord` `ghp-dns-internal` for `ghp.unkin.net` -> **A 198.18.200.4** (traefik-internal gateway VIP), mirroring the active `identity-dns-internal` record in the same file.
## ⚠️ NOTE — GitHub App OAuth callback must be updated on github.com
The GitHub App's OAuth callback URL must be changed on github.com to **`https://ghp.unkin.net/auth/github/callback`**. It was set to the k8s host in the runbook; OAuth login will break until this is updated.
## Note on DNS authority
The bind-operator apex `unkin.net` zone comments warn that some apex names may still be served by the puppet DNS master (records from PuppetDB), with the k8s bind cluster not yet confirmed as the live authority for `unkin.net`. This DNSRecord mirrors the already-active `identity`/`lb1` apex records in the same file, so it is consistent with them — but if the k8s bind cluster is not the live authority for `unkin.net`, `ghp.unkin.net` will not resolve from this Record alone and the puppet master record must be added instead.
## Validation
`kustomize build` of the ghp and bind-internal overlays render clean; kubeconform (strict) valid; yamllint + pre-commit clean. Not applied.
Reviewed-on: #360
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The `ghp` app is deployed but its pods are stuck 0/1 Ready (and were ImagePullBackOff), for three separate reasons this PR fixes:
- **ImagePullBackOff:** kubelet anonymous pulls fail on the artifactapi ghcr pull-through because ghcr.io's per-scope token auth is not proxied. The direct public image pulls anonymously, so switch to it.
- **Pods never Ready:** ghp serves its metrics endpoint over **HTTPS** (TLS is configured globally), but the liveness/readiness probes used the default HTTP scheme, so the kubelet probe got an HTTPS-server error and the pods never went Ready.
- **Scrape failure:** the VMServiceScrape hits that same HTTPS endpoint and needs a matching scheme/TLS config, or VM scraping of ghp fails.
- The `GHP_ADMINS` value was still a placeholder.
## How
- `deployment.yaml`: image -> `ghcr.io/goodtune/ghp:0.20.0`; liveness + readiness probe `scheme: HTTP` -> `HTTPS` (kubelet does not verify the probe cert).
- `migrate-job.yaml`: image -> `ghcr.io/goodtune/ghp:0.20.0` (shared image).
- `vmservicescrape.yaml`: endpoint `scheme: https` + `tlsConfig.insecureSkipVerify: true` (internal-CA cert; pod-IP target not in SANs).
- `configmap.yaml`: `GHP_ADMINS` -> `neoloc`.
Validated: `kustomize build apps/overlays/au-syd1/ghp` renders clean, kubeconform + pre-commit pass. Not applied.
## Follow-up (not fixed here)
The artifactapi ghcr pull-through does not proxy ghcr.io's per-scope token auth for anonymous kubelet pulls — worth closing that gap so estate images can go back through artifactapi.
Reviewed-on: #359
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Ship artifactapi v3.11.0: mirrorlist support, least-connections balancing, and cache-flush. Both `artifactapi` and `artifactapi-ui` images are built and present in the Gitea registry (release pipeline green).
## How
Bump the api and ui image pins v3.10.1 -> v3.11.0:
- `apps/base/artifactapi/api-deployment.yaml`: `git.unkin.net/unkin/artifactapi:v3.10.1` -> `:v3.11.0`
- `apps/base/artifactapi/ui-deployment.yaml`: `git.unkin.net/unkin/artifactapi-ui:v3.10.1` -> `:v3.11.0`
Overlay `apps/overlays/au-syd1/artifactapi` renders clean via `kustomize build`.
Reviewed-on: #356
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Ships the #117 local-repodata determinism fix (rpm + deb/apk) now that the v3.10.1 images are built and present in the Gitea registry. Simple one-release patch bump from the deployed v3.10.0 pin.
How:
- Bump artifactapi api image pin v3.10.0 -> v3.10.1
- Bump artifactapi-ui image pin v3.10.0 -> v3.10.1
Reviewed-on: #355
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
artifactapi v3.10.0 ships Alpine/apk support — an apk local repository, a `github_alpine` remote, and metadata-only handling — plus the new UI how-to sections. Rolling the deployed images forward from v3.9.1 delivers apk repo support and the UI docs to the cluster.
## What
- Bump `apps/base/artifactapi/api-deployment.yaml` API image `git.unkin.net/unkin/artifactapi` from `v3.9.1` to `v3.10.0`.
- Bump `apps/base/artifactapi/ui-deployment.yaml` UI image `git.unkin.net/unkin/artifactapi-ui` from `v3.9.1` to `v3.10.0`.
The v3.10.0 images are published/building from the `v3.10.0` tag. Only the two image tags change; `kustomize build`, yamllint, and pre-commit all pass.
Reviewed-on: #354
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
artifactapi v3.9.1 ships Debian/apt repository support (local, remote, and metadata-only repos) plus the UI "how to use" deb sections. Roll it out to the deployed cluster. The v3.9.1 images are published/building from the tag.
## How
- Bump api image `git.unkin.net/unkin/artifactapi` to `v3.9.1`.
- Bump ui image `git.unkin.net/unkin/artifactapi-ui` to `v3.9.1`.
Reviewed-on: #353
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
The jellyfin-ha-src Woodpecker build+test pipeline sets a `serviceAccountName`
that must exist in the `woodpecker` namespace.
## How
- Add a `jellyfin-ha-src` ServiceAccount under `apps/base/woodpecker/`,
mirroring the `kea-operator-ci` build/test account (no Vault binding needed;
the pipeline only restores/builds/tests).
- Register it in the woodpecker kustomization.
Reviewed-on: #352
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
artifactapi v3.8.0 ships the github_rpm remote (GitHub releases exposed as a dnf/yum repo via synthesized repodata, a background metadata syncer with multi-replica leasing, and a server-level GitHub machine credential for authenticated requests). This rolls the running deployment onto that release.
## How
- Bump the api image `git.unkin.net/unkin/artifactapi` from v3.7.7 to v3.8.0.
- Bump the ui image `git.unkin.net/unkin/artifactapi-ui` from v3.7.7 to v3.8.0.
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #351
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
ArgoCD fails to sync the puppet app with:
Deployment.apps "puppetserver-master" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is 'Recreate'
The manifest is already correct: #341 changed the master to `spec.strategy.type: Recreate` with no rollingUpdate block. The failure is a live-object artifact. When the master ran RollingUpdate the API server defaulted `spec.strategy.rollingUpdate` (maxSurge/maxUnavailable) onto the object. That defaulted field is owned by no applier, so neither a client-side merge nor server-side apply drops it when the desired manifest omits it. The live object therefore keeps `rollingUpdate` while gaining `type: Recreate`, which the API server rejects — blocking every sync.
k8s forbids any `rollingUpdate` field when `strategy.type` is `Recreate`; the two are mutually exclusive, so the sync cannot converge until the stale field is removed from the live object.
## Changes
- Annotate the `puppetserver-master` Deployment with `argocd.argoproj.io/sync-options: Replace=true`. Replace performs a full PUT that overwrites the whole object, dropping the stale `rollingUpdate` field and letting the Recreate strategy apply cleanly. The annotation is scoped to this one resource, so puppetdb/puppetboard/compiler keep the app-wide ServerSideApply behaviour.
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/puppet` renders `puppetserver-master` with `strategy: { type: Recreate }` (no rollingUpdate) and the new sync-options annotation.
- `make kubeconform`: puppet overlay 34/34 valid; only the known cattle-system rancher kubeVersion incompatibility fails.
- `pre-commit`: all hooks pass.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #349
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Kubernetes defaults `apiVersion: v1` and `kind: PersistentVolumeClaim` onto every StatefulSet `volumeClaimTemplates` entry, but neither the raw manifests nor the rendered helm charts in this repo emit those fields. Live StatefulSets therefore carry TypeMeta that git lacks, and ArgoCD reports a diff that removes it.
`volumeClaimTemplates` are immutable on an existing StatefulSet, so ArgoCD can never apply the removal. The affected StatefulSets stay perpetually OutOfSync and the un-appliable diff can contribute to sync failures.
Rendering the overlays confirms this is a chronic render-vs-live mismatch, not a regression: the source manifests/charts have never emitted the TypeMeta. Affected StatefulSets (render lacks TypeMeta, live defaults it in):
- `consul-server` (consul, helm render)
- `nats` (logging, helm render)
- `kanidm` (kanidm, raw manifest)
`vault` and the `woodpecker` StatefulSets already emit TypeMeta from their charts and are unaffected.
## How
- Add a fleet-wide `resource.customizations.ignoreDifferences.apps_StatefulSet` to the `argocd-cm` patch, using `jqPathExpressions` to ignore the defaulted `apiVersion` and `kind` under every `volumeClaimTemplates` entry.
A single global customization is chosen over per-manifest edits because the affected StatefulSets span both raw manifests (kanidm) and helm renders (consul, nats) whose output cannot be edited; it is inert for StatefulSets that already emit TypeMeta (vault, woodpecker) and future-proof for new ones. The live StatefulSets are left untouched — their `volumeClaimTemplates` are immutable, and recreation would orphan PVCs.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #350
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
- The cert-manager v1.20.2 controller crashloops: strict decoding of its ControllerConfiguration rejects the unknown field `acmeDNS01` (`failed to load config file ... strict decoding error: unknown field "acmeDNS01"`), so `/var/cert-manager/config/config.yaml` fails to load and the controller never starts. The rollout is stuck with only the old pod running.
- PR #337 placed the DNS-01 recursive-nameserver settings under `acmeDNS01`, but the field in the `controller.config.cert-manager.io/v1alpha1` schema is `acmeDNS01Config` (`ACMEDNS01Config`, with `recursiveNameservers` / `recursiveNameserversOnly`). The recursive-ns settings belong in the config file, not `extraArgs`; the CLI flags feed the same struct but the chart already renders a `--config` ControllerConfiguration, so the correct fix is the correct field name.
## How
- Rename the `config:` block `acmeDNS01` to `acmeDNS01Config`, keeping `recursiveNameservers` (`8.8.8.8:53`, `1.1.1.1:53`) and `recursiveNameserversOnly: true` so DNS-01 resolution and self-checks still use the public DNS view for the split-horizon delegation.
Rendered `kustomize build --enable-helm` confirms the ConfigMap `config.yaml` now carries a valid `acmeDNS01Config` block and no longer contains the invalid `acmeDNS01`; the cert-manager overlay is kubeconform-clean (55 valid, 0 invalid).
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #347
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
NetBox 4.6.5 refuses to save v2 API tokens without `API_TOKEN_PEPPERS` ("Unable to save v2 tokens: API_TOKEN_PEPPERS is not defined"), which blocks creating the superuser token the NetBox Vault engine needs (it defaults to v2 tokens). The chart only auto-generates a pepper when it creates the config secret itself; it does not do that while `existingSecret` (`netbox-secret-key`) is set, so the config secret carries no `api_token_peppers` key.
## Changes
- Document `api_token_peppers` on the `netbox-secret-key` VaultStaticSecret: a JSON pepper map `{"1": "<random>"}` seeded once into Vault alongside `secret_key`. VSO syncs every key at the path into the config secret, which the chart already mounts as an optional file into `API_TOKEN_PEPPERS`.
- Add a reloader annotation via `commonAnnotations` so the `netbox` and `netbox-worker` Deployments roll when `netbox-secret-key` changes, picking up the seeded pepper (and any rotated `secret_key`) without a manual restart.
## Follow-up (out of band)
seed the pepper once (rotating it invalidates existing v2 tokens):
```
PEP=$(openssl rand -base64 48 | tr -d '\n')
vault kv patch kv/kubernetes/namespace/netbox/default/netbox-secret-key \
api_token_peppers="{\"1\": \"$PEP\"}"
```
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #346
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
kea-dhcp4 crash-loops on a cold container start: the HA hook resolves the StatefulSet peer URL hostnames once at config load, but the peer DNS records aren't resolvable in the first instant of a fresh container, and kea exits hard instead of retrying. Verified in-cluster that the rendered config validates once DNS is warm, so it's a startup race. kea-operator v0.1.3 gates dhcp4 startup on a bounded `kea-dhcp4 -t` retry (~120s, then proceeds/fails loud).
## How
- bump kea-operator, kea, and kea-api images to v0.1.3
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #344
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: shrink the blast radius of the Puppet control-plane pods (CA/eyaml keys, compiled catalogs) per the security sweep in #307 — remove root where it is not required and strip cargo-culted capabilities.
How:
- puppetboard cert-generator init: root+APE:true -> uid 1000, drop:[all], APE:false; pod fsGroup 1000; removed trailing `chown -R 1000:1000` (PVC now group-owned).
- puppetdb create-log-dir init: root -> uid 999, drop:[all], APE:false; pod fsGroup 999; removed `chown 999:999`.
- All OpenVox capability add-lists: removed the duplicate CAP_-prefixed spellings (k8s normalises both to the same kernel cap) and dropped the unused AUDIT_WRITE.
- Added allowPrivilegeEscalation:false and seccompProfile RuntimeDefault across the workloads.
Stays root (evidence-backed, class-B fallback): the puppetserver master/compiler and puppetdb main containers, plus the perms-and-dirs and generate-types root containers. The OpenVox image entrypoint runs `chown -R puppet:puppet` over root-owned baked-in dirs and drops the JVM to the puppet user via `runuser` (needs CHOWN/SETUID/SETGID); a non-root start crashloops. Their cap sets are reduced to the minimum justified (CHOWN/DAC_OVERRIDE/FOWNER[/SETUID/SETGID]).
Validation: `kustomize build --enable-helm` clean; kubeconform 0 invalid / 0 errors; pre-commit (yamllint etc.) green. Confirmed against live pods: puppetserver/puppetdb JVMs already run as puppet/puppetdb via `runuser`; `pam_loginuid` is absent from the su/runuser PAM stacks and loginuid is unset, so dropping AUDIT_WRITE is safe.
Post-merge smoke test (puppet had an outage this session — watch closely): after argocd sync, confirm puppetserver master + a compiler reach `running` at /status/v1/simple, puppetdb reaches `running`, puppetboard serves 200, and the generate-types + g10k CronJobs complete — i.e. catalogs still compile and reports still ingest.
Closes#307https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #319
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
kea-dhcp4 and kea-ctrl-agent crash-loop because kea 2.6.5 refuses a unix-socket directory more relaxed than 0750, but the operator's shared emptyDir mounts `/var/run/kea` at 0777 (`'socket-name' is invalid: socket path:/var/run/kea ... more relaxed permissions than 750`). kea-operator v0.1.2 renders entrypoints that tighten it.
## How
- bump kea-operator, kea, and kea-api images to v0.1.2
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #343
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: extend the Tier-1 survey with 7 more high-value log sources so they parse into logs.raw columns/.fields for real querying instead of the generic catch-all. **Stacks on #318 — merge after it.**
How:
- 7 mutually-exclusive app_route conditions + parse transforms into the ClickHouse sink: **bind_query** (k8s bind-* + VM named), **rancher_audit** (cattle-system sidecar JSON), **cnpg_pg** (ONE transform for all 10 CNPG clusters via the `.postgres` container), **gitea** (router+access, k8s+VM), **puppet** (openvoxserver/openvoxdb logback + access), **litellm** (JSON request logs), **postfix** (per-line maillog).
- Carve `.postgres` out of the Tier-1 authentik route + new puppet/gitea/litellm routes so the single cnpg_pg route claims every CNPG pod without double-insert (keeps app_route mutually exclusive). Catch-all intact.
- Companion k8s flips in this PR: litellm `JSON_LOGS=True`; bind `querylog yes` on both bind-internal BindClusters; gitea router+access logging to stdout. Rancher auditLog was already on.
- 15 new `vector test` cases (routing + field extraction + authentik-postgres→cnpg exclusivity proof); all 35 green (vector 0.57). Fields go into the existing `fields Map(String,String)` — no DDL change.
Puppet-side follow-ups (out of scope for argocd): enable named query logging (profiles/dns/server.pp); ship the VM vector rollout with `.file`/`.SYSLOG_IDENTIFIER` tags for named/gitea/puppetserver(+multiline logback join)/postfix maillog.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #320
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The puppet MASTER is the singleton CA/master. A second master, even transiently during a rolling update, races on CA/cert signing and shared state (the CA lives on a shared PVC mounted by every master pod). The master was previously driven by an HPA with `minReplicas: 2`, `maxReplicas: 5` and a `RollingUpdate` strategy, so 2-5 masters could coexist normally and a rollout would briefly run old+new masters against the same CA data — a latent CA-corruption/split-brain bug. Recreate guarantees the old pod terminates before the new one starts, so two masters never coexist.
## Changes
- Set `puppetserver-master` `spec.replicas: 1` and `spec.strategy.type: Recreate` (drops RollingUpdate).
- Remove the `puppetserver-masters-autoscaler` HPA and its kustomization entry, which forced 2-5 master replicas and would otherwise override `replicas: 1`.
- Refresh the `puppetserver-master-vpa` note to reflect the pinned-singleton, no-HPA state (VPA stays `updateMode: Off`, recommendation-only).
The compiler (`puppetserver-compiler`) remains the horizontally-scalable tier with its own HPA — untouched. puppetdb/puppetboard untouched.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #341
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The re-keyed internal `unkin.net` intermediate broke CA consumers (CNPG->RGW backups, subPath/startup-cached CA mounts) and needed manual pod restarts, because Reloader was deployed with `ignoreSecrets: true` and could not restart on the `vault-ca-cert` Secret. Enabling secret watching naively is unsafe: many workloads carry the generic `reloader.stakater.com/auto`, and the estate rotates numerous Secrets via Vault/VSO — those would restart on every rotation. This enables secret watching but scopes existing `auto` to ConfigMaps, making secret-reload opt-in per Secret.
## Changes
- Set `reloader.ignoreSecrets: false` (au-syd1 reloader-system values) so Secrets are watched.
- Convert every generic `reloader.stakater.com/auto: "true"` to the ConfigMap-only `configmap.reloader.stakater.com/auto: "true"` — 22 annotations across 19 files. Existing ConfigMap-reload behaviour is preserved; Vault/VSO Secret rotations no longer restart these workloads.
- Add explicit `secret.reloader.stakater.com/reload: "vault-ca-cert"` to the CA consumers that mount the CA and carry a Reloader annotation: `artifactapi/api`, `cephrgw-operator`, `puppetserver-master`, `puppetserver-compiler`, `litellm`, `logarchiver`.
- Add `secret.reloader.stakater.com/reload: "kanidm-tls"` so kanidm rolls when cert-manager renews its leaf.
- Add `docs/ca-rotation.md` runbook (indexed in `docs/README.md`).
## Safety review (secret-only / CA workloads)
`vault-ca-cert` is a plain reflected Secret that bootstraps Vault trust (not VSO-rotated; changes only on intermediate re-key). `kanidm-tls` is a cert-manager leaf. Everything else mounted (`environment`, `*-credentials`, `eyaml-keys`, `puppetboard-secrets`, `s3-credentials`, `nats-auth`, `clickhouse-credentials`, `woodpecker-*`) is VSO/CNPG Vault-rotated and deliberately excluded.
- `cephrgw-operator` — mounts only Secrets (`cephrgw-credentials` VSO + `vault-ca-cert`), no ConfigMap. Its old comment said "restart when the credentials Secret rotates"; `cephrgw-credentials` is VSO so that is now excluded, and reload is scoped to `vault-ca-cert` only. Comment updated.
- `nats` (logging) — old comment "Roll the StatefulSet when nats-auth changes"; `nats-auth` is VSO, so this is now ConfigMap-only (deliberately no roll on rotation). Comment updated. Same for the vector agent/aggregator/vm-ingest (VSO `nats-auth`/`clickhouse-credentials`).
- `artifactapi/ui` — mounts neither a ConfigMap nor a Secret; its `auto` was already a no-op. Left as ConfigMap-only.
- `puppetdb` / `puppetboard` — mount a ConfigMap plus VSO Secrets (postgres creds / puppetboard-secrets); ConfigMap-only is correct, no secret reload added.
CA consumers that mount `vault-ca-cert` but have **no** Reloader annotation (CRD-managed or startup-cached) are documented in `docs/ca-rotation.md` for manual restart rather than annotated here: `grafana`, `observability/vmagent`, `paperclip`, `argocd-repo-server`, plus CNPG clusters (`kubectl cnpg restart`).
## Notes / coordination
- Annotations left in their existing location (some sit on the pod template, e.g. `litellm`, `puppetdb`; Reloader reads controller-level metadata — placement unchanged from before, no regression).
- Touches `apps/overlays/au-syd1/logging/values-vector-*.yaml`, which overlap open PR #320 (Tier-2 Vector pipelines) — only the one-line reloader annotation is changed here.
## Validation
- `make kubeconform` — touched overlays (reloader-system, logging, woodpecker, authentik) valid; only the known-unrelated cattle-system rancher chart kubeVersion failure remains.
- `uvx pre-commit run --all-files` — all hooks pass.
Closes#326
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #339
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
kea-0/kea-1 crash-looped after the dhcp-system deploy. Two root causes:
1. **kea-dhcp4** rejected the `ntp-servers` option (DHCP code 42) because that option carries IPv4 addresses only, but the KeaCluster supplied rotating `pool.ntp.org` hostnames (`DHCP4_CONFIG_LOAD_FAIL ... Failed to convert string to address '0.au.pool.ntp.org'`).
2. **kea-ctrl-agent/dhcp4** rejected the `/run/kea` unix socket path — kea 2.6.5 permits only `/var/run/kea` (exact-string check). Fixed in kea-operator v0.1.1 (`RunDir=/var/run/kea`).
- Remove `ntpServers` from the KeaCluster (not representable via DHCP option 42; add concrete NTP server IPs if ever needed).
- Bump kea-operator, kea, and kea-api images v0.1.0 -> v0.1.1 (socket-path fix).
kubeconform + pre-commit green.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #338
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
unkin.net is split-horizon: the `_acme-challenge.unkin.net -> _acme-challenge.acme.unkin.net` delegation CNAME exists only in the public Google Cloud DNS view. cert-manager's CNAME following (`cnameStrategy: Follow`) resolves via in-cluster CoreDNS to the nodes' internal resolver, which serves an internal view of unkin.net lacking that CNAME; Follow therefore finds no delegation and still sends the rfc2136 UPDATE to zone unkin.net on bind-external (only authoritative for acme.unkin.net), returning NOTAUTH. Follow needs a public-view resolver for both the CNAME chase and the propagation self-check. TSIG is proven fine.
- Set `acmeDNS01.recursiveNameservers` to `8.8.8.8:53` and `1.1.1.1:53` with `acmeDNS01.recursiveNameserversOnly: true` in the cert-manager ControllerConfiguration so DNS-01 resolution and self-checks use the public DNS view.
- Keeps `cnameStrategy: Follow` on the ClusterIssuers (merged in #331); this PR gives that following a resolver that can see the delegation.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #337
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Early-boot clients — anaconda/kickstart and yum in %post, PXE environments — need direct HTTP access to the artifactapi rpm repos. The current setup returns a 301 redirect from HTTP to HTTPS, which those minimal clients cannot follow (or downgrade insecurely), breaking rpm installs.
## Changes
- Attach the `api-route` HTTPRoute to the Gateway's `http` (port 80) listener alongside `https`, so `http://artifactapi.k8s.syd1.au.unkin.net/...` serves app content directly (200/40x from the app, no Location header).
- Remove the `http-redirect` HTTPRoute (RequestRedirect 301 `http`->`https`), which was the sole redirect mechanism — the traefik `web` entrypoint has no global `redirections`, so this is scoped strictly to artifactapi and does not affect other apps.
- Leave HTTPS unchanged.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #336
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Replaces the isc-dhcpd PXE-boot VM with the kea-operator + an HA kea pair, GitOps-managed. Deploys on a new, unused anycast IP so nothing is cut over yet; the production cutover off the current dhcpd address (198.18.19.18) is a separate later task.
- Add `apps/base/dhcp-system`: namespace, kea-operator RBAC + Deployment (v0.1.0), VPA, and the 4 kea.unkin.net CRDs pulled from the operator repo at tag v0.1.0.
- Add CRs translating the legacy dhcpd config (source: puppet `roles/infra/dhcp/server.yaml`): KeaCluster `kea` (2 replicas, hot-standby HA, main.unkin.net, 1200/86400 leases, AU ntp pool); five KeaSubnets 198.18.13-17.0/24 with .200-.220 pools, gateways .254 except .17->.1, next-server 198.18.19.19; Legacy/UEFI-64 PXE client classes; KeaAPI.
- DHCP-advertised DNS points at the in-cluster bind-resolvers cluster (PureLB 198.18.200.7), not the legacy 198.18.19.15 forwarder.
- Pin the DHCP LoadBalancer Service to the free common-pool IP 198.18.200.10 via PureLB.
- KeaAPI bearer token is operator-generated (no plain Secret committed).
- Commit generated kea.unkin.net JSON schemas for kubeconform; register dhcp-system in the platform ApplicationSet + AppProject.
Client-class object names are lowercased (`legacy`/`uefi-64`) to satisfy RFC1123 since the operator renders the kea class name from metadata.name.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #333
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
ServiceAccount `terraform-ipam` in the `woodpecker` namespace for the terraform-ipam pipeline. The Vault k8s auth role `woodpecker_terraform_ipam` (terraform-vault PR) binds it. Mirrors the other terraform-* CI ServiceAccounts and is wired into the woodpecker kustomization.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #334
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Vault's kubernetes secret engine will mint scoped tokens for a static \`agent-dns\` service account instead of generating cluster-wide RBAC, so agent DNS access is confined to exactly the bind namespaces. This is the GitOps half of the terraform-vault agent-dns role rework (PR unkin/terraform-vault#109). Ordering: this must sync before the Vault \`agent-dns\` creds are usable — Vault mints tokens for an SA that must already exist.
## How
- Add ServiceAccount \`agent-dns\` + ClusterRole \`agent-dns\` (definition only, no ClusterRoleBinding) in \`bind-system\`: full verbs on \`bind.unkin.net\` CRDs, get/list/watch pods/services/configmaps/events, get pods/log.
- Add RoleBinding \`agent-dns\` in each of \`bind-system\`, \`bind-internal\`, \`bind-external\`, \`externaldns\`, binding the SA to the ClusterRole in that namespace — confining all access (reads included) to those four namespaces.
Whitelist note: the platform AppProject already permits ClusterRole/ClusterRoleBinding and all four namespace destinations, so no project change is needed.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #332
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
A live DNS-01 smoke test returned NOTAUTH because the solver walked _acme-challenge.unkin.net to zone unkin.net and sent the rfc2136 UPDATE there, but bind-external is only authoritative for acme.unkin.net; without cnameStrategy: Follow the solver does not chase the delegation CNAME.
- Set `cnameStrategy: Follow` on the `letsencrypt` and `letsencrypt-staging` ClusterIssuer DNS-01 solvers so cert-manager follows the `_acme-challenge.unkin.net -> _acme-challenge.acme.unkin.net` CNAME and updates the `acme.unkin.net` zone.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #331
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
The new kea-operator repo's Woodpecker CI pipelines run under a dedicated Kubernetes ServiceAccount that must exist in the woodpecker namespace (cross-repo dependency; the .woodpecker/*.yaml steps set `serviceAccountName: kea-operator-ci`).
- Adds ServiceAccount `kea-operator-ci` in the `woodpecker` namespace
- Registers it in the woodpecker kustomization resources
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #330
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The \`vault-issuer\` ClusterIssuer is applied out-of-band (Helm release \`cert-manager-clusterissuer\`) and is referenced by ~15 Gateways, but is not tracked in GitOps — so the live, load-bearing issuer is drift. This imports it so ArgoCD manages it. The committed spec matches the live object exactly (verified against \`kubectl get clusterissuer vault-issuer -o yaml\`), so adoption is a no-op.
## Changes
- Add \`apps/base/cert-manager/clusterissuer_vault-issuer.yaml\` capturing the live spec byte-faithfully: server \`https://vault.service.consul:8200\`, path \`pki_int/sign/servers_default\`, k8s auth mount \`/v1/auth/k8s/au/syd1\`, role \`cert_manager_issuer\`, serviceAccountRef \`cert-manager-vault-issuer\` (audience \`vault\`), caBundleSecretRef \`vault-ca-cert\`/\`ca.crt\`. Helm ownership labels/annotations kept so adoption produces zero diff.
- Register the manifest in the cert-manager base kustomization (inserted between \`clusterrolebinding.yaml\` and \`vmservicescrape.yaml\` to avoid the lines #327 touches).
## Depends on #327
ArgoCD can only adopt this resource once \`{group: cert-manager.io, kind: ClusterIssuer}\` is in the platform project \`clusterResourceWhitelist\`. That whitelist entry is added by #327, not here (to avoid a duplicate/conflicting change). **Merge #327 first.** There may be a small merge conflict with #327 in \`apps/base/cert-manager/kustomization.yaml\` (both append to the \`resources\` list); rebase on main after #327 merges.
## Note: SA name discrepancy (not fixed here — committing live spec unchanged)
The live issuer authenticates as SA **\`cert-manager-vault-issuer\`**, but the repo scaffolding \`serviceaccount.yaml\` creates SA **\`vault-issuer\`**. Both SAs exist live in \`cert-manager\` (192d and 136d). The issuer uses \`cert-manager-vault-issuer\`, so this PR commits that name (live truth). The repo-managed \`vault-issuer\` SA appears unused by this issuer — worth a follow-up to reconcile which SA is canonical, but out of scope for a zero-change import.
https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #328
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
We self-delegate `_acme-challenge.unkin.net` into an `acme.unkin.net` zone we serve ourselves, so cert-manager can solve Let's Encrypt DNS-01 over RFC2136/TSIG. That needs a publicly-reachable authoritative BIND, separate from the internal estate.
- Add app `bind-external` (base + au-syd1 overlay); register it in the platform ApplicationSet and AppProject destinations (bind-operator already watches all namespaces).
- Add BindCluster `bind-external`: authoritative-only, recursion off, no forwarding, transfers denied except the keyed catalog/zone AXFR; 2 replicas; primaryService is a dmz-pinned PureLB LoadBalancer at `198.18.199.53`.
- Add BindZone `acme.unkin.net` (primary, dynamicUpdate) and BindTSIGKey `certmanager` (hmac-sha256), whose Secret `certmanager-tsig` reflects into the `cert-manager` namespace for the rfc2136 solver.
Pairs with argocd-apps #327 (the ClusterIssuers) and a one-time Google Cloud DNS delegation + NAT of the public IP :53 to `198.18.199.53`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #329
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Publicly-trusted wildcard certs via Let's Encrypt DNS-01, solved over RFC2136/TSIG against our own BIND. A one-time CNAME self-delegates `_acme-challenge.unkin.net` into the `acme.unkin.net` zone served by bind-external; cert-manager writes the challenge TXT there. No GCP/clouddns and no Vault secret involved. The existing `vault-issuer` (internal PKI) is untouched.
- Add ClusterIssuers `letsencrypt` (prod) and `letsencrypt-staging`, both using a dns01 rfc2136 solver: nameserver `198.18.199.53:53`, key `certmanager`, HMACSHA256, `tsigSecretSecretRef` -> reflected Secret `certmanager-tsig` key `secret`.
- Whitelist `cert-manager.io ClusterIssuer` in the platform AppProject.
Depends on #329 (bind-external: the acme.unkin.net zone, the certmanager TSIG key reflected into cert-manager, and the 198.18.199.53 nameserver) and on the one-time Google Cloud DNS delegation + NAT of the public IP :53 to 198.18.199.53. Earlier clouddns/Vault commits on this branch are reverted.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #327
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Why: the logging aggregator wrote every event through the generic catch-all shape. The Tier-1 survey picked six high-value log sources that warrant structured parsing into logs.raw columns/fields for real querying.
How:
- Two-stage routing in `aggregator.yaml`: `app_route` peels off the six Tier-1 streams by subject / VM source tag (mutually exclusive — no double-insert); everything else falls through `app_route._unmatched` to the unchanged generic k8s/vm catch-all.
- Six parse transforms emit the full `logs.raw` shape plus structured `.fields` (Map(String,String) — no DDL change): authentik (JSON), traefik (JSON access), vault audit (JSON), nginx access+error (regex), haproxy httplog (regex), glauth (JSON).
- Companion flip: traefik-system access logs to `format: json` (both overlays) so `traefik_parse` has structured input.
- 15 new `vector test` cases (routing + field extraction) in `aggregator-tests.yaml`; all green locally (vector 0.57).
Live now: authentik + traefik (k8s). Awaiting the puppet-side vector rollout (logs.vm.* with `.file`/`.SYSLOG_IDENTIFIER` tags per the documented convention): vault-file, nginx, haproxy, glauth — transforms are present and unit-tested so they light up automatically.
Note: geoip enrichment for nginx/traefik client IPs is a separate prerequisite — no enrichment table exists in the aggregator yet; these transforms extract `client_ip` ready for it.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #318
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Points the new gitea at its own k8s route so it can be exercised (login, browse, OIDC callback) before the git.unkin.net data cutover — ROOT_URL currently resolves to the live VM forge, which would break links on the k8s route. Flips back to git.unkin.net at cutover.
- set gitea DOMAIN/ROOT_URL/SSH_DOMAIN to git.k8s.syd1.au.unkin.net
Reviewed-on: #317
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Move the estate's own service/operator image pulls off the Gitea container registry (git.unkin.net/unkin) to the artifactapi local docker registry (docker-internal), ahead of the git.unkin.net forge migration which disables Gitea's container registry. The images were copied digest-for-digest into docker-internal and pulls verified before this repoint.
- repoint age-api, bind-operator, bind-tsig-api, cephrgw-operator, encapi, logarchiver, pdbmux image pulls to artifactapi.k8s.syd1.au.unkin.net/docker-internal
Deliberately not repointed here: artifactapi's own api/ui images (circular — it can't pull itself from itself), and the almalinux9-* base/CI images (huge, and their per-repo Woodpecker push targets move in a batched follow-up). Forge raw-CRD URLs and git clone sources are unaffected (those stay on the forge).
Reviewed-on: #310
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md.
- add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute
- add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222
- register gitea in the platform ApplicationSet + AppProject
- add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback)
Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc.
Reviewed-on: #309
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Final step of the ec migration: the old buckets were purged in #315, so the operator will now create fresh ones on the ec placement target. Restores the nine Bucket CRs with placementTarget: ec and retainOnDelete: true (purge disabled again for safety).
- re-add the nine cnpg backup Bucket resources on ec, retainOnDelete: true
Reviewed-on: #316
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Step 2 of the ec placement migration: with purge-on-delete now allowed (#313), removing the Bucket CRs makes the operator delete the underlying RGW buckets and their objects, freeing the names to be recreated on ec in step 3.
- remove the nine cnpg backup Bucket resources
- keep ObjectStoreUser and ScheduledBackup so the backup-s3 secrets and schedules survive
Reviewed-on: #315
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Authentik is canonical at https://identity.unkin.net, served by the internal
unkin.net CA. Grafana, LiteLLM and NetBox failed OIDC discovery because their
images don't trust that CA (x509: unknown authority); NetBox also still pointed
at the secondary admin host.
- grafana: mount the reflected vault-ca-cert; set generic_oauth `tls_client_ca`.
- litellm: `combine-certs` init builds public+internal CA bundle; `SSL_CERT_FILE`
+ `REQUESTS_CA_BUNDLE` point at it.
- netbox: flip OIDC issuer to identity.unkin.net; same combine bundle for
python-social-auth (`requests`).
- docs: record the Rancher manual runtime step (issuer + CA in the auth config).
Validated: kustomize build + kubeconform + pre-commit.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #314
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Step 1 of moving the CNPG backup buckets to ec placement: RGW can't move an existing bucket, so they must be dropped and recreated. This lets the operator actually delete the buckets (with their objects) when the CRs are removed in step 2.
- set retainOnDelete: false and purgeOnDelete: true on all nine cnpg backup Bucket CRs
- leave ObjectStoreUser/BucketAccess untouched so the backup-s3 secrets persist
Reviewed-on: #313
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
cephrgw-operator v0.4.0 adds immutable placementTarget selection on Buckets; the nine CNPG backup buckets should live on the ec (4/1) placement instead of 3-replica. Existing buckets cannot change placement, so after this merges the buckets get deleted and recreated on ec and fresh base backups are triggered (day-old backups are accepted losses, per Ben).
- bump cephrgw-operator image and CRD ref to v0.4.0
- add placementTarget: ec to all nine cnpg backup Bucket CRs
Reviewed-on: #312
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Final-mile bringup: after #301/#306/#308 the auth chain was fixed and logs flowed, but the `nats-bootstrap` PostSync hook **failed** with:
```
nats: error: could not pick a Stream to operate on: ... could not load schema { ... }: stat .: permission denied
```
The nats CLI stats its **working directory** when loading response-validation schemas. Under the Job's `readOnlyRootFilesystem: true` + `runAsUser: 1000`, the nats-box image's default WORKDIR isn't accessible to uid 1000, so every `nats stream/consumer` call errored. (A throwaway pod using default securityContext worked, which is why manual stream creation succeeded.)
Consequence: the PostSync hook never completes → `logging-logging` stays **OutOfSync**. The `LOGS` stream + consumers persist in JetStream once created, so log flow is unaffected — but GitOps convergence is blocked and the hook would keep retrying.
## What
Set `workingDir: /tmp` on the bootstrap container (the writable emptyDir already mounted for `HOME`). The nats CLI can then stat/operate normally.
**Verified on the live cluster:** a nats-box pod with the Job's exact restrictive securityContext + `workingDir: /tmp` runs `nats stream info LOGS` cleanly (fails without it).
## Note (separate, pre-existing)
There is also a first-deploy ordering deadlock: the `nats-bootstrap` PostSync hook runs only after the Sync-phase resources are healthy, but the vector consumer Deployments can't become healthy until the hook creates the `LOGS` stream. On this deploy I broke the deadlock by creating the stream/consumers manually (idempotent with the Job); the stream now persists so it won't recur on normal re-syncs, but a fresh cluster / PVC loss would hit it again. A durable fix (sync-waves so bootstrap runs after NATS but before the consumers) is worth a follow-up — flagged, not included here to keep this fix minimal.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #311
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Vector archiver leg wrote gzip NDJSON to S3 with no index or encryption. logarchiver replaces it with a Go service that seals raw logs to S3 as zstd + OpenPGP objects and indexes each object in ClickHouse (`logs.archive_index`), acking JetStream only after the object is stored and indexed.
## Changes
- Add logarchiver Deployment (`git.unkin.net/unkin/logarchiver:v0.1.0`), ConfigMap, and dedicated ServiceAccount, reusing the archiver's NATS (`log-consumer` / durable `archiver` / `ARCHIVE_SUBJECTS=logs.k8s.vault.>`), S3 (`logs-archive-s3`), ClickHouse (`clickhouse-credentials`) and `vault-ca` wiring.
- Encrypts to the `logarchive` gpg public key, fetched from the gpg engine via k8s auth (role `logging_logarchiver`, projected vault-audience token). `ack_wait` (5m) > batch `max_age` (2m) so messages aren't redelivered mid-batch.
- Add `logs.archive_index` DDL to the clickhouse-schema bootstrap Job (no TTL — outlives `logs.raw`).
- Remove the vector-archiver Helm release, values and pipeline ConfigMap.
Cross-repo: apply **terraform-vault #106** (gpg key + role/policy) before this syncs, or the pod can't fetch the public key. Sequencing: apply after #306 (already merged).
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #308
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
After #301 merged, the stack was still broken. Live diagnosis found the **actual** NATS auth root cause (my earlier interpolation fix in #301 was necessary but not sufficient).
### Evidence
- Every NATS client failed with `authorization violation`: the `nats-bootstrap` PostSync Job hung 30 min as `log-admin` then failed `DeadlineExceeded` (its `until nats account info` loop never authenticated), and `vector-aggregator`/`vector-archiver` crash-looped.
- The nats-0 container env **matched** the Vault secret exactly (all three password SHAs), yet auth was rejected.
- **Decisive test:** authenticating as `log-admin` with the **literal string** `$NATS_ADMIN_PASSWORD` **succeeded** — proving the server stored the passwords **un-expanded**.
### Root cause
The nats chart renders `config.merge` as JSON, so a plain `password: $NATS_ADMIN_PASSWORD` becomes the quoted literal `"$NATS_ADMIN_PASSWORD"` in `nats.conf`, and **NATS does not expand variables inside quoted strings**. Per the chart README, env vars must be wrapped in `<< $VAR >>` to render **unquoted** so NATS expands them.
## What
Wrap all three user passwords in `<< >>`:
```
password: << $NATS_ADMIN_PASSWORD >> # (+ producer, consumer)
```
Rendered `nats.conf` now emits `"password": $NATS_ADMIN_PASSWORD` (unquoted).
This is the **server-side** half; **#301** (merged) fixed the **client-side** half (Vector 0.57 needs `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION` to send the real password). Both are required — with both, server-expanded password == vector-interpolated password.
## Verified end-to-end
nats-server with unquoted `$VAR` config + env, plus vector with the interpolation flag: admin `account info` OK, `LOGS` stream + `transform` consumer created, and the vector consumer connects successfully.
## Expected recovery after merge + sync
1. `nats-config` CM updates → the config-reloader reloads NATS with the **real** (expanded) passwords.
2. The stuck `logging-logging` sync retries; the Sync phase applies #301's vector env + this config.
3. `nats-bootstrap` PostSync hook now authenticates as admin → creates the `LOGS` stream + `transform`/`archiver` consumers → sync completes.
4. Vector pods roll with interpolation enabled → producers publish, aggregator/archiver bind their durable consumers and write to ClickHouse / S3.
5. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` rises.
ClickHouse itself is already healthy (chi-logs Running, schema Job Complete) thanks to #301's watchNamespaces fix.
## Validation
kustomize + kubeconform clean (logging 40); rendered `nats.conf` shows unquoted `$VAR`; pre-commit clean.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #306
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
PR #296 merged and deployed, but the stack came up partially broken. Diagnosed live (cluster-admin) — two independent root causes, both fixed here.
## Root cause 1 — Vector env-var interpolation is off by default in 0.57
`vector-aggregator` and `vector-archiver` crash-looped with `async_nats::connector: authorization violation`; `vector-agent` / `vector-vm-ingest` were "Running" but silently failing to publish.
Diagnosis (evidence):
- The NATS server config **does** expand `$NATS_*_PASSWORD` (a `nats` CLI login with the real secret value authenticated fine), and the ACL was **not** the problem (a local repro with the narrow ACL + no stream connects cleanly and returns "stream not found", not an auth violation).
- The failure reproduces locally: a hardcoded password connects; the **same value via `${NATS_CONSUMER_PASSWORD}` fails**. Configuring the server to expect the literal string `${NATS_CONSUMER_PASSWORD}` makes Vector connect — proving **Vector sends the literal, un-interpolated string**.
- `vector --help` shows `--dangerously-allow-env-var-interpolation` — in 0.57 `${VAR}` interpolation is **opt-in**. An unset-var test confirms interpolation is off (no "unknown env var" error).
- Verified fix: with `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true` → **connects and authenticates**.
Every tier uses `${...}` for auth (`${NATS_*_PASSWORD}`, `${CLICKHOUSE_*}`), so the env var is added to **all four** vector deployments. (This slipped past CI because `vector test` never opens the NATS connection.)
## Root cause 2 — operator watches only its own namespace
`kubectl get chi -n logging` showed the `logs` CHI existed but with **empty status / no finalizer** — the operator never touched it, so the `logging-logging` Argo sync was stuck `Progressing` on *"waiting for healthy state of ClickHouseInstallation/logs"*, and the PostSync hooks (nats-bootstrap stream+consumers, clickhouse-schema) never ran (no stream → the consumers had nothing to bind even once auth is fixed).
Diagnosis: forcing an update event on the CHI produced zero operator reaction; a full operator restart didn't help. The Altinity chart README states `watchNamespaces: []` (our value) makes the operator **watch only its own namespace** (`clickhouse-system`). The CHI is in `logging`.
Fix: `watchNamespaces: ["logging"]` → operator config `watch.namespaces.include: [logging]`.
## Changes
- `apps/overlays/au-syd1/logging/values-vector-{agent,vm-ingest,aggregator,archiver}.yaml`: add `VECTOR_DANGEROUSLY_ALLOW_ENV_VAR_INTERPOLATION=true`.
- `apps/overlays/au-syd1/clickhouse-system/values.yaml`: `watchNamespaces: ["logging"]`.
No NATS ACL change (the original narrow ACL is correct). No secret/base changes.
## Expected recovery after merge + sync
1. clickhouse-system syncs → operator config gains `logging` → operator restarts → reconciles the `logs` CHI → CHI pod comes up healthy.
2. `logging-logging` sync unblocks → PostSync hooks run → JetStream `LOGS` stream + `transform`/`archiver` consumers created; `logs.raw` table created.
3. Vector pods roll with interpolation enabled → agents/vm-ingest authenticate and publish; aggregator/archiver authenticate, bind their durable consumers, and write to ClickHouse / S3.
4. Verify: `nats stream info LOGS` shows messages; `SELECT count() FROM logs.raw` increases.
## Validation
kustomize build + kubeconform clean (clickhouse-system 22, logging 40); operator config renders `watch.namespaces.include: [logging]`; all 4 vector deployments carry the interpolation env; pre-commit clean. The interpolation fix was verified end-to-end against a real nats-server (fails without the flag, connects with it).
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #301
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
CNPG WAL archiving to Ceph RGW fails with CERTIFICATE_VERIFY_FAILED on six clusters because the reflected vault-ca-cert bundle carries a corrupt intermediate: the genuinely-signed cert has a typo'd AIA URL (vault.servuce.consul), and the committed copy was text-edited at some point to fix the typo — flipping one byte of signed data and invalidating the signature (openssl verify: error 7 certificate signature failure). Only radosgw surfaces it because it serves a bare leaf, forcing clients to verify the stored intermediate against the root; services presenting their own intermediate never exercised the corrupt copy. terraform-k8s's copy is defunct per Ben — this file is the authoritative source.
- restore the original signed intermediate (one base64 character; sha256 E0:13:1B..., verified against the root, and the resulting bundle validates the live s3.ceph.unkin.net leaf)
- add an explicit allow-plain-secret marker mechanism to ci/validate-no-secrets.sh for public-data bootstrap secrets, and mark vault-ca-cert.yaml with it (a CA bundle is public and cannot be Vault-sourced since it establishes Vault trust)
After merge+sync the reflector propagates to all namespaces and barman's next retry (~1min) succeeds; base backups run on tonight's schedule. Follow-ups worth considering: re-issue the intermediate in Vault with a corrected AIA URL, and/or configure radosgw to serve its intermediate.
Reviewed-on: #305
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The CNPG buckets are empty after PR #298. Live diagnosis found **two** independent
causes; this PR fixes one of them.
`litellm`, `puppet` and `paperclip` run the CNPG `17-minimal-trixie` image, which
omits the `barman-cloud` CLI. In-tree `barmanObjectStore` archiving shells out to
`barman-cloud-wal-archive`, so their archiver dies immediately:
```
ContinuousArchiving=False :: unexpected failure invoking barman-cloud-wal-archive:
exec: "barman-cloud-wal-archive": executable file not found in $PATH
```
(verified on the live `puppet-postgres` primary: `which barman-cloud-wal-archive` →
not found; on a `-system` cluster it resolves to `/usr/local/bin/barman-cloud-wal-archive`).
## How
Switch those three clusters from `17-minimal-trixie` to `17-system-trixie` — the
`-system` variant already used by the other six clusters, which bundles the
barman-cloud tools. Tag confirmed present upstream (ghcr manifest HTTP 200).
```
- imageName: ghcr.io/cloudnative-pg/postgresql:17-minimal-trixie
+ imageName: ghcr.io/cloudnative-pg/postgresql:17-system-trixie
```
CNPG applies this as a rolling image update (switchover, no data change).
## Not fixed here (separate, primary blocker)
The other six clusters (full image, barman present) fail with a **TLS trust**
error — the reflected `vault-ca-cert` bundle carries a **stale intermediate CA**,
so barman can't verify `s3.ceph.unkin.net`:
```
SSL: CERTIFICATE_VERIFY_FAILED ... certificate signature failure
```
That is a shared trust-anchor refresh (likely owned by terraform-k8s /
`config/certificates/secret.yaml`, `managed-by: terragrunt`), handled separately —
it also gates litellm/puppet once they have barman. See the investigation report.
## Validation
- `kustomize build --enable-helm` + `kubeconform` pass on `litellm`, `puppet`
overlays and the `paperclip` base (paperclip has no overlay yet).
- `pre-commit run` passes on all changed files.
## Follow-ups
- Longer term, the Barman Cloud Plugin (sidecar) would let minimal images keep
their size while still archiving — track with the plugin migration.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #304
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
ArgoCD SSO fails with `failed to query provider "https://identity.unkin.net/application/o/argocd/": tls: failed to verify certificate: x509: certificate signed by unknown authority`. argocd-server does OIDC discovery to identity.unkin.net over TLS served by the internal `unkin.net` CA. Unlike argocd-repo-server (which mounts `vault-ca-cert`), argocd-server has no internal CA in its trust store and no `rootCA` in `oidc.config`, so it never trusted the issuer.
## Change
- argocd-cm `oidc.config`: add `rootCA` (inline PEM) = the internal `unkin.net` root CA. argocd-server hot-reloads argocd-cm, so no rollout restart is required.
## Why the root, not the cluster vault-ca-cert bundle
The `unkin.net Intermediate Authority` was recently **re-keyed** (same serial, new key: bundle SHA1 `C4:48:78…` vs served `F1:DD:34…`). The cluster `vault-ca-cert` bundle still carries the **stale** intermediate and fails `openssl verify` against the currently-served identity cert. identity.unkin.net presents its current intermediate in the handshake, so anchoring on the long-lived, stable `unkin.net` root (valid to 2034, matches the host trust anchor) is both correct and rotation-proof. Verified: `openssl verify -CAfile <root> -untrusted <served-intermediate> <served-leaf>` = OK; the embedded PEM round-trips through the YAML patch and validates the served leaf.
## Verify after merge
argocd-server picks up argocd-cm live; retest SSO login. (Separately, the cluster `vault-ca-cert` reflected secret carries a stale intermediate and should be refreshed, but that is out of scope here.)
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #303
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
2026-07-28 22:17:32 +10:00
223 changed files with 9751 additions and 309 deletions
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
unkin-agent
unkin-agent
unkinben