argocd-apps/docs/gitea-migration.md

Gitea VM → k8s migration

Replaces the Puppet-managed git.unkin.net VM (roles::infra::git::server, 3 nodes on shared /shared/apps/gitea + Patroni Postgres + Redis) with the gitea namespace here. The forge is critical infra (Woodpecker, tea, terraform-git, teabot, ArgoCD repo sources, r10k), so cutover is staged with the VM left intact for rollback.

What's deployed

Piece	Here	VM today
App	gitea chart 12.6.0, Gitea 1.26.2 rootless, 2 replicas	binary 1.26.2, 3 nodes
Repo/LFS storage	RWX PVC gitea-shared-storage (cephfs-raid6-delete, 20Gi)	/shared/apps/gitea
DB	CNPG gitea-postgres (2 instances) + pgbouncer, S3 backup cnpg-gitea 04:00/30d	Patroni-shared Postgres
Cache/session/queue	standalone Valkey (db 0/1/2, AOF)	Redis sentinel
SSH	disabled (DISABLE_SSH) — HTTPS clones only	host sshd, port 2222
Ingress	Gateway API → git.unkin.net + git.k8s.syd1.au.unkin.net	HAProxy be_gitea → git.unkin.net
Actions	disabled (CI stays Woodpecker)	runner deployed separately
Container registry	disabled (moved to artifactapi docker-internal)	built-in, in use
Other package types	enabled (generic archives for *arr etc. still used)	built-in

HA note: this is the same shape the VM already runs (multiple app nodes on shared storage + external DB + external Redis), which the Gitea chart's docs/ha-setup.md endorses for replicaCount > 1. Genuine multi-replica HA, not single-replica failover.

One-time Vault seeds (before first sync)

# app DB user (CNPG bootstrap + Gitea)
vault kv put kv/kubernetes/namespace/gitea/default/postgres-credentials \
  username=gitea password="$(openssl rand -hex 24)"

# local admin fallback (survives OIDC cutover)
vault kv put kv/kubernetes/namespace/gitea/default/gitea-admin \
  username=benvin password="$(openssl rand -hex 24)" email=benvin@unkin.net

# Gitea internal secrets — pinned so all replicas match. At data cutover (stage 1/2)
# REPLACE these with the VM's app.ini [security] SECRET_KEY + INTERNAL_TOKEN so the
# restored DB's encrypted columns (2FA, mirrors, oauth) stay decryptable.
vault kv put kv/kubernetes/namespace/gitea/default/gitea-inner \
  SECRET_KEY="$(gitea generate secret SECRET_KEY)" \
  INTERNAL_TOKEN="$(gitea generate secret INTERNAL_TOKEN)"

# Authentik OIDC client — `client_secret` read by terraform-authentik; `key`/`secret`
# consumed by the chart's oauth login-source (existingSecret needs both keys).
S=$(openssl rand -hex 40)
vault kv put kv/kubernetes/namespace/gitea/default/oauth-credentials \
  client_secret="$S" key=gitea secret="$S"

Stages

0 — Validate on the admin route. Merge this PR + the terraform-authentik PR (gitea OIDC app). ArgoCD syncs the gitea app; external-dns publishes git.k8s.syd1.au.unkin.net → the gateway VIP (safe — new name, empty instance). git.unkin.net stays on the live VM until stage 2. Confirm on the admin route: local-admin login, OIDC login (Authentik → auto-register/link), create a test repo, HTTPS push/clone (SSH is disabled — HTTPS only), and a Woodpecker webhook fires against git.k8s.syd1.au.unkin.net.

1 — Data migration rehearsal. Gitea is already Postgres→Postgres, so no DB engine conversion — a plain SQL dump/restore plus a filesystem copy of the data dir. On a VM node:

# repos + LFS + attachments + avatars (the /shared/apps/gitea tree, minus the DB)
gitea dump -c /etc/gitea/app.ini --skip-db --file /tmp/gitea-data.zip     # or rsync /shared/apps/gitea
pg_dump -h master.patroni-shared-prod.service.au-syd1.consul -U gitea gitea > /tmp/gitea.sql

Restore into k8s:

# DB
kubectl -n gitea exec -i gitea-postgres-1 -- psql -U gitea gitea < /tmp/gitea.sql
# data dir into the shared PVC (via a throwaway pod mounting gitea-shared-storage)
kubectl -n gitea cp /shared/apps/gitea/repos <helper-pod>:/data/git/repositories
kubectl -n gitea cp /shared/apps/gitea/lfs   <helper-pod>:/data/git/lfs

Then set gitea-inner to the VM's real SECRET_KEY/INTERNAL_TOKEN (above) and restart the deployment. Verify repos/issues/LFS/users. Rehearse fully before the freeze so the freeze window is just a delta.

2 — Freeze + cutover (the git.unkin.net DNS flip). Put the VM read-only (stop Woodpecker/webhooks), take a final dump/restore of the delta. The Gateway already serves git.unkin.net (listener + cert SAN) and the app already advertises ROOT_URL https://git.unkin.net/, so cutover is only the DNS flip:

• Repoint git.unkin.net → the internal Traefik VIP 198.18.200.4.
• Where the record lives (reconciled): git.unkin.net resolves TODAY via the Puppet DNS master (profiles::dns::master, records from PuppetDB exported resources) to the HAProxy VRRP VIP 198.18.19.17 → the VM. The GitOps home for the unkin.net apex is the k8s bind-operator unkin-net BindZone (apps/base/bind-internal/authoritative/), where identity.unkin.net already lives as a DNSRecord — but that zone currently holds only SOA+NS + a few records (the bulk apex migration off Puppet is a pending follow-up). So at cutover, flip whichever is the live authority for unkin.net at that moment: either uncomment the git-dns-internal DNSRecord in records.yaml (prepared, gated) if the k8s bind cluster is live, or change the Puppet-side record. Not external-dns — it owns only the k8s.syd1.au.unkin.net zone.

3 — Consumer verification. After DNS/cert flips:

Consumer	Check / knob
Woodpecker	forge URL (WOODPECKER_GITEA_URL) still https://git.unkin.net; OAuth app client id/secret unchanged; forge_remote_id per-repo is stable only if the Gitea repo IDs are preserved by the DB restore — a full SQL restore keeps them, a re-migration would not. Verify a webhook + a build.
tea	~/.config/tea/config.yml host unchanged; token still valid (tokens are in the restored DB).
terraform-git provider	base_url https://git.unkin.net, admin token from restored DB works.
teabot / vault-plugin-secrets-gitea	admin user+password (Basic-Auth API) — the rotate-root password is in Vault, and the admin local user must exist post-restore. Re-run rotate-root if needed.
ArgoCD repo creds	repoURL https://git.unkin.net/unkin/argocd-apps unchanged; the repo credential secret still valid.
r10k / puppet	r10k_repo https://git.unkin.net/unkin/puppet-r10k.git unchanged.
Container registry	pulls already moved to artifactapi docker-internal (see the registry-migration PR). Nothing should still pull git.unkin.net/unkin/<image>.

4 — Rollback. DNS git.unkin.net back to 198.18.19.17, un-freeze the VM. The VM was never modified, so rollback is a single record change.

Open decisions / hard problems

• SSH is disabled (DISABLE_SSH) — HTTPS clones only, the estate norm. No SSH VIP, no SSH DNS, nothing to cut over.
• artifactapi's own image is circular — it currently pulls git.unkin.net/unkin/ artifactapi. It can't pull itself from docker-internal. Keep it on an external/bootstrap-safe registry (or node-preloaded) before the VM registry goes away; not repointed in the registry-migration PR.
• forge_remote_id stability (Woodpecker) hinges on preserving Gitea repo IDs — guaranteed by a full SQL restore, broken by any re-create/re-migrate path.
• Cutover window: freeze = final delta dump/restore + DNS TTL. Lower the git.unkin.net record TTL (600s in both zones) a day ahead so the flip is quick.