argocd-apps/docs/gitea-migration.md

Gitea VM → k8s migration

Replaces the Puppet-managed git.unkin.net VM (roles::infra::git::server, 3 nodes on shared /shared/apps/gitea + Patroni Postgres + Redis) with the gitea namespace here. The forge is critical infra (Woodpecker, tea, terraform-git, teabot, ArgoCD repo sources, r10k), so cutover is staged with the VM left intact for rollback.

What's deployed

Piece	Here	VM today
App	gitea chart 12.6.0, Gitea 1.26.2 rootless, 2 replicas	binary 1.26.2, 3 nodes
Repo/LFS storage	RWX PVC gitea-shared-storage (cephfs-raid6-delete, 20Gi)	/shared/apps/gitea
DB	CNPG gitea-postgres (2 instances) + pgbouncer, S3 backup cnpg-gitea 04:00/30d	Patroni-shared Postgres
Cache/session/queue	standalone Valkey (db 0/1/2, AOF)	Redis sentinel
SSH	LoadBalancer VIP 198.18.200.10:2222 (built-in per-pod SSH server)	host sshd, port 2222
Ingress	Gateway API → git2.k8s.syd1.au.unkin.net (temp)	HAProxy be_gitea → git.unkin.net
Actions	disabled (CI stays Woodpecker)	runner deployed separately
Container registry	disabled (moved to artifactapi docker-internal)	built-in, in use
Other package types	enabled (generic archives for *arr etc. still used)	built-in

HA note: this is the same shape the VM already runs (multiple app nodes on shared storage + external DB + external Redis), which the Gitea chart's docs/ha-setup.md endorses for replicaCount > 1. Genuine multi-replica HA, not single-replica failover.

One-time Vault seeds (before first sync)

# app DB user (CNPG bootstrap + Gitea)
vault kv put kv/kubernetes/namespace/gitea/default/postgres-credentials \
  username=gitea password="$(openssl rand -hex 24)"

# local admin fallback (survives OIDC cutover)
vault kv put kv/kubernetes/namespace/gitea/default/gitea-admin \
  username=benvin password="$(openssl rand -hex 24)" email=benvin@unkin.net

# Gitea internal secrets — pinned so all replicas match. At data cutover (stage 1/2)
# REPLACE these with the VM's app.ini [security] SECRET_KEY + INTERNAL_TOKEN so the
# restored DB's encrypted columns (2FA, mirrors, oauth) stay decryptable.
vault kv put kv/kubernetes/namespace/gitea/default/gitea-inner \
  SECRET_KEY="$(gitea generate secret SECRET_KEY)" \
  INTERNAL_TOKEN="$(gitea generate secret INTERNAL_TOKEN)"

# Authentik OIDC client — `client_secret` read by terraform-authentik; `key`/`secret`
# consumed by the chart's oauth login-source (existingSecret needs both keys).
S=$(openssl rand -hex 40)
vault kv put kv/kubernetes/namespace/gitea/default/oauth-credentials \
  client_secret="$S" key=gitea secret="$S"

Stages

0 — Validate on the temp host. Merge this PR + the terraform-authentik PR (gitea OIDC app). ArgoCD syncs the gitea app. Confirm: git2.k8s.syd1.au.unkin.net serves, local-admin login, OIDC login (Authentik → auto-register/link), create a test repo, HTTPS push/clone, SSH push/clone via 198.18.200.10:2222, and a Woodpecker webhook fires against the temp host.

1 — Data migration rehearsal. Gitea is already Postgres→Postgres, so no DB engine conversion — a plain SQL dump/restore plus a filesystem copy of the data dir. On a VM node:

# repos + LFS + attachments + avatars (the /shared/apps/gitea tree, minus the DB)
gitea dump -c /etc/gitea/app.ini --skip-db --file /tmp/gitea-data.zip     # or rsync /shared/apps/gitea
pg_dump -h master.patroni-shared-prod.service.au-syd1.consul -U gitea gitea > /tmp/gitea.sql

Restore into k8s:

# DB
kubectl -n gitea exec -i gitea-postgres-1 -- psql -U gitea gitea < /tmp/gitea.sql
# data dir into the shared PVC (via a throwaway pod mounting gitea-shared-storage)
kubectl -n gitea cp /shared/apps/gitea/repos <helper-pod>:/data/git/repositories
kubectl -n gitea cp /shared/apps/gitea/lfs   <helper-pod>:/data/git/lfs

Then set gitea-inner to the VM's real SECRET_KEY/INTERNAL_TOKEN (above) and restart the deployment. Verify repos/issues/LFS/users. Rehearse fully before the freeze so the freeze window is just a delta.

2 — Freeze + cutover. Put the VM read-only (stop Woodpecker/webhooks), take a final dump/restore of the delta. Then:

• Add git.unkin.net to the Gateway: append it as a second listener hostname + cert SAN (cert-manager.io/common-name stays, add the host to the listeners and a matching HTTPRoute), and set gitea.config.server.{DOMAIN,ROOT_URL,SSH_DOMAIN} back to git.unkin.net.
• Repoint DNS git.unkin.net → the internal Traefik VIP 198.18.200.4. Record owner: git.unkin.net currently resolves to the HAProxy VRRP anycast VIP 198.18.19.17 (roles::infra::halb::haproxy2 anycast_ip); the A record lives in the main.unkin.net zone (Puppet DNS roles::infra::dns::master). Update it there (or via the bind estate) — not external-dns, which only owns the *.k8s.… records.

3 — Consumer verification. After DNS/cert flips:

Consumer	Check / knob
Woodpecker	forge URL (WOODPECKER_GITEA_URL) still https://git.unkin.net; OAuth app client id/secret unchanged; forge_remote_id per-repo is stable only if the Gitea repo IDs are preserved by the DB restore — a full SQL restore keeps them, a re-migration would not. Verify a webhook + a build.
tea	~/.config/tea/config.yml host unchanged; token still valid (tokens are in the restored DB).
terraform-git provider	base_url https://git.unkin.net, admin token from restored DB works.
teabot / vault-plugin-secrets-gitea	admin user+password (Basic-Auth API) — the rotate-root password is in Vault, and the admin local user must exist post-restore. Re-run rotate-root if needed.
ArgoCD repo creds	repoURL https://git.unkin.net/unkin/argocd-apps unchanged; the repo credential secret still valid.
r10k / puppet	r10k_repo https://git.unkin.net/unkin/puppet-r10k.git unchanged.
Container registry	pulls already moved to artifactapi docker-internal (see the registry-migration PR). Nothing should still pull git.unkin.net/unkin/<image>.

4 — Rollback. DNS git.unkin.net back to 198.18.19.17, un-freeze the VM. The VM was never modified, so rollback is a single record change.

Open decisions / hard problems

• SSH on git.unkin.net:2222. One A record can't point at both the HTTPS gateway VIP (.4) and a separate SSH VIP. Options: (a) HTTPS-only clones (estate norm, SSH dropped); (b) a Traefik TCPRoute so .4:2222 reaches the SSH service (share one VIP); (c) keep SSH on a dedicated host/VIP. Validation uses (c) via git2-ssh.k8s.…:2222.
• artifactapi's own image is circular — it currently pulls git.unkin.net/unkin/ artifactapi. It can't pull itself from docker-internal. Keep it on an external/bootstrap-safe registry (or node-preloaded) before the VM registry goes away; not repointed in the registry-migration PR.
• forge_remote_id stability (Woodpecker) hinges on preserving Gitea repo IDs — guaranteed by a full SQL restore, broken by any re-create/re-migrate path.
• Cutover window: freeze = final delta dump/restore + DNS TTL. Keep the main.unkin.net record TTL low a day ahead.