argocd-apps/docs/gitea-migration.md

# Gitea VM → k8s migration

Replaces the Puppet-managed `git.unkin.net` VM (`roles::infra::git::server`, 3 nodes
on shared `/shared/apps/gitea` + Patroni Postgres + Redis) with the `gitea`
namespace here. The forge is critical infra (Woodpecker, tea, terraform-git,
teabot, ArgoCD repo sources, r10k), so cutover is staged with the VM left intact
for rollback.

## What's deployed

| Piece | Here | VM today |
|---|---|---|
| App | gitea chart 12.6.0, Gitea **1.26.2** rootless, **2 replicas** | binary 1.26.2, 3 nodes |
| Repo/LFS storage | RWX PVC `gitea-shared-storage` (`cephfs-raid6-delete`, 20Gi) | `/shared/apps/gitea` |
| DB | CNPG `gitea-postgres` (2 instances) + pgbouncer, S3 backup `cnpg-gitea` 04:00/30d | Patroni-shared Postgres |
| Cache/session/queue | standalone Valkey (db 0/1/2, AOF) | Redis sentinel |
| SSH | **disabled** (`DISABLE_SSH`) — HTTPS clones only | host sshd, port 2222 |
| Ingress | Gateway API → `git.unkin.net` + `git.k8s.syd1.au.unkin.net` | HAProxy `be_gitea` → git.unkin.net |
| Actions | **disabled** (CI stays Woodpecker) | runner deployed separately |
| Container registry | **disabled** (moved to artifactapi `docker-internal`) | built-in, in use |
| Other package types | enabled (generic archives for *arr etc. still used) | built-in |

HA note: this is the **same shape the VM already runs** (multiple app nodes on
shared storage + external DB + external Redis), which the Gitea chart's
`docs/ha-setup.md` endorses for `replicaCount > 1`. Genuine multi-replica HA, not
single-replica failover.

## One-time Vault seeds (before first sync)

```sh
# app DB user (CNPG bootstrap + Gitea)
vault kv put kv/kubernetes/namespace/gitea/default/postgres-credentials \
  username=gitea password="$(openssl rand -hex 24)"

# local admin fallback (survives OIDC cutover)
vault kv put kv/kubernetes/namespace/gitea/default/gitea-admin \
  username=benvin password="$(openssl rand -hex 24)" email=benvin@unkin.net

# Gitea internal secrets — pinned so all replicas match. At data cutover (stage 1/2)
# REPLACE these with the VM's app.ini [security] SECRET_KEY + INTERNAL_TOKEN so the
# restored DB's encrypted columns (2FA, mirrors, oauth) stay decryptable.
vault kv put kv/kubernetes/namespace/gitea/default/gitea-inner \
  SECRET_KEY="$(gitea generate secret SECRET_KEY)" \
  INTERNAL_TOKEN="$(gitea generate secret INTERNAL_TOKEN)"

# Authentik OIDC client — `client_secret` read by terraform-authentik; `key`/`secret`
# consumed by the chart's oauth login-source (existingSecret needs both keys).
S=$(openssl rand -hex 40)
vault kv put kv/kubernetes/namespace/gitea/default/oauth-credentials \
  client_secret="$S" key=gitea secret="$S"
```

## Stages

**0 — Validate on the admin route.** Merge this PR + the terraform-authentik PR
(gitea OIDC app). ArgoCD syncs the `gitea` app; external-dns publishes
`git.k8s.syd1.au.unkin.net` → the gateway VIP (safe — new name, empty instance).
`git.unkin.net` stays on the live VM until stage 2. Confirm on the admin route:
local-admin login, OIDC login (Authentik → auto-register/link), create a test
repo, HTTPS push/clone (SSH is disabled — HTTPS only), and a Woodpecker webhook
fires against `git.k8s.syd1.au.unkin.net`.

**1 — Data migration rehearsal.** Gitea is already Postgres→Postgres, so **no DB
engine conversion** — a plain SQL dump/restore plus a filesystem copy of the data
dir. On a VM node:

```sh
# repos + LFS + attachments + avatars (the /shared/apps/gitea tree, minus the DB)
gitea dump -c /etc/gitea/app.ini --skip-db --file /tmp/gitea-data.zip     # or rsync /shared/apps/gitea
pg_dump -h master.patroni-shared-prod.service.au-syd1.consul -U gitea gitea > /tmp/gitea.sql
```

Restore into k8s:

```sh
# DB
kubectl -n gitea exec -i gitea-postgres-1 -- psql -U gitea gitea < /tmp/gitea.sql
# data dir into the shared PVC (via a throwaway pod mounting gitea-shared-storage)
kubectl -n gitea cp /shared/apps/gitea/repos <helper-pod>:/data/git/repositories
kubectl -n gitea cp /shared/apps/gitea/lfs   <helper-pod>:/data/git/lfs
```

Then set `gitea-inner` to the VM's real `SECRET_KEY`/`INTERNAL_TOKEN` (above) and
restart the deployment. Verify repos/issues/LFS/users. Rehearse fully before the
freeze so the freeze window is just a delta.

**2 — Freeze + cutover (the `git.unkin.net` DNS flip).** Put the VM read-only (stop
Woodpecker/webhooks), take a final dump/restore of the delta. The Gateway already
serves `git.unkin.net` (listener + cert SAN) and the app already advertises
`ROOT_URL https://git.unkin.net/`, so cutover is **only the DNS flip**:
- Repoint `git.unkin.net` → the internal Traefik VIP `198.18.200.4`.
- **Where the record lives (reconciled):** `git.unkin.net` resolves TODAY via the
  **Puppet DNS master** (`profiles::dns::master`, records from PuppetDB exported
  resources) to the HAProxy VRRP VIP `198.18.19.17` → the VM. The GitOps home for
  the `unkin.net` apex is the **k8s bind-operator** `unkin-net` BindZone
  (`apps/base/bind-internal/authoritative/`), where `identity.unkin.net` already
  lives as a DNSRecord — but that zone currently holds only SOA+NS + a few records
  (the bulk apex migration off Puppet is a pending follow-up). So at cutover, flip
  **whichever is the live authority for `unkin.net` at that moment**: either
  uncomment the `git-dns-internal` DNSRecord in `records.yaml` (prepared, gated) if
  the k8s bind cluster is live, or change the Puppet-side record. **Not**
  external-dns — it owns only the `k8s.syd1.au.unkin.net` zone.

**3 — Consumer verification.** After DNS/cert flips:

| Consumer | Check / knob |
|---|---|
| Woodpecker | forge URL (`WOODPECKER_GITEA_URL`) still `https://git.unkin.net`; OAuth app client id/secret unchanged; **`forge_remote_id` per-repo is stable** only if the Gitea repo IDs are preserved by the DB restore — a full SQL restore keeps them, a re-migration would not. Verify a webhook + a build. |
| tea | `~/.config/tea/config.yml` host unchanged; token still valid (tokens are in the restored DB). |
| terraform-git provider | `base_url https://git.unkin.net`, admin token from restored DB works. |
| teabot / vault-plugin-secrets-gitea | admin user+password (Basic-Auth API) — the rotate-root password is in Vault, and the admin **local** user must exist post-restore. Re-run rotate-root if needed. |
| ArgoCD repo creds | `repoURL https://git.unkin.net/unkin/argocd-apps` unchanged; the repo credential secret still valid. |
| r10k / puppet | `r10k_repo https://git.unkin.net/unkin/puppet-r10k.git` unchanged. |
| Container registry | pulls already moved to artifactapi `docker-internal` (see the registry-migration PR). Nothing should still pull `git.unkin.net/unkin/<image>`. |

**4 — Rollback.** DNS `git.unkin.net` back to `198.18.19.17`, un-freeze the VM. The
VM was never modified, so rollback is a single record change.

## Open decisions / hard problems

- **SSH is disabled** (`DISABLE_SSH`) — HTTPS clones only, the estate norm. No SSH
  VIP, no SSH DNS, nothing to cut over.
- **artifactapi's own image is circular** — it currently pulls `git.unkin.net/unkin/
  artifactapi`. It can't pull itself from `docker-internal`. Keep it on an
  external/bootstrap-safe registry (or node-preloaded) before the VM registry goes
  away; not repointed in the registry-migration PR.
- **`forge_remote_id` stability** (Woodpecker) hinges on preserving Gitea repo IDs —
  guaranteed by a full SQL restore, broken by any re-create/re-migrate path.
- **Cutover window**: freeze = final delta dump/restore + DNS TTL. Lower the
  `git.unkin.net` record TTL (600s in both zones) a day ahead so the flip is quick.