Add k8s Gitea deployment (migration target for git.unkin.net) (#309)
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md. - add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute - add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222 - register gitea in the platform ApplicationSet + AppProject - add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback) Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc. Reviewed-on: #309 Co-authored-by: Ben Vincent <ben@unkin.net> Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #309.
This commit is contained in:
@@ -0,0 +1,130 @@
|
||||
# Gitea VM → k8s migration
|
||||
|
||||
Replaces the Puppet-managed `git.unkin.net` VM (`roles::infra::git::server`, 3 nodes
|
||||
on shared `/shared/apps/gitea` + Patroni Postgres + Redis) with the `gitea`
|
||||
namespace here. The forge is critical infra (Woodpecker, tea, terraform-git,
|
||||
teabot, ArgoCD repo sources, r10k), so cutover is staged with the VM left intact
|
||||
for rollback.
|
||||
|
||||
## What's deployed
|
||||
|
||||
| Piece | Here | VM today |
|
||||
|---|---|---|
|
||||
| App | gitea chart 12.6.0, Gitea **1.26.2** rootless, **2 replicas** | binary 1.26.2, 3 nodes |
|
||||
| Repo/LFS storage | RWX PVC `gitea-shared-storage` (`cephfs-raid6-delete`, 20Gi) | `/shared/apps/gitea` |
|
||||
| DB | CNPG `gitea-postgres` (2 instances) + pgbouncer, S3 backup `cnpg-gitea` 04:00/30d | Patroni-shared Postgres |
|
||||
| Cache/session/queue | standalone Valkey (db 0/1/2, AOF) | Redis sentinel |
|
||||
| SSH | **disabled** (`DISABLE_SSH`) — HTTPS clones only | host sshd, port 2222 |
|
||||
| Ingress | Gateway API → `git.unkin.net` + `git.k8s.syd1.au.unkin.net` | HAProxy `be_gitea` → git.unkin.net |
|
||||
| Actions | **disabled** (CI stays Woodpecker) | runner deployed separately |
|
||||
| Container registry | **disabled** (moved to artifactapi `docker-internal`) | built-in, in use |
|
||||
| Other package types | enabled (generic archives for *arr etc. still used) | built-in |
|
||||
|
||||
HA note: this is the **same shape the VM already runs** (multiple app nodes on
|
||||
shared storage + external DB + external Redis), which the Gitea chart's
|
||||
`docs/ha-setup.md` endorses for `replicaCount > 1`. Genuine multi-replica HA, not
|
||||
single-replica failover.
|
||||
|
||||
## One-time Vault seeds (before first sync)
|
||||
|
||||
```sh
|
||||
# app DB user (CNPG bootstrap + Gitea)
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/postgres-credentials \
|
||||
username=gitea password="$(openssl rand -hex 24)"
|
||||
|
||||
# local admin fallback (survives OIDC cutover)
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/gitea-admin \
|
||||
username=benvin password="$(openssl rand -hex 24)" email=benvin@unkin.net
|
||||
|
||||
# Gitea internal secrets — pinned so all replicas match. At data cutover (stage 1/2)
|
||||
# REPLACE these with the VM's app.ini [security] SECRET_KEY + INTERNAL_TOKEN so the
|
||||
# restored DB's encrypted columns (2FA, mirrors, oauth) stay decryptable.
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/gitea-inner \
|
||||
SECRET_KEY="$(gitea generate secret SECRET_KEY)" \
|
||||
INTERNAL_TOKEN="$(gitea generate secret INTERNAL_TOKEN)"
|
||||
|
||||
# Authentik OIDC client — `client_secret` read by terraform-authentik; `key`/`secret`
|
||||
# consumed by the chart's oauth login-source (existingSecret needs both keys).
|
||||
S=$(openssl rand -hex 40)
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/oauth-credentials \
|
||||
client_secret="$S" key=gitea secret="$S"
|
||||
```
|
||||
|
||||
## Stages
|
||||
|
||||
**0 — Validate on the admin route.** Merge this PR + the terraform-authentik PR
|
||||
(gitea OIDC app). ArgoCD syncs the `gitea` app; external-dns publishes
|
||||
`git.k8s.syd1.au.unkin.net` → the gateway VIP (safe — new name, empty instance).
|
||||
`git.unkin.net` stays on the live VM until stage 2. Confirm on the admin route:
|
||||
local-admin login, OIDC login (Authentik → auto-register/link), create a test
|
||||
repo, HTTPS push/clone (SSH is disabled — HTTPS only), and a Woodpecker webhook
|
||||
fires against `git.k8s.syd1.au.unkin.net`.
|
||||
|
||||
**1 — Data migration rehearsal.** Gitea is already Postgres→Postgres, so **no DB
|
||||
engine conversion** — a plain SQL dump/restore plus a filesystem copy of the data
|
||||
dir. On a VM node:
|
||||
|
||||
```sh
|
||||
# repos + LFS + attachments + avatars (the /shared/apps/gitea tree, minus the DB)
|
||||
gitea dump -c /etc/gitea/app.ini --skip-db --file /tmp/gitea-data.zip # or rsync /shared/apps/gitea
|
||||
pg_dump -h master.patroni-shared-prod.service.au-syd1.consul -U gitea gitea > /tmp/gitea.sql
|
||||
```
|
||||
|
||||
Restore into k8s:
|
||||
|
||||
```sh
|
||||
# DB
|
||||
kubectl -n gitea exec -i gitea-postgres-1 -- psql -U gitea gitea < /tmp/gitea.sql
|
||||
# data dir into the shared PVC (via a throwaway pod mounting gitea-shared-storage)
|
||||
kubectl -n gitea cp /shared/apps/gitea/repos <helper-pod>:/data/git/repositories
|
||||
kubectl -n gitea cp /shared/apps/gitea/lfs <helper-pod>:/data/git/lfs
|
||||
```
|
||||
|
||||
Then set `gitea-inner` to the VM's real `SECRET_KEY`/`INTERNAL_TOKEN` (above) and
|
||||
restart the deployment. Verify repos/issues/LFS/users. Rehearse fully before the
|
||||
freeze so the freeze window is just a delta.
|
||||
|
||||
**2 — Freeze + cutover (the `git.unkin.net` DNS flip).** Put the VM read-only (stop
|
||||
Woodpecker/webhooks), take a final dump/restore of the delta. The Gateway already
|
||||
serves `git.unkin.net` (listener + cert SAN) and the app already advertises
|
||||
`ROOT_URL https://git.unkin.net/`, so cutover is **only the DNS flip**:
|
||||
- Repoint `git.unkin.net` → the internal Traefik VIP `198.18.200.4`.
|
||||
- **Where the record lives (reconciled):** `git.unkin.net` resolves TODAY via the
|
||||
**Puppet DNS master** (`profiles::dns::master`, records from PuppetDB exported
|
||||
resources) to the HAProxy VRRP VIP `198.18.19.17` → the VM. The GitOps home for
|
||||
the `unkin.net` apex is the **k8s bind-operator** `unkin-net` BindZone
|
||||
(`apps/base/bind-internal/authoritative/`), where `identity.unkin.net` already
|
||||
lives as a DNSRecord — but that zone currently holds only SOA+NS + a few records
|
||||
(the bulk apex migration off Puppet is a pending follow-up). So at cutover, flip
|
||||
**whichever is the live authority for `unkin.net` at that moment**: either
|
||||
uncomment the `git-dns-internal` DNSRecord in `records.yaml` (prepared, gated) if
|
||||
the k8s bind cluster is live, or change the Puppet-side record. **Not**
|
||||
external-dns — it owns only the `k8s.syd1.au.unkin.net` zone.
|
||||
|
||||
**3 — Consumer verification.** After DNS/cert flips:
|
||||
|
||||
| Consumer | Check / knob |
|
||||
|---|---|
|
||||
| Woodpecker | forge URL (`WOODPECKER_GITEA_URL`) still `https://git.unkin.net`; OAuth app client id/secret unchanged; **`forge_remote_id` per-repo is stable** only if the Gitea repo IDs are preserved by the DB restore — a full SQL restore keeps them, a re-migration would not. Verify a webhook + a build. |
|
||||
| tea | `~/.config/tea/config.yml` host unchanged; token still valid (tokens are in the restored DB). |
|
||||
| terraform-git provider | `base_url https://git.unkin.net`, admin token from restored DB works. |
|
||||
| teabot / vault-plugin-secrets-gitea | admin user+password (Basic-Auth API) — the rotate-root password is in Vault, and the admin **local** user must exist post-restore. Re-run rotate-root if needed. |
|
||||
| ArgoCD repo creds | `repoURL https://git.unkin.net/unkin/argocd-apps` unchanged; the repo credential secret still valid. |
|
||||
| r10k / puppet | `r10k_repo https://git.unkin.net/unkin/puppet-r10k.git` unchanged. |
|
||||
| Container registry | pulls already moved to artifactapi `docker-internal` (see the registry-migration PR). Nothing should still pull `git.unkin.net/unkin/<image>`. |
|
||||
|
||||
**4 — Rollback.** DNS `git.unkin.net` back to `198.18.19.17`, un-freeze the VM. The
|
||||
VM was never modified, so rollback is a single record change.
|
||||
|
||||
## Open decisions / hard problems
|
||||
|
||||
- **SSH is disabled** (`DISABLE_SSH`) — HTTPS clones only, the estate norm. No SSH
|
||||
VIP, no SSH DNS, nothing to cut over.
|
||||
- **artifactapi's own image is circular** — it currently pulls `git.unkin.net/unkin/
|
||||
artifactapi`. It can't pull itself from `docker-internal`. Keep it on an
|
||||
external/bootstrap-safe registry (or node-preloaded) before the VM registry goes
|
||||
away; not repointed in the registry-migration PR.
|
||||
- **`forge_remote_id` stability** (Woodpecker) hinges on preserving Gitea repo IDs —
|
||||
guaranteed by a full SQL restore, broken by any re-create/re-migrate path.
|
||||
- **Cutover window**: freeze = final delta dump/restore + DNS TTL. Lower the
|
||||
`git.unkin.net` record TTL (600s in both zones) a day ahead so the flip is quick.
|
||||
Reference in New Issue
Block a user