Add k8s Gitea deployment (HA-shaped, migration target for git.unkin.net)
Stand up the forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache): official Gitea chart 12.6.0 (app 1.26.2) at 2 replicas on RWX CephFS, CNPG Postgres with S3 backup, standalone Valkey for cache/session/queue, Authentik OIDC, Actions disabled and the container registry moved to artifactapi. Serves a temporary git2.k8s.syd1.au.unkin.net host; cutover is staged (see the doc). - add apps/base/gitea (namespace, CNPG cluster+backup+pooler, Valkey, VaultAuth, VaultStaticSecrets, Gateway, HTTPRoute) - add apps/overlays/au-syd1/gitea (chart 12.6.0 via helm-through-kustomize + values, drop the chart test Pod) - register gitea in the platform ApplicationSet and AppProject - add docs/gitea-migration.md staged cutover plan Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
@@ -7,3 +7,4 @@ Operational notes for the manifests in this repo.
|
||||
| [cnpg-backups.md](cnpg-backups.md) | How CNPG Postgres backups (WAL archiving + nightly base backups) to Ceph RGW are configured. |
|
||||
| [cnpg-restore.md](cnpg-restore.md) | Restoring a CNPG cluster: full recovery, point-in-time recovery, cutover, and gotchas. |
|
||||
| [authentik-rancher-sso.md](authentik-rancher-sso.md) | Manual runtime step to point Rancher's OIDC auth at the canonical `identity.unkin.net` issuer and trust the internal CA. |
|
||||
| [gitea-migration.md](gitea-migration.md) | Staged cutover of the git.unkin.net forge from the Puppet VM to the `gitea` namespace. |
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
# Gitea VM → k8s migration
|
||||
|
||||
Replaces the Puppet-managed `git.unkin.net` VM (`roles::infra::git::server`, 3 nodes
|
||||
on shared `/shared/apps/gitea` + Patroni Postgres + Redis) with the `gitea`
|
||||
namespace here. The forge is critical infra (Woodpecker, tea, terraform-git,
|
||||
teabot, ArgoCD repo sources, r10k), so cutover is staged with the VM left intact
|
||||
for rollback.
|
||||
|
||||
## What's deployed
|
||||
|
||||
| Piece | Here | VM today |
|
||||
|---|---|---|
|
||||
| App | gitea chart 12.6.0, Gitea **1.26.2** rootless, **2 replicas** | binary 1.26.2, 3 nodes |
|
||||
| Repo/LFS storage | RWX PVC `gitea-shared-storage` (`cephfs-raid6-delete`, 20Gi) | `/shared/apps/gitea` |
|
||||
| DB | CNPG `gitea-postgres` (2 instances) + pgbouncer, S3 backup `cnpg-gitea` 04:00/30d | Patroni-shared Postgres |
|
||||
| Cache/session/queue | standalone Valkey (db 0/1/2, AOF) | Redis sentinel |
|
||||
| SSH | LoadBalancer VIP `198.18.200.10:2222` (built-in per-pod SSH server) | host sshd, port 2222 |
|
||||
| Ingress | Gateway API → `git2.k8s.syd1.au.unkin.net` (temp) | HAProxy `be_gitea` → git.unkin.net |
|
||||
| Actions | **disabled** (CI stays Woodpecker) | runner deployed separately |
|
||||
| Container registry | **disabled** (moved to artifactapi `docker-internal`) | built-in, in use |
|
||||
| Other package types | enabled (generic archives for *arr etc. still used) | built-in |
|
||||
|
||||
HA note: this is the **same shape the VM already runs** (multiple app nodes on
|
||||
shared storage + external DB + external Redis), which the Gitea chart's
|
||||
`docs/ha-setup.md` endorses for `replicaCount > 1`. Genuine multi-replica HA, not
|
||||
single-replica failover.
|
||||
|
||||
## One-time Vault seeds (before first sync)
|
||||
|
||||
```sh
|
||||
# app DB user (CNPG bootstrap + Gitea)
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/postgres-credentials \
|
||||
username=gitea password="$(openssl rand -hex 24)"
|
||||
|
||||
# local admin fallback (survives OIDC cutover)
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/gitea-admin \
|
||||
username=benvin password="$(openssl rand -hex 24)" email=benvin@unkin.net
|
||||
|
||||
# Gitea internal secrets — pinned so all replicas match. At data cutover (stage 1/2)
|
||||
# REPLACE these with the VM's app.ini [security] SECRET_KEY + INTERNAL_TOKEN so the
|
||||
# restored DB's encrypted columns (2FA, mirrors, oauth) stay decryptable.
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/gitea-inner \
|
||||
SECRET_KEY="$(gitea generate secret SECRET_KEY)" \
|
||||
INTERNAL_TOKEN="$(gitea generate secret INTERNAL_TOKEN)"
|
||||
|
||||
# Authentik OIDC client — `client_secret` read by terraform-authentik; `key`/`secret`
|
||||
# consumed by the chart's oauth login-source (existingSecret needs both keys).
|
||||
S=$(openssl rand -hex 40)
|
||||
vault kv put kv/kubernetes/namespace/gitea/default/oauth-credentials \
|
||||
client_secret="$S" key=gitea secret="$S"
|
||||
```
|
||||
|
||||
## Stages
|
||||
|
||||
**0 — Validate on the temp host.** Merge this PR + the terraform-authentik PR
|
||||
(gitea OIDC app). ArgoCD syncs the `gitea` app. Confirm: `git2.k8s.syd1.au.unkin.net`
|
||||
serves, local-admin login, OIDC login (Authentik → auto-register/link), create a
|
||||
test repo, HTTPS push/clone, SSH push/clone via `198.18.200.10:2222`, and a
|
||||
Woodpecker webhook fires against the temp host.
|
||||
|
||||
**1 — Data migration rehearsal.** Gitea is already Postgres→Postgres, so **no DB
|
||||
engine conversion** — a plain SQL dump/restore plus a filesystem copy of the data
|
||||
dir. On a VM node:
|
||||
|
||||
```sh
|
||||
# repos + LFS + attachments + avatars (the /shared/apps/gitea tree, minus the DB)
|
||||
gitea dump -c /etc/gitea/app.ini --skip-db --file /tmp/gitea-data.zip # or rsync /shared/apps/gitea
|
||||
pg_dump -h master.patroni-shared-prod.service.au-syd1.consul -U gitea gitea > /tmp/gitea.sql
|
||||
```
|
||||
|
||||
Restore into k8s:
|
||||
|
||||
```sh
|
||||
# DB
|
||||
kubectl -n gitea exec -i gitea-postgres-1 -- psql -U gitea gitea < /tmp/gitea.sql
|
||||
# data dir into the shared PVC (via a throwaway pod mounting gitea-shared-storage)
|
||||
kubectl -n gitea cp /shared/apps/gitea/repos <helper-pod>:/data/git/repositories
|
||||
kubectl -n gitea cp /shared/apps/gitea/lfs <helper-pod>:/data/git/lfs
|
||||
```
|
||||
|
||||
Then set `gitea-inner` to the VM's real `SECRET_KEY`/`INTERNAL_TOKEN` (above) and
|
||||
restart the deployment. Verify repos/issues/LFS/users. Rehearse fully before the
|
||||
freeze so the freeze window is just a delta.
|
||||
|
||||
**2 — Freeze + cutover.** Put the VM read-only (stop Woodpecker/webhooks), take a
|
||||
final dump/restore of the delta. Then:
|
||||
- Add `git.unkin.net` to the Gateway: append it as a second listener hostname +
|
||||
cert SAN (`cert-manager.io/common-name` stays, add the host to the listeners and
|
||||
a matching HTTPRoute), and set `gitea.config.server.{DOMAIN,ROOT_URL,SSH_DOMAIN}`
|
||||
back to `git.unkin.net`.
|
||||
- Repoint DNS `git.unkin.net` → the internal Traefik VIP `198.18.200.4`. **Record
|
||||
owner:** `git.unkin.net` currently resolves to the HAProxy VRRP anycast VIP
|
||||
`198.18.19.17` (`roles::infra::halb::haproxy2 anycast_ip`); the A record lives in
|
||||
the `main.unkin.net` zone (Puppet DNS `roles::infra::dns::master`). Update it
|
||||
there (or via the bind estate) — **not** external-dns, which only owns the
|
||||
`*.k8s.…` records.
|
||||
|
||||
**3 — Consumer verification.** After DNS/cert flips:
|
||||
|
||||
| Consumer | Check / knob |
|
||||
|---|---|
|
||||
| Woodpecker | forge URL (`WOODPECKER_GITEA_URL`) still `https://git.unkin.net`; OAuth app client id/secret unchanged; **`forge_remote_id` per-repo is stable** only if the Gitea repo IDs are preserved by the DB restore — a full SQL restore keeps them, a re-migration would not. Verify a webhook + a build. |
|
||||
| tea | `~/.config/tea/config.yml` host unchanged; token still valid (tokens are in the restored DB). |
|
||||
| terraform-git provider | `base_url https://git.unkin.net`, admin token from restored DB works. |
|
||||
| teabot / vault-plugin-secrets-gitea | admin user+password (Basic-Auth API) — the rotate-root password is in Vault, and the admin **local** user must exist post-restore. Re-run rotate-root if needed. |
|
||||
| ArgoCD repo creds | `repoURL https://git.unkin.net/unkin/argocd-apps` unchanged; the repo credential secret still valid. |
|
||||
| r10k / puppet | `r10k_repo https://git.unkin.net/unkin/puppet-r10k.git` unchanged. |
|
||||
| Container registry | pulls already moved to artifactapi `docker-internal` (see the registry-migration PR). Nothing should still pull `git.unkin.net/unkin/<image>`. |
|
||||
|
||||
**4 — Rollback.** DNS `git.unkin.net` back to `198.18.19.17`, un-freeze the VM. The
|
||||
VM was never modified, so rollback is a single record change.
|
||||
|
||||
## Open decisions / hard problems
|
||||
|
||||
- **SSH on git.unkin.net:2222.** One A record can't point at both the HTTPS gateway
|
||||
VIP (`.4`) and a separate SSH VIP. Options: (a) HTTPS-only clones (estate norm,
|
||||
SSH dropped); (b) a Traefik `TCPRoute` so `.4:2222` reaches the SSH service
|
||||
(share one VIP); (c) keep SSH on a dedicated host/VIP. Validation uses (c) via
|
||||
`git2-ssh.k8s.…:2222`.
|
||||
- **artifactapi's own image is circular** — it currently pulls `git.unkin.net/unkin/
|
||||
artifactapi`. It can't pull itself from `docker-internal`. Keep it on an
|
||||
external/bootstrap-safe registry (or node-preloaded) before the VM registry goes
|
||||
away; not repointed in the registry-migration PR.
|
||||
- **`forge_remote_id` stability** (Woodpecker) hinges on preserving Gitea repo IDs —
|
||||
guaranteed by a full SQL restore, broken by any re-create/re-migrate path.
|
||||
- **Cutover window**: freeze = final delta dump/restore + DNS TTL. Keep the
|
||||
`main.unkin.net` record TTL low a day ahead.
|
||||
Reference in New Issue
Block a user