Add k8s Gitea deployment (migration target for git.unkin.net) (#309)

Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md.

- add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute
- add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222
- register gitea in the platform ApplicationSet + AppProject
- add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback)

Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc.

Reviewed-on: #309
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #309.
This commit is contained in:
2026-07-31 20:03:43 +10:00
committed by BenVincent
parent 6a13ca758a
commit a04dcc2975
19 changed files with 949 additions and 0 deletions
@@ -20,6 +20,32 @@ spec:
# identity.unkin.net hostname there.
- 198.18.200.4
---
# PRODUCTION CUTOVER RECORD — intentionally commented out.
# git.unkin.net currently resolves to the LIVE VM forge (HAProxy VRRP VIP
# 198.18.19.17), which holds every repo the estate depends on. Uncommenting this
# repoints the whole org's git.unkin.net at the new k8s Gitea gateway VIP, so it
# is the FINAL step of the forge migration — gated on the data migration (gitea
# dump/restore + SECRET_KEY copy) in argocd-apps docs/gitea-migration.md.
# NOTE: the live git.unkin.net answer is served by the puppet DNS master today
# (profiles::dns::master, records from PuppetDB); this k8s apex zone holds only
# SOA+NS + a few DNSRecords so far. Confirm the k8s bind cluster is the live
# authority for unkin.net (or update the puppet record instead) before relying
# on this CR at cutover.
# ---
# apiVersion: bind.unkin.net/v1alpha1
# kind: DNSRecord
# metadata:
# name: git-dns-internal
# namespace: bind-internal
# spec:
# zoneRef: unkin-net
# name: git
# type: A
# ttl: 600
# values:
# # traefik-internal gateway VIP; the gitea Gateway serves git.unkin.net there.
# - 198.18.200.4
---
apiVersion: bind.unkin.net/v1alpha1
kind: DNSRecord
metadata:
+46
View File
@@ -0,0 +1,46 @@
---
# Ceph RGW (S3) backup target for the gitea CNPG cluster, provisioned by the
# in-estate cephrgw-operator. One dedicated bucket + owner user per cluster.
apiVersion: ceph.unkin.net/v1alpha1
kind: ObjectStoreUser
metadata:
name: cnpg-gitea-backup
namespace: gitea
spec:
displayName: "CNPG backup owner (gitea)"
# RGW users are global; keep the uid namespace-qualified so it never collides.
uid: cnpg-gitea-backup
maxBuckets: 5
# Operator writes AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ RGW_UID,
# S3_ENDPOINT) into this Secret; the Cluster's barmanObjectStore consumes it.
secretName: cnpg-gitea-backup-s3
retainOnDelete: true
---
apiVersion: ceph.unkin.net/v1alpha1
kind: Bucket
metadata:
name: cnpg-gitea
namespace: gitea
spec:
bucketName: cnpg-gitea
ownerRef: cnpg-gitea-backup
versioning: false
tags:
app: gitea
purpose: cnpg-backup
retainOnDelete: true
---
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: cnpg-gitea-nightly
namespace: gitea
spec:
# 6-field CNPG cron (seconds first). 04:00 — next free slot after netbox
# (03:40), keeping the estate's 20-minute stagger.
schedule: "0 0 4 * * *"
immediate: false
backupOwnerReference: self
method: barmanObjectStore
cluster:
name: gitea-postgres
+90
View File
@@ -0,0 +1,90 @@
---
# Postgres for the k8s Gitea (replaces the Patroni-shared DB the VM uses). Gitea
# already runs on Postgres, so cutover is a plain pg dump/restore (no engine
# conversion). App-user creds come from the postgres-credentials Vault secret.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: gitea-postgres
namespace: gitea
spec:
affinity:
podAntiAffinityType: preferred
backup:
# 30-day retention. Enforced by CNPG against the object store on each
# successful base backup.
retentionPolicy: 30d
barmanObjectStore:
# Dedicated per-cluster Ceph RGW bucket (cephrgw-operator provisions it).
destinationPath: s3://cnpg-gitea
endpointURL: https://s3.ceph.unkin.net
# radosgw serves a Vault-PKI cert; trust the internal CA (reflected into
# every namespace as the vault-ca-cert Secret).
endpointCA:
name: vault-ca-cert
key: ca.crt
# Keys minted by the ObjectStoreUser in cnpg_backup.yaml; never hardcoded.
s3Credentials:
accessKeyId:
name: cnpg-gitea-backup-s3
key: AWS_ACCESS_KEY_ID
secretAccessKey:
name: cnpg-gitea-backup-s3
key: AWS_SECRET_ACCESS_KEY
# Path prefix within the bucket; keep stable across restores (see docs).
serverName: gitea
data:
compression: bzip2
jobs: 2
wal:
compression: zstd
maxParallel: 2
bootstrap:
initdb:
database: gitea
encoding: UTF8
localeCType: C
localeCollate: C
owner: gitea
secret:
name: postgres-credentials
enablePDB: true
enableSuperuserAccess: false
failoverDelay: 0
imageName: ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie
instances: 2
logLevel: info
monitoring:
customQueriesConfigMap:
- key: queries
name: cnpg-default-monitoring
disableDefaultQueries: false
enablePodMonitor: false
postgresql:
parameters:
max_connections: "200"
shared_buffers: 256MB
primaryUpdateMethod: restart
primaryUpdateStrategy: unsupervised
replicationSlots:
highAvailability:
enabled: true
slotPrefix: _cnpg_
synchronizeReplicas:
enabled: true
updateInterval: 30
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: 250m
memory: 512Mi
smartShutdownTimeout: 180
startDelay: 3600
stopDelay: 1800
storage:
resizeInUseVolumes: true
size: 20Gi
storageClass: cephrbd-fast-delete
switchoverDelay: 3600
+36
View File
@@ -0,0 +1,36 @@
---
# pgbouncer in front of the primary. Gitea opens a connection per request and
# benefits from pooling under multiple app replicas. Session mode keeps Gitea's
# occasional session-scoped state (advisory locks, LISTEN/NOTIFY) working.
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
name: gitea-postgres-pooler-rw
namespace: gitea
spec:
cluster:
name: gitea-postgres
instances: 2
pgbouncer:
parameters:
default_pool_size: "50"
max_client_conn: "200"
paused: false
poolMode: session
template:
metadata:
labels:
app: pooler-rw
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- pooler-rw
topologyKey: kubernetes.io/hostname
containers: []
type: rw
+69
View File
@@ -0,0 +1,69 @@
---
# HTTPS front for the k8s Gitea, served on two names:
# git.unkin.net — canonical/production (apex, bind-operator zone;
# DNS flip is the gated cutover step, see the doc)
# git.k8s.syd1.au.unkin.net — admin/backup route (external-dns k8s.syd1 zone),
# same dual-name pattern as identity.unkin.net.
# The cert-manager Certificate (vault-issuer) takes CN git.unkin.net and gets a
# DNS SAN for each TLS listener hostname automatically.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: gitea
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
traefik.io/instance: internal
annotations:
cert-manager.io/cluster-issuer: vault-issuer
cert-manager.io/common-name: git.unkin.net
cert-manager.io/private-key-size: "4096"
# Only the k8s admin route is published by external-dns (it owns just the
# k8s.syd1.au.unkin.net zone). git.unkin.net lives in the apex zone and is
# flipped at cutover — NOT managed here.
external-dns.alpha.kubernetes.io/hostname: git.k8s.syd1.au.unkin.net
external-dns.alpha.kubernetes.io/target: 198.18.200.4
spec:
gatewayClassName: traefik-internal
listeners:
- name: http-primary
port: 80
protocol: HTTP
hostname: git.unkin.net
allowedRoutes:
namespaces:
from: Same
- name: https-primary
port: 443
protocol: HTTPS
hostname: git.unkin.net
allowedRoutes:
namespaces:
from: Same
tls:
mode: Terminate
certificateRefs:
- group: ""
kind: Secret
name: gitea-tls
- name: http-admin
port: 80
protocol: HTTP
hostname: git.k8s.syd1.au.unkin.net
allowedRoutes:
namespaces:
from: Same
- name: https-admin
port: 443
protocol: HTTPS
hostname: git.k8s.syd1.au.unkin.net
allowedRoutes:
namespaces:
from: Same
tls:
mode: Terminate
certificateRefs:
- group: ""
kind: Secret
name: gitea-tls
+65
View File
@@ -0,0 +1,65 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: gitea-http-redirect
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
spec:
hostnames:
- git.unkin.net
- git.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: gitea
sectionName: http-primary
- group: gateway.networking.k8s.io
kind: Gateway
name: gitea
sectionName: http-admin
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: gitea
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/instance: gitea
spec:
hostnames:
- git.unkin.net
- git.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: gitea
sectionName: https-primary
- group: gateway.networking.k8s.io
kind: Gateway
name: gitea
sectionName: https-admin
rules:
- backendRefs:
- group: ""
kind: Service
name: gitea-http
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
+16
View File
@@ -0,0 +1,16 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- cnpg_cluster.yaml
- cnpg_backup.yaml
- cnpg_pooler.yaml
- valkey-deployment.yaml
- valkey-pvc.yaml
- valkey-service.yaml
- vaultauth.yaml
- vaultstaticsecret.yaml
- gateway.yaml
- httproute.yaml
+7
View File
@@ -0,0 +1,7 @@
---
apiVersion: v1
kind: Namespace
metadata:
labels:
app.kubernetes.io/name: gitea
name: gitea
+89
View File
@@ -0,0 +1,89 @@
---
# Standalone Valkey (Redis-compatible) for Gitea's session store, cache and
# queue. The Gitea chart bundles a redis-cluster subchart, but we run our own
# standalone Valkey here: it keeps image control in-estate (valkey/valkey,
# already allowlisted through the artifactapi dockerhub mirror) and matches the
# standalone-cache pattern used by litellm/netbox. One instance serves three
# logical DBs: DB 0 = session, DB 1 = cache, DB 2 = queue. AOF persistence is
# enabled so queued actions/webhook deliveries survive a restart.
apiVersion: apps/v1
kind: Deployment
metadata:
name: gitea-valkey
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/component: valkey
spec:
replicas: 1
selector:
matchLabels:
app: gitea-valkey
strategy:
type: Recreate
template:
metadata:
labels:
app: gitea-valkey
app.kubernetes.io/name: gitea
app.kubernetes.io/component: valkey
spec:
securityContext:
fsGroup: 999
containers:
- name: valkey
image: valkey/valkey:8-alpine
imagePullPolicy: IfNotPresent
command:
- valkey-server
- --appendonly
- "yes"
- --save
- "60"
- "1"
ports:
- containerPort: 6379
name: valkey
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 999
capabilities:
drop:
- ALL
livenessProbe:
exec:
command:
- valkey-cli
- ping
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
exec:
command:
- valkey-cli
- ping
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 50m
memory: 128Mi
volumeMounts:
- mountPath: /data
name: data
restartPolicy: Always
volumes:
- name: data
persistentVolumeClaim:
claimName: gitea-valkey-data
+14
View File
@@ -0,0 +1,14 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gitea-valkey-data
namespace: gitea
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: cephrbd-fast-delete
volumeMode: Filesystem
+20
View File
@@ -0,0 +1,20 @@
---
apiVersion: v1
kind: Service
metadata:
name: gitea-valkey
namespace: gitea
labels:
app.kubernetes.io/name: gitea
app.kubernetes.io/component: valkey
spec:
internalTrafficPolicy: Cluster
ports:
- name: valkey
port: 6379
protocol: TCP
targetPort: valkey
selector:
app: gitea-valkey
sessionAffinity: None
type: ClusterIP
+18
View File
@@ -0,0 +1,18 @@
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
name: default
namespace: gitea
spec:
allowedNamespaces:
- gitea
kubernetes:
audiences:
- vault
role: default
serviceAccount: default
tokenExpirationSeconds: 600
method: kubernetes
mount: k8s/au/syd1
vaultConnectionRef: vso-system/default
+83
View File
@@ -0,0 +1,83 @@
---
# CNPG app-user credentials (keys: username, password). Consumed by the Cluster
# bootstrap (initdb.secret) AND by Gitea (gitea.config.database.PASSWD via the
# chart's existingSecret wiring). One-time Vault seed — see the PR description.
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: postgres-credentials
namespace: gitea
spec:
destination:
create: true
name: postgres-credentials
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/gitea/default/postgres-credentials
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
---
# Initial Gitea admin (keys: username, password, email). Applied by the chart's
# init job on first boot (gitea.admin.existingSecret). Local fallback account
# that survives the Authentik OIDC cutover. One-time Vault seed.
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: gitea-admin
namespace: gitea
spec:
destination:
create: true
name: gitea-admin
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/gitea/default/gitea-admin
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
---
# Gitea internal secrets (keys: SECRET_KEY, INTERNAL_TOKEN). Pinned here rather
# than chart-generated so all replicas share identical values AND so the data
# cutover can replace them with the VM's app.ini values (SECRET_KEY encrypts
# 2FA/mirror/oauth secrets in the DB — it MUST match the restored database).
# One-time Vault seed.
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: gitea-inner
namespace: gitea
spec:
destination:
create: true
name: gitea-inner
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/gitea/default/gitea-inner
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
---
# Authentik OIDC client secret (key: client_secret). Read by the
# terraform-authentik provider runner (policy already grants
# kv/.../namespace/+/default/oauth-credentials) AND mounted into Gitea to
# register the OIDC login source. One-time Vault seed.
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
name: oauth-credentials
namespace: gitea
spec:
destination:
create: true
name: oauth-credentials
overwrite: true
hmacSecretData: true
mount: kv
path: kubernetes/namespace/gitea/default/oauth-credentials
refreshAfter: 5m
type: kv-v2
vaultAuthRef: default
@@ -0,0 +1,51 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../../base/gitea
helmCharts:
- name: gitea
repo: oci://docker.gitea.com/charts
version: "12.6.0"
releaseName: gitea
namespace: gitea
valuesFile: values.yaml
# The chart renders a `helm.sh/hook: test` connection Pod (busybox). We deploy
# via kustomize+ArgoCD (not `helm test`), so drop it rather than leave an orphan
# Pod pulling an unallowlisted image.
patches:
- target:
version: v1
kind: Pod
name: gitea-test-connection
patch: |-
$patch: delete
apiVersion: v1
kind: Pod
metadata:
name: gitea-test-connection
# The chart renders the http Service with `targetPort: null` (it defaults the
# target to the port name, which kustomize drops); pin it to 3000 so the
# manifest is valid and the HTTPRoute backend resolves.
- target:
version: v1
kind: Service
name: gitea-http
patch: |-
- op: replace
path: /spec/ports/0/targetPort
value: 3000
# SSH is disabled (DISABLE_SSH); drop the ssh Service the chart still renders.
- target:
version: v1
kind: Service
name: gitea-ssh
patch: |-
$patch: delete
apiVersion: v1
kind: Service
metadata:
name: gitea-ssh
+184
View File
@@ -0,0 +1,184 @@
# Gitea helm-gitea chart 12.6.0 (appVersion 1.26.1), app pinned to 1.26.2 to
# match the VM being replaced. HA-shaped: 2 replicas on shared RWX CephFS, with
# Postgres (CNPG), cache/session/queue (Valkey), and OIDC all externalised —
# exactly the shape the chart's docs/ha-setup.md requires for replicaCount > 1.
replicaCount: 2
image:
registry: docker.io
repository: gitea/gitea
tag: "1.26.2" # chart appends "-rootless" because rootless: true below
rootless: true
pullPolicy: IfNotPresent
# All bundled stateful subcharts OFF — we run CNPG Postgres + standalone Valkey.
postgresql-ha:
enabled: false
postgresql:
enabled: false
valkey-cluster:
enabled: false
valkey:
enabled: false
# Shared repo/LFS/attachment storage. RWX is mandatory for >1 replica so every
# pod sees the same /data (repos, lfs, avatars, attachments).
persistence:
enabled: true
create: true
mount: true
accessModes:
- ReadWriteMany
storageClass: cephfs-raid6-delete
size: 20Gi
service:
http:
type: ClusterIP
port: 3000
# git-over-ssh is disabled — HTTPS clones only (the estate norm). DISABLE_SSH
# below stops the in-pod SSH server; the chart still renders a gitea-ssh
# Service, which the overlay kustomization $patch-deletes.
ssh:
type: ClusterIP
port: 22
# Gateway API (HTTPRoute in the base) fronts HTTP; the chart Ingress is unused.
ingress:
enabled: false
# Rolling replicas on slow shared storage: keep one old pod up and give new pods
# generous startup headroom so migrations on boot don't trip the probes.
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: "100%"
maxUnavailable: 0
resources:
requests:
cpu: "500m"
memory: 512Mi
limits:
cpu: "2"
memory: 2Gi
# Trust the internal unkin.net CA that signs identity.unkin.net's cert. Gitea is
# Go, so SSL_CERT_DIR adds the mounted vault-ca-cert to the system trust pool
# (additive — public roots stay via the default bundle). Needed for the OIDC
# discovery/JWKS fetch at login.
deployment:
env:
- name: SSL_CERT_DIR
value: /etc/gitea/tls-ca
extraVolumes:
- name: vault-ca-cert
secret:
secretName: vault-ca-cert
items:
- key: ca.crt
path: ca.crt
extraContainerVolumeMounts:
- name: vault-ca-cert
mountPath: /etc/gitea/tls-ca
readOnly: true
extraInitVolumeMounts:
- name: vault-ca-cert
mountPath: /etc/gitea/tls-ca
readOnly: true
gitea:
# Local admin fallback (survives the OIDC cutover). Secret keys: username,
# password. Seeded in Vault -> synced to the gitea-admin Secret by VSO.
admin:
existingSecret: gitea-admin
email: "benvin@unkin.net"
passwordMode: keepUpdated
# DB password (and Gitea internal secrets) injected as env from VSO Secrets,
# never rendered into app.ini in git. GITEA__<section>__<KEY> maps to app.ini.
additionalConfigFromEnvs:
- name: GITEA__database__PASSWD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: password
- name: GITEA__security__SECRET_KEY
valueFrom:
secretKeyRef:
name: gitea-inner
key: SECRET_KEY
- name: GITEA__security__INTERNAL_TOKEN
valueFrom:
secretKeyRef:
name: gitea-inner
key: INTERNAL_TOKEN
# Authentik OIDC login source, registered declaratively on boot. existingSecret
# must carry keys `key` (client id) and `secret` (client secret) — seeded at the
# same Vault path terraform-authentik reads client_secret from.
oauth:
- name: "authentik"
provider: "openidConnect"
existingSecret: oauth-credentials
autoDiscoverUrl: "https://identity.unkin.net/application/o/gitea/.well-known/openid-configuration"
config:
server:
DOMAIN: git.unkin.net
ROOT_URL: https://git.unkin.net/
SSH_DOMAIN: git.unkin.net
DISABLE_SSH: true
START_SSH_SERVER: false
LFS_START_SERVER: true
database:
DB_TYPE: postgres
HOST: gitea-postgres-pooler-rw:5432
NAME: gitea
USER: gitea
SSL_MODE: disable
# Standalone Valkey: session on db 0, cache on db 1, queue on db 2.
session:
PROVIDER: redis
PROVIDER_CONFIG: "redis://gitea-valkey:6379/0"
cache:
ENABLED: true
ADAPTER: redis
HOST: "redis://gitea-valkey:6379/1"
queue:
TYPE: redis
CONN_STR: "redis://gitea-valkey:6379/2"
# DB-backed issue indexer is replica-safe on shared storage (matches the VM);
# avoids the on-disk bleve indexer that HA can't share.
indexer:
ISSUE_INDEXER_TYPE: db
REPO_INDEXER_ENABLED: false
# CI stays on Woodpecker — Gitea Actions disabled.
actions:
ENABLED: false
repository:
DEFAULT_BRANCH: main
DEFAULT_PRIVATE: last
# OIDC auto-registration + account linking so existing local users (benvin,
# bots) link to their Authentik identity by matching username on first login.
service:
ENABLE_AUTO_REGISTRATION: true
oauth2_client:
ENABLE_AUTO_REGISTRATION: true
ACCOUNT_LINKING: auto
USERNAME: preferred_username
UPDATE_AVATAR: true
# Startup headroom for migrations on RWX storage across a rolling update.
startupProbe:
enabled: true
livenessProbe:
enabled: true
readinessProbe:
enabled: true
+1
View File
@@ -24,6 +24,7 @@ spec:
- path: apps/overlays/*/elastic-system
- path: apps/overlays/*/encapi
- path: apps/overlays/*/externaldns
- path: apps/overlays/*/gitea
- path: apps/overlays/*/grafana-system
- path: apps/overlays/*/inteldeviceplugins-system
- path: apps/overlays/*/jfrog
+3
View File
@@ -13,6 +13,7 @@ spec:
- oci://gcr.io/k8s-staging-nfd/charts
- oci://ghcr.io/woodpecker-ci/helm/woodpecker
- oci://ghcr.io/netbox-community/netbox-chart/netbox
- oci://docker.gitea.com/charts
destinations:
- namespace: '*-system'
server: https://kubernetes.default.svc
@@ -34,6 +35,8 @@ spec:
server: https://kubernetes.default.svc
- namespace: 'externaldns'
server: https://kubernetes.default.svc
- namespace: 'gitea'
server: https://kubernetes.default.svc
- namespace: 'jfrog'
server: https://kubernetes.default.svc
- namespace: 'kanidm'
+1
View File
@@ -7,3 +7,4 @@ Operational notes for the manifests in this repo.
| [cnpg-backups.md](cnpg-backups.md) | How CNPG Postgres backups (WAL archiving + nightly base backups) to Ceph RGW are configured. |
| [cnpg-restore.md](cnpg-restore.md) | Restoring a CNPG cluster: full recovery, point-in-time recovery, cutover, and gotchas. |
| [authentik-rancher-sso.md](authentik-rancher-sso.md) | Manual runtime step to point Rancher's OIDC auth at the canonical `identity.unkin.net` issuer and trust the internal CA. |
| [gitea-migration.md](gitea-migration.md) | Staged cutover of the git.unkin.net forge from the Puppet VM to the `gitea` namespace. |
+130
View File
@@ -0,0 +1,130 @@
# Gitea VM → k8s migration
Replaces the Puppet-managed `git.unkin.net` VM (`roles::infra::git::server`, 3 nodes
on shared `/shared/apps/gitea` + Patroni Postgres + Redis) with the `gitea`
namespace here. The forge is critical infra (Woodpecker, tea, terraform-git,
teabot, ArgoCD repo sources, r10k), so cutover is staged with the VM left intact
for rollback.
## What's deployed
| Piece | Here | VM today |
|---|---|---|
| App | gitea chart 12.6.0, Gitea **1.26.2** rootless, **2 replicas** | binary 1.26.2, 3 nodes |
| Repo/LFS storage | RWX PVC `gitea-shared-storage` (`cephfs-raid6-delete`, 20Gi) | `/shared/apps/gitea` |
| DB | CNPG `gitea-postgres` (2 instances) + pgbouncer, S3 backup `cnpg-gitea` 04:00/30d | Patroni-shared Postgres |
| Cache/session/queue | standalone Valkey (db 0/1/2, AOF) | Redis sentinel |
| SSH | **disabled** (`DISABLE_SSH`) — HTTPS clones only | host sshd, port 2222 |
| Ingress | Gateway API → `git.unkin.net` + `git.k8s.syd1.au.unkin.net` | HAProxy `be_gitea` → git.unkin.net |
| Actions | **disabled** (CI stays Woodpecker) | runner deployed separately |
| Container registry | **disabled** (moved to artifactapi `docker-internal`) | built-in, in use |
| Other package types | enabled (generic archives for *arr etc. still used) | built-in |
HA note: this is the **same shape the VM already runs** (multiple app nodes on
shared storage + external DB + external Redis), which the Gitea chart's
`docs/ha-setup.md` endorses for `replicaCount > 1`. Genuine multi-replica HA, not
single-replica failover.
## One-time Vault seeds (before first sync)
```sh
# app DB user (CNPG bootstrap + Gitea)
vault kv put kv/kubernetes/namespace/gitea/default/postgres-credentials \
username=gitea password="$(openssl rand -hex 24)"
# local admin fallback (survives OIDC cutover)
vault kv put kv/kubernetes/namespace/gitea/default/gitea-admin \
username=benvin password="$(openssl rand -hex 24)" email=benvin@unkin.net
# Gitea internal secrets — pinned so all replicas match. At data cutover (stage 1/2)
# REPLACE these with the VM's app.ini [security] SECRET_KEY + INTERNAL_TOKEN so the
# restored DB's encrypted columns (2FA, mirrors, oauth) stay decryptable.
vault kv put kv/kubernetes/namespace/gitea/default/gitea-inner \
SECRET_KEY="$(gitea generate secret SECRET_KEY)" \
INTERNAL_TOKEN="$(gitea generate secret INTERNAL_TOKEN)"
# Authentik OIDC client — `client_secret` read by terraform-authentik; `key`/`secret`
# consumed by the chart's oauth login-source (existingSecret needs both keys).
S=$(openssl rand -hex 40)
vault kv put kv/kubernetes/namespace/gitea/default/oauth-credentials \
client_secret="$S" key=gitea secret="$S"
```
## Stages
**0 — Validate on the admin route.** Merge this PR + the terraform-authentik PR
(gitea OIDC app). ArgoCD syncs the `gitea` app; external-dns publishes
`git.k8s.syd1.au.unkin.net` → the gateway VIP (safe — new name, empty instance).
`git.unkin.net` stays on the live VM until stage 2. Confirm on the admin route:
local-admin login, OIDC login (Authentik → auto-register/link), create a test
repo, HTTPS push/clone (SSH is disabled — HTTPS only), and a Woodpecker webhook
fires against `git.k8s.syd1.au.unkin.net`.
**1 — Data migration rehearsal.** Gitea is already Postgres→Postgres, so **no DB
engine conversion** — a plain SQL dump/restore plus a filesystem copy of the data
dir. On a VM node:
```sh
# repos + LFS + attachments + avatars (the /shared/apps/gitea tree, minus the DB)
gitea dump -c /etc/gitea/app.ini --skip-db --file /tmp/gitea-data.zip # or rsync /shared/apps/gitea
pg_dump -h master.patroni-shared-prod.service.au-syd1.consul -U gitea gitea > /tmp/gitea.sql
```
Restore into k8s:
```sh
# DB
kubectl -n gitea exec -i gitea-postgres-1 -- psql -U gitea gitea < /tmp/gitea.sql
# data dir into the shared PVC (via a throwaway pod mounting gitea-shared-storage)
kubectl -n gitea cp /shared/apps/gitea/repos <helper-pod>:/data/git/repositories
kubectl -n gitea cp /shared/apps/gitea/lfs <helper-pod>:/data/git/lfs
```
Then set `gitea-inner` to the VM's real `SECRET_KEY`/`INTERNAL_TOKEN` (above) and
restart the deployment. Verify repos/issues/LFS/users. Rehearse fully before the
freeze so the freeze window is just a delta.
**2 — Freeze + cutover (the `git.unkin.net` DNS flip).** Put the VM read-only (stop
Woodpecker/webhooks), take a final dump/restore of the delta. The Gateway already
serves `git.unkin.net` (listener + cert SAN) and the app already advertises
`ROOT_URL https://git.unkin.net/`, so cutover is **only the DNS flip**:
- Repoint `git.unkin.net` → the internal Traefik VIP `198.18.200.4`.
- **Where the record lives (reconciled):** `git.unkin.net` resolves TODAY via the
**Puppet DNS master** (`profiles::dns::master`, records from PuppetDB exported
resources) to the HAProxy VRRP VIP `198.18.19.17` → the VM. The GitOps home for
the `unkin.net` apex is the **k8s bind-operator** `unkin-net` BindZone
(`apps/base/bind-internal/authoritative/`), where `identity.unkin.net` already
lives as a DNSRecord — but that zone currently holds only SOA+NS + a few records
(the bulk apex migration off Puppet is a pending follow-up). So at cutover, flip
**whichever is the live authority for `unkin.net` at that moment**: either
uncomment the `git-dns-internal` DNSRecord in `records.yaml` (prepared, gated) if
the k8s bind cluster is live, or change the Puppet-side record. **Not**
external-dns — it owns only the `k8s.syd1.au.unkin.net` zone.
**3 — Consumer verification.** After DNS/cert flips:
| Consumer | Check / knob |
|---|---|
| Woodpecker | forge URL (`WOODPECKER_GITEA_URL`) still `https://git.unkin.net`; OAuth app client id/secret unchanged; **`forge_remote_id` per-repo is stable** only if the Gitea repo IDs are preserved by the DB restore — a full SQL restore keeps them, a re-migration would not. Verify a webhook + a build. |
| tea | `~/.config/tea/config.yml` host unchanged; token still valid (tokens are in the restored DB). |
| terraform-git provider | `base_url https://git.unkin.net`, admin token from restored DB works. |
| teabot / vault-plugin-secrets-gitea | admin user+password (Basic-Auth API) — the rotate-root password is in Vault, and the admin **local** user must exist post-restore. Re-run rotate-root if needed. |
| ArgoCD repo creds | `repoURL https://git.unkin.net/unkin/argocd-apps` unchanged; the repo credential secret still valid. |
| r10k / puppet | `r10k_repo https://git.unkin.net/unkin/puppet-r10k.git` unchanged. |
| Container registry | pulls already moved to artifactapi `docker-internal` (see the registry-migration PR). Nothing should still pull `git.unkin.net/unkin/<image>`. |
**4 — Rollback.** DNS `git.unkin.net` back to `198.18.19.17`, un-freeze the VM. The
VM was never modified, so rollback is a single record change.
## Open decisions / hard problems
- **SSH is disabled** (`DISABLE_SSH`) — HTTPS clones only, the estate norm. No SSH
VIP, no SSH DNS, nothing to cut over.
- **artifactapi's own image is circular** — it currently pulls `git.unkin.net/unkin/
artifactapi`. It can't pull itself from `docker-internal`. Keep it on an
external/bootstrap-safe registry (or node-preloaded) before the VM registry goes
away; not repointed in the registry-migration PR.
- **`forge_remote_id` stability** (Woodpecker) hinges on preserving Gitea repo IDs —
guaranteed by a full SQL restore, broken by any re-create/re-migrate path.
- **Cutover window**: freeze = final delta dump/restore + DNS TTL. Lower the
`git.unkin.net` record TTL (600s in both zones) a day ahead so the flip is quick.