diff --git a/apps/base/bind-internal/authoritative/records.yaml b/apps/base/bind-internal/authoritative/records.yaml index 6ef0599..b97dbec 100644 --- a/apps/base/bind-internal/authoritative/records.yaml +++ b/apps/base/bind-internal/authoritative/records.yaml @@ -20,6 +20,32 @@ spec: # identity.unkin.net hostname there. - 198.18.200.4 --- +# PRODUCTION CUTOVER RECORD — intentionally commented out. +# git.unkin.net currently resolves to the LIVE VM forge (HAProxy VRRP VIP +# 198.18.19.17), which holds every repo the estate depends on. Uncommenting this +# repoints the whole org's git.unkin.net at the new k8s Gitea gateway VIP, so it +# is the FINAL step of the forge migration — gated on the data migration (gitea +# dump/restore + SECRET_KEY copy) in argocd-apps docs/gitea-migration.md. +# NOTE: the live git.unkin.net answer is served by the puppet DNS master today +# (profiles::dns::master, records from PuppetDB); this k8s apex zone holds only +# SOA+NS + a few DNSRecords so far. Confirm the k8s bind cluster is the live +# authority for unkin.net (or update the puppet record instead) before relying +# on this CR at cutover. +# --- +# apiVersion: bind.unkin.net/v1alpha1 +# kind: DNSRecord +# metadata: +# name: git-dns-internal +# namespace: bind-internal +# spec: +# zoneRef: unkin-net +# name: git +# type: A +# ttl: 600 +# values: +# # traefik-internal gateway VIP; the gitea Gateway serves git.unkin.net there. +# - 198.18.200.4 +--- apiVersion: bind.unkin.net/v1alpha1 kind: DNSRecord metadata: diff --git a/apps/base/gitea/cnpg_backup.yaml b/apps/base/gitea/cnpg_backup.yaml new file mode 100644 index 0000000..a4418b4 --- /dev/null +++ b/apps/base/gitea/cnpg_backup.yaml @@ -0,0 +1,46 @@ +--- +# Ceph RGW (S3) backup target for the gitea CNPG cluster, provisioned by the +# in-estate cephrgw-operator. One dedicated bucket + owner user per cluster. +apiVersion: ceph.unkin.net/v1alpha1 +kind: ObjectStoreUser +metadata: + name: cnpg-gitea-backup + namespace: gitea +spec: + displayName: "CNPG backup owner (gitea)" + # RGW users are global; keep the uid namespace-qualified so it never collides. + uid: cnpg-gitea-backup + maxBuckets: 5 + # Operator writes AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (+ RGW_UID, + # S3_ENDPOINT) into this Secret; the Cluster's barmanObjectStore consumes it. + secretName: cnpg-gitea-backup-s3 + retainOnDelete: true +--- +apiVersion: ceph.unkin.net/v1alpha1 +kind: Bucket +metadata: + name: cnpg-gitea + namespace: gitea +spec: + bucketName: cnpg-gitea + ownerRef: cnpg-gitea-backup + versioning: false + tags: + app: gitea + purpose: cnpg-backup + retainOnDelete: true +--- +apiVersion: postgresql.cnpg.io/v1 +kind: ScheduledBackup +metadata: + name: cnpg-gitea-nightly + namespace: gitea +spec: + # 6-field CNPG cron (seconds first). 04:00 — next free slot after netbox + # (03:40), keeping the estate's 20-minute stagger. + schedule: "0 0 4 * * *" + immediate: false + backupOwnerReference: self + method: barmanObjectStore + cluster: + name: gitea-postgres diff --git a/apps/base/gitea/cnpg_cluster.yaml b/apps/base/gitea/cnpg_cluster.yaml new file mode 100644 index 0000000..2857e0b --- /dev/null +++ b/apps/base/gitea/cnpg_cluster.yaml @@ -0,0 +1,90 @@ +--- +# Postgres for the k8s Gitea (replaces the Patroni-shared DB the VM uses). Gitea +# already runs on Postgres, so cutover is a plain pg dump/restore (no engine +# conversion). App-user creds come from the postgres-credentials Vault secret. +apiVersion: postgresql.cnpg.io/v1 +kind: Cluster +metadata: + name: gitea-postgres + namespace: gitea +spec: + affinity: + podAntiAffinityType: preferred + backup: + # 30-day retention. Enforced by CNPG against the object store on each + # successful base backup. + retentionPolicy: 30d + barmanObjectStore: + # Dedicated per-cluster Ceph RGW bucket (cephrgw-operator provisions it). + destinationPath: s3://cnpg-gitea + endpointURL: https://s3.ceph.unkin.net + # radosgw serves a Vault-PKI cert; trust the internal CA (reflected into + # every namespace as the vault-ca-cert Secret). + endpointCA: + name: vault-ca-cert + key: ca.crt + # Keys minted by the ObjectStoreUser in cnpg_backup.yaml; never hardcoded. + s3Credentials: + accessKeyId: + name: cnpg-gitea-backup-s3 + key: AWS_ACCESS_KEY_ID + secretAccessKey: + name: cnpg-gitea-backup-s3 + key: AWS_SECRET_ACCESS_KEY + # Path prefix within the bucket; keep stable across restores (see docs). + serverName: gitea + data: + compression: bzip2 + jobs: 2 + wal: + compression: zstd + maxParallel: 2 + bootstrap: + initdb: + database: gitea + encoding: UTF8 + localeCType: C + localeCollate: C + owner: gitea + secret: + name: postgres-credentials + enablePDB: true + enableSuperuserAccess: false + failoverDelay: 0 + imageName: ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie + instances: 2 + logLevel: info + monitoring: + customQueriesConfigMap: + - key: queries + name: cnpg-default-monitoring + disableDefaultQueries: false + enablePodMonitor: false + postgresql: + parameters: + max_connections: "200" + shared_buffers: 256MB + primaryUpdateMethod: restart + primaryUpdateStrategy: unsupervised + replicationSlots: + highAvailability: + enabled: true + slotPrefix: _cnpg_ + synchronizeReplicas: + enabled: true + updateInterval: 30 + resources: + limits: + cpu: "2" + memory: 2Gi + requests: + cpu: 250m + memory: 512Mi + smartShutdownTimeout: 180 + startDelay: 3600 + stopDelay: 1800 + storage: + resizeInUseVolumes: true + size: 20Gi + storageClass: cephrbd-fast-delete + switchoverDelay: 3600 diff --git a/apps/base/gitea/cnpg_pooler.yaml b/apps/base/gitea/cnpg_pooler.yaml new file mode 100644 index 0000000..c976f52 --- /dev/null +++ b/apps/base/gitea/cnpg_pooler.yaml @@ -0,0 +1,36 @@ +--- +# pgbouncer in front of the primary. Gitea opens a connection per request and +# benefits from pooling under multiple app replicas. Session mode keeps Gitea's +# occasional session-scoped state (advisory locks, LISTEN/NOTIFY) working. +apiVersion: postgresql.cnpg.io/v1 +kind: Pooler +metadata: + name: gitea-postgres-pooler-rw + namespace: gitea +spec: + cluster: + name: gitea-postgres + instances: 2 + pgbouncer: + parameters: + default_pool_size: "50" + max_client_conn: "200" + paused: false + poolMode: session + template: + metadata: + labels: + app: pooler-rw + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app + operator: In + values: + - pooler-rw + topologyKey: kubernetes.io/hostname + containers: [] + type: rw diff --git a/apps/base/gitea/gateway.yaml b/apps/base/gitea/gateway.yaml new file mode 100644 index 0000000..1e4d675 --- /dev/null +++ b/apps/base/gitea/gateway.yaml @@ -0,0 +1,69 @@ +--- +# HTTPS front for the k8s Gitea, served on two names: +# git.unkin.net — canonical/production (apex, bind-operator zone; +# DNS flip is the gated cutover step, see the doc) +# git.k8s.syd1.au.unkin.net — admin/backup route (external-dns k8s.syd1 zone), +# same dual-name pattern as identity.unkin.net. +# The cert-manager Certificate (vault-issuer) takes CN git.unkin.net and gets a +# DNS SAN for each TLS listener hostname automatically. +apiVersion: gateway.networking.k8s.io/v1 +kind: Gateway +metadata: + name: gitea + namespace: gitea + labels: + app.kubernetes.io/name: gitea + app.kubernetes.io/instance: gitea + traefik.io/instance: internal + annotations: + cert-manager.io/cluster-issuer: vault-issuer + cert-manager.io/common-name: git.unkin.net + cert-manager.io/private-key-size: "4096" + # Only the k8s admin route is published by external-dns (it owns just the + # k8s.syd1.au.unkin.net zone). git.unkin.net lives in the apex zone and is + # flipped at cutover — NOT managed here. + external-dns.alpha.kubernetes.io/hostname: git.k8s.syd1.au.unkin.net + external-dns.alpha.kubernetes.io/target: 198.18.200.4 +spec: + gatewayClassName: traefik-internal + listeners: + - name: http-primary + port: 80 + protocol: HTTP + hostname: git.unkin.net + allowedRoutes: + namespaces: + from: Same + - name: https-primary + port: 443 + protocol: HTTPS + hostname: git.unkin.net + allowedRoutes: + namespaces: + from: Same + tls: + mode: Terminate + certificateRefs: + - group: "" + kind: Secret + name: gitea-tls + - name: http-admin + port: 80 + protocol: HTTP + hostname: git.k8s.syd1.au.unkin.net + allowedRoutes: + namespaces: + from: Same + - name: https-admin + port: 443 + protocol: HTTPS + hostname: git.k8s.syd1.au.unkin.net + allowedRoutes: + namespaces: + from: Same + tls: + mode: Terminate + certificateRefs: + - group: "" + kind: Secret + name: gitea-tls diff --git a/apps/base/gitea/httproute.yaml b/apps/base/gitea/httproute.yaml new file mode 100644 index 0000000..62b6202 --- /dev/null +++ b/apps/base/gitea/httproute.yaml @@ -0,0 +1,65 @@ +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: gitea-http-redirect + namespace: gitea + labels: + app.kubernetes.io/name: gitea + app.kubernetes.io/instance: gitea +spec: + hostnames: + - git.unkin.net + - git.k8s.syd1.au.unkin.net + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: gitea + sectionName: http-primary + - group: gateway.networking.k8s.io + kind: Gateway + name: gitea + sectionName: http-admin + rules: + - filters: + - type: RequestRedirect + requestRedirect: + scheme: https + statusCode: 301 + matches: + - path: + type: PathPrefix + value: / +--- +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: gitea + namespace: gitea + labels: + app.kubernetes.io/name: gitea + app.kubernetes.io/instance: gitea +spec: + hostnames: + - git.unkin.net + - git.k8s.syd1.au.unkin.net + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: gitea + sectionName: https-primary + - group: gateway.networking.k8s.io + kind: Gateway + name: gitea + sectionName: https-admin + rules: + - backendRefs: + - group: "" + kind: Service + name: gitea-http + port: 3000 + weight: 1 + matches: + - path: + type: PathPrefix + value: / diff --git a/apps/base/gitea/kustomization.yaml b/apps/base/gitea/kustomization.yaml new file mode 100644 index 0000000..03a8b42 --- /dev/null +++ b/apps/base/gitea/kustomization.yaml @@ -0,0 +1,16 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - namespace.yaml + - cnpg_cluster.yaml + - cnpg_backup.yaml + - cnpg_pooler.yaml + - valkey-deployment.yaml + - valkey-pvc.yaml + - valkey-service.yaml + - vaultauth.yaml + - vaultstaticsecret.yaml + - gateway.yaml + - httproute.yaml diff --git a/apps/base/gitea/namespace.yaml b/apps/base/gitea/namespace.yaml new file mode 100644 index 0000000..96fa32b --- /dev/null +++ b/apps/base/gitea/namespace.yaml @@ -0,0 +1,7 @@ +--- +apiVersion: v1 +kind: Namespace +metadata: + labels: + app.kubernetes.io/name: gitea + name: gitea diff --git a/apps/base/gitea/valkey-deployment.yaml b/apps/base/gitea/valkey-deployment.yaml new file mode 100644 index 0000000..22725dd --- /dev/null +++ b/apps/base/gitea/valkey-deployment.yaml @@ -0,0 +1,89 @@ +--- +# Standalone Valkey (Redis-compatible) for Gitea's session store, cache and +# queue. The Gitea chart bundles a redis-cluster subchart, but we run our own +# standalone Valkey here: it keeps image control in-estate (valkey/valkey, +# already allowlisted through the artifactapi dockerhub mirror) and matches the +# standalone-cache pattern used by litellm/netbox. One instance serves three +# logical DBs: DB 0 = session, DB 1 = cache, DB 2 = queue. AOF persistence is +# enabled so queued actions/webhook deliveries survive a restart. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gitea-valkey + namespace: gitea + labels: + app.kubernetes.io/name: gitea + app.kubernetes.io/component: valkey +spec: + replicas: 1 + selector: + matchLabels: + app: gitea-valkey + strategy: + type: Recreate + template: + metadata: + labels: + app: gitea-valkey + app.kubernetes.io/name: gitea + app.kubernetes.io/component: valkey + spec: + securityContext: + fsGroup: 999 + containers: + - name: valkey + image: valkey/valkey:8-alpine + imagePullPolicy: IfNotPresent + command: + - valkey-server + - --appendonly + - "yes" + - --save + - "60" + - "1" + ports: + - containerPort: 6379 + name: valkey + protocol: TCP + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 999 + capabilities: + drop: + - ALL + livenessProbe: + exec: + command: + - valkey-cli + - ping + failureThreshold: 3 + initialDelaySeconds: 30 + periodSeconds: 30 + successThreshold: 1 + timeoutSeconds: 5 + readinessProbe: + exec: + command: + - valkey-cli + - ping + failureThreshold: 3 + initialDelaySeconds: 5 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 5 + resources: + limits: + cpu: 500m + memory: 512Mi + requests: + cpu: 50m + memory: 128Mi + volumeMounts: + - mountPath: /data + name: data + restartPolicy: Always + volumes: + - name: data + persistentVolumeClaim: + claimName: gitea-valkey-data diff --git a/apps/base/gitea/valkey-pvc.yaml b/apps/base/gitea/valkey-pvc.yaml new file mode 100644 index 0000000..11f3283 --- /dev/null +++ b/apps/base/gitea/valkey-pvc.yaml @@ -0,0 +1,14 @@ +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: gitea-valkey-data + namespace: gitea +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi + storageClassName: cephrbd-fast-delete + volumeMode: Filesystem diff --git a/apps/base/gitea/valkey-service.yaml b/apps/base/gitea/valkey-service.yaml new file mode 100644 index 0000000..5b97d26 --- /dev/null +++ b/apps/base/gitea/valkey-service.yaml @@ -0,0 +1,20 @@ +--- +apiVersion: v1 +kind: Service +metadata: + name: gitea-valkey + namespace: gitea + labels: + app.kubernetes.io/name: gitea + app.kubernetes.io/component: valkey +spec: + internalTrafficPolicy: Cluster + ports: + - name: valkey + port: 6379 + protocol: TCP + targetPort: valkey + selector: + app: gitea-valkey + sessionAffinity: None + type: ClusterIP diff --git a/apps/base/gitea/vaultauth.yaml b/apps/base/gitea/vaultauth.yaml new file mode 100644 index 0000000..52d3da1 --- /dev/null +++ b/apps/base/gitea/vaultauth.yaml @@ -0,0 +1,18 @@ +--- +apiVersion: secrets.hashicorp.com/v1beta1 +kind: VaultAuth +metadata: + name: default + namespace: gitea +spec: + allowedNamespaces: + - gitea + kubernetes: + audiences: + - vault + role: default + serviceAccount: default + tokenExpirationSeconds: 600 + method: kubernetes + mount: k8s/au/syd1 + vaultConnectionRef: vso-system/default diff --git a/apps/base/gitea/vaultstaticsecret.yaml b/apps/base/gitea/vaultstaticsecret.yaml new file mode 100644 index 0000000..6890f68 --- /dev/null +++ b/apps/base/gitea/vaultstaticsecret.yaml @@ -0,0 +1,83 @@ +--- +# CNPG app-user credentials (keys: username, password). Consumed by the Cluster +# bootstrap (initdb.secret) AND by Gitea (gitea.config.database.PASSWD via the +# chart's existingSecret wiring). One-time Vault seed — see the PR description. +apiVersion: secrets.hashicorp.com/v1beta1 +kind: VaultStaticSecret +metadata: + name: postgres-credentials + namespace: gitea +spec: + destination: + create: true + name: postgres-credentials + overwrite: true + hmacSecretData: true + mount: kv + path: kubernetes/namespace/gitea/default/postgres-credentials + refreshAfter: 5m + type: kv-v2 + vaultAuthRef: default +--- +# Initial Gitea admin (keys: username, password, email). Applied by the chart's +# init job on first boot (gitea.admin.existingSecret). Local fallback account +# that survives the Authentik OIDC cutover. One-time Vault seed. +apiVersion: secrets.hashicorp.com/v1beta1 +kind: VaultStaticSecret +metadata: + name: gitea-admin + namespace: gitea +spec: + destination: + create: true + name: gitea-admin + overwrite: true + hmacSecretData: true + mount: kv + path: kubernetes/namespace/gitea/default/gitea-admin + refreshAfter: 5m + type: kv-v2 + vaultAuthRef: default +--- +# Gitea internal secrets (keys: SECRET_KEY, INTERNAL_TOKEN). Pinned here rather +# than chart-generated so all replicas share identical values AND so the data +# cutover can replace them with the VM's app.ini values (SECRET_KEY encrypts +# 2FA/mirror/oauth secrets in the DB — it MUST match the restored database). +# One-time Vault seed. +apiVersion: secrets.hashicorp.com/v1beta1 +kind: VaultStaticSecret +metadata: + name: gitea-inner + namespace: gitea +spec: + destination: + create: true + name: gitea-inner + overwrite: true + hmacSecretData: true + mount: kv + path: kubernetes/namespace/gitea/default/gitea-inner + refreshAfter: 5m + type: kv-v2 + vaultAuthRef: default +--- +# Authentik OIDC client secret (key: client_secret). Read by the +# terraform-authentik provider runner (policy already grants +# kv/.../namespace/+/default/oauth-credentials) AND mounted into Gitea to +# register the OIDC login source. One-time Vault seed. +apiVersion: secrets.hashicorp.com/v1beta1 +kind: VaultStaticSecret +metadata: + name: oauth-credentials + namespace: gitea +spec: + destination: + create: true + name: oauth-credentials + overwrite: true + hmacSecretData: true + mount: kv + path: kubernetes/namespace/gitea/default/oauth-credentials + refreshAfter: 5m + type: kv-v2 + vaultAuthRef: default diff --git a/apps/overlays/au-syd1/gitea/kustomization.yaml b/apps/overlays/au-syd1/gitea/kustomization.yaml new file mode 100644 index 0000000..799ab0a --- /dev/null +++ b/apps/overlays/au-syd1/gitea/kustomization.yaml @@ -0,0 +1,51 @@ +--- +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../../../base/gitea + +helmCharts: + - name: gitea + repo: oci://docker.gitea.com/charts + version: "12.6.0" + releaseName: gitea + namespace: gitea + valuesFile: values.yaml + +# The chart renders a `helm.sh/hook: test` connection Pod (busybox). We deploy +# via kustomize+ArgoCD (not `helm test`), so drop it rather than leave an orphan +# Pod pulling an unallowlisted image. +patches: + - target: + version: v1 + kind: Pod + name: gitea-test-connection + patch: |- + $patch: delete + apiVersion: v1 + kind: Pod + metadata: + name: gitea-test-connection + # The chart renders the http Service with `targetPort: null` (it defaults the + # target to the port name, which kustomize drops); pin it to 3000 so the + # manifest is valid and the HTTPRoute backend resolves. + - target: + version: v1 + kind: Service + name: gitea-http + patch: |- + - op: replace + path: /spec/ports/0/targetPort + value: 3000 + # SSH is disabled (DISABLE_SSH); drop the ssh Service the chart still renders. + - target: + version: v1 + kind: Service + name: gitea-ssh + patch: |- + $patch: delete + apiVersion: v1 + kind: Service + metadata: + name: gitea-ssh diff --git a/apps/overlays/au-syd1/gitea/values.yaml b/apps/overlays/au-syd1/gitea/values.yaml new file mode 100644 index 0000000..5f69015 --- /dev/null +++ b/apps/overlays/au-syd1/gitea/values.yaml @@ -0,0 +1,184 @@ +# Gitea helm-gitea chart 12.6.0 (appVersion 1.26.1), app pinned to 1.26.2 to +# match the VM being replaced. HA-shaped: 2 replicas on shared RWX CephFS, with +# Postgres (CNPG), cache/session/queue (Valkey), and OIDC all externalised — +# exactly the shape the chart's docs/ha-setup.md requires for replicaCount > 1. +replicaCount: 2 + +image: + registry: docker.io + repository: gitea/gitea + tag: "1.26.2" # chart appends "-rootless" because rootless: true below + rootless: true + pullPolicy: IfNotPresent + +# All bundled stateful subcharts OFF — we run CNPG Postgres + standalone Valkey. +postgresql-ha: + enabled: false +postgresql: + enabled: false +valkey-cluster: + enabled: false +valkey: + enabled: false + +# Shared repo/LFS/attachment storage. RWX is mandatory for >1 replica so every +# pod sees the same /data (repos, lfs, avatars, attachments). +persistence: + enabled: true + create: true + mount: true + accessModes: + - ReadWriteMany + storageClass: cephfs-raid6-delete + size: 20Gi + +service: + http: + type: ClusterIP + port: 3000 + # git-over-ssh is disabled — HTTPS clones only (the estate norm). DISABLE_SSH + # below stops the in-pod SSH server; the chart still renders a gitea-ssh + # Service, which the overlay kustomization $patch-deletes. + ssh: + type: ClusterIP + port: 22 + +# Gateway API (HTTPRoute in the base) fronts HTTP; the chart Ingress is unused. +ingress: + enabled: false + +# Rolling replicas on slow shared storage: keep one old pod up and give new pods +# generous startup headroom so migrations on boot don't trip the probes. +strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: "100%" + maxUnavailable: 0 + +resources: + requests: + cpu: "500m" + memory: 512Mi + limits: + cpu: "2" + memory: 2Gi + +# Trust the internal unkin.net CA that signs identity.unkin.net's cert. Gitea is +# Go, so SSL_CERT_DIR adds the mounted vault-ca-cert to the system trust pool +# (additive — public roots stay via the default bundle). Needed for the OIDC +# discovery/JWKS fetch at login. +deployment: + env: + - name: SSL_CERT_DIR + value: /etc/gitea/tls-ca +extraVolumes: + - name: vault-ca-cert + secret: + secretName: vault-ca-cert + items: + - key: ca.crt + path: ca.crt +extraContainerVolumeMounts: + - name: vault-ca-cert + mountPath: /etc/gitea/tls-ca + readOnly: true +extraInitVolumeMounts: + - name: vault-ca-cert + mountPath: /etc/gitea/tls-ca + readOnly: true + +gitea: + # Local admin fallback (survives the OIDC cutover). Secret keys: username, + # password. Seeded in Vault -> synced to the gitea-admin Secret by VSO. + admin: + existingSecret: gitea-admin + email: "benvin@unkin.net" + passwordMode: keepUpdated + + # DB password (and Gitea internal secrets) injected as env from VSO Secrets, + # never rendered into app.ini in git. GITEA__
__ maps to app.ini. + additionalConfigFromEnvs: + - name: GITEA__database__PASSWD + valueFrom: + secretKeyRef: + name: postgres-credentials + key: password + - name: GITEA__security__SECRET_KEY + valueFrom: + secretKeyRef: + name: gitea-inner + key: SECRET_KEY + - name: GITEA__security__INTERNAL_TOKEN + valueFrom: + secretKeyRef: + name: gitea-inner + key: INTERNAL_TOKEN + + # Authentik OIDC login source, registered declaratively on boot. existingSecret + # must carry keys `key` (client id) and `secret` (client secret) — seeded at the + # same Vault path terraform-authentik reads client_secret from. + oauth: + - name: "authentik" + provider: "openidConnect" + existingSecret: oauth-credentials + autoDiscoverUrl: "https://identity.unkin.net/application/o/gitea/.well-known/openid-configuration" + + config: + server: + DOMAIN: git.unkin.net + ROOT_URL: https://git.unkin.net/ + SSH_DOMAIN: git.unkin.net + DISABLE_SSH: true + START_SSH_SERVER: false + LFS_START_SERVER: true + + database: + DB_TYPE: postgres + HOST: gitea-postgres-pooler-rw:5432 + NAME: gitea + USER: gitea + SSL_MODE: disable + + # Standalone Valkey: session on db 0, cache on db 1, queue on db 2. + session: + PROVIDER: redis + PROVIDER_CONFIG: "redis://gitea-valkey:6379/0" + cache: + ENABLED: true + ADAPTER: redis + HOST: "redis://gitea-valkey:6379/1" + queue: + TYPE: redis + CONN_STR: "redis://gitea-valkey:6379/2" + + # DB-backed issue indexer is replica-safe on shared storage (matches the VM); + # avoids the on-disk bleve indexer that HA can't share. + indexer: + ISSUE_INDEXER_TYPE: db + REPO_INDEXER_ENABLED: false + + # CI stays on Woodpecker — Gitea Actions disabled. + actions: + ENABLED: false + + repository: + DEFAULT_BRANCH: main + DEFAULT_PRIVATE: last + + # OIDC auto-registration + account linking so existing local users (benvin, + # bots) link to their Authentik identity by matching username on first login. + service: + ENABLE_AUTO_REGISTRATION: true + oauth2_client: + ENABLE_AUTO_REGISTRATION: true + ACCOUNT_LINKING: auto + USERNAME: preferred_username + UPDATE_AVATAR: true + + # Startup headroom for migrations on RWX storage across a rolling update. + startupProbe: + enabled: true + livenessProbe: + enabled: true + readinessProbe: + enabled: true diff --git a/argocd/applicationsets/platform.yaml b/argocd/applicationsets/platform.yaml index 43194d8..9b5a9db 100644 --- a/argocd/applicationsets/platform.yaml +++ b/argocd/applicationsets/platform.yaml @@ -24,6 +24,7 @@ spec: - path: apps/overlays/*/elastic-system - path: apps/overlays/*/encapi - path: apps/overlays/*/externaldns + - path: apps/overlays/*/gitea - path: apps/overlays/*/grafana-system - path: apps/overlays/*/inteldeviceplugins-system - path: apps/overlays/*/jfrog diff --git a/argocd/projects/platform.yaml b/argocd/projects/platform.yaml index 27f469b..1eabece 100644 --- a/argocd/projects/platform.yaml +++ b/argocd/projects/platform.yaml @@ -13,6 +13,7 @@ spec: - oci://gcr.io/k8s-staging-nfd/charts - oci://ghcr.io/woodpecker-ci/helm/woodpecker - oci://ghcr.io/netbox-community/netbox-chart/netbox + - oci://docker.gitea.com/charts destinations: - namespace: '*-system' server: https://kubernetes.default.svc @@ -34,6 +35,8 @@ spec: server: https://kubernetes.default.svc - namespace: 'externaldns' server: https://kubernetes.default.svc + - namespace: 'gitea' + server: https://kubernetes.default.svc - namespace: 'jfrog' server: https://kubernetes.default.svc - namespace: 'kanidm' diff --git a/docs/README.md b/docs/README.md index 8ab0681..1e790f4 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,3 +7,4 @@ Operational notes for the manifests in this repo. | [cnpg-backups.md](cnpg-backups.md) | How CNPG Postgres backups (WAL archiving + nightly base backups) to Ceph RGW are configured. | | [cnpg-restore.md](cnpg-restore.md) | Restoring a CNPG cluster: full recovery, point-in-time recovery, cutover, and gotchas. | | [authentik-rancher-sso.md](authentik-rancher-sso.md) | Manual runtime step to point Rancher's OIDC auth at the canonical `identity.unkin.net` issuer and trust the internal CA. | +| [gitea-migration.md](gitea-migration.md) | Staged cutover of the git.unkin.net forge from the Puppet VM to the `gitea` namespace. | diff --git a/docs/gitea-migration.md b/docs/gitea-migration.md new file mode 100644 index 0000000..4e7ef31 --- /dev/null +++ b/docs/gitea-migration.md @@ -0,0 +1,130 @@ +# Gitea VM → k8s migration + +Replaces the Puppet-managed `git.unkin.net` VM (`roles::infra::git::server`, 3 nodes +on shared `/shared/apps/gitea` + Patroni Postgres + Redis) with the `gitea` +namespace here. The forge is critical infra (Woodpecker, tea, terraform-git, +teabot, ArgoCD repo sources, r10k), so cutover is staged with the VM left intact +for rollback. + +## What's deployed + +| Piece | Here | VM today | +|---|---|---| +| App | gitea chart 12.6.0, Gitea **1.26.2** rootless, **2 replicas** | binary 1.26.2, 3 nodes | +| Repo/LFS storage | RWX PVC `gitea-shared-storage` (`cephfs-raid6-delete`, 20Gi) | `/shared/apps/gitea` | +| DB | CNPG `gitea-postgres` (2 instances) + pgbouncer, S3 backup `cnpg-gitea` 04:00/30d | Patroni-shared Postgres | +| Cache/session/queue | standalone Valkey (db 0/1/2, AOF) | Redis sentinel | +| SSH | **disabled** (`DISABLE_SSH`) — HTTPS clones only | host sshd, port 2222 | +| Ingress | Gateway API → `git.unkin.net` + `git.k8s.syd1.au.unkin.net` | HAProxy `be_gitea` → git.unkin.net | +| Actions | **disabled** (CI stays Woodpecker) | runner deployed separately | +| Container registry | **disabled** (moved to artifactapi `docker-internal`) | built-in, in use | +| Other package types | enabled (generic archives for *arr etc. still used) | built-in | + +HA note: this is the **same shape the VM already runs** (multiple app nodes on +shared storage + external DB + external Redis), which the Gitea chart's +`docs/ha-setup.md` endorses for `replicaCount > 1`. Genuine multi-replica HA, not +single-replica failover. + +## One-time Vault seeds (before first sync) + +```sh +# app DB user (CNPG bootstrap + Gitea) +vault kv put kv/kubernetes/namespace/gitea/default/postgres-credentials \ + username=gitea password="$(openssl rand -hex 24)" + +# local admin fallback (survives OIDC cutover) +vault kv put kv/kubernetes/namespace/gitea/default/gitea-admin \ + username=benvin password="$(openssl rand -hex 24)" email=benvin@unkin.net + +# Gitea internal secrets — pinned so all replicas match. At data cutover (stage 1/2) +# REPLACE these with the VM's app.ini [security] SECRET_KEY + INTERNAL_TOKEN so the +# restored DB's encrypted columns (2FA, mirrors, oauth) stay decryptable. +vault kv put kv/kubernetes/namespace/gitea/default/gitea-inner \ + SECRET_KEY="$(gitea generate secret SECRET_KEY)" \ + INTERNAL_TOKEN="$(gitea generate secret INTERNAL_TOKEN)" + +# Authentik OIDC client — `client_secret` read by terraform-authentik; `key`/`secret` +# consumed by the chart's oauth login-source (existingSecret needs both keys). +S=$(openssl rand -hex 40) +vault kv put kv/kubernetes/namespace/gitea/default/oauth-credentials \ + client_secret="$S" key=gitea secret="$S" +``` + +## Stages + +**0 — Validate on the admin route.** Merge this PR + the terraform-authentik PR +(gitea OIDC app). ArgoCD syncs the `gitea` app; external-dns publishes +`git.k8s.syd1.au.unkin.net` → the gateway VIP (safe — new name, empty instance). +`git.unkin.net` stays on the live VM until stage 2. Confirm on the admin route: +local-admin login, OIDC login (Authentik → auto-register/link), create a test +repo, HTTPS push/clone (SSH is disabled — HTTPS only), and a Woodpecker webhook +fires against `git.k8s.syd1.au.unkin.net`. + +**1 — Data migration rehearsal.** Gitea is already Postgres→Postgres, so **no DB +engine conversion** — a plain SQL dump/restore plus a filesystem copy of the data +dir. On a VM node: + +```sh +# repos + LFS + attachments + avatars (the /shared/apps/gitea tree, minus the DB) +gitea dump -c /etc/gitea/app.ini --skip-db --file /tmp/gitea-data.zip # or rsync /shared/apps/gitea +pg_dump -h master.patroni-shared-prod.service.au-syd1.consul -U gitea gitea > /tmp/gitea.sql +``` + +Restore into k8s: + +```sh +# DB +kubectl -n gitea exec -i gitea-postgres-1 -- psql -U gitea gitea < /tmp/gitea.sql +# data dir into the shared PVC (via a throwaway pod mounting gitea-shared-storage) +kubectl -n gitea cp /shared/apps/gitea/repos :/data/git/repositories +kubectl -n gitea cp /shared/apps/gitea/lfs :/data/git/lfs +``` + +Then set `gitea-inner` to the VM's real `SECRET_KEY`/`INTERNAL_TOKEN` (above) and +restart the deployment. Verify repos/issues/LFS/users. Rehearse fully before the +freeze so the freeze window is just a delta. + +**2 — Freeze + cutover (the `git.unkin.net` DNS flip).** Put the VM read-only (stop +Woodpecker/webhooks), take a final dump/restore of the delta. The Gateway already +serves `git.unkin.net` (listener + cert SAN) and the app already advertises +`ROOT_URL https://git.unkin.net/`, so cutover is **only the DNS flip**: +- Repoint `git.unkin.net` → the internal Traefik VIP `198.18.200.4`. +- **Where the record lives (reconciled):** `git.unkin.net` resolves TODAY via the + **Puppet DNS master** (`profiles::dns::master`, records from PuppetDB exported + resources) to the HAProxy VRRP VIP `198.18.19.17` → the VM. The GitOps home for + the `unkin.net` apex is the **k8s bind-operator** `unkin-net` BindZone + (`apps/base/bind-internal/authoritative/`), where `identity.unkin.net` already + lives as a DNSRecord — but that zone currently holds only SOA+NS + a few records + (the bulk apex migration off Puppet is a pending follow-up). So at cutover, flip + **whichever is the live authority for `unkin.net` at that moment**: either + uncomment the `git-dns-internal` DNSRecord in `records.yaml` (prepared, gated) if + the k8s bind cluster is live, or change the Puppet-side record. **Not** + external-dns — it owns only the `k8s.syd1.au.unkin.net` zone. + +**3 — Consumer verification.** After DNS/cert flips: + +| Consumer | Check / knob | +|---|---| +| Woodpecker | forge URL (`WOODPECKER_GITEA_URL`) still `https://git.unkin.net`; OAuth app client id/secret unchanged; **`forge_remote_id` per-repo is stable** only if the Gitea repo IDs are preserved by the DB restore — a full SQL restore keeps them, a re-migration would not. Verify a webhook + a build. | +| tea | `~/.config/tea/config.yml` host unchanged; token still valid (tokens are in the restored DB). | +| terraform-git provider | `base_url https://git.unkin.net`, admin token from restored DB works. | +| teabot / vault-plugin-secrets-gitea | admin user+password (Basic-Auth API) — the rotate-root password is in Vault, and the admin **local** user must exist post-restore. Re-run rotate-root if needed. | +| ArgoCD repo creds | `repoURL https://git.unkin.net/unkin/argocd-apps` unchanged; the repo credential secret still valid. | +| r10k / puppet | `r10k_repo https://git.unkin.net/unkin/puppet-r10k.git` unchanged. | +| Container registry | pulls already moved to artifactapi `docker-internal` (see the registry-migration PR). Nothing should still pull `git.unkin.net/unkin/`. | + +**4 — Rollback.** DNS `git.unkin.net` back to `198.18.19.17`, un-freeze the VM. The +VM was never modified, so rollback is a single record change. + +## Open decisions / hard problems + +- **SSH is disabled** (`DISABLE_SSH`) — HTTPS clones only, the estate norm. No SSH + VIP, no SSH DNS, nothing to cut over. +- **artifactapi's own image is circular** — it currently pulls `git.unkin.net/unkin/ + artifactapi`. It can't pull itself from `docker-internal`. Keep it on an + external/bootstrap-safe registry (or node-preloaded) before the VM registry goes + away; not repointed in the registry-migration PR. +- **`forge_remote_id` stability** (Woodpecker) hinges on preserving Gitea repo IDs — + guaranteed by a full SQL restore, broken by any re-create/re-migrate path. +- **Cutover window**: freeze = final delta dump/restore + DNS TTL. Lower the + `git.unkin.net` record TTL (600s in both zones) a day ahead so the flip is quick.