Replaces the isc-dhcpd PXE-boot VM with the kea-operator (v0.1.0) and an
HA kea pair, managed by ArgoCD. Deploys on a new, unused anycast IP; the
production cutover off the current dhcpd address is a separate later task.
- Add apps/base/dhcp-system: namespace, kea-operator RBAC + Deployment,
VPA, and the kea.unkin.net CRDs pulled from the operator repo at v0.1.0.
- Add the CRs translating the legacy dhcpd config: KeaCluster (2 replicas,
hot-standby HA, main.unkin.net, 1200/86400 leases, AU ntp pool), five
KeaSubnets 198.18.13-17.0/24 (gateways .254 except .17->.1 per the puppet
dhcp hieradata), Legacy/UEFI-64 PXE client classes, and the KeaAPI.
- Pin the DHCP LoadBalancer Service to the free common-pool IP 198.18.200.10
via PureLB (not the current dhcpd anycast 198.18.19.18).
- Provision the KeaAPI bearer token via an operator-generated Secret.
- Commit generated kea.unkin.net JSON schemas for kubeconform.
- Register dhcp-system in the platform ApplicationSet and AppProject.
Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
We self-delegate `_acme-challenge.unkin.net` into an `acme.unkin.net` zone we serve ourselves, so cert-manager can solve Let's Encrypt DNS-01 over RFC2136/TSIG. That needs a publicly-reachable authoritative BIND, separate from the internal estate.
- Add app `bind-external` (base + au-syd1 overlay); register it in the platform ApplicationSet and AppProject destinations (bind-operator already watches all namespaces).
- Add BindCluster `bind-external`: authoritative-only, recursion off, no forwarding, transfers denied except the keyed catalog/zone AXFR; 2 replicas; primaryService is a dmz-pinned PureLB LoadBalancer at `198.18.199.53`.
- Add BindZone `acme.unkin.net` (primary, dynamicUpdate) and BindTSIGKey `certmanager` (hmac-sha256), whose Secret `certmanager-tsig` reflects into the `cert-manager` namespace for the rfc2136 solver.
Pairs with argocd-apps #327 (the ClusterIssuers) and a one-time Google Cloud DNS delegation + NAT of the public IP :53 to `198.18.199.53`.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #329
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Publicly-trusted wildcard certs via Let's Encrypt DNS-01, solved over RFC2136/TSIG against our own BIND. A one-time CNAME self-delegates `_acme-challenge.unkin.net` into the `acme.unkin.net` zone served by bind-external; cert-manager writes the challenge TXT there. No GCP/clouddns and no Vault secret involved. The existing `vault-issuer` (internal PKI) is untouched.
- Add ClusterIssuers `letsencrypt` (prod) and `letsencrypt-staging`, both using a dns01 rfc2136 solver: nameserver `198.18.199.53:53`, key `certmanager`, HMACSHA256, `tsigSecretSecretRef` -> reflected Secret `certmanager-tsig` key `secret`.
- Whitelist `cert-manager.io ClusterIssuer` in the platform AppProject.
Depends on #329 (bind-external: the acme.unkin.net zone, the certmanager TSIG key reflected into cert-manager, and the 198.18.199.53 nameserver) and on the one-time Google Cloud DNS delegation + NAT of the public IP :53 to 198.18.199.53. Earlier clouddns/Vault commits on this branch are reverted.
---------
Co-authored-by: Ben Vincent <neotheo@gmail.com>
Reviewed-on: #327
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Stand up the git.unkin.net forge on k8s to replace the Puppet VM. Deployed HA-shaped to match what the VM already runs (multi-replica on shared storage + external DB/cache), so this is genuine multi-replica HA rather than single-replica failover. Serves a temporary git2.k8s.syd1.au.unkin.net host; the git.unkin.net cutover is staged in docs/gitea-migration.md.
- add apps/base/gitea: namespace, CNPG gitea-postgres (2 instances, S3 backup bucket cnpg-gitea, nightly 04:00/30d), pgbouncer pooler, standalone Valkey (session/cache/queue, AOF), VaultAuth + VaultStaticSecrets, Gateway + HTTPRoute
- add apps/overlays/au-syd1/gitea: official Gitea chart 12.6.0 (app 1.26.2, rootless, 2 replicas) via helm-through-kustomize; RWX CephFS repo storage, external CNPG + Valkey, Actions disabled, container registry disabled (moved to artifactapi), Authentik OIDC with auto-register/account-linking; SSH via LoadBalancer VIP 198.18.200.10:2222
- register gitea in the platform ApplicationSet + AppProject
- add docs/gitea-migration.md staged cutover plan (VM Postgres->CNPG dump/restore, DNS in main.unkin.net zone, consumer checklist, rollback)
Depends on: terraform-authentik gitea OIDC app, and terraform-artifactapi ^gitea/ dockerhub allowlist (both separate PRs). One-time Vault seeds are listed in the migration doc.
Reviewed-on: #309
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
NetBox becomes the source of truth for host/interface/IPAM data as part of retiring Cobbler. The ENC role already moved to encapi; a kickstart-generation microservice that consumes NetBox comes later and is intentionally **out of scope** here.
## Change
Adds the `netbox` app (namespace `netbox`, platform project) using the netbox-community helm chart via the OCI helm-through-kustomize pattern, plus estate-native data stores:
- **NetBox** — chart `8.3.40` (appVersion `v4.6.5`), `oci://ghcr.io/netbox-community/netbox-chart`. 2 web replicas, 1 worker; bundled Postgres/Valkey subcharts disabled. Media on RWX CephFS so replicas share uploads. Chart `helm.sh/hook: test` Pod dropped via a kustomize delete patch (we deploy through ArgoCD, not `helm test`).
- **CNPG Postgres** — `netbox-postgres`, 2 instances, PG 18, `cephrbd-fast-delete`. Standard per-cluster S3 backup: `barmanObjectStore -> s3://cnpg-netbox`, cephrgw `ObjectStoreUser` + `Bucket`, nightly `ScheduledBackup` at **03:40** (`0 40 3 * * *` — next free slot after grafana's 03:20), 30d retention. A pgbouncer `Pooler` (session mode) fronts it; NetBox connects via `netbox-postgres-pooler-rw`.
- **Valkey** — standalone Deployment (`valkey/valkey:8-alpine`), AOF-persistent PVC on `cephrbd-fast-delete`. One instance: DB 0 = RQ task queue, DB 1 = cache. No auth (in-cluster, namespace-isolated). Chosen over the bundled Bitnami subchart to keep image control in-estate and avoid Bitnami's legacy-image churn; mirrors the litellm standalone-cache pattern.
- **Ingress** — `Gateway` + `HTTPRoute` at `netbox.k8s.syd1.au.unkin.net` (`traefik-internal`, `vault-issuer` cert into `netbox-tls`, external-dns to the internal VIP), HTTP->HTTPS 301.
- **Secrets** — all via VSO `VaultStaticSecret` (`postgres-credentials`, `netbox-secret-key`, `netbox-superuser`, `oauth-credentials`); no plain Secrets committed. The shared `default` k8s-auth role already binds `*` namespaces with a namespace-templated KV policy, so **no terraform-vault change is needed**.
- **Authentik OIDC SSO** — `remoteAuth` wires `REMOTE_AUTH_ENABLED` + the `OpenIdConnectAuth` backend via chart values; `SOCIAL_AUTH_OIDC_*` via `extraConfig` (the chart's config loader globs `/run/config/extra/*/*.yaml`). The client secret is injected as a YAML fragment mounted from the Vault-synced `oauth-credentials` secret. New users auto-provision on first login.
- Registers `netbox` in the platform ApplicationSet and AppProject (destination namespace + chart sourceRepo).
## Image table (source -> artifactapi -> DHI decision)
All images flow through the estate's containerd registry mirrors; the allowlist patterns gate them. DHI (Docker Hardened Images) require authenticated pulls from a Docker Hub `dhi/` org and are **not** reachable through the anonymous mirror, so upstream official is used throughout.
| Image | Upstream | Mirror / allowlist | Allowlisted? | DHI decision |
|---|---|---|---|---|
| NetBox app/worker/housekeeping | `ghcr.io/netbox-community/netbox:v4.6.5` | ghcr remote, `^netbox-community/` | **added in terraform-artifactapi #17** | No DHI published; upstream official |
| CNPG Postgres | `ghcr.io/cloudnative-pg/postgresql:18.1-system-trixie` | ghcr, `^cloudnative-pg/` | already | estate-standard CNPG image |
| Valkey | `docker.io/valkey/valkey:8-alpine` | dockerhub, `^valkey/valkey` | already | DHI not anon-pullable; upstream official |
| NetBox init (perms) | `docker.io/busybox:1.38.0` | dockerhub, `^library/busybox` | already | upstream official |
| Worker wait-for-backend | `docker.io/rancher/kubectl:v1.36.2` | dockerhub, `^rancher/` | already | upstream official |
## Cross-repo PRs (merge order)
1. **terraform-artifactapi #17** (`^netbox-community/` ghcr allowlist) — **merge before** this PR so the NetBox image pulls on first sync.
2. **terraform-authentik #11** (OIDC provider/application) — independent; SSO works once applied + the secret is seeded.
3. This PR.
> Note: the CNPG S3-backup stanza pattern here is identical to PR #298 (already in main); it merges cleanly regardless of ordering.
## One-time Vault seeds (before/at first sync)
```
# App DB user (CNPG bootstrap + NetBox both consume this)
vault kv put kv/kubernetes/namespace/netbox/default/postgres-credentials \
username=netbox password="$(openssl rand -base64 30)"
# Django SECRET_KEY
vault kv put kv/kubernetes/namespace/netbox/default/netbox-secret-key \
secret_key="$(python3 -c 'import secrets;print(secrets.token_urlsafe(60))')"
# Bootstrap superuser
vault kv put kv/kubernetes/namespace/netbox/default/netbox-superuser \
username=admin email=admin@unkin.net \
password="$(openssl rand -base64 24)" api_token="$(openssl rand -hex 20)"
# OIDC client secret — ONE value stored two ways (raw for Authentik, YAML for NetBox)
CS="$(openssl rand -base64 30)"
vault kv put kv/kubernetes/namespace/netbox/default/oauth-credentials \
client_secret="$CS" oidc.yaml="SOCIAL_AUTH_OIDC_SECRET: \"$CS\""
```
Seed `oauth-credentials` **before** applying terraform-authentik #11 (that apply reads `client_secret`).
## Validation
- `kustomize build --enable-helm apps/overlays/au-syd1/netbox` — clean.
- `kubeconform` (CI args, k8s 1.33.7): **27/27 valid, 0 invalid**.
- `pre-commit` (yamllint + no-plain-secrets) on all changed files — pass.
## Out of scope
The PXE/kickstart microservice that will consume NetBox — not scaffolded here.
https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
Reviewed-on: #299
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
During the VM -> k8s Puppet migration, two PuppetDBs coexist and nodes move
between them as they migrate. `node-lookup` (and `pblastreport`) need a single,
consistent PuppetDB v4 view spanning both. `pdbmux` is a small merging proxy
that provides exactly that. Per the all-in-kubernetes estate direction it runs
as an in-cluster service, not a per-VM systemd unit.
pdbmux now lives in its own repository (https://git.unkin.net/unkin/pdbmux) —
split out of the earlier node-lookup prototype — and is released as a container
image on its own `v*` tags.
## Changes
- Add `apps/base/pdbmux/` (namespace, configmap, deployment, service, gateway,
httproute), modeled directly on the encapi app.
- Deployment: 2 replicas, image `git.unkin.net/unkin/pdbmux:v0.1.0`, port 8080,
`/healthz` liveness + readiness, config via `PDBMUX_*` env from a ConfigMap.
- Backends: `old=http://puppetdbapi.service.consul:8080`,
`new=http://puppetdb.puppet.svc.cluster.local:8080` (in-cluster, verified
against `apps/base/puppet/service_puppetdb.yaml` port `pdb-http`/8080 — the
in-cluster address is preferred over the external gateway). `new` is
primary/prefer, merge = freshness.
- Expose over HTTPS at `pdbmux.k8s.syd1.au.unkin.net` via a `traefik-internal`
Gateway (cert-manager `vault-issuer`, external-dns), plain-HTTP backend on a
port-80 Service — same shape as the puppetdb/encapi gateways — so
VM/workstation `node-lookup` can reach it.
- Add `apps/overlays/au-syd1/pdbmux/` and wire pdbmux into the platform
ApplicationSet (`apps/overlays/*/pdbmux`) and the platform AppProject
(`pdbmux` namespace destination), exactly as encapi is wired.
No new woodpecker ServiceAccount is required: the pdbmux image push uses the
`docker-buildx` plugin against the Gitea registry with the `default` SA (same as
encapi), not artifactapi.
## Verification
- `kubectl kustomize apps/overlays/au-syd1/pdbmux` builds clean (image resolves
to `git.unkin.net/unkin/pdbmux:v0.1.0`).
- ApplicationSet + AppProject YAML validated.
## Merge gates
1. The pdbmux repo initial-content PR
(unkin/pdbmux#1) must merge first.
2. `v0.1.0` must then be tagged on the pdbmux repo so the image
`git.unkin.net/unkin/pdbmux:v0.1.0` is built and pushed by that repo`s
`.woodpecker/docker.yaml`.
3. Then merge this PR. (If the first release tag differs from `v0.1.0`, update
the image tag in `apps/base/pdbmux/deployment.yaml` to match before merging.)
---------
Co-authored-by: benvin <neotheo@gmail.com>
Reviewed-on: #275
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
**HOLD until v0.1.3 is tagged/built** (operator #4 merged + tagged) — this PR bumps the operator to v0.1.3, whose CRD adds the `clusterRef` field these keys use.
## Why
Put all BIND DNS services in one `bind-internal` namespace and name the StatefulSets clearly.
## Changes
- 3 clusters consolidated into `bind-internal`, StatefulSets renamed **bind-authoritative** / **bind-resolvers** / **bind-externaldns**; LBs kept on 198.18.200.6/.7/.8; external-dns hostnames renamed to match
- `clusterRef` added to `transfer-key` (→ bind-authoritative) and `externaldns-key` (→ bind-externaldns) so keys are scoped per cluster
- removed the old `ns-auth`/`ns-resolver`/`ns-externaldns` apps; ApplicationSet + AppProject now list `bind-internal`
- bumped `bind-system` operator to **v0.1.3** (CRD link + image)
- operator stays in `bind-system`
## Deploy impact
ArgoCD prunes the old ns-* namespaces (StatefulSets/PVCs — data is only seed SOA+NS, no migrated records yet) and creates the renamed clusters in bind-internal.
## Validated
`kustomize build` → 28 docs (3 BindCluster, 20 BindZone, 2 catalog, 2 keys, ns); kubeconform clean.
Reviewed-on: #225
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Renames the three BIND DNS app namespaces `binddns-{auth,resolver,externaldns}` -> `ns-{auth,resolver,externaldns}`.
## Why
Shorter, clearer namespace names for the DNS tiers.
## Changes
- `argocd/applicationsets/platform.yaml`: overlay path registrations renamed (the ApplicationSet derives each app's namespace from its overlay dir name)
- `argocd/projects/platform.yaml`: destination namespaces renamed
## Coupled with
The per-tier PRs (#220/#221/#222) rename the overlay dirs + namespaces + external-dns hostnames to match. No app deploys to a renamed namespace until both this and the tier PR are merged (harmless before then — the ApplicationSet only instantiates apps for existing dirs).
Reviewed-on: #223
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
First of a 4-PR split of the bind rollout (was #216). Deploys just the operator control plane so it can be verified before any DNS clusters exist.
## Why
Roll out incrementally: operator + CRDs first, then each BIND tier as its own PR.
## Changes
- `apps/base/bind-system`: operator Deployment (`git.unkin.net/unkin/bind-operator:v0.1.1`), RBAC, namespace; CRDs pulled from the operator repo by raw URL (`config/crd/install.yaml` @ v0.1.1)
- au-syd1 `bind-system` overlay
- register all four bind apps in `argocd/applicationsets/platform.yaml` (DNS overlays instantiate only when their dirs land in the follow-up PRs)
- add `binddns-*` namespaces to `argocd/projects/platform.yaml`
- add `schemas/bind.unkin.net/*.json` for kubeconform
## Deploy impact
Operator pod + CRDs only. No DNS services yet — the operator is idle until BindClusters exist.
## Follow-ups (merge after this)
binddns-auth, binddns-resolver, binddns-externaldns — one PR each.
Reviewed-on: #219
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- Deploy age-api to the au-syd1 cluster
- Uses configMapGenerator for people config with jaidi, ben, and sudaporn
- Includes gateway, httproute, service, and deployment
- Image: git.unkin.net/unkin/age-api:v0.1.0
Reviewed-on: #210
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- New `apps/base/priority-classes/` app with four `PriorityClass` objects managed via the `platform` ArgoCD project
- Adds `apps/overlays/*/priority-classes` to the platform ApplicationSet generator
- Adds `priority-classes` namespace to platform AppProject destinations (required even for cluster-scoped resources)
| Class | Value | PreemptionPolicy | Intent |
|---|---|---|---|
| `low` | 100 | Never | Background work; evictable, won't preempt others |
| `power` | 100 | Never | Compute-heavy but expendable (e.g. AI/ML workloads) |
| `medium` | 10000 | PreemptLowerPriority | Standard services |
| `high` | 100000 | PreemptLowerPriority | Critical services; preempts lower-priority pods |
`PriorityClass` is already in the platform project's `clusterResourceWhitelist` so no project policy changes were needed.
## Test plan
- ArgoCD syncs `platform-priority-classes` successfully
- `kubectl get priorityclasses low power medium high` shows all four classes
Reviewed-on: #174
Vault and consul namespaces were missing from the platform AppProject
allowed destinations, causing ArgoCD sync failures with:
destination server 'https://kubernetes.default.svc' and namespace
'vault' do not match any of the allowed destinations in project 'platform'
Reviewed-on: #152
Migrate PureLB load balancer from Terragrunt to ArgoCD/Kustomize.
Deploys purelb v0.13.0 with two LBNodeAgent and two ServiceGroup CRs
(common: 198.18.200.0/24, dmz: 198.18.199.0/24).
Adds LBNodeAgent and ServiceGroup to kubeconform skip list (no CRD catalog schema).
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #84
Migrate Vault Secrets Operator from Terragrunt to ArgoCD/Kustomize.
Deploys vault-secrets-operator v1.2.0 with 3 replicas, plus ClusterRole,
ClusterRoleBindings, and vault-admin ServiceAccount.
Note: static service account tokens (kubernetes.io/service-account-token)
cannot be stored in git; create manually or via Vault after deployment.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #81
Migrate Victoria Metrics operator from Terragrunt to ArgoCD/Kustomize.
Deploys victoria-metrics-operator v0.57.1 with 2 replicas in vm-system.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #80
Migrate ECK operator from Terragrunt to ArgoCD/Kustomize.
Deploys eck-operator v3.2.0 with 2 replicas and PodDisruptionBudget
in the elastic-system namespace.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #79
Migrate repository sync cronjobs from Terragrunt to ArgoCD/Kustomize.
Adds four daily CronJobs (almalinux9-baseos, almalinux9-appstream, epel9,
openvox7) with associated PVCs and ConfigMaps in the reposync namespace.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #78
- Add node-feature-discovery and inteldeviceplugins-system to platform project
- Convert intel-nfd-rules from local Helm chart to static NodeFeatureRule manifests
- Add required Helm repositories (NFD OCI registry and Intel charts)
- Create base configurations with Helm charts and overlay structures
- Update platform ApplicationSet and project permissions
Reviewed-on: #48
- Add cnpg-system base ArgoCD application with namespace
- Create cnpg-system overlay for au-syd1 with CloudNativePG Helm chart
- Update platform ApplicationSet to include cnpg-system deployment
- Configure cloudnative-pg operator v0.27.0 with HA and resource limits
- Maintain one-to-one migration from Terraform configuration
Reviewed-on: #44
- Add externaldns base ArgoCD application with namespace and Vault integration
- Create externaldns overlay for au-syd1 with Helm chart configuration
- Update platform ApplicationSet to include externaldns deployment
- Configure external-dns v1.19.0 with RFC2136 provider for DNS updates
- Maintain one-to-one migration from Terraform configuration including TSIG secrets
Reviewed-on: #43
- Add cattle-system base ArgoCD application with namespace, Vault integration, and ingress
- Create cattle-system overlay for au-syd1 with Rancher Helm chart configuration
- Update platform ApplicationSet to include cattle-system deployment
- Update platform project to include Rancher Helm repository as source
- Configure Rancher v2.13.1 with HA, TLS, audit logging, and bootstrap secret from Vault
- Maintain one-to-one migration from Terraform configuration
Reviewed-on: #39
working towards a larger, redundant, autoscaling and simple puppet
implementation in kubernetes. this was originally based on the openvox
helm chart with several improvements (not all in this pr)
- use of cnpg instead of single bitnamilegacy postgres container
- use for g10k instead of r10k
- run one instance of g10k per namespace, instead of per-pod
- store only keep one copy of the environments/branches (instead of per-pod)
- change g10k to native cronjob instead of hacky implementation
- use vault secrets
part one adds:
- cnpg puppetdb pgsql cluster
- cnpg puppetdb pgpooler
- persistent volume claims for puppet, puppetdb, the code repository, etc
Reviewed-on: #25
Ben Vincent
unkinben
benvin