v3.23.0 adds virtual alpine repositories and drops local files and metadata when a repo is deleted.
- pin api, ui and image-keeper to v3.23.0
Reviewed-on: #571
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v3.22.0 adds the vcs remote type for git provider source archives and release assets.
- pin api, ui and image-keeper to v3.22.0
Reviewed-on: #570
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
bind-operator v0.4.0 renders a forward BindZone without forwarders as `forwarders { };`, so the RBL direct-resolution zones no longer need raw zone clauses in the openforwarder view extraOptions.
- add rbl-zones.yaml with one forwarder-less forward BindZone per RBL domain
- remove the raw RBL zone lines from the openforwarder view
Reviewed-on: #569
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
encapi v0.2.0 authenticates writes with Vault identity OIDC tokens; it must validate the issuer, which is served with the internal Vault CA.
- bump encapi image to v0.2.0
- set ENCAPI_OIDC_ISSUER to the Vault identity OIDC issuer
- trust the Vault CA via a combined bundle and SSL_CERT_FILE, matching artifactapi
Reviewed-on: #567
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v3.18.0 adds the ansible Galaxy remote plus local and virtual ansible repositories.
- pin api, ui and image-keeper to v3.18.0
Reviewed-on: #562
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v3.16.0 adds local and virtual npm, the gems remote, HEAD on remotes, and fixes mutable refresh and per-request host handling.
- pin api, ui and image-keeper to v3.16.0
Reviewed-on: #560
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v3.15.0 adds local helm and puppet repos, the nuget remote and a shared virtual merge cache.
- pin api, ui and image-keeper to v3.15.0
Reviewed-on: #559
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
bind-operator v0.4.0 applies config changes with `rndc reconfig` instead of restarting pods, which records the result in a new `status.appliedConfig` field and a `ConfigApplied` condition. It also renders `forwarders { };` for forward zones with no forwarders.
- bump the bind-operator image to v0.4.0
- bump the CRD install URL to the v0.4.0 tag
Reviewed-on: #558
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The edge gateways now exist, so the smarthost and rspamd trust their real public IPs instead of the TEST-NET placeholders.
- set mta1 (66.226.148.207) and mta2 (66.226.149.50) in smarthost gateways.cidr
- allow both in the smarthost NetworkPolicy
- trust both in the rspamd external_relay ip_map
Reviewed-on: #556
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The lab resolver is authoritative for unkin.net and main.unkin.net, so names that exist only in public DNS return NXDOMAIN internally. The edge MTAs and the smarthost resolve these names through it.
- add mail-in.main.unkin.net A 103.216.191.185 (home WAN)
- add mta1.edge.unkin.net A 66.226.148.207
- add mta2.edge.unkin.net A 66.226.149.50
- records mirror public DNS
Reviewed-on: #557
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Recursing public names in-cluster SERVFAILs a few percent of cold lookups and loses the cache on every restart, which breaks CI. DNSBLs refuse or poison queries relayed via public resolvers, so rspamd's RBL lookups must keep resolving directly.
- add global forwarders 1.1.1.1 and 8.8.8.8 (forward first) to bind-resolvers
- add empty-forwarder zones to the openforwarder view for the RBL domains rspamd queries, so they resolve iteratively
Reviewed-on: #555
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The second mail edge site moves to BinaryLane Brisbane (VPC 172.18.32.0/24), whose hosts resolve through the internal resolvers over the vpn.
- add 172.18.32.0/24 to acl-dmz
Reviewed-on: #554
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The in-cluster BIND resolvers have no IPv6 route but still try every AAAA nameserver. Each attempt fails with "network unreachable" yet counts toward the per-query recursion budget, so cold lookups of names with many dual-stack NS (registry.opentofu.org, cloudfront, crates.io, gcr.io) hit `quota reached` and SERVFAIL, surfacing as transient DNS errors in CI.
- mark all IPv6 servers bogus in the openforwarder view so recursion is IPv4-only
Reviewed-on: #553
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
artifactapi v3.14.0 can require a Vault-issued bearer JWT for writes to /api/v2/remotes and /api/v2/virtuals, so config changes stop being open to anything in-cluster.
- bump artifactapi and artifactapi-ui images to v3.14.0
- set ARTIFACTAPI_OIDC_ISSUER to the Vault identity OIDC issuer and ARTIFACTAPI_OIDC_AUDIENCE to artifactapi
Requires terraform-artifactapi #64 merged first.
Reviewed-on: #552
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
BinaryLane only accepts RFC 1918 VPC ranges, so the syd and mel mail DMZ VPCs move from 198.18.30.0/24 and 198.18.31.0/24 to 172.18.30.0/24 and 172.18.31.0/24.
- replace the 198.18.30-31.0/24 entries in acl-dmz with 172.18.30.0/24 and 172.18.31.0/24
Reviewed-on: #551
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The edge MTAs present certificates from the internal Vault PKI, so the smarthost cannot verify mta1/mta2.edge.unkin.net against the system CA bundle.
- pin both edge relay TLS policies to the internal CA via `tafile=`
- mount `vault-ca-cert` into the smarthost pods and reload on change
---------
Co-authored-by: BenVincent <benvin@main.unkin.net>
Reviewed-on: #548
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
main.unkin.net is the test mail domain: it receives into the existing unkin.net mailboxes, and its outbound mail is DKIM-signed as d=unkin.net, which aligns under relaxed DMARC.
- relay main.unkin.net on the smarthost and route it to Stalwart over LMTP
- add main.unkin.net as a domain alias of unkin.net in Stalwart
Reviewed-on: #550
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Stalwart only files mail into Junk natively when its own spam filter marks it, so rspamd-tagged mail reached Junk only for accounts with a Sieve script, and first boot loads upstream DNSBLs, HTTP lists and URL lookups.
- enable the spam filter with one header rule scoring `X-Spam: Yes` above the spam threshold, covering every account including newly provisioned ones
- remove the upstream spam rules, tags, DNSBL servers, HTTP lists and lookup keys
- disable the classifier model and Pyzor
Reviewed-on: #544
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The mail edge VMs in the BinaryLane DMZ VPCs query the internal resolvers over the VPN, but the resolver match-clients ACLs do not include their ranges.
- add 198.18.30.0/24 (syd) and 198.18.31.0/24 (mel) to `acl-dmz`
Reviewed-on: #545
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The stalwart-apply PostSync Job fails with "Read-only file system": stalwart-cli caches the server schema under $HOME/.cache, which sits on the read-only root. The plan never applies, so Stalwart runs without its certificate, listeners or stdout tracer.
- mount a /tmp emptyDir in both Job containers
- set HOME=/tmp for the cli and kubectl (discovery cache)
Reviewed-on: #549
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Browsers coalesce HTTP/2 connections across *.unkin.net hosts sharing the traefik-external IP. A request for a haproxy-fronted host (e.g. git.unkin.net) on a grafana connection skips SNI passthrough and hits Traefik with no HTTP router, returning 404.
- replace the TLS passthrough listener with an HTTPS listener terminating the three LE wildcards
- replace the haproxy TLSRoute with an HTTPRoute for the same hosts
- add a BackendTLSPolicy re-encrypting to haproxy:443
Reviewed-on: #547
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Stalwart is the mailbox server for the mail migration and runs multi-node from the start, so it needs a coordinator and a shared in-memory store.
- run Stalwart as a 3-replica StatefulSet with a headless Service, topology spread and a PDB
- add the `mail-valkey` ValkeyCluster as in-memory store and coordinator
- keep data in CNPG Postgres and blobs in cephrgw S3
- converge settings from plan.ndjson in a PostSync Job, then roll the StatefulSet
- add a global `spam-to-junk` Sieve script filing rspamd-tagged mail into Junk and stopping
- exempt LMTP from the inbound throttles, as all inbound mail comes from the smarthost
- label pods as smarthost clients and move the client LoadBalancer to 198.18.200.18
Reviewed-on: #539
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
DNSBLs such as Spamhaus refuse queries that arrive via public resolvers, and forwarding everything to 8.8.8.8/1.1.1.1 leaves the estate dependent on them. The lab resolvers now do full recursion themselves (DNSSEC-validated via the operator default `dnssec-validation auto` and BIND built-in root hints).
- drop the global forwarders from the bind-resolvers BindCluster
- keep the per-zone forwarders for unkin.net, consul, k8s and reverse zones unchanged
Reviewed-on: #546
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The smarthost routes all mail through one rspamd: inbound that survived the edge gets the AI (LiteLLM Haiku) pass and is tagged for Stalwart's Junk filing; outbound from the lab and pods is DKIM-signed.
- add rspamd (milter via ClusterIP only), ClamAV and a 3-node sentinel Valkey
- tag inbound (`X-Spam`, `X-Spam-Status`, `X-Spamd-Result`), reject only viruses, tempfail when ClamAV fails
- score inbound against the hop before the edge relays (`external_relay`)
- DKIM-sign and skip GPT only for the smarthost's mynetworks sources
- elect the highest-offset Valkey on full restart; RDB, 384mb `volatile-ttl`
- read a hashed controller password (`password_hash`)
Requires `password_hash` in kv `mail/default/rspamd-controller`.
Reviewed-on: #542
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Mail needs one routing hub between the edge gateways, Stalwart and the lab: inbound to Stalwart over LMTP, outbound to the gateways, all of it through rspamd.
- add a 2-replica postfix StatefulSet `mail-in.main.unkin.net` (50Gi spool per pod, node spread, PDB)
- serve port 25 with the LE `*.main.unkin.net` wildcard, reflected into `mail`
- gateways (TLS, XFORWARD) relay to relay_domains with LMTP recipient verification; lab and pods relay anywhere
- pass all mail through the rspamd milter, tempfailing when it is down
- limit port 25 via NetworkPolicy to gateways, lab and `mail.unkin.net/smarthost-client=true` pods
- relay outbound via mta1/mta2.edge.unkin.net with verified TLS; add ClusterIP + LB 198.18.200.16
Requires #539 merged.
Reviewed-on: #541
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The internal A record for mail.unkin.net used 198.18.200.11, which Consul (#531) claims; Stalwart's LoadBalancer moved to 198.18.200.18 in #539.
- point mail.unkin.net A record at 198.18.200.18
Merge with or after #539.
Reviewed-on: #543
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Mail clients on the lab network reach Stalwart at mail.unkin.net, served by the stalwart-lb PureLB VIP.
- add an internal unkin.net A record for mail pointing at 198.18.200.11
Requires argocd-apps #539 for the VIP to answer.
Reviewed-on: #540
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Stalwart needs a Postgres metadata store and an S3 blob store before it can be deployed into the new `mail` namespace.
- add CNPG Cluster `stalwart-db` (db/owner `stalwart`, CNPG-minted `stalwart-db-app` creds) with barman backups to a dedicated `cnpg-mail` bucket and a nightly ScheduledBackup
- add cephrgw Bucket `stalwart-blobs` with an owner user writing RW keys to `stalwart-blobs-s3`
- add the au-syd1 overlay, platform ApplicationSet path and project destination
Reviewed-on: #538
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The new terraform-binarylane repo needs a Woodpecker ServiceAccount for its CI pipelines to authenticate to Vault.
- add serviceaccount_terraform_binarylane.yaml in the woodpecker namespace
- add it to the woodpecker base kustomization
Reviewed-on: #537
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v0.5.0 answers AST name queries for pdbmux_source and reports the backend host as its value.
- bump pdbmux image to v0.5.0
Reviewed-on: #536
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
artifactapi pulls its own images from itself, so a cold boot with every api pod down cannot pull them. Kubelet GC never removes images referenced by pods on the node, so a pod on every node holding the pinned tags keeps them local.
- add image-keeper DaemonSet referencing the api and ui images as no-op init containers
- run a static busybox as `true` inside the distroless api image
- tolerate all taints, run at low priority with tiny non-root resources
Reviewed-on: #535
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v3.13.1 adds rpm virtual repos, the cargo remote type and GitHub scan retry; its images now publish to docker-internal instead of the Gitea registry.
- pin api and ui to docker-internal/artifactapi{,-ui}:v3.13.1
Reviewed-on: #534
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The v3.13.0 images were never published (registry push failed), so the current pin cannot be pulled.
- pin api and ui images back to v3.12.0
Reviewed-on: #532
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Bump artifactapi API and UI images from v3.12.0 to v3.13.0.
- Update git.unkin.net/unkin/artifactapi to v3.13.0
- Update git.unkin.net/unkin/artifactapi-ui to v3.13.0
Reviewed-on: #529
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The standalone k8s Consul runs its own raft under DC `au-syd1`, the same DC name as the VM cluster that k8s servers are about to join. Its leftover state risks a split brain, so it goes before the replacement lands.
- remove `apps/overlays/au-syd1/consul`, dropping `platform-consul` from the platform ApplicationSet and cascading deletion of the `consul` namespace and PVCs
- keep `apps/base/consul` for reuse by the replacement
Reviewed-on: #530
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
v0.1.7 stops the operator reconcile hot-loop, which was hot-reloading Kea about 9 times a second and flooding the dhcp-system logs. Images now publish to docker-internal.
- bump kea-operator, kea and kea-api to v0.1.7
- pull all three from artifactapi docker-internal
Reviewed-on: #527
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The consul overlay fetches its chart directly from helm.releases.hashicorp.com, while the other 15 helm overlays on the estate mirror use the artifactapi virtual helm repo, which already proxies hashicorp-helm.
- point consul helmCharts repo at the artifactapi virtual helm repo
Reviewed-on: #528
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Querying the VLCluster from outside the cluster (CLI tools, puppet VMs) needs a resolvable TLS endpoint for vlselect, mirroring the existing logs-ingest endpoint for vlinsert.
- add `vlselect` Gateway on `traefik-internal` with vault-issuer TLS and external-dns for `vlselect.k8s.syd1.au.unkin.net`
- add HTTP->HTTPS redirect route and HTTPS route to `vlselect-logs:9471`, unauthenticated
Reviewed-on: #525
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
authentik-server requests only 50m CPU, so on busy nodes it is starved, `/-/health/live/` exceeds the 3s liveness timeout and kubelet restarts it (~100 restarts per pod).
- raise `server.resources.requests.cpu` from 50m to 500m
Reviewed-on: #526
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Compiler pods average ~2Gi memory but request 1Gi, so the scheduler overpacks nodes and the HPA's utilisation target is skewed.
- raise compiler requests to 1 cpu / 2Gi and limits to 4 cpu / 4Gi
- scale up faster (30s window, 75%) and down slower (600s window)
Reviewed-on: #524
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
The VM haproxies are being decommissioned; the sites they published via puppet CNAMEs are already served by the k8s haproxy edge, so k8s DNS publishes them instead.
- add main.unkin.net A records for sonarr, radarr, lidarr, readarr, prowlarr, nzbget, jellyfin -> 198.18.199.0
- add unkin.net A records for fafflix and git (git-edge.yaml) -> 198.18.199.0, auth -> 198.18.200.4
- point the commented git.yaml cutover at replacing git-edge.yaml
Merge alongside puppet-prod halb DNS removal; A records only take effect once the puppet CNAMEs are gone.
Reviewed-on: #523
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
grafana.unkin.net still routes through the puppet haproxy edge to the old grafana VMs; the k8s grafana should serve it directly like identity and vlogs.
- add grafana-external Gateway (traefik-external, *.unkin.net wildcard) with redirect + main HTTPRoutes
- reflect wildcard-unkin-net-tls into grafana
- set grafana root_url to https://grafana.unkin.net
- add grafana A record -> 198.18.199.0 in the bind-operator unkin.net zone
- drop grafana.unkin.net from the k8s haproxy routes and config
Requires terraform-authentik grafana redirect URI PR applied first, and the puppet halb vrrp_cnames grafana.unkin.net CNAME removed.
Reviewed-on: #522
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
artifactapi v3.12.0 makes DELETE eviction clear every cache layer, so stale mutable indexes (e.g. epel repomd) no longer survive eviction. It also adds `?prefix=` filtering to object listings.
- bump artifactapi and artifactapi-ui images to v3.12.0
Reviewed-on: #521
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Both traefik HPAs report `ScalingActive False: missing request for cpu` because the chart values set no container resources, so the CPU-utilisation metric can't be computed and neither Deployment ever scales.
- traefik-external requests 200m CPU / 128Mi, memory limit 512Mi
- traefik-internal requests 500m CPU / 256Mi, memory limit 1Gi
- No CPU limits, to avoid throttling the proxy
- The chart derives GOMEMLIMIT (90% of the memory limit) from the new limits
Reviewed-on: #520
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
unkin-agent