Phase 2 of the consul migration: expose the HTTP API (not just the UI) at consul.k8s.syd1.au.unkin.net, now rebased onto main post-#280 (ACLs enabled).
- Adds a consul-http ClusterIP service targeting the server pods on 8500 (API + UI share the port, so the UI stays reachable at /ui/)
- Repoints the consul and consul-svc HTTPRoutes from consul-ui:80 to consul-http:8500
- Documents ACL-authenticated access in apps/base/consul/README.md: token from kv/kubernetes/namespace/consul/default/bootstrap-acl-token (VSO-synced), X-Consul-Token curl and consul CLI usage, UI token login, and the prefer-vault-minted-tokens note
Verification post-merge (ACLs are live, so authenticated): CONSUL_HTTP_TOKEN=$(vault kv get -field=token kv/kubernetes/namespace/consul/default/bootstrap-acl-token) && curl -H "X-Consul-Token: $CONSUL_HTTP_TOKEN" https://consul.k8s.syd1.au.unkin.net/v1/status/leader
Rollback: revert the HTTPRoute backends to consul-ui:80.
Reviewed-on: #282
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with `default_policy: deny` and `down_policy: extend-cache`; the k8s cluster currently runs with **ACLs disabled**. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the **same** `initial_management` token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git.
## Changes
- Enable `global.acls.manageSystemACLs` so the chart manages system ACL tokens/policies for consul components.
- Point `global.acls.bootstrapToken` at a pre-existing Kubernetes secret `consul-bootstrap-acl-token` (key `token`); chart 1.9.7 supports this, and when the secret is populated the `server-acl-init` job **skips bootstrapping** and adopts that token as the management token (renders `-bootstrap-token-secret-name`/`-bootstrap-token-secret-key`, verified in the kustomize output).
- Add a `VaultAuth` (mount `k8s/au/syd1`, role `default`, SA `default`) and `VaultStaticSecret` in the `consul` namespace that sync `kv/kubernetes/namespace/consul/default/bootstrap-acl-token` into the `consul-bootstrap-acl-token` k8s secret via VSO (mirrors the encapi pattern).
- Merge the `acl` block (`enabled`, `default_policy: deny`, `down_policy: extend-cache`, `enable_token_persistence`) into the server `extraConfig` to match the VM posture.
## OPERATIONAL NOTE — required BEFORE merge
The user MUST place the VM cluster's `initial_management` token in Vault first, or ACL bootstrapping will generate a *different* token and break the mirror:
```
vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token>
```
VSO then syncs it into the `consul-bootstrap-acl-token` secret before the `server-acl-init` job runs. No terraform-vault change is needed: the wildcard `default` k8s-auth role (`bound_service_account_namespaces: ['*']`) plus the templated `kv/kubernetes/default` policy already grant the `consul` namespace `default` SA read on `kv/kubernetes/namespace/consul/default/*`.
## Risk / expected behavior
- Enabling ACLs **rolls the 5 servers** (StatefulSet update) and runs a `server-acl-init` job.
- With `default_policy: deny`, previously-anonymous operations are denied. The `vault` service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace `vault`) via their `service_registration "consul"` stanza (catalog entry has `ServiceMeta.external-source: vault`, port 8200). After the flip this registration will be **denied** unless Vault is given a Consul ACL token with `service:write` on `vault` (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration.
- Anonymous HTTP API reads (e.g. `/v1/status/leader`) will also be denied post-merge unless a token is supplied — see PR 2 verification note.
## Ordering
Independent of the API-route PR (`benvin/consul-api-route`). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done.
Reviewed-on: #280
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Summary
- Adds `group: gateway.networking.k8s.io` and `kind: Gateway` to all `parentRefs` entries
- Adds `group: ""`, `kind: Service`, and `weight: 1` to all `backendRefs` entries
- Affects 9 HTTPRoute files across artifactapi, cattle-system, consul, kanidm, litellm, paperclip, puppet, and vault
## Why
ArgoCD diffs the desired manifest against the live Kubernetes object. The Gateway API controller defaults these fields when creating/updating objects, so the live state always has them — causing persistent OutOfSync for every HTTPRoute. Same root cause as #153 (certificateRefs).
## Test plan
- [ ] All affected ArgoCD applications show Synced after merge
Reviewed-on: #162
The Gateway API admission server defaults certificateRefs[].group to ""
when it is omitted. ArgoCD diffed the desired state (no group field) against
the live state (group: "") and flagged every gateway as out of sync.
Fix: explicitly set group: "" in all certificateRefs entries so the
rendered manifest matches the API server's canonical form exactly.
Affected: artifactapi, cattle-system, consul, litellm, paperclip,
puppet (puppetboard + puppetdb), vault.
Reviewed-on: #153
## Summary
- Deploys HashiCorp Consul 1.22.7 using Helm chart 1.9.7 with 5 server replicas
- Configuration modelled on production consul: \`datacenter=au-syd1\`, \`connect=true\`, \`raft_multiplier=10\`, HTTP on 8500, GRPC on 8502, HTTPS disabled
- 5-replica server cluster with \`bootstrapExpect=5\`
- 10Gi cephrbd-fast-delete PVC per server pod
- Gateway API: HTTPS gateway + HTTPRoute (443→consul-consul-ui:80→8500) at \`consul.k8s.syd1.au.unkin.net\`
- PodDisruptionBudget patched from \`policy/v1beta1\` to \`policy/v1\` (k8s 1.25+ compatibility)
- ArgoCD platform ApplicationSet updated to include consul overlay path
- Clients disabled (server-only deployment)
- ConnectInject disabled (can be enabled later for service mesh)
## Requires
- PR #147 (artifactapi: add hashicorp/consul to docker immutable patterns) to be merged first
## Test plan
- [ ] Sandbox tested in \`sandbox-consul\`: all 5 server pods 1/1 Running, cluster formed
- [ ] After merge: ArgoCD syncs consul namespace
- [ ] Verify \`consul.k8s.syd1.au.unkin.net\` is accessible via Gateway
Reviewed-on: #149
unkinben