unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Enable ACLs on k8s consul with Vault-sourced bootstrap token #280

Merged
benvin merged 1 commits from benvin/consul-acls into main 2026-07-25 17:39:18 +10:00
Conversation 0 Commits 1 Files Changed 4
+56
unkinben commented 2026-07-25 17:06:38 +10:00
Owner
Copy Link
Copy Source

Why

Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with default_policy: deny and down_policy: extend-cache; the k8s cluster currently runs with ACLs disabled. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the same initial_management token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git.

Changes

  • Enable global.acls.manageSystemACLs so the chart manages system ACL tokens/policies for consul components.
  • Point global.acls.bootstrapToken at a pre-existing Kubernetes secret consul-bootstrap-acl-token (key token); chart 1.9.7 supports this, and when the secret is populated the server-acl-init job skips bootstrapping and adopts that token as the management token (renders -bootstrap-token-secret-name/-bootstrap-token-secret-key, verified in the kustomize output).
  • Add a VaultAuth (mount k8s/au/syd1, role default, SA default) and VaultStaticSecret in the consul namespace that sync kv/kubernetes/namespace/consul/default/bootstrap-acl-token into the consul-bootstrap-acl-token k8s secret via VSO (mirrors the encapi pattern).
  • Merge the acl block (enabled, default_policy: deny, down_policy: extend-cache, enable_token_persistence) into the server extraConfig to match the VM posture.

OPERATIONAL NOTE — required BEFORE merge

The user MUST place the VM cluster's initial_management token in Vault first, or ACL bootstrapping will generate a different token and break the mirror:

vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token>

VSO then syncs it into the consul-bootstrap-acl-token secret before the server-acl-init job runs. No terraform-vault change is needed: the wildcard default k8s-auth role (bound_service_account_namespaces: ['*']) plus the templated kv/kubernetes/default policy already grant the consul namespace default SA read on kv/kubernetes/namespace/consul/default/*.

Risk / expected behavior

  • Enabling ACLs rolls the 5 servers (StatefulSet update) and runs a server-acl-init job.
  • With default_policy: deny, previously-anonymous operations are denied. The vault service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace vault) via their service_registration "consul" stanza (catalog entry has ServiceMeta.external-source: vault, port 8200). After the flip this registration will be denied unless Vault is given a Consul ACL token with service:write on vault (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration.
  • Anonymous HTTP API reads (e.g. /v1/status/leader) will also be denied post-merge unless a token is supplied — see PR 2 verification note.

Ordering

Independent of the API-route PR (benvin/consul-api-route). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done.

## Why Phase 1 of the consul VM to k8s migration: bring the k8s consul cluster to ACL parity with the authoritative VM cluster before the eventual snapshot-restore. The VM cluster runs ACLs enabled with `default_policy: deny` and `down_policy: extend-cache`; the k8s cluster currently runs with **ACLs disabled**. Sourcing the bootstrap/management token from Vault lets the k8s cluster bootstrap with the **same** `initial_management` token as the VM cluster, so puppet automation and the snapshot-restore line up. No token material is placed in git. ## Changes - Enable `global.acls.manageSystemACLs` so the chart manages system ACL tokens/policies for consul components. - Point `global.acls.bootstrapToken` at a pre-existing Kubernetes secret `consul-bootstrap-acl-token` (key `token`); chart 1.9.7 supports this, and when the secret is populated the `server-acl-init` job **skips bootstrapping** and adopts that token as the management token (renders `-bootstrap-token-secret-name`/`-bootstrap-token-secret-key`, verified in the kustomize output). - Add a `VaultAuth` (mount `k8s/au/syd1`, role `default`, SA `default`) and `VaultStaticSecret` in the `consul` namespace that sync `kv/kubernetes/namespace/consul/default/bootstrap-acl-token` into the `consul-bootstrap-acl-token` k8s secret via VSO (mirrors the encapi pattern). - Merge the `acl` block (`enabled`, `default_policy: deny`, `down_policy: extend-cache`, `enable_token_persistence`) into the server `extraConfig` to match the VM posture. ## OPERATIONAL NOTE — required BEFORE merge The user MUST place the VM cluster's `initial_management` token in Vault first, or ACL bootstrapping will generate a *different* token and break the mirror: ``` vault kv put kv/kubernetes/namespace/consul/default/bootstrap-acl-token token=<VM initial_management token> ``` VSO then syncs it into the `consul-bootstrap-acl-token` secret before the `server-acl-init` job runs. No terraform-vault change is needed: the wildcard `default` k8s-auth role (`bound_service_account_namespaces: ['*']`) plus the templated `kv/kubernetes/default` policy already grant the `consul` namespace `default` SA read on `kv/kubernetes/namespace/consul/default/*`. ## Risk / expected behavior - Enabling ACLs **rolls the 5 servers** (StatefulSet update) and runs a `server-acl-init` job. - With `default_policy: deny`, previously-anonymous operations are denied. The `vault` service in the k8s catalog is self-registered by the Vault/OpenBao servers (namespace `vault`) via their `service_registration "consul"` stanza (catalog entry has `ServiceMeta.external-source: vault`, port 8200). After the flip this registration will be **denied** unless Vault is given a Consul ACL token with `service:write` on `vault` (and the anonymous token is not granted that). This must be handled as part of the cutover — either grant the anonymous token limited write, or configure a token in Vault's consul service_registration. - Anonymous HTTP API reads (e.g. `/v1/status/leader`) will also be denied post-merge unless a token is supplied — see PR 2 verification note. ## Ordering Independent of the API-route PR (`benvin/consul-api-route`). Both precede phase 3 (snapshot). Do not merge until the Vault step above is done.
unkinben added 1 commit 2026-07-25 17:06:38 +10:00
Enable ACLs on k8s consul with Vault-sourced bootstrap token
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
b7d976a82b 
Phase 1 of the consul VM->k8s migration: bring the k8s consul cluster to
ACL parity with the authoritative VM cluster before snapshot-restore. The
VM cluster runs ACLs enabled with default_policy deny and down_policy
extend-cache; the k8s cluster currently runs with ACLs disabled. Sourcing
the bootstrap/management token from Vault lets the k8s cluster bootstrap
with the SAME initial_management token as the VM cluster, so puppet
automation and the eventual snapshot-restore line up. No token material is
placed in git.

- Enable global.acls.manageSystemACLs so the chart manages system ACL
  tokens/policies for consul components.
- Point global.acls.bootstrapToken at a pre-existing Kubernetes secret
  consul-bootstrap-acl-token (key token); when populated the
  server-acl-init job skips bootstrapping and adopts that token as the
  management token.
- Add a VaultAuth (mount k8s/au/syd1, role default) and VaultStaticSecret
  in the consul namespace that sync kv/kubernetes/namespace/consul/default/
  bootstrap-acl-token into the consul-bootstrap-acl-token secret via VSO.
- Merge the acl block (enabled, default_policy deny, down_policy
  extend-cache, enable_token_persistence) into the server extraConfig to
  match the VM cluster posture.
benvin merged commit a633835550 into main 2026-07-25 17:39:18 +10:00
benvin deleted branch benvin/consul-acls 2026-07-25 17:39:19 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#280
Delete Branch "benvin/consul-acls"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?