unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Enable TLS on consul servers and drop WAN retry join #548

Closed
unkin-agent wants to merge 4 commits from benvin/consul-server-tls-fed into develop
pull from: benvin/consul-server-tls-fed
merge into: unkin:develop
Conversation 4 Commits 4 Files Changed 4
+17 -2
unkin-agent commented 2026-10-07 11:48:35 +11:00
Member
Copy Link
Copy Source

Consul servers need TLS and a gRPC TLS port before mesh gateways can federate the datacenters. The WAN retry join is dropped ahead of federation; drw1 has no servers yet, so nothing depends on it.

  • add a tls parameter to profiles::consul::server and set CA, cert and key from the vault PKI files
  • open grpc_tls on 8503
  • enable auto_reload_config so cert rotation reloads without a restart
  • set join_remote_regions: false in syd1 and drw1 to stop WAN retry join
Consul servers need TLS and a gRPC TLS port before mesh gateways can federate the datacenters. The WAN retry join is dropped ahead of federation; drw1 has no servers yet, so nothing depends on it. - add a `tls` parameter to profiles::consul::server and set CA, cert and key from the vault PKI files - open `grpc_tls` on 8503 - enable `auto_reload_config` so cert rotation reloads without a restart - set `join_remote_regions: false` in syd1 and drw1 to stop WAN retry join
unkin-agent commented 2026-10-07 11:52:29 +11:00
Author
Member
Copy Link
Copy Source
  • hieradata/roles/infra/storage/consul.yaml:19 — tls.defaults.cert_file is inherited by tls.grpc; with ports.grpc: 8502 and no ports.grpc_tls, Consul >=1.14 refuses to start ("the ports.grpc listener no longer supports TLS ... ports.grpc_tls is not defined"), so every server dies on its next puppet run → add grpc_tls: 8503 to profiles::consul::server::ports (plaintext 8502 stays as is).
  • site/profiles/manifests/consul/server.pp:135 — enable_mesh_gateway_wan_federation is hardcoded for every region, but drw1 hiera keeps join_remote_regions: true, rendering retry_join_wan, which Consul rejects alongside WAN fed ("'retry_join_wan' is incompatible") → set join_remote_regions: false in hieradata/country/au/region/drw1/infra/storage/consul.yaml or make the WAN-fed flag a param enabled only for syd1.
- hieradata/roles/infra/storage/consul.yaml:19 — `tls.defaults.cert_file` is inherited by `tls.grpc`; with `ports.grpc: 8502` and no `ports.grpc_tls`, Consul >=1.14 refuses to start ("the `ports.grpc` listener no longer supports TLS ... `ports.grpc_tls` is not defined"), so every server dies on its next puppet run → add `grpc_tls: 8503` to `profiles::consul::server::ports` (plaintext 8502 stays as is). - site/profiles/manifests/consul/server.pp:135 — `enable_mesh_gateway_wan_federation` is hardcoded for every region, but drw1 hiera keeps `join_remote_regions: true`, rendering `retry_join_wan`, which Consul rejects alongside WAN fed ("'retry_join_wan' is incompatible") → set `join_remote_regions: false` in hieradata/country/au/region/drw1/infra/storage/consul.yaml or make the WAN-fed flag a param enabled only for syd1.
unkin-agent commented 2026-10-07 11:59:18 +11:00
Author
Member
Copy Link
Copy Source
  • PR body — change list omits two diff changes: join_remote_regions: false also set in syd1 (hieradata/country/au/region/syd1/infra/storage/consul.yaml:5), and auto_reload_config => true added (site/profiles/manifests/consul/server.pp:117) → list both, with why (syd1: retry_join_wan is rejected with wanfed; auto_reload: picks up the 90-day cert renewals without a restart)
- PR body — change list omits two diff changes: `join_remote_regions: false` also set in syd1 (hieradata/country/au/region/syd1/infra/storage/consul.yaml:5), and `auto_reload_config => true` added (site/profiles/manifests/consul/server.pp:117) → list both, with why (syd1: retry_join_wan is rejected with wanfed; auto_reload: picks up the 90-day cert renewals without a restart)
unkin-agent commented 2026-10-07 12:02:14 +11:00
Author
Member
Copy Link
Copy Source
  • site/profiles/manifests/consul/server.pp:135 — enable_mesh_gateway_wan_federation switches cross-DC RPC/WAN gossip to gateway-only, but no mesh gateway exists until #549; syd1→drw1 traffic fails (ErrDCNotAvailable) from the first server restart → move the flag into #549 (after gateways are registered) or merge/roll #548+#549 together and say so in the rollout steps.
- site/profiles/manifests/consul/server.pp:135 — `enable_mesh_gateway_wan_federation` switches cross-DC RPC/WAN gossip to gateway-only, but no mesh gateway exists until #549; syd1→drw1 traffic fails (ErrDCNotAvailable) from the first server restart → move the flag into #549 (after gateways are registered) or merge/roll #548+#549 together and say so in the rollout steps.
unkin-agent changed title from Enable TLS and mesh gateway WAN federation on consul servers to Enable TLS on consul servers and drop WAN retry join 2026-10-09 19:31:19 +11:00
unkin-agent commented 2026-10-09 19:44:47 +11:00
Author
Member
Copy Link
Copy Source

No findings.

No findings.
benvin changed target branch from benvin/consul-server-fed-sans to develop 2026-10-09 21:51:52 +11:00
benvin added 4 commits 2026-10-09 21:51:52 +11:00
Add consul server federation SANs to consul server certs
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
f35b385714
Enable TLS and mesh gateway WAN federation on consul servers
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
9d25ba1d9a
Add consul grpc_tls port and drop drw1 WAN retry join
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
674f51a44b
Defer mesh gateway WAN federation to the gateway change
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
b1ee318394
unkin-agent closed this pull request 2026-10-09 22:30:42 +11:00
All checks were successful
ci/woodpecker/pr/ruby-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Required
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Required
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/puppet-prod#548
Delete Branch "benvin/consul-server-tls-fed"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?