From 9e7687fccb94d97e107763489e828dbccaa5f2af Mon Sep 17 00:00:00 2001 From: Ben Vincent Date: Wed, 12 Aug 2026 00:03:13 +1000 Subject: [PATCH] Mint the netbox user-management credential dynamically from the single admin token (#119) ## Why netbox_user_management authenticates to NetBox to reconcile service users + permissions on every apply. It must not depend on a second static admin token, and it must not break when the engine rotates its admin seed (`netbox/config/rotate` mints a fresh admin token and deletes the old one). The durable shape: keep exactly ONE static admin token, and have the netbox engine mint an ephemeral, user-admin-capable token that the e-breuninger provider uses to manage users. ## How - `module.netbox_user_mgmt_role` creates `netbox/roles/vault-user-mgmt`, a write-enabled role for a pre-existing NetBox superuser named by `user_mgmt_username`. Minted tokens authenticate AS that superuser (NetBox tokens carry no scope beyond `write_enabled`; the user's permissions apply), so they can create users. - `netbox_user_management` reads `netbox/creds/vault-user-mgmt` and configures the netbox provider with the minted token. When `user_mgmt_username` is unset it falls back to the single static `admin_token` (a `check` block warns that rotation would then break it) - a bootstrap/degraded path, never a second static token. - Grant the deployer `read` on `netbox/creds/vault-user-mgmt` (the one deliberate exception to the admin policy's `netbox/creds/*` exclusion). - Keep the bare-token + `token_version`-match postconditions on the single static admin token. ## Feasibility constraints (worked through, documented in-module) 1. **The engine CAN mint a user-admin token** - roles map to a pre-existing user with only a `write_enabled` gate (`vault-plugin-secrets-netbox` `path_roles.go`, `client.go` `MintToken`); point it at a superuser and minted tokens can manage users. 2. **Token transits state.** The hashicorp/vault provider (5.6.0) exposes ephemeral resources for KV only, not dynamic engine creds, so the mint is read via the `vault_generic_secret` DATA source: the short-lived token is written to state (sensitive, lease-revoked) and re-minted each plan. Migrate to an ephemeral resource once the vault provider ships a dynamic-secret one. 3. **A clean single fresh apply is not possible.** A provider cannot be configured from a role created in the same run (data sources don't defer; OpenTofu 1.11 defers only ephemeral resources, which the vault provider doesn't offer here). So enabling the dynamic path on a backend needs a one-time targeted bootstrap of the mount + role, then normal applies. Documented in `config/netbox_secret_backend/netbox.yaml`. ## Operator follow-up - Repair the live mount first (unchanged): `vault write netbox/config token=` (the mount uses `ignore_changes=[token]`), keep `token_version=2`. - To enable dynamic minting: set `user_mgmt_username` to the pre-existing superuser, apply the deployer creds policy, then bootstrap once: `tofu apply -target=...netbox_secret_backend -target=...netbox_user_mgmt_role`, then apply normally. Until then user management stays on the static token (non-breaking, with a warning). Reviewed-on: https://git.unkin.net/unkin/terraform-vault/pulls/119 Co-authored-by: Ben Vincent Co-committed-by: Ben Vincent --- config/netbox_secret_backend/netbox.yaml | 44 ++++++++-- modules/vault_cluster/main.tf | 36 ++++++++- .../modules/netbox_secret_backend/main.tf | 21 +++++ .../modules/netbox_user_management/main.tf | 80 +++++++++++++++++-- .../netbox_user_management/variables.tf | 3 + modules/vault_cluster/variables.tf | 6 ++ policies/netbox/creds/vault-user-mgmt.yaml | 23 ++++++ 7 files changed, 198 insertions(+), 15 deletions(-) create mode 100644 policies/netbox/creds/vault-user-mgmt.yaml diff --git a/config/netbox_secret_backend/netbox.yaml b/config/netbox_secret_backend/netbox.yaml index 4e22f42..9c2ff49 100644 --- a/config/netbox_secret_backend/netbox.yaml +++ b/config/netbox_secret_backend/netbox.yaml @@ -1,16 +1,48 @@ # Mounts the netbox token secrets engine at "netbox" and writes its config. # The seeded NetBox admin token is sensitive and read from KV, not stored here: # kv/service/vault/au/syd1/secret_backend/netbox/config -# -> key: admin_token (required) -# Populate that KV path with a purpose-built NetBox service token that has -# add_token + grant_token (or superuser) BEFORE applying, then run -# `vault write -f netbox/config/rotate` after the first apply so only Vault -# holds the live admin token. +# -> key: admin_token (required) the SINGLE static admin credential +# +# admin_token must be a BARE NetBox token with NO scheme prefix: do not prepend +# "Bearer " or "Token ". NetBox infers the version from the value's nbt_ prefix, +# so one bare token authenticates under either scheme; the plugin adds the keyword +# itself. A prefixed value yields a malformed header + 403. +# +# Populate admin_token with a purpose-built NetBox superuser token (add_user + +# add_token + grant_token, or superuser) BEFORE applying, then run +# `vault write -f netbox/config/rotate` after the first apply so only Vault holds +# the live admin token. +# +# Only ONE static admin token exists. netbox_user_management does NOT re-read this +# token; instead the engine mints it a short-lived user-admin token per apply from +# netbox/roles/vault-user-mgmt (see user_mgmt_username below), so rotating +# admin_token never breaks user management. Set user_mgmt_username to the +# pre-existing NetBox superuser the static admin_token belongs to (or another +# superuser). Leaving it unset falls back to using admin_token directly, which is +# only a bootstrap/degraded path and breaks after rotation. +# +# Bootstrap ordering: the vault-user-mgmt role must exist before the netbox +# provider is configured from its creds, so on a brand-new backend apply the mount +# + role first (e.g. `tofu apply -target=...netbox_secret_backend +# -target=...netbox_user_mgmt_role`) once, then apply normally. # # token_version 2 is the NetBox 4.6.5 default and requires API_TOKEN_PEPPERS to # be configured on the NetBox server; set token_version: 1 here if the server -# has no peppers. +# has no peppers. token_version does NOT change how the plugin authenticates its +# own calls (that scheme comes from the admin_token value's nbt_ prefix); it only +# sets the version of the per-user tokens the engine mints. It must still MATCH +# the admin_token kind: nbt_ v2 token -> token_version 2; bare v1 token -> 1. +# +# The mount uses ignore_changes=[token], so editing KV alone does NOT reach the +# live mount. To push a corrected/rotated admin token into a running mount: +# vault write netbox/config token= +# (netbox_url/token_version are preserved on a partial update). Do NOT -replace +# the mount to force a re-read - that recreates it and drops all roles/config. description: "NetBox ephemeral scoped API token engine" netbox_url: "https://netbox.k8s.syd1.au.unkin.net" token_version: 2 request_timeout_seconds: 30 +# Set to the pre-existing NetBox superuser admin_token belongs to, to mint the +# user-management credential dynamically (recommended). Until set, user management +# uses admin_token directly and a check block warns that rotation will break it. +# user_mgmt_username: "vault-netbox-admin" diff --git a/modules/vault_cluster/main.tf b/modules/vault_cluster/main.tf index bfd7e0d..459a4f5 100644 --- a/modules/vault_cluster/main.tf +++ b/modules/vault_cluster/main.tf @@ -475,10 +475,34 @@ module "netbox_secret_backend" { depends_on = [module.plugin] } +# Dedicated engine role that mints an ephemeral, user-admin-capable token for the +# pre-existing NetBox superuser named on each backend (user_mgmt_username). +# netbox_user_management reads netbox/creds/vault-user-mgmt from it, so it +# authenticates with a short-lived Vault-minted token derived from the single +# static admin token - never a second static credential, and unaffected by +# rotation of the engine's admin seed. Created before user management so the role +# exists when it reads creds. +module "netbox_user_mgmt_role" { + source = "./modules/netbox_secret_backend_role" + + for_each = { for k, v in var.netbox_secret_backend : k => v if v.user_mgmt_username != null } + + backend = each.key + name = "vault-user-mgmt" + netbox_username = each.value.user_mgmt_username + write_enabled = true + description = "Ephemeral user-admin token for netbox_user_management (Vault-minted per apply)" + ttl = 600 + max_ttl = 1200 + + depends_on = [module.netbox_secret_backend] +} + # Declaratively manage the NetBox service users + object permissions the engine -# roles mint tokens for, using the seeded admin token (mirrors consul_acl_management). -# Consumes the SAME role config as netbox_secret_backend_role: one file per identity, -# filename-derived username, inline permissions. +# roles mint tokens for, authenticating with the Vault-minted user-admin token +# above (mirrors consul_acl_management). Consumes the SAME role config as +# netbox_secret_backend_role: one file per identity, filename-derived username, +# inline permissions. module "netbox_user_management" { source = "./modules/netbox_user_management" @@ -487,6 +511,12 @@ module "netbox_user_management" { netbox_backends = var.netbox_secret_backend netbox_roles = var.netbox_secret_backend_role netbox_backend_aliases = var.netbox_backend_aliases + + # This module declares its own netbox provider, so it is a legacy module and + # cannot take depends_on. Ordering vs the vault-user-mgmt role is not needed on + # steady state (the role pre-exists, so reading its creds succeeds regardless); + # on first enablement the role must be created first via the one-time targeted + # bootstrap documented in config/netbox_secret_backend/netbox.yaml. } module "netbox_secret_backend_role" { diff --git a/modules/vault_cluster/modules/netbox_secret_backend/main.tf b/modules/vault_cluster/modules/netbox_secret_backend/main.tf index 19c99b0..585d6d9 100644 --- a/modules/vault_cluster/modules/netbox_secret_backend/main.tf +++ b/modules/vault_cluster/modules/netbox_secret_backend/main.tf @@ -8,6 +8,27 @@ data "vault_kv_secret_v2" "config" { mount = "kv" name = "service/vault/${var.country}/${var.region}/secret_backend/${var.path}/config" + + lifecycle { + # The plugin builds its own Authorization header from the token VALUE, not + # token_version: a value starting with the nbt_ prefix is sent as + # "Bearer " (v2), otherwise "Token " (v1). So admin_token must + # be the BARE token - a literal `Bearer `/`Token ` scheme prefix yields a + # malformed three-part header and 403s on the plugin's own NetBox calls. + # + # token_version does NOT change that header; it only selects the version of + # the per-user tokens the engine mints for roles. It must still MATCH the + # admin token's kind so the mount and its minted creds line up: an nbt_ v2 + # admin token pairs with token_version=2, a bare v1 token with token_version=1. + postcondition { + condition = nonsensitive( + !startswith(self.data["admin_token"], "Bearer ") && + !startswith(self.data["admin_token"], "Token ") && + startswith(self.data["admin_token"], "nbt_") == (var.token_version == 2) + ) + error_message = "KV admin_token for netbox backend '${var.path}' must be a BARE NetBox token with no 'Bearer '/'Token ' scheme prefix, AND its version must match token_version: a v2 token (nbt_.) requires token_version=2; a v1 token (bare 40-char value) requires token_version=1." + } + } } resource "netbox_secret_backend" "this" { diff --git a/modules/vault_cluster/modules/netbox_user_management/main.tf b/modules/vault_cluster/modules/netbox_user_management/main.tf index a5692d0..c79c813 100644 --- a/modules/vault_cluster/modules/netbox_user_management/main.tf +++ b/modules/vault_cluster/modules/netbox_user_management/main.tf @@ -1,20 +1,88 @@ -# Read the seeded NetBox admin token for each backend from KV. This is the same -# token the netbox secrets engine is configured with (key admin_token), and it -# must carry add_token + grant_token (or superuser) to create users/permissions. +# netbox_user_management reconciles NetBox service users + permissions on every +# apply, so it needs an admin credential each run. That credential is minted +# DYNAMICALLY by the netbox engine from the SINGLE static admin token, so no +# second static credential exists and it survives rotation of the engine seed: +# +# 1. module.netbox_user_mgmt_role creates netbox/roles/vault-user-mgmt, a +# write-enabled role for a pre-existing NetBox superuser (user_mgmt_username). +# 2. Reading netbox/creds/vault-user-mgmt mints a short-lived, user-admin-capable +# token for that superuser; the e-breuninger provider uses it to CRUD users. +# +# The hashicorp/vault provider ships ephemeral resources for KV only, not for +# dynamic engine creds, so the mint is read via the vault_generic_secret DATA +# source: the short-lived token transits Terraform state (sensitive, lease-revoked) +# and is re-minted each plan. This is the closest single-static-token shape the +# current providers allow; move to an ephemeral resource once the vault provider +# ships a dynamic-secret one. Ordering note: the vault-user-mgmt role must already +# exist when this reads creds, so on a brand-new backend bootstrap the mount + +# role first (targeted apply) - a fresh single apply cannot configure the netbox +# provider from a role created in the same run. +locals { + # Backends that mint a dynamic user-admin token (a pre-existing superuser named). + netbox_dynamic_backends = { for k, v in var.netbox_backends : k => v if v.user_mgmt_username != null } + # Backends still using the single static admin_token directly (until a superuser + # is named). Bootstrap/degraded path - the same one token, not a second static. + netbox_static_backends = { for k, v in var.netbox_backends : k => v if v.user_mgmt_username == null } +} + +# Dynamic path: the engine mints a user-admin token for the superuser. Requires +# the deployer to read netbox/creds/vault-user-mgmt (policies/netbox/creds). +data "vault_generic_secret" "user_admin" { + for_each = local.netbox_dynamic_backends + + path = "${each.key}/creds/vault-user-mgmt" +} + +# Static fallback: the single admin_token from KV, used only until a superuser is +# named. NetBox derives the token version from the value's `nbt_` prefix, not the +# keyword, so the same BARE token works under either scheme; reject a value that +# carries a literal `Bearer `/`Token ` scheme prefix (a malformed header -> 403). data "vault_kv_secret_v2" "netbox_backend_configs" { - for_each = var.netbox_backends + for_each = local.netbox_static_backends mount = "kv" name = "service/vault/${var.country}/${var.region}/secret_backend/${each.key}/config" + + lifecycle { + postcondition { + condition = nonsensitive( + !startswith(self.data["admin_token"], "Bearer ") && + !startswith(self.data["admin_token"], "Token ") + ) + error_message = "KV admin_token for netbox backend '${each.key}' must be a BARE NetBox token with no 'Bearer '/'Token ' scheme prefix (v2: nbt_.; v1: the 40-char value)." + } + } } -# One NetBox provider instance per backend, authenticated with its admin token. +# Warn (non-fatal) for any backend still on the static token: rotating the engine +# admin seed would then break user management. Set user_mgmt_username to switch to +# the dynamic, rotation-proof mint. +check "netbox_user_mgmt_dynamic" { + assert { + condition = length(local.netbox_static_backends) == 0 + error_message = "A netbox backend has no user_mgmt_username, so user management uses the static admin_token directly and will break if that token is rotated (netbox/config/rotate). Set user_mgmt_username to a pre-existing NetBox superuser to mint the credential dynamically." + } +} + +locals { + # Per backend: the dynamically-minted user-admin token, else the static seed. + netbox_admin_tokens = { + for k, v in var.netbox_backends : k => ( + v.user_mgmt_username != null + ? data.vault_generic_secret.user_admin[k].data["token"] + : data.vault_kv_secret_v2.netbox_backend_configs[k].data["admin_token"] + ) + } +} + +# One NetBox provider instance per backend, authenticated with its (dynamic or +# static) admin token. provider "netbox" { alias = "by_backend" for_each = var.netbox_backend_aliases server_url = var.netbox_backends[each.key].netbox_url - api_token = data.vault_kv_secret_v2.netbox_backend_configs[each.key].data["admin_token"] + api_token = local.netbox_admin_tokens[each.key] allow_insecure_https = var.netbox_backends[each.key].tls_skip_verify # NetBox is internal and not always reachable at plan time; the resource CRUD # calls surface any real incompatibility, so skip the startup version probe. diff --git a/modules/vault_cluster/modules/netbox_user_management/variables.tf b/modules/vault_cluster/modules/netbox_user_management/variables.tf index 051907c..313d48d 100644 --- a/modules/vault_cluster/modules/netbox_user_management/variables.tf +++ b/modules/vault_cluster/modules/netbox_user_management/variables.tf @@ -3,6 +3,9 @@ variable "netbox_backends" { type = map(object({ netbox_url = string tls_skip_verify = optional(bool, false) + # Pre-existing NetBox superuser the engine mints a dynamic user-admin token + # for; unset means fall back to the static admin_token from KV. + user_mgmt_username = optional(string) })) } diff --git a/modules/vault_cluster/variables.tf b/modules/vault_cluster/variables.tf index be7b04d..01814f8 100644 --- a/modules/vault_cluster/variables.tf +++ b/modules/vault_cluster/variables.tf @@ -426,6 +426,12 @@ variable "netbox_secret_backend" { ca_cert = optional(string) tls_skip_verify = optional(bool, false) request_timeout_seconds = optional(number, 30) + # Pre-existing NetBox superuser (or add_user + add_token + grant_token) the + # engine mints an ephemeral user-admin token for, so netbox_user_management + # authenticates with a Vault-minted credential derived from the single static + # admin token instead of a second static one. Unset = use the static + # admin_token directly (bootstrap/degraded; breaks after admin-token rotation). + user_mgmt_username = optional(string) })) default = {} } diff --git a/policies/netbox/creds/vault-user-mgmt.yaml b/policies/netbox/creds/vault-user-mgmt.yaml new file mode 100644 index 0000000..ec02ea7 --- /dev/null +++ b/policies/netbox/creds/vault-user-mgmt.yaml @@ -0,0 +1,23 @@ +# Allow the vault deployer to mint the ephemeral user-admin token that +# netbox_user_management authenticates with (netbox/creds/vault-user-mgmt). The +# engine mints it from the single static admin token, so the deployer never holds +# a second static NetBox credential. +# +# The netbox admin policy (policies/netbox/admin.yaml) deliberately excludes +# netbox/creds/* - minting is normally for consumers, not the deployer. This is +# the one deliberate exception: the deployer needs a user-admin token during the +# run to reconcile NetBox users. Scoped to the single vault-user-mgmt role only. +# +# Bound to the same principals as the admin policy: the tf_vault AppRole and its +# Woodpecker k8s auth role. +--- +rules: + - path: "netbox/creds/vault-user-mgmt" + capabilities: + - read + +auth: + approle: + - tf_vault + k8s/au/syd1: + - woodpecker_terraform_vault