Mint the netbox user-management credential dynamically from the single admin token (#119)

## Why

netbox_user_management authenticates to NetBox to reconcile service users + permissions on every apply. It must not depend on a second static admin token, and it must not break when the engine rotates its admin seed (`netbox/config/rotate` mints a fresh admin token and deletes the old one). The durable shape: keep exactly ONE static admin token, and have the netbox engine mint an ephemeral, user-admin-capable token that the e-breuninger provider uses to manage users.

## How

- `module.netbox_user_mgmt_role` creates `netbox/roles/vault-user-mgmt`, a write-enabled role for a pre-existing NetBox superuser named by `user_mgmt_username`. Minted tokens authenticate AS that superuser (NetBox tokens carry no scope beyond `write_enabled`; the user's permissions apply), so they can create users.
- `netbox_user_management` reads `netbox/creds/vault-user-mgmt` and configures the netbox provider with the minted token. When `user_mgmt_username` is unset it falls back to the single static `admin_token` (a `check` block warns that rotation would then break it) - a bootstrap/degraded path, never a second static token.
- Grant the deployer `read` on `netbox/creds/vault-user-mgmt` (the one deliberate exception to the admin policy's `netbox/creds/*` exclusion).
- Keep the bare-token + `token_version`-match postconditions on the single static admin token.

## Feasibility constraints (worked through, documented in-module)

1. **The engine CAN mint a user-admin token** - roles map to a pre-existing user with only a `write_enabled` gate (`vault-plugin-secrets-netbox` `path_roles.go`, `client.go` `MintToken`); point it at a superuser and minted tokens can manage users.
2. **Token transits state.** The hashicorp/vault provider (5.6.0) exposes ephemeral resources for KV only, not dynamic engine creds, so the mint is read via the `vault_generic_secret` DATA source: the short-lived token is written to state (sensitive, lease-revoked) and re-minted each plan. Migrate to an ephemeral resource once the vault provider ships a dynamic-secret one.
3. **A clean single fresh apply is not possible.** A provider cannot be configured from a role created in the same run (data sources don't defer; OpenTofu 1.11 defers only ephemeral resources, which the vault provider doesn't offer here). So enabling the dynamic path on a backend needs a one-time targeted bootstrap of the mount + role, then normal applies. Documented in `config/netbox_secret_backend/netbox.yaml`.

## Operator follow-up

- Repair the live mount first (unchanged): `vault write netbox/config token=<BARE>` (the mount uses `ignore_changes=[token]`), keep `token_version=2`.
- To enable dynamic minting: set `user_mgmt_username` to the pre-existing superuser, apply the deployer creds policy, then bootstrap once: `tofu apply -target=...netbox_secret_backend -target=...netbox_user_mgmt_role`, then apply normally. Until then user management stays on the static token (non-breaking, with a warning).

Reviewed-on: #119
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>

modules/vault_cluster/main.tf

@@ -475,10 +475,34 @@ module "netbox_secret_backend" {
   depends_on = [module.plugin]
 }
 
+# Dedicated engine role that mints an ephemeral, user-admin-capable token for the
+# pre-existing NetBox superuser named on each backend (user_mgmt_username).
+# netbox_user_management reads netbox/creds/vault-user-mgmt from it, so it
+# authenticates with a short-lived Vault-minted token derived from the single
+# static admin token - never a second static credential, and unaffected by
+# rotation of the engine's admin seed. Created before user management so the role
+# exists when it reads creds.
+module "netbox_user_mgmt_role" {
+  source = "./modules/netbox_secret_backend_role"
+
+  for_each = { for k, v in var.netbox_secret_backend : k => v if v.user_mgmt_username != null }
+
+  backend         = each.key
+  name            = "vault-user-mgmt"
+  netbox_username = each.value.user_mgmt_username
+  write_enabled   = true
+  description     = "Ephemeral user-admin token for netbox_user_management (Vault-minted per apply)"
+  ttl             = 600
+  max_ttl         = 1200
+
+  depends_on = [module.netbox_secret_backend]
+}
+
 # Declaratively manage the NetBox service users + object permissions the engine
-# roles mint tokens for, using the seeded admin token (mirrors consul_acl_management).
-# Consumes the SAME role config as netbox_secret_backend_role: one file per identity,
-# filename-derived username, inline permissions.
+# roles mint tokens for, authenticating with the Vault-minted user-admin token
+# above (mirrors consul_acl_management). Consumes the SAME role config as
+# netbox_secret_backend_role: one file per identity, filename-derived username,
+# inline permissions.
 module "netbox_user_management" {
   source = "./modules/netbox_user_management"
 
@@ -487,6 +511,12 @@ module "netbox_user_management" {
   netbox_backends        = var.netbox_secret_backend
   netbox_roles           = var.netbox_secret_backend_role
   netbox_backend_aliases = var.netbox_backend_aliases
+
+  # This module declares its own netbox provider, so it is a legacy module and
+  # cannot take depends_on. Ordering vs the vault-user-mgmt role is not needed on
+  # steady state (the role pre-exists, so reading its creds succeeds regardless);
+  # on first enablement the role must be created first via the one-time targeted
+  # bootstrap documented in config/netbox_secret_backend/netbox.yaml.
 }
 
 module "netbox_secret_backend_role" {

modules/vault_cluster/modules/netbox_secret_backend/main.tf

@@ -8,6 +8,27 @@
 data "vault_kv_secret_v2" "config" {
   mount = "kv"
   name  = "service/vault/${var.country}/${var.region}/secret_backend/${var.path}/config"
+
+  lifecycle {
+    # The plugin builds its own Authorization header from the token VALUE, not
+    # token_version: a value starting with the nbt_ prefix is sent as
+    # "Bearer <token>" (v2), otherwise "Token <token>" (v1). So admin_token must
+    # be the BARE token - a literal `Bearer `/`Token ` scheme prefix yields a
+    # malformed three-part header and 403s on the plugin's own NetBox calls.
+    #
+    # token_version does NOT change that header; it only selects the version of
+    # the per-user tokens the engine mints for roles. It must still MATCH the
+    # admin token's kind so the mount and its minted creds line up: an nbt_ v2
+    # admin token pairs with token_version=2, a bare v1 token with token_version=1.
+    postcondition {
+      condition = nonsensitive(
+        !startswith(self.data["admin_token"], "Bearer ") &&
+        !startswith(self.data["admin_token"], "Token ") &&
+        startswith(self.data["admin_token"], "nbt_") == (var.token_version == 2)
+      )
+      error_message = "KV admin_token for netbox backend '${var.path}' must be a BARE NetBox token with no 'Bearer '/'Token ' scheme prefix, AND its version must match token_version: a v2 token (nbt_<key>.<secret>) requires token_version=2; a v1 token (bare 40-char value) requires token_version=1."
+    }
+  }
 }
 
 resource "netbox_secret_backend" "this" {

modules/vault_cluster/modules/netbox_user_management/main.tf

@@ -1,20 +1,88 @@
-# Read the seeded NetBox admin token for each backend from KV. This is the same
-# token the netbox secrets engine is configured with (key admin_token), and it
-# must carry add_token + grant_token (or superuser) to create users/permissions.
+# netbox_user_management reconciles NetBox service users + permissions on every
+# apply, so it needs an admin credential each run. That credential is minted
+# DYNAMICALLY by the netbox engine from the SINGLE static admin token, so no
+# second static credential exists and it survives rotation of the engine seed:
+#
+#   1. module.netbox_user_mgmt_role creates netbox/roles/vault-user-mgmt, a
+#      write-enabled role for a pre-existing NetBox superuser (user_mgmt_username).
+#   2. Reading netbox/creds/vault-user-mgmt mints a short-lived, user-admin-capable
+#      token for that superuser; the e-breuninger provider uses it to CRUD users.
+#
+# The hashicorp/vault provider ships ephemeral resources for KV only, not for
+# dynamic engine creds, so the mint is read via the vault_generic_secret DATA
+# source: the short-lived token transits Terraform state (sensitive, lease-revoked)
+# and is re-minted each plan. This is the closest single-static-token shape the
+# current providers allow; move to an ephemeral resource once the vault provider
+# ships a dynamic-secret one. Ordering note: the vault-user-mgmt role must already
+# exist when this reads creds, so on a brand-new backend bootstrap the mount +
+# role first (targeted apply) - a fresh single apply cannot configure the netbox
+# provider from a role created in the same run.
+locals {
+  # Backends that mint a dynamic user-admin token (a pre-existing superuser named).
+  netbox_dynamic_backends = { for k, v in var.netbox_backends : k => v if v.user_mgmt_username != null }
+  # Backends still using the single static admin_token directly (until a superuser
+  # is named). Bootstrap/degraded path - the same one token, not a second static.
+  netbox_static_backends = { for k, v in var.netbox_backends : k => v if v.user_mgmt_username == null }
+}
+
+# Dynamic path: the engine mints a user-admin token for the superuser. Requires
+# the deployer to read netbox/creds/vault-user-mgmt (policies/netbox/creds).
+data "vault_generic_secret" "user_admin" {
+  for_each = local.netbox_dynamic_backends
+
+  path = "${each.key}/creds/vault-user-mgmt"
+}
+
+# Static fallback: the single admin_token from KV, used only until a superuser is
+# named. NetBox derives the token version from the value's `nbt_` prefix, not the
+# keyword, so the same BARE token works under either scheme; reject a value that
+# carries a literal `Bearer `/`Token ` scheme prefix (a malformed header -> 403).
 data "vault_kv_secret_v2" "netbox_backend_configs" {
-  for_each = var.netbox_backends
+  for_each = local.netbox_static_backends
 
   mount = "kv"
   name  = "service/vault/${var.country}/${var.region}/secret_backend/${each.key}/config"
+
+  lifecycle {
+    postcondition {
+      condition = nonsensitive(
+        !startswith(self.data["admin_token"], "Bearer ") &&
+        !startswith(self.data["admin_token"], "Token ")
+      )
+      error_message = "KV admin_token for netbox backend '${each.key}' must be a BARE NetBox token with no 'Bearer '/'Token ' scheme prefix (v2: nbt_<key>.<secret>; v1: the 40-char value)."
+    }
+  }
 }
 
-# One NetBox provider instance per backend, authenticated with its admin token.
+# Warn (non-fatal) for any backend still on the static token: rotating the engine
+# admin seed would then break user management. Set user_mgmt_username to switch to
+# the dynamic, rotation-proof mint.
+check "netbox_user_mgmt_dynamic" {
+  assert {
+    condition     = length(local.netbox_static_backends) == 0
+    error_message = "A netbox backend has no user_mgmt_username, so user management uses the static admin_token directly and will break if that token is rotated (netbox/config/rotate). Set user_mgmt_username to a pre-existing NetBox superuser to mint the credential dynamically."
+  }
+}
+
+locals {
+  # Per backend: the dynamically-minted user-admin token, else the static seed.
+  netbox_admin_tokens = {
+    for k, v in var.netbox_backends : k => (
+      v.user_mgmt_username != null
+      ? data.vault_generic_secret.user_admin[k].data["token"]
+      : data.vault_kv_secret_v2.netbox_backend_configs[k].data["admin_token"]
+    )
+  }
+}
+
+# One NetBox provider instance per backend, authenticated with its (dynamic or
+# static) admin token.
 provider "netbox" {
   alias    = "by_backend"
   for_each = var.netbox_backend_aliases
 
   server_url           = var.netbox_backends[each.key].netbox_url
-  api_token            = data.vault_kv_secret_v2.netbox_backend_configs[each.key].data["admin_token"]
+  api_token            = local.netbox_admin_tokens[each.key]
   allow_insecure_https = var.netbox_backends[each.key].tls_skip_verify
   # NetBox is internal and not always reachable at plan time; the resource CRUD
   # calls surface any real incompatibility, so skip the startup version probe.

modules/vault_cluster/modules/netbox_user_management/variables.tf

@@ -3,6 +3,9 @@ variable "netbox_backends" {
   type = map(object({
     netbox_url      = string
     tls_skip_verify = optional(bool, false)
+    # Pre-existing NetBox superuser the engine mints a dynamic user-admin token
+    # for; unset means fall back to the static admin_token from KV.
+    user_mgmt_username = optional(string)
   }))
 }
 

modules/vault_cluster/variables.tf

@@ -426,6 +426,12 @@ variable "netbox_secret_backend" {
     ca_cert                 = optional(string)
     tls_skip_verify         = optional(bool, false)
     request_timeout_seconds = optional(number, 30)
+    # Pre-existing NetBox superuser (or add_user + add_token + grant_token) the
+    # engine mints an ephemeral user-admin token for, so netbox_user_management
+    # authenticates with a Vault-minted credential derived from the single static
+    # admin token instead of a second static one. Unset = use the static
+    # admin_token directly (bootstrap/degraded; breaks after admin-token rotation).
+    user_mgmt_username = optional(string)
   }))
   default = {}
 }