Mint the user-management credential dynamically from the single admin token

Replace the second static user_mgmt_token with a Vault-minted, user-admin-capable
token so only ONE static NetBox admin credential exists. A dedicated engine role
(netbox/roles/vault-user-mgmt) mints a short-lived token for a pre-existing NetBox
superuser (user_mgmt_username); netbox_user_management authenticates the
e-breuninger provider with that minted token to reconcile the service users. This
also resolves rotation-divergence structurally: the credential is always derived
from the current static admin token, so rotating it never strands user management.

Constraints this design works within (documented in the module):
- The hashicorp/vault provider ships ephemeral resources for KV only, not dynamic
  engine creds, so the mint is read via the vault_generic_secret data source; the
  short-lived token transits state (sensitive, lease-revoked) and is re-minted each
  plan. Migrate to an ephemeral resource once the provider ships one.
- A provider cannot be configured from a role created in the same fresh apply, so a
  brand-new backend needs a one-time targeted bootstrap of the mount + role.

- Add user_mgmt_username to the netbox backend config; when set, mint dynamically,
  else fall back to the single static admin_token (a check block warns that
  rotation would then break user management).
- Add module.netbox_user_mgmt_role (vault-user-mgmt, write-enabled, short TTL).
- Grant the deployer read on netbox/creds/vault-user-mgmt (the one deliberate
  exception to the admin policy's creds exclusion).
- Keep the bare-token + token_version postconditions on the single admin token.

config/netbox_secret_backend/netbox.yaml

@@ -1,24 +1,30 @@
 # Mounts the netbox token secrets engine at "netbox" and writes its config.
 # The seeded NetBox admin token is sensitive and read from KV, not stored here:
 #   kv/service/vault/au/syd1/secret_backend/netbox/config
-#   -> key: admin_token     (required) engine bootstrap seed; rotated by Vault
-#   -> key: user_mgmt_token (recommended) stable token for netbox_user_management
+#   -> key: admin_token (required) the SINGLE static admin credential
 #
-# Both keys must be BARE NetBox tokens with NO scheme prefix: do not prepend
+# admin_token must be a BARE NetBox token with NO scheme prefix: do not prepend
 # "Bearer " or "Token ". NetBox infers the version from the value's nbt_ prefix,
-# so one bare token authenticates under either scheme; the plugin and provider
-# add the keyword themselves. A prefixed value yields a malformed header + 403.
+# so one bare token authenticates under either scheme; the plugin adds the keyword
+# itself. A prefixed value yields a malformed header + 403.
 #
-# Populate admin_token with a purpose-built NetBox service token that has
-# add_token + grant_token (or superuser) BEFORE applying, then run
-# `vault write -f netbox/config/rotate` after the first apply so only Vault
-# holds the live admin token.
+# Populate admin_token with a purpose-built NetBox superuser token (add_user +
+# add_token + grant_token, or superuser) BEFORE applying, then run
+# `vault write -f netbox/config/rotate` after the first apply so only Vault holds
+# the live admin token.
 #
-# Rotation caveat: config/rotate mints a fresh admin_token and DELETES the old
-# one, so the KV admin_token becomes a dead token. netbox_user_management reads
-# its credential every apply, so seed a SEPARATE, never-rotated user_mgmt_token
-# (also add_user + add_token + grant_token, or superuser) BEFORE rotating;
-# otherwise user management breaks once admin_token is rotated away.
+# Only ONE static admin token exists. netbox_user_management does NOT re-read this
+# token; instead the engine mints it a short-lived user-admin token per apply from
+# netbox/roles/vault-user-mgmt (see user_mgmt_username below), so rotating
+# admin_token never breaks user management. Set user_mgmt_username to the
+# pre-existing NetBox superuser the static admin_token belongs to (or another
+# superuser). Leaving it unset falls back to using admin_token directly, which is
+# only a bootstrap/degraded path and breaks after rotation.
+#
+# Bootstrap ordering: the vault-user-mgmt role must exist before the netbox
+# provider is configured from its creds, so on a brand-new backend apply the mount
+# + role first (e.g. `tofu apply -target=...netbox_secret_backend
+# -target=...netbox_user_mgmt_role`) once, then apply normally.
 #
 # token_version 2 is the NetBox 4.6.5 default and requires API_TOKEN_PEPPERS to
 # be configured on the NetBox server; set token_version: 1 here if the server
@@ -36,3 +42,7 @@ description: "NetBox ephemeral scoped API token engine"
 netbox_url: "https://netbox.k8s.syd1.au.unkin.net"
 token_version: 2
 request_timeout_seconds: 30
+# Set to the pre-existing NetBox superuser admin_token belongs to, to mint the
+# user-management credential dynamically (recommended). Until set, user management
+# uses admin_token directly and a check block warns that rotation will break it.
+# user_mgmt_username: "vault-netbox-admin"

modules/vault_cluster/main.tf

@@ -475,10 +475,34 @@ module "netbox_secret_backend" {
   depends_on = [module.plugin]
 }
 
+# Dedicated engine role that mints an ephemeral, user-admin-capable token for the
+# pre-existing NetBox superuser named on each backend (user_mgmt_username).
+# netbox_user_management reads netbox/creds/vault-user-mgmt from it, so it
+# authenticates with a short-lived Vault-minted token derived from the single
+# static admin token - never a second static credential, and unaffected by
+# rotation of the engine's admin seed. Created before user management so the role
+# exists when it reads creds.
+module "netbox_user_mgmt_role" {
+  source = "./modules/netbox_secret_backend_role"
+
+  for_each = { for k, v in var.netbox_secret_backend : k => v if v.user_mgmt_username != null }
+
+  backend         = each.key
+  name            = "vault-user-mgmt"
+  netbox_username = each.value.user_mgmt_username
+  write_enabled   = true
+  description     = "Ephemeral user-admin token for netbox_user_management (Vault-minted per apply)"
+  ttl             = 600
+  max_ttl         = 1200
+
+  depends_on = [module.netbox_secret_backend]
+}
+
 # Declaratively manage the NetBox service users + object permissions the engine
-# roles mint tokens for, using the seeded admin token (mirrors consul_acl_management).
-# Consumes the SAME role config as netbox_secret_backend_role: one file per identity,
-# filename-derived username, inline permissions.
+# roles mint tokens for, authenticating with the Vault-minted user-admin token
+# above (mirrors consul_acl_management). Consumes the SAME role config as
+# netbox_secret_backend_role: one file per identity, filename-derived username,
+# inline permissions.
 module "netbox_user_management" {
   source = "./modules/netbox_user_management"
 
@@ -487,6 +511,12 @@ module "netbox_user_management" {
   netbox_backends        = var.netbox_secret_backend
   netbox_roles           = var.netbox_secret_backend_role
   netbox_backend_aliases = var.netbox_backend_aliases
+
+  # This module declares its own netbox provider, so it is a legacy module and
+  # cannot take depends_on. Ordering vs the vault-user-mgmt role is not needed on
+  # steady state (the role pre-exists, so reading its creds succeeds regardless);
+  # on first enablement the role must be created first via the one-time targeted
+  # bootstrap documented in config/netbox_secret_backend/netbox.yaml.
 }
 
 module "netbox_secret_backend_role" {

modules/vault_cluster/modules/netbox_user_management/main.tf

@@ -1,59 +1,82 @@
-# Read the NetBox admin credential for each backend from KV. User management runs
-# on EVERY apply (the e-breuninger provider refreshes/reconciles the users and
-# permissions), so it needs a STABLE admin token, not the engine's bootstrap seed.
+# netbox_user_management reconciles NetBox service users + permissions on every
+# apply, so it needs an admin credential each run. That credential is minted
+# DYNAMICALLY by the netbox engine from the SINGLE static admin token, so no
+# second static credential exists and it survives rotation of the engine seed:
 #
-# The netbox secrets engine seeds its `admin_token` once and then rotates it in
-# place (`vault write -f netbox/config/rotate`), which mints a fresh admin token
-# and DELETES the old one - so after the first rotation the KV `admin_token` names
-# a token that no longer exists in NetBox. Binding user management to it would make
-# every post-rotation apply fail to authenticate. We therefore prefer a dedicated,
-# never-rotated key `user_mgmt_token` and fall back to `admin_token` only as a
-# bootstrap convenience (valid until the first rotate). Seed `user_mgmt_token`
-# (bare token, add_user + add_token + grant_token, or superuser) BEFORE rotating.
+#   1. module.netbox_user_mgmt_role creates netbox/roles/vault-user-mgmt, a
+#      write-enabled role for a pre-existing NetBox superuser (user_mgmt_username).
+#   2. Reading netbox/creds/vault-user-mgmt mints a short-lived, user-admin-capable
+#      token for that superuser; the e-breuninger provider uses it to CRUD users.
+#
+# The hashicorp/vault provider ships ephemeral resources for KV only, not for
+# dynamic engine creds, so the mint is read via the vault_generic_secret DATA
+# source: the short-lived token transits Terraform state (sensitive, lease-revoked)
+# and is re-minted each plan. This is the closest single-static-token shape the
+# current providers allow; move to an ephemeral resource once the vault provider
+# ships a dynamic-secret one. Ordering note: the vault-user-mgmt role must already
+# exist when this reads creds, so on a brand-new backend bootstrap the mount +
+# role first (targeted apply) - a fresh single apply cannot configure the netbox
+# provider from a role created in the same run.
+locals {
+  # Backends that mint a dynamic user-admin token (a pre-existing superuser named).
+  netbox_dynamic_backends = { for k, v in var.netbox_backends : k => v if v.user_mgmt_username != null }
+  # Backends still using the single static admin_token directly (until a superuser
+  # is named). Bootstrap/degraded path - the same one token, not a second static.
+  netbox_static_backends = { for k, v in var.netbox_backends : k => v if v.user_mgmt_username == null }
+}
+
+# Dynamic path: the engine mints a user-admin token for the superuser. Requires
+# the deployer to read netbox/creds/vault-user-mgmt (policies/netbox/creds).
+data "vault_generic_secret" "user_admin" {
+  for_each = local.netbox_dynamic_backends
+
+  path = "${each.key}/creds/vault-user-mgmt"
+}
+
+# Static fallback: the single admin_token from KV, used only until a superuser is
+# named. NetBox derives the token version from the value's `nbt_` prefix, not the
+# keyword, so the same BARE token works under either scheme; reject a value that
+# carries a literal `Bearer `/`Token ` scheme prefix (a malformed header -> 403).
 data "vault_kv_secret_v2" "netbox_backend_configs" {
-  for_each = var.netbox_backends
+  for_each = local.netbox_static_backends
 
   mount = "kv"
   name  = "service/vault/${var.country}/${var.region}/secret_backend/${each.key}/config"
 
   lifecycle {
-    # NetBox derives the token version from the value's `nbt_` prefix, not the
-    # Authorization keyword, so the same BARE token works whether a client sends
-    # it as `Token <t>` (e-breuninger provider) or `Bearer <t>` (the plugin). A
-    # value carrying a literal `Bearer `/`Token ` scheme prefix produces a
-    # malformed three-part header and a 403; reject it here with a clear message.
     postcondition {
       condition = nonsensitive(
-        !startswith(lookup(self.data, "user_mgmt_token", self.data["admin_token"]), "Bearer ") &&
-        !startswith(lookup(self.data, "user_mgmt_token", self.data["admin_token"]), "Token ")
+        !startswith(self.data["admin_token"], "Bearer ") &&
+        !startswith(self.data["admin_token"], "Token ")
       )
-      error_message = "KV admin credential for netbox backend '${each.key}' must be a BARE NetBox token with no 'Bearer '/'Token ' scheme prefix (v2: nbt_<key>.<secret>; v1: the 40-char value)."
+      error_message = "KV admin_token for netbox backend '${each.key}' must be a BARE NetBox token with no 'Bearer '/'Token ' scheme prefix (v2: nbt_<key>.<secret>; v1: the 40-char value)."
     }
   }
 }
 
-# Warn (non-fatal) when only the rotating bootstrap seed is present, so operators
-# seed a stable user_mgmt_token before enabling engine rotation.
-check "netbox_user_mgmt_token_seeded" {
+# Warn (non-fatal) for any backend still on the static token: rotating the engine
+# admin seed would then break user management. Set user_mgmt_username to switch to
+# the dynamic, rotation-proof mint.
+check "netbox_user_mgmt_dynamic" {
   assert {
-    condition = nonsensitive(alltrue([
-      for cfg in data.vault_kv_secret_v2.netbox_backend_configs :
-      contains(keys(cfg.data), "user_mgmt_token")
-    ]))
-    error_message = "A netbox backend has no dedicated user_mgmt_token; user management is falling back to the rotating admin_token seed. Seed a stable user_mgmt_token before running netbox/config/rotate, or user management will fail after the seed is rotated away."
+    condition     = length(local.netbox_static_backends) == 0
+    error_message = "A netbox backend has no user_mgmt_username, so user management uses the static admin_token directly and will break if that token is rotated (netbox/config/rotate). Set user_mgmt_username to a pre-existing NetBox superuser to mint the credential dynamically."
   }
 }
 
 locals {
-  # Prefer the stable, never-rotated user-management token; fall back to the
-  # bootstrap admin_token only until a user_mgmt_token is seeded.
+  # Per backend: the dynamically-minted user-admin token, else the static seed.
   netbox_admin_tokens = {
-    for backend, cfg in data.vault_kv_secret_v2.netbox_backend_configs :
-    backend => lookup(cfg.data, "user_mgmt_token", cfg.data["admin_token"])
+    for k, v in var.netbox_backends : k => (
+      v.user_mgmt_username != null
+      ? data.vault_generic_secret.user_admin[k].data["token"]
+      : data.vault_kv_secret_v2.netbox_backend_configs[k].data["admin_token"]
+    )
   }
 }
 
-# One NetBox provider instance per backend, authenticated with its admin token.
+# One NetBox provider instance per backend, authenticated with its (dynamic or
+# static) admin token.
 provider "netbox" {
   alias    = "by_backend"
   for_each = var.netbox_backend_aliases

modules/vault_cluster/modules/netbox_user_management/variables.tf

@@ -3,6 +3,9 @@ variable "netbox_backends" {
   type = map(object({
     netbox_url      = string
     tls_skip_verify = optional(bool, false)
+    # Pre-existing NetBox superuser the engine mints a dynamic user-admin token
+    # for; unset means fall back to the static admin_token from KV.
+    user_mgmt_username = optional(string)
   }))
 }
 

modules/vault_cluster/variables.tf

@@ -426,6 +426,12 @@ variable "netbox_secret_backend" {
     ca_cert                 = optional(string)
     tls_skip_verify         = optional(bool, false)
     request_timeout_seconds = optional(number, 30)
+    # Pre-existing NetBox superuser (or add_user + add_token + grant_token) the
+    # engine mints an ephemeral user-admin token for, so netbox_user_management
+    # authenticates with a Vault-minted credential derived from the single static
+    # admin token instead of a second static one. Unset = use the static
+    # admin_token directly (bootstrap/degraded; breaks after admin-token rotation).
+    user_mgmt_username = optional(string)
   }))
   default = {}
 }

policies/netbox/creds/vault-user-mgmt.yaml

@@ -0,0 +1,23 @@
+# Allow the vault deployer to mint the ephemeral user-admin token that
+# netbox_user_management authenticates with (netbox/creds/vault-user-mgmt). The
+# engine mints it from the single static admin token, so the deployer never holds
+# a second static NetBox credential.
+#
+# The netbox admin policy (policies/netbox/admin.yaml) deliberately excludes
+# netbox/creds/* - minting is normally for consumers, not the deployer. This is
+# the one deliberate exception: the deployer needs a user-admin token during the
+# run to reconcile NetBox users. Scoped to the single vault-user-mgmt role only.
+#
+# Bound to the same principals as the admin policy: the tf_vault AppRole and its
+# Woodpecker k8s auth role.
+---
+rules:
+  - path: "netbox/creds/vault-user-mgmt"
+    capabilities:
+      - read
+
+auth:
+  approle:
+    - tf_vault
+  k8s/au/syd1:
+    - woodpecker_terraform_vault