vault: move openbao plugin sockets off /tmp onto /run (#509)

## Why

The litellm secrets engine on the OpenBao cluster died with `rpc Unavailable / dial unix /tmp/pluginNNN: no such file` (terraform-vault#112), fixed only by a manual `sys/plugins/reload/backend`.

Root cause (post-incident log audit): go-plugin puts each plugin's control socket under the process TMPDIR — `/tmp/pluginNNN` — and `vault.service` runs without PrivateTmp. The daily `systemd-tmpfiles-clean` reaps aged `/tmp` files; bao ran a single systemd invocation for 3+ weeks, so the socket long outlived the `/tmp` cleanup age and got deleted out from under the still-running plugin process (no panic/OOM/signal/exit in the bao journal — the process was healthy, just unreachable). The risk is shared by every OpenBao plugin (gpg, rancher, gitea, ...), not just litellm.

Supersedes the earlier tmpfiles-exclude approach (#508, closed) with the permanent fix: move the sockets off `/tmp` entirely.

## Change

- Add a `vault.service` drop-in (`systemd::manage_dropin`) that sets `Environment=TMPDIR=/run/vault-plugins` and `RuntimeDirectory=vault-plugins` (mode 0700).
- Point plugin sockets at `/run` (tmpfs, no age-based cleanup); `RuntimeDirectory` creates/owns the dir per service start.
- Notify a vault service restart (module handles daemon-reload) so the new TMPDIR takes effect and plugins respawn with sockets under `/run`.

## Heads-up

Puppet rolls a **bao restart per node** when this lands (the drop-in notifies `Service['vault']`). With auto-unseal (this cluster runs `profiles::vault::unseal`) it is a rolling non-event; if any node relies on manual unseal it will come back **sealed** and need unsealing. Merge consciously / stagger if needed.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
Reviewed-on: #509
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
This commit was merged in pull request #509.
This commit is contained in:
2026-08-09 12:14:02 +10:00
committed by BenVincent
parent d859daead4
commit 7102ef2b34
+14
View File
@@ -130,6 +130,20 @@ class profiles::vault::server (
mode => '0600', mode => '0600',
} }
# go-plugin creates each secrets plugin's control socket under TMPDIR
# (/tmp/pluginNNN by default); systemd-tmpfiles-clean reaps aged /tmp files
# and severs the socket of a long-lived plugin, orphaning the process. Point
# TMPDIR at a per-start RuntimeDirectory on /run (tmpfs, no age cleanup).
systemd::manage_dropin { 'plugin-tmpdir.conf':
unit => 'vault.service',
service_entry => {
'RuntimeDirectory' => 'vault-plugins',
'RuntimeDirectoryMode' => '0700',
'Environment' => 'TMPDIR=/run/vault-plugins',
},
notify => Service['vault'],
}
service { 'vault': service { 'vault':
ensure => true, ensure => true,
enable => true, enable => true,