From 7102ef2b34736a95c83c54aa68d493b5cd74c6a9 Mon Sep 17 00:00:00 2001 From: Ben Vincent Date: Sun, 9 Aug 2026 12:14:02 +1000 Subject: [PATCH] vault: move openbao plugin sockets off /tmp onto /run (#509) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Why The litellm secrets engine on the OpenBao cluster died with `rpc Unavailable / dial unix /tmp/pluginNNN: no such file` (terraform-vault#112), fixed only by a manual `sys/plugins/reload/backend`. Root cause (post-incident log audit): go-plugin puts each plugin's control socket under the process TMPDIR — `/tmp/pluginNNN` — and `vault.service` runs without PrivateTmp. The daily `systemd-tmpfiles-clean` reaps aged `/tmp` files; bao ran a single systemd invocation for 3+ weeks, so the socket long outlived the `/tmp` cleanup age and got deleted out from under the still-running plugin process (no panic/OOM/signal/exit in the bao journal — the process was healthy, just unreachable). The risk is shared by every OpenBao plugin (gpg, rancher, gitea, ...), not just litellm. Supersedes the earlier tmpfiles-exclude approach (#508, closed) with the permanent fix: move the sockets off `/tmp` entirely. ## Change - Add a `vault.service` drop-in (`systemd::manage_dropin`) that sets `Environment=TMPDIR=/run/vault-plugins` and `RuntimeDirectory=vault-plugins` (mode 0700). - Point plugin sockets at `/run` (tmpfs, no age-based cleanup); `RuntimeDirectory` creates/owns the dir per service start. - Notify a vault service restart (module handles daemon-reload) so the new TMPDIR takes effect and plugins respawn with sockets under `/run`. ## Heads-up Puppet rolls a **bao restart per node** when this lands (the drop-in notifies `Service['vault']`). With auto-unseal (this cluster runs `profiles::vault::unseal`) it is a rolling non-event; if any node relies on manual unseal it will come back **sealed** and need unsealing. Merge consciously / stagger if needed. https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT Reviewed-on: https://git.unkin.net/unkin/puppet-prod/pulls/509 Co-authored-by: Ben Vincent Co-committed-by: Ben Vincent --- site/profiles/manifests/vault/server.pp | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/site/profiles/manifests/vault/server.pp b/site/profiles/manifests/vault/server.pp index 665b2ea..3a7ecff 100644 --- a/site/profiles/manifests/vault/server.pp +++ b/site/profiles/manifests/vault/server.pp @@ -130,6 +130,20 @@ class profiles::vault::server ( mode => '0600', } + # go-plugin creates each secrets plugin's control socket under TMPDIR + # (/tmp/pluginNNN by default); systemd-tmpfiles-clean reaps aged /tmp files + # and severs the socket of a long-lived plugin, orphaning the process. Point + # TMPDIR at a per-start RuntimeDirectory on /run (tmpfs, no age cleanup). + systemd::manage_dropin { 'plugin-tmpdir.conf': + unit => 'vault.service', + service_entry => { + 'RuntimeDirectory' => 'vault-plugins', + 'RuntimeDirectoryMode' => '0700', + 'Environment' => 'TMPDIR=/run/vault-plugins', + }, + notify => Service['vault'], + } + service { 'vault': ensure => true, enable => true,