unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

vault: move openbao plugin sockets off /tmp onto /run #509

Merged
benvin merged 1 commits from benvin/vault-plugin-tmpdir into develop 2026-08-09 12:14:03 +10:00
Conversation 0 Commits 1 Files Changed 1
+14
unkinben commented 2026-08-06 22:42:12 +10:00
Owner
Copy Link
Copy Source

Why

The litellm secrets engine on the OpenBao cluster died with rpc Unavailable / dial unix /tmp/pluginNNN: no such file (terraform-vault#112), fixed only by a manual sys/plugins/reload/backend.

Root cause (post-incident log audit): go-plugin puts each plugin's control socket under the process TMPDIR — /tmp/pluginNNN — and vault.service runs without PrivateTmp. The daily systemd-tmpfiles-clean reaps aged /tmp files; bao ran a single systemd invocation for 3+ weeks, so the socket long outlived the /tmp cleanup age and got deleted out from under the still-running plugin process (no panic/OOM/signal/exit in the bao journal — the process was healthy, just unreachable). The risk is shared by every OpenBao plugin (gpg, rancher, gitea, ...), not just litellm.

Supersedes the earlier tmpfiles-exclude approach (#508, closed) with the permanent fix: move the sockets off /tmp entirely.

Change

  • Add a vault.service drop-in (systemd::manage_dropin) that sets Environment=TMPDIR=/run/vault-plugins and RuntimeDirectory=vault-plugins (mode 0700).
  • Point plugin sockets at /run (tmpfs, no age-based cleanup); RuntimeDirectory creates/owns the dir per service start.
  • Notify a vault service restart (module handles daemon-reload) so the new TMPDIR takes effect and plugins respawn with sockets under /run.

Heads-up

Puppet rolls a bao restart per node when this lands (the drop-in notifies Service['vault']). With auto-unseal (this cluster runs profiles::vault::unseal) it is a rolling non-event; if any node relies on manual unseal it will come back sealed and need unsealing. Merge consciously / stagger if needed.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

## Why The litellm secrets engine on the OpenBao cluster died with `rpc Unavailable / dial unix /tmp/pluginNNN: no such file` (terraform-vault#112), fixed only by a manual `sys/plugins/reload/backend`. Root cause (post-incident log audit): go-plugin puts each plugin's control socket under the process TMPDIR — `/tmp/pluginNNN` — and `vault.service` runs without PrivateTmp. The daily `systemd-tmpfiles-clean` reaps aged `/tmp` files; bao ran a single systemd invocation for 3+ weeks, so the socket long outlived the `/tmp` cleanup age and got deleted out from under the still-running plugin process (no panic/OOM/signal/exit in the bao journal — the process was healthy, just unreachable). The risk is shared by every OpenBao plugin (gpg, rancher, gitea, ...), not just litellm. Supersedes the earlier tmpfiles-exclude approach (#508, closed) with the permanent fix: move the sockets off `/tmp` entirely. ## Change - Add a `vault.service` drop-in (`systemd::manage_dropin`) that sets `Environment=TMPDIR=/run/vault-plugins` and `RuntimeDirectory=vault-plugins` (mode 0700). - Point plugin sockets at `/run` (tmpfs, no age-based cleanup); `RuntimeDirectory` creates/owns the dir per service start. - Notify a vault service restart (module handles daemon-reload) so the new TMPDIR takes effect and plugins respawn with sockets under `/run`. ## Heads-up Puppet rolls a **bao restart per node** when this lands (the drop-in notifies `Service['vault']`). With auto-unseal (this cluster runs `profiles::vault::unseal`) it is a rolling non-event; if any node relies on manual unseal it will come back **sealed** and need unsealing. Merge consciously / stagger if needed. https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
unkinben added 1 commit 2026-08-06 22:42:13 +10:00
vault: move openbao plugin sockets off /tmp onto a /run RuntimeDirectory
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
26c5235d00 
go-plugin creates each secrets plugin's control socket under the process
TMPDIR (/tmp/pluginNNN by default). The daily systemd-tmpfiles-clean
deletes aged /tmp files, severing the socket of a long-lived plugin while
its process keeps running, so every request fails with rpc Unavailable /
dial unix /tmp/pluginNNN: no such file until a reload respawns it. This
took out the litellm engine (terraform-vault#112); the risk is shared by
every OpenBao plugin (gpg, rancher, gitea, ...).

Add a vault.service drop-in setting TMPDIR to a per-start RuntimeDirectory
on /run (tmpfs, no age-based cleanup), so plugin sockets can never be
reaped. The drop-in notifies a vault restart so the new TMPDIR takes
effect and plugins respawn with sockets under /run.

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
benvin merged commit 7102ef2b34 into develop 2026-08-09 12:14:03 +10:00
benvin deleted branch benvin/vault-plugin-tmpdir 2026-08-09 12:14:03 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/puppet-prod#509
Delete Branch "benvin/vault-plugin-tmpdir"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?