unkin/puppet-prod
4
0
Fork 0
Code Issues 15 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

vault: stop systemd-tmpfiles reaping openbao plugin sockets #508

Closed
unkinben wants to merge 1 commits from benvin/openbao-plugin-socket-tmpfiles into develop
pull from: benvin/openbao-plugin-socket-tmpfiles
merge into: unkin:develop
Conversation 1 Commits 1 Files Changed 1
+9
unkinben commented 2026-08-06 22:32:17 +10:00
Owner
Copy Link
Copy Source

Why

The litellm secrets engine died on the OpenBao cluster (terraform-vault#112): every request returned rpc Unavailable / dial unix /tmp/pluginNNN: no such file, fixed only by a manual sys/plugins/reload/backend.

Root cause (evidenced from central logs): go-plugin puts each plugin's control socket at /tmp/pluginNNN. vault.service runs without PrivateTmp, and systemd-tmpfiles-clean runs daily on every bao node deleting aged /tmp files. bao had a single systemd invocation across 3+ weeks (no restart), so the socket was far older than the cleanup age and got reaped. The plugin process never crashed (no panic/OOM/signal/exit in the journal until the manual reload) — the cleaner just removed the socket out from under a live process. Node 2002's clean ran 08-04 14:50Z; issue #112 was filed ~10 min later.

This is not litellm-specific: every OpenBao plugin (gpg, rancher, gitea, ...) shares the same /tmp socket and the same fate.

Change

  • Add a tmpfiles.d exclude x /tmp/plugin* on all vault servers so plugin control sockets survive systemd-tmpfiles-clean.

Takes effect on the next clean; no vault restart required.

https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT

## Why The litellm secrets engine died on the OpenBao cluster (terraform-vault#112): every request returned `rpc Unavailable / dial unix /tmp/pluginNNN: no such file`, fixed only by a manual `sys/plugins/reload/backend`. Root cause (evidenced from central logs): go-plugin puts each plugin's control socket at `/tmp/pluginNNN`. `vault.service` runs without PrivateTmp, and `systemd-tmpfiles-clean` runs daily on every bao node deleting aged `/tmp` files. bao had a single systemd invocation across 3+ weeks (no restart), so the socket was far older than the cleanup age and got reaped. The plugin process never crashed (no panic/OOM/signal/exit in the journal until the manual reload) — the cleaner just removed the socket out from under a live process. Node 2002's clean ran 08-04 14:50Z; issue #112 was filed ~10 min later. This is not litellm-specific: every OpenBao plugin (gpg, rancher, gitea, ...) shares the same `/tmp` socket and the same fate. ## Change - Add a tmpfiles.d exclude `x /tmp/plugin*` on all vault servers so plugin control sockets survive `systemd-tmpfiles-clean`. Takes effect on the next clean; no vault restart required. https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
unkinben added 1 commit 2026-08-06 22:32:17 +10:00
vault: keep systemd-tmpfiles from reaping openbao plugin sockets
ci/woodpecker/pr/ruby-validate Pipeline was successful
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Details
5e8cff921c 
go-plugin gives each secrets plugin a control socket at /tmp/pluginNNN.
The daily systemd-tmpfiles-clean deletes /tmp files older than the OS
policy, which severs the socket of a long-lived plugin while its process
keeps running. Every request then fails with rpc Unavailable / dial unix
/tmp/pluginNNN: no such file, until a plugin reload respawns it. This hit
the litellm engine (terraform-vault#112) but the risk is shared by every
OpenBao plugin (gpg, rancher, gitea, ...).

Ship a tmpfiles.d exclude (x /tmp/plugin*) on all vault servers so plugin
sockets survive the cleaner. Takes effect on the next clean, no restart.

Claude-Session: https://claude.ai/code/session_01JUoARVdmhxKQHyyyp1pxeT
unkinben commented 2026-08-06 22:40:20 +10:00
Author
Owner
Copy Link
Copy Source

Superseded by the permanent TMPDIR drop-in approach (per user): instead of excluding plugin sockets from the /tmp cleaner, move them off /tmp entirely onto /run (tmpfs, no age-based cleanup) via a vault.service drop-in. Closing in favour of the new PR.

Superseded by the permanent TMPDIR drop-in approach (per user): instead of excluding plugin sockets from the /tmp cleaner, move them off /tmp entirely onto /run (tmpfs, no age-based cleanup) via a vault.service drop-in. Closing in favour of the new PR.
unkinben closed this pull request 2026-08-06 22:40:32 +10:00
All checks were successful
ci/woodpecker/pr/ruby-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/puppet-lint Pipeline was successful
Required
Details
ci/woodpecker/pr/yamllint Pipeline was successful
Required
Details
ci/woodpecker/pr/bolt-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/erb-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/epp-validate Pipeline was successful
Required
Details
ci/woodpecker/pr/ruby-check Pipeline was successful
Required
Details
ci/woodpecker/pr/puppet-validate Pipeline was successful
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/puppet-prod#508
Delete Branch "benvin/openbao-plugin-socket-tmpfiles"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?