terraform-vault plan CI broken: litellm secret backend returns 500 (blocks all PRs) #112

New Issue

2026-08-05T01:00:01+10:00

unkinben commented

2026-08-05 01:00:01 +10:00

Problem

The plan pipeline on master PRs fails for every terraform-vault PR right now, on an untouched resource:

Error: failed to read litellm config
  with module.litellm_secret_backend["litellm"].litellm_secret_backend.this,
  on modules/litellm_secret_backend/main.tf line 7 ...
Code: 500. Errors: * rpc error: code = Unavailable desc = connection error:
  "transport: error while dialing: dial unix /tmp/pluginNNN: no such file or directory"

The litellm provider makes an API call that returns HTTP 500 (the litellm vault secret engine), then the provider plugin process dies, failing tofu plan. Reproduced on pipelines 134 and 135 (PR #111), identical each time — not transient.

Impact

Branch protection requires the plan status check, so no terraform-vault PR can merge until this is fixed (e.g. blocks the terraform-infra rename PR #111).

Proposal

Restore the litellm secret engine backend (vault-plugin-secrets-litellm) so its API stops returning 500, or
make module.litellm_secret_backend resilient to a 500 during read (the provider should surface an error, not crash the plugin RPC).

Acceptance

make plan on a no-op terraform-vault PR is green.

## Problem The `plan` pipeline on `master` PRs fails for every terraform-vault PR right now, on an untouched resource: ``` Error: failed to read litellm config with module.litellm_secret_backend["litellm"].litellm_secret_backend.this, on modules/litellm_secret_backend/main.tf line 7 ... Code: 500. Errors: * rpc error: code = Unavailable desc = connection error: "transport: error while dialing: dial unix /tmp/pluginNNN: no such file or directory" ``` The `litellm` provider makes an API call that returns HTTP 500 (the litellm vault secret engine), then the provider plugin process dies, failing `tofu plan`. Reproduced on pipelines 134 and 135 (PR #111), identical each time — not transient. ## Impact Branch protection requires the `plan` status check, so **no terraform-vault PR can merge** until this is fixed (e.g. blocks the terraform-infra rename PR #111). ## Proposal - Restore the litellm secret engine backend (vault-plugin-secrets-litellm) so its API stops returning 500, or - make `module.litellm_secret_backend` resilient to a 500 during read (the provider should surface an error, not crash the plugin RPC). ## Acceptance - `make plan` on a no-op terraform-vault PR is green.

unkinben referenced this issue

2026-08-05 01:02:21 +10:00

Rename terraform-ipam CI Vault access -> terraform-infra #111

unkinben commented

2026-08-06 22:32:48 +10:00

Root cause found (post-incident audit). The plugin never crashed: it was systemd-tmpfiles-clean on the bao nodes reaping the go-plugin control socket /tmp/pluginNNN out from under a still-running plugin process (vault.service has no PrivateTmp; bao had a single systemd invocation for 3+ weeks so the socket long outlived the /tmp cleanup age). Central-log evidence: zero panic/OOM/signal/exit in the bao journal until the manual reload, and node 2002's daily clean at 08-04 14:50Z lines up ~10 min before this issue was filed. The manual sys/plugins/reload/backend worked because it respawns the plugin with a fresh socket. Fleet fix (protects all plugins, not just litellm): puppet-prod#508 adds a tmpfiles.d x /tmp/plugin* exclude on all vault servers. Nothing to change in the litellm plugin itself.

Root cause found (post-incident audit). The plugin never crashed: it was `systemd-tmpfiles-clean` on the bao nodes reaping the go-plugin control socket `/tmp/pluginNNN` out from under a still-running plugin process (vault.service has no PrivateTmp; bao had a single systemd invocation for 3+ weeks so the socket long outlived the /tmp cleanup age). Central-log evidence: zero panic/OOM/signal/exit in the bao journal until the manual reload, and node 2002's daily clean at 08-04 14:50Z lines up ~10 min before this issue was filed. The manual `sys/plugins/reload/backend` worked because it respawns the plugin with a fresh socket. Fleet fix (protects all plugins, not just litellm): puppet-prod#508 adds a tmpfiles.d `x /tmp/plugin*` exclude on all vault servers. Nothing to change in the litellm plugin itself.

unkinben commented

2026-08-06 22:42:26 +10:00

Fix PR updated: puppet-prod#508 (tmpfiles exclude) is closed in favour of the permanent fix puppet-prod#509 — a vault.service drop-in that sets TMPDIR=/run/vault-plugins (RuntimeDirectory on tmpfs, no age-based cleanup), so plugin sockets can never be reaped by systemd-tmpfiles-clean. Applies to all OpenBao plugins. Note: landing #509 triggers a rolling per-node bao restart (auto-unseal makes it a non-event).

unkinben commented

2026-08-06 23:14:22 +10:00

terraform-vault plan is green again (PR #113 plan succeeded; #111 also merged), so the litellm secret-backend 500 appears to have recovered. Leaving open for the owner to confirm the root cause is fixed vs. a transient litellm outage.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/terraform-vault#112