Move JetStream limits to a ConfigMap; 7d retention, honest sizing
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/vector-test Pipeline failed
ci/woodpecker/pr/kubeconform Pipeline was successful

- Retention -> 7 days (max_age=168h), still retention=limits/discard=old so the
  transform tier and the archiver each independently see every message; reading
  never deletes.
- Put the tunable stream limits (max_age, max_bytes, dupe_window) in the
  nats-stream-limits ConfigMap. The bootstrap Job reads them and does an
  idempotent create-or-UPDATE (nats stream add || nats stream edit), so changing
  the ConfigMap + re-sync applies new limits with no manual surgery. The
  ConfigMap keeps its kustomize content-hash suffix, so an edit renames it and
  rewrites the Job's env refs -> the PostSync hook Job's spec changes and Argo
  re-runs it (on top of hooks running each sync). Verified end-to-end against a
  real nats-server: create, idempotent re-run, and a max_age change all apply.
- Honest 7d sizing: assume ~1500 events/s avg @ ~1 KiB/event; with S2 stream
  compression (~4x) that's ~33 GiB/day -> ~230 GiB/7d per replica. Enable S2
  compression on the stream, set max_bytes=300 GiB, and raise the file-store PVC
  to 400Gi/node (3 replicas = 1.2 TiB) so the byte cap can't silently truncate
  retention below 7d. Numbers + assumptions flagged in the PR body.
- Update runbook/comments: replay window is now 7d.

Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
2026-07-27 21:40:10 +10:00
parent 1202aae06f
commit b18e9a669f
5 changed files with 68 additions and 17 deletions
+2 -1
View File
@@ -7,7 +7,8 @@
# insert a transform and append its id to the clickhouse sink `inputs` — no edge
# or VM rollout required.
#
# Durability model: JetStream (72h / 40GiB) is the SOLE durability layer. This
# Durability model: JetStream (7d / 300 GiB, S2-compressed) is the SOLE
# durability layer and the replay window. This
# tier is stateless (no PVC, memory buffer). If ClickHouse is down the sink
# blocks (when_full=block); back-pressure stops the source pulling, so unpulled
# messages stay in JetStream and are redelivered. NB: Vector's NATS source has
+1 -1
View File
@@ -4,7 +4,7 @@
# from the ClickHouse transform path (archive lag can never stall ingest — true
# fan-out). Writes RAW, pre-transform events (as they sit in JetStream) as
# gzipped NDJSON, partitioned by subject + date. This is the long-horizon replay
# source beyond JetStream's 72h retention window.
# source beyond JetStream's 7d retention window.
data_dir: /vector-data-dir
api: