Move JetStream limits to a ConfigMap; 7d retention, honest sizing
- Retention -> 7 days (max_age=168h), still retention=limits/discard=old so the transform tier and the archiver each independently see every message; reading never deletes. - Put the tunable stream limits (max_age, max_bytes, dupe_window) in the nats-stream-limits ConfigMap. The bootstrap Job reads them and does an idempotent create-or-UPDATE (nats stream add || nats stream edit), so changing the ConfigMap + re-sync applies new limits with no manual surgery. The ConfigMap keeps its kustomize content-hash suffix, so an edit renames it and rewrites the Job's env refs -> the PostSync hook Job's spec changes and Argo re-runs it (on top of hooks running each sync). Verified end-to-end against a real nats-server: create, idempotent re-run, and a max_age change all apply. - Honest 7d sizing: assume ~1500 events/s avg @ ~1 KiB/event; with S2 stream compression (~4x) that's ~33 GiB/day -> ~230 GiB/7d per replica. Enable S2 compression on the stream, set max_bytes=300 GiB, and raise the file-store PVC to 400Gi/node (3 replicas = 1.2 TiB) so the byte cap can't silently truncate retention below 7d. Numbers + assumptions flagged in the PR body. - Update runbook/comments: replay window is now 7d. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
@@ -7,7 +7,8 @@
|
||||
# insert a transform and append its id to the clickhouse sink `inputs` — no edge
|
||||
# or VM rollout required.
|
||||
#
|
||||
# Durability model: JetStream (72h / 40GiB) is the SOLE durability layer. This
|
||||
# Durability model: JetStream (7d / 300 GiB, S2-compressed) is the SOLE
|
||||
# durability layer and the replay window. This
|
||||
# tier is stateless (no PVC, memory buffer). If ClickHouse is down the sink
|
||||
# blocks (when_full=block); back-pressure stops the source pulling, so unpulled
|
||||
# messages stay in JetStream and are redelivered. NB: Vector's NATS source has
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# from the ClickHouse transform path (archive lag can never stall ingest — true
|
||||
# fan-out). Writes RAW, pre-transform events (as they sit in JetStream) as
|
||||
# gzipped NDJSON, partitioned by subject + date. This is the long-horizon replay
|
||||
# source beyond JetStream's 72h retention window.
|
||||
# source beyond JetStream's 7d retention window.
|
||||
data_dir: /vector-data-dir
|
||||
|
||||
api:
|
||||
|
||||
Reference in New Issue
Block a user