Move JetStream limits to a ConfigMap; 7d retention, honest sizing
- Retention -> 7 days (max_age=168h), still retention=limits/discard=old so the transform tier and the archiver each independently see every message; reading never deletes. - Put the tunable stream limits (max_age, max_bytes, dupe_window) in the nats-stream-limits ConfigMap. The bootstrap Job reads them and does an idempotent create-or-UPDATE (nats stream add || nats stream edit), so changing the ConfigMap + re-sync applies new limits with no manual surgery. The ConfigMap keeps its kustomize content-hash suffix, so an edit renames it and rewrites the Job's env refs -> the PostSync hook Job's spec changes and Argo re-runs it (on top of hooks running each sync). Verified end-to-end against a real nats-server: create, idempotent re-run, and a max_age change all apply. - Honest 7d sizing: assume ~1500 events/s avg @ ~1 KiB/event; with S2 stream compression (~4x) that's ~33 GiB/day -> ~230 GiB/7d per replica. Enable S2 compression on the stream, set max_bytes=300 GiB, and raise the file-store PVC to 400Gi/node (3 replicas = 1.2 TiB) so the byte cap can't silently truncate retention below 7d. Numbers + assumptions flagged in the PR body. - Update runbook/comments: replay window is now 7d. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
This commit is contained in:
@@ -16,6 +16,19 @@ resources:
|
||||
# Vector pipelines are the single source of truth (also validated by
|
||||
# `vector test` in CI). Mounted into each tier via `existingConfigMaps`.
|
||||
configMapGenerator:
|
||||
# Tunable JetStream stream limits (the nats-bootstrap Job reads these and does
|
||||
# create-or-update). Hash suffix is INTENTIONALLY left on: editing a value
|
||||
# renames the ConfigMap, which rewrites the Job's env reference, which changes
|
||||
# the PostSync hook Job's spec and forces Argo to re-run it -> new limits apply.
|
||||
# Sizing assumes ~1500 events/s avg @ ~1 KiB/event with S2 compression (~4x):
|
||||
# ~33 GiB/day compressed -> ~230 GiB/7d per replica. max_bytes 300 GiB sits
|
||||
# under the 400Gi/node PVC (see values-nats.yaml). Raising retention beyond the
|
||||
# PVC requires bumping BOTH max_bytes here and fileStore PVC size in values.
|
||||
- name: nats-stream-limits
|
||||
literals:
|
||||
- max_age=168h
|
||||
- max_bytes=322122547200
|
||||
- dupe_window=2m
|
||||
- name: vector-agent-config
|
||||
files:
|
||||
- agent.yaml=vector/agent.yaml
|
||||
|
||||
Reference in New Issue
Block a user