unkinben
5ae14b3a38
Ben: 1.2 TiB is too much. Cut both stores to 3d and resize honestly. - NATS LOGS stream max_age 168h -> 72h in the nats-stream-limits ConfigMap. At the stated ~33 GiB/day compressed (S2), 3d ~= 100 GiB, so max_bytes -> 130 GiB and the file-store PVC -> 180Gi/node. 3 replicas = ~0.5 TiB total (down from 1.2 TiB). - ClickHouse logs.raw TTL 30d -> 3d in the bootstrap DDL. At ~130 GiB/day raw, LZ4/ZSTD (~6x) stores ~20-25 GiB/day => ~60-75 GiB/3d; with merge headroom the CHI PVC -> 150Gi (from 200Gi). logs.raw is the only table. - Long-term retention now lives EXCLUSIVELY in the S3 archive (logarchiver) for the configured subjects; everything else is gone after 3 days — accepted design, documented in the runbook + PR body. - Update all 7d/30d comments and the replay window (now 3d). PVC shrink is a plan-time change: this stack is not deployed yet, so it is clean. Once deployed, PVCs cannot shrink in place (would need recreate). Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
52 lines
1.7 KiB
YAML
52 lines
1.7 KiB
YAML
---
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
|
|
resources:
|
|
- namespace.yaml
|
|
- vaultauth.yaml
|
|
- vaultstaticsecret.yaml
|
|
- clickhouseinstallation.yaml
|
|
- job_clickhouse-schema.yaml
|
|
- nats-bootstrap-job.yaml
|
|
- cephrgw.yaml
|
|
- gateway.yaml
|
|
- httproute.yaml
|
|
|
|
# Vector pipelines are the single source of truth (also validated by
|
|
# `vector test` in CI). Mounted into each tier via `existingConfigMaps`.
|
|
configMapGenerator:
|
|
# Tunable JetStream stream limits (the nats-bootstrap Job reads these and does
|
|
# create-or-update). Hash suffix is INTENTIONALLY left on: editing a value
|
|
# renames the ConfigMap, which rewrites the Job's env reference, which changes
|
|
# the PostSync hook Job's spec and forces Argo to re-run it -> new limits apply.
|
|
# Sizing assumes ~1500 events/s avg @ ~1 KiB/event with S2 compression (~4x):
|
|
# ~33 GiB/day compressed -> ~100 GiB/3d per replica. max_bytes 130 GiB sits
|
|
# under the 180Gi/node PVC (see values-nats.yaml). Raising retention beyond the
|
|
# PVC requires bumping BOTH max_bytes here and fileStore PVC size in values.
|
|
- name: nats-stream-limits
|
|
literals:
|
|
- max_age=72h
|
|
- max_bytes=139586437120
|
|
- dupe_window=2m
|
|
- name: vector-agent-config
|
|
files:
|
|
- agent.yaml=vector/agent.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|
|
- name: vector-aggregator-config
|
|
files:
|
|
- aggregator.yaml=vector/aggregator.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|
|
- name: vector-vm-ingest-config
|
|
files:
|
|
- vm-ingest.yaml=vector/vm-ingest.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|
|
- name: vector-archiver-config
|
|
files:
|
|
- archiver.yaml=vector/archiver.yaml
|
|
options:
|
|
disableNameSuffixHash: true
|