unkinben
c39af2f9c3
Rework the logging pipeline around a durable message bus so logs survive a ClickHouse outage, can be replayed after a bad transform, and fan out to multiple independent consumers. Add long-term raw-log backup to S3. Topology becomes edge -> JetStream -> consumers -> sinks: - Dedicated JetStream NATS cluster (3 replicas, file storage) in the logging namespace. Deliberately separate from app messaging (streamstack) for blast-radius isolation. Stream LOGS (subjects logs.>, retention=limits, 40GiB / 72h) is the outage buffer; durable consumers give independent offsets. - Edge publishers (thin): the k8s DaemonSet and a new VM-ingest Deployment (HTTP NDJSON front door behind the logs-ingest Gateway) publish into JetStream (logs.k8s.<ns>.<container> / logs.vm.<host>). No parsing on the edge. - Transform tier (StatefulSet): pulls the whole stream via the durable `transform` consumer, routes by subject, shapes, and remains the sole ClickHouse writer. Its disk buffer shrinks (JetStream is the outage buffer). - Archiver (Deployment): its OWN durable `archiver` consumer (independent offsets — archive lag never affects the ClickHouse path) writes RAW, pre-transform events to a Ceph RGW S3 bucket (cephrgw-operator ObjectStoreUser + Bucket + BucketAccess) as gzipped NDJSON keyed by raw/<subject>/YYYY/MM/DD/. Default subject filter is Vault audit (logs.k8s.vault.>), configurable. Auth: distinct NATS users (producer publish-only, consumer pull+ack, admin for the stream/consumer bootstrap Job) with passwords from Vault (nats-auth Secret); S3 creds from the BucketAccess Secret. Streams/consumers are provisioned by an idempotent PostSync bootstrap Job. Add local kubeconform schemas for the ceph.unkin.net CRDs (datreeio lacks them) and extend the vector-test CI to cover the agent, VM-ingest and archiver configs. Verified end-to-end locally: NATS ACLs, vector JetStream publish, and durable-consumer pull+ack (at-least-once) all work. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
123 lines
4.7 KiB
YAML
123 lines
4.7 KiB
YAML
---
|
|
# Declarative JetStream provisioning: the LOGS stream + durable consumers.
|
|
# ArgoCD PostSync hook, idempotent (add-or-converge), re-runs each sync.
|
|
#
|
|
# Stream LOGS: file storage, 3 replicas, retention=limits (NOT workqueue) so
|
|
# multiple durable consumers fan out and can independently replay within the
|
|
# window. Sized as the ClickHouse-outage buffer: 40 GiB / 72h (per-replica PVC
|
|
# is 50Gi, see values-nats.yaml). Beyond that window the S3 archive is the
|
|
# long-term replay source.
|
|
#
|
|
# Consumers (independent offsets = true fan-out):
|
|
# transform -> whole log stream, feeds the ClickHouse transform tier
|
|
# archiver -> configurable security-relevant subset, feeds the S3 archiver.
|
|
# Default filter is Vault audit (logs.k8s.vault.>); ADD subjects
|
|
# by editing ARCHIVE_SUBJECTS (space-separated -> repeated
|
|
# --filter). Exact default set is an open decision for Ben.
|
|
#
|
|
# Runbook (replay):
|
|
# (a) reprocess from JetStream (within 72h): scale the transform tier to 0,
|
|
# then `nats consumer rm LOGS transform` and re-run this Job (recreates at
|
|
# DeliverAll), or `nats consumer edit`/`--replay` from a start seq/time.
|
|
# (b) long-horizon (beyond JetStream): re-ingest S3 archive objects back
|
|
# through the transform tier (vector aws_s3 source or a one-shot Job).
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: nats-bootstrap
|
|
namespace: logging
|
|
annotations:
|
|
argocd.argoproj.io/hook: PostSync
|
|
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
|
|
labels:
|
|
app.kubernetes.io/name: nats-bootstrap
|
|
app.kubernetes.io/component: bootstrap
|
|
spec:
|
|
backoffLimit: 20
|
|
activeDeadlineSeconds: 1800
|
|
ttlSecondsAfterFinished: 3600
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: nats-bootstrap
|
|
vector.dev/exclude: "true"
|
|
spec:
|
|
restartPolicy: OnFailure
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
containers:
|
|
- name: nats-bootstrap
|
|
image: natsio/nats-box:0.18.0
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
env:
|
|
- name: HOME
|
|
value: /tmp
|
|
- name: NATS_URL
|
|
value: "nats://nats.logging.svc.cluster.local:4222"
|
|
- name: NATS_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nats-auth
|
|
key: admin_password
|
|
# Space-separated subject filters for the archiver consumer.
|
|
- name: ARCHIVE_SUBJECTS
|
|
value: "logs.k8s.vault.>"
|
|
command:
|
|
- /bin/sh
|
|
- -ec
|
|
- |
|
|
export NATS_USER=log-admin NATS_PASSWORD="$NATS_ADMIN_PASSWORD"
|
|
echo "Waiting for NATS + JetStream ..."
|
|
until nats --server "$NATS_URL" account info >/dev/null 2>&1; do
|
|
echo " not ready, retry in 5s"; sleep 5
|
|
done
|
|
|
|
echo "Ensuring stream LOGS ..."
|
|
nats stream add LOGS \
|
|
--subjects='logs.>' --storage=file --replicas=3 \
|
|
--retention=limits --discard=old \
|
|
--max-age=72h --max-bytes=42949672960 \
|
|
--max-msgs=-1 --max-msgs-per-subject=-1 --max-msg-size=-1 \
|
|
--max-consumers=-1 --dupe-window=2m --defaults 2>/dev/null \
|
|
|| nats stream edit -f LOGS \
|
|
--subjects='logs.>' --discard=old \
|
|
--max-age=72h --max-bytes=42949672960 --dupe-window=2m
|
|
|
|
echo "Ensuring consumer transform ..."
|
|
nats consumer add LOGS transform \
|
|
--pull --filter='logs.>' --deliver=all --ack=explicit \
|
|
--max-deliver=-1 --replay=instant --defaults 2>/dev/null \
|
|
|| echo " transform already exists"
|
|
|
|
echo "Ensuring consumer archiver (filters: $ARCHIVE_SUBJECTS) ..."
|
|
filter_args=""
|
|
for s in $ARCHIVE_SUBJECTS; do filter_args="$filter_args --filter=$s"; done
|
|
# shellcheck disable=SC2086
|
|
nats consumer add LOGS archiver \
|
|
--pull $filter_args --deliver=all --ack=explicit \
|
|
--max-deliver=-1 --replay=instant --defaults 2>/dev/null \
|
|
|| echo " archiver already exists"
|
|
|
|
echo "Done."
|
|
nats stream info LOGS
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 64Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 256Mi
|
|
volumeMounts:
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
volumes:
|
|
- name: tmp
|
|
emptyDir: {}
|