unkinben
1202aae06f
Three changes from review: 1. Pull every container image through the artifactapi dockerhub remote instead of direct upstream: clickhouse-server, altinity operator + metrics-exporter, bitnami/kubectl (crdHook), nats + nats-server-config-reloader, nats-box (bootstrap Job), and vector (all tiers + the CI image). Requires terraform-artifactapi#16 (dockerhub allowlist patterns) merged first. 2. Keep upstream official images (no Docker Hardened Images). DHI exists for clickhouse-server and vector but is subscription-gated and served from a private org namespace not reachable via the anonymous artifactapi dockerhub proxy; its shell-less images would also break the bash bootstrap Jobs and the shell-based vector-test CI step. Use vector's distroless-libc for runtime pods (near-hardened) and the debian variant only for CI. 3. Make the transform tier a stateless Deployment (was a StatefulSet): no PVC, no disk buffer — JetStream is the sole durability layer. The ClickHouse sink uses an in-memory block buffer so a ClickHouse outage back-pressures the JetStream pull source (unpulled messages are retained/redelivered). Add a CPU HPA (2-8) — safe because JetStream pull consumers distribute work across N replicas on the one durable consumer. Caveat documented: vector's NATS source has no end-to-end acks (acks on receipt), so a pod killed mid-outage can lose its in-memory buffer window; accepted trade for a stateless autoscaling tier. Claude-Session: https://claude.ai/code/session_015ur3i7D2azsMAWTSVABApv
123 lines
4.7 KiB
YAML
123 lines
4.7 KiB
YAML
---
|
|
# Declarative JetStream provisioning: the LOGS stream + durable consumers.
|
|
# ArgoCD PostSync hook, idempotent (add-or-converge), re-runs each sync.
|
|
#
|
|
# Stream LOGS: file storage, 3 replicas, retention=limits (NOT workqueue) so
|
|
# multiple durable consumers fan out and can independently replay within the
|
|
# window. Sized as the ClickHouse-outage buffer: 40 GiB / 72h (per-replica PVC
|
|
# is 50Gi, see values-nats.yaml). Beyond that window the S3 archive is the
|
|
# long-term replay source.
|
|
#
|
|
# Consumers (independent offsets = true fan-out):
|
|
# transform -> whole log stream, feeds the ClickHouse transform tier
|
|
# archiver -> configurable security-relevant subset, feeds the S3 archiver.
|
|
# Default filter is Vault audit (logs.k8s.vault.>); ADD subjects
|
|
# by editing ARCHIVE_SUBJECTS (space-separated -> repeated
|
|
# --filter). Exact default set is an open decision for Ben.
|
|
#
|
|
# Runbook (replay):
|
|
# (a) reprocess from JetStream (within 72h): scale the transform tier to 0,
|
|
# then `nats consumer rm LOGS transform` and re-run this Job (recreates at
|
|
# DeliverAll), or `nats consumer edit`/`--replay` from a start seq/time.
|
|
# (b) long-horizon (beyond JetStream): re-ingest S3 archive objects back
|
|
# through the transform tier (vector aws_s3 source or a one-shot Job).
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: nats-bootstrap
|
|
namespace: logging
|
|
annotations:
|
|
argocd.argoproj.io/hook: PostSync
|
|
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
|
|
labels:
|
|
app.kubernetes.io/name: nats-bootstrap
|
|
app.kubernetes.io/component: bootstrap
|
|
spec:
|
|
backoffLimit: 20
|
|
activeDeadlineSeconds: 1800
|
|
ttlSecondsAfterFinished: 3600
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: nats-bootstrap
|
|
vector.dev/exclude: "true"
|
|
spec:
|
|
restartPolicy: OnFailure
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
runAsUser: 1000
|
|
runAsGroup: 1000
|
|
containers:
|
|
- name: nats-bootstrap
|
|
image: artifactapi.k8s.syd1.au.unkin.net/dockerhub/natsio/nats-box:0.18.0
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
readOnlyRootFilesystem: true
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
env:
|
|
- name: HOME
|
|
value: /tmp
|
|
- name: NATS_URL
|
|
value: "nats://nats.logging.svc.cluster.local:4222"
|
|
- name: NATS_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nats-auth
|
|
key: admin_password
|
|
# Space-separated subject filters for the archiver consumer.
|
|
- name: ARCHIVE_SUBJECTS
|
|
value: "logs.k8s.vault.>"
|
|
command:
|
|
- /bin/sh
|
|
- -ec
|
|
- |
|
|
export NATS_USER=log-admin NATS_PASSWORD="$NATS_ADMIN_PASSWORD"
|
|
echo "Waiting for NATS + JetStream ..."
|
|
until nats --server "$NATS_URL" account info >/dev/null 2>&1; do
|
|
echo " not ready, retry in 5s"; sleep 5
|
|
done
|
|
|
|
echo "Ensuring stream LOGS ..."
|
|
nats stream add LOGS \
|
|
--subjects='logs.>' --storage=file --replicas=3 \
|
|
--retention=limits --discard=old \
|
|
--max-age=72h --max-bytes=42949672960 \
|
|
--max-msgs=-1 --max-msgs-per-subject=-1 --max-msg-size=-1 \
|
|
--max-consumers=-1 --dupe-window=2m --defaults 2>/dev/null \
|
|
|| nats stream edit -f LOGS \
|
|
--subjects='logs.>' --discard=old \
|
|
--max-age=72h --max-bytes=42949672960 --dupe-window=2m
|
|
|
|
echo "Ensuring consumer transform ..."
|
|
nats consumer add LOGS transform \
|
|
--pull --filter='logs.>' --deliver=all --ack=explicit \
|
|
--max-deliver=-1 --replay=instant --defaults 2>/dev/null \
|
|
|| echo " transform already exists"
|
|
|
|
echo "Ensuring consumer archiver (filters: $ARCHIVE_SUBJECTS) ..."
|
|
filter_args=""
|
|
for s in $ARCHIVE_SUBJECTS; do filter_args="$filter_args --filter=$s"; done
|
|
# shellcheck disable=SC2086
|
|
nats consumer add LOGS archiver \
|
|
--pull $filter_args --deliver=all --ack=explicit \
|
|
--max-deliver=-1 --replay=instant --defaults 2>/dev/null \
|
|
|| echo " archiver already exists"
|
|
|
|
echo "Done."
|
|
nats stream info LOGS
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 64Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 256Mi
|
|
volumeMounts:
|
|
- name: tmp
|
|
mountPath: /tmp
|
|
volumes:
|
|
- name: tmp
|
|
emptyDir: {}
|