unkin-agent
74354e847d
The node scrape jobs labelmapped every node label (~50+ NFD labels) onto each series, exceeding vminsert's 40-label default so 100% of kubelet and cadvisor series were silently dropped (vm_rows_ignored_total reason=too_many_labels). kube-state-metrics, VMAlert and the kube-prometheus k8s.rules recording rules were also absent, so dashboard queries like node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate had no producer. - replaces the blanket node labelmap with a single node-name relabel - renames node jobs so series carry job=kubelet (+ metrics_path) matching kube-prometheus selectors - adds kube-state-metrics (v2.20.0 via artifactapi k8s-registry remote) with RBAC, Service and VMServiceScrape - adds VMAlert wired to vmselect/vminsert, recording rules only - adds VMRule with the kube-prometheus k8s.rules recording group
28 lines
739 B
YAML
28 lines
739 B
YAML
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAlert
|
|
metadata:
|
|
name: main
|
|
namespace: observability
|
|
spec:
|
|
replicaCount: 1
|
|
selectAllByDefault: true
|
|
evaluationInterval: 30s
|
|
datasource:
|
|
url: http://vmselect-main.observability.svc.cluster.local:8481/select/0/prometheus/
|
|
remoteWrite:
|
|
url: http://vminsert-main.observability.svc.cluster.local:8480/insert/0/prometheus/
|
|
remoteRead:
|
|
url: http://vmselect-main.observability.svc.cluster.local:8481/select/0/prometheus/
|
|
extraArgs:
|
|
loggerFormat: json
|
|
# recording rules only for now; no alertmanager deployed
|
|
notifier.blackhole: "true"
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 512Mi
|