unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

observability: fix dropped kubelet/cadvisor metrics, add kube-state-metrics + VMAlert k8s.rules #399

Merged
benvin merged 2 commits from benvin/vm-k8s-metrics into main 2026-08-23 21:38:07 +10:00
Conversation 0 Commits 2 Files Changed 5
+376 -6

2 Commits

Author SHA1 Message Date
unkin-agent a0a6881194 Use canonical registry.k8s.io image for kube-state-metrics (containerd mirror handles artifactapi redirect)
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
2026-08-23 21:33:05 +10:00
unkin-agent 74354e847d observability: fix dropped kubelet/cadvisor metrics, add KSM + vmalert k8s.rules
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details 
The node scrape jobs labelmapped every node label (~50+ NFD labels) onto
each series, exceeding vminsert's 40-label default so 100% of kubelet and
cadvisor series were silently dropped (vm_rows_ignored_total
reason=too_many_labels). kube-state-metrics, VMAlert and the
kube-prometheus k8s.rules recording rules were also absent, so dashboard
queries like
node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
had no producer.

- replaces the blanket node labelmap with a single node-name relabel
- renames node jobs so series carry job=kubelet (+ metrics_path) matching
  kube-prometheus selectors
- adds kube-state-metrics (v2.20.0 via artifactapi k8s-registry remote)
  with RBAC, Service and VMServiceScrape
- adds VMAlert wired to vmselect/vminsert, recording rules only
- adds VMRule with the kube-prometheus k8s.rules recording group
 2026-08-23 12:37:04 +10:00