unkin/argocd-apps
3
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

observability: fix dropped kubelet/cadvisor metrics, add kube-state-metrics + VMAlert k8s.rules #399

Merged
benvin merged 2 commits from benvin/vm-k8s-metrics into main 2026-08-23 21:38:07 +10:00
Conversation 0 Commits 2 Files Changed 5
+376 -6
unkin-agent commented 2026-08-23 12:37:46 +10:00
Member
Copy Link
Copy Source

Why

Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The kubernetes-nodes/kubernetes-nodes-cadvisor scrape jobs use a blanket labelmap on __meta_kubernetes_node_label_(.+), copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (vm_rows_ignored_total{reason="too_many_labels"} ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus k8s.rules recording rules were never deployed, so node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate (and companions) have no producer.

Changes

  • replaces the blanket node-label labelmap in both node scrape jobs with a single __meta_kubernetes_node_name -> node relabel
  • renames the node jobs so series carry job="kubelet" plus metrics_path (/metrics and /metrics/cadvisor), matching kube-prometheus selectors; the cadvisor job keeps a distinct job_name (uniqueness requirement) and forces job=kubelet via relabel
  • adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with jobLabel yielding job="kube-state-metrics"), image routed through the artifactapi k8s-registry remote
  • adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, notifier.blackhole since it only evaluates recording rules)
  • adds a VMRule with the kube-prometheus k8s.rules recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, kube_pod_owner workload relabel rules)

Prerequisite

The artifactapi k8s-registry remote exists but its patterns allowlist (^autoscaling/vpa-, ^external-dns/, ^sig-storage/) does not cover kube-state-metrics/ — terraform-artifactapi needs ^kube-state-metrics/ added to config/remote_docker/k8s-registry.yaml before/with this merge, or the KSM pod will ImagePullBackOff.

Notes

  • validated with kustomize build (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
  • the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
## Why Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer. ## Changes - replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel - renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel - adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote - adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules) - adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules) ## Prerequisite The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff. ## Notes - validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped) - the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
unkin-agent added 1 commit 2026-08-23 12:37:46 +10:00
observability: fix dropped kubelet/cadvisor metrics, add KSM + vmalert k8s.rules
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
74354e847d 
The node scrape jobs labelmapped every node label (~50+ NFD labels) onto
each series, exceeding vminsert's 40-label default so 100% of kubelet and
cadvisor series were silently dropped (vm_rows_ignored_total
reason=too_many_labels). kube-state-metrics, VMAlert and the
kube-prometheus k8s.rules recording rules were also absent, so dashboard
queries like
node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
had no producer.

- replaces the blanket node labelmap with a single node-name relabel
- renames node jobs so series carry job=kubelet (+ metrics_path) matching
  kube-prometheus selectors
- adds kube-state-metrics (v2.20.0 via artifactapi k8s-registry remote)
  with RBAC, Service and VMServiceScrape
- adds VMAlert wired to vmselect/vminsert, recording rules only
- adds VMRule with the kube-prometheus k8s.rules recording group
unkinben added 1 commit 2026-08-23 21:33:15 +10:00
Use canonical registry.k8s.io image for kube-state-metrics (containerd mirror handles artifactapi redirect)
ci/woodpecker/pr/vector-test Pipeline was successful
Details
ci/woodpecker/pr/pre-commit Pipeline was successful
Details
ci/woodpecker/pr/kubeconform Pipeline was successful
Details
a0a6881194
benvin merged commit e608dadae7 into main 2026-08-23 21:38:07 +10:00
benvin deleted branch benvin/vm-k8s-metrics 2026-08-23 21:38:07 +10:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: unkin/argocd-apps#399
Delete Branch "benvin/vm-k8s-metrics"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?