unkin-agent
e608dadae7
## Why
Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer.
## Changes
- replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel
- renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel
- adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote
- adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules)
- adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules)
## Prerequisite
The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff.
## Notes
- validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
- the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
Reviewed-on: #399
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
134 lines
6.3 KiB
YAML
134 lines
6.3 KiB
YAML
---
|
|
# kube-prometheus k8s.rules recording rules (dashboard queries depend on these).
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMRule
|
|
metadata:
|
|
name: k8s-rules
|
|
namespace: observability
|
|
spec:
|
|
groups:
|
|
- name: k8s.rules
|
|
rules:
|
|
- record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate
|
|
expr: |
|
|
sum by (cluster, namespace, pod, container) (
|
|
irate(container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}[5m])
|
|
) * on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
|
|
1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=""})
|
|
)
|
|
- record: node_namespace_pod_container:container_memory_working_set_bytes
|
|
expr: |
|
|
container_memory_working_set_bytes{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}
|
|
* on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
|
|
1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=""})
|
|
)
|
|
- record: node_namespace_pod_container:container_memory_rss
|
|
expr: |
|
|
container_memory_rss{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}
|
|
* on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
|
|
1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=""})
|
|
)
|
|
- record: node_namespace_pod_container:container_memory_cache
|
|
expr: |
|
|
container_memory_cache{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}
|
|
* on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
|
|
1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=""})
|
|
)
|
|
- record: node_namespace_pod_container:container_memory_swap
|
|
expr: |
|
|
container_memory_swap{job="kubelet", metrics_path="/metrics/cadvisor", image!=""}
|
|
* on (cluster, namespace, pod) group_left(node) topk by (cluster, namespace, pod) (
|
|
1, max by (cluster, namespace, pod, node) (kube_pod_info{node!=""})
|
|
)
|
|
- record: namespace_memory:kube_pod_container_resource_requests:sum
|
|
expr: |
|
|
sum by (namespace, cluster) (
|
|
sum by (namespace, pod, cluster) (
|
|
max by (namespace, pod, container, cluster) (
|
|
kube_pod_container_resource_requests{resource="memory",job="kube-state-metrics"}
|
|
) * on (namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (
|
|
kube_pod_status_phase{phase=~"Pending|Running"} == 1
|
|
)
|
|
)
|
|
)
|
|
- record: namespace_cpu:kube_pod_container_resource_requests:sum
|
|
expr: |
|
|
sum by (namespace, cluster) (
|
|
sum by (namespace, pod, cluster) (
|
|
max by (namespace, pod, container, cluster) (
|
|
kube_pod_container_resource_requests{resource="cpu",job="kube-state-metrics"}
|
|
) * on (namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (
|
|
kube_pod_status_phase{phase=~"Pending|Running"} == 1
|
|
)
|
|
)
|
|
)
|
|
- record: namespace_memory:kube_pod_container_resource_limits:sum
|
|
expr: |
|
|
sum by (namespace, cluster) (
|
|
sum by (namespace, pod, cluster) (
|
|
max by (namespace, pod, container, cluster) (
|
|
kube_pod_container_resource_limits{resource="memory",job="kube-state-metrics"}
|
|
) * on (namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (
|
|
kube_pod_status_phase{phase=~"Pending|Running"} == 1
|
|
)
|
|
)
|
|
)
|
|
- record: namespace_cpu:kube_pod_container_resource_limits:sum
|
|
expr: |
|
|
sum by (namespace, cluster) (
|
|
sum by (namespace, pod, cluster) (
|
|
max by (namespace, pod, container, cluster) (
|
|
kube_pod_container_resource_limits{resource="cpu",job="kube-state-metrics"}
|
|
) * on (namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (
|
|
kube_pod_status_phase{phase=~"Pending|Running"} == 1
|
|
)
|
|
)
|
|
)
|
|
- record: namespace_workload_pod:kube_pod_owner:relabel
|
|
labels:
|
|
workload_type: deployment
|
|
expr: |
|
|
max by (cluster, namespace, workload, pod) (
|
|
label_replace(
|
|
label_replace(
|
|
kube_pod_owner{job="kube-state-metrics", owner_kind="ReplicaSet"},
|
|
"replicaset", "$1", "owner_name", "(.*)"
|
|
) * on (replicaset, namespace) group_left(owner_name) topk by (replicaset, namespace) (
|
|
1, max by (replicaset, namespace, owner_name) (
|
|
kube_replicaset_owner{job="kube-state-metrics"}
|
|
)
|
|
),
|
|
"workload", "$1", "owner_name", "(.*)"
|
|
)
|
|
)
|
|
- record: namespace_workload_pod:kube_pod_owner:relabel
|
|
labels:
|
|
workload_type: daemonset
|
|
expr: |
|
|
max by (cluster, namespace, workload, pod) (
|
|
label_replace(
|
|
kube_pod_owner{job="kube-state-metrics", owner_kind="DaemonSet"},
|
|
"workload", "$1", "owner_name", "(.*)"
|
|
)
|
|
)
|
|
- record: namespace_workload_pod:kube_pod_owner:relabel
|
|
labels:
|
|
workload_type: statefulset
|
|
expr: |
|
|
max by (cluster, namespace, workload, pod) (
|
|
label_replace(
|
|
kube_pod_owner{job="kube-state-metrics", owner_kind="StatefulSet"},
|
|
"workload", "$1", "owner_name", "(.*)"
|
|
)
|
|
)
|
|
- record: namespace_workload_pod:kube_pod_owner:relabel
|
|
labels:
|
|
workload_type: job
|
|
expr: |
|
|
max by (cluster, namespace, workload, pod) (
|
|
label_replace(
|
|
kube_pod_owner{job="kube-state-metrics", owner_kind="Job"},
|
|
"workload", "$1", "owner_name", "(.*)"
|
|
)
|
|
)
|