unkin-agent
e608dadae7
## Why
Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer.
## Changes
- replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel
- renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel
- adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote
- adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules)
- adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules)
## Prerequisite
The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff.
## Notes
- validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
- the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
Reviewed-on: #399
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
202 lines
4.7 KiB
YAML
202 lines
4.7 KiB
YAML
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: kube-state-metrics
|
|
namespace: observability
|
|
labels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: kube-state-metrics
|
|
labels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources:
|
|
- configmaps
|
|
- secrets
|
|
- nodes
|
|
- pods
|
|
- services
|
|
- serviceaccounts
|
|
- resourcequotas
|
|
- replicationcontrollers
|
|
- limitranges
|
|
- persistentvolumeclaims
|
|
- persistentvolumes
|
|
- namespaces
|
|
- endpoints
|
|
verbs: [list, watch]
|
|
- apiGroups: [apps]
|
|
resources:
|
|
- statefulsets
|
|
- daemonsets
|
|
- deployments
|
|
- replicasets
|
|
verbs: [list, watch]
|
|
- apiGroups: [batch]
|
|
resources:
|
|
- cronjobs
|
|
- jobs
|
|
verbs: [list, watch]
|
|
- apiGroups: [autoscaling]
|
|
resources:
|
|
- horizontalpodautoscalers
|
|
verbs: [list, watch]
|
|
- apiGroups: [authentication.k8s.io]
|
|
resources:
|
|
- tokenreviews
|
|
verbs: [create]
|
|
- apiGroups: [authorization.k8s.io]
|
|
resources:
|
|
- subjectaccessreviews
|
|
verbs: [create]
|
|
- apiGroups: [policy]
|
|
resources:
|
|
- poddisruptionbudgets
|
|
verbs: [list, watch]
|
|
- apiGroups: [certificates.k8s.io]
|
|
resources:
|
|
- certificatesigningrequests
|
|
verbs: [list, watch]
|
|
- apiGroups: [discovery.k8s.io]
|
|
resources:
|
|
- endpointslices
|
|
verbs: [list, watch]
|
|
- apiGroups: [storage.k8s.io]
|
|
resources:
|
|
- storageclasses
|
|
- volumeattachments
|
|
verbs: [list, watch]
|
|
- apiGroups: [admissionregistration.k8s.io]
|
|
resources:
|
|
- mutatingwebhookconfigurations
|
|
- validatingwebhookconfigurations
|
|
verbs: [list, watch]
|
|
- apiGroups: [networking.k8s.io]
|
|
resources:
|
|
- networkpolicies
|
|
- ingressclasses
|
|
- ingresses
|
|
verbs: [list, watch]
|
|
- apiGroups: [coordination.k8s.io]
|
|
resources:
|
|
- leases
|
|
verbs: [list, watch]
|
|
- apiGroups: [rbac.authorization.k8s.io]
|
|
resources:
|
|
- clusterrolebindings
|
|
- clusterroles
|
|
- rolebindings
|
|
- roles
|
|
verbs: [list, watch]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: kube-state-metrics
|
|
labels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: kube-state-metrics
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: kube-state-metrics
|
|
namespace: observability
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: kube-state-metrics
|
|
namespace: observability
|
|
labels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
spec:
|
|
serviceAccountName: kube-state-metrics
|
|
automountServiceAccountToken: true
|
|
containers:
|
|
- name: kube-state-metrics
|
|
image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0
|
|
ports:
|
|
- name: http-metrics
|
|
containerPort: 8080
|
|
- name: telemetry
|
|
containerPort: 8081
|
|
livenessProbe:
|
|
httpGet:
|
|
path: /livez
|
|
port: http-metrics
|
|
initialDelaySeconds: 5
|
|
timeoutSeconds: 5
|
|
readinessProbe:
|
|
httpGet:
|
|
path: /readyz
|
|
port: telemetry
|
|
initialDelaySeconds: 5
|
|
timeoutSeconds: 5
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop: [ALL]
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
runAsUser: 65534
|
|
seccompProfile:
|
|
type: RuntimeDefault
|
|
resources:
|
|
requests:
|
|
cpu: 50m
|
|
memory: 128Mi
|
|
limits:
|
|
cpu: 500m
|
|
memory: 512Mi
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: kube-state-metrics
|
|
namespace: observability
|
|
labels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
spec:
|
|
clusterIP: None
|
|
selector:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
ports:
|
|
- name: http-metrics
|
|
port: 8080
|
|
targetPort: http-metrics
|
|
- name: telemetry
|
|
port: 8081
|
|
targetPort: telemetry
|
|
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMServiceScrape
|
|
metadata:
|
|
name: kube-state-metrics
|
|
namespace: observability
|
|
spec:
|
|
# jobLabel yields job="kube-state-metrics" as kube-prometheus selectors expect.
|
|
jobLabel: app.kubernetes.io/name
|
|
selector:
|
|
matchLabels:
|
|
app.kubernetes.io/name: kube-state-metrics
|
|
endpoints:
|
|
- port: http-metrics
|
|
honorLabels: true
|
|
- port: telemetry
|