unkin-agent
e608dadae7
## Why
Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer.
## Changes
- replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel
- renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel
- adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote
- adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules)
- adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules)
## Prerequisite
The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff.
## Notes
- validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
- the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
Reviewed-on: #399
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
133 lines
4.6 KiB
YAML
133 lines
4.6 KiB
YAML
---
|
|
apiVersion: operator.victoriametrics.com/v1beta1
|
|
kind: VMAgent
|
|
metadata:
|
|
name: main
|
|
namespace: observability
|
|
spec:
|
|
replicaCount: 2
|
|
scrapeInterval: 15s
|
|
# Also consume VMServiceScrape / VMPodScrape / VMNodeScrape from every namespace
|
|
# (the operator auto-converts Prometheus ServiceMonitors -> VMServiceScrape).
|
|
selectAllByDefault: true
|
|
extraArgs:
|
|
loggerFormat: json
|
|
remoteWrite:
|
|
- url: http://vminsert-main.observability.svc.cluster.local:8480/insert/0/prometheus/
|
|
resources:
|
|
requests:
|
|
cpu: 250m
|
|
memory: 512Mi
|
|
limits:
|
|
cpu: "1"
|
|
memory: 2Gi
|
|
# Reflected Vault intermediate CA (unkin.net) for verifying puppet Consul + metrics targets.
|
|
volumes:
|
|
- name: vault-ca
|
|
secret:
|
|
secretName: vault-ca-cert
|
|
volumeMounts:
|
|
- name: vault-ca
|
|
mountPath: /etc/vmagent-tls
|
|
readOnly: true
|
|
inlineScrapeConfig: |
|
|
- job_name: vmagent
|
|
static_configs:
|
|
- targets: ["localhost:8429"]
|
|
- job_name: "kubernetes-apiservers"
|
|
kubernetes_sd_configs:
|
|
- role: endpoints
|
|
scheme: https
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
insecure_skip_verify: true
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
relabel_configs:
|
|
- source_labels:
|
|
- __meta_kubernetes_namespace
|
|
- __meta_kubernetes_service_name
|
|
- __meta_kubernetes_endpoint_port_name
|
|
action: keep
|
|
regex: default;kubernetes;https
|
|
# Only the node name goes onto series: a blanket node-label labelmap pushes past vminsert's 40-label limit and drops everything.
|
|
- job_name: "kubelet"
|
|
scheme: https
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
insecure_skip_verify: true
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
kubernetes_sd_configs:
|
|
- role: node
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_node_name]
|
|
target_label: node
|
|
- source_labels: [__metrics_path__]
|
|
target_label: metrics_path
|
|
- job_name: "kubelet-cadvisor"
|
|
scheme: https
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
insecure_skip_verify: true
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
kubernetes_sd_configs:
|
|
- role: node
|
|
metrics_path: /metrics/cadvisor
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_node_name]
|
|
target_label: node
|
|
- source_labels: [__metrics_path__]
|
|
target_label: metrics_path
|
|
# job_name must be unique; kube-prometheus selectors expect job="kubelet"
|
|
- target_label: job
|
|
replacement: kubelet
|
|
metric_relabel_configs:
|
|
- action: replace
|
|
source_labels: [pod]
|
|
regex: '(.+)'
|
|
target_label: pod_name
|
|
replacement: '${1}'
|
|
- action: replace
|
|
source_labels: [container]
|
|
regex: '(.+)'
|
|
target_label: container_name
|
|
replacement: '${1}'
|
|
- action: replace
|
|
target_label: name
|
|
replacement: k8s_stub
|
|
- action: replace
|
|
source_labels: [id]
|
|
regex: '^/system\.slice/(.+)\.service$'
|
|
target_label: systemd_service_name
|
|
replacement: '${1}'
|
|
# puppet-prod Consul service discovery (same targets as the puppet vmagent).
|
|
# consul.service.consul resolves to the puppet Consul from in-cluster pods.
|
|
- job_name: consul
|
|
consul_sd_configs:
|
|
- server: consul.service.consul:443
|
|
scheme: https
|
|
# Set datacenter explicitly so VM does not call /v1/agent/self
|
|
# (which needs agent:read that the anonymous token lacks); catalog
|
|
# and health reads work anonymously.
|
|
datacenter: au-syd1
|
|
tls_config:
|
|
ca_file: /etc/vmagent-tls/ca.crt
|
|
relabel_configs:
|
|
- source_labels: [__meta_consul_tagpresent_metrics]
|
|
regex: "true"
|
|
action: keep
|
|
- source_labels: [__meta_consul_node, __meta_consul_service_port]
|
|
separator: ":"
|
|
target_label: __address__
|
|
replacement: "${1}:${2}"
|
|
action: replace
|
|
- source_labels: [__meta_consul_tag_metrics_scheme]
|
|
target_label: __scheme__
|
|
action: replace
|
|
- target_label: __metrics_path__
|
|
replacement: /metrics
|
|
- source_labels: [__meta_consul_tag_metrics_job]
|
|
target_label: job
|
|
action: replace
|
|
tls_config:
|
|
ca_file: /etc/vmagent-tls/ca.crt
|