## Why
Kubernetes container metrics are entirely missing from VictoriaMetrics, leaving the CNPG dashboard CPU panels empty. The `kubernetes-nodes`/`kubernetes-nodes-cadvisor` scrape jobs use a blanket `labelmap` on `__meta_kubernetes_node_label_(.+)`, copying ~50+ node-feature-discovery labels onto every series; every kubelet/cadvisor series then exceeds vminsert's 40-label default and 100% are silently dropped (`vm_rows_ignored_total{reason="too_many_labels"}` ~38B). On top of that, kube-state-metrics, VMAlert and the kube-prometheus `k8s.rules` recording rules were never deployed, so `node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate` (and companions) have no producer.
## Changes
- replaces the blanket node-label `labelmap` in both node scrape jobs with a single `__meta_kubernetes_node_name -> node` relabel
- renames the node jobs so series carry `job="kubelet"` plus `metrics_path` (`/metrics` and `/metrics/cadvisor`), matching kube-prometheus selectors; the cadvisor job keeps a distinct `job_name` (uniqueness requirement) and forces `job=kubelet` via relabel
- adds kube-state-metrics v2.20.0 (Deployment, standard ClusterRole/Binding, headless Service, VMServiceScrape with `jobLabel` yielding `job="kube-state-metrics"`), image routed through the artifactapi `k8s-registry` remote
- adds a VMAlert CR (datasource vmselect-main, remoteWrite/remoteRead vminsert/vmselect-main, `notifier.blackhole` since it only evaluates recording rules)
- adds a VMRule with the kube-prometheus `k8s.rules` recording group (cpu sum_irate, memory working_set/rss/cache/swap, namespace requests/limits sums, `kube_pod_owner` workload relabel rules)
## Prerequisite
The artifactapi `k8s-registry` remote exists but its `patterns` allowlist (`^autoscaling/vpa-`, `^external-dns/`, `^sig-storage/`) does not cover `kube-state-metrics/` — terraform-artifactapi needs `^kube-state-metrics/` added to `config/remote_docker/k8s-registry.yaml` before/with this merge, or the KSM pod will ImagePullBackOff.
## Notes
- validated with `kustomize build` (base + au-syd1 overlay) and kubeconform (VM CRD schemas skipped)
- the grafana CNPG dashboard also has stale datasource uids — separate follow-up, deliberately not touched here
Reviewed-on: #399
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
## Why
Both traefik instances (traefik-internal/traefik-external, chart 40.0.0, image v3.7.0) already expose prometheus metrics on container port `metrics` (9100), but nothing scrapes them — VictoriaMetrics has no traefik series (verified empty). The observability VMAgent runs with selectAllByDefault and consumes VM*Scrape CRs only, so a VMPodScrape is all that is needed.
## Changes
- Adds `apps/base/observability/vmpodscrape-traefik.yaml`: scrapes port `metrics` /metrics on `app.kubernetes.io/name: traefik` pods in traefik-system, relabeling `app.kubernetes.io/instance` to a stable `traefik_instance` label to split internal vs external series
- Registers the new CR in the observability base kustomization
- Enables `metrics.prometheus.addEntryPointsLabels` and `addRoutersLabels` in both au-syd1 traefik values files for per-entrypoint/per-route dashboards (pods restart on rollout — acceptable)
Validated with `kustomize build` (observability base + overlay, traefik-system overlay with --enable-helm) and kubeconform (unknown CRDs skipped); rendered deployments carry the new metric flags and the VMPodScrape renders in the overlay.
Reviewed-on: #397
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
Follow-up to #239. Every CNPG instance pod already exposes Prometheus metrics on `:9187` (`metrics` port), but nothing scraped them. Adds a single namespace-wide `VMPodScrape` (`namespaceSelector.any`, `selector cnpg.io/podRole=instance`) so the observability VMAgent collects postgres metrics for **all ~23 CNPG clusters** across the estate (authentik, grafana, woodpecker, artifactapi, puppet, litellm, …). No chart changes. Pairs well with the CNPG grafana.com dashboard (20417) as a further follow-up.
Reviewed-on: #242
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Consul SD job discovered **0 targets**. vmagent logs showed it reaching the puppet Consul fine (TLS/connectivity OK) but getting `403` on `GET /v1/agent/self`: the anonymous token `lacks permission 'agent:read'`. VictoriaMetrics calls `/v1/agent/self` only to auto-detect the datacenter; catalog/health reads (what SD actually needs) work anonymously.
## Fix
Set `datacenter: au-syd1` on the consul_sd_config so VM skips the `agent/self` call. No consul token needed.
## Verify after sync
vmagent `/targets` → `consul` job shows the puppet targets (haproxy/ceph/gitea/node/…) up.
Reviewed-on: #240
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The k8s au-syd1 VictoriaMetrics stack ran as two helm charts and only scraped in-cluster targets. The victoria-metrics-operator already runs in vm-system, so this moves the stack onto operator-managed CRDs. That unlocks VMServiceScrape/VMPodScrape (auto-converted from Prometheus ServiceMonitors, used by a follow-up PR) and adds Consul service discovery so the cluster scrapes the **same puppet-prod targets** as the puppet vmagent. Also shrinks vmstorage 3 → 2 (Ceph-backed, replicationFactor 2).
## Changes
- Add **VMCluster `main`**: vmstorage 2 replicas (cephrbd-fast-delete 200Gi, 180d retention, replicationFactor 2), vminsert/vmselect 2 replicas + HPA (2–10, 60% cpu).
- Add **VMAgent `main`**: retains the kubernetes SD jobs (apiservers/nodes/cadvisor), `selectAllByDefault` for VMServiceScrape/VMPodScrape, and a **Consul SD job** against `consul.service.consul` (resolves to the puppet Consul from pods) replicating the puppet vmagent relabels — keep tag `metrics`, `__scheme__` from `metrics_scheme`, `job` from `metrics_job`. TLS is **verified against the reflected `vault-ca-cert`** (no insecure skip-verify).
- Expose vmselect/vminsert/vmagent via **Gateway API** (traefik-internal Gateway + HTTPRoute, http→https redirect), same hostnames.
- Remove the two helm charts, their values files, and vendored charts.
## Notes
- Data wipe on cutover is acceptable (confirmed) — old helm PVCs can be deleted.
- Verify at rollout: pods resolve `*.main.unkin.net` node FQDNs (needed for CA SAN match on scrape targets); `/targets` shows `job=consul`.
Reviewed-on: #234
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Migrate Victoria Metrics cluster and agent from Terragrunt to ArgoCD/Kustomize.
Creates new observability AppProject and ApplicationSet.
Deploys victoria-metrics-cluster v0.33.0 (vmselect/vminsert/vmstorage with
HPA, PDB, ingress) and victoria-metrics-agent v0.30.0 (3 replicas, k8s scrape
configs) in the observability namespace.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #82
unkin-agent
unkinben