observability: migrate VictoriaMetrics from helm charts to operator CRDs
ci/woodpecker/pr/pre-commit Pipeline was successful
ci/woodpecker/pr/kubeconform Pipeline was successful

The k8s au-syd1 VictoriaMetrics stack ran as two helm charts
(victoria-metrics-cluster + victoria-metrics-agent) and only scraped
in-cluster targets. The victoria-metrics-operator already runs in
vm-system, so move the stack onto operator-managed CRDs. This lets the
VMAgent consume VMServiceScrape/VMPodScrape (auto-converted from
Prometheus ServiceMonitors) and adds Consul service discovery so the
cluster scrapes the same puppet-prod targets as the puppet vmagent.

Changes:
- Add VMCluster `main`: vmstorage 2 replicas (down from 3, replicationFactor
  2, cephrbd-fast-delete 200Gi, 180d retention), vminsert/vmselect 2 replicas
  + HPA (2-10, 60% cpu).
- Add VMAgent `main`: keeps the kubernetes SD jobs (apiservers/nodes/cadvisor),
  selectAllByDefault for VMServiceScrape/VMPodScrape, and a Consul SD job
  against consul.service.consul (puppet Consul) replicating the puppet vmagent
  relabels (keep tag `metrics`, scheme from `metrics_scheme`, job from
  `metrics_job`). TLS verified against the reflected vault-ca-cert (no
  insecure skip-verify).
- Expose vmselect/vminsert/vmagent via Gateway API (traefik-internal Gateway +
  HTTPRoute, http->https redirect), same hostnames as before.
- Remove the two helm charts, their values files, and vendored charts.
This commit is contained in:
2026-07-05 22:09:59 +10:00
parent 53b55419a7
commit 41ab3ff614
8 changed files with 523 additions and 301 deletions
+165
View File
@@ -0,0 +1,165 @@
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vmselect-http-redirect
namespace: observability
labels:
app.kubernetes.io/name: vmselect
app.kubernetes.io/instance: victoria-metrics
spec:
hostnames:
- vmselect.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: vmselect
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vmselect
namespace: observability
labels:
app.kubernetes.io/name: vmselect
app.kubernetes.io/instance: victoria-metrics
spec:
hostnames:
- vmselect.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: vmselect
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: vmselect-main
port: 8481
weight: 1
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vminsert-http-redirect
namespace: observability
labels:
app.kubernetes.io/name: vminsert
app.kubernetes.io/instance: victoria-metrics
spec:
hostnames:
- vminsert.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: vminsert
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vminsert
namespace: observability
labels:
app.kubernetes.io/name: vminsert
app.kubernetes.io/instance: victoria-metrics
spec:
hostnames:
- vminsert.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: vminsert
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: vminsert-main
port: 8480
weight: 1
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vmagent-http-redirect
namespace: observability
labels:
app.kubernetes.io/name: vmagent
app.kubernetes.io/instance: victoria-metrics
spec:
hostnames:
- vmagent.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: vmagent
sectionName: http
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vmagent
namespace: observability
labels:
app.kubernetes.io/name: vmagent
app.kubernetes.io/instance: victoria-metrics
spec:
hostnames:
- vmagent.k8s.syd1.au.unkin.net
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: vmagent
sectionName: https
rules:
- backendRefs:
- group: ""
kind: Service
name: vmagent-main
port: 8429
weight: 1
matches:
- path:
type: PathPrefix
value: /