From 9a56612ae6fa040c89605b953d6285c818ffb90a Mon Sep 17 00:00:00 2001
From: Ben Vincent <ben@unkin.net>
Date: Mon, 6 Jul 2026 23:57:54 +1000
Subject: [PATCH] observability: scrape existing app metrics (audit + ceph-csi
 batch) (#239)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

## Why
Audit of existing helm deployments for (a) metrics collection and (b) bundled Grafana dashboards, per request. This PR enables the first, cleanest batch (ceph-csi) and documents the rest for follow-up review.

## Key constraint
The cluster has **no Prometheus-Operator CRDs** (`servicemonitors.monitoring.coreos.com` absent) — only the VM-native `VMServiceScrape`/`VMPodScrape`. So flipping charts' `serviceMonitor: true` toggles would fail. The correct mechanism here is native **VMServiceScrape** CRs, which the observability VMAgent picks up (`selectAllByDefault`, added in #234). Bundled chart dashboards ship as ConfigMaps that grafana-operator v5 ignores (it uses CRs), so "enabling" them means adding `GrafanaDashboard` CRs.

## This PR
- `VMServiceScrape` for **csi-cephfs** and **csi-cephrbd** (their `*-http-metrics` services already exist).

## Audit — metrics (apps already exposing a metrics service; enable via VMServiceScrape)
| App | Metrics service / port | Note |
|-----|------------------------|------|
| csi-cephfs / csi-cephrbd | `*-http-metrics` :8080 | **done here** |
| cert-manager | webhook :9402 `metrics`, cainjector :9402 `http-metrics` | multi-port; easy follow-up |
| argocd | `argocd-*-metrics` :8082/8083/8084/9001 | many components |
| media-apps | `prowlarr/radarr/sonarr-metrics` :9707 (exportarr) | follow-up |
| vm-system / grafana-operator | operator metrics | low value |

## Audit — charts with metrics *toggles* (need endpoint enabled + VMServiceScrape)
authentik, vault, purelb, externaldns, cnpg-system (`enablePodMonitor`), reloader, traefik. These expose `serviceMonitor`/`podMonitor` toggles that assume Prometheus CRDs — recommend enabling the metrics endpoint + a VMServiceScrape rather than the toggle.

## Audit — bundled Grafana dashboards
- **CNPG**: ships a dashboard (grafana.com id 20417) — add as `GrafanaDashboard` (grafanaCom).
- **traefik**: `dashboard: true` option + community dashboards — add as `GrafanaDashboard`.
- Others: none bundled.

## Review ask
Confirm the VMServiceScrape approach; I'll extend to the remaining apps (cert-manager/argocd/media-apps + CNPG/traefik dashboards) in follow-up batches.

Reviewed-on: https://git.unkin.net/unkin/argocd-apps/pulls/239
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
---
 apps/base/csi-cephfs/kustomization.yaml    |  1 +
 apps/base/csi-cephfs/vmservicescrape.yaml  | 15 +++++++++++++++
 apps/base/csi-cephrbd/kustomization.yaml   |  1 +
 apps/base/csi-cephrbd/vmservicescrape.yaml | 15 +++++++++++++++
 4 files changed, 32 insertions(+)
 create mode 100644 apps/base/csi-cephfs/vmservicescrape.yaml
 create mode 100644 apps/base/csi-cephrbd/vmservicescrape.yaml

diff --git a/apps/base/csi-cephfs/kustomization.yaml b/apps/base/csi-cephfs/kustomization.yaml
index 5849aad..cd444d8 100644
--- a/apps/base/csi-cephfs/kustomization.yaml
+++ b/apps/base/csi-cephfs/kustomization.yaml
@@ -7,3 +7,4 @@ resources:
   - vaultauth.yaml
   - vaultstaticsecret.yaml
   - storageclass.yaml
+  - vmservicescrape.yaml
diff --git a/apps/base/csi-cephfs/vmservicescrape.yaml b/apps/base/csi-cephfs/vmservicescrape.yaml
new file mode 100644
index 0000000..3362347
--- /dev/null
+++ b/apps/base/csi-cephfs/vmservicescrape.yaml
@@ -0,0 +1,15 @@
+---
+# Scrape the ceph-csi-cephfs nodeplugin + provisioner http-metrics endpoints.
+# Picked up by the observability VMAgent (selectAllByDefault).
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMServiceScrape
+metadata:
+  name: ceph-csi-cephfs
+  namespace: csi-cephfs
+spec:
+  selector:
+    matchLabels:
+      app: ceph-csi-cephfs
+  endpoints:
+    - port: http-metrics
+      path: /metrics
diff --git a/apps/base/csi-cephrbd/kustomization.yaml b/apps/base/csi-cephrbd/kustomization.yaml
index 5849aad..cd444d8 100644
--- a/apps/base/csi-cephrbd/kustomization.yaml
+++ b/apps/base/csi-cephrbd/kustomization.yaml
@@ -7,3 +7,4 @@ resources:
   - vaultauth.yaml
   - vaultstaticsecret.yaml
   - storageclass.yaml
+  - vmservicescrape.yaml
diff --git a/apps/base/csi-cephrbd/vmservicescrape.yaml b/apps/base/csi-cephrbd/vmservicescrape.yaml
new file mode 100644
index 0000000..ed08260
--- /dev/null
+++ b/apps/base/csi-cephrbd/vmservicescrape.yaml
@@ -0,0 +1,15 @@
+---
+# Scrape the ceph-csi-rbd nodeplugin + provisioner http-metrics endpoints.
+# Picked up by the observability VMAgent (selectAllByDefault).
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMServiceScrape
+metadata:
+  name: ceph-csi-rbd
+  namespace: csi-cephrbd
+spec:
+  selector:
+    matchLabels:
+      app: ceph-csi-rbd
+  endpoints:
+    - port: http-metrics
+      path: /metrics