## Why
Both traefik instances (traefik-internal/traefik-external, chart 40.0.0, image v3.7.0) already expose prometheus metrics on container port `metrics` (9100), but nothing scrapes them — VictoriaMetrics has no traefik series (verified empty). The observability VMAgent runs with selectAllByDefault and consumes VM*Scrape CRs only, so a VMPodScrape is all that is needed.
## Changes
- Adds `apps/base/observability/vmpodscrape-traefik.yaml`: scrapes port `metrics` /metrics on `app.kubernetes.io/name: traefik` pods in traefik-system, relabeling `app.kubernetes.io/instance` to a stable `traefik_instance` label to split internal vs external series
- Registers the new CR in the observability base kustomization
- Enables `metrics.prometheus.addEntryPointsLabels` and `addRoutersLabels` in both au-syd1 traefik values files for per-entrypoint/per-route dashboards (pods restart on rollout — acceptable)
Validated with `kustomize build` (observability base + overlay, traefik-system overlay with --enable-helm) and kubeconform (unknown CRDs skipped); rendered deployments carry the new metric flags and the VMPodScrape renders in the overlay.
Reviewed-on: #397
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>
No redis/valkey instance in the estate exports metrics today. Adds a
pinned redis_exporter (v1.89.0 via the artifactapi dockerhub remote)
next to every instance and two cluster-wide VMPodScrapes so the
observability VMAgent picks them all up.
- enables the valkey-operator exporter sidecar on the jellyfin
ValkeyCluster, overriding the operator's bare-dockerhub default image
with the artifactapi-proxied pin
- adds a metrics-exporter sidecar (:9121, port name metrics) to the
artifactapi/authentik/litellm redis and gitea/netbox valkey
Deployments, with the monitoring.unkin.net/redis-exporter opt-in pod
label
- adds VMPodScrape redis-exporters selecting that label in any
namespace, and VMPodScrape valkey-operator-clusters for
operator-managed pods whose labels are fixed (the ValkeyCluster CR has
no pod-label passthrough)
Follow-up to #239. Every CNPG instance pod already exposes Prometheus metrics on `:9187` (`metrics` port), but nothing scraped them. Adds a single namespace-wide `VMPodScrape` (`namespaceSelector.any`, `selector cnpg.io/podRole=instance`) so the observability VMAgent collects postgres metrics for **all ~23 CNPG clusters** across the estate (authentik, grafana, woodpecker, artifactapi, puppet, litellm, …). No chart changes. Pairs well with the CNPG grafana.com dashboard (20417) as a further follow-up.
Reviewed-on: #242
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The Consul SD job discovered **0 targets**. vmagent logs showed it reaching the puppet Consul fine (TLS/connectivity OK) but getting `403` on `GET /v1/agent/self`: the anonymous token `lacks permission 'agent:read'`. VictoriaMetrics calls `/v1/agent/self` only to auto-detect the datacenter; catalog/health reads (what SD actually needs) work anonymously.
## Fix
Set `datacenter: au-syd1` on the consul_sd_config so VM skips the `agent/self` call. No consul token needed.
## Verify after sync
vmagent `/targets` → `consul` job shows the puppet targets (haproxy/ceph/gitea/node/…) up.
Reviewed-on: #240
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
## Why
The k8s au-syd1 VictoriaMetrics stack ran as two helm charts and only scraped in-cluster targets. The victoria-metrics-operator already runs in vm-system, so this moves the stack onto operator-managed CRDs. That unlocks VMServiceScrape/VMPodScrape (auto-converted from Prometheus ServiceMonitors, used by a follow-up PR) and adds Consul service discovery so the cluster scrapes the **same puppet-prod targets** as the puppet vmagent. Also shrinks vmstorage 3 → 2 (Ceph-backed, replicationFactor 2).
## Changes
- Add **VMCluster `main`**: vmstorage 2 replicas (cephrbd-fast-delete 200Gi, 180d retention, replicationFactor 2), vminsert/vmselect 2 replicas + HPA (2–10, 60% cpu).
- Add **VMAgent `main`**: retains the kubernetes SD jobs (apiservers/nodes/cadvisor), `selectAllByDefault` for VMServiceScrape/VMPodScrape, and a **Consul SD job** against `consul.service.consul` (resolves to the puppet Consul from pods) replicating the puppet vmagent relabels — keep tag `metrics`, `__scheme__` from `metrics_scheme`, `job` from `metrics_job`. TLS is **verified against the reflected `vault-ca-cert`** (no insecure skip-verify).
- Expose vmselect/vminsert/vmagent via **Gateway API** (traefik-internal Gateway + HTTPRoute, http→https redirect), same hostnames.
- Remove the two helm charts, their values files, and vendored charts.
## Notes
- Data wipe on cutover is acceptable (confirmed) — old helm PVCs can be deleted.
- Verify at rollout: pods resolve `*.main.unkin.net` node FQDNs (needed for CA SAN match on scrape targets); `/targets` shows `job=consul`.
Reviewed-on: #234
Co-authored-by: Ben Vincent <ben@unkin.net>
Co-committed-by: Ben Vincent <ben@unkin.net>
Migrate Victoria Metrics cluster and agent from Terragrunt to ArgoCD/Kustomize.
Creates new observability AppProject and ApplicationSet.
Deploys victoria-metrics-cluster v0.33.0 (vmselect/vminsert/vmstorage with
HPA, PDB, ingress) and victoria-metrics-agent v0.30.0 (3 replicas, k8s scrape
configs) in the observability namespace.
💘 Generated with Crush
Assisted-by: Claude Sonnet 4.6 via Crush <crush@charm.land>
Reviewed-on: #82
unkin-agent
unkinben