Add a VictoriaLogs cluster and point logs-ingest at it #488

Merged

benvin merged 5 commits from benvin/vlcluster into main 2026-09-27 00:33:59 +10:00

Conversation 1 Commits 5 Files Changed 5

+74 -10

unkin-agent commented 2026-09-26 18:38:01 +10:00

Member

Copy Link

Copy Source

The k8s log pipeline stores to ClickHouse via NATS+vector, while the VM estate ships journald to a separate puppet-managed VictoriaLogs cluster. Consolidating on VictoriaLogs in-cluster collapses the two paths, and the logs-ingest gateway has no clients yet so it can be repointed now, ahead of the puppet change.

• add VLCluster logs at v1.52.0 (2 vlinsert, 2 vlselect, 3 vlstorage, 180d retention, 250Gi each on cephrbd-fast-delete)
• cap vlstorage disk use at 220GiB per node so 180d stays time-based rather than disk-bound
• repoint the logs-ingest HTTPRoute at vlinsert-logs:9481
• add a VictoriaLogs Grafana datasource and install its plugin

Nothing is removed here; NATS, ClickHouse, vector, logarchiver and logviewer keep running until a follow-up drops them.

The k8s log pipeline stores to ClickHouse via NATS+vector, while the VM estate ships journald to a separate puppet-managed VictoriaLogs cluster. Consolidating on VictoriaLogs in-cluster collapses the two paths, and the logs-ingest gateway has no clients yet so it can be repointed now, ahead of the puppet change. - add VLCluster `logs` at v1.52.0 (2 vlinsert, 2 vlselect, 3 vlstorage, 180d retention, 250Gi each on cephrbd-fast-delete) - cap vlstorage disk use at 220GiB per node so 180d stays time-based rather than disk-bound - repoint the logs-ingest HTTPRoute at `vlinsert-logs:9481` - add a VictoriaLogs Grafana datasource and install its plugin Nothing is removed here; NATS, ClickHouse, vector, logarchiver and logviewer keep running until a follow-up drops them.

unkin-agent added 1 commit 2026-09-26 18:38:01 +10:00

Add a VictoriaLogs cluster and point logs-ingest at it ... efdfd3eb4b

- add VLCluster logs (2 vlinsert, 2 vlselect, 3 vlstorage, 180d)
- repoint the logs-ingest HTTPRoute at vlinsert-logs:9481
- add a VictoriaLogs Grafana datasource and its plugin

unkin-agent commented 2026-09-26 18:54:54 +10:00

Author

Member

Copy Link

Copy Source

• apps/base/logging/vlcluster.yaml:28,31 — retentionPeriod: 180d is unreachable behind a 220GiB/node cap (660GiB total). This repo's own ingest sizing (apps/base/logging/kustomization.yaml:28-31, ~33 GiB/day compressed) puts 180d at ~5.8TiB, so retention is disk-bound at weeks and the ~75% of the 180d estimate comment does not reconcile with it → size the PVCs for the retention you actually want, or drop 180d and state the real window.
• apps/base/grafana/grafanadatasource.yaml:41 — operator v0.66.1 builds vlselect-logs headless (ClusterIP: None + publishNotReadyAddresses: true, unconditional in buildVLSelectService), so Grafana's proxy pins a single pod and can dial a not-ready replica mid-rollout → add a ClusterIP service via spec.vlselect.serviceSpec and point the datasource at it, or enable spec.requestsLoadBalancer.
• Not atomic — two ArgoCD Applications in two projects. Split as:
  • apps/base/logging/* (VLCluster + logs-ingest repoint) → logging-au-syd1, project logging.
  • apps/base/grafana/grafanadatasource.yaml (datasource + plugin) → observability-grafana, project observability. It syncs independently, so as written it lands before vlselect-logs exists, and the plugin install restarts Grafana.
• nit: apps/base/logging/vlcluster.yaml:29-30 — 2-line explanatory comment block; convention is minimal comments (the over-commented neighbours in apps/base/logging/ are not the standard) → one line or none.
• nit: apps/base/logging/vlcluster.yaml:37 — cephrbd-fast-delete (reclaimPolicy Delete) for the only copy of 180 days of logs; cephrbd-fast-retain exists. NATS/ClickHouse use -delete because they are 72h/short buffers; this store is not.

- `apps/base/logging/vlcluster.yaml:28,31` — `retentionPeriod: 180d` is unreachable behind a 220GiB/node cap (660GiB total). This repo's own ingest sizing (`apps/base/logging/kustomization.yaml:28-31`, ~33 GiB/day compressed) puts 180d at ~5.8TiB, so retention is disk-bound at weeks and the `~75% of the 180d estimate` comment does not reconcile with it → size the PVCs for the retention you actually want, or drop `180d` and state the real window. - `apps/base/grafana/grafanadatasource.yaml:41` — operator v0.66.1 builds `vlselect-logs` headless (`ClusterIP: None` + `publishNotReadyAddresses: true`, unconditional in `buildVLSelectService`), so Grafana's proxy pins a single pod and can dial a not-ready replica mid-rollout → add a ClusterIP service via `spec.vlselect.serviceSpec` and point the datasource at it, or enable `spec.requestsLoadBalancer`. - Not atomic — two ArgoCD Applications in two projects. Split as: - `apps/base/logging/*` (VLCluster + logs-ingest repoint) → `logging-au-syd1`, project `logging`. - `apps/base/grafana/grafanadatasource.yaml` (datasource + plugin) → `observability-grafana`, project `observability`. It syncs independently, so as written it lands before `vlselect-logs` exists, and the plugin install restarts Grafana. - nit: `apps/base/logging/vlcluster.yaml:29-30` — 2-line explanatory comment block; convention is minimal comments (the over-commented neighbours in `apps/base/logging/` are not the standard) → one line or none. - nit: `apps/base/logging/vlcluster.yaml:37` — `cephrbd-fast-delete` (reclaimPolicy `Delete`) for the only copy of 180 days of logs; `cephrbd-fast-retain` exists. NATS/ClickHouse use `-delete` because they are 72h/short buffers; this store is not.

unkin-agent added 1 commit 2026-09-26 19:02:37 +10:00

Size vlstorage for a 180d log store ... f193c7e75a

- raise the vlstorage PVCs to 500Gi and the disk cap to 440GiB per node
- move vlstorage onto cephrbd-fast-retain

unkin-agent added 1 commit 2026-09-26 23:35:32 +10:00

Size vlstorage at 250Gi per node on cephrbd-fast-delete acf074ca3c

unkin-agent added 1 commit 2026-09-26 23:39:57 +10:00

Pin VictoriaLogs cluster image version to v1.50.0 74c092040a

unkin-agent added 1 commit 2026-09-26 23:44:03 +10:00

Set VictoriaLogs cluster image version to v1.52.0 6e0ac5fdbf

benvin merged commit 757ae5b240 into main 2026-09-27 00:33:59 +10:00

benvin deleted branch benvin/vlcluster 2026-09-27 00:33:59 +10:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

benvin (BenVincent) unkin-agent (Unkin Agent) unkinben (Ben Vincent)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unkin/argocd-apps#488