grafana: scale to 3 replicas, raise limits (#508)

A single Grafana pod with a 1 cpu/1Gi limit is a single point of failure and gets throttled under dashboard load. State lives in Postgres, so extra replicas are safe; spreading them keeps a node loss from taking out Grafana.

- set `replicas: 3` on the grafana deployment
- raise grafana container limits to 2 cpu / 4Gi
- spread grafana pods across nodes with a soft hostname topology constraint

Reviewed-on: #508
Co-authored-by: unkin-agent <unkin-agent@unkin.net>
Co-committed-by: unkin-agent <unkin-agent@unkin.net>

apps/base/grafana/grafana.yaml

@@ -9,8 +9,16 @@ metadata:
 spec:
   deployment:
     spec:
+      replicas: 3
       template:
         spec:
+          topologySpreadConstraints:
+            - maxSkew: 1
+              topologyKey: kubernetes.io/hostname
+              whenUnsatisfiable: ScheduleAnyway
+              labelSelector:
+                matchLabels:
+                  app: grafana
           containers:
             - name: grafana
               env:
@@ -31,8 +39,8 @@ spec:
                   cpu: 100m
                   memory: 256Mi
                 limits:
-                  cpu: "1"
-                  memory: 1Gi
+                  cpu: "2"
+                  memory: 4Gi
   config:
     server:
       root_url: "https://grafana.k8s.syd1.au.unkin.net"