Alerting

​

How alerts get to people

​

When to use which severity

• Active customer impact (elevated error rates, full outage)
• Data loss or risk of data loss
• Pods that should be running but aren’t
• Anything where waiting until morning makes it worse

• Elevated latency that hasn’t crossed into “customers are mad” territory
• Resource (high memory, goroutine counts, connection pools filling up)
• Indicators that could become critical if left alone
• Anything you’d look at during business hours but wouldn’t lose sleep over

​

Current alerts

​

Custom (defined in this repo)

​

kube-prometheus-stack built-ins

• All etcd alerts
• KubeAPIDown, KubeAPIErrorBudgetBurn, KubeAPITerminatedRequests
• KubeAggregatedAPIDown, KubeAggregatedAPIErrors
• KubeControllerManagerDown, KubeSchedulerDown, KubeProxyDown
• KubeClientCertificateExpiration, KubeVersionMismatch
• KubeletClient/ServerCertificateExpiration and RenewalErrors
• NodeRAIDDegraded, NodeRAIDDiskFailure, NodeBondingDegraded

• KubePodCrashLooping — replaced by PodNotRunning which excludes krane-managed customer workloads
• KubePodNotReady — was firing for customer deployment containers

​

Kubernetes — critical

​

Kubernetes — warning

​

Kubelet — critical

​

Kubelet — warning

​

Node — critical

​

Node — warning

​

Alertmanager — critical

​

Alertmanager — warning

​

Prometheus — critical

​

Prometheus — warning

​

Prometheus Operator — warning

​

kube-state-metrics — critical

​

General

​

Info

​

Adding a new alert

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: <service>-alerts
  namespace: {{ .Release.Namespace }}
  labels:
    app.kubernetes.io/name: <service>-alerts
spec:
  groups:
    - name: <service>-health
      rules:
        - alert: SomethingBad
          expr: <your promql>
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Short description with {{ "{{" }} $labels.region {{ "}}" }}"
            description: "Longer explanation of what's happening."

​

Things to keep in mind

• Always set severity to critical or warning. No severity = no routing.
• Exclude customer workloads if your alert is pod-level. Add container!="deployment" to the metric selector and use unless on(namespace, pod) kube_pod_labels{label_app_kubernetes_io_managed_by="krane"} to skip krane-managed pods. We don’t want to page ourselves for a customer’s broken container.
• Include region in the summary when the data has it. Getting paged with “SomethingBad (us-east-1 / api)” is a lot more useful than just “SomethingBad.”
• Set a reasonable for duration. Too short and you get flapping alerts. Too long and you find out late. 2-5 minutes is a good starting point for most things.
• Think about staging vs. production. The environment external label is available on all metrics (production001 or staging). If you want different severity per environment, write two rules with different environment filters — one critical for production, one warning for staging.

​

incident.io