Devops
DevopsIntermediate

Monitoring Kubernetes Clusters with Prometheus and Grafana

DeveloperHat Team
4 min read
KubernetesMonitoringPrometheusGrafana

TL;DR

A comprehensive guide to implementing effective monitoring for your Kubernetes clusters using Prometheus and Grafana.

Monitoring Kubernetes Clusters with Prometheus and Grafana

Effective monitoring is crucial for maintaining healthy Kubernetes clusters. This guide will show you how to set up comprehensive monitoring using Prometheus and Grafana.

$1

$1

1. Node metrics

2. Pod metrics

3. Container metrics

4. Application metrics

5. Network metrics

$1

``mermaid

graph LR

A[Kubernetes Cluster] --> B[Prometheus]

B --> C[Grafana]

D[Node Exporter] --> B

E[kube-state-metrics] --> B

`

$1

$1

`bash

Add Prometheus Helm repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

Install Prometheus

helm install prometheus prometheus-community/prometheus \

--namespace monitoring \

--create-namespace

`

$1

`yaml

prometheus-values.yaml

server:

persistentVolume:

size: 50Gi

retention: 15d

alertmanager:

enabled: true

persistence:

enabled: true

size: 10Gi

nodeExporter:

enabled: true

kubeStateMetrics:

enabled: true

`

$1

$1

`bash

Add Grafana Helm repository

helm repo add grafana https://grafana.github.io/helm-charts

helm repo update

Install Grafana

helm install grafana grafana/grafana \

--namespace monitoring \

--set persistence.enabled=true \

--set persistence.size=10Gi

`

$1

`json

{

"dashboard": {

"id": null,

"title": "Kubernetes Cluster Overview",

"panels": [

{

"title": "CPU Usage",

"type": "graph",

"targets": [

{

"expr": "sum(rate(container_cpu_usage_seconds_total{container!=\"\"}[5m])) by (pod)"

}

]

}

]

}

}

`

$1

$1

`yaml

Node alerts

groups:

  • name: node.rules

    • rules:

    - alert: NodeHighCPU

    expr: instance:node_cpu_utilisation:rate5m > 0.8

    for: 10m

    labels:

    severity: warning

    annotations:

    description: "CPU usage on {{ $labels.instance }} is above 80%"

    `

    $1

    `yaml

    Pod alerts

    groups:

  • name: pod.rules

    • rules:

    - alert: PodHighMemory

    expr: container_memory_usage_bytes > 1.5 1024 * 1024 1024

    for: 5m

    labels:

    severity: warning

    annotations:

    description: "Pod {{ $labels.pod }} is using too much memory"

    `

    $1

    $1

    `yaml

    apiVersion: monitoring.coreos.com/v1

    kind: ServiceMonitor

    metadata:

    name: app-monitor

    spec:

    selector:

    matchLabels:

    app: myapp

    endpoints:

    - port: metrics

    `

    $1

    `yaml

    groups:

  • name: custom.rules

    • rules:

    - record: job:http_requests_total:rate5m

    expr: rate(http_requests_total[5m])

    `

    $1

    $1

    `yaml

    resources:

    requests:

    cpu: 200m

    memory: 256Mi

    limits:

    cpu: 500m

    memory: 512Mi

    `

    $1

    `yaml

    prometheus:

    retention:

    time: 15d

    size: 50GB

    `

    $1

    `yaml

    prometheus:

    replicaCount: 2

    affinity:

    podAntiAffinity:

    requiredDuringSchedulingIgnoredDuringExecution:

    - topologyKey: kubernetes.io/hostname

    `

    $1

    $1

    1. High memory usage

    2. Metric collection gaps

    3. Slow query performance

    4. Dashboard loading issues

    $1

    `bash

    Check Prometheus status

    kubectl get pods -n monitoring

    kubectl logs -f prometheus-server-xyz -n monitoring

    Check Grafana status

    kubectl get pods -n monitoring

    kubectl logs -f grafana-xyz -n monitoring

    `

    $1

    $1

    `promql

    Bad query

    rate(http_requests_total[5m])

    Better query

    sum by (service) (rate(http_requests_total[5m]))

    `

    $1

    `yaml

    prometheus:

    resources:

    requests:

    cpu: 1

    memory: 2Gi

    limits:

    cpu: 2

    memory: 4Gi

    ``

    $1

    Effective Kubernetes monitoring with Prometheus and Grafana requires:

    1. Proper setup and configuration

    2. Regular maintenance

    3. Performance optimization

    4. Alert tuning

    5. Resource management

    Follow these guidelines to maintain a healthy and observable Kubernetes cluster.

    $1

    Here are valuable resources for Kubernetes monitoring:

    1. [Prometheus Documentation](https://prometheus.io/docs/introduction/overview/) - Official Prometheus documentation

    2. [Grafana Documentation](https://grafana.com/docs/) - Official Grafana documentation

    3. [Kubernetes Monitoring Guide](https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/) - Official K8s monitoring guide

    4. [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) - Automated Prometheus setup

    5. [Grafana Dashboards](https://grafana.com/grafana/dashboards/) - Pre-built Grafana dashboards

    6. [Kubernetes Metrics](https://kubernetes.io/docs/reference/instrumentation/metrics/) - K8s metrics reference

    7. [PromQL Basics](https://prometheus.io/docs/prometheus/latest/querying/basics/) - Prometheus Query Language guide

    8. [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) - Prometheus alerting

    9. [Kubernetes Best Practices](https://kubernetes.io/docs/concepts/cluster-administration/monitoring/) - K8s monitoring best practices

    10. [Service Monitoring](https://prometheus.io/docs/guides/go-application/) - Application monitoring guide

    These resources provide comprehensive information about monitoring Kubernetes clusters effectively.

    Why This Matters

    Understanding the business and technical context helps you make informed decisions rather than blindly following patterns.

    Trade-offs to Consider

    Every architectural decision involves trade-offs. Consider your specific requirements, team expertise, and scale when evaluating options.

    When NOT to Use This

    Knowing when a solution doesn't apply is as valuable as knowing when it does. Consider alternatives for your specific situation.

    Decision Framework

    Use this framework to evaluate whether this approach is right for your use case based on your specific constraints and requirements.

    Related Articles