Monitoring Kubernetes Clusters with Prometheus and Grafana

Effective monitoring is crucial for maintaining healthy Kubernetes clusters. This guide will show you how to set up comprehensive monitoring using Prometheus and Grafana.

Understanding Kubernetes Monitoring

Key Metrics to Monitor

1. Node metrics
2. Pod metrics
3. Container metrics
4. Application metrics
5. Network metrics

Monitoring Architecture

Flowchart Diagram

100%

Copy

Rendering diagram...

Setting Up Prometheus

Installation with Helm

# Add Prometheus Helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install Prometheus
helm install prometheus prometheus-community/prometheus \
  --namespace monitoring \
  --create-namespace

Basic Configuration

# prometheus-values.yaml
server:
  persistentVolume:
    size: 50Gi
  retention: 15d

alertmanager:
  enabled: true
  
  persistence:
    enabled: true
    size: 10Gi

nodeExporter:
  enabled: true

kubeStateMetrics:
  enabled: true

Configuring Grafana

Installation

# Add Grafana Helm repository
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Grafana
helm install grafana grafana/grafana \
  --namespace monitoring \
  --set persistence.enabled=true \
  --set persistence.size=10Gi

Dashboard Setup

{
  "dashboard": {
    "id": null,
    "title": "Kubernetes Cluster Overview",
    "panels": [
      {
        "title": "CPU Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(container_cpu_usage_seconds_total{container!=\"\"}[5m])) by (pod)"
          }
        ]
      }
    ]
  }
}

Essential Metrics and Alerts

Node Monitoring

# Node alerts
groups:
- name: node.rules
  rules:
  - alert: NodeHighCPU
    expr: instance:node_cpu_utilisation:rate5m > 0.8
    for: 10m
    labels:
      severity: warning
    annotations:
      description: "CPU usage on {{ $labels.instance }} is above 80%"

Pod Monitoring

# Pod alerts
groups:
- name: pod.rules
  rules:
  - alert: PodHighMemory
    expr: container_memory_usage_bytes > 1.5 * 1024 * 1024 * 1024
    for: 5m
    labels:
      severity: warning
    annotations:
      description: "Pod {{ $labels.pod }} is using too much memory"

Advanced Monitoring Features

Custom Metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-monitor
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: metrics

Recording Rules

groups:
- name: custom.rules
  rules:
  - record: job:http_requests_total:rate5m
    expr: rate(http_requests_total[5m])

Best Practices

1. Resource Management

resources:
  requests:
    cpu: 200m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

2. Data Retention

prometheus:
  retention:
    time: 15d
    size: 50GB

3. High Availability

prometheus:
  replicaCount: 2
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - topologyKey: kubernetes.io/hostname

Troubleshooting

Common Issues

1. High memory usage
2. Metric collection gaps
3. Slow query performance
4. Dashboard loading issues

Solutions

# Check Prometheus status
kubectl get pods -n monitoring
kubectl logs -f prometheus-server-xyz -n monitoring

# Check Grafana status
kubectl get pods -n monitoring
kubectl logs -f grafana-xyz -n monitoring

Performance Optimization

Query Optimization

# Bad query
rate(http_requests_total[5m])

# Better query
sum by (service) (rate(http_requests_total[5m]))

Resource Tuning

prometheus:
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 2
      memory: 4Gi

Conclusion

Effective Kubernetes monitoring with Prometheus and Grafana requires:

1. Proper setup and configuration
2. Regular maintenance
3. Performance optimization
4. Alert tuning
5. Resource management

Follow these guidelines to maintain a healthy and observable Kubernetes cluster.

References

Here are valuable resources for Kubernetes monitoring:

1. Prometheus Documentation - Official Prometheus documentation
2. Grafana Documentation - Official Grafana documentation
3. Kubernetes Monitoring Guide - Official K8s monitoring guide
4. Prometheus Operator - Automated Prometheus setup
5. Grafana Dashboards - Pre-built Grafana dashboards
6. Kubernetes Metrics - K8s metrics reference
7. PromQL Basics - Prometheus Query Language guide
8. Alertmanager - Prometheus alerting
9. Kubernetes Best Practices - K8s monitoring best practices
10. Service Monitoring - Application monitoring guide

These resources provide comprehensive information about monitoring Kubernetes clusters effectively.