Optimizing Amazon EKS: A Practical Guide for Experienced Engineers

TL;DR: This guide details best practices for architecting and operating Amazon EKS clusters, focusing on security, scalability, and operational efficiency. It covers node group configuration, high availability, IAM roles, network policies, resource management, monitoring, cost optimization with Spot Instances and autoscaling, and operational patterns like GitOps and disaster recovery. Practical examples and a decision checklist are provided.

---

$1

Amazon Elastic Kubernetes Service (EKS) abstracts away the complexities of managing the Kubernetes control plane, allowing engineers to focus on deploying and managing containerized applications at scale. Achieving optimal performance, security, and cost-effectiveness in EKS requires a deliberate design approach. This guide outlines key best practices and provides concrete examples for experienced engineers.

``mermaid

%%{init: {'theme': 'base', 'themeVariables': { 'background': 'transparent', 'primaryColor': '#FF9900', 'primaryTextColor': '#232F3E', 'lineColor': '#147EB4' }}}%%

flowchart TB

subgraph Network["VPC & Networking"]

direction TB

subgraph AZ1["Availability Zone 1"]

direction TB

Subnet1["fa:fa-network-wired Public Subnet"]

Subnet2["fa:fa-network-wired Private Subnet"]

MNG1["fa:fa-server Managed Node Group"]

end

subgraph AZ2["Availability Zone 2"]

direction TB

Subnet3["fa:fa-network-wired Public Subnet"]

Subnet4["fa:fa-network-wired Private Subnet"]

MNG2["fa:fa-server Managed Node Group"]

end

end

subgraph Control["Control Plane & Security"]

direction LR

EKS["fa:fa-cogs EKS Control Plane"]

OIDC["fa:fa-key OIDC Provider"]

IAM["fa:fa-shield-alt IAM Roles"]

SG["fa:fa-lock Security Groups"]

end

subgraph Workloads["Workload Management"]

direction TB

subgraph Resources["Resource Management"]

direction LR

HPA["fa:fa-balance-scale HPA"]

VPA["fa:fa-arrows-alt VPA"]

CA["fa:fa-expand Cluster Autoscaler"]

end

subgraph Storage["Storage Solutions"]

direction LR

EBS["fa:fa-hdd EBS CSI"]

EFS["fa:fa-folder EFS CSI"]

S3["fa:fa-database S3"]

end

end

subgraph Observability["Monitoring & Logging"]

direction LR

CW["fa:fa-chart-line CloudWatch"]

CT["fa:fa-history CloudTrail"]

Prometheus["fa:fa-chart-bar Prometheus"]

Fluentbit["fa:fa-stream Fluent Bit"]

end

EKS --> MNG1

EKS --> MNG2

EKS --> OIDC

OIDC --> IAM

MNG1 --> Resources

MNG2 --> Resources

Resources --> Storage

EKS --> Observability

%% Styling

classDef networkNode fill:#FF9900,stroke:#FF9900,color:#232F3E,stroke-width:2px

classDef controlNode fill:#232F3E,stroke:#232F3E,color:#FFFFFF,stroke-width:2px

classDef workloadNode fill:#147EB4,stroke:#147EB4,color:#FFFFFF,stroke-width:2px

classDef observeNode fill:#147EB4,stroke:#147EB4,color:#FFFFFF,stroke-width:2px

classDef groupStyle fill:transparent,stroke:#147EB4,stroke-width:2px

class Subnet1,Subnet2,Subnet3,Subnet4,MNG1,MNG2 networkNode

class EKS,OIDC,IAM,SG controlNode

class HPA,VPA,CA,EBS,EFS,S3 workloadNode

class CW,CT,Prometheus,Fluentbit observeNode

class Network,Control,Workloads,Resources,Storage,Observability groupStyle

`

---

$1

$1

Managed Node Groups simplify the management of worker nodes. Properly configuring them is crucial for stability and scalability.

* Instance Types: Choose instance types based on workload requirements (CPU, memory, network I/O, GPU). Consider using instance families optimized for compute, memory, or storage.

* Subnet Placement: Distribute node groups across multiple Availability Zones (AZs) for high availability. Place nodes in private subnets for enhanced security, with only necessary access to public endpoints.

* Labels and Taints: Use labels to identify node groups for scheduling (e.g., role: application, environment: production). Employ taints to repel specific workloads from certain nodes, reserving them for critical system components or specialized applications.

Example Configuration:

`yaml

apiVersion: eks.amazonaws.com/v1alpha1

kind: NodeGroup

metadata:

name: production-nodes

spec:

clusterName: production-cluster

# IAM role for EKS nodes. This role must have the necessary permissions

# to join the EKS cluster and interact with AWS services.

nodeRole: arn:aws:iam::111122223333:role/eks-node-role

subnets:

- subnet-0123456789abcdef0 # Private subnet in AZ1

- subnet-0123456789abcdef1 # Private subnet in AZ2

instanceTypes:

- m5.large

- m5a.large

scaling:

minSize: 2

maxSize: 10

desiredSize: 3

# Labels for workload scheduling and identification

labels:

role: application

environment: production

# Taints to prevent unwanted pods from scheduling on these nodes

taints:

- key: dedicated

value: production

effect: NoSchedule # Pods without a matching toleration will not be scheduled here

`

$1

Achieving high availability for applications involves distributing replicas across nodes and AZs and ensuring graceful updates.

* Pod Anti-Affinity: Configure Pod Anti-Affinity to prevent multiple replicas of the same application from being scheduled on the same node or in the same AZ.

* Topology Spread Constraints: Use Topology Spread Constraints to ensure pods are evenly distributed across zones or nodes, maximizing availability in case of zone failures.

* Rolling Updates: Configure deployments with Rolling Updates and define maxSurge and maxUnavailable to control the deployment process and minimize downtime.

Example Configuration:

`yaml

apiVersion: apps/v1

kind: Deployment

metadata:

name: sample-app

spec:

replicas: 3

strategy:

type: RollingUpdate

rollingUpdate:

maxSurge: 1 # Allow one extra pod to be created before any are removed

maxUnavailable: 0 # Ensure no pods are unavailable during the update

selector:

matchLabels:

app: sample-app

template:

metadata:

labels:

app: sample-app

spec:

topologySpreadConstraints:

# Distribute pods across availability zones

- maxSkew: 1

topologyKey: topology.kubernetes.io/zone

whenUnsatisfied: DoNotSchedule # Pod will not be scheduled if constraints are violated

labelSelector:

matchLabels:

app: sample-app

affinity:

# Ensure pods are not scheduled on the same node

podAntiAffinity:

preferredDuringSchedulingIgnoredDuringExecution:

- weight: 100

podAffinityTerm:

labelSelector:

matchExpressions:

- key: app

operator: In

values:

- sample-app

topologyKey: kubernetes.io/hostname # Distribute across different nodes

`

---

$1

$1

IRSA allows Kubernetes service accounts to assume IAM roles, granting pods fine-grained access to AWS services without the need for AWS access keys. This is a significant security improvement over other methods.

* Principle of Least Privilege: Assign only the necessary IAM permissions to the role assumed by the service account.

* Dedicated Roles: Create distinct IAM roles for different applications or microservices, isolating their AWS access.

Example Configuration:

`yaml

apiVersion: v1

kind: ServiceAccount

metadata:

name: app-service-account

# This annotation tells EKS to associate this ServiceAccount with the specified IAM role.

annotations:

eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/app-role

`

$1

Network Policies are Kubernetes-native resources that control traffic flow at the IP address or port level (OSI layer 3 or 4). They are essential for implementing micro-segmentation within the cluster.

* Default Deny: Start with a default deny policy for ingress and egress and then explicitly allow necessary traffic.

* Namespace Scoping: Use namespaceSelector to control cross-namespace communication, enforcing isolation between different environments or teams.

Example Configuration:

`yaml

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

name: restrict-traffic

namespace: default # Apply this policy to pods in the 'default' namespace

spec:

podSelector:

matchLabels:

app: secure-app # Apply to pods with the label 'app: secure-app'

policyTypes:

- Ingress # Control incoming traffic

- Egress # Control outgoing traffic

ingress:

- from:

# Allow traffic from pods in namespaces with the label 'environment: production'

- namespaceSelector:

matchLabels:

environment: production

# Allow traffic from pods with the label 'role: frontend' in the same namespace

- podSelector:

matchLabels:

role: frontend

ports:

- protocol: TCP

port: 8080 # Only allow traffic on TCP port 8080

egress:

- to:

# Allow traffic to pods in namespaces with the label 'environment: production'

- namespaceSelector:

matchLabels:

environment: production

ports:

- protocol: TCP

port: 5432 # Only allow outbound traffic to TCP port 5432 (e.g., to a database)

`

---

$1

Effective resource management prevents noisy neighbor issues, ensures application stability, and optimizes costs.

$1

* Resource Quotas: Apply quotas at the namespace level to limit the total amount of compute resources (CPU, memory) and object counts (pods, services) that can be consumed.

* Limit Ranges: Define default resource requests and limits for pods within a namespace, ensuring that containers have at least a minimum resource allocation and a maximum cap.

Example Resource Quota:

`yaml

apiVersion: v1

kind: ResourceQuota

metadata:

name: compute-resources

namespace: production # Apply to the 'production' namespace

spec:

hard:

requests.cpu: "4" # Total CPU requests across all pods cannot exceed 4 cores

requests.memory: 8Gi # Total memory requests cannot exceed 8 GiB

limits.cpu: "8" # Total CPU limits cannot exceed 8 cores

limits.memory: 16Gi # Total memory limits cannot exceed 16 GiB

pods: "20" # Maximum number of pods allowed

`

$1

* Horizontal Pod Autoscaler (HPA): Automatically scales the number of pod replicas based on observed metrics like CPU or memory utilization.

* Vertical Pod Autoscaler (VPA): Automatically adjusts the CPU and memory requests/limits for pods. VPA can be deployed in "recommendation" mode to suggest optimal values, or in "auto" mode for automatic adjustments (use with caution, especially with stateful applications).

* Cluster Autoscaler (CA): Automatically adjusts the number of nodes in the cluster based on pending pods that cannot be scheduled due to resource constraints.

Example Horizontal Pod Autoscaler:

`yaml

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

name: app-hpa

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

name: sample-app # Target the 'sample-app' deployment

minReplicas: 2

maxReplicas: 10

metrics:

# Scale based on average CPU utilization

- type: Resource

resource:

name: cpu

target:

type: Utilization # Target is a percentage of the pod's request

averageUtilization: 70 # Scale up when average CPU utilization reaches 70%

# Scale based on average memory utilization

- type: Resource

resource:

name: memory

target:

type: Utilization

averageUtilization: 80 # Scale up when average memory utilization reaches 80%

`

---

$1

Comprehensive observability is crucial for understanding cluster health, troubleshooting issues, and optimizing performance.

$1

Container Insights provides enhanced monitoring for EKS clusters, collecting, aggregating, and summarizing metrics and logs from containerized applications and the underlying infrastructure.

* Deployment: Deploy the CloudWatch agent as a DaemonSet to collect metrics and logs from each node.

* Configuration: Customize the agent's configuration to specify which metrics and logs to collect, cluster name, and collection intervals.

Example CloudWatch Agent ConfigMap:

`yaml

apiVersion: v1

kind: ConfigMap

metadata:

name: cwagent-config

namespace: amazon-cloudwatch # Ensure the agent is deployed in this namespace

data:

cwagentconfig.json: |

{

"logs": {

"metrics_collected": {

"kubernetes": {

"cluster_name": "production-cluster", # Your EKS cluster name

"metrics_collection_interval": 60 # Collect metrics every 60 seconds

}

},

"force_flush_interval": 5 # Flush logs every 5 seconds

}

}

`

$1

Prometheus is a popular open-source monitoring and alerting system. Grafana is a leading visualization tool. Together, they provide a powerful platform for metrics collection and dashboarding.

* ServiceMonitor: Use ServiceMonitor resources (from the Prometheus Operator) to automatically discover and scrape metrics endpoints exposed by your applications.

* Dashboards: Create Grafana dashboards to visualize key metrics for applications, nodes, and the EKS control plane.

Example ServiceMonitor:

`yaml

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: app-monitor

labels:

release: prometheus # Label to be selected by the Prometheus Operator

spec:

selector:

matchLabels:

app: sample-app # Selects Services associated with the 'sample-app'

namespaceSelector:

matchNames:

- default # Scrapes services in the 'default' namespace

endpoints:

- port: metrics # Name of the port defined in the Service that exposes metrics

interval: 15s # Scrape interval

path: /metrics # The HTTP path where metrics are exposed

`

---

$1

Effectively managing costs in EKS is a continuous process involving strategic use of AWS resources.

$1

Amazon EC2 Spot Instances offer significant cost savings (up to 90% compared to On-Demand prices) for fault-tolerant workloads.

* Node Group Configuration: Specify capacityType: SPOT in your Managed Node Group configuration.

* Workload Suitability: Use Spot Instances for stateless, fault-tolerant applications (e.g., batch processing, CI/CD workers, web servers with multiple replicas). Avoid using them for critical stateful applications or workloads that cannot tolerate interruptions without proper graceful handling.

* Mixed Instance Pools: Consider using mixed instance groups (On-Demand and Spot) to balance cost savings with reliability.

Example Spot Instance Node Group:

`yaml

apiVersion: eks.amazonaws.com/v1alpha1

kind: NodeGroup

metadata:

name: spot-nodes

spec:

clusterName: production-cluster

nodeRole: arn:aws:iam::111122223333:role/eks-node-role

instanceTypes:

- m5.large

- m5a.large

- m5d.large # Example of mixed instance types for better availability

capacityType: SPOT # Crucial for utilizing Spot Instances

scaling:

minSize: 2

maxSize: 10

`

$1

Optimizing the Cluster Autoscaler's behavior can directly impact costs by ensuring that the cluster scales down efficiently when resources are underutilized.

* scaleDownUnneededTime: Defines how long a node must be underutilized before the Cluster Autoscaler considers scaling it down. A shorter duration leads to faster scale-down and cost savings, but might lead to more frequent scale-up/down cycles.

* scaleDownDelayAfterAdd: A delay after adding a node before the autoscaler will consider removing it. This prevents a new node from being immediately scaled down if it's just filling a temporary need.

* scaleDownUtilizationThreshold: The minimum node utilization required to prevent scale-down.

Example Cluster Autoscaler ConfigMap:

`yaml

apiVersion: v1

kind: ConfigMap

metadata:

name: cluster-autoscaler-config

namespace: kube-system # Typically deployed in kube-system

data:

config.yaml: |

# How long a node should be unneeded before it can be scaled down.

scaleDownUnneededTime: 5m

# How long after adding a node before it can be considered for scale down.

scaleDownDelayAfterAdd: 5m

# Node utilization threshold below which a node can be considered for scale down.

scaleDownUtilizationThreshold: 0.5 # Scale down if node utilization is below 50%

# If true, the cluster autoscaler will try not to remove nodes with pods that have

# PDBs that would be violated.

skipNodesWithSystemPods: true

# Example: Specify AWS specific tags for node groups

# node-group-auto-discovery:

# tags:

# - tag:

# key: k8s.io/cluster-autoscaler/enabled

# value: "true"

# - tag:

# key: k8s.io/cluster-autoscaler/production-cluster

# value: "owned"

`

---

$1

Adopting robust operational practices ensures reliability, maintainability, and agility.

$1

GitOps is a paradigm for managing infrastructure and applications using Git as the single source of truth. Tools like Flux or Argo CD automate the deployment and synchronization of cluster state with Git repositories.

* Declarative Configuration: Store all cluster configurations (Kubernetes manifests, Helm charts) in Git.

* Automated Synchronization: Use a GitOps operator to continuously monitor the Git repository and reconcile the cluster state with the desired state defined in Git.

Example FluxCD Configuration (GitRepository and Kustomization):

`yaml

apiVersion: source.toolkit.fluxcd.io/v1beta2

kind: GitRepository

metadata:

name: app-source

namespace: flux-system # Namespace where Flux components are installed

spec:

interval: 1m # How often to poll Git for changes

url: https://github.com/org/app-config # URL of your Git repository

ref:

branch: main # Branch to monitor

---

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2

kind: Kustomization

metadata:

name: app-config

namespace: flux-system

spec:

interval: 10m # How often to apply changes to the cluster

path: ./overlays/production # Path within the Git repository containing your Kustomization files

prune: true # Remove resources that are no longer present in Git

sourceRef:

kind: GitRepository

name: app-source # Reference to the GitRepository defined above

`

$1

Implementing a robust backup and disaster recovery strategy is critical for business continuity.

* Velero: Velero is a popular open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes.

* Scheduled Backups: Configure Velero to perform regular backups of namespaces, cluster resources, and PV snapshots.

* Cross-Region Backups: For disaster recovery, consider backing up to a different AWS region.

Example Velero Schedule:

`yaml

apiVersion: velero.io/v1

kind: Schedule

metadata:

name: daily-backup

namespace: velero # Ensure Velero is installed in this namespace

spec:

schedule: "0 0 * " # Cron syntax for daily backup at midnight UTC

template:

includedNamespaces:

- production # Back up all resources in the 'production' namespace

# Specify the storage location defined in Velero's configuration

storageLocation: default

# Specify the volume snapshot location

volumeSnapshotLocations:

- default

# How long to retain backups (in hours)

ttl: 720h # 30 days

``

---

$1

While EKS is a powerful platform, it may not be the ideal choice for every scenario:

* Very Small, Non-Critical Workloads: For simple applications with minimal scaling needs and low criticality, the operational overhead and cost of EKS might be disproportionate compared to simpler solutions like AWS Elastic Beanstalk or even serverless compute (AWS Lambda).

* Teams Lacking Kubernetes Expertise: EKS still requires a foundational understanding of Kubernetes concepts. Teams completely new to container orchestration or Kubernetes may face a steep learning curve.

* Strict Compliance Requirements with Air-Gapped Environments: While EKS can be secured to high standards, achieving true air-gapped environments with managed services can be complex. On-premises Kubernetes deployments might be more straightforward in such niche cases.

Minimal Infrastructure Management Desired: If the goal is to abstract away *all infrastructure management, including node provisioning and OS patching, fully managed container services like AWS Fargate (without EC2 nodes) might be a better fit, though with different trade-offs in flexibility and cost.

---

$1

Use this checklist to evaluate your EKS deployment against the best practices outlined in this guide.

$1

* [ ] Are you using Managed Node Groups for EKS worker nodes?

* [ ] Is your cluster deployed across multiple Availability Zones for high availability?

* [ ] Is the EKS control plane version kept up-to-date with minor version updates?

* [ ] Is the Cluster Autoscaler configured and actively managing node capacity?

* [ ] Are node groups configured with appropriate labels and taints?

$1

* [ ] Are IAM Roles for Service Accounts (IRSA) implemented for pod AWS access?

* [ ] Are Network Policies in place to enforce micro-segmentation within the cluster?

* [ ] Are Security Groups configured correctly for EKS nodes and the control plane endpoint?

* [ ] Is EKS audit logging enabled and integrated with a log analysis tool?

* [ ] Are RBAC roles and permissions reviewed regularly for least privilege?

$1

* [ ] Are Resource Quotas applied to namespaces to limit resource consumption?

* [ ] Are Limit Ranges configured to set default resource requests/limits for pods?

* [ ] Is Horizontal Pod Autoscaling (HPA) configured for stateless applications?

* [ ] Is Vertical Pod Autoscaling (VPA) being used for resource optimization recommendations or automatic adjustments?

* [ ] Are Pod Disruption Budgets (PDBs) configured for critical applications to ensure availability during voluntary disruptions (e.g., node upgrades)?

$1

* [ ] Is CloudWatch Container Insights enabled for EKS?

* [ ] Is a Prometheus/Grafana stack deployed for comprehensive metrics monitoring?

* [ ] Are alerts configured for critical cluster and application events?

* [ ] Is application and cluster logging aggregated and sent to a centralized logging solution (e.g., CloudWatch Logs, Elasticsearch)?

$1

* [ ] Are Spot Instances being utilized for suitable, fault-tolerant workloads?

* [ ] Are Cluster Autoscaler settings tuned for efficient scale-down and cost savings?

* [ ] Is regular cost analysis performed on EKS resources?

* [ ] Are unused resources (e.g., idle deployments, old AMIs) identified and cleaned up?

$1

* [ ] Is a GitOps workflow (e.g., Flux, Argo CD) implemented for managing cluster configuration?

* [ ] Is a backup and disaster recovery strategy in place using tools like Velero?

* [ ] Are deployment processes automated and repeatable?

* [ ] Is infrastructure-as-code used for EKS cluster provisioning and management?

---

$1

1. [AWS EKS Best Practices Guide](https://aws.github.io/aws-eks-best-practices/)

2. [EKS Workshop](https://www.eksworkshop.com/)

3. [Kubernetes Documentation](https://kubernetes.io/docs/)

4. [AWS EKS Documentation](https://docs.aws.amazon.com/eks/)

5. [CloudWatch Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights.html)

6. [EKS Security Best Practices](https://docs.aws.amazon.com/eks/latest/user