Building Resilient Architectures with AWS Elastic Load Balancing and Auto Scaling

TL;DR: This guide details how to construct highly available and scalable applications on AWS using Application Load Balancers (ALBs) and Auto Scaling Groups (ASGs). It covers essential concepts, architectural components, implementation steps, monitoring strategies, and disaster recovery considerations, providing practical Terraform configurations.

1. Introduction to High Availability

High availability (HA) in cloud architectures is not merely about uptime but ensuring applications remain accessible and performant under varying loads and during component failures. This involves designing systems that can withstand disruptions and scale dynamically.

Core Pillars of High Availability

• Redundancy: Eliminating single points of failure (SPOFs) by distributing resources across multiple fault domains, typically AWS Availability Zones (AZs).
• Scalability: The ability to adjust the capacity of an application to meet demand. This includes both horizontal scaling (adding more instances) and vertical scaling (increasing resources of existing instances), with a focus on automated horizontal scaling for HA.
• Fault Tolerance: Designing systems to continue operating, possibly at a reduced capacity, when failures occur. This is achieved through automatic failover, self-healing mechanisms, and robust health monitoring.

2. Architectural Components

A robust HA architecture on AWS integrates several key services.

2.1. Virtual Private Cloud (VPC) Configuration

A multi-AZ VPC is foundational for distributing resources. This ensures that even if one AZ experiences an issue, your application can continue to function from other AZs.

# Example: Multi-AZ VPC and public subnets using Terraform

resource "aws_vpc" "ha_vpc" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_support   = true
  enable_dns_hostnames = true
  tags = {
    Name = "ha-vpc"
  }
}

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_subnet" "public_subnets" {
  count             = 3 # Distribute across at least 3 AZs
  vpc_id            = aws_vpc.ha_vpc.id
  cidr_block        = "10.0.${count.index}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = false # Instances will get IPs via NAT or ALB

  tags = {
    Name = "public-subnet-${count.index + 1}"
  }
}

resource "aws_subnet" "private_subnets" {
  count             = 3
  vpc_id            = aws_vpc.ha_vpc.id
  cidr_block        = "10.0.${count.index + 3}.0/24" # Different CIDR block for private subnets
  availability_zone = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = false

  tags = {
    Name = "private-subnet-${count.index + 1}"
  }
}

# Internet Gateway and Route Table for Public Subnets
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.ha_vpc.id
  tags = {
    Name = "main-igw"
  }
}

resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.ha_vpc.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }
  tags = {
    Name = "public-rt"
  }
}

resource "aws_route_table_association" "public_assoc" {
  count          = 3
  subnet_id      = aws_subnet.public_subnets[count.index].id
  route_table_id = aws_route_table.public_rt.id
}

# NAT Gateway and Private Route Table (for outbound access from private subnets)
resource "aws_eip" "nat_eip" {
  count = 3 # One EIP per AZ for NAT Gateway
  domain = "vpc"
}

resource "aws_nat_gateway" "nat_gw" {
  count         = 3
  allocation_id = aws_eip.nat_eip[count.index].id
  subnet_id     = aws_subnet.public_subnets[count.index].id
  tags = {
    Name = "nat-gw-${count.index + 1}"
  }
  depends_on = [aws_internet_gateway.main]
}

resource "aws_route_table" "private_rt" {
  count  = 3
  vpc_id = aws_vpc.ha_vpc.id
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gw[count.index].id
  }
  tags = {
    Name = "private-rt-${count.index + 1}"
  }
}

resource "aws_route_table_association" "private_assoc" {
  count          = 3
  subnet_id      = aws_subnet.private_subnets[count.index].id
  route_table_id = aws_route_table.private_rt[count.index].id
}

2.2. Elastic Load Balancing (ELB) - Application Load Balancer (ALB)

ALBs distribute incoming application traffic across multiple targets, such as EC2 instances, in multiple AZs. This is critical for distributing load and routing around unhealthy instances.

# Example: ALB and Listener configuration using Terraform

resource "aws_security_group" "alb_sg" {
  name        = "ha-alb-sg"
  description = "Allow HTTP/S traffic to ALB"
  vpc_id      = aws_vpc.ha_vpc.id

  ingress {
    description = "HTTP from anywhere"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTPS from anywhere"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "ha-alb-sg"
  }
}

resource "aws_lb" "ha_alb" {
  name                       = "ha-application-alb"
  internal                   = false
  load_balancer_type         = "application"
  security_groups            = [aws_security_group.alb_sg.id]
  subnets                    = aws_subnet.public_subnets[*].id # ALB in public subnets
  enable_deletion_protection = true

  tags = {
    Environment = "production"
  }
}

resource "aws_lb_target_group" "app_tg" {
  name     = "ha-app-target-group"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.ha_vpc.id
  target_type = "instance" # Or "ip" for ECS/EKS

  health_check {
    enabled             = true
    protocol            = "HTTP"
    path                = "/health" # Define a health check endpoint on your application
    port                = "traffic-port"
    matcher             = "200" # Expect HTTP 200 OK
    interval            = 30
    timeout             = 5
    healthy_threshold   = 2
    unhealthy_threshold = 2
  }

  tags = {
    Name = "ha-app-target-group"
  }
}

resource "aws_lb_listener" "http_listener" {
  load_balancer_arn = aws_lb.ha_alb.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app_tg.arn
  }
}

# Optional: HTTPS Listener
/*
resource "aws_acm_certificate" "default" {
  domain_name       = "*.example.com"
  validation_method = "DNS"
}

resource "aws_lb_listener" "https_listener" {
  load_balancer_arn = aws_lb.ha_alb.arn
  port              = "443"
  protocol          = "HTTPS"
  certificate_arn   = aws_acm_certificate.default.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app_tg.arn
  }
}
*/

2.3. Auto Scaling Group (ASG)

ASGs automatically adjust the number of EC2 instances based on demand or schedule. Combined with an ALB, they ensure that enough healthy instances are available to handle traffic and that unhealthy instances are replaced.

# Example: Launch Template and Auto Scaling Group configuration using Terraform

resource "aws_security_group" "instance_sg" {
  name        = "ha-instance-sg"
  description = "Allow HTTP from ALB and SSH from specific IP"
  vpc_id      = aws_vpc.ha_vpc.id

  ingress {
    description     = "HTTP from ALB"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb_sg.id] # Allow traffic ONLY from ALB SG
  }

  ingress {
    description = "SSH from management IP"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["YOUR_MGMT_IP/32"] # Restrict SSH access
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"] # Allow outbound to internet (via NAT)
  }

  tags = {
    Name = "ha-instance-sg"
  }
}

resource "aws_launch_template" "app_lt" {
  name_prefix   = "ha-app-template-"
  image_id      = "ami-0abcdef1234567890" # Replace with a current Amazon Linux 2 AMI or your chosen OS
  instance_type = "t3.medium" # Choose instance type based on workload
  key_name      = "your-ssh-key-pair" # Optional: for SSH access

  network_interfaces {
    associate_public_ip_address = false # Instances in private subnets will not have public IPs
    security_groups               = [aws_security_group.instance_sg.id]
  }

  user_data = base64encode(<<-EOF
              #!/bin/bash
              yum update -y
              yum install -y httpd
              systemctl start httpd
              systemctl enable httpd
              echo "<h1>Hello from $(hostname -f)</h1>" > /var/www/html/index.html
              echo "OK" > /health # Simple health check file
              EOF
  )

  tags {
    Name = "ha-app-instance-template"
  }
}

resource "aws_autoscaling_group" "app_asg" {
  desired_capacity    = 2
  max_size           = 10
  min_size           = 2
  health_check_type  = "ELB" # ASG will use ALB's health checks
  health_check_grace_period = 300 # Wait up to 5 minutes for new instances to pass health checks

  # Instances will be launched in these private subnets
  vpc_zone_identifier = aws_subnet.private_subnets[*].id

  launch_template {
    id      = aws_launch_template.app_lt.id
    version = "$Latest" # Always use the latest version of the launch template
  }

  target_group_arns  = [aws_lb_target_group.app_tg.arn] # Register instances with the ALB target group

  tag {
    key                 = "Name"
    value               = "ha-app-instance"
    propagate_at_launch = true # Tag new instances automatically
  }
}

3. Implementation Steps

Building an HA architecture involves careful configuration of each component.

3.1. Network Foundation

• Multi-AZ VPC: Define VPC CIDR, subnets across multiple AZs (public for ALB, private for instances).
• Routing: Configure an Internet Gateway and route table for public subnets. For private subnets, set up NAT Gateways (one per AZ for resilience) with their own EIPs and route tables for outbound internet access.

3.2. Security Configuration

• ALB Security Group: Allow inbound traffic on ports 80/443 from
  0.0.0.0/0
  .
• Instance Security Group: Allow inbound traffic on application ports (e.g., 80) only from the ALB's security group. Restrict SSH access to trusted IPs. Allow necessary outbound traffic (e.g., for updates, API calls via NAT Gateway).

3.3. Load Balancer Configuration

• ALB: Create an ALB in your public subnets, associated with its security group.
• Target Group: Define a target group that specifies the protocol, port, and VPC for your application. Crucially, configure health checks here to monitor instance health.
• Listener: Create a listener (e.g., HTTP on port 80) that forwards traffic to the target group.

3.4. Auto Scaling Configuration

• Launch Template: Define the EC2 instance configuration: AMI, instance type, security group, user data for bootstrapping applications, SSH key pair.
• Auto Scaling Group: Configure the ASG with
  min_size
  ,
  max_size
  , and
  desired_capacity
  . Link it to the launch template and the ALB target group. Specify the private subnets where instances should be launched. Set
  health_check_type
  to
  ELB
  to leverage ALB health checks.

4. Monitoring and Health Checks

Robust monitoring is essential to understand system health and performance, and to trigger scaling or alerts.

4.1. Load Balancer Health Checks

• Target Group Health Checks: Configure health check paths (e.g.,
  /health
  ), intervals, thresholds, and expected response codes (e.g.,
  200
  ) on the ALB target group. Unhealthy instances will be automatically removed from the target group's rotation by the ALB.

4.2. CloudWatch Alarms

Set up alarms to monitor key metrics and trigger actions.

# Example: CloudWatch Alarm for high CPU utilization triggering an Auto Scaling policy
resource "aws_cloudwatch_metric_alarm" "high_cpu_alarm" {
  alarm_name          = "ha-asg-high-cpu-alarm"
  alarm_description   = "Scale up ASG when CPU utilization exceeds 70% for 5 minutes"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2" # Evaluate over 10 minutes (2 * 5 min periods)
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 300 # 5 minutes
  statistic           = "Average"
  threshold           = "70"
  treat_missing_data  = "notBreaching" # Important for scaling

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.app_asg.name
  }

  # Actions to take when the alarm is triggered
  alarm_actions = [aws_autoscaling_policy.scale_up_policy.arn]
}

# Example: CloudWatch Alarm for low CPU utilization to scale down
resource "aws_cloudwatch_metric_alarm" "low_cpu_alarm" {
  alarm_name          = "ha-asg-low-cpu-alarm"
  alarm_description   = "Scale down ASG when CPU utilization falls below 30% for 15 minutes"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = "3" # Evaluate over 15 minutes (3 * 5 min periods)
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 300 # 5 minutes
  statistic           = "Average"
  threshold           = "30"
  treat_missing_data  = "notBreaching"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.app_asg.name
  }

  alarm_actions = [aws_autoscaling_policy.scale_down_policy.arn]
}

5. Scaling Policies

Define how your ASG should scale based on observed metrics.

5.1. Target Tracking Scaling

This is often the simplest and most effective scaling method for maintaining a target metric.

# Example: Target Tracking Scaling policy for average CPU utilization

resource "aws_autoscaling_policy" "target_tracking_cpu" {
  name                   = "ha-asg-target-tracking-cpu"
  autoscaling_group_name = aws_autoscaling_group.app_asg.name
  policy_type           = "TargetTrackingScaling"

  target_tracking_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ASGAverageCPUUtilization"
    }
    target_value = 50.0 # Aim to keep average CPU at 50%
    scale_in_cooldown  = 300 # Cooldown period after scale-in
    scale_out_cooldown = 300 # Cooldown period after scale-out
  }
}

5.2. Step Scaling

Step scaling allows for more granular control by defining different scaling adjustments based on metric thresholds. This can be used in conjunction with or as an alternative to target tracking.

# Example: Step Scaling policy

# This assumes you have defined CloudWatch alarms for custom metrics or specific conditions
# For simplicity, we'll create policies that can be attached to alarms.

# Alarm that triggers scale-up
resource "aws_cloudwatch_metric_alarm" "high_request_count_per_target" {
  alarm_name          = "ha-asg-high-request-count-alarm"
  alarm_description   = "Scale up when RequestCountPerTarget exceeds 1000"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "RequestCountPerTarget"
  namespace           = "AWS/ApplicationELB"
  period              = 60
  statistic           = "Average"
  threshold           = "1000"
  treat_missing_data  = "notBreaching"

  dimensions = {
    LoadBalancer      = aws_lb.ha_alb.name
    TargetGroup       = aws_lb_target_group.app_tg.name
  }

  alarm_actions = [aws_autoscaling_policy.step_scale_out.arn]
}

# Scale-out policy
resource "aws_autoscaling_policy" "step_scale_out" {
  name                   = "ha-asg-step-scale-out"
  autoscaling_group_name = aws_autoscaling_group.app_asg.name
  policy_type           = "StepScaling"
  adjustment_type       = "ChangeInCapacity"
  cooldown              = 300

  step_adjustment {
    scaling_adjustment       = 2 # Add 2 instances
    metric_interval_lower_bound = 0
    metric_interval_upper_bound = 1000 # If RequestCountPerTarget is between 0 and 1000
  }

  step_adjustment {
    scaling_adjustment       = 4 # Add 4 instances
    metric_interval_lower_bound = 1000 # If RequestCountPerTarget is >= 1000
  }
}

# Alarm that triggers scale-down
resource "aws_cloudwatch_metric_alarm" "low_request_count_per_target" {
  alarm_name          = "ha-asg-low-request-count-alarm"
  alarm_description   = "Scale down when RequestCountPerTarget falls below 200"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = "3"
  metric_name         = "RequestCountPerTarget"
  namespace           = "AWS/ApplicationELB"
  period              = 300
  statistic           = "Average"
  threshold           = "200"
  treat_missing_data  = "notBreaching"

  dimensions = {
    LoadBalancer      = aws_lb.ha_alb.name
    TargetGroup       = aws_lb_target_group.app_tg.name
  }

  alarm_actions = [aws_autoscaling_policy.step_scale_in.arn]
}

# Scale-in policy
resource "aws_autoscaling_policy" "step_scale_in" {
  name                   = "ha-asg-step-scale-in"
  autoscaling_group_name = aws_autoscaling_group.app_asg.name
  policy_type           = "StepScaling"
  adjustment_type       = "ChangeInCapacity"
  cooldown              = 600 # Longer cooldown for scale-in

  step_adjustment {
    scaling_adjustment       = -1 # Remove 1 instance
    metric_interval_upper_bound = 200 # If RequestCountPerTarget is <= 200
  }

  step_adjustment {
    scaling_adjustment       = -2 # Remove 2 instances
    metric_interval_lower_bound = 0
    metric_interval_upper_bound = 100 # If RequestCountPerTarget is <= 100
  }
}

6. Disaster Recovery (DR) Considerations

While this guide focuses on HA within a region, DR involves strategies for region-wide failures.

6.1. Data Backup and Restoration

• Automated Backups: Utilize AWS Backup or native snapshot capabilities (EBS, RDS) for regular, automated backups.
• Cross-Region Replication: For critical data stores (S3, RDS), configure cross-region replication to have copies in a different AWS region.

# Example: S3 Cross-Region Replication (Conceptual)
# This requires separate S3 buckets in different regions and IAM roles.

resource "aws_s3_bucket" "primary_data_bucket" {
  bucket = "my-app-data-bucket-primary-us-east-1"
  acl    = "private"
  # ... other configurations
}

resource "aws_s3_bucket" "replica_data_bucket" {
  bucket = "my-app-data-bucket-replica-us-west-2"
  acl    = "private"
  # ... other configurations
}

resource "aws_iam_role" "s3_replication_role" {
  name = "s3-replication-role"
  # ... assumes permissions for replication
}

resource "aws_s3_bucket_replication_configuration" "replication_config" {
  role   = aws_iam_role.s3_replication_role.arn
  bucket = aws_s3_bucket.primary_data_bucket.id

  rule {
    id     = "replication-rule-1"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.replica_data_bucket.arn
      storage_class = "STANDARD_IA" # Or appropriate class
    }

    source_selection_criteria {
      sse_kms_encrypted_objects {
        status = "Disabled"
      }
    }

    filter {
      prefix = "" # Replicate all objects
    }
  }
}

6.2. Multi-Region Deployment (Advanced)

For true disaster recovery, consider deploying identical or mirrored environments in separate AWS regions. This can involve:

• Active-Passive: A warm standby environment in another region that can be promoted during a disaster.
• Active-Active: Distributing traffic across multiple regions simultaneously (more complex and costly).

7. Trade-offs and Considerations

• Complexity: Implementing HA adds complexity to your infrastructure. While services like ALB and ASG abstract much of it, understanding the interplay is crucial.
• Cost: Running redundant resources across multiple AZs and potentially multiple regions, along with load balancers and auto-scaling, incurs higher costs than a single-instance deployment.
• Latency: While ALBs and ASGs distribute traffic efficiently within a region, cross-region replication and multi-region deployments can introduce latency for data synchronization or user requests.
• Application Design: HA heavily relies on your application being stateless or managing state externally (e.g., in RDS, ElastiCache, S3) so that any instance can serve any request.
• Health Check Granularity: The effectiveness of ASG and ALB relies on accurate and comprehensive health checks. A poorly configured health check might incorrectly mark healthy instances as unhealthy or vice-versa.