TL;DR
Learn how to design and implement highly available architectures using AWS Elastic Load Balancing and Auto Scaling Groups for resilient applications.
Building Resilient Architectures with AWS Elastic Load Balancing and Auto Scaling
TL;DR: This guide details how to construct highly available and scalable applications on AWS using Application Load Balancers (ALBs) and Auto Scaling Groups (ASGs). It covers essential concepts, architectural components, implementation steps, monitoring strategies, and disaster recovery considerations, providing practical Terraform configurations.
$1
High availability (HA) in cloud architectures is not merely about uptime but ensuring applications remain accessible and performant under varying loads and during component failures. This involves designing systems that can withstand disruptions and scale dynamically.
$1
* Redundancy: Eliminating single points of failure (SPOFs) by distributing resources across multiple fault domains, typically AWS Availability Zones (AZs).
* Scalability: The ability to adjust the capacity of an application to meet demand. This includes both horizontal scaling (adding more instances) and vertical scaling (increasing resources of existing instances), with a focus on automated horizontal scaling for HA.
* Fault Tolerance: Designing systems to continue operating, possibly at a reduced capacity, when failures occur. This is achieved through automatic failover, self-healing mechanisms, and robust health monitoring.
$1
A robust HA architecture on AWS integrates several key services.
$1
A multi-AZ VPC is foundational for distributing resources. This ensures that even if one AZ experiences an issue, your application can continue to function from other AZs.
``hcl
Example: Multi-AZ VPC and public subnets using Terraform
resource "aws_vpc" "ha_vpc" {
cidr_block = "10.0.0.0/16"
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "ha-vpc"
}
}
data "aws_availability_zones" "available" {
state = "available"
}
resource "aws_subnet" "public_subnets" {
count = 3 # Distribute across at least 3 AZs
vpc_id = aws_vpc.ha_vpc.id
cidr_block = "10.0.${count.index}.0/24"
availability_zone = data.aws_availability_zones.available.names[count.index]
map_public_ip_on_launch = false # Instances will get IPs via NAT or ALB
tags = {
Name = "public-subnet-${count.index + 1}"
}
}
resource "aws_subnet" "private_subnets" {
count = 3
vpc_id = aws_vpc.ha_vpc.id
cidr_block = "10.0.${count.index + 3}.0/24" # Different CIDR block for private subnets
availability_zone = data.aws_availability_zones.available.names[count.index]
map_public_ip_on_launch = false
tags = {
Name = "private-subnet-${count.index + 1}"
}
}
Internet Gateway and Route Table for Public Subnets
resource "aws_internet_gateway" "main" {
vpc_id = aws_vpc.ha_vpc.id
tags = {
Name = "main-igw"
}
}
resource "aws_route_table" "public_rt" {
vpc_id = aws_vpc.ha_vpc.id
route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.main.id
}
tags = {
Name = "public-rt"
}
}
resource "aws_route_table_association" "public_assoc" {
count = 3
subnet_id = aws_subnet.public_subnets[count.index].id
route_table_id = aws_route_table.public_rt.id
}
NAT Gateway and Private Route Table (for outbound access from private subnets)
resource "aws_eip" "nat_eip" {
count = 3 # One EIP per AZ for NAT Gateway
domain = "vpc"
}
resource "aws_nat_gateway" "nat_gw" {
count = 3
allocation_id = aws_eip.nat_eip[count.index].id
subnet_id = aws_subnet.public_subnets[count.index].id
tags = {
Name = "nat-gw-${count.index + 1}"
}
depends_on = [aws_internet_gateway.main]
}
resource "aws_route_table" "private_rt" {
count = 3
vpc_id = aws_vpc.ha_vpc.id
route {
cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.nat_gw[count.index].id
}
tags = {
Name = "private-rt-${count.index + 1}"
}
}
resource "aws_route_table_association" "private_assoc" {
count = 3
subnet_id = aws_subnet.private_subnets[count.index].id
route_table_id = aws_route_table.private_rt[count.index].id
}
`
$1
ALBs distribute incoming application traffic across multiple targets, such as EC2 instances, in multiple AZs. This is critical for distributing load and routing around unhealthy instances.
`hcl
Example: ALB and Listener configuration using Terraform
resource "aws_security_group" "alb_sg" {
name = "ha-alb-sg"
description = "Allow HTTP/S traffic to ALB"
vpc_id = aws_vpc.ha_vpc.id
ingress {
description = "HTTP from anywhere"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
ingress {
description = "HTTPS from anywhere"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "ha-alb-sg"
}
}
resource "aws_lb" "ha_alb" {
name = "ha-application-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.alb_sg.id]
subnets = aws_subnet.public_subnets[*].id # ALB in public subnets
enable_deletion_protection = true
tags = {
Environment = "production"
}
}
resource "aws_lb_target_group" "app_tg" {
name = "ha-app-target-group"
port = 80
protocol = "HTTP"
vpc_id = aws_vpc.ha_vpc.id
target_type = "instance" # Or "ip" for ECS/EKS
health_check {
enabled = true
protocol = "HTTP"
path = "/health" # Define a health check endpoint on your application
port = "traffic-port"
matcher = "200" # Expect HTTP 200 OK
interval = 30
timeout = 5
healthy_threshold = 2
unhealthy_threshold = 2
}
tags = {
Name = "ha-app-target-group"
}
}
resource "aws_lb_listener" "http_listener" {
load_balancer_arn = aws_lb.ha_alb.arn
port = "80"
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.app_tg.arn
}
}
Optional: HTTPS Listener
/*
resource "aws_acm_certificate" "default" {
domain_name = "*.example.com"
validation_method = "DNS"
}
resource "aws_lb_listener" "https_listener" {
load_balancer_arn = aws_lb.ha_alb.arn
port = "443"
protocol = "HTTPS"
certificate_arn = aws_acm_certificate.default.arn
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.app_tg.arn
}
}
*/
`
$1
ASGs automatically adjust the number of EC2 instances based on demand or schedule. Combined with an ALB, they ensure that enough healthy instances are available to handle traffic and that unhealthy instances are replaced.
`hcl
Example: Launch Template and Auto Scaling Group configuration using Terraform
resource "aws_security_group" "instance_sg" {
name = "ha-instance-sg"
description = "Allow HTTP from ALB and SSH from specific IP"
vpc_id = aws_vpc.ha_vpc.id
ingress {
description = "HTTP from ALB"
from_port = 80
to_port = 80
protocol = "tcp"
security_groups = [aws_security_group.alb_sg.id] # Allow traffic ONLY from ALB SG
}
ingress {
description = "SSH from management IP"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["YOUR_MGMT_IP/32"] # Restrict SSH access
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"] # Allow outbound to internet (via NAT)
}
tags = {
Name = "ha-instance-sg"
}
}
resource "aws_launch_template" "app_lt" {
name_prefix = "ha-app-template-"
image_id = "ami-0abcdef1234567890" # Replace with a current Amazon Linux 2 AMI or your chosen OS
instance_type = "t3.medium" # Choose instance type based on workload
key_name = "your-ssh-key-pair" # Optional: for SSH access
network_interfaces {
associate_public_ip_address = false # Instances in private subnets will not have public IPs
security_groups = [aws_security_group.instance_sg.id]
}
user_data = base64encode(<<-EOF
#!/bin/bash
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd
echo "
Hello from $(hostname -f)
" > /var/www/html/index.htmlecho "OK" > /health # Simple health check file
EOF
)
tags {
Name = "ha-app-instance-template"
}
}
resource "aws_autoscaling_group" "app_asg" {
desired_capacity = 2
max_size = 10
min_size = 2
health_check_type = "ELB" # ASG will use ALB's health checks
health_check_grace_period = 300 # Wait up to 5 minutes for new instances to pass health checks
# Instances will be launched in these private subnets
vpc_zone_identifier = aws_subnet.private_subnets[*].id
launch_template {
id = aws_launch_template.app_lt.id
version = "$Latest" # Always use the latest version of the launch template
}
target_group_arns = [aws_lb_target_group.app_tg.arn] # Register instances with the ALB target group
tag {
key = "Name"
value = "ha-app-instance"
propagate_at_launch = true # Tag new instances automatically
}
}
`
$1
Building an HA architecture involves careful configuration of each component.
$1
* Multi-AZ VPC: Define VPC CIDR, subnets across multiple AZs (public for ALB, private for instances).
* Routing: Configure an Internet Gateway and route table for public subnets. For private subnets, set up NAT Gateways (one per AZ for resilience) with their own EIPs and route tables for outbound internet access.
$1
* ALB Security Group: Allow inbound traffic on ports 80/443 from 0.0.0.0/0 Instance Security Group: Allow inbound traffic on application ports (e.g., 80) *only from the ALB's security group. Restrict SSH access to trusted IPs. Allow necessary outbound traffic (e.g., for updates, API calls via NAT Gateway)..
$1
* ALB: Create an ALB in your public subnets, associated with its security group.
* Target Group: Define a target group that specifies the protocol, port, and VPC for your application. Crucially, configure health checks here to monitor instance health.
* Listener: Create a listener (e.g., HTTP on port 80) that forwards traffic to the target group.
$1
* Launch Template: Define the EC2 instance configuration: AMI, instance type, security group, user data for bootstrapping applications, SSH key pair.
* Auto Scaling Group: Configure the ASG with min_size, max_size, and desired_capacity. Link it to the launch template and the ALB target group. Specify the private subnets where instances should be launched. Set health_check_type to ELB to leverage ALB health checks.
$1
Robust monitoring is essential to understand system health and performance, and to trigger scaling or alerts.
$1
* Target Group Health Checks: Configure health check paths (e.g., /health), intervals, thresholds, and expected response codes (e.g., 200) on the ALB target group. Unhealthy instances will be automatically removed from the target group's rotation by the ALB.
$1
Set up alarms to monitor key metrics and trigger actions.
` resource "aws_cloudwatch_metric_alarm" "high_cpu_alarm" {
alarm_name = "ha-asg-high-cpu-alarm"
alarm_description = "Scale up ASG when CPU utilization exceeds 70% for 5 minutes"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = "2" # Evaluate over 10 minutes (2 * 5 min periods)
metric_name = "CPUUtilization"
namespace = "AWS/EC2"
period = 300 # 5 minutes
statistic = "Average"
threshold = "70"
treat_missing_data = "notBreaching" # Important for scaling dimensions = {
AutoScalingGroupName = aws_autoscaling_group.app_asg.name
} # Actions to take when the alarm is triggered
alarm_actions = [aws_autoscaling_policy.scale_up_policy.arn]
} resource "aws_cloudwatch_metric_alarm" "low_cpu_alarm" {
alarm_name = "ha-asg-low-cpu-alarm"
alarm_description = "Scale down ASG when CPU utilization falls below 30% for 15 minutes"
comparison_operator = "LessThanThreshold"
evaluation_periods = "3" # Evaluate over 15 minutes (3 * 5 min periods)
metric_name = "CPUUtilization"
namespace = "AWS/EC2"
period = 300 # 5 minutes
statistic = "Average"
threshold = "30"
treat_missing_data = "notBreaching" dimensions = {
AutoScalingGroupName = aws_autoscaling_group.app_asg.name
} alarm_actions = [aws_autoscaling_policy.scale_down_policy.arn]
}
hcl
`Example: CloudWatch Alarm for high CPU utilization triggering an Auto Scaling policy
Example: CloudWatch Alarm for low CPU utilization to scale down
$1
Define how your ASG should scale based on observed metrics.
$1
This is often the simplest and most effective scaling method for maintaining a target metric.
`hcl
Example: Target Tracking Scaling policy for average CPU utilization
resource "aws_autoscaling_policy" "target_tracking_cpu" {
name = "ha-asg-target-tracking-cpu"
autoscaling_group_name = aws_autoscaling_group.app_asg.name
policy_type = "TargetTrackingScaling"
target_tracking_configuration {
predefined_metric_specification {
predefined_metric_type = "ASGAverageCPUUtilization"
}
target_value = 50.0 # Aim to keep average CPU at 50%
scale_in_cooldown = 300 # Cooldown period after scale-in
scale_out_cooldown = 300 # Cooldown period after scale-out
}
}
`
$1
Step scaling allows for more granular control by defining different scaling adjustments based on metric thresholds. This can be used in conjunction with or as an alternative to target tracking.
`hcl
Example: Step Scaling policy
This assumes you have defined CloudWatch alarms for custom metrics or specific conditions
For simplicity, we'll create policies that can be attached to alarms.
Alarm that triggers scale-up
resource "aws_cloudwatch_metric_alarm" "high_request_count_per_target" {
alarm_name = "ha-asg-high-request-count-alarm"
alarm_description = "Scale up when RequestCountPerTarget exceeds 1000"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = "2"
metric_name = "RequestCountPerTarget"
namespace = "AWS/ApplicationELB"
period = 60
statistic = "Average"
threshold = "1000"
treat_missing_data = "notBreaching"
dimensions = {
LoadBalancer = aws_lb.ha_alb.name
TargetGroup = aws_lb_target_group.app_tg.name
}
alarm_actions = [aws_autoscaling_policy.step_scale_out.arn]
}
Scale-out policy
resource "aws_autoscaling_policy" "step_scale_out" {
name = "ha-asg-step-scale-out"
autoscaling_group_name = aws_autoscaling_group.app_asg.name
policy_type = "StepScaling"
adjustment_type = "ChangeInCapacity"
cooldown = 300
step_adjustment {
scaling_adjustment = 2 # Add 2 instances
metric_interval_lower_bound = 0
metric_interval_upper_bound = 1000 # If RequestCountPerTarget is between 0 and 1000
}
step_adjustment {
scaling_adjustment = 4 # Add 4 instances
metric_interval_lower_bound = 1000 # If RequestCountPerTarget is >= 1000
}
}
Alarm that triggers scale-down
resource "aws_cloudwatch_metric_alarm" "low_request_count_per_target" {
alarm_name = "ha-asg-low-request-count-alarm"
alarm_description = "Scale down when RequestCountPerTarget falls below 200"
comparison_operator = "LessThanThreshold"
evaluation_periods = "3"
metric_name = "RequestCountPerTarget"
namespace = "AWS/ApplicationELB"
period = 300
statistic = "Average"
threshold = "200"
treat_missing_data = "notBreaching"
dimensions = {
LoadBalancer = aws_lb.ha_alb.name
TargetGroup = aws_lb_target_group.app_tg.name
}
alarm_actions = [aws_autoscaling_policy.step_scale_in.arn]
}
Scale-in policy
resource "aws_autoscaling_policy" "step_scale_in" {
name = "ha-asg-step-scale-in"
autoscaling_group_name = aws_autoscaling_group.app_asg.name
policy_type = "StepScaling"
adjustment_type = "ChangeInCapacity"
cooldown = 600 # Longer cooldown for scale-in
step_adjustment {
scaling_adjustment = -1 # Remove 1 instance
metric_interval_upper_bound = 200 # If RequestCountPerTarget is <= 200
}
step_adjustment {
scaling_adjustment = -2 # Remove 2 instances
metric_interval_lower_bound = 0
metric_interval_upper_bound = 100 # If RequestCountPerTarget is <= 100
}
}
`
$1
While this guide focuses on HA within a region, DR involves strategies for region-wide failures.
$1
* Automated Backups: Utilize AWS Backup or native snapshot capabilities (EBS, RDS) for regular, automated backups.
* Cross-Region Replication: For critical data stores (S3, RDS), configure cross-region replication to have copies in a different AWS region.
`hcl
Example: S3 Cross-Region Replication (Conceptual)
This requires separate S3 buckets in different regions and IAM roles.
resource "aws_s3_bucket" "primary_data_bucket" {
bucket = "my-app-data-bucket-primary-us-east-1"
acl = "private"
# ... other configurations
}
resource "aws_s3_bucket" "replica_data_bucket" {
bucket = "my-app-data-bucket-replica-us-west-2"
acl = "private"
# ... other configurations
}
resource "aws_iam_role" "s3_replication_role" {
name = "s3-replication-role"
# ... assumes permissions for replication
}
resource "aws_s3_bucket_replication_configuration" "replication_config" {
role = aws_iam_role.s3_replication_role.arn
bucket = aws_s3_bucket.primary_data_bucket.id
rule {
id = "replication-rule-1"
status = "Enabled"
destination {
bucket = aws_s3_bucket.replica_data_bucket.arn
storage_class = "STANDARD_IA" # Or appropriate class
}
source_selection_criteria {
sse_kms_encrypted_objects {
status = "Disabled"
}
}
filter {
prefix = "" # Replicate all objects
}
}
}
``
$1
For true disaster recovery, consider deploying identical or mirrored environments in separate AWS regions. This can involve:
* Active-Passive: A warm standby environment in another region that can be promoted during a disaster.
* Active-Active: Distributing traffic across multiple regions simultaneously (more complex and costly).
$1
* Complexity: Implementing HA adds complexity to your infrastructure. While services like ALB and ASG abstract much of it, understanding the interplay is crucial.
* Cost: Running redundant resources across multiple AZs and potentially multiple regions, along with load balancers and auto-scaling, incurs higher costs than a single-instance deployment.
* Latency: While ALBs and ASGs distribute traffic efficiently within a region, cross-region replication and multi-region deployments can introduce latency for data synchronization or user requests.
* Application Design: HA heavily relies on your application being stateless or managing state externally (e.g., in RDS, ElastiCache, S3) so that any instance can serve any request.
* Health Check Granularity: The effectiveness of ASG and ALB relies on accurate and comprehensive health checks. A poorly configured health check might incorrectly mark healthy instances as unhealthy or vice-versa.
##
Why This Matters
Understanding the business and technical context helps you make informed decisions rather than blindly following patterns.
Trade-offs to Consider
Every architectural decision involves trade-offs. Consider your specific requirements, team expertise, and scale when evaluating options.
When NOT to Use This
Knowing when a solution doesn't apply is as valuable as knowing when it does. Consider alternatives for your specific situation.
Decision Framework
Use this framework to evaluate whether this approach is right for your use case based on your specific constraints and requirements.
