TL;DR
Learn how to implement zero-downtime deployments using the blue-green deployment strategy in various environments and platforms
import { MermaidDiagram } from '@/components/mermaid-diagram'
Learn how to achieve zero-downtime deployments using the blue-green deployment strategy. This comprehensive guide covers implementation patterns, best practices, and real-world examples across different platforms.
graph TB
subgraph "Load Balancer"
LB["Load Balancer"]
end
subgraph "Blue Environment"
B1["Blue Pod 1"]
B2["Blue Pod 2"]
B3["Blue Pod 3"]
end
subgraph "Green Environment"
G1["Green Pod 1"]
G2["Green Pod 2"]
G3["Green Pod 3"]
end
Users["Users"] --> LB
LB -->|"Active Traffic"| B1
LB -->|"Active Traffic"| B2
LB -->|"Active Traffic"| B3
LB -.->|"No Traffic"| G1
LB -.->|"No Traffic"| G2
LB -.->|"No Traffic"| G3
style LB fill:#3b82f6,stroke:#2563eb,color:white
style B1 fill:#3b82f6,stroke:#2563eb,color:white
style B2 fill:#3b82f6,stroke:#2563eb,color:white
style B3 fill:#3b82f6,stroke:#2563eb,color:white
style G1 fill:#f1f5f9,stroke:#64748b
style G2 fill:#f1f5f9,stroke:#64748b
style G3 fill:#f1f5f9,stroke:#64748b
style Users fill:#f1f5f9,stroke:#64748b
/>
}
$1
Blue-green deployment is a deployment strategy that maintains two identical production environments:
1. Blue Environment: Currently active environment serving production traffic
2. Green Environment: New environment with updated application version
Key benefits include:
sequenceDiagram
participant LB as Load Balancer
participant Blue as Blue Environment
participant Green as Green Environment
participant Tests as Test Suite
Note over Blue: Active Environment
Note over Green: Inactive Environment
activate Blue
LB->>Blue: Route Production Traffic
Note over Green: Deploy New Version
activate Green
Green->>Tests: Run Health Checks
Tests-->>Green: Health Checks Pass
Note over LB,Green: Switch Traffic
LB->>Green: Route Production Traffic
LB->>Blue: Stop Traffic
Note over Blue: Previous Version
Note over Green: New Active Version
deactivate Blue
Note over Blue: Available for Rollback
/>
}
$1
$1
Here's a detailed example of implementing blue-green deployments in Kubernetes:
`` apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-blue
labels:
app: myapp
version: blue
environment: production
spec:
replicas: 3
selector:
matchLabels:
app: myapp
version: blue
template:
metadata:
labels:
app: myapp
version: blue
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
# Security context
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
containers:
- name: myapp
image: myapp:1.0
imagePullPolicy: Always
# Resource management
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "200m"
memory: "256Mi"
# Port configuration
ports:
- name: http
containerPort: 8080
protocol: TCP
# Health checks
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 10
# Environment variables
env:
- name: VERSION
value: "blue"
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: db_host
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: app-secrets
key: db_password ---
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-green
labels:
app: myapp
version: green
environment: production
spec:
replicas: 3
selector:
matchLabels:
app: myapp
version: green
template:
metadata:
labels:
app: myapp
version: green
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
# Security context
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 2000
containers:
- name: myapp
image: myapp:2.0
imagePullPolicy: Always
# Resource management
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "200m"
memory: "256Mi"
# Port configuration
ports:
- name: http
containerPort: 8080
protocol: TCP
# Health checks
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: http
initialDelaySeconds: 5
periodSeconds: 10
# Environment variables
env:
- name: VERSION
value: "green"
- name: DB_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: db_host
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: app-secrets
key: db_password
yaml
`blue-deployment.yaml
Purpose: Define the blue environment deployment
green-deployment.yaml
Purpose: Define the green environment deployment
$1
Configure the service to switch between blue and green environments:
` apiVersion: v1
kind: Service
metadata:
name: myapp-service
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
selector:
app: myapp
version: blue # Switch to green during deployment
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
- name: metrics
protocol: TCP
port: 9090
targetPort: 9090
type: LoadBalancer
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
yaml
`service.yaml
Purpose: Define the service for routing traffic
$1
Here's a Python script to automate the deployment process:
`python
deploy.py
Purpose: Automate blue-green deployment process
import kubernetes
import time
from typing import Dict, List, Optional
class BlueGreenDeployer:
"""Manages blue-green deployments in Kubernetes."""
def __init__(self, namespace: str = "default"):
"""Initialize the deployer with Kubernetes configuration."""
kubernetes.config.load_kube_config()
self.api = kubernetes.client.AppsV1Api()
self.core_api = kubernetes.client.CoreV1Api()
self.namespace = namespace
def get_active_version(self) -> str:
"""Determine currently active version (blue/green)."""
try:
service = self.core_api.read_namespaced_service(
name="myapp-service",
namespace=self.namespace
)
return service.spec.selector["version"]
except kubernetes.client.rest.ApiException as e:
print(f"Error getting active version: {e}")
raise
def deploy_new_version(self, version: str, image: str) -> None:
"""Deploy new version of the application."""
try:
# Update deployment with new image
deployment = self.api.read_namespaced_deployment(
name=f"myapp-{version}",
namespace=self.namespace
)
deployment.spec.template.spec.containers[0].image = image
self.api.patch_namespaced_deployment(
name=f"myapp-{version}",
namespace=self.namespace,
body=deployment
)
# Wait for deployment to be ready
self._wait_for_deployment(f"myapp-{version}")
except kubernetes.client.rest.ApiException as e:
print(f"Error deploying new version: {e}")
raise
def switch_traffic(self, version: str) -> None:
"""Switch traffic to the specified version."""
try:
# Update service selector
service = self.core_api.read_namespaced_service(
name="myapp-service",
namespace=self.namespace
)
service.spec.selector["version"] = version
self.core_api.patch_namespaced_service(
name="myapp-service",
namespace=self.namespace,
body=service
)
print(f"Traffic switched to {version} version")
except kubernetes.client.rest.ApiException as e:
print(f"Error switching traffic: {e}")
raise
def _wait_for_deployment(self, name: str, timeout: int = 300) -> None:
"""Wait for deployment to be ready."""
start = time.time()
while time.time() - start < timeout:
try:
deployment = self.api.read_namespaced_deployment(
name=name,
namespace=self.namespace
)
if (deployment.status.available_replicas == deployment.spec.replicas and
deployment.status.ready_replicas == deployment.spec.replicas):
print(f"Deployment {name} is ready")
return
except kubernetes.client.rest.ApiException as e:
print(f"Error checking deployment status: {e}")
raise
time.sleep(5)
raise TimeoutError(f"Deployment {name} not ready after {timeout} seconds")
def verify_deployment(self, version: str) -> bool:
"""Verify the deployment health."""
try:
deployment = self.api.read_namespaced_deployment(
name=f"myapp-{version}",
namespace=self.namespace
)
# Check deployment status
if (deployment.status.available_replicas == deployment.spec.replicas and
deployment.status.ready_replicas == deployment.spec.replicas):
return True
return False
except kubernetes.client.rest.ApiException as e:
print(f"Error verifying deployment: {e}")
return False
def rollback(self, from_version: str, to_version: str) -> None:
"""Rollback to the previous version."""
try:
# Switch traffic back
self.switch_traffic(to_version)
# Scale down the problematic deployment
deployment = self.api.read_namespaced_deployment(
name=f"myapp-{from_version}",
namespace=self.namespace
)
deployment.spec.replicas = 0
self.api.patch_namespaced_deployment(
name=f"myapp-{from_version}",
namespace=self.namespace,
body=deployment
)
print(f"Rolled back to {to_version} version")
except kubernetes.client.rest.ApiException as e:
print(f"Error during rollback: {e}")
raise
def main():
"""Main deployment process."""
deployer = BlueGreenDeployer()
try:
# Get current active version
current_version = deployer.get_active_version()
new_version = "green" if current_version == "blue" else "blue"
print(f"Current version: {current_version}")
print(f"Deploying new version: {new_version}")
# Deploy new version
deployer.deploy_new_version(new_version, f"myapp:2.0")
# Verify deployment
if deployer.verify_deployment(new_version):
# Switch traffic
deployer.switch_traffic(new_version)
# Monitor new version
time.sleep(60) # Monitor for 1 minute
if not deployer.verify_deployment(new_version):
print("New version unstable, rolling back...")
deployer.rollback(new_version, current_version)
else:
print("Deployment successful!")
else:
print("Deployment verification failed, aborting...")
deployer.rollback(new_version, current_version)
except Exception as e:
print(f"Deployment failed: {e}")
raise
if __name__ == "__main__":
main()
`
$1
Implement comprehensive health checks:
`python
health_checks.py
Purpose: Verify application health during deployment
import requests
import time
from typing import Dict, List, Optional
from dataclasses import dataclass
@dataclass
class HealthCheckResult:
"""Represents the result of a health check."""
success: bool
message: str
metrics: Dict[str, float]
class HealthChecker:
"""Manages application health checks."""
def __init__(self, base_url: str, timeout: int = 5):
"""Initialize the health checker."""
self.base_url = base_url
self.timeout = timeout
def check_application_health(self) -> HealthCheckResult:
"""Perform comprehensive health check."""
try:
# Basic health check
response = requests.get(
f"{self.base_url}/health",
timeout=self.timeout
)
response.raise_for_status()
# Get application metrics
metrics_response = requests.get(
f"{self.base_url}/metrics",
timeout=self.timeout
)
metrics = metrics_response.json()
# Analyze metrics
error_rate = metrics.get("error_rate", 0)
response_time = metrics.get("response_time_p95", 0)
cpu_usage = metrics.get("cpu_usage", 0)
memory_usage = metrics.get("memory_usage", 0)
# Define health criteria
is_healthy = (
error_rate < 0.01 and # Less than 1% error rate
response_time < 500 and # Response time under 500ms
cpu_usage < 80 and # CPU usage under 80%
memory_usage < 80 # Memory usage under 80%
)
return HealthCheckResult(
success=is_healthy,
message="Application healthy" if is_healthy else "Health check failed",
metrics={
"error_rate": error_rate,
"response_time": response_time,
"cpu_usage": cpu_usage,
"memory_usage": memory_usage
}
)
except requests.exceptions.RequestException as e:
return HealthCheckResult(
success=False,
message=f"Health check failed: {str(e)}",
metrics={}
)
def monitor_deployment(url: str, duration: int = 300, interval: int = 10) -> bool:
"""Monitor deployment health for a specified duration."""
checker = HealthChecker(url)
start_time = time.time()
while time.time() - start_time < duration:
result = checker.check_application_health()
print(f"Health check result: {result.message}")
print("Metrics:", result.metrics)
if not result.success:
return False
time.sleep(interval)
return True
`
$1
#### 1. Jenkins Pipeline
` // Jenkinsfile
// Purpose: Define CI/CD pipeline for blue-green deployment pipeline {
agent any
environment {
DOCKER_REGISTRY = 'myregistry.azurecr.io'
APP_NAME = 'myapp'
NAMESPACE = 'production'
}
stages {
stage('Build') {
steps {
script {
// Build Docker image
docker.build("${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER}")
}
}
}
stage('Test') {
steps {
script {
// Run tests
sh 'python -m pytest tests/'
// Run security scan
sh 'trivy image ${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER}'
}
}
}
stage('Push') {
steps {
script {
// Push to registry
docker.withRegistry('https://${DOCKER_REGISTRY}', 'registry-credentials') {
docker.image("${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER}").push()
}
}
}
}
stage('Deploy') {
steps {
script {
// Deploy using blue-green strategy
def deployer = load 'deploy.py'
try {
// Get current version
def currentVersion = sh(
script: "kubectl get svc myapp-service -n ${NAMESPACE} -o jsonpath='{.spec.selector.version}'",
returnStdout: true
).trim()
// Determine new version
def newVersion = currentVersion == 'blue' ? 'green' : 'blue'
// Deploy new version
sh """
python deploy.py \
--namespace ${NAMESPACE} \
--version ${newVersion} \
--image ${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER}
"""
// Verify deployment
def healthCheck = load 'health_checks.py'
def isHealthy = healthCheck.monitor_deployment(
"http://myapp-${newVersion}.${NAMESPACE}",
300, // 5 minutes monitoring
10 // 10 seconds interval
)
if (!isHealthy) {
error "Deployment verification failed"
}
} catch (Exception e) {
// Rollback on failure
sh """
python deploy.py \
--namespace ${NAMESPACE} \
--rollback \
--version ${currentVersion}
"""
error "Deployment failed: ${e.message}"
}
}
}
}
stage('Cleanup') {
steps {
script {
// Cleanup old resources
sh """
# Remove old images
docker rmi ${DOCKER_REGISTRY}/${APP_NAME}:${BUILD_NUMBER}
# Clean up old deployments
kubectl delete pods -n ${NAMESPACE} --field-selector status.phase=Succeeded
"""
}
}
}
}
post {
success {
// Notify on success
slackSend(
color: 'good',
message: "Deployment successful: ${APP_NAME} version ${BUILD_NUMBER}"
)
}
failure {
// Notify on failure
slackSend(
color: 'danger',
message: "Deployment failed: ${APP_NAME} version ${BUILD_NUMBER}"
)
}
}
}
groovy
`
#### 2. GitHub Actions
`yaml
.github/workflows/deploy.yml
Purpose: GitHub Actions workflow for blue-green deployment
name: Blue-Green Deployment
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
env:
REGISTRY: ghcr.io
APP_NAME: myapp
NAMESPACE: production
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Login to Container Registry
uses: docker/login-action@v2
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push image
uses: docker/build-push-action@v4
with:
context: .
push: true
tags: ${{ env.REGISTRY }}/${{ env.APP_NAME }}:${{ github.sha }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: python -m pytest tests/
- name: Deploy
run: |
# Get current version
CURRENT_VERSION=$(kubectl get svc myapp-service -n ${NAMESPACE} -o jsonpath='{.spec.selector.version}')
NEW_VERSION=$([ "$CURRENT_VERSION" = "blue" ] && echo "green" || echo "blue")
# Deploy new version
python deploy.py \
--namespace ${NAMESPACE} \
--version ${NEW_VERSION} \
--image ${REGISTRY}/${APP_NAME}:${GITHUB_SHA}
# Monitor deployment
python health_checks.py \
--url http://myapp-${NEW_VERSION}.${NAMESPACE} \
--duration 300 \
--interval 10
env:
KUBECONFIG: ${{ secrets.KUBECONFIG }}
- name: Notify on success
if: success()
uses: rtCamp/action-slack-notify@v2
env:
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
SLACK_MESSAGE: 'Deployment successful! :rocket:'
SLACK_COLOR: good
- name: Notify on failure
if: failure()
uses: rtCamp/action-slack-notify@v2
env:
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
SLACK_MESSAGE: 'Deployment failed! :x:'
SLACK_COLOR: danger
`
$1
#### 1. Pre-Switch Testing
` def pre_switch_tests():
"""Tests to run before switching traffic."""
return {
'smoke_tests': run_smoke_tests(),
'integration_tests': run_integration_tests(),
'performance_tests': run_performance_tests(),
'security_scans': run_security_scans()
} def validate_test_results(results):
"""Validate test results before proceeding."""
thresholds = {
'smoke_tests': 1.0,
'integration_tests': 0.95,
'performance_tests': 0.9,
'security_scans': 1.0
}
return all(results[test] >= threshold
for test, threshold in thresholds.items())
python
`
#### 2. Post-Switch Monitoring
` def post_switch_monitoring():
"""Monitor application after traffic switch."""
metrics = [
'request_rate',
'error_rate',
'latency_p95',
'cpu_usage',
'memory_usage'
]
baseline = get_baseline_metrics()
current = get_current_metrics()
return compare_metrics(baseline, current, metrics)
python
`
$1
#### 1. Automated Rollback
` def rollback_deployment():
"""Rollback to previous stable version."""
try:
# Switch traffic back to blue environment
switch_traffic('green', 'blue')
# Scale down green deployment
scale_deployment('green', 0)
# Verify blue environment health
if not health_check('blue'):
raise RollbackError('Blue environment unhealthy')
except Exception as e:
notify_team(f'Rollback failed: {str(e)}')
raise
python
`
#### 2. Manual Intervention
` manual_rollback_steps:
1: Verify blue environment status
2: Update service selector to blue
3: Monitor blue environment metrics
4: Scale down green deployment
5: Notify stakeholders
6: Document incident
yaml
`
$1
#### 1. Deployment Guidelines
` deployment_guidelines:
preparation:
- Verify both environments are identical
- Ensure sufficient resources
- Update deployment documentation
execution:
- Deploy during low-traffic periods
- Use automated deployment scripts
- Implement proper monitoring
validation:
- Run comprehensive tests
- Monitor key metrics
- Have rollback plan ready
yaml
`
#### 2. Monitoring Setup
` monitoring_setup:
metrics:
- Response time
- Error rates
- Resource utilization
- Business metrics
alerts:
- Error rate threshold
- Latency threshold
- Resource exhaustion
dashboards:
- Deployment status
- Application health
- System metrics
yaml
``
$1
Successful blue-green deployments require:
1. Proper infrastructure setup
2. Automated deployment scripts
3. Comprehensive testing
4. Robust monitoring
5. Quick rollback capability
Remember to:
$1
1. [Kubernetes Documentation](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
2. [AWS Blue/Green Deployments](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/blue-green.html)
3. [Jenkins Pipeline Documentation](https://www.jenkins.io/doc/book/pipeline/)
4. [GitHub Actions Documentation](https://docs.github.com/en/actions)
5. [Docker Documentation](https://docs.docker.com/)
6. [Python Kubernetes Client](https://github.com/kubernetes-client/python)
7. [Prometheus Monitoring](https://prometheus.io/docs/introduction/overview/)
8. [ELK Stack for Logging](https://www.elastic.co/what-is/elk-stack)
Why This Matters
Understanding the business and technical context helps you make informed decisions rather than blindly following patterns.
Trade-offs to Consider
Every architectural decision involves trade-offs. Consider your specific requirements, team expertise, and scale when evaluating options.
When NOT to Use This
Knowing when a solution doesn't apply is as valuable as knowing when it does. Consider alternatives for your specific situation.
Decision Framework
Use this framework to evaluate whether this approach is right for your use case based on your specific constraints and requirements.