Comprehensive AWS Resource Monitoring with Amazon CloudWatch

TL;DR: Amazon CloudWatch is AWS's primary service for collecting, visualizing, and acting upon metrics and logs from your AWS resources. This guide covers its fundamental components (metrics, logs, alarms), practical setup for common resources, advanced techniques like custom metrics and log analytics, and specialized applications in performance, security, and cost monitoring.

---

$1

Amazon CloudWatch is a foundational observability service for AWS. It aggregates and tracks metrics, collects and monitors log files, sets alarms based on metric thresholds, and automates responses to these alarms. CloudWatch provides a unified view of your AWS infrastructure and application performance.

``mermaid

flowchart TB

A[AWS Resources & Applications] --> B(CloudWatch)

B --> C{Observability Data}

C --> D[Metrics]

C --> E[Logs]

D --> F[Dashboards]

E --> F

B --> G[Alarms]

G --> H{Automated Actions}

H --> I[SNS Notifications]

H --> J[Auto Scaling Actions]

H --> K[Lambda Functions]

`

$1

$1

CloudWatch metrics are time-series data representing the performance and health of AWS resources. They are organized hierarchically, enabling detailed filtering and aggregation.

* Namespace: A container for CloudWatch metrics. It distinguishes metrics from different services or applications (e.g., AWS/EC2, AWS/RDS, custom namespaces like CustomAppMetrics).

* Metric Name: The specific metric being tracked (e.g., CPUUtilization, NetworkIn, Invocations).

* Dimensions: Key-value pairs that uniquely identify a metric. They allow you to segment metric data by specific attributes of a resource. A single metric can have multiple dimensions.

*Example: For an EC2 instance, dimensions might include InstanceId. For an RDS instance, dimensions could include DBInstanceIdentifier.

* Value: The numerical data point collected for the metric.

* Timestamp: The date and time the metric data point was recorded.

* Unit: The unit of measurement for the metric (e.g., Percent, Bytes, Count, Seconds).

Example: Publishing Custom Metrics via Lambda

This example demonstrates how to publish a custom metric from a Lambda function.

`yaml

Resources:

# IAM Role for Lambda to put metric data

LambdaBasicExecutionRole:

Type: AWS::IAM::Role

Properties:

AssumeRolePolicyDocument:

Version: '2012-10-17'

Statement:

- Effect: Allow

Principal:

Service: lambda.amazonaws.com

Action: sts:AssumeRole

ManagedPolicyArns:

- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

# Lambda Function to publish custom metric

CustomMetricPublisher:

Type: AWS::Lambda::Function

Properties:

FunctionName: CustomMetricPublisher

Handler: index.handler

Role: !GetAtt LambdaBasicExecutionRole.Arn

Code:

ZipFile: |

const AWS = require('aws-sdk');

const cloudwatch = new AWS.CloudWatch();

exports.handler = async (event) => {

console.log("Received event:", JSON.stringify(event, null, 2));

try {

const params = {

Namespace: 'MyAppMetrics', // Custom namespace for your application

MetricData: [

{

MetricName: 'SuccessfulOrders',

Dimensions: [

{

Name: 'Environment',

Value: process.env.ENVIRONMENT || 'Development' // Example dimension

},

{

Name: 'Region',

Value: process.env.AWS_REGION

}

],

Value: 1, // Increment count for a successful order

Unit: 'Count'

}

]

};

await cloudwatch.putMetricData(params).promise();

console.log("Successfully put metric data.");

return { statusCode: 200, body: 'Metric published successfully' };

} catch (error) {

console.error("Error publishing metric:", error);

return { statusCode: 500, body: 'Error publishing metric' };

}

};

Runtime: nodejs18.x

Environment:

Variables:

ENVIRONMENT: Production # Set your desired environment

`

$1

CloudWatch Logs allows you to centralize logs from various AWS services and applications.

* Log Streams: Sequences of log events from a specific source (e.g., a single EC2 instance, a Lambda function invocation).

* Log Groups: Collections of log streams that share the same retention, monitoring, and access control settings.

* Log Events: Individual log records, each containing a timestamp and the log message.

Example: Configuring Log Group Retention and Metric Filter

This example sets up a log group for application logs with a 30-day retention policy and a metric filter to count specific error messages.

`yaml

Resources:

# Log Group for application logs

ApplicationLogGroup:

Type: AWS::Logs::LogGroup

Properties:

LogGroupName: /aws/application/my-app/production # Consistent naming is crucial

RetentionInDays: 30 # Set retention policy

Tags:

- Key: Environment

Value: Production

# Metric Filter to count specific error messages

ApplicationErrorMetricFilter:

Type: AWS::Logs::MetricFilter

Properties:

LogGroupName: !Ref ApplicationLogGroup

FilterPattern: '[timestamp, level, message="ERROR"]' # Pattern to match log entries containing "ERROR"

MetricTransformations:

- MetricName: ApplicationErrors # Name of the custom metric

MetricNamespace: MyAppMetrics # Use the same namespace as custom metrics for consistency

MetricValue: '1' # Increment this value for each matching log event

Unit: 'Count'

`

$1

$1

CloudWatch automatically collects standard metrics for most AWS resources. These metrics provide a baseline for understanding resource behavior.

| Resource Type | Key Default Metrics | Default Collection Frequency |

| :------------ | :--------------------------------------------------- | :--------------------------- |

| EC2 | CPUUtilization, NetworkIn, NetworkOut, DiskReadOps, DiskWriteOps | 5 minutes (standard) / 1 minute (detailed monitoring, extra cost) |

| RDS | CPUUtilization, FreeStorageSpace, DatabaseConnections, ReadIOPS, WriteIOPS | 1 minute |

| Lambda | Invocations, Errors, Duration, Throttles, ConcurrentExecutions | 1 minute |

| ECS | CPUUtilization, MemoryUtilization (task-level) | 1 minute |

| ELB (ALB/NLB) | RequestCount, TargetResponseTime, HTTPCode_Target_5XX_Count | 1 minute |

Example: Enabling Detailed Monitoring for EC2 Instances

Detailed monitoring (1-minute intervals) provides more granular insights but incurs additional costs. The CloudWatch Agent can also be used to collect custom metrics from EC2 instances.

`yaml

Resources:

# IAM Role for EC2 instance to interact with CloudWatch Agent and SSM

EC2InstanceProfile:

Type: AWS::IAM::InstanceProfile

Properties:

Roles:

- !Ref EC2InstanceRole

EC2InstanceRole:

Type: AWS::IAM::Role

Properties:

AssumeRolePolicyDocument:

Version: '2012-10-17'

Statement:

- Effect: Allow

Principal:

Service: ec2.amazonaws.com

Action: sts:AssumeRole

Policies:

- PolicyName: CloudWatchAgentPolicy

PolicyDocument:

Version: '2012-10-17'

Statement:

- Effect: Allow

Action:

- cloudwatch:PutMetricData

- logs:PutLogEvents

- logs:CreateLogStream

- logs:DescribeLogStreams

Resource: "*"

- PolicyName: SSMParameterPolicy # For fetching agent configuration via SSM

PolicyDocument:

Version: '2012-10-17'

Statement:

- Effect: Allow

Action:

- ssm:GetParameters

Resource: "arn:aws:ssm::*:parameter/AmazonCloudWatch/Config/"

# EC2 Instance configured with CloudWatch Agent

MyMonitoredEC2Instance:

Type: AWS::EC2::Instance

Properties:

InstanceType: t3.medium

ImageId: ami-0abcdef1234567890 # Replace with a valid AMI ID

SubnetId: subnet-xxxxxxxxxxxxxxxxx # Replace with your Subnet ID

SecurityGroupIds:

- sg-xxxxxxxxxxxxxxxxx # Replace with your Security Group ID

IamInstanceProfile: !Ref EC2InstanceProfile

UserData:

Fn::Base64: !Sub |

#!/bin/bash -xe

# Install CloudWatch Agent

curl https://s3.amazonaws.com/amazoncloudwatch-agent/amazon-linux/amd64/latest/amazon-cloudwatch-agent.rpm -o amazon-cloudwatch-agent.rpm

sudo rpm -U ./amazon-cloudwatch-agent.rpm

# Fetch configuration from SSM Parameter Store

# Ensure a parameter named /AmazonCloudWatch/Config/MyEC2Config exists in SSM

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c ssm:/AmazonCloudWatch/Config/MyEC2Config

# Start and enable CloudWatch Agent

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start

# Example SSM Parameter for CloudWatch Agent Configuration

CloudWatchAgentConfigParameter:

Type: AWS::SSM::Parameter

Properties:

Name: /AmazonCloudWatch/Config/MyEC2Config

Type: String

Value: |

{

"agent": {

"metrics_collection_interval": 60,

"run_as_user": "cwagent"

},

"logs": {

"configs": [

{

"log_group_name": "/aws/ec2/my-instance-logs",

"log_stream_name": "${aws:InstanceId}",

"file_path": "/var/log/messages",

"log_group_retention_in_days": 14

}

]

},

"metrics": {

"namespace": "EC2CustomMetrics",

"metrics_collected": {

"disk": {

"measurement": [

"used_percent"

],

"metrics_collection_interval": 60,

"resources": [

"*"

],

"ignore_file_system_types": [

"sysfs",

"devtmpfs",

"tmpfs",

"squashfs"

]

},

"cpu_usage": {

"metrics_collection_interval": 60

}

}

}

}

`

$1

CloudWatch Alarms monitor a single metric over a specified period and perform one or more actions based on the value of the metric relative to a given threshold.

* States: OK, ALARM, INSUFFICIENT_DATA.

* Evaluation Periods: The number of consecutive periods the metric must breach the threshold to trigger an alarm.

* Datapoints to Alarm: The number of datapoints within the evaluation periods that must be breaching.

* Actions: What happens when the alarm state changes (e.g., send a notification to an SNS topic, trigger an Auto Scaling action, invoke a Lambda function).

Alarm States Transition Diagram:

`mermaid

stateDiagram-v2

[*] --> OK: Initial State / Metric in range

OK --> ALARM: Threshold Breached (Evaluation Periods Met)

ALARM --> OK: Threshold Restored (Evaluation Periods Met)

OK --> INSUFFICIENT_DATA: Missing Data (for a period)

INSUFFICIENT_DATA --> OK: Data Resumed (within range)

ALARM --> INSUFFICIENT_DATA: Missing Data (while in ALARM)

INSUFFICIENT_DATA --> ALARM: Threshold Breached (after data resumes)

`

Example: High CPU Utilization Alarm with SNS Notification

This alarm triggers when the average CPU utilization for a specific EC2 instance exceeds 80% for two consecutive 5-minute periods, sending a notification to an SNS topic.

`yaml

Resources:

# Assume EC2Instance is defined elsewhere and its ID is available (e.g., via !Ref)

# Assume AlertSNSTopic is an existing SNS Topic ARN

HighCPUUtilizationAlarm:

Type: AWS::CloudWatch::Alarm

Properties:

AlarmName: ProductionEC2HighCPUAlarm

AlarmDescription: "Alarm when CPU utilization exceeds 80% for more than 10 minutes."

Namespace: AWS/EC2

MetricName: CPUUtilization

Dimensions:

- Name: InstanceId

Value: !Ref MyMonitoredEC2Instance # Reference to the EC2 instance defined previously

Statistic: Average

Period: 300 # 5 minutes

EvaluationPeriods: 2 # Two consecutive 5-minute periods

Threshold: 80

ComparisonOperator: GreaterThanThreshold

TreatMissingData: ignore # or "breaching", "notBreaching", "missing"

AlarmActions:

- !Ref AlertSNSTopic # ARN of the SNS topic to notify

OKActions:

- !Ref AlertSNSTopic # Optional: Notify when OK

`

$1

$1

* Custom Metrics: Beyond automatic collection, you can publish your own metrics to CloudWatch, ideal for application-specific KPIs or business logic. These can be published programmatically (e.g., from Lambda, EC2 using SDKs) or via the CloudWatch Agent.

* Dashboards: Create custom dashboards to visualize a collection of metrics, logs, and alarms in a single pane of glass. This is essential for providing a consolidated view for different teams or stakeholders.

Dashboard Layout Example:

`mermaid

graph TB

subgraph Application Health Dashboard

A[EC2 Instance Metrics] --> E(Widgets)

B[RDS Performance Metrics] --> E

C[Lambda Function Metrics] --> E

D[Custom Business Metrics] --> E

end

subgraph Widgets

E --> F[CPU/Memory Usage (Line Graph)]

E --> G[Error Counts (Number)]

E --> H[Latency (Gauge)]

E --> I[Request Throughput (Line Graph)]

end

`

Example: Creating a CloudWatch Dashboard via CloudFormation

`yaml

Resources:

ApplicationMonitoringDashboard:

Type: AWS::CloudWatch::Dashboard

Properties:

DashboardName: ApplicationPerformanceOverview

DashboardBody: |

{

"widgets": [

{

"type": "metric",

"properties": {

"metrics": [

["AWS/EC2", "CPUUtilization", "InstanceId", "${MyMonitoredEC2Instance}"],

["AWS/EC2", "NetworkIn", "InstanceId", "${MyMonitoredEC2Instance}"],

["AWS/EC2", "NetworkOut", "InstanceId", "${MyMonitoredEC2Instance}"]

],

"view": "timeSeries",

"stacked": false,

"region": "${AWS::Region}",

"stat": "Average",

"period": 300,

"title": "EC2 Instance Performance"

}

},

{

"type": "log.metrics",

"properties": {

"metrics": [

[ "MyAppMetrics", "ApplicationErrors", "Environment", "Production" ]

],

"view": "timeSeries",

"stacked": false,

"region": "${AWS::Region}",

"title": "Application Errors (from Log Filter)"

}

},

{

"type": "alarm",

"properties": {

"alarms": [

"${HighCPUUtilizationAlarm}"

]

}

}

]

}

`

$1

CloudWatch Logs Insights is a powerful query language that allows you to interactively search and analyze your log data. You can query logs from various sources in one place, identify the root cause of issues, and discover trends.

Log Analysis Workflow:

`mermaid

sequenceDiagram

participant AppServer as Application Servers

participant LambdaFunc as Lambda Functions

participant CWLogs as CloudWatch Logs

participant CWInsights as CloudWatch Logs Insights

participant CWAlarms as CloudWatch Alarms

participant LambdaTrigger as Triggering Lambda

AppServer->>CWLogs: Send log events

LambdaFunc->>CWLogs: Send log events

CWLogs-->>CWInsights: Query log data

CWInsights-->>CWAlarms: Create metrics from queries

CWAlarms-->>LambdaTrigger: Trigger actions based on metric thresholds

`

Example CloudWatch Logs Insights Query:

To find all error messages from a specific time range:

`sql

fields @timestamp, @message

| filter @message like /ERROR/

| sort @timestamp desc

| limit 20

`

To count occurrences of specific error codes:

`sql

filter @message like /ERROR/

| parse @message " * ErrorCode: *, Message: " as timestamp, level, errorCode, message

| stats count(*) by errorCode

| sort count(*) desc

`

$1

$1

Monitor application-level metrics to understand user experience and internal application health.

* Key Metrics: Latency, error rates, request throughput, transaction success rates, custom business metrics.

* Integrated Services: CloudWatch often works with AWS X-Ray for distributed tracing and ServiceLens for visualizing service dependencies and performance.

Application Monitoring Architecture:

`mermaid

graph TB

subgraph Application Tier

A[Web Servers]

B[Application Servers]

C[Databases]

D[Microservices]

end

subgraph Observability Layer

E[CloudWatch Metrics & Logs]

F[AWS X-Ray]

G[CloudWatch Synthetics]

H[ServiceLens]

end

A --> E

B --> E

C --> E

D --> E

E --> F

F --> H

G --> E

G --> H

`

$1

Monitor resource utilization metrics to ensure optimal performance and to inform scaling decisions.

Example: Configuring Auto Scaling Group Metrics Collection

This configuration enables CloudWatch to collect metrics related to the size and state of an Auto Scaling group.

`yaml

Resources:

MyAutoScalingGroup:

Type: AWS::AutoScaling::AutoScalingGroup

Properties:

# ... other ASG properties ...

MinSize: 1

MaxSize: 5

DesiredCapacity: 2

LaunchTemplate:

LaunchTemplateId: !Ref MyLaunchTemplate # Assuming a LaunchTemplate exists

Version: "$Latest"

# Enable CloudWatch metrics collection

MetricsCollection:

- Granularity: 1Minute # Available: 1Minute, 5Minute

Metrics:

- GroupMinSize

- GroupMaxSize

- GroupDesiredCapacity

- GroupInServiceInstances

- GroupPendingInstances

- GroupTerminatingInstances

- GroupTotalInstances

`

$1

$1

Leverage CloudWatch to aggregate and analyze security-related data from various AWS services.

Security Monitoring Workflow:

`mermaid

graph LR

A[AWS CloudTrail] -->|API Logs| B(CloudWatch Logs)

C[GuardDuty] -->|Findings| B

D[Security Hub] -->|Alerts| B

E[VPC Flow Logs] -->|Network Traffic| B

F[AWS Config] -->|Configuration Changes| B

B --> G(CloudWatch Logs Insights)

G --> H[Security Analysis]

G --> I[Alarms on Security Events]

I --> J[Automated Remediation Lambda]

`

Example: CloudTrail Event Monitoring with Lambda

This setup triggers a Lambda function for specific CloudTrail events, such as console logins.

`yaml

Resources:

# IAM Role for Lambda to process events and log them

SecurityEventProcessorRole:

Type: AWS::IAM::Role

Properties:

AssumeRolePolicyDocument:

Version: '2012-10-17'

Statement:

- Effect: Allow

Principal:

Service: lambda.amazonaws.com

Action: sts:AssumeRole

ManagedPolicyArns:

- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

# Lambda Function to process security events

SecurityEventProcessor:

Type: AWS::Lambda::Function

Properties:

FunctionName: SecurityEventProcessor

Handler: index.handler

Role: !GetAtt SecurityEventProcessorRole.Arn

Code:

ZipFile: |

const AWS = require('aws-sdk');

const cloudwatch = new AWS.CloudWatch();

exports.handler = async (event) => {

console.log("Received event:", JSON.stringify(event, null, 2));

const eventSource = event.source;

const detailType = event['detail-type'];

const detail = event.detail;

if (eventSource === 'aws.signin' && detailType === 'AWS Console Sign In via CloudTrail') {

console.log('AWS Console Login Event:');

console.log('User Identity:', detail.userIdentity.principalId);

console.log('User Type:', detail.userIdentity.type);

console.log('IP Address:', detail.sourceIPAddress);

console.log('Login Status:', detail.responseElements ? detail.responseElements.ConsoleLogin : 'N/A');

// Publish a custom metric for console logins

try {

await cloudwatch.putMetricData({

Namespace: 'SecurityMetrics',

MetricData: [{

MetricName: 'ConsoleLogins',

Dimensions: [{ Name: 'LoginStatus', Value: detail.responseElements ? detail.responseElements.ConsoleLogin : 'Unknown' }],

Value: 1,

Unit: 'Count'

}]

}).promise();

console.log("Published ConsoleLogins metric.");

} catch (error) {

console.error("Error publishing metric:", error);

}

}

// Add more event handling logic here for other security events

};

Runtime: nodejs18.x

# Event Rule to capture specific CloudTrail events

SecurityEventRule:

Type: AWS::Events::Rule

Properties:

RuleName: CaptureConsoleLogins

Description: Trigger Lambda for AWS Console Sign In events

EventPattern:

source:

- aws.signin

detail-type:

- AWS Console Sign In via CloudTrail

State: ENABLED

Targets:

- Arn: !GetAtt SecurityEventProcessor.Arn

Id: SecurityEventProcessorTarget

`

$1

Monitor authentication events to detect suspicious login attempts or unauthorized access. This is often done by analyzing CloudTrail logs for ConsoleLogin or ConsoleLoginFailure events.

$1

$1

While CloudWatch itself incurs costs, it's also a critical tool for monitoring AWS spending.

* Cost Explorer: Integrates with CloudWatch to visualize spending trends and identify cost drivers.

* AWS Budgets: Set custom budgets and receive alerts when costs or usage exceed (or are forecasted to exceed) your budgeted amounts. CloudWatch alarms can be configured based on budget thresholds.

Cost Monitoring Dashboard Components:

`mermaid

graph TB

A[AWS Cost Explorer API] --> B(CloudWatch Metrics)

C[AWS Budgets API] --> B

B --> D[CloudWatch Dashboards]

D --> E[Daily Spend Visualization]

D --> F[Monthly Spend Trends]

D --> G[Service-wise Cost Breakdown]

D --> H[Budget Threshold Alerts]

`

$1

Identify underutilized or idle resources by analyzing their metrics. CloudWatch alarms can trigger automated actions to scale down or terminate these resources.

Example: Lambda Function for Resource Cleanup

This example provides a skeletal Lambda function that could be extended to identify and clean up resources.

``yaml

Resources:

# IAM Role for Lambda to describe and terminate EC2 instances

ResourceCleanupRole:

Type: AWS::IAM::Role

Properties:

AssumeRolePolicyDocument:

Version: '2012-10-17'

Statement:

- Effect: Allow

Principal:

Service: lambda.amazonaws.com

Action: sts:AssumeRole

Policies:

- PolicyName: EC2CleanupPermissions

PolicyDocument:

Version: '2012-10-17'

Statement:

- Effect: Allow

Action:

- ec2:DescribeInstances

- ec2:TerminateInstances

- ec2:DescribeVolumes

- ec2:DeleteVolume

- ec2:DescribeSnapshots

- ec2:DeleteSnapshot

- ec2:DescribeSecurityGroups

- ec2:DeleteSecurityGroup

# Add other resource types as needed

Resource: "*"

ResourceCleanupFunction:

Type: AWS::Lambda::Function

Properties: