Building observability to increase resiliency.pdf

Building observability to
increase resiliency
David Yanacek
C O P 3 4 3 - R
(he/him)
Sr. Principal Engineer
AWS Monitoring and Observability

Navigating the seas of observability

My sailing proficiency

Diagnose
issues
Uncover
hidden issues
Prevent
future issues
Agenda

Diagnose
issues
Learning objectives
Find patterns in high-cardinality
metrics
Use dimensionality to compute
the right metric
Navigate distributed systems
using tracing

Learning objectives
Uncover
hidden issues
Measure from everywhere
Aggregate metrics in customer-
oriented ways

Learning objectives
Auto scale and track the
utilization of everything
Monitor game days just like
production
Prevent
future issues

All aboard!

Diagnose issues

Diagnose issues
Bad dependency Bad component Bad deployment Traffic spike

A leak has sprung!

A “leak” in the real world
Navigation
Product
Search
Browse
[Your website logo] Cart

(Un)awareness of a problem
Time
Error
rate
Site-wide error rate
Alarm
Website errors

(Un)awareness of a problem
Time
Error
rate
Problem
Alarm
Website errors

Considering dimensionality
“Show me [metric] per [dimension]”

Latency
Measurements
Requests
Errors

Latency Website
Measurements Attributes
Requests
Errors

Latency Website
Measurements Attributes
Requests Webpage
Errors

A deflated error rate
Product requests vs errors Cart requests vs errors
Requests
Requests
Time Time
Cart requests
Product requests
Cart errors
Product errors

A deflated error rate
Cart requests
Product requests
Cart errors
Product errors +
+
=

Per-API error rate
Cart requests
Product requests
Cart errors
Product errors
= =

Per-API error rate
Time
Error
rate
Per-widget error rate Cart errors
Product errors

Per-API alarms
Time
Error
rate
Per-widget error rate
Cart alarm
Cart errors
Product errors
Product alarm

Per-API alarms
Time
Error
rate
Per-widget error rate Cart errors
Product errors
Cart alarm
Product alarm

Per-API alarm noise
Cart alarms
Product alarms Browse alarms Search alarms Nav alarms

Composite alarms
Cart alarms
Overall alarm

Alarms for everything
Cart alarms
Overall alarm
Latency
Errors
Volume
Latency
Errors
Volume
Latency
Errors
Volume
Latency
Errors
Volume
Latency
Errors
Volume

Diagnose issues: Dimensions and alarms
Split key application health metrics on separate
dimensions for each customer use case, like per-
webpage or per-widget

Diagnose issues: Dimensions and alarms
dimensions for each customer use case, like per-
webpage or per-widget
Combine many alarm signals together to avoid
fatigue by using CloudWatch Composite Alarms

Diagnose issues
Bad dependency

Find the source of the problem
Navigation
Product
Search
Browse
[Your website logo] Cart

Triangulate the problem

Website architecture
Users Orders queue
Search index
Product DB
Product
info service
Ordering
service
Product search
service
Website

Website architecture
Users Orders queue
PlaceOrder()
Ordering
service
Website

Placing an order: Detailed architecture
AWS Cloud
Web server stack Cart service stack
Users Website Ordering
service
Orders queue

Placing an order: Detailed architecture
Application
Load Balancer
Availability Zone 1
Instances
AWS Cloud
Availability Zone 2
Instances
Availability Zone 3
Instances
Web server stack
Orders queue
Application
Load Balancer
Availability Zone 1
Instances
Availability Zone 2
Instances
Availability Zone 3
Instances
Cart service stack
Users

Trace propagation
Application
Load Balancer
Availability Zone 1
Instances
AWS Cloud
Availability Zone 2
Instances
Availability Zone 3
Instances
Web server stack
Orders queue
Application
Load Balancer
Availability Zone 1
Instances
Availability Zone 2
Instances
Availability Zone 3
Instances
Cart service stack
Trace-Id: 1-5759e988
Users

Trace propagation
Application
Load Balancer
Availability Zone 1
Instances
AWS Cloud
Availability Zone 2
Instances
Web server stack
Orders queue
Application
Load Balancer
Availability Zone 1
Instances
Availability Zone 2
Instances
Cart service stack
… …
AWS X-Ray
Users

Service map
Users Orders queue
Search index
Product DB
Product
info service
Ordering
service
Product search
service
Website

CloudWatch ServiceLens Map using AWS X-Ray

Diagnose issues: Bad dependency
Propagate the incoming trace context to every
outbound client call you make to every dependency

Enable collection of trace segments from all AWS
services, and from your apps
AWS X-Ray

Enable collection of trace segments from all AWS
services, and from your apps
Use service maps derived from traces to triangulate
the failing dependency during incidents
AWS X-Ray

Diagnose issues
Bad component

A web server process crash
Application
Load Balancer
AWS Cloud
Web server environment
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers

Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers
…

Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers
~11%

Health checks to the rescue
Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers

The deadlock
Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 2
Web servers
Availability Zone 2
Web servers
Product info service Ordering service
Product search service

Health checks to the rescue?
Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers

Health checks to the rescue?
Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers
https://aws.amazon.com/builders-library/implementing-health-checks/

Different types of challenges
Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers

Measuring availability
Time
Error
rate
Website errors

Dimensionality again
Latency
Errors Webpage
Website
Instance Id
Measurements
Attributes

Creating an alarm for a gray-failed instance
Time
Error
rate
Per-instance error rate
ghi789
abc123
def456
jkl012
…

CloudWatch Metrics Insights query and alarm
SELECT SUM(Failure)
FROM SCHEMA(MyWebsite, InstanceId)
GROUP BY InstanceId
ORDER BY SUM() DESC
LIMIT 10

SELECT SUM(Failure)
FROM SCHEMA(MyWebsite, InstanceId)
GROUP BY InstanceId
ORDER BY SUM() DESC
LIMIT 10
q1 =
FIRST(q1) > 0.01
PERIOD
DATAPOINTS
= 1 minute
= 2

Different types of challenges
Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers
Cache nodes Cache nodes Cache nodes
Product catalog
cache cluster

Dimensionality again
Latency
Errors Availability Zone
Website
Webpage
Measurements
Attributes
Instance Id

Recovery with zonal shifts
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 3
Web servers
Availability Zone 1
Web servers
Cache nodes Cache nodes Cache nodes
Product catalog
cache cluster
Application
Load Balancer
Route 53
Application Recovery Controller
Operator

Diagnose issues: Bad component
dimensions for each infrastructure boundary, like EC2
Instance or Availability Zone

Diagnose issues: Bad component
dimensions for each infrastructure boundary, like EC2
Instance or Availability Zone
Find and alarm on the poorest performing parts of
your infrastructure using Metrics Insights queries

Diagnose issues
Bad deployment

How (not) to deploy

Let’s try that again
Scene: Safe deployment
Take: Two

Roll back first, ask questions later

Service in alarm!

Rollback started

Rollback complete

No rollback
Time
Error
rate
Website errors
Alarm

No rollback
Time
Error
rate
Website errors
Alarm
One-box deployment

No rollback
Time
Error
rate
Website errors
Alarm
One-box deployment
Wave 1 deployment

No rollback
Time
Error
rate
Website errors
Alarm
One-box deployment
Wave 1 deployment
Wave 2, 3 deployments

Manual rollback
Time
Error
rate
Website errors
Alarm

Manual rollback
Time
Error
rate
Website errors
Alarm
Start investigating

Manual rollback
Time
Error
rate
Website errors
Alarm
Start investigating
Realize there’s a deployment

Manual rollback
Time
Error
rate
Website errors
Alarm
Start rollback
Start investigating
Realize there’s a deployment

Manual rollback
Time
Error
rate
Website errors
Alarm
Time-to-detect Time-to-mitigate

Automatic rollback
Time
Error
rate
Website errors
Alarm

Automatic rollback
Time
Error
rate
Website errors
Alarm
Auto rollback started

Automatic rollback
Time
Error
rate
Website errors
Alarm
Time-to-detect
Time-to-mitigate

Automatic rollback in CloudFormation

Automatic rollback
Time
Error
rate
Website errors
Alarm
?

Dimensionality (yet) again
Latency
Errors
Measurements
Attributes
Code revision
Website
Availability Zone
Webpage
Instance Id

Faster detection using dimensions
Time
Error
rate
Overall
New alarm
New code
Overall alarm

Time
Error
rate
Overall
New alarm
New code
Overall alarm
Auto rollback started

Time
Error
rate
Overall
New alarm
New code
Overall alarm
Time-to-detect
Time-to-mitigate

Composite rollback alarms
New code
version alarms
Overall alarms
Per-webpage
alarms
Infrastructure
alarms
Literally all of
your alarms
Overall alarm

Diagnose issues: Bad deployment
Split key application health metrics on logical boundaries like
DeploymentId to minimize the time to detect bad changes
B
A →

Roll back all types of changes automatically to minimize
time to recover
B
A →

Roll back all types of changes automatically to minimize
time to recover
Combine literally all of your alarms into a single CloudWatch
composite alarm for triggering rollbacks
B
A →

Diagnose issues
Traffic spike

Troubleshooting
Latency alarm

Troubleshooting
Latency alarm
Average latency

Troubleshooting
Latency alarm
Average latency Deployments
4 days ago
5 days ago
…

Troubleshooting
Latency alarm
Average latency
Fleet CPU
Deployments
4 days ago
5 days ago
…

Troubleshooting
Latency alarm
Fleet size
Average latency
Fleet CPU
Deployments
4 days ago
5 days ago
…

Troubleshooting
Latency alarm
Fleet size
Average latency
Fleet CPU
Request volume
Deployments
4 days ago
5 days ago
…

Where/who are these requests coming from?
Request volume
∆?

Attributes
Code revision
Website
Availability Zone
Webpage
Instance Id
Customer Id
Dimensions with high cardinality
Requests
Latency
Errors
Measurements

Attributes
Code revision
Website
Availability Zone
Webpage
Instance Id
Customer Id
Requests
Cardinality
Latency
Errors
Measurements

Attributes
Code revision
Website
Availability Zone
Webpage
Instance Id
Customer Id
Requests
Cardinality
(Few of these)
Latency
Errors
Measurements

Attributes
Code revision
Website
Availability Zone
Webpage
Instance Id
Customer Id
Requests
Cardinality
(Many of these)
Latency
Errors
Measurements

Top-N metrics with CloudWatch
Contributor Insights
Per-customer requests
Time
Request
count

Top-N metrics with CloudWatch
Contributor Insights
Time
Request
count
(top-N)
Time
Request
count

Telemetry in Amazon CloudWatch
Amazon CloudWatch
Alarms
Logs Metrics
Application
{ "structured":"log" }
… …

Contributor Insights: How it’s made
{
"Timestamp": 1574109732004,
"TraceId": "Root=1-5759e988-...",
"CustomerId": "abcde12345",
"ClientIp": "192.168.131.39",
"InstanceId": "i-0012341EXAMPLE",
"Operation": "GetProducts",
"Time": 98,
"Error": 0,
"Failure": 0,
"DB.Time": 49,
"CacheHit": 0
}
A request log entry

{
"Timestamp": 1574109732004,
"TraceId": "Root=1-5759e988-...",
"ClientIp": "192.168.131.39",
"Time": 98,
"Error": 0,
"Failure": 0,
"DB.Time": 49,
"CacheHit": 0
}
(Things about the request)
A request log entry

{
"Timestamp": 1574109732004,
"TraceId": "Root=1-5759e988-...",
"ClientIp": "192.168.131.39",
"Time": 98,
"Error": 0,
"Failure": 0,
"DB.Time": 49,
"CacheHit": 0
}
(Things about the request)
A request log entry
(Things that happened during the request)

{
"Timestamp": 1574109732004,
"TraceId": "Root=1-5759e988-...",
"ClientIp": "192.168.131.39",
"Time": 98,
"Error": 0,
"Failure": 0,
"DB.Time": 49,
"CacheHit": 0
}
{
"AggregateOn": "Count",
"Contribution": {
"Keys": [ "$.CustomerId" ]
},
"LogFormat": "JSON",
"Schema": {
"Name": "CloudWatchLogRule",
"Version": 1
},
"LogGroupARNs": [ "arn:..." ]
}
A request log entry A Contributor Insights rule

Top-N metrics with Contributor Insights

On-demand metrics with Logs Insights
filter ClientIp = '192.168.131.39'
| stats count(*) by bin(60s)

Diagnose issues: Traffic spike
Emit logs that are rich with data so that you can cut your
metrics on many dimensions
{ "foo":"bar" }

Record and analyze high-cardinality metrics like per-
customer request volume by configuring Contributor Insights
{ "foo":"bar" }

Record and analyze high-cardinality metrics like per-
customer request volume by configuring Contributor Insights
Slice and dice metrics that you didn’t materialize up front by
writing Logs Insights queries
→
{ "foo":"bar" }

Uncover hidden issues

The hidden storm

External issues Misattributed faults

External issues

Measuring from where?
Users Website
…

Users
Application
Load Balancer
Availability Zone 1
Instances
AWS Cloud
Availability Zone 2
Instances
Availability Zone 3
Instances
(other services)
Your web server environment

Users
Application
Load Balancer
AWS Cloud

An incompatible change
Users
AWS Cloud
Amazon CloudFront
Distribution
Your CI/CD pipeline
Version 1
Code
Instances
Application
Load Balancer

Users
AWS Cloud
Amazon CloudFront
Distribution
Your CI/CD pipeline
Version 1
Code
Instances
Application
Load Balancer
{}

Users
AWS Cloud
Amazon CloudFront
Distribution
Your CI/CD pipeline
Version 1
Code
Instances
Application
Load Balancer
Version 2
Code

Users
AWS Cloud
Amazon CloudFront
Distribution
Your CI/CD pipeline
Version 1
Code
Version 2
Code
Instances
Application
Load Balancer

Users
AWS Cloud
Amazon CloudFront
Distribution
Your CI/CD pipeline
Version 1
Code
Version 2
Code
Instances
Application
Load Balancer
{???}

Users
Application
Load Balancer
AWS Cloud
Amazon CloudWatch
Real-User Monitoring

A botched migration
Application
Load Balancer
Instances
AWS Cloud
Containers
(other services)
Old web server environment
Users
Application
Load Balancer
New web server environment

A botched migration
Application
Load Balancer
Instances
AWS Cloud
Containers
(other services)
Users
Application
Load Balancer
Amazon Route 53
Old: 100%
New: 0%

A botched migration
Application
Load Balancer
Instances
AWS Cloud
Containers
(other services)
Users
Application
Load Balancer
Amazon Route 53
Old: 99%
New: 1%

Synthetic workload measurement

Users
Application
Load Balancer
AWS Cloud
Amazon CloudWatch
Synthetics

Uncover hidden issues: External issues
Look at metrics emitted by synthetic workloads and
real clients of your application to catch issues outside
of your own measurement

Misattributed faults

Input validation bugs
public void validate() {
- if(input.length() > 140) {
+ if(input.length() > 100) {
throw new ClientError();
}
}
(Bug: Making input validation more strict)

Client vs server fault
Client fault (4XX)
Server fault (5XX)

An increase due to a bad deployment
Time
Error
rate
Error rate by type (4XX vs 5XX)
Server fault

An increase due to a bad deployment
Time
Error
rate
Error rate by type (4XX vs 5XX)
Client fault
Server fault

The same increase, but for a different reason
Time
Error
rate
Customer 789
Customer 123
Customer 456
Customer 012
…
Customer 345
Overall errors
Per-client fault rate (4XX)

Alarming when lots of customers change
Time
Error
rate
Per-client fault rate (4XX)
Customer 789
Customer 123
Customer 456
Customer 012
…
Customer 345
Overall errors

Metrics from another dimension
Percent of requests
with errors
Percent of clients
with errors

Percent of requests
with errors
# 𝑜𝑓 𝒓𝒆𝒒𝒖𝒆𝒔𝒕𝒔 𝑤𝑖𝑡ℎ 𝑒𝑟𝑟𝑜𝑟𝑠
# 𝑜𝑓 𝒓𝒆𝒒𝒖𝒆𝒔𝒕𝒔

Percent of clients
with errors
# 𝑜𝑓 𝒄𝒖𝒔𝒕𝒐𝒎𝒆𝒓𝒔 𝑤𝑖𝑡ℎ 𝑒𝑟𝑟𝑜𝑟𝑠
# 𝑜𝑓 𝒄𝒖𝒔𝒕𝒐𝒎𝒆𝒓𝒔

Dimensionality and cardinality together

Dimensionality and cardinality together
𝐼𝑁𝑆𝐼𝐺𝐻𝑇_𝑅𝑈𝐿𝐸_𝑀𝐸𝑇𝑅𝐼𝐶(𝐸𝑟𝑟𝑜𝑟𝑠𝑃𝑒𝑟𝐶𝑢𝑠𝑡𝑜𝑚𝑒𝑟, 𝑈𝑛𝑖𝑞𝑢𝑒𝐶𝑜𝑛𝑡𝑟𝑖𝑏𝑢𝑡𝑜𝑟𝑠)
𝐼𝑁𝑆𝐼𝐺𝐻𝑇_𝑅𝑈𝐿𝐸_𝑀𝐸𝑇𝑅𝐼𝐶(𝑅𝑒𝑞𝑢𝑒𝑠𝑡𝑠𝑃𝑒𝑟𝐶𝑢𝑠𝑡𝑜𝑚𝑒𝑟, 𝑈𝑛𝑖𝑞𝑢𝑒𝐶𝑜𝑛𝑡𝑟𝑖𝑏𝑢𝑡𝑜𝑟𝑠)

A stable, alarmable metric
Percent of clients
with errors

Uncover hidden issues: Misattributed
When many customers suddenly see errors that are
categorized as their fault, it may actually be your fault

Uncover hidden issues: Misattributed
When many customers suddenly see errors that are
categorized as their fault, it may actually be your fault
Calculate metrics like “percent of customers” instead
of “percent of requests” by using Contributor Insights
rules to estimate the cardinality of a dimension

Prevent future issues

Run game days
Monitor utilization

Monitor utilization

Monitoring resource utilization

Elastic building blocks
Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 2
Web servers
Auto Scaling Group
…

Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 2
Web servers
Auto Scaling Group
Amazon
Elastic Compute Cloud
(Amazon EC2)
…

Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 2
Web servers
Auto Scaling Group
Elastic Load
Balancing
Amazon
(Amazon EC2)
…

Application
Load Balancer
AWS Cloud
Users
Availability Zone 2
Web servers
Availability Zone 2
Web servers
Auto Scaling Group
Elastic Load
Balancing
Amazon EC2
Auto Scaling
Amazon
(Amazon EC2)
…

Monitoring the utilization
CPU Memory Thread pools
File system

File system
Alarms

File system
Alarms Dashboards

File system
Amazon EC2
Auto Scaling
Alarms Dashboards

myASG:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
MaxSize: '50'
MinSize: '20'
myScalingPolicy:
...
Amazon EC2
Auto Scaling

Amazon EC2
Auto Scaling
GroupMaxSize
Instances
Time
GroupInServiceInstances
Number of instances
in an Auto Scaling Group

Amazon EC2
Auto Scaling
GroupMaxSize
Instances
Time
Number of instances
× 100
=

Amazon EC2
Auto Scaling
GroupMaxSize
Instances
%
Time Time
Number of instances
Percent of instances
in an Auto Scaling group
× 100
=
100% -

Monitoring utilization across abstractions
AWS Lambda
Amazon EC2
Amazon RDS Amazon DynamoDB
Amazon SQS
Amazon MQ

Monitoring utilization across abstractions
AWS Lambda
Amazon EC2
CPU Memory
File system Thread pools
Concurrency

Automatic usage metrics

Configure Auto Scaling on all of your elastic
resources to react quickly to changes in load and
maintain healthy headroom

Configure Auto Scaling on all of your elastic
resources to react quickly to changes in load and
maintain healthy headroom
Measure the utilization of everything – from CPU,
to thread pools, to quotas – by creating alarms and
a capacity dashboard

Run game days

Game day drills

Properties of a good game day
Regular
Reasoned
Controlled
Realistic

Properties of a good game day
Regular
Reasoned
Controlled
Realistic
AWS Fault Injection
Service

Game day observability
Replicate production observability into test environments
Verify behavior of metrics, alarms, dashboards, logs
Add new instrumentation, metrics, alarms afterward

Prevent future issues: Game days
Regularly perform controlled experiments of failure
modes by using the AWS Fault Injection Service

Prevent future issues: Game days
Regularly perform controlled experiments of failure
modes by using the AWS Fault Injection Service
Use the same observability tools during experiments
as you do in production by including things like alarm
and dashboard definition in your infrastructure as code
AWS Cloud Development Kit
(AWS CDK)

Diagnose
issues
Recap
Find patterns in
high-cardinality metrics
Use dimensionality
to compute the right metric
Navigate distributed systems
using tracing

Recap
Uncover
hidden issues
Aggregate metrics in customer-
oriented ways

Recap
Auto scale and track the
utilization of everything
Monitor game days just like
production
Prevent
future issues

Further reading: Amazon Builders’ Library
https://aws.amazon.com/builders-library/

Thank you!
Please complete the session
survey in the mobile app
Thank you!
Please complete the session
survey in the mobile app
David Yanacek
david-yanacek
@dyanacek
@dyanacek@hachyderm.io
dyanacek.bsky.social

Building observability to increase resiliency.pdf

Recommended

Recommended

More Related Content

Similar to Building observability to increase resiliency.pdf

Similar to Building observability to increase resiliency.pdf (20)

Recently uploaded

Recently uploaded (20)

Building observability to increase resiliency.pdf