PatternsResiliency_DevDays2019.pdf

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
N O R D I C S

Resiliency and Availability Design
Patterns for the Cloud
Adrian Hornsby
Sr. Technical Evangelist
Amazon Web Services

Distributed Systems
are hard

Complex systems
Amazon Twitter Netflix

Partial failure mode

Resiliency: Ability for a system to handle and
eventually recover from unexpected conditions

How do we build resilient software
systems?

…

“Quality is not an act, it is a habit”
Aristotle, some time around 350BC

https://aws.amazon.com/wellarchitected

Let’s talk about Multi-AZ

Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby

Region
standby
X

Region
DB Instance
X

• Enables fault-tolerant applications
• AWS regional services designed to
withstand AZ failures
• Leveraged by most AWS regional
services (S3, DynamoDB, ELBs, …)
Region
standby

Region
Application

Cascading Failures

Better to react without reacting

But Why?

3 AZ’s is better than 2

Region
Application
Requires 8 Instances

Region
Application
Requires 6 Instances

Let’s talk about auto scaling

Auto-Scaling
FixedVariable

Availability zone 1
Auto Scaling group
AWS Region
Availability zone 2
Auto-scaling for self-healing
X

Let’s talk about decoupling and async

Process A Process B Process A Process B
Synchronous Asynchronous
Waiting
Working
Continues
get or fetch resultGet result
Decoupling with async pattern

API: {DO foo}
PUT JOB: {JobID: 0001, Task: DO foo}
API: {JobID: 0001}
GET JOB: {JobID: 0001, Task: DO foo}
{JobID: 0001, Result: bar}
Cache node
Worker
Instance
Worker
Instance
Queue/Streaming
API
Instance
API
Instance
API
Instance

Push Notification
User
Worker
Instance
Worker
Instance
API
Instance
API
Instance
Cache node
Fetch results
API
Instance
Queue/Streaming

Degrade & prioritize traffic
with queues
Worker
Instance
Worker
Instance
API
Instance
API
Instance
API
Instance
HighPriorityQueue
LowPriorityQueue

Let’s talk about timeouts, backoff &
retries!

Users
App
DB
Conn
Pool
INSERT
INSERT
INSERT
INSERT
What happens if the DB “slows down”?
Timeout client side Timeout backend side ??

User 1
App
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = default = Infinite
Retry INSERT
Retry INSERT
ERROR: Failed to get connection from pool
Retry

https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html

Set the timeouts!

How else could we have prevented the error?
User 1
DB
Conn
Pool
INSERT
Retry INSERT
Retry INSERT
Retry
ERROR: Failed to get connection from pool

User 1
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = 10s
Wait 2s before Retry
INSERT
INSERT
Backing off between retries
Releasing connectionsBackoff

No jitter With jitter
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Simple Exponential Backoff is not enough: Add Jitter

Example: add jitter 0-1000ms
def get_item(self, url, n=1):
MAX_TRIES = 12
try:
res = requests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0))
return self.get_item(url, n)
else:
return res

Idempotent operation
No additional effect if it is called more than
once with the same input parameters.

Circuit Breaker
• Wrap a protected function
call in a circuit breaker
object, which monitors for
failures.
• If failures reach a certain
threshold, the circuit
breaker trips.
Producer Circuit Breaker Consumer
Connection
Monitoring
Timeouts
Breaking Circuit

https://spring.io/guides/gs/circuit-breaker/

Let’s talk about health checking!

Auto Scaling group
Service A
Availability zone 1
Auto Scaling group
AWS Region
Service A
Availability zone 2
Service BService B
database Email
Probing for health
Cluster

Shallow health check
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes

Deep health check
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes
Are you healthy?
yes
yes
yes
yes

Deep health check
Instance
Cache node
Email
database
Cluster
Are you healthy?
no
Are you healthy?
no
yes
yes
yes

Prioritize shallow health checks during
hard times.
Cache.

Let’s talk about load shedding.

Cheaply reject excess work

Be careful when selecting the right
metric

Don’t be overly optimistic and take on more than you can.
Find an operational metric to reject what you cannot take in.
Favor cached and static content
Prioritize ELB health check (shallow) pings
In an overload situation you have precious resources, do not
let any of it go to waste.
Load Shedding

Service Degradation & Fallbacks

Let’s talk about databases!

Database Federation
Users
DB
Products
DB
Instance InstanceInstance
DB Instance
DB instance read
replica
DB Instance
DB instance read
replica

Database Sharding User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica

Read / Write separation
DB Instance DB instance read
replica
DB instance read
replica
DB instance read
replica
Supports degradation through Read-Only mode

Transient state does not
belong in the database.

Let’s talk about shuffle sharding.

X X X X X X XX
♤♡♢ ⚀ ⚁ ⚂ ⚃♧♢

Measure for this: blast radius

Cell-based architecture
XX
♤♡♢ ⚀ ⚁ ⚂ ⚃♧♢

Shuffle sharding
XX
♤♡♢ ⚀ ⚁⚂ ⚃♡ ♤ ♧♢ ⚀⚂♧ ⚁⚃♢ ♢
♡ ♧♢

Shuffle sharding
Nodes = 8
Shard size = 2
Combinations = 28
Overlap % customers
0 53.6%
1 42.8%
2 3.6%

Shuffle sharding
Nodes = 100
Shard size = 5
Combinations = 75 million!
Overlap % customers
0 77%
1 21%
2 1.8%
3 0.06%
4 0.0006%
5 0.0000013%

Shuffle sharding

Let’s talk about chaos!

Fire Drills

GameDay at Amazon
Creating Resiliency Through Destruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

Chaos engineering
https://github.com/Netflix/SimianArmy

“Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the system’s
capability to withstand turbulent conditions in
production.”
http://principlesofchaos.org

Failure injection
• Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks
• “Paul” attack

Plan for the worst, prepare for the
unexpected.

Thank you!
Adrian Hornsby
@adhorn
https://medium.com/@adhorn
https://speakerdeck.com/adhorn/patterns-for-building-resilient-software-systems-2019

PatternsResiliency_DevDays2019.pdf

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to PatternsResiliency_DevDays2019.pdf

Similar to PatternsResiliency_DevDays2019.pdf (20)

More from Amazon Web Services

More from Amazon Web Services (20)

PatternsResiliency_DevDays2019.pdf