The talks covers various design patterns to make services more resilient. For more detail, please also follow https://twitter.com/adhorn who created the deck.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
V I E N N A
24.10.19
Resiliency and availability design
patterns for the cloud
Cobus Bernard
Senior Technical Evangelist
Amazon Web Services
@cobusbernard
cobusbernard
cobusbernard
B A R 3

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Can you guess whatwillhappen?

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distributed Systems are hard

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failures areagivenand
everythingwilleventuallyfail
over time.
Werner Vogels
CTO – Amazon.com
“ “

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resiliency:Abilityfor asystemtohandle and
eventuallyrecover from unexpected conditions
conditions

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partialfailure mode

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why do we build resilient software systems?

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thecostof downtime
Annual Fortune
1000 application
downtime costs
(IDC)
$1.25 to
$2.5B
Average cost of a
data breach
(Ponemon
Institute)
$3.6M
Cost/hr of a
critical
application
failure (IDC)
$500K
to $1M
Average cost/hr
of downtime
(Ponemon
Institute)
$474K
Average cost per
lost or stolen
record (Ponemon
Institute)
$141

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do we build resilient software systems?

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
People
Application
Network & Data
Infrastructure

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk aboutAvailability

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Systemavailability
Availability =
Normal Operation Time
Total Time
MTBF**
MTBF** + MTTR*
=
* Mean Time To Repair (MTTR)
**Mean Time Between Failure (MTBF)

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reading homework

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availabilityinparallel
A = 1 – (1 – Ax)2
Part X
Part X

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availabilityinparallel
Component Availability Downtime
X 99% (2-nines) 3 days 15 hours
Two X in parallel 99.99% (4-nines) 52 minutes
Three X in parallel 99.9999% (6-nines) 31 seconds

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Componentredundancyincreases availability
significantly!

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Global Infrastructure
• 22 Regions with 69 Availability Zones
• 3 Regions coming soon: Cape Town
Jakarta and Milan
• 100Gbps redundant network
• 99.99% SLA

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fully-scaledAvailabilityZone

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Highlyredundant regional network

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region and availabilityzones
Region
Availability zone a Availability zone b Availability zone c
data center
data center
data center
1 or more data centers per AZ
2 or more AZs per region (new regions min 3)
data center
data center
data center
data center
data center
data center

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How about a global architecture?

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Once upon a time …
Origin

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
And Now …
Origin
~300ms

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Origin
Improve latency for end-users
Origin

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Improve availabilityand disasterrecovery
Applications in
US West
Applications in
US East
Users from
San Francisco
Users from
New York
Service 1
Service 2
Service 3
Service 4
Service 1
Service 2
Service 3
Service 4

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
So should we go for a global architecture?

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Perfect your regional architecture first!

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about Multi-AZ

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
new master
Elastic Load
Balancing (ELB)

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
• Enables fault-tolerant applications
• AWS regional services designed to
withstand AZ failures
• Leveraged by AWS regional
services such as Amazon S3,
Amazon DynamoDB, Amazon
Aurora, Amazon ELBs, etc.
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about auto scaling

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto-Scaling
FixedVariable

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability zone 1
Auto Scaling group
AWS Region
Availability zone 2
Auto-scaling for self-healing
Elastic Load
Balancing (ELB)
X

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about the AWS responsibility
models

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWSoperationalresponsibilitymodels
On-Premises Cloud
Less More
Compute Virtual Machine
EC2 Elastic Beanstalk AWS LambdaFargate
Databases MySQL MySQL on EC2
RDS MySQL RDS Aurora Aurora Serverless DynamoDB
Storage Storage
S3
Messaging ESBs
Amazon MQ Kinesis SQS / SNS
Analytics
Hadoop Hadoop on EC2 EMR Elasticsearch Service Athena
Firehose

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about databases

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common resiliencyissueswithDatabases??
REPLICATION BACKUPSSCALING

41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB: purpose-built for thecloud
Fast and flexible key value database service for any scale
Comprehensive
security
Encrypts all data by default
and fully integrates with
AWS Identity and Access
Management for robust
security
Performance at scale
Consistent, single-digit
millisecond response times at any
scale; build applications with
virtually unlimited throughput
Global database for
global users and apps
Build global applications with fast
access to local data by easily
replicating tables across multiple
AWS Regions
Serverless
No hardware provisioning,
software patching, or upgrades;
scales up or down
automatically; continuously
backs up your data

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
AZ 1
AZ 2
AZ 3
Network
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
Storage
Node
Leader
PutItem

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
AZ 1
AZ 2
AZ 3
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
Storage
Node
Leader
GetItem
Network

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon.com, Nike, Netflix, Duolingo, Lyft, Airbnb, Samsung,
Toyota, and Capital One depend on the scale and performance of
DynamoDB to support their workloads.
10 trillion requests
per day
20 million
requests per second

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AmazonAurora: purpose-built for thecloud
MySQL and PostgreSQL compatible relational database
Availability
and durability
Fault-tolerant, self-healing
storage; six copies of data
across three AZs; continuous
backup to S3
Fully managed
Managed by RDS:
no hardware provisioning,
software patching, setup,
configuration, or backups
Highly secure
Network isolation,
encryption at rest/transit
Performance
and scalability
5x throughput of standard MySQL
and 3x of standard PostgreSQL;
scale-out up to
15 read replicas

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BINLOG DATA DOUBLE-WRITELOG FRM FILES
MYSQL WITH REPLICA
EBS mirrorEBS mirror
AZ 1 AZ 2
Amazon S3
EBS
Amazon Elastic
Block Store (EBS)
Primary
Instance
Replica
Instance
1
2
3
4
5
AZ 1 AZ 3
Primary
Instance
AZ 2
Replica
Instance
Replica
Instance
AMAZON AURORA
TYPE OF WRITE
AmazonAurora: distributed architecture

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Capital One, GE, Dow Jones, Netflix, Verizon, Expedia and Amazon depend on
the scale and performance of Aurora to support their workloads.
5X the
throughput of
standard
MySQL
15 low
latency read
replicas
Failover in
less than 30
seconds
3X the
throughput of
standard
PostgreSQL

49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Purpose-built databases
Relational Key-value Document In-memory Graph Time-series Ledger
DynamoDB NeptuneAmazon RDS
Aurora CommercialCommunity
Timestream QLDBElastiCacheDocumentDB

50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Read-Writeseparation
Master Read Replica Read Replica Read Replica
Instance InstanceInstance

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DatabaseFederation
Users
DB
Products
DB
Master
(Read) Replica
Master
(Read) Replica
Instance InstanceInstance

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DatabaseSharding User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
Master
(Read) Replica
Master
(Read) Replica
Master
(Read) Replica
Instance InstanceInstance

53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about backups

54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Backup service

56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PreventingAccidentalTableDeletion
https://aws.amazon.com/blogs/database/preventing-accidental-table-deletion-in-dynamodb/

57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PreventingAccidentalTableDeletion (sql)

58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Practice and testrecoveryfrom your backups!!

59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about timeouts, backoff &
retries!

60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users
App
DB
Conn
Pool
INSERT
INSERT
INSERT
INSERT
What happens if the DB “slows down”?
Timeout client side Timeout backend side ??

61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
App
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = default = Infinite
Retry INSERT
Retry INSERT
ERROR: Failed to get connection from pool
Retry

62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.timeout

63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html

65. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
@timeout_decorator.timeout(5, timeout_exception=StopIteration)
def timed_get(url):
return requests.get(url)
https://pypi.org/project/timeout-decorator/

66. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Set thetimeouts!

67. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How elsecould wehaveprevented theerror?
User 1
DB
Conn
Pool
INSERT
Retry INSERT
Retry INSERT
Retry
ERROR: Failed to get connection from pool

68. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = 10s
Wait 2s before Retry
INSERT
INSERT
Wait 4s before Retry
Wait 8s before Retry
Wait 16s before Retry
Backing off betweenretries
Releasing connectionsBackoff

69. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
No jitter With jitter
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
SimpleExponentialBackoffisnotenough:AddJitter

70. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: add jitter0-1000ms
def get_item(self, url, n=1):
MAX_TRIES = 12
try:
res = requests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0))
return self.get_item(url, n)
else:
return res

71. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Idempotent operation
No additional effect if it is called more than
once with the same input parameters.

72. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about health checking!

73. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto Scaling group
Service A
Availability zone 1
Auto Scaling group
AWS Region
Service A
Availability zone 2
Service BService B
database Email
Probing for health
Cluster

74. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shallowhealthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes

75. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shallowhealthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes

76. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep healthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes
Are you healthy?
yes
yes
yes
yes

77. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep healthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
no
Are you healthy?
no
yes
yes
yes

78. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prioritize shallow health checks during
hard times.
Cache.

79. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about load shedding.

80. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

81. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

82. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cheaply reject excess work

83. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

84. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Be careful when selecting the right metric

85. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Don’tbeoverlyoptimisticandtakeonmorethanyoucan.
Findanoperationalmetrictorejectwhatyoucannottakein.
Favorcachedandstaticcontent
PrioritizeELBhealthcheck(shallow)pings
Inanoverloadsituationyouhavepreciousresources,donotletany
ofitgotowaste.
Load Shedding

86. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk aboutresiliency (chaos) engineering

87. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fire Drills

88. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GameDay atAmazon
CreatingResiliencyThroughDestruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

89. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaosengineering
https://github.com/Netflix/SimianArmy

90. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Chaos Engineeringis the discipline of
experimentingon a distributedsystem
in orderto buildconfidence in the system’s
capabilitytowithstand turbulentconditionsin
production.”
http://principlesofchaos.org

91. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failureinjection
• Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks
• “Paul” attack
https://www.gremlin.comhttps://github.com/Netflix/SimianArmy https://chaostoolkit.org

92. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
STEADY
STATE
HYPOTHESIS
RUN
EXPERIMENT
VERIFY
FIX!
PhasesofChaosEngineering

93. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://aws.amazon.com/wellarchitected

94. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
@cobusbernard
cobusbernard
cobusbernard