We have traditionally built robust software systems by trying to avoid mistakes and by dodging failures when they occur in production or by testing parts of the system in isolation from one another. Modern methods and techniques take a very different approach based on resiliency, which promotes embracing failure instead of trying to avoid it. Resilient architectures enhance observability, leverage well-known patterns such as graceful degradation, timeouts and circuit breakers but also new patterns like cell-based architecture and shuffle sharding. In this session, will review the most useful patterns for building resilient software systems and especially show the audience how they can benefit from the patterns.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STEVEN BRYEN | AWS TECHNICAL & DEVELOPER EVANGELISM | @steven_bryen
sbryen@amazon.com
LONDON – MARCH 2019
Resiliency & Availability Design Patterns
for the Cloud

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RELIABILITY AND RESILIENCY

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
99.99%

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Everything fails all the
time.
Werner Vogels
CTO – Amazon.com
“ “

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reliability:
handles the “known, unknowns”
Resiliency:
handles the “unknown, unknowns”

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How do we build resilient software
systems?

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“Quality is not an act, it is a habit”
Aristotle, some time around
350BC

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Distributed Systems
are hard

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Complex systems
Amazon Twitter Netflix

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk about Multi-AZ for
Maximum availability

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Region
Availability zone a Availability zone b Availability zone c
Application

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Region
Availability zone a Availability zone b Availability zone c
Application

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Better to react without reacting

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Region
Availability zone a Availability zone b Availability zone c
Application

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
But Why?

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3 AZ’s is better than 2

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Region
Availability zone a Availability zone b Availability zone c
Application
Requires 8 Instances

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Region
Availability zone a Availability zone b Availability zone c
Application
Requires 6 Instances

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://aws.amazon.com/wellarchitected

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk about auto scaling

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Auto-Scaling
FixedVariable

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Availability zone 1
Auto Scaling group
AWS Region
Availability zone 2
Auto-scaling for self-healing
• Set min > 0
X

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk about decoupling and async

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Process A Process B Process A Process B
Synchronous Asynchronous
Waiting
Working
Continues
get or fetch resultGet result
Pattern 5: Decoupling with async pattern

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
API: {DO foo}
PUT JOB: {JobID: 0001, Task: DO foo}
API: {JobID: 0001}
GET JOB: {JobID: 0001, Task: DO foo}
{JobID: 0001, Result: bar}
Cache node
Worker
Instance
Worker
Instance
Queue/Streaming
API
Instance
API
Instance
API
Instance

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Push Notification
User
Worker
Instance
Worker
Instance
API
Instance
API
Instance
Cache node
Fetch results
API
Instance
Queue/Streaming

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Degrade & prioritize traffic
with queues
Worker
Instance
Worker
Instance
API
Instance
API
Instance
API
Instance
HighPriorityQueue
LowPriorityQueue

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk about timeouts, backoff &
retries!

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Users
App
DB
Conn
Pool
INSERT
INSERT
INSERT
INSERT
What happens if the DB “slows down”?
Timeout client side Timeout backend side ??

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
User 1
App
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = Not implemented
Retry INSERT
Retry INSERT
ERROR: Failed to get connection from pool
Retry

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.http://docs.python-requests.org/en/master/user/advanced/#timeouts

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
http://docs.python-requests.org/en/master/user/advanced/#timeouts

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How else could we have prevented the error?
User 1
DB
Conn
Pool
INSERT
Retry INSERT
Retry INSERT
Retry
ERROR: Failed to get connection from pool

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Wait 16s before Retry
Wait 8s before Retry
Wait 4s before Retry
Wait 2s before Retry
User 1
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = 10s
INSERT
INSERT
Exponential Backoff?
Releasing connectionsBackoff

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No jitter With jitter
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Simple Exponential Backoff is not enough: Add Jitter

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example: add jitter 0-1000ms
def get_item(self, url, n=1):
MAX_TRIES = 12
try:
res = requests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0))
return self.get_item(url, n)
else:
return res

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk about databases.

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Federation
Users
DB
Product
s DB
Instance InstanceInstance
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Sharding User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
Instance InstanceInstance
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Read / Write separation
DB Instance DB instance
read replica
DB instance
read replica
DB instance
read replica
Instance InstanceInstance
Supports degradation through Read-Only mode

44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk about shuffle sharding.

45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

47. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cascading Failures

49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Measure for this: blast radius

50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1 12 23 34 4 556 677 88

52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No of Nodes: 8
Shard Size: 2
Blast Radius
Overlap % of Customers Impacted
0 53.6%
1 42.8%
2 3.6%

53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No of Nodes: 100
Shard Size: 5
Blast Radius
Overlap % of Customers Impacted
0 77%
1 21%
2 1.8%
3 0.06%
4 0.0006%
5 0.0000013%

54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Needs a client that retries or is fault tolerant
Works for servers, queues or other resources
Needs a routing mechanism such as per customer
DNS Names
Shuffle Sharding

55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s talk about chaos!

56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GameDay at Amazon
Creating Resiliency Through Destruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chaos engineering
https://github.com/Netflix/SimianArmy

58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the system’s
capability to withstand turbulent conditions
in production.”
http://principlesofchaos.org

59. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Failure injection
Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks
• “Paul” attack

60. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
@adhorn

61. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
@adhorn

62. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Plan for the worst, prepare for the
unexpected.

63. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STEVEN BRYEN | AWS TECHNICAL & DEVELOPER EVANGELISM | @steven_bryen
sbryen@amazon.com
LONDON – MARCH 2019
Thank You!