Chaos Engineering: Why Breaking Things Should Be Practiced - AWS Developer Workshop at Web Summit 2018 Ever wondered how companies delivering global services like Amazon or Netflix are architecting and testing their software systems? If you are curious and want to learn how they do it - this session is for you! With the rise of micro-services and large-scale distributed architectures, software systems have grow increasingly complex and hard to understand. Adding to that complexity, the velocity of software delivery has also dramatically increased, resulting in failures being harder to predict and contain. While the cloud allows for high availability, redundancy and fault-tolerance, no single component can guarantee 100% uptime. Therefore, we have to understand availability but especially learn how to design architectures with failure in mind. And since failures have become more and more chaotic in nature, we must turn to chaos engineering in order to identify failures before they become outages. In this talk, I will deep dive into availability, reliability and large-scale architectures and make an introduction to chaos engineering, a discipline that promotes breaking things on purpose in order to learn how to build more resilient systems. Speaker: Adrian Hornsby - Technical Evangelist, AWS

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Adrian Hornsby, Cloud Architecture Evangelist
@adhorn
Chaos Engineering:
Why Breaking Things Should Be Practiced.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://xkcd.com/1428/

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Complex systems
Amazon Twitter Netflix

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partial failure mode

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resiliency: Ability for a system to handle and
eventually recover from unexpected conditions

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
People
Application
Network & Data
Infrastructure

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building confidence through testing
Unit testing of components:
• Tested in isolation to ensure function meets expectations.
Functional testing of integrations:
• Each execution path tested to assure expected results.
Is it enough???

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GameDay at Amazon
Creating Resiliency Through Destruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaos engineering
https://github.com/Netflix/SimianArmy

10. Failure injection
• Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks
• “Paul” attack
https://www.gremlin.comhttps://github.com/Netflix/SimianArmy https://chaostoolkit.org

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Chaos Engineering is the discipline of
experimenting on a distributed system
in order to build confidence in the system’s
capability to withstand turbulent conditions in
production.”
http://principlesofchaos.org

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Break your systems on purpose.
Find out their weaknesses and fix
them before they break when
least expected.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Chaos doesn’t cause problems.
It reveals them.”
Nora Jones
Senior Chaos Engineer, Netflix

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://bit.ly/2uKOJMQ

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Phases of Chaos Engineering

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Steady
State
Hypothesis
Design & Run
Experiment
Verify
& Learn
Fix

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Steady
State

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is steady state?
• ”normal” behavior of your system
https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is steady state?
• ”normal” behavior of your system
• Business Metric
https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Business metrics at work
Amazon: 100 ms of extra load time caused a 1% drop in sales (Greg Linden).
Google: 500 ms of extra load time caused 20% fewer searches (Marissa Mayer).
Yahoo!: 400 ms of extra load time caused a 5–9% increase in the number
of people who clicked “back” before the page even loaded (Nicole Sullivan).

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hypothesis

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What if…?
“What if this load balancer breaks?”
“What if Redis becomes slow?”
“What if a host on Cassandra goes away?”
”What if latency increases by 300ms?”
”What if the database stops?”
Make it everyone’s problem!

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Disclaimer!
Don’t make an hypothesis that you know
will break you!

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Design & Run
Experiment

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Designing experiment
• Pick hypothesis
• Scope the experiment
• Identify metrics
• Notify the organization

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Rules of thumbs
• Start with very small
• As close as possible to production
• Minimize the blast radius.
• Have an emergency STOP!

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Running Chaos Experiment
Users
Canary deployment
Normal Version
99%
Users
1%
Users
Start with ..

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Verify
& Learn

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Quantifying the result of the experiment
• Time to detect?
• Time for notification? And escalation?
• Time to public notification?
• Time for graceful degradation to kick-in?
• Time for self healing to happen?
• Time to recovery – partial and full?
• Time to all-clear and stable?

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DON’T blame that one person …

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PostMortems – COE (Correction of Errors)
The 5 WHYs
Outage
Because
of …
Because
of …
Because
of …
Because
of …
NOT
ENOUGH

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
More questions to ask
• Can you clarify if there were any preceding events?
• Why would they believe acting in this way was the best course of action to
deliver the desired outcome?
• Is there another failure mode that could present here?
• What decisions or events prior to this made this work before?
• Why stop there – are there places to dig deeper that could shine a light more
on this?
• Did others step in to help, to advise, or to intercede?

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Rules to remember!
1. Failure requires multiple faults
2. There is no isolated ‘cause’ of an accident.
3. There are multiple contributors to accidents.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fix

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fix

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Patterns for Resilient Architecture

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability in parallel
A = 1 – (1 – Ax)2
Part X
Part X

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability in parallel
Component Availability Downtime
X 99% (2-nines) 3 days 15 hours
Two X in parallel 99.99% (4-nines) 52 minutes
Three X in parallel 99.9999% (6-nines) 31 seconds

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Application

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto Scaling group
Service A
Availability zone 1
Auto Scaling group
AWS Region
Service A
Availability zone 2
Service BService B
Stateless Services

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto-Scaling • Compute efficiency
• Node failure
• Traffic spikes
• Performance bugs

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto Scaling group
Service A
Availability zone 1
Auto Scaling group
AWS Region
Service A
Availability zone 2
Service BService B
Auto-scaling

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Decoupling with async pattern
Listener
Pub-Sub
Queue
Queue
A
A
B
B

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
API: {DO foo}
PUT JOB: {JobID: 0001, Task: DO foo}
API: {JobID: 0001}
GET JOB: {JobID: 0001, Task: DO foo}
{JobID: 0001, Result: bar}
Cache node
Worker
Instance
Worker
Instance
Queue
API
Instance
API
Instance
API
Instance

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Push Notification
User
Worker
Instance
Worker
Instance
Queue
API
Instance
API
Instance
Cache node
Fetch results
API
Instance

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Degrade & prioritize traffic
with queues
Worker
Instance
Worker
Instance
API
Instance
API
Instance
API
Instance
HighPriorityQueue
LowPriorityQueue

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Read / Write Sharding
DB Instance DB instance read
replica
DB instance read
replica
DB instance read
replica
Instance InstanceInstance

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database Federation
Users
DB
Products
DB
Instance InstanceInstance
DB Instance
DB instance read
replica
DB Instance
DB instance read
replica

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database Sharding User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
Instance InstanceInstance
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica
DB Instance
DB instance
read replica

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transient state does not
belong in the database.

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cascading Failures

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about timeouts & retries!

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users
App
DB
Conn
Pool
INSERT
INSERT
INSERT
INSERT
What happens if the DB “slows down”?
Timeout client side Timeout backend side ??

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
App
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = Not implemented
Retry INSERT
Retry INSERT
ERROR: Failed to get connection from pool
Retry

56. http://docs.python-requests.org/en/master/user/advanced/#timeouts

57. http://docs.python-requests.org/en/master/user/advanced/#timeouts

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
@timeout_decorator.timeout(5, timeout_exception=StopIteration)
def timed_get(url):
return requests.get(url)
https://pypi.org/project/timeout-decorator/

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Timeouts

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How else could we have prevented the error?
User 1
DB
Conn
Pool
INSERT
Retry INSERT
Retry INSERT
Retry
ERROR: Failed to get connection from pool

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = 10s
Wait 2s before Retry
INSERT
INSERT
Wait 4s before Retry
Wait 8s before Retry
Wait 16s before Retry
Backing off between retries
Releasing connectionsBackoff

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
No jitter With jitter
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Simple Exponential Backoff is not enough: Add Jitter

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: add jitter 0-1000ms
def get_item(self, url, n=1):
MAX_TRIES = 12
try:
res = requests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0))
return self.get_item(url, n)
else:
return res

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
@backoff.on_exception(backoff.full_jitter, max_time=60)
def poll_for_message(queue):
return queue.get()
https://pypi.org/project/backoff/
As of version 1.2, the default jitter function backoff.full_jitter implements the ‘Full Jitter’ algorithm as defined in the
AWS Architecture Blog’s Exponential Backoff And Jitter post.

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Idempotent operation
No additional effect if it is called more than
once with the same input parameters.

66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service Degradation & Fallbacks

67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Circuit Breaker
• Wrap a protected function
call in a circuit breaker
object, which monitors for
failures.
• If failures reach a certain
threshold, the circuit
breaker trips.
Producer Circuit Breaker Consumer
Connection
Monitoring
Timeouts
Breaking Circuit

68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Non-blocking UI
https://medium.com/@sophie_paxtonUX/stop-getting-in-my-way-non-blocking-ux-5cbbfe0f0158

69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fire Drills

70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Big challenges to chaos engineering
Mostly Cultural
• no time or flexibility to simulate disasters.
• teams already spending all of its time fixing things.
• can be very political.
• might force deep conversations.
• deeply invested in a specific technical roadmap (micro-services) that
chaos engineering tests show is not as resilient to failures as originally
predicted.

71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Changing culture takes time!
Be patient…

73. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Thanks you!
@adhorn
https://medium.com/@adhorn

74. Did We Scan Your Badge?
Remember to opt-in to AWS
communications and you will receive a
post-event email with a link to:
• AWS Developer Workshop Slides
• $200 in AWS Credits