1. Applying Principles of Chaos Engineering to Serverless
Yan Cui @theburningmonk

2. Agenda
▪What is chaos engineering?
▪New challenges with serverless
▪Applying latency injection to serverless
▪Applying error injection to serverless

3. After the talk
▪Slides will be available on slideshare
▪Find links to slides and video at https://theburningmonk.com/
oreillysa2018

4. What is chaos engineering?

5. Chaos Engineering is the discipline of experimenting on a distributed system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.
- principlesofchaos.org

6. Smallpox
Earliest evidence of disease in 3rd Century BC Egyptian Mummy.
est. 400K deaths per year in 18th Century Europe.

7. History of vaccination
First vaccine was developed in
1798 by Edward Jenner.
https://en.wikipedia.org/wiki/Edward_Jenner

8. History of vaccination
First vaccine was developed in
1798 by Edward Jenner.
WHO certified global eradication
in 1980.
https://en.wikipedia.org/wiki/Edward_Jenner

9. https://en.wikipedia.org/wiki/Vaccine
History of vaccination

10. History of vaccination
Vaccination is the most effective method to prevent infectious diseases.

11. History of vaccination
Vaccination is the most effective method to prevent infectious diseases.
Vaccines stimulate the immune system to recognise and destroy the
disease before contracting it for real.

12. Chaos engineering
Use controlled experiments to inject failures into our system.

13. Chaos engineering
Use controlled experiments to inject failures into our system.
Help us learn about our system’s behaviour and uncover unknown
failure modes, before they manifest like wildfire in production.

14. Chaos engineering
Use controlled experiments to inject failures into our system.
Help us learn about our system’s behaviour and uncover unknown
failure modes, before they manifest like wildfire in production.
Lets us build confidence in its ability to withstand turbulent conditions.

15. Chaos Engineering is the vaccine to frailties in modern software.

16. * https://bit.ly/production-ready-serverless
** https://theburningmonk.com
Who am I?
Principal Engineer at DAZN.
AWS Serverless Hero.
Author of Production-Ready Serverless* course by Manning.
Blogger**, speaker.

19. https://www.ft.com/content/07d375ee-6ee5-11e8-92d3-6c13e5c92914

20. https://www.theguardian.com/media/2018/may/14/streaming-service-dazn-netflix-sport-us-boxing-eddie-hearn

21. About DAZN
Available in 7 countries - Austria, Switzerland, Germany,
Japan, Canada, Italy and USA.
Available on 30+ platforms.

22. About DAZN
Around 1,000,000 concurrent viewers at peak.

23. Chaos engineering has an image problem

24. Chaos engineering has an image problem

25. Chaos engineering has an image problem
Too much emphasis is on breaking things.

26. Chaos engineering has an image problem
Too much emphasis is on breaking things.
Easy to conflate the action of injecting failures with the payback.

27. Why did you break
production?

28. Because I can!

29. Chaos engineering has an image problem
The goal is to learn about the system and build confidence.

30. Chaos engineering has an image problem
The goal is to learn about the system and build confidence.
The goal is not to break things.

31. Chaos in practice
4 steps to start running chaos
experiments yourself.

32. STEP 1. Define “steady state”
aka. what does normal, working
condition looks like?

33. this is not a
steady state

34. Hypothesise steady state will
continue in both control group
& the experiment group
ie. you should have a reasonable degree of confidence the
system would handle the failure before you proceed with
the experiment
STEP 2.

35. Chaos in practice
Explore unknown unknowns away from production.

36. Chaos in practice
Explore unknown unknowns away from production.
Experiments that graduate to production should be
carefully considered and planned.
You should have reasonable confidence in the system
before running experiments in production.

37. Chaos in practice
Treat production with the care it deserves.
The goal is not to break things.

38. Chaos in practice
If you know the system would break and you did it anyway,
then it’s not a chaos experiment!
It’s called being irresponsible.

39. STEP 3. Inject realistic failures
e.g. server crash, network error, HD
malfunction, etc.

40. https://github.com/Netflix/SimianArmy http://oreil.ly/2tZU1Sn

42. STEP 4. Disprove hypothesis
i.e. look for difference in steady state

43. Chaos in practice
Look for evidence that steady state was impacted by the injected failure.

44. Chaos in practice
Look for evidence that steady state was impacted by the injected failure.
Address weaknesses before failures happen for real.

45. Containment
Experiments needs to be controlled.
The goal is not to break things.

46. Containment
Ensure everyone knows what you are doing.
Don’t surprise your teammates.

47. Containment
Run experiments during office hours.

48. Containment
Run experiments during office hours.
Avoid important dates.

49. Containment
Make the smallest change necessary to prove or disprove hypothesis.

50. Containment
Make the smallest change necessary to prove or disprove hypothesis.
Have a rollback plan.
Stop the experiment right away if things start to go wrong.

51. Containment
Don’t start in production.
Can learn a lot by running experiments in staging.

52. by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

53. New challenges with serverless

54. chaos monkey kills an EC2 instance
latency monkey induces
artificial delay in APIs
chaos gorilla kills an AWS
Availability Zone
chaos kong kills an entire
AWS region

56. Serverless challenges
There are no servers that you can access and kill.

59. There are more inherent chaos and complexity in
a serverless architecture.

60. Serverless challenges
Smaller unit of deployment, but a lot more of them.

61. serverful
serverlessServerless challenges

62. Serverless challenges
Every function needs to be correctly configured and secured.

63. Kinesis
?
SNS
CloudWatch
Events
CloudWatch
LogsIoT
Core
DynamoDB
S3 SES
Serverless challenges
Lambda Lambda

64. Serverless challenges
A lot of managed, intermediate services.
Each with its own set of failure modes.

65. Serverless challenges
Unknown failure modes in the infrastructure we don’t control.

66. Serverless challenges
Unknown failure modes in the infrastructure we don’t control.
Often there’s little we can do when an outage occurs in the platform.

67. Common weaknesses

68. Common weaknesses
Improperly tuned timeouts.

69. Common weaknesses
Missing error handling.

70. Common weaknesses
Missing fallback.

71. Common weaknesses
Missing regional failover.

72. Latency injection with serverless

73. STEP 1. Define “steady state”
aka. what does normal, working
condition looks like?

74. Defining steady state
What metrics do you use?

75. Defining steady state
What metrics do you use?
p95/p99 latencies, error count, backlog size, yield*, harvest**
* percentage of requests completed
** completeness of the returned response

76. Hypothesise steady state will
continue in both control group
& the experiment group
ie. you should have a reasonable degree of confidence the
system would handle the failure before you proceed with
the experiment
STEP 2.

77. API Gateway
Serverless considerations

78. Serverless considerations
Consider the effect of cold starts.
How does it affect your strategy
for handling slow responses.

79. Request timeouts
Strategy should:

80. Request timeouts
Strategy should:
1. Give requests the best chance to succeed

81. Request timeouts
Strategy should:
1. Give requests the best chance to succeed
2. Do not allow slow response to timeout the caller function

82. Request timeouts
Finding the right timeout value is tricky.

83. Request timeouts
Finding the right timeout value is tricky.
Too short : requests not given the best chance to succeed.

84. Request timeouts
Finding the right timeout value is tricky.
Too short : requests not given the best chance to succeed.
Too long : risk timing out the calling function.

85. Request timeouts
Finding the right timeout value is tricky.
Too short : requests not given the best chance to succeed.
Too long : risk timing out the calling function.
Even more complicated when you have multiple integration points.

86. Approach 1: split invocation time equally
(e.g. 3 requests, 6s function timeout = 2s timeout per request)

87. Approach 2: every request is given nearly all the invocation time
(e.g. 3 requests, 6s function timeout = 5s timeout per request)

88. Request timeouts
Proposal: set request timeouts dynamically based on invocation time left

89. Request timeouts

90. Set timeout based on remaining invocation time

91. Set timeout based on remaining invocation time

92. Recovery steps
Log the timeout with as much context as possible.
The API, timeout value, correlation IDs, request object, etc.

93. Recovery steps
Log the timeout with as much context as possible.
The API, timeout value, correlation IDs, request object, etc.
Record custom metrics.

94. Recovery steps
Log the timeout with as much context as possible.
The API, timeout value, correlation IDs, request object, etc.
Record custom metrics.
Use fallbacks.

96. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

98. Recovery steps
Be mindful when you sacrifice precision for availability.
User experience is the king.

99. STEP 3. Inject realistic failures
e.g. server crash, network error, HD
malfunction, etc.

100. Where to inject latency?

101. Where to inject latency?

102. hypothesis:
Function has appropriate timeout on its HTTP
communications and can degrade gracefully
when these requests time out

103. Where to inject latency?

104. Where to inject latency?
Should be applied to 3rd party services too.
DynamoDB, Twillio, Auth0, …

105. Where to inject latency?

106. Where to inject latency?
Be mindful of the blast radius of the experiment.
The goal is not to break things.

107. http client
public-api-a
http client
public-api-b
internal-api
Where to inject latency?

108. hypothesis:
All functions have appropriate timeout on their HTTP
communications to this internal API, and can degrade
gracefully when requests are timed out

109. Where to inject latency?

110. Where to inject latency?

111. Large blast radius, can cause cascade failures unintentionally
Where to inject latency?

113. development

114. development production

115. Priming (psychology):
Priming is a technique whereby exposure to one stimulus
influences a response to a subsequent stimulus, without
conscious guidance or intention.
It is a technique in psychology used to train a person's
memory both in positive and negative ways.

117. Use failure injection to programme your colleagues into
thinking about failure modes early.

118. Where to inject latency?
Make X% of all requests slow
in the dev environment.

119. hypothesis:
The client app has appropriate timeout on their
HTTP communication with the server, and can
degrade gracefully when requests are timed out

120. Where to inject latency?

121. Where to inject latency?

123. Where to inject latency?

124. STEP 4. Disprove hypothesis
i.e. look for difference in steady state

125. How to inject latency?

126. How to inject latency?
Static weavers (e.g. PostSharp, AspectJ).
Dynamic proxies.

127. https://theburningmonk.com/2015/04/design-for-latency-issues/

128. How to inject latency?
Manually crafted wrapper libraries.

134. configured in SSM Parameter Store

136. no injected latency

138. with injected latency

140. factory wrapper function
(think bluebird’s promisifyAll function)

146. Error injection with serverless

147. Common errors
HTTP 5XX
DynamoDB provisioned throughput exceeded
Throttled Lambda invocations

148. hypothesis:
Function has appropriate error handling on its HTTP
communications and can degrade gracefully when
downstream dependencies fail

149. Where to inject errors?

150. hypothesis:
Function has appropriate error handling on
DynamoDB operations and can degrade gracefully
when DynamoDB throughputs are exceeded

151. Where to inject errors?

152. Where to inject errors?
Induce Lambda throttling by temporarily setting reserved concurrency.

153. Recap

154. failures are INEVITABLE

155. the only way to truly know your system’s
resilience against failures is to test it
through CONTROLLED experiments

157. the goal of chaos engineering is NOT to
actually break production

158. CONTAINMENT should be front and
centre of your thinking

160. STEP 1. Define “steady state”
aka. what does normal, working
condition looks like?

161. Hypothesise steady state will
continue in both control group
& the experiment group
ie. you should have a reasonable degree of confidence the
system would handle the failure before you proceed with
the experiment
STEP 2.

162. STEP 3. Inject realistic failures
e.g. server crash, network error, HD
malfunction, etc.

163. STEP 4. Disprove hypothesis
i.e. look for difference in steady state

164. there are more inherent chaos and
complexity in a serverless application

165. even without servers, you can still inject
CONTROLLED failures at the application level

170. @theburningmonk
theburningmonk.com
github.com/theburningmonk

171. API Gateway and Kinesis
Authentication & authorisation (IAM, Cognito)
Testing
Running & Debugging functions locally
Log aggregation
Monitoring & Alerting
X-Ray
Correlation IDs
CI/CD
Performance and Cost optimisation
Error Handling
Configuration management
VPC
Security
Leading practices (API Gateway, Kinesis, Lambda)
Canary deployments
http://bit.ly/prod-ready-serverless
get 40% off
with: ytcui