Download to read offline
Uploaded byYan Cui
Applying principles of chaos engineering to serverless
The document discusses chaos engineering principles, particularly in serverless architectures, emphasizing the importance of controlled experiments to enhance system resilience against failures. It outlines a step-by-step approach for conducting these experiments, including defining steady states, hypothesizing outcomes, injecting realistic failures, and analyzing results. Key practices include effective communication, careful timing, and assessing the impacts of various failures to improve system robustness.
More Related Content
Similar to Applying principles of chaos engineering to serverless
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Applying principles of chaos engineering to serverless
- 1. APPLYING PRINCIPLES to SERVERLESSt a b chaosengineering of A E S of
- 2.
- 3. history of Smallpox est.400K deaths per year in 18th Century Europe. earliest evidence of disease in 3rd Century BC Egyptian Mummy
- 4. history of Smallpox est.400K deaths per year in 18th Century Europe. earliest evidence of disease in 3rd Century BC Egyptian Mummy 1798 first vaccine developed Edward Jenner
- 5. 1798 first vaccine developed 1980 historyof Smallpox Edward Jenner WHO certified global eradication est. 400K deaths per year in 18th Century Europe. earliest evidence of disease in 3rd Century BC Egyptian Mummy
- 7. Vaccination is themost effective method of preventing infectious diseases
- 8. stimulates the immunesystem to recognize and destroy the disease before contracting the disease for real
- 9. Chaos Engineering controlled experimentsto help us learn about our system’s behaviour and build confidence in its ability to withstand turbulent conditions
- 13. it’s about buildingconfidence, NOT breaking things
- 14. I’m gonna inject youwith a deadly disease now
- 15.
- 16. STEP 1. define“Steady State” aka. what does normal, working condition looks like?
- 17. this is nota steady state
- 18. STEP 2. hypothesize steadystate will continue in both control group & the experiment group ie. you should have a reasonable degree of confidence the system would handle the failure before you proceed with the experiment
- 19. explore unknown unknowns awayfrom production
- 20. treat production withthe care it deserves
- 21. the goal isNOT, to actually hurt production
- 22. If you knowthe system would break, and you did it anyway… then it’s NOT a chaos experiment. It’s called being IRRESPONSIBLE.
- 24. STEP 3. inject realisticfailures e.g. server crash, network error, HD malfunction, etc.
- 25.
- 26.
- 28. STEP 4. disprove hypothesis i.e.look for difference with steady state
- 29. if a WEANESSis uncovered, improve it before the behaviour manifests in the system at large
- 30. Chaos Engineering controlled experimentsto help us learn about our system’s behaviour and build confidence in its ability to withstand turbulent conditions
- 32. Chaos Engineering controlled experimentsto help us learn about our system’s behaviour and build confidence in its ability to withstand turbulent conditions
- 33.
- 34. ensure everyone knowswhat you’re doing
- 35. ensure everyone knowswhat you’re doing NO surprises!
- 36.
- 37. run experiments duringoffice hours
- 38.
- 39.
- 40. smallest change thatallows you to detect a signal that steady state is disrupted
- 41. rollback at thefirst sign of TROUBLE!
- 42.
- 43. Don’t try torun before you know how to walk.
- 44. by Russ Miles@russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361
- 45. chaos monkey killsan EC2 instance latency monkey induces artificial delay in APIs chaos gorilla kills an AWS Availability Zone chaos kong kills an entire AWS region
- 48. there is noserver…
- 49. there is noserver… that you can kill
- 52. there are moreinherent chaos and complexity in a Serverless architecture
- 53. smaller units ofdeployment and A LOT more of them!
- 54. more difficult toharden around boundaries serverful serverless
- 55.
- 56. ? SNS Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 SES more intermediaryservices, and greater variety too
- 57. ? SNS Kinesis CloudWatch Events CloudWatch LogsIoT DynamoDB S3 SES more intermediaryservices, and greater variety too each with its own set of failure modes
- 58. serverful serverless more configurations, more opportunitiesfor misconfiguration
- 59. more unknown failuremodes in infrastructure that we don’t control
- 60. often there’s littlewe can do when an outage occurs in the platform
- 61.
- 62.
- 63. missing fallback whendownstream is unavailable
- 64.
- 65. STEP 1. define“Steady State” aka. what does normal, working condition looks like?
- 66. what metrics doyou monitor?
- 67. 9X-percentile latency error count yield(% of requests completed) harvest (completeness of results)
- 68. STEP 2. hypothesize steadystate will continue in both control group & the experiment group ie. you should have a reasonable degree of confidence the system would handle the failure before you proceed with the experiment
- 69.
- 70. consider the effectof cold-starts & API Gateway overhead
- 71. use short timeoutfor API calls
- 72. the goal ofa timeout strategy is to give HTTP requests the best chance to succeed, provided that doing so does not cause the calling function itself to err
- 73. fixed timeout aretricky to get right…
- 74. fixed timeout aretricky to get right… too short and you don’t give requests the best chance to succeed
- 75. fixed timeout aretricky to get right… too long and you run the risk of letting the request timeout the calling function
- 76. and it getsworse when you make multiple API calls in one function…
- 79. set the requesttimeout based on the amount of invocation time left
- 84. log the timeoutincident with as much context as possible e.g. timeout value, correlation IDs, request object, …
- 85.
- 89. be mindful whenyou sacrifice precision for availability, user experience is the king
- 90. STEP 3. inject realisticfailures e.g. server crash, network error, HD malfunction, etc.
- 91. where to injectlatency?
- 92. hypothesis: function has appropriatetimeout on its HTTP communications and can degrade gracefully when these requests time out
- 94. should also beapplied to 3rd parties services we depend on, e.g. DynamoDB
- 96. what’s the blastradius?
- 97.
- 98. hypothesis: all functions haveappropriate timeout on their HTTP communications to this internal API, and can degrade gracefully when requests are timed out
- 101. large blast radius,risky..
- 103. could be effectivewhen used away from production environment, to weed out weaknesses quickly
- 104. not priming developersto build more resilient systems
- 105.
- 106.
- 107. Priming (psychology): Priming isa technique whereby exposure to one stimulus influences a response to a subsequent stimulus, without conscious guidance or intention. It is a technique in psychology used to train a person's memory both in positive and negative ways.
- 110. make dev environmentsbetter resemble the turbulent conditions you should realistically expect your system to survive in production
- 111. hypothesis: the client apphas appropriate timeout on their HTTP communication with the server, and can degrade gracefully when requests are timed out
- 116. STEP 4. disprove hypothesis i.e.look for difference with steady state
- 117. how to injectlatency?
- 118. static weaver (e.g.AspectJ, PostSharp), or dynamic proxies
- 119.
- 120. manually crafted wrapperlibrary
- 126. configured in SSMParameter Store
- 128.
- 130.
- 132. factory wrapper function (thinkbluebird’s promisifyAll function)
- 138.
- 139.
- 140. the only wayto truly know your system’s resilience against failures is to test it through controlled experiments
- 144. vaccinate your serverless architectureagainst failures
- 145.
- 146.