An introduction to chaos engineering as part of DevOps at XP2019 - Presented by Gurtej Pal Singh @ XP2019 Montreal Canada https://xp2019.sched.com/event/MeQs/introducing-chaos-programming-as-part-of-devops-gurtej-pal-singh

1. Introducing Chaos Engineering
as part of DevOps

3. What is Chaos Engineering ?
4

4. “Everything fails all the time. We lose whole data centers!
Those things happen.”
Werner Vogels (CTO @Amazon.com)
What ?
5

5. “Everything fails all the time. We lose whole data centers!
Those things happen.”
Werner Vogels (CTO @Amazon.com)
Who am I ?
6
Twitter: @GurtejPalSingh
LinkedIn: https://www.linkedin.com/in/gurtejpalsingh/
Blog: https://gurtejpsingh.wordpress.com/
Associate Director – Agile Service Delivery @ LoyaltyOne
Enterprise Digital Strategy (Agile Coaching & DevOps Consulting) | Planning & Execution | Technology Consulting
Gurtej Pal Singh

6. Agenda
• What ?
• Chaos Engineering VS Testing
• The Story of How & Why
• Chaos Engineering & DevOps
• Industry Presence
7

7. Ideal Architecture
Fault
Tolerant
Highly
Available
Auto
Scalable
Gurtej Pal Singh | @GurtejPalSingh
8

8. Let us see…
Database Database
Database
Request DB Request
Fetch from Cache
Web Server
Users
Gurtej Pal Singh | @GurtejPalSingh
9

9. Then to now…
Gurtej Pal Singh | @GurtejPalSingh
10Pic courtesy: https://blog.knoldus.com

10. Let us see…..
Gurtej Pal Singh | @GurtejPalSingh
11

11. Gurtej Pal Singh | @GurtejPalSingh
11
Chaos Engineering Software Testing
Chaos Engineering VS Testing

12. Chaos Engineering VS Testing
Gurtej Pal Singh | @GurtejPalSingh
13

13. Models of the two…
Testing == Validation
Chaos Engineering == Experimentation
Gurtej Pal Singh | @GurtejPalSingh
14

14. Gurtej Pal Singh | @GurtejPalSingh
14
The story of How & Why…

15. Prerequisites
• Review the application architecture to identify :
•Dependencies
•Customer impact
•Failure Points
•Recovery procedures
Gurtej Pal Singh | @GurtejPalSingh
16

16. Principles of Chaos
• Build a Hypothesis around Steady State Behavior
• Vary real world events
• Experiment in production
• Automate experiments to run continuously
• Minimize Blast Radius
Gurtej Pal Singh | @GurtejPalSingh
17

17. Sample Chaos Experiments*
• Terminate random virtual server in a region.
• Subject entire fleet to high CPU/Memory within a region.
• Increase latency in one or more servers.
• Block access to a storage system.
• Failover a database to its secondary.
• Random killing of critical processes.
*For a generic three tier application, It is not comprehensive and depends on the application tooling support and maturity, but it is
a good starting point.
Gurtej Pal Singh | @GurtejPalSingh
18

18. Approach
Web Setup
Sane Setup
Chaos Setup
Requests
Expected behavior
Expected
benchmarked
behavior
Hypothesis
(dis)proved behavior
Incorporate
Learnings
Gurtej Pal Singh | @GurtejPalSingh
19
ForComparingtheresults

19. Levels of Chaos
Sophistication
Adoption
Gurtej Pal Singh | @GurtejPalSingh
20
No chaos Story Frequent Crashes
Presumed Resiliency True Resiliency

20. Why ?
Gurtej Pal Singh | @GurtejPalSingh
21Image: www.salonuready.info

21. Gurtej Pal Singh | @GurtejPalSingh
19
Chaos Engineering & DevOps

22. Chaos Engineering in CI CD Pipeline
Gurtej Pal Singh | @GurtejPalSingh
23

23. Simian Army :
• Chaos Monkey
• Chaos Gorilla
• Chaos Kong
• ChAP (Chaos Automation Platform)
Few tools to get started with
Chaos Engineering Gurtej Pal Singh | @GurtejPalSingh
24

24. Chaos Engineering as a part
of DevOps Culture
Post the Chaos Experiments has been continuously validated in a CICD pipeline:
Gurtej Pal Singh | @GurtejPalSingh
25
Developers & stakeholders continually embrace failure as a way to prepare for and prevent
it, resulting in stronger and more resilient applications.
• Enable it in production , incrementally, in a random fashion (without notifying the support teams)
• Tooling must be mature enough to stop the experiment without further impacts.
• Initialize Chaos experiments introduced during random non-peak times, post maturity , move to peak
times.

25. Industry Presence
Gurtej Pal Singh | @GurtejPalSingh
26

26. Industry Presence
Gurtej Pal Singh | @GurtejPalSingh
24
Chaos Engineering
Companies, People,
Tools and Practices

27. What ?
Discipline of experimenting in a distributed system to build confidence
in system capability in turbulent conditions in “PRODUCTION”.
https://principlesofchaos.org/
Gurtej Pal Singh | @GurtejPalSingh
28

28. Gurtej Pal Singh | @GurtejPalSingh
29
- Werner Vogels –
VP & CTO at Amazon.com
The main point is no more to avoid failures,
but to limit impact of those failures.

29. References & Resources
• https://principlesofchaos.org/
• https://medium.com/capital-one-tech/continuous-chaos-introducing-chaos-
engineering-into-devops-practices-75757e1cca6d
• https://www.gremlin.com/uploads/20171210-
Chaos_Engineering_White_Paper.pdf
• https://www.youtube.com/watch?v=6ilMZqKdMMU
• https://www.youtube.com/watch?v=qHykK5pFRW4
• https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116
• https://medium.com/netflix-techblog/chap-chaos-automation-platform-
53e6d528371f
• https://www.oreilly.com/library/view/chaos-engineering/9781491988459/
Gurtej Pal Singh | @GurtejPalSingh
30

30. Gurtej Pal Singh | @GurtejPalSingh
29
Twitter: @GurtejPalSingh
LinkedIn: https://www.linkedin.com/in/gurtejpalsingh/