The document discusses the history and principles of chaos engineering. It began in 2004 at Amazon and was further developed and popularized at Netflix in 2010-2012 when they created tools like Chaos Monkey and open sourced their Simian Army. Key aspects of chaos engineering discussed include defining the steady state of a system, monitoring key metrics, starting with small and reversible experiments, automating experiments to run often, and shifting mindsets to proactively address failures. The overall goal is to build confidence in a system's ability to withstand failures through experimentation.

1. CHAOS ENGINEERING
– OR LET'S SHAKE THE
TREE
J I M M Y D A H L Q V I S T , K N O W I T

3. " Failures are given and
everything will eventually
fail over time "
Werner Vogels
CTO – Amazon.com

4. TRIBUTE
• Nora Jones
• Adrian Cockroft
• Adrian Hornsby

5. History and background
2004 Amazon – Jesse Robbins. Master of disaster
2010 Netflix – Greg Orzell. Chaos Monkey
2012 Netflix – Open sources simian army
2016 Gremlin Inc is founded
2017 Netflix Chaos engineering book. Chaos toolset.
2018 Concept is spread, ChaosConf started.

7. History and background
2004 Amazon – Jesse Robbins. Master of disaster
2010 Netflix – Greg Orzell. Chaos Monkey
2012 Netflix – Open sources simian army
2016 Gremlin Inc is founded
2017 Netflix Chaos engineering book. Chaos toolset.
2018 Concept is spread, ChaosConf started.

9. " By running experiments on a
regular basis that simulate a
Regional outage, we were able to
identify any systemic weaknesses
early and fix them. "
Netflix Blog

10. History and background
2004 Amazon – Jesse Robbins. Master of disaster
2010 Netflix – Greg Orzell. Chaos Monkey
2012 Netflix – Open sources simian army
2016 Gremlin Inc is founded
2017 Netflix Chaos engineering book. Chaos toolset.
2018 Concept is spread, ChaosConf started.

12. History and background
2004 Amazon – Jesse Robbins. Master of disaster
2010 Netflix – Greg Orzell. Chaos Monkey
2012 Netflix – Open sources simian army
2016 Gremlin Inc is founded
2017 Netflix – Chaos engineering book.
2018 Concept is spread, ChaosConf started.

13. History and background
2004 Amazon – Jesse Robbins. Master of disaster
2010 Netflix – Greg Orzell. Chaos Monkey
2012 Netflix – Open sources simian army
2016 Gremlin Inc is founded
2017 Netflix – Chaos engineering book.
2018 Concept is spread, ChaosConf started.

14. Chaos Engineering is the discipline of
experimenting on a system
in order to build confidence in the system’s
capability to withstand turbulent conditions
in production.

15. " Failures are given and
everything will eventually
fail over time "
Werner Vogels
CTO – Amazon.com

16. Chaos Engineering is the discipline of
experimenting on a system
in order to build confidence in the system’s
capability to withstand turbulent conditions
in production.

18. Unit testing
Component X
Input Output

19. Integration testing
Component A
Input OutputOutput / Input
Component B

20. Distributed System
Input
Output

22. Distributed System
Input
Output Corrupt?

23. Chaos Engineering is the discipline of
experimenting on a system
in order to build confidence in the system’s
capability to withstand turbulent
conditions in production.

25. " Chaos doesn't cause problems.
It reveals them. "
Nora Jones
Slack - Head of chaos engineering
and human factors

26. Before practicing chaos

27. Socialize

28. Start small
Use an opt in model, not an opt out.
Only include services that like to be chaosed.
Start with a success!
Don't start in production.

29. Steady state
Define the steady state. Build a hypothesis about the
steady state. What does our system look like when it's
behaving normally.

30. Monitoring
Understand your key business metrics and KPIs.
Netflix key business metric is SPS.

31. First experiment
Graceful restarts and degradations

32. Design your next experiments
" You have to know the past to
understand the present. "
Carl Sagan

33. Move to production
Don't forget about your customers!
Don't destroy the customer experience!
Make sure you can abort!
Only run during business hour.

34. Automate everything
Run often
Automatic safeguards
Percentage of traffic
Netflix Chaos Automation Platform (ChAP)

35. Change of mindset
What happens IF this fails to,
what happens WHEN this fails.

36. Lesson learnt

40. Takeaways
Everyone can be doing Chaos Engineering
Chaos Engineering is a learning opportunity
Be conscious about customers, involve business

41. " Chaos doesn't cause problems.
It reveals them. "
Nora Jones
Slack - Head of chaos engineering
and human factors

42. Tack!