Chaos Engineering is listed as 'Trial' in the ThoughtWorks Tech Radar, but what is it really and how is it different from traditional testing? When and why should you get started with Chaos Engineering and is Chaos Monkey the right place to start when you do?

1. Chaos Engineering
When should you release the monkeys?
Steve Upton

2. What is Chaos Engineering?
Why Chaos Engineering?
Getting started with Chaos Engineering

3. What is Chaos
Engineering?

5. Netflix Principles

6. Netflix Principles
1. No service should be a Single Point Of Failure

7. Netflix Principles
1. No service should be a Single Point Of Failure
2. Never trust that you’ve done #1 correctly

9. "It is better to be in a constant
state of minor failure [than
occasionally catastrophic failure]”
- Richard Rodger, CEO @ nearForm

14. Principles of Chaos Engineering
source: principlesofchaos.org

15. Principles of Chaos Engineering
source: principlesofchaos.org
1. Start by defining ‘steady state’ as some measurable output of a system that
indicates normal behavior.

16. Principles of Chaos Engineering
source: principlesofchaos.org
1. Start by defining ‘steady state’ as some measurable output of a system that
indicates normal behavior.
2. Hypothesize that this steady state will continue in both the control group and
the experimental group.

17. Principles of Chaos Engineering
source: principlesofchaos.org
1. Start by defining ‘steady state’ as some measurable output of a system that
indicates normal behavior.
2. Hypothesize that this steady state will continue in both the control group and
the experimental group.
3. Introduce variables that reflect real world events like servers that crash, hard
drives that malfunction, network connections that are severed, etc.

18. Principles of Chaos Engineering
1. Start by defining ‘steady state’ as some measurable output of a system that
indicates normal behavior.
2. Hypothesize that this steady state will continue in both the control group and
the experimental group.
3. Introduce variables that reflect real world events like servers that crash, hard
drives that malfunction, network connections that are severed, etc.
4. Try to disprove the hypothesis by looking for a difference in steady state
between the control group and the experimental group.
source: principlesofchaos.org

19. Principles of Chaos Engineering
1. Make sure your system is working.
2. Try to break it.
3. Did it break?
4. Repeat.

20. Principles of Chaos Engineering

21. Principles of Chaos Engineering

22. Chaos Engineering needs...
A way of validating that
the system is working
and
a way of inducing failures.

23. “Isn’t this just
stress testing?”
- Everyone

24. Real world
failure modes

25. Real world
failure modes

26. Real world
failure modes

27. “Everything fails all the time”
- Werner Vogels, VP + CTO @ AWS

28. “A distributed system is one in
which the failure of a computer you
didn't even know existed can render
your own computer unusable.”
- Leslie Lamport

29. Decoupled cause and effect

30. Emphasis on monitoring

31. Why Chaos
Engineering?

32. Image credit: UX Brighton
Cynefin
Framework

34. Simple
Sense
Categorize
Respond
Best Practice

35. Simple
Sense
Categorize
Respond
Best Practice

36. Complicated
Sense
Analyze
Respond
Good Practice
Simple
Sense
Categorize
Respond
Best Practice

37. Complicated
Sense
Analyze
Respond
Good Practice
Simple
Sense
Categorize
Respond
Best Practice

38. Complex
Probe
Sense
Respond
Emergent
Complicated
Sense
Analyze
Respond
Good Practice
Simple
Sense
Categorize
Respond
Best Practice

39. Complex
Probe
Sense
Respond
Emergent
Complicated
Sense
Analyze
Respond
Good Practice
Simple
Sense
Categorize
Respond
Best Practice

40. Complex
Probe
Sense
Respond
Emergent
Complicated
Sense
Analyze
Respond
Good Practice
Chaotic
Act
Sense
Respond
Novel
Simple
Sense
Categorize
Respond
Best Practice

41. Complex
Probe
Sense
Respond
Emergent
Complicated
Sense
Analyze
Respond
Good Practice
Chaotic
Act
Sense
Respond
Novel
Simple
Sense
Categorize
Respond
Best Practice

42. credit: @johncutlefish

43. Complex
Probe
Sense
Respond
Emergent
Complicated
Sense
Analyze
Respond
Good Practice
Chaotic
Act
Sense
Respond
Novel
Simple
Sense
Categorize
Respond
Best Practice

47. Complex Complicated
Chaotic Simple

48. Complex Complicated
Chaotic Simple
Chaos Engineering

49. Complex Complicated
Chaotic Simple

50. Complex
Chaotic
?

54. Complex Complicated
Chaotic Simple
Unified Logging

55. Complex Complicated
Chaotic Simple
Code Freeze

56. “An evolving [software] system
increases in complexity unless
work is done to reduce it.”
- Lehman's laws of software evolution

57. Complex Complicated
Chaotic Simple
Where are
you today?

58. Complex Complicated
Chaotic Simple
Where will
you be
tomorrow?

59. Getting started with
Chaos Engineering

61. {
"version": "1.0.0",
"title": "What is the impact of an expired certificate on our application chain?",
"description": "If a certificate expires, we should gracefully deal with the issue.",
"steady-state-hypothesis": {
"title": "Application responds",
"probes": [
{
"type": "probe",
"name": "we-can-request-sunset",
"tolerance": 200,
"provider": {
"type": "http",
"timeout": 3,
"verify_tls": false,
"url": "https://localhost:8443/city/Paris"
}
}
]
}

62. "method": [
{
"type": "action",
"name": "swap-to-expired-cert",
"provider": {
"type": "process",
"path": "cp",
"arguments": "expired-cert.pem cert.pem"
}
},
{
"type": "action",
"name": "restart-astre-service-to-pick-up-certificate",
"provider": {
"type": "process",
"path": "pkill",
"arguments": "--echo -HUP -F astre.pid"
}
}
],

63. "rollbacks": [
{
"type": "action",
"name": "swap-to-vald-cert",
"provider": {
"type": "process",
"path": "cp",
"arguments": "valid-cert.pem cert.pem"
}
},
{
"ref": "restart-astre-service-to-pick-up-certificate"
},
{
"ref": "restart-sunset-service-to-pick-up-certificate"
}
]

67. Chaos Engineering
You must be
this tall to use
Microservices

68. Chaos Engineering
You must be
this tall to use
Chaos Engineering

71. monitoring

74. https://artillery.io/chaos-lambda/

77. Steve Upton
Thank you
@Steve_Upton