Chaos engineering is a proactive approach to testing system resilience by conducting controlled experiments to identify weaknesses, rather than waiting for outages. The document outlines various levels of maturity for conducting chaos experiments, starting from basic host failure preparedness to advanced fault injections. It emphasizes the importance of thoughtful design in experimentation to improve system reliability and user experience.

1. Introduction to
Chaos Engineering
Kolton Andrus
CEO and Co-Founder
Gremlin

2. J. Crew Website down during Black
Friday 2018 sales Newsweek, 11/23/18
Alaska Airlines Flights halted nationwide due
to outage in Seattle Business Insider, 1/6/19
Slack Outage Connectivity issues
hit workplaces WSJ, 6/27/18

3. What is Chaos Engineering?

4. Inject something harmful to
build an immunity.

5. Chaos Engineering
Thoughtful, planned experiments
designed to reveal the
weakness in our systems.

6. We test proactively, instead of
waiting for an outage.

7. Distributed Systems
are fragile.
THE PROBLEM

8. INTERNETDATA CENTER DATA CENTER
BUSINESS LOGICWEB SERVER DATABASE
The “Old” way

9. Kolton Andrus
CO-FOUNDER & CEO
Matthew Fornaciari
CO-FOUNDER & CTO
We’ve done this before.

11. How?

12. Design
Thoughtful
Experiments

13. Chaos Experiment | Example
Hypothesis (Expected Outcome) No customer impact expected
Attack Condition
Duration: 600s (10 min)
Latency: 400ms
Targets: 50% of available instances
Result (Actual Outcome)
1.2 to 1.6 second latency
Degraded user experience
Returning cached data with 200s
“That’s a real miss in alerting and metrics.”
Slow Response from Database Primary

14. THE BEGINNING
Chaos Monkey
Level 0
MATURITY REQUIRED
Low
APPROACH TAKEN
Random
VALUE PROVIDED
Prepare for host failures in the cloud

15. THE FIRST STEP
Infrastructure
Failures
Level 1
MATURITY REQUIRED
Basic Operations
APPROACH TAKEN
Disciplined
VALUE PROVIDED
Prepare for host-level failures

16. Intermediate
Network Failures
Level 1.5
MATURITY REQUIRED
Networking expertise
APPROACH TAKEN
Gameday
VALUE PROVIDED
Prepare for high impact events

17. Truly comprehensive
fault injection.
Consume Resources
Change the State
Confuse the Network

18. World-class safety & security.
SUPPORT FOR SSO & MFA NEVER RUN AS ROOT REVERT

19. Astonishingly simple
and easy to use.
Intuitive CLI
Elegant API

20. Trusted by teams worldwide

21. Thank you.