https://www.reactivesummit.org/2018/schedule/the-practice-of-chaos-engineering

1. The Practice of
Chaos Engineering
Ana Medina
Chaos Engineer at Gremlin
@ana_m_medina

2. @ana_m_medina#reactive18
@ana_m_medina
Gremlin
Uber
SFEFCU Google Quicken Loans
Stanford University Miami
Dade College

3. @ana_m_medina#reactive18
How many of you have heard
of Chaos Engineering?

4. @ana_m_medina#reactive18
How many of you have run a
Chaos Engineering
experiment?

5. @ana_m_medina#reactive18
What is
Chaos Engineering?

6. @ana_m_medina#reactive18
Thoughtful, planned
experiments designed to reveal
the weakness in our systems.
Chaos Engineering

7. @ana_m_medina#reactive18
Inject something harmful to
build an immunity.
-@KoltonAndrus
Gremlin Founder and CEO
Chaos Engineering

8. @ana_m_medina#reactive18
Why?
● Microservices
● Systems are scaling
fast
● Downtime is really
expensive
● Our dependencies
will fail
● Pager fatigue and
burnout really hurts

9. @ana_m_medina#reactive18
Use Cases:
● Outage
reproduction
● On-call training
● Strengthen new
products
● Battle test new
infrastructure and
services

10. @ana_m_medina#reactive18
What do you need before doing Chaos
Engineering?
● Monitoring/Observability
● On-Call and Incident Management
● Cost of Downtime Per Hour

11. @ana_m_medina#reactive18
Chaos Engineering is not
● Unexpected or unmonitored experiments
● Creating outages

12. @ana_m_medina#reactive18
“Chaos Engineering
Without Observability ...
Is Just Chaos”
-@mipsytipsy
Charity Majors
CEO of honeycomb

13. @ana_m_medina#reactive18
Minimize the
Blast radius

14. THE BEGINNING
Chaos Monkey
Level 0
MATURITY REQUIRED
Low
APPROACH TAKEN
Random
VALUE PROVIDED
Prepare for host failures in the cloud
@ana_m_medina#reactive18

15. THE FIRST STEP
Infrastructure
Failures
Level 1
MATURITY REQUIRED
Basic Operations
APPROACH TAKEN
Disciplined
VALUE PROVIDED
Prepare for host-level failures
@ana_m_medina#reactive18

16. INTERMEDIATE
Network Failures
Level 1.5
MATURITY REQUIRED
Networking expertise
APPROACH TAKEN
Gameday
VALUE PROVIDED
Prepare for high impact events
@ana_m_medina#reactive18

17. THE NEXT STEP
Application
Failures
Level 2
MATURITY REQUIRED
Advanced
APPROACH TAKEN
Precision Experiments
VALUE PROVIDED
Safely validate the user experience
@ana_m_medina#reactive18

18. Latency added to 50% of android traffic

19. Exceptions - 50% of android traffic failed

20. @ana_m_medina#reactive18
You can and should inject chaos at every layer
of your stack
● Application
● API
● Caching
● Database
● Hardware
● Cloud Infrastructure / Bare metal

21. @ana_m_medina#reactive18
Top places to inject chaos

22. @ana_m_medina#reactive18

23. @ana_m_medina#reactive18
https://www.gremlin.com/community/tutorials/what-i-learned-running-the-
chaos-lab-kafka-breaks/

24. @ana_m_medina#reactive18
Getting Started:
● Identify top 5 critical systems
● Choose system
● Whiteboard the system
● Determine what experiment you want to run:
(resource, state, network)
● Determine Blast Radius

25. @ana_m_medina#reactive18
Companies doing
Chaos Engineering

26. @ana_m_medina#reactive18
Chaos Days

27. @ana_m_medina#reactive18
Chaos Days: Dedicated day for your
entire company to focus on building
resilience instead of new products.
https://www.gremlin.com/community/tutorials/planning-your-own-chaos-day/

28. @ana_m_medina#reactive18
“What could go wrong?”
“Do we know what will
happen if this breaks?”

29. @ana_m_medina#reactive18
Chaos Day Crew:
VP Engineering / CTO / COO
Executive Assistant
Engineering Director / Manager
Senior Engineer
New Grad / Intern Engineer

30. @ana_m_medina#reactive18
What experiments can you
run?
• Reproduce outage conditions
• Unpredictable circumstances
• Large traffic spikes
• Race conditions
• Datacenter failure
• Time travel - system clocks to be out of sync
• Network errors
• CPU overloads

31. @ana_m_medina#reactive18

32. @ana_m_medina#reactive18

33. @ana_m_medina#reactive18

34. @ana_m_medina#reactive18

35. @ana_m_medina#reactive18

36. @ana_m_medina#reactive18
gremlin.com/chaos-monkey/

37. @ana_m_medina#reactive18
Join the Chaos, Join Slack:
bit.ly/chaos-eng-slack
1,900+ members across the world
Learn more:

38. @ana_m_medina#reactive18
THANKS!
@ana_m_medina
ana@gremlin.com