This document discusses chaos engineering and building resilient systems. It defines chaos engineering as experimenting in production to reveal weaknesses and build confidence in resilience. Some key principles of chaos engineering are discussed, such as having steady state periods between experiments and formulating hypotheses before experiments. Game days are mentioned where engineers take on roles like master of disaster to experiment with failures. The goal of chaos engineering is to design systems that can withstand failures through practices like circuit breaking and observability.

1. Chaos Engineering
Injecting failure for building
resilience in systems

2. Nice to meet you
YURY NIÑO
Software Engineer and Chaos
Engineer Advocate.
Loves building software applications, solving
resilience issues and teaching. Passionate about
reading, writing and cycling.

3. Agenda
● Resilience vs Reliability
● Why the world needs Resilience and
Reliability?
● Chaos Engineering
● Principles of Chaos
● Chaos in Practice
● Game Days

4. How many of you
Have encountered a crash of
your systems on production?

5. A recognition for ...
This talk is dedicated to
the #SystemAdministrators well
caffeinated, who get woken up in the
middle of the night when “things go
bump”.
#EngineeringTeam #DigitalFactory
@jnhernandz @

6. What is a
Resilient System?

7. A resilient system can maintain an acceptable level
of service in the face of failure.
A resilient system can weather the storm such a
large scale natural disaster or a controlled chaos
engineering.
Tammy Bütow Principal SRE at Gremlin

8. https://securethegrid.com

9. A distributed system on production needs to be
resilient in order to be reliable and this is precisely
a target that we Software Engineers, Systems
Engineers, Site Reliability Engineers and Chaos
Engineers always aim.
Mine :)

10. Why the world needs
Resilient Systems?

11. Because ...
We are surrounded by
distributed systems.
When we read the news in our
cellphones, send an email or buy our
lunch ...
We do not tolerate that
they fail!

13. February 28th, 2017 will be remembered
● Simple Storage Service (S3) went down in US-EAST.
● Outage lasted about 4 hrs.
● > 100.000 websites across the world were impacted.

14. Me :(

15. The World is Chaotic!
● Distributed systems contains moving
parts.
● Many things can go wrong.
○ Hard disks can fail.
○ The network can go down.
○ Customer traffic can overload.

16. How many of you know
What is Chaos
Engineering?

17. Chaos Engineering
It is the discipline of experimenting in
production on a distributed system in
order to reveal their weakness and to
build confidence in their resilience
capability.
https://principlesofchaos.org/

18. Chaos Engineering
It is deliberately inducing stress or
fault into software and/or hardware as
a way of learning/verifying things
about systems.
https://www.gremlin.com

19. Chaos Engineering is about
● Simulating the failure of a datacenter.
● Injecting latency between services.
● Randomly causing exceptions.
● Changing time travel.
● Emulating I/O errors.
http://principlesofchaos.org/

20. 2008
Chaos Engineering
began at Netflix
2010
Chaos Monkey was
launched
2018
A lot of resources for
Chaos Engineering.
2014
Role of Chaos
Engineer was created.
History of Chaos Engineering
Kolton Andrus

21. Chaos in Practice

22. Principles of
Chaos
https://principlesofchaos.org/

23. 1. Steady Stead

25. 2. Hypothesis:
Circuit
Breaker
builds
Resilience

26. 2. Hypothesis:
Circuit
Breaker
builds
Resilience

27. 4. Run the Experiment
Application
Name Finer Observability DataDog
Hypothesis Circuit Breaker works
Environment My Home Results
Duration 5 - 10 seconds
Load 1 request
Actions

28. 4. Run the Experiment
Application
Name Finer Observability DataDog
Hypothesis Circuit Breaker works
Facing latencies > 5 seconds between
dashboard_api and smart_api to open
the circuit.
Environment My Home Results
Duration 20 milliseconds
Load 1 request
Issue #4356
Configure the proper hystrix parameters
according the results.
Implement a fallback.
Actions

29. Game Days

30. Game Day: Roles
Master of Disaster First on-call Team
https://www.pinterest.es/pin/824299538021645731/

31. Game Days can Transform our Teams
Even though Game Days are not real! they
make Engineers gain confidence.

32. Since we, Engineers are experiencing the failure as part
of our job, we should start designing for failure.
Me :)
The best time to learn about fire
is when you’re on fire.
—Jen Hammond, New Relic engineering manager

33. How to begin ...
https://chaosengineering.slack.com
https://github.com/dastergon/awesome-chaos
-engineering
https://www.infoq.com/chaos-engineering
@yurynino