Athletes, Firemen and Doctors train everyday to be the best at their chosen profession. As engineers, we spend much of our time getting stuff to production and making sure our infrastructure doesn’t burn down out right. In this talk, we'll discuss the need for and the options of creating a game day culture. Where we as engineers not only write, maintain and operate our software platforms but actively pursue ways to learn and predict its (non-functional) behavior. We'll look at tools like toxiproxy and the simian army for ways to prepare teams to tweak their testing and monitoring setup and work instructions to quickly observe, react to and resolve problems.

1. Gamification of Chaos Testing
Bram Vogelaar
@attachmentgenie

2. What could this number represent ?
29.74 Seconds

3. That is a weird SLO???
99.97 % Uptime

4. Nope, it is this guy!

5. Well…Actually…it is this guy

6. Because of this!
“It takes 10K hrs to become
an expert”
Malcom Gladwell

7. What he really said!
“Experts have on average spent
10K hrs learning their craft”
Malcom Gladwell

8. Pilots and Fireman do this most of their time

9. They even spend a lot of time doing this

10. Can you honestly answer this question?
Can you figure out if your
platform is in an error
state?

11. Chaos Engineering
The discipline of experimenting
on a distributed system in
order to build confidence in
the system’s capability to
withstand turbulent conditions
in production

12. The Grand Daddy
https://netflix.github.io/chaosmonkey/

13. Kubernete is so complex
it grew an entire (chaos) ecosystem

14. Look we did some science!
Published Result

15. How it started
Experiment #1

16. Experimental Design Matters
How representative is one gel to the next?
How many replicates are significant proof?
How about negative controls?
Is one side of this gel representative for the other?
Are the proteins separated enough?
Are these really the proteins we think they are?
Was this the right technique to begin with?
How much sample do we have for repeats?

17. Game Day Exercises
https://www.youtube.com/watch?v=xdiGW-RSb2w

18. Game analytics help guide training

19. Make that sidecar work for you

20. What have we learned so far
1. Athletes and Musicians
- Practice makes perfect
- Pre and Post game analytics will point at options to adjust and win
2. Pilots/Doctors/Fireman
- People respond differently to stress
- Life like simulations are critical
- Checklist will help you prevent making mistakes
3. Scientists
- Break problems down into known tasks
- Lies, Damn Lies and Statistics….and Biases
- Experimental Design matters

21. That’s all mighty fine, but...
Checklists are also
incredibly boring!

22. So let’s translate it to IT engineering
l Game Day Exercises
l (Non)-Destructive Tests
- Configuration Management / Infrastructure as Code
l Observability
- Metrics
- Logs
- Traces
l Living in the year 3000: Breaking production on purpose

23. Convince Management that:
Failure is normal and
expected behavior

24. Let engineers be scientists
Training
==
Testing
==
Monitoring

25. Get Engineers to be comfortable facing failur

26. When pressing $NewThing ™ into service
Create “real” DoD’s,
SLO’s + SLI’s
and Runbooks

27. Large complex systems will always
be in a (somewhat) degraded state

28. Contact
bram@attachmentgenie.com
@attachmentgenie
#2021-sre
https://www.slideshare.net/attachmentgenie