1. Practice and experimentation are important for building expertise, whether for athletes, musicians, pilots, doctors, or scientists. Pre- and post-analysis helps identify areas for improvement. 2. Simulations that mimic real-world stressors are critical for pilots, doctors, and firefighters to practice their skills and prevent mistakes. Checklists can help ensure important steps are not missed. 3. When experimenting scientifically, experimental design and accounting for biases are important. Replicates, controls, and representative samples help establish confidence in results.

1. Gamification of Chaos
Testing
Bram Vogelaar
@attachmentgenie

2. 29.74 Seconds
What could this number represent?

3. 99.97% Uptime?
That is a weird SLO?

4. Nope, it is this guy!

5. Well... actually...it is this guy

6. “It takes 10K hrs to
become an expert”
Malcom Gladwell
Because of this!

7. “Experts have on
average spent 10K hrs
learning their craft”
Malcom Gladwell
What he really said!

8. Pilots and Fireman do this most of their time

9. They even spend a lot of time doing this

10. Large complex systems will always be in a
degraded state

11. Can you figure out if your
platform is in an error
state?
Can you honestly answer this question?

12. The discipline of experimenting
on a distributed system in
order to build confidence in
the system’s capability to
withstand turbulent conditions
in production
Chaos Engineering

13. The grand daddy

14. K8S is so complex it grew an entire ecosystem

15. Look we did some science!
Published Result

16. How it started
Experiment #1

17. Experimental design matters
How representative is one gel to the next?
How many replicates are significant proof?
How about negative controls?
Is one side of this gel representative for the other?
Are the proteins separated enough?
Are these really the proteins we think they are?
Was this the right technique to begin with?
How much sample do we have for repeats?

18. Game Day Exercises
https://www.youtube.com/watch?v=xdiGW-RSb2w

19. Game analytics help guide training

20. Toxiproxy

21. Make that sidecar work for you

22. 1. Athletes and Musicians
– Practice makes perfect
– Pre and Post game analytics will point at options to adjust and win
2. Pilots/Doctors/Fireman
– People respond differently to stress
– Life like simulations are critical
– Checklist will help you prevent making mistakes
3. Scientists
– Break problems down into known tasks
– Lies, Damn Lies and Statistics….and Biases
– Experimental Design matters
What have we learned so far

23. Checklists are also
incredibly boring!
That’s all mighty fine, but...

24. ●
Game Day Exercises
●
(Non)-Destructive Tests
– Configuration Management / Infrastructure as Code
●
Observability
– Metrics
– Logs
– Traces
●
Living in the year 3000: Breaking production on purpose
So let’s translate it to IT engineering

25. Failure is normal and
expected behavior
Convince Management that:

26. Training
==
Testing
==
Monitoring
Let engineers be scientists

27. Get Engineers to be comfortable facing failure

28. Create “real” DoD’s
And
Runbooks
When pressing $NewThing ™ into service

29. bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie
Contact