The document outlines five things not to do when ensuring high availability of systems and services. Specifically: 1) Don't make systems impossible to break as human error is inevitable. 2) Don't put up artificial barriers that just create more process without solving problems. 3) Don't think of failures as exceptional but rather assume they will occur. 4) Don't try to make problems go away by panicking or rebooting without understanding the issue. 5) Don't harass debuggers by constantly demanding status updates or fixes without giving them time to properly diagnose and resolve issues. The document recommends trusting engineers, planning for failures, gathering evidence before taking action, and investing in effective incident management.

1. Five NEINs
of Availability
Tomer Gabel
DevOpsDays Tel-Aviv, 2019

2. Who am I?

3. MAKE IT IMPOSSIBLE TO BREAK
1. Don’t…
Image by Bert Heymans on Flickr (CC BY-NC-ND 2.0)

4. • Human error is inevitable
• More process won’t solve the problem
• More process will screw you over
• Avoid the process entirely, or provide
means of circumventing it

5. PUT ARTIFICIAL BARRIERS IN PLACE
2. Don’t…
Image by Jessica BKK on Flickr (CC BY 2.0)

6. • Regulation
– Apply only where required, and/or
– Allow access with proper logging/auditing. It’ll
end up cheaper
• Lack of trust
– You trust them to build it, but not to operate it?
– Won’t help you anyway
Barriers

7. THINK OF
FAILURES AS
EXCEPTIONAL
3. Don’t…
Image by Piratenmensch on Flickr (CC BY-SA 2.0)

8. Source: AWS case study by Slack

11. MAKE PROBLEMS
GO AWAY
4. Don’t…
Top image by chaouki on Flickr (CC BY-SA 2.0), bottom image by Caffeinatrix on Flickr (CC BY-NC-ND 2.0)

12. Source: Imgflip

13. Panic Reboot
Nasty side
effects
Lost
opportunity
Bryan Cantrill
Debugging Microservices in
Production, QCon SF 2015

14. HARASS YOUR
DEBUGGERS
5. Don’t…
Image by mariana neri on Flickr (CC BY-NC-ND 2.0)

15. There’s an
issue! Fix it!
WHY ISN’T
IT FIXED
YET?
Is it
fixed yet?
Can you send a
status update?
Don’t forget to
save the
screenshots!

16. In conclusion
Don't…
1. Make it impossible to break
2. Put artificial barriers in place
3. Think of failures as exceptional
4. Make problems go away
5. Harass your debuggers

17. In conclusion
Don't…
1. Make it impossible to break
2. Put artificial barriers in place
3. Think of failures as exceptional
4. Make problems go away
5. Harass your debuggers
Do…
• Trust your engineers
• Assume and plan for failure
• Gather evidence before acting
• Invest in incident management

18. QUESTIONS?
Thank you for listening
tomer@tomergabel.com
@tomerg
On GitHub:
https://github.com/holograph
This work is licensed under a Creative
Commons Attribution-ShareAlike 4.0
International License.