Note: https://www.slideshare.net/TanyaReilly/the-history-of-fire-escapes-103869960 is an updated version of this talk.
This talk was a keynote at DevOps Days New York in 2018. If you like watching videos more than reading, the video is at https://www.youtube.com/watch?v=02KEKtc-5Dc.
When a datacenter goes offline, a server gets overloaded, or a binary hits a crashing bug, we usually have a contingency plan. We reduce damage, redirect traffic, page someone, drop low-priority requests, follow documented procedures. But why do many failures still come as a surprise? In this talk, we look at some real life analogs to preventing and managing software failures. Fire partitions. Public safety campaigns. Smoke alarms.
Sprinkler systems. Doors that say “This is not an exit”. And fire escapes. What can we learn from the real world about expecting failure and designing for it?