The Upside of Downtime (Velocity 2010)

http://gapingvoid.com/ Sunday, June 20, 2010

The Upside of Downtime Turning disaster into opportunity Sunday, June 20, 2010

Who’s had a site go down? Sunday, June 20, 2010

Who’s hasn’t had a site go down? Sunday, June 20, 2010

There’s always that one guy! Sunday, June 20, 2010

Sunday, June 20, 2010

Downtime sucks Source: http://www.motivatedphotos.com/?id=8080 Sunday, June 20, 2010

Why downtime sucks Business $3,000 $2,250 $1,500 Sales $750 $0 0 2 4 6 8 10 12 14 16 18 20 22 Sunday, June 20, 2010

Why downtime sucks Business Brand Sunday, June 20, 2010

Why downtime sucks Business Brand You Sunday, June 20, 2010

Why downtime sucks Business Brand You Users Sunday, June 20, 2010

Downtime = Bad! (Duh) Sunday, June 20, 2010

Approach #1 Don’t fail Sunday, June 20, 2010

Source: http://kansansforlife.ﬁles.wordpress.com/2009/12/titanic.jpg Sunday, June 20, 2010

“Everything fails all the time” -- Werner Vogels (Amazon, CTO) Sunday, June 20, 2010

Your site will fail Werner Vogels (Amazon, CTO) Sunday, June 20, 2010

Why?!? Sunday, June 20, 2010

Why Failure Happens Risk Homeostasis Source: http://joshuahind.ﬁles.wordpress.com/2009/09/bicycle-crash.jpg Sunday, June 20, 2010

Why Failure Happens Risk Homeostasis Black Swan Source: Amazon.com Sunday, June 20, 2010

Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Source: http://www.apoliticus.com/wp-content/uploads/2009/01/6_21_080306_rumsfeld.jpg Sunday, June 20, 2010

Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Change Source: http://bozark.net/wordpress/wp-content/uploads/2008/09/barack_obama_change_fairey.jpg Sunday, June 20, 2010

Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Change Many small failures Source: http://www.biojobblog.com/uploads/image/dominos.jpg Sunday, June 20, 2010

Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Change Many small failures Humans Source: http://www.librarian.net/talks/clc/CLC.key/SJ_Shoulder_Shrug.jpg Sunday, June 20, 2010

Polisher blocked Not unusual Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Polisher Moisture leaks into blocked air system Not unusual Not expected Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Not good Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Dammit Relief valve broken Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Dammit Relief valve broken WTF Gauge broken Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Dammit Relief valve broken Meltdown Gauge broken Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010

Source: http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/03-Advanced_Failover_Architecture Sunday, June 20, 2010

“accidental power failure” Source: http://www.datacenterknowledge.com/archives/2010/06/16/power-failure-kos-intuit-sites-for-24-hours/ Sunday, June 20, 2010

“trafﬁc accident damaged a nearby utility transformer” Source: http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-ofﬂine/ Sunday, June 20, 2010

“unfortunate code change” Source: http://www.datacenterknowledge.com/archives/2010/06/11/errant-code-change-crashes-10-million-blogs/ Sunday, June 20, 2010

“Unhappy customers may get some attention, but unhappy networked customers can quickly impact your business” -- Clay Shirky Source: http://happenupon.ﬁles.wordpress.com/2009/02/technology-guru-clay-shir-001.jpg, http://scholarlykitchen.sspnet.org/2010/03/02/shirky-at-nfais-how-abundance-breaks-everything/ Sunday, June 20, 2010

http://labs.webmetrics.com/crowdsourceduptime Sunday, June 20, 2010

Recap Sunday, June 20, 2010

Your site will fail Sunday, June 20, 2010

Your site will fail + Downtime is bad Sunday, June 20, 2010

Your site will fail + Downtime is bad + Everyone will ﬁnd out Sunday, June 20, 2010

Your site will fail + Downtime is bad + Everyone will ﬁnd out = Screw it, I’ll become a lumberjack Source: http://sbadrinath.ﬁles.wordpress.com/2009/03/different26rqcu3.jpg Sunday, June 20, 2010

“Embrace fear of outages and degradation. Use it to guide your architecture, your code, your infrastructure. So lean into it.” -- John Allspaw, VP Tech. Ops at Etsy Sunday, June 20, 2010

Approach #2 Prepare for downtime Sunday, June 20, 2010

Disclaimer: Try hard to avoid downtime Sunday, June 20, 2010

Learning by example... Sunday, June 20, 2010

Case Study #1 Facebook Sunday, June 20, 2010

“The larger issue here isn't just that a portion of Facebook's platform has gone down - numerous web services have issues from time to time, including everything from Gmail to Twitter. An outage of this length, however, with no ofﬁcial communication from the company itself is disturbing.” -- N.Y. Times Sunday, June 20, 2010

Facebook Downtime Disturbing Sunday, June 20, 2010

Case Study #2 Google App Engine Sunday, June 20, 2010

Google App Engine Downtime Kudos Sunday, June 20, 2010

Case Study #3 Atlassian Sunday, June 20, 2010

Atlassian Downtime Bravo Sunday, June 20, 2010

http://atlassian.com/ Sunday, June 20, 2010

Downtime: Opportunity to Build Trust Sunday, June 20, 2010

Downtime: Opportunity to Destroy Trust Sunday, June 20, 2010

How To: Prepare for Downtime Sunday, June 20, 2010

Something > Nothing Sunday, June 20, 2010

Upside of Downtime Framework 1.0 Life is good Oh crap That sucked Time Sunday, June 20, 2010

Upside of Downtime Framework 1.0 Prepare Communicate Explain Time Sunday, June 20, 2010

Prepare Communicate Explain Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Something is Can’t tell if it’s I’ll assume it’s wrong me or you you You suck Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Something is Can’t tell if it’s I’ll assume it’s wrong me or you you Tell me when You suck a lot I know it’s you you’re back less Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Easy to ﬁnd Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Easy to ﬁnd Hosted off-site Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Easy to ﬁnd Hosted off-site Real-time / automated Sunday, June 20, 2010

7 keys for public health dashboards 1. Must show current status for each “service” 2. Data must be accurate and timely 3. Must be easy to ﬁnd 4. Must provide details for events in real time 5. Provide historical uptime and performance data 6. Provide a way to be notiﬁed of status changes 7. Provide details on the data is gathered Source: http://www.transparentuptime.com/2008/11/rules-for-successful-public-health.html Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Easy to ﬁnd Hosted off-site Real-time / automated 2. Process Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Easy to ﬁnd Hosted off-site Real-time / automated 2. Process Authority Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Easy to ﬁnd Hosted off-site Real-time / automated 2. Process Authority Mean-Time-To-Communicate (MTTC) Sunday, June 20, 2010

Prepare Communicate Explain 1. Communication channel Easy to ﬁnd Hosted off-site Real-time / automated 2. Process Authority Mean-Time-To-Communicate (MTTC) On-call/drills/escalations/etc. Sunday, June 20, 2010

Your servers Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Use communication channel Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Use communication channel MTTC Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started ETA Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started ETA Update regularly Sunday, June 20, 2010

Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started ETA Update regularly 2. Fix it! Sunday, June 20, 2010

Phew, close one! Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Source: http://en.blog.wordpress.com/2010/02/19/wp-com-downtime-summary/ Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Source: http://www.bureauofcommunication.com/compose/apology Sunday, June 20, 2010

Prepare Communicate Explain “We apologize for any inconvenience this may have caused” Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted Source: http://techcrunch.com/2009/11/02/large-scale-downtime-at-rackspace-cloud/ Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Source: http://www.zendesk.com/2010/03/tuesday-double-whammy.html Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Lessons learned Source: http://graysky.org/2010/02/downtime-postmortem/ Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Lessons learned Sunday, June 20, 2010

Prepare Communicate Explain “I was completely overwhelmed by the amount of positive feedback and support I received.” Sunday, June 20, 2010

Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Lessons learned 2. Improve for the future Sunday, June 20, 2010

Prepare Communicate Explain “Google is not just saying sorry, they are actually implementing serious changes which probably represents millions of dollars of development to help make sure this doesn't happen again.” Source: http://news.ycombinator.com/item?id=1168493 Sunday, June 20, 2010

Prepare Communicate Explain Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf Sunday, June 20, 2010

Prepare Communicate Explain Be human Sunday, June 20, 2010

Prepare Communicate Explain Be authentic Sunday, June 20, 2010

Prepare Communicate Explain Be transparent Sunday, June 20, 2010

Prepare Communicate Explain Accept responsibility Sunday, June 20, 2010

Prepare Communicate Explain Learn and improve Sunday, June 20, 2010

Prepare Communicate Explain Trust Sunday, June 20, 2010

Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to ﬁnd - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010

Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to ﬁnd - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Be Prepared + Be Transparent + Be Human Sunday, June 20, 2010

Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to ﬁnd - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Be Prepared + Be Transparent + Be Human = Sunday, June 20, 2010 Trust

Disclaimer: Don’t screw up too often Sunday, June 20, 2010

Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Not Caught Sunday, June 20, 2010

Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Not Caught Win Sunday, June 20, 2010

Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Big Loss Not Caught Win Sunday, June 20, 2010

Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Big Win Big Loss Not Caught Win Sunday, June 20, 2010

Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Big Win Big Loss Not Caught Win Win Sunday, June 20, 2010

Beneﬁts Gain trust Reduce churn, increase loyalty Reduce support costs Ability to control the message Competitive advantage More time to focus on the actual problem Reduce stress Sunday, June 20, 2010

Change != Easy Sunday, June 20, 2010

Change != Impossible Sunday, June 20, 2010

Keys to Adoption Getting past a culture of “hide the problem” Sunday, June 20, 2010

Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Sunday, June 20, 2010

Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Available resources to improve Sunday, June 20, 2010

Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Available resources to improve Pain Sunday, June 20, 2010

Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Available resources to improve Pain Buy-in Sunday, June 20, 2010

Product Management Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Sales/ Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Reality: Opportunity to learn/improve Sales/ Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Reality: Opportunity to learn/improve Sales/ Default: I don’t want my customers to know Marketing Sunday, June 20, 2010

Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Reality: Opportunity to learn/improve Sales/ Default: I don’t want my customers to know Marketing Reality: They’ll ﬁnd out, better from us Sunday, June 20, 2010

Source: http://delicious.com/lennysan/healthdashboard Sunday, June 20, 2010

Simple as that! Sunday, June 20, 2010

Your site will still fail! Sunday, June 20, 2010

“The measure of a society is how well it transforms pain and suffering into something worthwhile.” -- Fredrick Nietzsche Sunday, June 20, 2010

“The measure of a company is how well it transforms pain of downtime into something worthwhile.” -- Lenny Rachitsky Source: Original quote inspired by Fredrick Nietzsche Sunday, June 20, 2010

Bare minimum: Register a Twitter account Sunday, June 20, 2010

Thank You Slides: http://bit.ly/upside-of-downtime Lenny Rachitsky @lennysan http://www.transparentuptime.com/ Webmetrics/Neustar @webmetrics http://www.webmetrics.com/ Sunday, June 20, 2010

Bonus Sunday, June 20, 2010

Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to ﬁnd - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve "Unlikely that an accidental surface or subsurface oil spill would occur from the proposed activities" -- Exploration and environmental impact plan Source: http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion Sunday, June 20, 2010

“Be not afraid of transparency; some are born transparent, some achieve transparency, and others have transparency thrust upon them.” -- Burrowed from William Shakespeare Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) 7. Tweak the environment - (create a simple process) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) 7. Tweak the environment - (create a simple process) 8. Build habits - (build process organically) Sunday, June 20, 2010

Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) 7. Tweak the environment - (create a simple process) 8. Build habits - (build process organically) 9. Rally the herd - (get buy in, rest will follow) Sunday, June 20, 2010

http://www.royans.net	289
http://www.transparentuptime.com	143
http://velocityconf.com	72
http://hutuworm.blogspot.com	28
http://guillaumelerouge.com	19
http://textontechs.com	18
http://xianguo.com	14
http://www.startupschool.tv	5
http://static.slidesharecdn.com	4
http://flavors.me	2
http://www.blogger.com	2
http://reader.youdao.com	2
http://www.slideshare.net	1
http://feeds2.feedburner.com	1
http://glerouge.tumblr.com	1
http://feeds.feedburner.com	1

The Upside of Downtime (Velocity 2010)

by Lenny Rachitsky

Accessibility

Categories

Upload Details

Usage Rights

16 Embeds 602

Statistics

The Upside of Downtime (Velocity 2010) Presentation Transcript