Successfully reported this slideshow.

The Upside of Downtime (Velocity 2010)

38

Share

•
Jun. 21, 2010
• 38 likes • 12,052 views
Upcoming SlideShare
NZISF Talk: Six essential security services
NZISF Talk: Six essential security services
Loading in …3
×
1 of 229
1 of 229

The Upside of Downtime (Velocity 2010)

Jun. 21, 2010
• 38 likes • 12,052 views

38

Share

Download to read offline

Technology Business

Your site will inevitably go down, and thanks to services like Twitter there's a good chance that all of your customers will find out about it. This presentation aims to help you turn that downtime into an opportunity to build trust with your users.

Your site will inevitably go down, and thanks to services like Twitter there's a good chance that all of your customers will find out about it. This presentation aims to help you turn that downtime into an opportunity to build trust with your users.

Technology Business
License: CC Attribution-ShareAlike License

Recommended

More Related Content

More from Lenny Rachitsky

Your Mind Is In Your Pants - Mind 2.0
Lenny Rachitsky
Street fight west summit mind 2.0
Lenny Rachitsky
Filter Failure
Lenny Rachitsky
Bitnorth 2011
Lenny Rachitsky
Mind 2.0: Your mind is #InYourPants
Lenny Rachitsky
TEDxConcordia - Losing Serendipity
Lenny Rachitsky
Localmind pitch at NewTech Montreal
Lenny Rachitsky
Losing Serendipity (Bitnorth 2010)
Lenny Rachitsky
Upside of Downtime Preparation Framework
Lenny Rachitsky
Google App Engine - Simple Introduction
Lenny Rachitsky
The Cloud - An introduction
Lenny Rachitsky
How to Trust the Cloud
Lenny Rachitsky
The Power of Story, Part 1
Lenny Rachitsky
Getting Things Done - Intro
Lenny Rachitsky
The White City - Chicago World Fair of 1893
Lenny Rachitsky
Influence - Robert Cialdini
Lenny Rachitsky
Twitter - An Intro
Lenny Rachitsky

Featured

What to Upload to SlideShare
SlideShare
Be A Great Product Leader (Amplify, Oct 2019)
Adam Nash
Trillion Dollar Coach Book (Bill Campbell)
Eric Schmidt
APIdays Paris 2019 - Innovation @ scale, APIs as Digital Factories' New Machi...
apidays
A few thoughts on work life-balance
Wim Vanderbauwhede
Is vc still a thing final
Mark Suster
The GaryVee Content Model
Gary Vaynerchuk
Mammalian Brain Chemistry Explains Everything
Loretta Breuning, PhD
Blockchain + AI + Crypto Economics Are We Creating a Code Tsunami?
Dinis Guarda
The AI Rush
Jean-Baptiste Dumont
AI and Machine Learning Demystified by Carol Smith at Midwest UX 2017
Carol Smith
10 facts about jobs in the future
Pew Research Center's Internet & American Life Project
Harry Surden - Artificial Intelligence and Law Overview
Harry Surden
Inside Google's Numbers in 2017
Rand Fishkin
Pinot: Realtime Distributed OLAP datastore
Kishore Gopalakrishna
How to Become a Thought Leader in Your Niche
Leslie Samuel
Visual Design with Data
Seth Familian
Designing Teams for Emerging Challenges
Aaron Irizarry
UX, ethnography and possibilities: for Libraries, Museums and Archives
Ned Potter
Winners and Losers - All the (Russian) President's Men
Ian Bremmer

Related Books

Free with a 14 day trial from Scribd

See all
The Future Is Faster Than You Think: How Converging Technologies Are Transforming Business, Industries, and Our Lives Peter H. Diamandis
(4.5/5)
Free
SAM: One Robot, a Dozen Engineers, and the Race to Revolutionize the Way We Build Jonathan Waldman
(5/5)
Free
From Gutenberg to Google: The History of Our Future Tom Wheeler
(3.5/5)
Free
Talk to Me: How Voice Computing Will Transform the Way We Live, Work, and Think James Vlahos
(4/5)
Free
So You Want to Start a Podcast: Finding Your Voice, Telling Your Story, and Building a Community That Will Listen Kristen Meinzer
(3.5/5)
Free
Bezonomics: How Amazon Is Changing Our Lives and What the World's Best Companies Are Learning from It Brian Dumaine
(4/5)
Free
No Filter: The Inside Story of Instagram Sarah Frier
(4.5/5)
Free
Autonomy: The Quest to Build the Driverless Car—And How It Will Reshape Our World Lawrence D. Burns
(5/5)
Free
Future Presence: How Virtual Reality Is Changing Human Connection, Intimacy, and the Limits of Ordinary Life Peter Rubin
(4/5)
Free
Live Work Work Work Die: A Journey into the Savage Heart of Silicon Valley Corey Pein
(4.5/5)
Free
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are Seth Stephens-Davidowitz
(4.5/5)
Free
Life After Google: The Fall of Big Data and the Rise of the Blockchain Economy George Gilder
(4/5)
Free
Believe IT: How to Go from Underestimated to Unstoppable Jamie Kern Lima
(4.5/5)
Free
We Should All Be Millionaires: A Woman’s Guide to Earning More, Building Wealth, and Gaining Economic Power Rachel Rodgers
(4.5/5)
Free
Authentic: A Memoir by the Founder of Vans Louise Maclellan
(4.5/5)
Free
Made in China: A Prisoner, an SOS Letter, and the Hidden Cost of America's Cheap Goods Amelia Pang
(4.5/5)
Free

Related Audiobooks

Free with a 14 day trial from Scribd

See all
If Then: How the Simulmatics Corporation Invented the Future Jill Lepore
(4.5/5)
Free
A Brief History of Motion: From the Wheel, to the Car, to What Comes Next Tom Standage
(4/5)
Free
An Ugly Truth: Inside Facebook’s Battle for Domination Sheera Frenkel
(4.5/5)
Free
Spooked: The Trump Dossier, Black Cube, and the Rise of Private Spies Barry Meier
(4/5)
Free
Second Nature: Scenes from a World Remade Nathaniel Rich
(5/5)
Free
Test Gods: Virgin Galactic and the Making of a Modern Astronaut Nicholas Schmidle
(5/5)
Free
Driven: The Race to Create the Autonomous Car Alex Davies
(4.5/5)
Free
Einstein's Fridge: How the Difference Between Hot and Cold Explains the Universe Paul Sen
(4.5/5)
Free
Dignity in a Digital Age: Making Tech Work for All of Us Ro Khanna
(4/5)
Free
After Steve: How Apple Became a Trillion-Dollar Company and Lost its Soul Tripp Mickle
(4.5/5)
Free
User Friendly: How the Hidden Rules of Design Are Changing the Way We Live, Work, and Play Cliff Kuang
(4.5/5)
Free
Uncanny Valley: A Memoir Anna Wiener
(4/5)
Free
Blockchain: The Next Everything Stephen P. Williams
(4/5)
Free
Lean Out: The Truth About Women, Power, and the Workplace Marissa Orr
(4.5/5)
Free
A World Without Work: Technology, Automation, and How We Should Respond Daniel Susskind
(4.5/5)
Free
The Players Ball: A Genius, a Con Man, and the Secret History of the Internet's Rise David Kushner
(4.5/5)
Free

The Upside of Downtime (Velocity 2010)

  1. 1. http://gapingvoid.com/ Sunday, June 20, 2010
  2. 2. The Upside of Downtime Turning disaster into opportunity Sunday, June 20, 2010
  3. 3. Who’s had a site go down? Sunday, June 20, 2010
  4. 4. Who’s hasn’t had a site go down? Sunday, June 20, 2010
  5. 5. There’s always that one guy! Sunday, June 20, 2010
  6. 6. Sunday, June 20, 2010
  7. 7. Sunday, June 20, 2010
  8. 8. Sunday, June 20, 2010
  9. 9. Sunday, June 20, 2010
  10. 10. Sunday, June 20, 2010
  11. 11. Sunday, June 20, 2010
  12. 12. Sunday, June 20, 2010
  13. 13. Sunday, June 20, 2010
  14. 14. Sunday, June 20, 2010
  15. 15. Downtime sucks Source: http://www.motivatedphotos.com/?id=8080 Sunday, June 20, 2010
  16. 16. Why downtime sucks Business $3,000 $2,250 $1,500 Sales $750 $0 0 2 4 6 8 10 12 14 16 18 20 22 Sunday, June 20, 2010
  17. 17. Why downtime sucks Business Brand Sunday, June 20, 2010
  18. 18. Why downtime sucks Business Brand You Sunday, June 20, 2010
  19. 19. Why downtime sucks Business Brand You Users Sunday, June 20, 2010
  20. 20. Downtime = Bad! (Duh) Sunday, June 20, 2010
  21. 21. Approach #1 Don’t fail Sunday, June 20, 2010
  22. 22. Source: http://kansansforlife.files.wordpress.com/2009/12/titanic.jpg Sunday, June 20, 2010
  23. 23. “Everything fails all the time” -- Werner Vogels (Amazon, CTO) Sunday, June 20, 2010
  24. 24. “Everything fails all the time” -- Werner Vogels (Amazon, CTO) Sunday, June 20, 2010
  25. 25. Your site will fail Werner Vogels (Amazon, CTO) Sunday, June 20, 2010
  26. 26. Why?!? Sunday, June 20, 2010
  27. 27. Why Failure Happens Risk Homeostasis Source: http://joshuahind.files.wordpress.com/2009/09/bicycle-crash.jpg Sunday, June 20, 2010
  28. 28. Why Failure Happens Risk Homeostasis Black Swan Source: Amazon.com Sunday, June 20, 2010
  29. 29. Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Source: http://www.apoliticus.com/wp-content/uploads/2009/01/6_21_080306_rumsfeld.jpg Sunday, June 20, 2010
  30. 30. Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Change Source: http://bozark.net/wordpress/wp-content/uploads/2008/09/barack_obama_change_fairey.jpg Sunday, June 20, 2010
  31. 31. Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Change Many small failures Source: http://www.biojobblog.com/uploads/image/dominos.jpg Sunday, June 20, 2010
  32. 32. Why Failure Happens Risk Homeostasis Black Swan Unknown unknowns Change Many small failures Humans Source: http://www.librarian.net/talks/clc/CLC.key/SJ_Shoulder_Shrug.jpg Sunday, June 20, 2010
  33. 33. Sunday, June 20, 2010
  34. 34. Sunday, June 20, 2010
  35. 35. Polisher blocked Not unusual Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  36. 36. Polisher Moisture leaks into blocked air system Not unusual Not expected Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  37. 37. Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Not good Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  38. 38. Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  39. 39. Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  40. 40. Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Dammit Relief valve broken Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  41. 41. Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Dammit Relief valve broken WTF Gauge broken Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  42. 42. Polisher Moisture leaks into Flow of cold water blocked air system stopped Not unusual Not expected Backup disabled Doh! Indicator blocked Dammit Relief valve broken Meltdown Gauge broken Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm Sunday, June 20, 2010
  43. 43. Sunday, June 20, 2010
  44. 44. Source: http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/03-Advanced_Failover_Architecture Sunday, June 20, 2010
  45. 45. “accidental power failure” Source: http://www.datacenterknowledge.com/archives/2010/06/16/power-failure-kos-intuit-sites-for-24-hours/ Sunday, June 20, 2010
  46. 46. “traffic accident damaged a nearby utility transformer” Source: http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-offline/ Sunday, June 20, 2010
  47. 47. “unfortunate code change” Source: http://www.datacenterknowledge.com/archives/2010/06/11/errant-code-change-crashes-10-million-blogs/ Sunday, June 20, 2010
  48. 48. Sunday, June 20, 2010
  49. 49. “Unhappy customers may get some attention, but unhappy networked customers can quickly impact your business” -- Clay Shirky Source: http://happenupon.files.wordpress.com/2009/02/technology-guru-clay-shir-001.jpg, http://scholarlykitchen.sspnet.org/2010/03/02/shirky-at-nfais-how-abundance-breaks-everything/ Sunday, June 20, 2010
  50. 50. Sunday, June 20, 2010
  51. 51. Sunday, June 20, 2010
  52. 52. Sunday, June 20, 2010
  53. 53. Sunday, June 20, 2010
  54. 54. Sunday, June 20, 2010
  55. 55. Sunday, June 20, 2010
  56. 56. http://labs.webmetrics.com/crowdsourceduptime Sunday, June 20, 2010
  57. 57. Sunday, June 20, 2010
  58. 58. Sunday, June 20, 2010
  59. 59. Sunday, June 20, 2010
  60. 60. Sunday, June 20, 2010
  61. 61. Recap Sunday, June 20, 2010
  62. 62. Your site will fail Sunday, June 20, 2010
  63. 63. Your site will fail + Downtime is bad Sunday, June 20, 2010
  64. 64. Your site will fail + Downtime is bad + Everyone will find out Sunday, June 20, 2010
  65. 65. Your site will fail + Downtime is bad + Everyone will find out = Screw it, I’ll become a lumberjack Source: http://sbadrinath.files.wordpress.com/2009/03/different26rqcu3.jpg Sunday, June 20, 2010
  66. 66. “Embrace fear of outages and degradation. Use it to guide your architecture, your code, your infrastructure. So lean into it.” -- John Allspaw, VP Tech. Ops at Etsy Sunday, June 20, 2010
  67. 67. Approach #2 Prepare for downtime Sunday, June 20, 2010
  68. 68. Disclaimer: Try hard to avoid downtime Sunday, June 20, 2010
  69. 69. Learning by example... Sunday, June 20, 2010
  70. 70. Case Study #1 Facebook Sunday, June 20, 2010
  71. 71. Sunday, June 20, 2010
  72. 72. Sunday, June 20, 2010
  73. 73. Sunday, June 20, 2010
  74. 74. Sunday, June 20, 2010
  75. 75. Sunday, June 20, 2010
  76. 76. Sunday, June 20, 2010
  77. 77. “The larger issue here isn't just that a portion of Facebook's platform has gone down - numerous web services have issues from time to time, including everything from Gmail to Twitter. An outage of this length, however, with no official communication from the company itself is disturbing.” -- N.Y. Times Sunday, June 20, 2010
  78. 78. Facebook Downtime Disturbing Sunday, June 20, 2010
  79. 79. Sunday, June 20, 2010
  80. 80. Case Study #2 Google App Engine Sunday, June 20, 2010
  81. 81. Sunday, June 20, 2010
  82. 82. Sunday, June 20, 2010
  83. 83. Sunday, June 20, 2010
  84. 84. Sunday, June 20, 2010
  85. 85. Sunday, June 20, 2010
  86. 86. Sunday, June 20, 2010
  87. 87. Sunday, June 20, 2010
  88. 88. Sunday, June 20, 2010
  89. 89. Sunday, June 20, 2010
  90. 90. Sunday, June 20, 2010
  91. 91. Sunday, June 20, 2010
  92. 92. Sunday, June 20, 2010
  93. 93. Sunday, June 20, 2010
  94. 94. Sunday, June 20, 2010
  95. 95. Google App Engine Downtime Kudos Sunday, June 20, 2010
  96. 96. Case Study #3 Atlassian Sunday, June 20, 2010
  97. 97. Sunday, June 20, 2010
  98. 98. Sunday, June 20, 2010
  99. 99. Sunday, June 20, 2010
  100. 100. Sunday, June 20, 2010
  101. 101. Sunday, June 20, 2010
  102. 102. Sunday, June 20, 2010
  103. 103. Sunday, June 20, 2010
  104. 104. Sunday, June 20, 2010
  105. 105. Sunday, June 20, 2010
  106. 106. Sunday, June 20, 2010
  107. 107. Sunday, June 20, 2010
  108. 108. Atlassian Downtime Bravo Sunday, June 20, 2010
  109. 109. http://atlassian.com/ Sunday, June 20, 2010
  110. 110. Downtime: Opportunity to Build Trust Sunday, June 20, 2010
  111. 111. Downtime: Opportunity to Destroy Trust Sunday, June 20, 2010
  112. 112. How To: Prepare for Downtime Sunday, June 20, 2010
  113. 113. Something > Nothing Sunday, June 20, 2010
  114. 114. Upside of Downtime Framework 1.0 Life is good Oh crap That sucked Time Sunday, June 20, 2010
  115. 115. Upside of Downtime Framework 1.0 Prepare Communicate Explain Time Sunday, June 20, 2010
  116. 116. Upside of Downtime Framework 1.0 Prepare Communicate Explain Time Sunday, June 20, 2010
  117. 117. Upside of Downtime Framework 1.0 Prepare Communicate Explain Time Sunday, June 20, 2010
  118. 118. Upside of Downtime Framework 1.0 Prepare Communicate Explain Time Sunday, June 20, 2010
  119. 119. Prepare Communicate Explain Sunday, June 20, 2010
  120. 120. Prepare Communicate Explain 1. Communication channel Sunday, June 20, 2010
  121. 121. Prepare Communicate Explain 1. Communication channel Something is Can’t tell if it’s I’ll assume it’s wrong me or you you You suck Sunday, June 20, 2010
  122. 122. Prepare Communicate Explain 1. Communication channel Something is Can’t tell if it’s I’ll assume it’s wrong me or you you Tell me when You suck a lot I know it’s you you’re back less Sunday, June 20, 2010
  123. 123. Sunday, June 20, 2010
  124. 124. Sunday, June 20, 2010
  125. 125. Sunday, June 20, 2010
  126. 126. Sunday, June 20, 2010
  127. 127. Sunday, June 20, 2010
  128. 128. Sunday, June 20, 2010
  129. 129. Sunday, June 20, 2010
  130. 130. Sunday, June 20, 2010
  131. 131. Prepare Communicate Explain 1. Communication channel Easy to find Sunday, June 20, 2010
  132. 132. Prepare Communicate Explain 1. Communication channel Easy to find Hosted off-site Sunday, June 20, 2010
  133. 133. Prepare Communicate Explain 1. Communication channel Easy to find Hosted off-site Real-time / automated Sunday, June 20, 2010
  134. 134. 7 keys for public health dashboards 1. Must show current status for each “service” 2. Data must be accurate and timely 3. Must be easy to find 4. Must provide details for events in real time 5. Provide historical uptime and performance data 6. Provide a way to be notified of status changes 7. Provide details on the data is gathered Source: http://www.transparentuptime.com/2008/11/rules-for-successful-public-health.html Sunday, June 20, 2010
  135. 135. Prepare Communicate Explain 1. Communication channel Easy to find Hosted off-site Real-time / automated 2. Process Sunday, June 20, 2010
  136. 136. Prepare Communicate Explain 1. Communication channel Easy to find Hosted off-site Real-time / automated 2. Process Authority Sunday, June 20, 2010
  137. 137. Prepare Communicate Explain 1. Communication channel Easy to find Hosted off-site Real-time / automated 2. Process Authority Mean-Time-To-Communicate (MTTC) Sunday, June 20, 2010
  138. 138. Prepare Communicate Explain 1. Communication channel Easy to find Hosted off-site Real-time / automated 2. Process Authority Mean-Time-To-Communicate (MTTC) On-call/drills/escalations/etc. Sunday, June 20, 2010
  139. 139. Your servers Sunday, June 20, 2010
  140. 140. Prepare Communicate Explain 1. Communicate Sunday, June 20, 2010
  141. 141. Prepare Communicate Explain 1. Communicate Use communication channel Sunday, June 20, 2010
  142. 142. Prepare Communicate Explain 1. Communicate Use communication channel MTTC Sunday, June 20, 2010
  143. 143. Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected Sunday, June 20, 2010
  144. 144. Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started Sunday, June 20, 2010
  145. 145. Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started ETA Sunday, June 20, 2010
  146. 146. Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started ETA Update regularly Sunday, June 20, 2010
  147. 147. Prepare Communicate Explain 1. Communicate Use communication channel MTTC Who/what is affected When the incident started ETA Update regularly 2. Fix it! Sunday, June 20, 2010
  148. 148. Phew, close one! Sunday, June 20, 2010
  149. 149. Prepare Communicate Explain 1. Postmortem Sunday, June 20, 2010
  150. 150. Prepare Communicate Explain 1. Postmortem Admit failure Source: http://en.blog.wordpress.com/2010/02/19/wp-com-downtime-summary/ Sunday, June 20, 2010
  151. 151. Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Source: http://www.bureauofcommunication.com/compose/apology Sunday, June 20, 2010
  152. 152. Prepare Communicate Explain “We apologize for any inconvenience this may have caused” Sunday, June 20, 2010
  153. 153. Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf Sunday, June 20, 2010
  154. 154. Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted Source: http://techcrunch.com/2009/11/02/large-scale-downtime-at-rackspace-cloud/ Sunday, June 20, 2010
  155. 155. Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Source: http://www.zendesk.com/2010/03/tuesday-double-whammy.html Sunday, June 20, 2010
  156. 156. Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Lessons learned Source: http://graysky.org/2010/02/downtime-postmortem/ Sunday, June 20, 2010
  157. 157. Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Lessons learned Sunday, June 20, 2010
  158. 158. Prepare Communicate Explain “I was completely overwhelmed by the amount of positive feedback and support I received.” Sunday, June 20, 2010
  159. 159. Prepare Communicate Explain 1. Postmortem Admit failure Sound like a human Start time and end time Who/what was impacted What went wrong Lessons learned 2. Improve for the future Sunday, June 20, 2010
  160. 160. Prepare Communicate Explain “Google is not just saying sorry, they are actually implementing serious changes which probably represents millions of dollars of development to help make sure this doesn't happen again.” Source: http://news.ycombinator.com/item?id=1168493 Sunday, June 20, 2010
  161. 161. Prepare Communicate Explain Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf Sunday, June 20, 2010
  162. 162. Prepare Communicate Explain Be human Sunday, June 20, 2010
  163. 163. Prepare Communicate Explain Be authentic Sunday, June 20, 2010
  164. 164. Prepare Communicate Explain Be transparent Sunday, June 20, 2010
  165. 165. Prepare Communicate Explain Accept responsibility Sunday, June 20, 2010
  166. 166. Prepare Communicate Explain Learn and improve Sunday, June 20, 2010
  167. 167. Prepare Communicate Explain Trust Sunday, June 20, 2010
  168. 168. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  169. 169. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Be Prepared + Be Transparent + Be Human Sunday, June 20, 2010
  170. 170. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Be Prepared + Be Transparent + Be Human = Sunday, June 20, 2010 Trust
  171. 171. Disclaimer: Don’t screw up too often Sunday, June 20, 2010
  172. 172. Sunday, June 20, 2010
  173. 173. Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Not Caught Sunday, June 20, 2010
  174. 174. Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Not Caught Win Sunday, June 20, 2010
  175. 175. Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Big Loss Not Caught Win Sunday, June 20, 2010
  176. 176. Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Big Win Big Loss Not Caught Win Sunday, June 20, 2010
  177. 177. Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Big Win Big Loss Not Caught Win Win Sunday, June 20, 2010
  178. 178. Downtime Prisoner’s Dilemma Transparent Not Transparent Caught Big Win Big Loss Not Caught Win Win Sunday, June 20, 2010
  179. 179. Benefits Gain trust Reduce churn, increase loyalty Reduce support costs Ability to control the message Competitive advantage More time to focus on the actual problem Reduce stress Sunday, June 20, 2010
  180. 180. Change != Easy Sunday, June 20, 2010
  181. 181. Change != Impossible Sunday, June 20, 2010
  182. 182. Keys to Adoption Getting past a culture of “hide the problem” Sunday, June 20, 2010
  183. 183. Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Sunday, June 20, 2010
  184. 184. Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Available resources to improve Sunday, June 20, 2010
  185. 185. Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Available resources to improve Pain Sunday, June 20, 2010
  186. 186. Keys to Adoption Getting past a culture of “hide the problem” Overriding commitment to want to improve Available resources to improve Pain Buy-in Sunday, June 20, 2010
  187. 187. Product Management Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010
  188. 188. Product Default: Lets wait for complaints Management Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010
  189. 189. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010
  190. 190. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010
  191. 191. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Operations Sales/ Marketing Sunday, June 20, 2010
  192. 192. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Sales/ Marketing Sunday, June 20, 2010
  193. 193. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Reality: Opportunity to learn/improve Sales/ Marketing Sunday, June 20, 2010
  194. 194. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Reality: Opportunity to learn/improve Sales/ Default: I don’t want my customers to know Marketing Sunday, June 20, 2010
  195. 195. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Reality: Opportunity to learn/improve Sales/ Default: I don’t want my customers to know Marketing Reality: They’ll find out, better from us Sunday, June 20, 2010
  196. 196. Product Default: Lets wait for complaints Management Reality: Proactiveness => Forgiveness Default: Too much work Support Reality: More upfront, less when it matters Engineering/ Default: Don’t want to look bad Operations Reality: Opportunity to learn/improve Sales/ Default: I don’t want my customers to know Marketing Reality: They’ll find out, better from us Sunday, June 20, 2010
  197. 197. Source: http://delicious.com/lennysan/healthdashboard Sunday, June 20, 2010
  198. 198. Simple as that! Sunday, June 20, 2010
  199. 199. Your site will still fail! Sunday, June 20, 2010
  200. 200. “The measure of a society is how well it transforms pain and suffering into something worthwhile.” -- Fredrick Nietzsche Sunday, June 20, 2010
  201. 201. “The measure of a company is how well it transforms pain of downtime into something worthwhile.” -- Lenny Rachitsky Source: Original quote inspired by Fredrick Nietzsche Sunday, June 20, 2010
  202. 202. Bare minimum: Register a Twitter account Sunday, June 20, 2010
  203. 203. Thank You Slides: http://bit.ly/upside-of-downtime Lenny Rachitsky @lennysan http://www.transparentuptime.com/ Webmetrics/Neustar @webmetrics http://www.webmetrics.com/ Sunday, June 20, 2010
  204. 204. Bonus Sunday, June 20, 2010
  205. 205. Sunday, June 20, 2010
  206. 206. Sunday, June 20, 2010
  207. 207. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  208. 208. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  209. 209. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  210. 210. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve "Unlikely that an accidental surface or subsurface oil spill would occur from the proposed activities" -- Exploration and environmental impact plan Source: http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion Sunday, June 20, 2010
  211. 211. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  212. 212. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  213. 213. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  214. 214. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  215. 215. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  216. 216. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  217. 217. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  218. 218. Upside of Downtime Framework 1.0 Prepare Communicate Explain 1. Communication channel 1. Communicate 1. Post-mortem - Easy to find - Use channel - Admit failure - Off-site - M.T.T.C. - Sound like a human - Real-time - Who/what affected - Start time and end time - When started - Who/what was impacted 2. Process - ETA to resolution - What went wrong - Give authority - Update regularly - Lessons learned - M.T.T.C. - On-call/escalations 2. Fix it! 2. Learn and improve Sunday, June 20, 2010
  219. 219. “Be not afraid of transparency; some are born transparent, some achieve transparency, and others have transparency thrust upon them.” -- Burrowed from William Shakespeare Sunday, June 20, 2010
  220. 220. Sunday, June 20, 2010
  221. 221. Making change 1. Find the bright spots - (this presentation has a bunch) Sunday, June 20, 2010
  222. 222. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) Sunday, June 20, 2010
  223. 223. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) Sunday, June 20, 2010
  224. 224. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) Sunday, June 20, 2010
  225. 225. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) Sunday, June 20, 2010
  226. 226. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) Sunday, June 20, 2010
  227. 227. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) 7. Tweak the environment - (create a simple process) Sunday, June 20, 2010
  228. 228. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) 7. Tweak the environment - (create a simple process) 8. Build habits - (build process organically) Sunday, June 20, 2010
  229. 229. Making change 1. Find the bright spots - (this presentation has a bunch) 2. Script the critical moves - (framework) 3. Point to the destination - (W.W.G.D.) 4. Find the feeling - (how would you feel?) 5. Shrink the change - (start small) 6. Grow your people - (everyone is learning as they go) 7. Tweak the environment - (create a simple process) 8. Build habits - (build process organically) 9. Rally the herd - (get buy in, rest will follow) Sunday, June 20, 2010

×