Sink or Swim: How Incident Communication Keeps You Afloat
•
1 like•8,250 views
Downtime is stressful. Don't let a lack of communication add extra weight to your issues. Join Scott Klein, co-founder and head of product for StatusPage, as he shares best practices for communicating effectively during an incident. He'll offer tips for increasing communication efficiency, reducing ticket load, and how to use StatusPage alongside the rest of your Atlassian toolset. Finally, Scott will discuss the expert ingredients of an effective incident response team: pre-incident preparation and post-incident followup with affected users. Scott Klein, Head of Product - StatusPage, Atlassian
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Sink or Swim: How Incident Communication Keeps You Afloat
Similar to Sink or Swim: How Incident Communication Keeps You Afloat (9)
More from Atlassian
More from Atlassian (20)
Recently uploaded
Recently uploaded (20)
Sink or Swim: How Incident Communication Keeps You Afloat
- 1. SCOTT KLEIN | PRINCIPAL PRODUCT MANAGER, STATUSPAGE | @SCOOTKLEIN Sink or Swim How Incident Communication Keeps You Afloat
- 7. Communication is the newest requirement of the software industry
- 8. Agenda The life of an incident Discovering the missing ingredient Incident communication best practices StatusPage tour Managing incidents the Atlassian way
- 9. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong!
- 10. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates
- 11. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team
- 12. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem
- 13. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal!
- 14. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 15. Meanwhile, over in customer land 1 Angry customers
- 16. Meanwhile, over in customer land 2 Angry customers
- 17. Meanwhile, over in customer land 3 Angry customers
- 18. Meanwhile, over in customer land 5 Angry customers
- 19. Meanwhile, over in customer land 8 Angry customers
- 20. Meanwhile, over in customer land 13 Angry customers
- 21. Meanwhile, over in customer land 21 Angry customers
- 22. Meanwhile, over in customer land 34 Angry customers
- 23. Meanwhile, over in customer land 55 Angry customers
- 24. Meanwhile, over in customer land 84 Angry customers
- 25. Meanwhile, over in customer land 99+ 2 Angry customers Support tickets
- 26. Meanwhile, over in customer land 99+ 3 Angry customers Support tickets
- 27. Meanwhile, over in customer land 99+ 5 Angry customers Support tickets
- 28. Meanwhile, over in customer land 99+ 7 Angry customers Support tickets
- 29. Meanwhile, over in customer land 99+ 11 Angry customers Support tickets
- 30. Meanwhile, over in customer land 99+ 13 Angry customers Support tickets
- 31. Meanwhile, over in customer land 99+ 17 Angry customers Support tickets
- 32. Meanwhile, over in customer land 99+ 25 Angry customers Support tickets
- 33. Meanwhile, over in customer land 99+ 41 Angry customers Support tickets
- 34. Meanwhile, over in customer land 99+ 57 Angry customers Support tickets
- 35. Meanwhile, over in customer land 3 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 36. Meanwhile, over in customer land 5 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 37. Meanwhile, over in customer land 7 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 38. Meanwhile, over in customer land 11 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 39. Meanwhile, over in customer land 13 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 40. Meanwhile, over in customer land 17 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 41. Meanwhile, over in customer land 19 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 42. Meanwhile, over in customer land 23 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 43. Meanwhile, over in customer land 27 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 44. Meanwhile, over in customer land 29 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 45. Meanwhile, over in customer land 29 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 46. Meanwhile, over in customer land 29 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 47. THE LIFE OF AN INCIDENT: WHAT YOUR CUSTOMERS SEE Something’s wrong!
- 48. THE LIFE OF AN INCIDENT: WHAT YOUR CUSTOMERS SEE Something’s wrong! What’s going on?! Time to panic.
- 49. THE LIFE OF AN INCIDENT: WHAT YOUR CUSTOMERS SEE Something’s wrong! What’s going on?! Time to panic. Emails Support Tickets
- 50. THE LIFE OF AN INCIDENT: WHAT YOUR CUSTOMERS SEE Something’s wrong! What’s going on?! Time to panic. Emails Support Tickets Phone Calls Tweets
- 51. THE LIFE OF AN INCIDENT: WHAT YOUR CUSTOMERS SEE Something’s wrong! What’s going on?! Time to panic. Emails Support Tickets Phone Calls Tweets Assume the worst, be angry!
- 52. THE LIFE OF AN INCIDENT: WHAT YOUR CUSTOMERS SEE Something’s wrong! What’s going on?! Time to panic. Emails Support Tickets Phone Calls Tweets Assume the worst, be angry!
- 53. Agenda Incident communication best practices StatusPage tour Managing incidents the Atlassian way The life of an incident Discovering the missing ingredient
- 54. THE LIFE OF AN INCIDENT: NOW WITH MORE WORDS Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 55. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 56. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 57. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 58. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 59. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 60. THE LIFE OF AN INCIDENT: WHAT YOU SEE Something’s wrong! Monitoring tool alerts and escalates On-call responds and leverages team Whole team swarms to fix the problem Back to normal! Document what failed, remediate
- 61. Communication is the newest requirement of the software industry
- 62. Agenda Incident communication best practices Managing incidents the Atlassian way The life of an incident Discovering the missing ingredient StatusPage tour
- 71. status.twilio.com Email subscribers Every update to the incident will get sent to your subscriber list. Keeps everyone in the loop proactively.
- 74. status.twilio.com SMS subscribers Get open and resolved messages via SMS, with link to view more info. More than 140 countries supported!
- 75. SUBSCRIBER NOTIFICATIONS Incident title Link for more info Branding intact
- 76. status.twilio.com Webhook subscribers Programmatically receive HTTP POST requests whenever incident is created/updated or components are updated. Automate internal systems!
- 81. statuspage.io/survey-results 300 customers / 3 stats
- 82. 25% AVERAGE REDUCTION IN SUPPORT TICKETS
- 83. 50% ORGANIZATIONS WITH FASTER INCIDENT RESPONSE
- 84. 67% ORGANIZATIONS THAT INCREASED USER TRUST
- 85. Meanwhile, over in customer land 29 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 86. Meanwhile, over in customer land 27 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 87. Meanwhile, over in customer land 23 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 88. Meanwhile, over in customer land 19 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 89. Meanwhile, over in customer land 17 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 90. Meanwhile, over in customer land 13 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 91. Meanwhile, over in customer land 11 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 92. Meanwhile, over in customer land 7 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 93. Meanwhile, over in customer land 5 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 94. Meanwhile, over in customer land 3 99+ 57 Angry customers Support tickets Phone calls, live chat, tweets
- 95. Meanwhile, over in customer land 99+ 57 Angry customers Support tickets
- 96. Meanwhile, over in customer land 99+ 41 Angry customers Support tickets
- 97. Meanwhile, over in customer land 99+ 25 Angry customers Support tickets
- 98. Meanwhile, over in customer land 99+ 17 Angry customers Support tickets
- 99. Meanwhile, over in customer land 99+ 13 Angry customers Support tickets
- 100. Meanwhile, over in customer land 99+ 11 Angry customers Support tickets
- 101. Meanwhile, over in customer land 99+ 7 Angry customers Support tickets
- 102. Meanwhile, over in customer land 99+ 5 Angry customers Support tickets
- 103. Meanwhile, over in customer land 99+ 3 Angry customers Support tickets
- 104. Meanwhile, over in customer land 99+ 2 Angry customers Support tickets
- 105. Meanwhile, over in customer land 84 Angry customers
- 106. Meanwhile, over in customer land 55 Angry customers
- 107. Meanwhile, over in customer land 34 Angry customers
- 108. Meanwhile, over in customer land 21 Angry customers
- 109. Meanwhile, over in customer land 13 Angry customers
- 110. Meanwhile, over in customer land 8 Angry customers
- 111. Meanwhile, over in customer land 5 Angry customers
- 112. Meanwhile, over in customer land 3 Angry customers
- 113. Meanwhile, over in customer land 2 Angry customers
- 114. Meanwhile, over in customer land 0 Angry customers
- 115. Agenda Managing incidents the Atlassian way The life of an incident Discovering the missing ingredient StatusPage tour Incident communication best practices
- 116. Sink or Swim Before the Incident
- 117. Team composition Commander Communicator Set up communication channels And make sure the right people are using them :) Delegate ALL information/repair Not involved in doing any of the actual work Drive post-incident review and work Make sure this incident doesn’t happen again
- 118. Team composition Communicator Commander Post all customer-facing communications StatusPage, Twitter, high-value customers, etc Notify IC of new customers affected Ensure commander knows scope of issues Relay postmortem to customers Build trust with customers through beautiful followup
- 119. Sink or Swim During the Incident
- 120. Communication golden rules Early Often Precise Consistent Get something up as early as possible It doesn’t have to be perfect, but it should be accurate, and reassure that you’re looking into it.
- 121. Communication golden rules Often Precision Together Early Uncertainty = anxiety Even if no new information, continue communicating and follow through on your commitments
- 122. Communication golden rules Often Together Early Precision “The problem seems to be database related” “We’re still tracking down the root cause of the issue” “We think we fixed it, but need more time” “Verification of the fix is still underway.”
- 123. Communication golden rules Often Together Early Precision “Amazon is still having trouble with things” “We continue to work with our hosting provider” “We’re still trying to find Bob, who got us into this mess anyway” “We continue to investigate the problem”
- 124. Communication golden rules Often Precision Early Together Consistency breeds trust In the heat of the moment you won’t know what to say. Plan ahead with the whole team
- 125. Sink or Swim After the Incident
- 126. Postmortem golden rules Say sorry Don’t name names Be personal Details inspire confidence Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Close the loop
- 127. Postmortem golden rules Say sorry Don’t name names Be personal Details inspire confidence Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Close the loop
- 128. Postmortem golden rules Say sorry Don’t name names Be personal Details inspire confidence Close the loop
- 129. Postmortem golden rules Say sorry Don’t name names Be personal Details inspire confidence Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Close the loop
- 130. Postmortem golden rules Say sorry Don’t name names Be personal Details inspire confidence Shape usage Please only use circles, rectangles, and rounded rectangles to call attention to a particular part of a screenshot, for the sake of consistency. Close the loop
- 131. Agenda The life of an incident Discovering the missing ingredient StatusPage tour Incident communication best practices Managing incidents the Atlassian way
- 132. RESPOND WITH Prerequisites and context To-do list and checkpoints for completion
- 134. Detail every step Reduce mental overhead in a stressful situation
- 135. Clear ownership Specify who’s responsible for each step and how to trigger a hand-off
- 136. COMMUNICATE WITH Single source-of trust for customers and colleagues Always up-to-date view of your systems
- 137. Detail every step Reduce mental overhead in a stressful situation Communicate early It’s okay if you don’t know what went wrong yet—just show you’re on it
- 138. Detail every step Reduce mental overhead in a stressful situation Communicate often As you learn more about the incident, provide frequent and detailed updates
- 139. SWARM WITH YOUR TEAM IN Plugins to keep all of the responders in sync together Reconstruct timelines for postmortems and forensic report
- 140. Detail every step Reduce mental overhead in a stressful situation Use JIRA and HipChat together to get to resolution, faster Create a HipChat incident room directly from your JIRA issue (which automatically tags watchers), and see updates on the issue in the room
- 141. Automatically surface incidents on your service desk portal SUPPORT WITH
- 142. Automatically surface incidents on your service desk portal SUPPORT WITH
- 143. TRACK REMEDIATION WITH Clear owners and linked HipChat, Confluence, and VC SLAs to ensure response timing
- 145. Building a culture of transparency and openness
- 146. Accept that failure happens Things will fail, caused by both internal and external forces. Even the most advanced organizations are not immune.
- 147. Commit to the culture Adopt values of learning and improvement, using automated and repeatable processes, and tearing down gates and bureaucracy
- 148. Thank you! SCOTT KLEIN | PRINCIPAL PRODUCT MANAGER, STATUSPAGE | @SCOOTKLEIN