Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Major Incident Management


Published on

January, 2019 North Coast HDI Chapter Meeting. Learn more on our website:

Published in: Technology
  • Be the first to comment

Major Incident Management

  1. 1. xMatters Major Incident Management Best practices Real world case studies Confidential and Proprietary
  2. 2. Hi…my name is Anne Deming Confidential and Proprietary 21/18/2019
  3. 3. How many MIM’s a month? Confidential and Proprietary 31/18/2019
  4. 4. A recent survey on Major Incidents •32% - once a year or less •42% - at least several times a year •13% - at least once a month •13% - at least once a week Confidential and Proprietary 41/18/2019
  5. 5. Must haves – we won’t bury the lead •Stakeholder Alignment •Rapid Engagement • Determine who and when to engage Identifying technical vs. management contacts •Intelligent Responses •Multi-modal Support •Integrations with Operations Confidential and Proprietary 51/18/2019
  6. 6. 6 Thechallengeyou face….(becauseyoursystemsare yourbrand) Confidential and Proprietary Poor communicationduringMajor Incidentscan seriouslyimpact IT
  7. 7. 7 CommonMajorIncidentCommunicationApproaches Confidential and Proprietary 7 Manual notifications Email and distribution lists mass notification No Precision No Reliability No Accountability“Kitchen Sink” approach results in alert fatigue Critical insights slip through the cracks Limited follow- through to ensure action
  8. 8. 8 Poor communicationduringa Major Incidenthas a bigimpact Extended downtimes & missed SLA Loss of revenue Employee burnout Tarnished image of IT Frustrated customers Resolvers not held accountable Confidential and Proprietary
  9. 9. A Case study for a digital communications company •Digital communications company •Over 32 million connections to its network •120 Million calls everyday in the network • Connects people, families & businesses (their customers) • Critical connectivity for tracking / payments on a transit system. Confidential and Proprietary 91/18/2019
  10. 10. HIGHLIGHTS on using xMatters •We transformed the way they manages incidents, with impressive results: •Reduced the number of clicks to engage stakeholders •2 minutes to send communications, an 88% reduction •3 minutes to engage stakeholders, an 85% reduction Confidential and Proprietary 101/18/2019
  11. 11. HOW? •A mobile app •Targeted messages •Automated escalations •Communications are easy-to-build, easy-to-send •Templated communications Confidential and Proprietary 111/18/2019
  12. 12. 1212Confidential and Proprietary There is a better way
  13. 13. Preparation - Creating effective communication plans •Impact of outsourcing within global teams and 3rd party vendors •Maintaining up-to-date contact info and schedules •Developing standard communication • Defining the communication content • pre-determined templates • pre-determined engagement timelines for communications • pre-determine the blast radius (all or nothing is a bad idea) Confidential and Proprietary 131/18/2019
  14. 14. Action – fire fighting •Activating the communication plan •Management of the recovery: Leadership roles, technical guidance •Inspiring rapid response from tech teams •Communicating effectively with necessary constituencies •Tracking communication plan status Confidential and Proprietary 141/18/2019
  15. 15. Reflection – the post mortem •Root Cause Analysis after a Major Incident •What types of information gathered during a major incident can be leveraged to avoid recurrence. •Tracking SLAs •Aggregating data for compliance/audit purposes •Remediating contact info errors •Driving improved performance – identifying weaknesses in the communication plan Confidential and Proprietary 151/18/2019
  16. 16. A cheat sheet of questions to start the dialogue • Who to contact and how to contact them. • Incident Manager…When you assemble folks how does this happen? • What about getting folks on the call? • Consider the Staffing schedule: skeleton crew, 8-5 or are you 24x7? • What is the timeframe for communication with the business? • What about escalations? How do you handle them? There is MTTE and time to become affective. Some MIMs take 30 hours. • A culture consideration: How to identify the right people to contact? Is it changing behavior or is enforcement an issue? • How do you measure if you are overwhelming one resource vs. another are you tracking? • How many steps are in your MIM process….Commonly we hear a 5 steps process. (Could yours be adjusted?) • How & when will you engage 3rd party (hosting providers or technology vendors you need to engage) in during a Major Incident? • When outsources escalate back to customer….How will you handle SLAs on the customer side? • Have you considered communication for upgrades or downgrades P2 to P1 or P1 to P2? • Is there criteria to go straight to MIM, red alert and bypass triage? Confidential and Proprietary 161/18/2019
  17. 17. Thank you….questions? Confidential and Proprietary 171/18/2019