Andrew Vermes: Major Incident Management
•Download as PPTX, PDF•
0 likes•351 views
This document provides guidance on effectively managing major incidents by separating work into four key streams: understanding what's happening, restoring service, investigating causes, and preventing future incidents. It emphasizes keeping all relevant information visible, updated, and organized on a shared incident dashboard. An example dashboard is shown and updated as a simulated incident involving a broken brick sorting robot is resolved over 60 minutes. Lessons include avoiding long calls, transparency of information, specificity from participants, and regular simulations to maintain skills.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Andrew Vermes: Major Incident Management
Similar to Andrew Vermes: Major Incident Management (20)
More from itSMF UK
More from itSMF UK (20)
Recently uploaded
Recently uploaded (20)
Andrew Vermes: Major Incident Management
- 2. Time for some fun! • Why are major incidents so hard? • Straightforward ways to share information • A live, interactive simulation
- 3. Caught in headlights • It’s easy to panic • Calm atmosphere helps
- 4. Keep questions helpful When will it be fixed? Is everything under control? Who’s responsible for this? Did you change anything? Why did we let this happen?
- 5. Too much vital information is hidden Some more Emails Magic Happens Case Closed Case Opened Some Emails & Calls Sometimes the magic is “Spoke to Mike & Steve” Most case documentation does not include the actual root cause of a problem, therefore knowledge reuse is minimised
- 6. Processes must be second nature Emergency services depend on: •Clear processes •Checklists •Repeated training Limited time to think: needs responses to be fast and automatic
- 7. Four things need effective control: • What’s happening? • How will we restore service? • What risks are there? • What investigation is needed? Major Incidents Understanding InvestigatingRestoring Preventing
- 8. Separate the workstreams Four key questions: What’s happening? What will we do about it? What risks are there? How do we find the cause? The goal is always- effective action DECISION ANALYSIS To select the right fix or workaround POTENTIAL PROBLEM ANALYSIS To manage risksSITUATION APPRAISAL To Sort Out Priority Actions PROBLEM ANALYSIS To Find True Cause
- 9. Keep your working information visible • Separate into 4 areas
- 10. Keep information updated • Ensure your dashboard is updated in real time
- 11. Major Incident Simulation Please go to: http://mindstormrobotics.com/ You will find information about the incident there SITUATION AUGUST 30 Tom Lewis, production manager at Stockholm Brick Company, has contacted you because one of their brick sorting robots, the TX72-6, has a problem. You repaired the conveyor belt yesterday, but now the sorting system has broken down again. The breakdown costs the company roughly 1,000 dollars per minute. Tom requests your assistance to solve the problem immediately.
- 12. Teamwork is key In teams: • Review your information • Complete the Major Incident Dashboard • Decide what to do next • Take the necessary actions and update the dashboard
- 13. Updating your dashboard • Keep information visible to your team
- 14. Check point • How much has your team spent so far? • What progress has been made in incident recovery? • Which moves did you make that added value?
- 15. Wrap up Start with a clear incident statement (what process, what symptom) Look early for similar but unaffected CIs for comparison The solution is…
- 16. First 10 minutes KT INCIDENT DASHBOARD time: August 30, 11:15 Incident Summary: Customer Issues, Priorities, Impact - What When Where Actions Needed Due Done Who Diagnostic Data we have Possible Causes Next Investigation stepDue Done Who Bricksorting robot TX72-6 problems Sorting system broken down Conveyor repaired yesterday Impact to client= $1,000/minute Goals fix must meet Possible fixes Best Fit? Due Who Risks and Opportunities list Preventive Actions Contingent Actions Due Done Who Restore fast Certainty of fix Avoid causing other incidents Bricksorting robot TX72-6 problems today Incident Overview & Action Plan Problem Investigation - Finding Cause Decisions to be made Risk Management
- 17. After 30 minutes KT INCIDENT DASHBOARD time: August 30 Incident Summary: Customer Issues, Priorities, Impact - What When Where Actions Needed Due Done Who Diagnostic Data we have Possible Causes Next Investigation step Due Done Who Bricksorting robot TX72-6 problems The chute assembly presses long against the touch sensor Application update yesterday Sorting system broken down IS NOT badly Sorting bricks when initially moving right Touchsensor sw update Conveyor repaired yesterday Happens during reset Database slow to respond Impact to client= $1,000/minute Started August 30 Goals fix must meet Possible fixes Best Fit? Due Who Risks and Opportunities list Preventive Actions Contingent Actions Due Done Who Restore fast Reload database Certainty of fix Replace touch sensor Avoid causing other incidents Repair conveyor (again) Replace control unit Replace Motor B Brick sorting robot sorting incorrectly; possible causes determined. Incident Overview & Action Plan Problem Investigation - Finding Cause Decisions to be made Risk Management
- 18. After 60 minutesKT INCIDENT DASHBOARD time: August 30 Incident Summary: Customer Issues, Priorities, Impact - What When Where Actions Needed Due Done Who Diagnostic Data we have Possible Causes Next Investigation step Due Done Who Bricksorting robot TX72-6 problems The chute assembly presses long against the touch sensor Application update yesterday Check log files for evidence Sorting system broken down IS NOT badly Sorting bricks when initially moving right Touchsensor sw update Check log files for evidence Conveyor repaired yesterday Happens during reset Database slow to respond Contact DB team Impact to client= $1,000/minute Started August 30 Goals fix must meet Possible fixes Best Fit? Due Who Risks and Opportunities list Preventive Actions Contingent Actions Due Done Who Restore fast Reload database Replacing Touch sensor Review work instructions Contact MR support Certainty of fix Replace touch sensor X Check part upfront for DOAAvoid causing other incidents Repair conveyor (again) temp Replace control unit Replace Motor B Brick sorting robot TX72-6 now sorting correctly Incident Overview & Action Plan Problem Investigation - Finding Cause Decisions to be made Risk Management
- 19. Takeaways for Major Incident Managers (and for everyone): 1. Avoid holding people on calls for many hours 2. Keep all information visible in real time 3. Require specific information from participants 4. Run regular simulations to keep skills high 5. Set clear update times and stick to them
- 20. Leaders in Problem Solving twitter.com @KepnerTregoe facebook.com/KepnerTregoe linkedin.com/company/kepner-tregoe Andrew Vermes Senior Consultant +44 (0) 7973506628 avermes@kepner-tregoe.com
Editor's Notes
- KT discovered early on that effective problem solvers focused on one thing at a time: Understanding what’s happening, the impact is a necessary prelude to the other four colours; in some cases we can go directly to a solution; in others we need to step back and understand the risk before we take action. If there are choices about the fix, we need to consider the goals and environment in a quick Decision Analysis. Sometimes, it’s unwise to move forward if we have no idea about the cause: we might make things worse- so Problem Analys needs to be done. This sessions is about speeding that path to root cause, so now a challenge for you.