SlideShare a Scribd company logo
1 of 21
TRIUMF Failure Investigation Tool
Violeta Toma
Accelerator Operations Manager, TRIUMF
October 17, 2017
• Introduce TapRooT®
• Incident investigation:
oMild consequences
oModerate consequences
• Summary
Outline
• Incident investigation system:
• Human errors
• Equipment failure
• Helps solve problems:
• Reactively
• Proactively
What is TapRooT®
TapRooT® Process
Collect Information
Understand what
happened
Identify problems
which caused the
incident
Analyze each
problem’s root cause
Search for systemic,
cultural &
organizational causal
factors
Propose corrective
actions
Present lessons learnt
to management to
obtain support for the
proposed corrective
actions
1) Your root cause analysis is only as good as the information you
collect
2) Your knowledge (not only the lack of it) can get in the way of a
good root cause analysis
3) You have to understand what happened before you can
understand why it happened
4) Interviews are NOT about asking questions
TapRooT® Best Practices
5) You can’t solve all human performance problems with
discipline, training, and procedures
6) Often, people can’t see effective corrective actions even if they
can find the root causes
7) All investigations are NOT equal (some investigation steps
can’t be skipped)
TapRooT® Best Practices
0) Incident: safety lock applied to a water pump electrical
disconnect was cut in error by somebody other than the lock owner
Steps 1 -4:
TapRooT® Human Performance
Investigation
Water pumps
Collect Information & Understand what Happened
11) Parts are ready
12) Water is shut off
13) PG supervisor leaves for the day before work is
completed. Mechanical Services (MS) group leader takes
over work coordination
15) Keys are handed over
Collect Information & Understand what Happened
17) Work finished, system brought to operations conditions
18) the MS group leader gives the 2 keys to an attendant to
remove the lock
19) MS and PG supervisor communicate and PG advices MS to
cut the padlock
Collect Information & Understand what Happened
Collect Information & Understand what Happened
TapRooT® Tool: SnapCharT
• Workers (PG & ES) unware of site lock-out procedure for multi-shift lock-out
• Use of personal lock for a multi-shift lock
• Padlocks hand inscribed, hard to read
• PG supervisor didn’t check which key he needs to hand to MS supervisor
• PG supervisor didn’t show MS supervisor the lock-out location
• MS supervisor didn’t tell attendant that he must remove one lock-out only
• MS supervisor didn’t tell attendant the name of the pump to be re-connected
Search for systemic, cultural & organizational causal
factors
• Attendant doesn’t relay that a padlock was already removed
• Attendant doesn’t ask which pump to re-connect
Search for systemic, cultural & organizational causal factors
a) Training Programme and policy/procedure
I. Multi-shift lock-out
II. Padlock label
b) Clear policies & procedures
c) Audits and Evaluations
d) Inforce policy on lock-out through internal audits and disciplinary actions when
applicable
e) Timely communication of policies and procedures
Propose Corrective Actions
• Ensure policies on Lock-out and on Device Disable are
consistent
• Group training plans to reference above policies when
appropriate
• Policy should be revised to include the acceptable methods of
marking a lockout/disable padlock
• All TRIUMF groups using shop padlocks as part of multi-shift
device disables should be made aware of current policy and
change all shop padlocks to padlocks from Operations Group
Propose Preventative Actions
• Although not required by WorkSafe BC, BC Safety Authority or
BC Electrical Code, it would be best practice for any electrical
wiring that has been made safe (de-energised and device disable
installed) and disconnected from a device to also have the device
end wires be covered appropriately
• Group supervisors to schedule periodic discussion sessions on
policies and procedures and invite subject matter experts able to
answer questions as needed
Propose Preventative Actions
• TapRooT dos and don’ts:
• Don’t assume
• Ask “what happened” not “why”
• Expect more than one root cause
• Human error is just the starting point of a root cause analysis
• Search for root causes that can be fixed
Summary
Thank You!
Merci!
Questions?
TRIUMF: Alberta | British Columbia|
Calgary | Carleton | Guelph | Manitoba |
McGill | McMaster | Montreal | Northern
British Columbia | Queen’s | Regina |
Saint Mary’s | Simon Fraser | Toronto |
Victoria | Western | Winnipeg | York

More Related Content

Similar to 3.TRIUMF_Failure_Investigation_Tool_Final_16_9.pptx

Similar to 3.TRIUMF_Failure_Investigation_Tool_Final_16_9.pptx (20)

CNIT 121: 17 Remediation Introduction (Part 1)
CNIT 121: 17 Remediation Introduction (Part 1)CNIT 121: 17 Remediation Introduction (Part 1)
CNIT 121: 17 Remediation Introduction (Part 1)
 
Engaging staff in Health and Safety, do rules work? - Golden rules thoughts 2...
Engaging staff in Health and Safety, do rules work? - Golden rules thoughts 2...Engaging staff in Health and Safety, do rules work? - Golden rules thoughts 2...
Engaging staff in Health and Safety, do rules work? - Golden rules thoughts 2...
 
Using data science to automate event correlation - June 2016 - Dan Turchin - ...
Using data science to automate event correlation - June 2016 - Dan Turchin - ...Using data science to automate event correlation - June 2016 - Dan Turchin - ...
Using data science to automate event correlation - June 2016 - Dan Turchin - ...
 
Australian speech pathologists in private practice: reduce stress by building...
Australian speech pathologists in private practice: reduce stress by building...Australian speech pathologists in private practice: reduce stress by building...
Australian speech pathologists in private practice: reduce stress by building...
 
Safeabilty: Analyzing the Relationship between Safety and Reliability
Safeabilty: Analyzing the Relationship between Safety and Reliability Safeabilty: Analyzing the Relationship between Safety and Reliability
Safeabilty: Analyzing the Relationship between Safety and Reliability
 
Iconuk 2016 - IBM Connections adoption Worst practices!
Iconuk 2016 - IBM Connections adoption Worst practices!Iconuk 2016 - IBM Connections adoption Worst practices!
Iconuk 2016 - IBM Connections adoption Worst practices!
 
The Forgotten Secret to DevOps Success: Measurement
The Forgotten Secret to DevOps Success: MeasurementThe Forgotten Secret to DevOps Success: Measurement
The Forgotten Secret to DevOps Success: Measurement
 
Incident Investigation and Analysis by HF&C
Incident Investigation and Analysis by HF&CIncident Investigation and Analysis by HF&C
Incident Investigation and Analysis by HF&C
 
STARCANADA 2013 Keynote: Lightning Strikes the Keynotes
STARCANADA 2013 Keynote: Lightning Strikes the KeynotesSTARCANADA 2013 Keynote: Lightning Strikes the Keynotes
STARCANADA 2013 Keynote: Lightning Strikes the Keynotes
 
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...
 
Normal accidents and outpatient surgeries
Normal accidents and outpatient surgeriesNormal accidents and outpatient surgeries
Normal accidents and outpatient surgeries
 
Will Robots Replace Testers?
Will Robots Replace Testers?Will Robots Replace Testers?
Will Robots Replace Testers?
 
5 Clear Signs You Need Security Policy Automation
5 Clear Signs You Need Security Policy Automation5 Clear Signs You Need Security Policy Automation
5 Clear Signs You Need Security Policy Automation
 
GDPR in practice
GDPR in practiceGDPR in practice
GDPR in practice
 
Deviation, OOS & complaint investigation and CAPA
Deviation, OOS & complaint investigation and CAPADeviation, OOS & complaint investigation and CAPA
Deviation, OOS & complaint investigation and CAPA
 
Information_Gathering_Tools
Information_Gathering_ToolsInformation_Gathering_Tools
Information_Gathering_Tools
 
8D Training Presentation (tai lieu tham khao)
8D Training Presentation (tai lieu tham khao)8D Training Presentation (tai lieu tham khao)
8D Training Presentation (tai lieu tham khao)
 
PSU Web 2013: User Research Power Tool: Pareto Principle Based User Research
PSU Web 2013: User Research Power Tool: Pareto Principle Based User ResearchPSU Web 2013: User Research Power Tool: Pareto Principle Based User Research
PSU Web 2013: User Research Power Tool: Pareto Principle Based User Research
 
QT final.pptx
QT final.pptxQT final.pptx
QT final.pptx
 
Atlassian Based DevOps Command Center: Adding Opsgenie to the Powerful Mix!
Atlassian Based DevOps Command Center: Adding Opsgenie to the Powerful Mix!Atlassian Based DevOps Command Center: Adding Opsgenie to the Powerful Mix!
Atlassian Based DevOps Command Center: Adding Opsgenie to the Powerful Mix!
 

Recently uploaded

Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
ZurliaSoop
 

Recently uploaded (20)

Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
Jual Obat Aborsi Hongkong ( Asli No.1 ) 085657271886 Obat Penggugur Kandungan...
 
General Principles of Intellectual Property: Concepts of Intellectual Proper...
General Principles of Intellectual Property: Concepts of Intellectual  Proper...General Principles of Intellectual Property: Concepts of Intellectual  Proper...
General Principles of Intellectual Property: Concepts of Intellectual Proper...
 
Food safety_Challenges food safety laboratories_.pdf
Food safety_Challenges food safety laboratories_.pdfFood safety_Challenges food safety laboratories_.pdf
Food safety_Challenges food safety laboratories_.pdf
 
Fostering Friendships - Enhancing Social Bonds in the Classroom
Fostering Friendships - Enhancing Social Bonds  in the ClassroomFostering Friendships - Enhancing Social Bonds  in the Classroom
Fostering Friendships - Enhancing Social Bonds in the Classroom
 
Sociology 101 Demonstration of Learning Exhibit
Sociology 101 Demonstration of Learning ExhibitSociology 101 Demonstration of Learning Exhibit
Sociology 101 Demonstration of Learning Exhibit
 
Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
Kodo Millet  PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...Kodo Millet  PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
 
Wellbeing inclusion and digital dystopias.pptx
Wellbeing inclusion and digital dystopias.pptxWellbeing inclusion and digital dystopias.pptx
Wellbeing inclusion and digital dystopias.pptx
 
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptxCOMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
COMMUNICATING NEGATIVE NEWS - APPROACHES .pptx
 
Accessible Digital Futures project (20/03/2024)
Accessible Digital Futures project (20/03/2024)Accessible Digital Futures project (20/03/2024)
Accessible Digital Futures project (20/03/2024)
 
Micro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdfMicro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdf
 
Exploring_the_Narrative_Style_of_Amitav_Ghoshs_Gun_Island.pptx
Exploring_the_Narrative_Style_of_Amitav_Ghoshs_Gun_Island.pptxExploring_the_Narrative_Style_of_Amitav_Ghoshs_Gun_Island.pptx
Exploring_the_Narrative_Style_of_Amitav_Ghoshs_Gun_Island.pptx
 
How to Manage Global Discount in Odoo 17 POS
How to Manage Global Discount in Odoo 17 POSHow to Manage Global Discount in Odoo 17 POS
How to Manage Global Discount in Odoo 17 POS
 
Google Gemini An AI Revolution in Education.pptx
Google Gemini An AI Revolution in Education.pptxGoogle Gemini An AI Revolution in Education.pptx
Google Gemini An AI Revolution in Education.pptx
 
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdfUGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
 
How to setup Pycharm environment for Odoo 17.pptx
How to setup Pycharm environment for Odoo 17.pptxHow to setup Pycharm environment for Odoo 17.pptx
How to setup Pycharm environment for Odoo 17.pptx
 
ICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptxICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptx
 
Understanding Accommodations and Modifications
Understanding  Accommodations and ModificationsUnderstanding  Accommodations and Modifications
Understanding Accommodations and Modifications
 
Jamworks pilot and AI at Jisc (20/03/2024)
Jamworks pilot and AI at Jisc (20/03/2024)Jamworks pilot and AI at Jisc (20/03/2024)
Jamworks pilot and AI at Jisc (20/03/2024)
 
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptxBasic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdf
 

3.TRIUMF_Failure_Investigation_Tool_Final_16_9.pptx

  • 1. TRIUMF Failure Investigation Tool Violeta Toma Accelerator Operations Manager, TRIUMF October 17, 2017
  • 2. • Introduce TapRooT® • Incident investigation: oMild consequences oModerate consequences • Summary Outline
  • 3. • Incident investigation system: • Human errors • Equipment failure • Helps solve problems: • Reactively • Proactively What is TapRooT®
  • 4. TapRooT® Process Collect Information Understand what happened Identify problems which caused the incident Analyze each problem’s root cause Search for systemic, cultural & organizational causal factors Propose corrective actions Present lessons learnt to management to obtain support for the proposed corrective actions
  • 5. 1) Your root cause analysis is only as good as the information you collect 2) Your knowledge (not only the lack of it) can get in the way of a good root cause analysis 3) You have to understand what happened before you can understand why it happened 4) Interviews are NOT about asking questions TapRooT® Best Practices
  • 6. 5) You can’t solve all human performance problems with discipline, training, and procedures 6) Often, people can’t see effective corrective actions even if they can find the root causes 7) All investigations are NOT equal (some investigation steps can’t be skipped) TapRooT® Best Practices
  • 7. 0) Incident: safety lock applied to a water pump electrical disconnect was cut in error by somebody other than the lock owner Steps 1 -4: TapRooT® Human Performance Investigation
  • 8. Water pumps Collect Information & Understand what Happened
  • 9. 11) Parts are ready 12) Water is shut off 13) PG supervisor leaves for the day before work is completed. Mechanical Services (MS) group leader takes over work coordination 15) Keys are handed over Collect Information & Understand what Happened
  • 10. 17) Work finished, system brought to operations conditions 18) the MS group leader gives the 2 keys to an attendant to remove the lock 19) MS and PG supervisor communicate and PG advices MS to cut the padlock Collect Information & Understand what Happened
  • 11. Collect Information & Understand what Happened
  • 12.
  • 14. • Workers (PG & ES) unware of site lock-out procedure for multi-shift lock-out • Use of personal lock for a multi-shift lock • Padlocks hand inscribed, hard to read • PG supervisor didn’t check which key he needs to hand to MS supervisor • PG supervisor didn’t show MS supervisor the lock-out location • MS supervisor didn’t tell attendant that he must remove one lock-out only • MS supervisor didn’t tell attendant the name of the pump to be re-connected Search for systemic, cultural & organizational causal factors
  • 15. • Attendant doesn’t relay that a padlock was already removed • Attendant doesn’t ask which pump to re-connect Search for systemic, cultural & organizational causal factors
  • 16. a) Training Programme and policy/procedure I. Multi-shift lock-out II. Padlock label b) Clear policies & procedures c) Audits and Evaluations d) Inforce policy on lock-out through internal audits and disciplinary actions when applicable e) Timely communication of policies and procedures Propose Corrective Actions
  • 17. • Ensure policies on Lock-out and on Device Disable are consistent • Group training plans to reference above policies when appropriate • Policy should be revised to include the acceptable methods of marking a lockout/disable padlock • All TRIUMF groups using shop padlocks as part of multi-shift device disables should be made aware of current policy and change all shop padlocks to padlocks from Operations Group Propose Preventative Actions
  • 18. • Although not required by WorkSafe BC, BC Safety Authority or BC Electrical Code, it would be best practice for any electrical wiring that has been made safe (de-energised and device disable installed) and disconnected from a device to also have the device end wires be covered appropriately • Group supervisors to schedule periodic discussion sessions on policies and procedures and invite subject matter experts able to answer questions as needed Propose Preventative Actions
  • 19. • TapRooT dos and don’ts: • Don’t assume • Ask “what happened” not “why” • Expect more than one root cause • Human error is just the starting point of a root cause analysis • Search for root causes that can be fixed Summary
  • 20.
  • 21. Thank You! Merci! Questions? TRIUMF: Alberta | British Columbia| Calgary | Carleton | Guelph | Manitoba | McGill | McMaster | Montreal | Northern British Columbia | Queen’s | Regina | Saint Mary’s | Simon Fraser | Toronto | Victoria | Western | Winnipeg | York

Editor's Notes

  1. TapRooT is a root cause analysis methodology developed and owned by System Improvements Inc. TapRooT was created by Mark Paradies who is the president of the System Improvements Inc. Mark started as a Admiral Rickover’s Nuclear Navy officer and worked for DuPont and Westinghouse before starting the System Improvements in 1988. The first TapRooT book was published in 1991 with a focus on human performance. It analyzed the human performance to find ways to correct human factors problems - not to assign blame. In 2000 the equipment performance investigation was integrated in the TapRooT book. System Improvements continues to improve the system, the written material, the software and training
  2. There are 7 steps in the TapRooT investigation process.
  3. TapRooT promotes 7 good practices Don’t start an investigation by guessing what happened: this will bias your questions and the outcome of the investigation. Collect all information available which can help understand the incident, the sequence of events, etc. When needed, use a SnapCharT: helps organize information collected, identify information missing. Previous experience can bias your investigation; Looking for cause/effect Start an investigation by asking “what happened” not “why it happen”. The purpose of the interview is to collect info not justifications for what happened. Ask the interviewee to provide the smallest and apparently least important details – they can be very valuable info for the investigation Listen: interviews are not only about asking questions - one must listen. Don’t interrupt to ask questions: it will break the remembering process and important info might be lost
  4. Statistically, most corrective actions for human performance problems are discipline, training and procedures. When these fail to improve performance it’s obvious that the CA were misused. The investigator should seek CAs based on the root causes which have the potential to safeguard against future human errors. Investigator must be able to think out of the box. Doing the same thing in the same way for a long time prevents investigators to see effective corrective actions. Root cause investigations must be tailored to the size of the problem. Simple incident investigation can be performed by one TapRooTer while more complex ones might need a team of 2, 3 or more. Some small incidents simply don’t need an investigation at all, just the implementation of a corrective action.
  5. 1) These are old type pumps which comprise a motor connected to the pump via a drive shaft.
  6. 1) Cooling system has 2 water pumps: P-1 & P-2 - none are labeled. 2) Pumps’ electrical disconnects are just behind pumps in poor lit area 3) Electrical disconnect boxes are hand inscribed with felt pen – easy to read under good lighting 4) P-1 failed and was removed for repair by Plant Group (PG). A PG lock was installed on the pump electrical disconnect 5) Decision made to buy a new pump as repair was not possible – PG lock is left in place 6) As the 2 pumps are of the same type and age decision made to replace both (buy 2 pumps) 7) New pumps are delivered 8) Electrical Services (ES) placed a lock on P-1 disconnect and removed the feed wires from the pump motor. Wires are left on the floor with bare ends exposed 9) Defective P-1 removed & new pump is installed by PG – it is found that parts need to be made (adaptor to the water line) 10) Work is halted. Two locks (PG & ES) are left on the pump electrical disconnect Left: old water pump Right: new water pump with integrated motor
  7. 12) PG supervisor locks the second pump (P-2) to prevent its damage by running it without water 15) PG supervisor goes to his desk to pick up the P-2 lock key: 2 keys on the desk – one for P-1 (failed) the other for P-2. PG supervisor cannot remember which is which and hands in the 2 keys to the MS group leader saying “one will unlock the padlock’ 16) PG supervisor does not show the MS group leader the padlock
  8. 17) Electrical connection to P-1 is finished, system is brought to operations conditions = valves opened 18) The attendant thinks there must be 2 padlocks on the disconnect because he was given 2 keys. He removes a padlock but cannot remove the second one
  9. 22) After the loud bang the MS supervisor locks back P
  10. The online Lockout/Tagout training did not emphasize that OG padlocks are to be used when work lasts more than one shift and the importance of labeling the padlock with the owner’s name (not only the group) Lock-out & Disable policies confusing Had an audit been performed, it would have shown that none of the groups were aware of the device lock-out procedures .. Analysis of previous NCRs showed that 38% of reported nonconformities were related to policies not being used or in need of improvement. This indicates that this failure of staff to be aware of, and competent in, the application of organizational policies is not unique.