SlideShare a Scribd company logo
Root Cause Analysis Report
Version as of May 6, 2004
Hoist Drop from Monorail 21 at K West
Spent Fuel Pool
March 10, 2004
Sponsor: D. M. Busche
Root Cause Analysis Team
W. J. Leonard, Team Leader
W. R. Corcoran, Senior Advisor
J. M. Lukes
M. E. Poole
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 2 of 117
Table of Contents
Executive Summary ........................................................................................................ 4
Glossary.......................................................................................................................... 5
Narrative.......................................................................................................................... 7
Eight Question (Phoenix) Analysis ................................................................................ 15
Missed Opportunity Matrix............................................................................................. 23
Influence-Consequence Matrices.................................................................................. 29
Hoist Drop .................................................................................................................. 30
Non-Notification.......................................................................................................... 31
Barrier Analysis Matrix .................................................................................................. 32
Discussion of Design Vulnerabilities.............................................................................. 38
Discussion of Process Vulnerabilities............................................................................ 38
Procedures................................................................................................................. 38
Training ...................................................................................................................... 39
Safety Vulnerability Audits.......................................................................................... 39
Discussion of Process Strength .................................................................................... 39
Discussion of Influences................................................................................................ 40
Corrective Action Recommendations ............................................................................ 42
Analysis of Expected Effectiveness of Recommended Corrective Action ..................... 47
Conclusions................................................................................................................... 47
Attachments .................................................................................................................. 48
Comparative TimeLine ©......................................................................................... 49
Why Staircase Trees............................................................................................... 73
Extent of Condition and Cause Matrix..................................................................... 83
Cause Codes .......................................................................................................... 92
Documents Reviewed ........................................................................................... 103
Personnel Contacted............................................................................................. 106
Analysts Comments .............................................................................................. 108
Structural Analysis Report..................................................................................... 111
Calculations........................................................................................................... 113
Unanswered Questions......................................................................................... 117
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 3 of 117
Table of Figures
Figure 1 - Fuel Storage Monorail System........................................................................ 8
Figure 2 - Locking Arm & Weighted Safety Arm.............................................................. 9
Figure 3 - Flexible Transfer Crane................................................................................... 9
Figure 4 - Hoist Trolley Stopped Against Weighted Safety Arm.................................... 10
Figure 5 - Weighted Safety Arms In “UP” Position ........................................................ 11
Figure 6 - Hoist/Trolley Assembly Lying on Grating ...................................................... 11
Figure 7 - WSA Clearance ............................................................................................ 12
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 4 of 117
Executive Summary
On March 10, 2004 at about 11:17 P.M. a monorail hoist fell from its monorail in the K-
West Basin Spent Fuel Pool and came to rest on a grating over the pool. There were no
personnel injuries, no radiological releases, and no significant equipment damage. It
was, however, a near miss because if workers had been in the fall trajectory of the hoist
they could have been injured, perhaps severely.
The Facility Manager and the Multi-Canister Overpack (MCO) Production Manager were
promptly notified, but the Department of Energy Facility Representative (DOE-FR) was
not promptly notified. This episode received press coverage and was the subject of a
letter from the DOE Richland Operations Office (RL) Manager to the Fluor Hanford (FH)
President.
The hoist fell because it was moved by a Radiological Control Technician (RCT) to
facilitate his regular duties and because the monorail hoist stop (Weighted Safety Arm)
that should have stopped the hoist was stuck in the up position. The causes of the
mispositioned Weighted Safety Arm are not fully known, but are believed to include
distortion of the Weighted Safety Arm or shaft by some kind of impact combined with
the failure to perform checks that would have identified the improper Weighted Safety
Arm conditions.
The prompt notification to the DOE-FR was not done because the personnel involved
incorrectly applied procedural requirements and failed to recognize the “near miss”
nature of the event shortly after it occurred. In the opinion of the team, the letter from
RL would not have been written had prompt notification of the event occurred.
Both RL and FH management have shown keen interest in this event and its
investigation.
Recommended corrective actions for the primary event include enhanced preventative
maintenance and enhanced operating requirements. Recommended corrective actions
for the reporting irregularity include clarification of the procedure and re-emphasis of
managements’ expectation for strict adherence to the contractual and procedural
requirement to notify the DOE-FR of any adverse events or conditions that are worthy of
being reported to the FH Facility Manager.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 5 of 117
Glossary
Item Description / Meaning
Contracts
Requirements
Document (CRD)
Contractual requirement with which Fluor Hanford must comply.
Specifically, the requirement that the DOE Facility
Representative must be promptly notified of non-reportable bad
news when the Facility Manager is notified. (CRD M 231.1-2
(Rev. 1), “Occurrence Reporting and Processing of Operations
Information”.)
Crane Inspection
Team (CIT)
Team formed by Spent Nuclear Fuel Project to evaluate crane
issues and make recommendations. Also called the Crane
Committee. (Letter 03-SNF-JKM-024, dated June 9, 2003,
“Establishment of the Spent Nuclear Fuel Crane Inspection
Team”)
Fuel Storage
Monorail System
(FSMS)
Extensive system over the K Basins for transporting spent
nuclear fuel stored under water. Components of the FSMS are
Hoists, WSAs, and FTCs. (See Figure 1)
Flexible Transfer
Crane (FTC)
Double rail trolley circling the perimeter of the fuel storage pool.
Used for transporting Spent Nuclear Fuel under water from one
parallel monorail to another. (See Figure 3)
Weighted Safety
Arm (WSA)
Safety Device for stopping errant hoist trolleys. Also called
Receiver Latch, Latch, and Rail Stop and Stop. WSAs are
mounted in parallel on a pivot shaft and hang from the top of the
monorail. When the FTC is latched with a monorail, the Locking
Arms automatically align the FTC and the WSAs rotate up so a
hoist can be rolled onto or off of the FTC. (See Figures 2, 4, and
5)
Locking Arm An extension of the WSA to align and secure an FTC to the
monorail. The Locking Arm is brazed to the top of the WSA. If
an FTC is not fully stopped before it is engaged with the
monorail, the Locking Arm may be bent outward from impact.
This results in inward bending of the WSA which can then
become stuck in the up position against the monorail. (See
Figures 2 and 7)
Non-Reportable
Event
An event or condition that does not require an occurrence report
in accordance with HNF-PRO-060, “Occurrence Reporting and
Processing of Operations Information”. RL has repeatedly
indicated that it wants to be notified of non-reportable events and
conditions that are the subjects of notifications to the Facility
Manager.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 6 of 117
Item Description / Meaning
Stand-Down Described in HNF-PRO-060 Appendix C “Occurrence Reporting
Categories and Criteria”, Group 4 “Facility Status”, Subgroup B
“Operations” (7) SC-4: “A facility or site stand-down resulting
from safety reasons reportable as an occurrence or occurrences.
NOTE: This is a secondary reporting criterion, and does not
require a separate occurrence report.” Compare with the next
higher level criterion, (6) SC-4*, ”A facility or operations
shutdown (i.e., a change of operational mode or curtailment of
work or process) directed by management for safety reasons.”
The latter is “reportable” (requires an occurrence report).
Production Pause A work stoppage not considered a stand-down, not defined in
HNF-PRO-060.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 7 of 117
Narrative
This event is the conjunction of two parallel developments, one having to do with
notifications and the other having to do with equipment. The ultimate consequence was
a letter of concern from DOE-RL to Fluor Hanford regarding delayed notification of
events and other issues.
Over the last few years, RL has become increasingly dissatisfied with the promptness of
notification of low-level “non-reportable” events that are sufficiently significant to require
informing facility management. This dissatisfaction eventually manifested itself by RL
imposing a change to the contract with FH. The Contract Requirements Document
(CRD) was implemented by FH in a change to its reporting process procedure, HNF-
PRO-060, “Reporting Occurrences and Processing Operations Information”.
The need for the change to HNF-PRO-060 was identified after the procedure was
through the review cycle. The need for the change was urgent and the time pressure
was sufficient that the CRD related addition was not sent back through the review cycle.
Thus, the reviewers were not afforded the opportunity to identify certain human factors
issues that would later turn out to be important.
The human factors issues involved are:
1) The use of the exact words from the CRD, and
2) The location of the change in a portion of the procedure called “Reportable
Occurrences”.
The wordy descriptions and location of the requirement in HNF-PRO-060 made it highly
likely that a person not specifically trained in the intent of the procedure change would
not perceive the basic simplicity of it. The basic simplicity of the intent is to have the
Facility Representative (FR) promptly notified of all non-reportable bad news that is
reported to the Facility Manager.
The FH Emergency Preparedness (EP) department had the responsibility to make and
implement the changes to HNF-PRO-060. There was no process to ensure that all
personnel who needed to change their notification behaviors would be trained to the
new requirements. In addition, there was no process to check that the new behaviors
would, in fact, take place. Thus, the Shift Operations Manager (SOM) who was on duty
when the hoist dropped had not been trained to the new requirements.
There was, however, a presentation by the EP Director to the Facility Managers Forum
covering the new requirements. Unfortunately, the person who was the K West Basin
Facility Manager (FM) on the night of the event was not in attendance because he was
not the K West Basin FM at the time of this presentation. In addition, the EP Director
did not cause checks to be made to ensure compliance with the new contract
requirements as listed in HNF-PRO-060.
Meanwhile, FH personnel had difficulties in recognizing and labeling “near miss”
(personnel injury) situations. Personnel are reluctant to declare near misses. This
reluctance is manifested in prolonged discussions as to whether or not a particular
event was a near miss.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 8 of 117
The K West Basin Fuel Storage Monorail System dates from the mid-1950s (Figure 1).
Recently the equipment has been used more frequently to support facility closure. The
use and misuse of this equipment has resulted in material deficiencies including tilted
monorails and distorted components.
Figure 1 - Fuel Storage Monorail System
The monorail Latch Assembly is made up of two key parts, the Locking Arm (Figure 2)
that engages a Flexible Transfer Crane (FTC) to the monorail (Figure 3), and the
Weighted Safety Arm (Figure 2) that acts as a rail stop when the FTC is not coupled.
“Normal” use of FTCs has included engaging the locking mechanism to the monorails
while the FTCs are still moving. This action causes the monorail’s Latch Assembly
Locking Arm to be impacted, likely bending the Locking Arm such that the Weighted
Safety Arm is bent in the opposite direction. This results in the WSAs contacting and
sticking in the up position on the monorail upper flange
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 9 of 117
Figure 2 - Locking Arms & Weighted Safety Arms
Figure 3 - Flexible Transfer Crane
(Photo taken in passageway at south end of Monorail 21, about where the hoist landed,
looking east)
The Weighted Safety Arms (WSAs) are engineered safety devices for preventing a
hoist/trolley assembly from falling off the end of a monorail (Figure 4). “Normal” use of
the hoist trolleys has included allowing the hoists to collide with the WSAs. The Root
Cause Analysis (RCA) Team considers that this is a use of a safety device for
operational purposes. There are two disadvantages to this practice. First, the collisions
Locking
Arms
Weighted
Safety
Arms
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 10 of 117
could damage the hoist trolley assembly, the WSAs, or the WSA shaft. (Note: For the
rest of this report, when we refer to bending to the WSA, please understand that the
same effects could have been caused by a bent shaft.) Second, if, for some reason, the
WSA is missing or stuck in the up position, the hoist trolley would not collide, but rather,
continue off the end of the monorail and fall in an area where personnel could be.
Figure 4 - Hoist Trolley Stopped Against Weighted Safety Arm
The annual Preventive Maintenance (PM) of the K West Basin monorail system was
completed in February, 2004, and did not result in the recording of any safety
discrepancies. Unfortunately, the records are not sufficient to provide adequate detail.
It is not clear whether discrepant conditions existed and actions were taken to correct
those conditions or whether no discrepant conditions existed. A recording of “No
Discrepancies” was made on the work record of the PM.
A variety of job classifications are needed in the K West Basin. Some, such as
Radiological Control Technicians (RCTs), do not receive hoisting and rigging training.
Many people, nevertheless, move hoists in order to do their assigned jobs unless they
are told to stop and get a qualified person.
On the night of March 10th
, 2004, an RCT moved a hoist on Monorail 21. Monorail 21
was tilted down toward the south. The RCT did not do any pre-use checks for this
seemingly trivial evolution. The RCT elected to push the hoist to the south (in the
direction of downward tilt). For unknown reasons, the WSA at the south end of Monorail
21 was stuck in the up position and was therefore unavailable to stop the traveling hoist
(Figure 5).
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 11 of 117
Figure 5 - Weighted Safety Arms Stuck In “UP” Position
The hoist fell off the end of the monorail and landed on a grating in a spot that is
sometimes occupied by workers. Fortunately, at the time of the impact, there were no
workers and no equipment in the trajectory of the hoist (Figure 6).
Figure 6 - Hoist/Trolley Assembly Lying on Grating
The hoist casing was damaged. No other physical consequences resulted.
The personnel in the area, the RCT and four Nuclear Chemical Operators (NCOs),
heard the sound of the impact and verified the source. They stopped work and
informed the SOM.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 12 of 117
The SOM sent an Operating Engineer (OE) and an NCO to investigate the fallen hoist.
They found that the WSA at the south end of Monorail 21 was inexplicably in the down
position. They also found that when the WSA was pushed up, could become stuck in
the up position. The WSA was left in the up position for further evaluation on dayshift.
The OE cordoned off the area. The hoist drop into the area sometimes occupied by
people was not considered to be a near miss personnel injury event.
Figure 7 - WSA Clearance
The scrap processing crew received permission from the SOM to re-enter the basin to
complete process work.
The SOM reviewed HNF-PRO-060 for reportability, but found no guidance that
prompted him to notify the FR or to advise the FM that the FR should be notified. The
SOM notified the FM about the event and the near miss aspect of the event was
discussed (determined not to be a near miss), but notification to the FR was not. The
SOM then notified the Multi-Canister Overpack (MCO) Production Manager. Again,
neither notification to the FR nor the near miss was discussed.
The next morning, March 11, 2004, at the K West Basin Plan of the Day management
meeting, the hoist fall was discussed and determination was made for the Crane
Inspection Team to investigate. The Crane Inspection Team investigated K West and
found 9 damaged WSAs. Action was taken to correct the identified discrepancies.
There was no overt evidence that the Crane Inspection Team (CIT) realized that their
activities were affecting root cause analysis evidence. This could have been particularly
significant since the possibility of sabotage would later be considered and the WSAs
were potentially criminal evidence. The CIT did take photographs of some of the
damaged WSAs. They did not, however, record “as found” dimensions of the distorted
WSAs prior to making adjustments (straightening).
Normal
Clearance:
described by
Structural
Engineer as
“almost zero.”
Wear caused by
engagement with
moving FTC.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 13 of 117
Meanwhile, the FR called the FM and discussed a number of production issues. During
this conversation, the FM mentioned that the hoist fell off the monorail.
Later that morning, following the CIT investigation of the stuck WSA, the Fuel Storage
Monorail System (FSMS) Design Authority and the Engineering Manager met with the
SNF Project Deputy Director. No certain cause for the WSAs to become bent was
determined. Additionally, no certain cause for the WSA having been moved up was
determined. However, one of the people who had been involved in the investigation
disclosed to the Deputy Director that the WSAs may have been damaged intentionally.
Following this meeting the CIT inspected the K East Basin and confirmed (based on a
list provided by operators) that several WSAs were capable of sticking in the up
position. The Crane Inspection Team straightened the bent WSAs, breaking one in the
attempt.
The Deputy Director notified the SNF Vice President of the event, stating that the CIT
thought the WSAs were bent on purpose. The SNF Vice President then notified the FH
Vice President. Subsequently, the FH President and the DOE-RL Manager were
notified. The RL Manager called the Facility Representative Manager, who notified the
K West Basin FR. The FR had not been aware that the hoist fall was a result of a bent
WSA stuck in the up position or that it was a suspected act of sabotage.
At about 2:00 in the afternoon, the FH Executive Vice President directed the
implementation of a “Production Pause”. Also, the SNF Vice President notified site
security and security notified local law enforcement.
The only evidence obtained by the RCA Team was a seven line hand written, undated
personal statement. Some interviews were conducted prior to the critique meeting;
however, there does not appear to be any formal record of these interviews.
A Critique Meeting was scheduled for 6:30 that evening. The event and corrective
actions were discussed, but no probable causes were determined. Security and law
enforcement personnel were present at the meeting.
The hoist fall event was determined reportable as a Near Miss Occurrence following the
Critique Meeting.
The next day, March 12, 2004 the Tri-City Herald printed an article titled “Hanford
Officials Investigate K Basin Accident”. The newspaper article stated that work was
stopped throughout Hanford’s K Basins after an accident lead to an investigation that
found mechanical problems at both the K East and K West Basins. Discussion of
possible sabotage was mentioned, however, the article went on to state that the
evidence was inconclusive.
The same day, the DOE-RL Manager sent the letter of concern to the FH President
stating that RL was “extremely concerned about the degradation of hoisting safety
equipment and subsequent hoist drop off the monorail to the basin grating, which was
initially identified as a purposeful act”. They went on to state that they were “particularly
dissatisfied with the response actions from the time the hoist fell on Wednesday night,
March 10, 2004, until RL was notified of issues on Thursday afternoon, March 11,
2004”.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 14 of 117
On March 15, 2004 at about 8:00 in the morning, the SNF Deputy Director informed the
FH Executive Vice President that corrective actions had been completed. The
Production Pause was terminated based on a restart plan that had been approved by
the Executive Vice President.
On March 18, 2004 the FH VP of Regulatory Compliance appointed an “Independent
Review of Recent Near Miss at SNF”, headed by the Manager of Assessments. The
review team was convened to gather information about the event for a response to the
RL letter of concern. No formal report was issued from this review.
On April 2, 2004, The Tri-City Herald printed an article titled “K Basins accident
concerns DOE”. The newspaper article described the RL letter and the FH response to
the letter. The article indicated that FH now believes that the problem was not caused
by worker sabotage, but was caused by wear and tear on the equipment at the end of
its life cycle.
On April 5, 2004, a Root Cause Analysis Team was chartered to investigate two primary
issues of concern:
1) The near miss event involving the falling hoist, and
2) The delayed notification to RL personnel.
The items to be included were:
• Determine the facts associated with the event;
• Analyze the causes for the event;
• Identify the potential for similar events to occur at the SNF Project and rest of the
site, and;
• Provide recommendations for corrective action to the SNF Project and the rest of the
site.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 15 of 117
Eight Question (Phoenix) Analysis
This analysis provides summary answers to the Eight Questions for Insight and is a
brief high-level summary of the entire episode.
Eight Question (Phoenix) Analysis of Hoist Drop
Impact
1. Consequences
(Tangible and intangible
adverse effects)
o Actual:
1) Adverse letter from RL to FH.
2) Adverse press coverage.
3) Costs of a Production Pause of several days.
4) On-site investigation Costs.
5) Costs incurred in interfacing with RL over this incident.
6) Reduced employee moral due to accusations of
wrongdoing.
o Expected:
1) Increased RL scrutiny of FH activities and associated
costs.
2) Probable withholding of fees to FH.
3) Costs of repairing/replacing distorted Weighted Safety
Arms.
o Potential Consequences:
1) Severe personnel injuries and an extensive production
pause or shut down.
2. Significance (What
does this mean for the
future of the facility?)
o Precursor to:
1) More serious personnel, economic or customer
satisfaction consequences. (The behaviors and conditions
involved in this one would, if not corrected, be capable of
causing a much more serious event.)
2) Potential breakdown of the Occurrence Reporting
element of the Safety Management Program.
o Temporal and Spatial Extent:
1) Monorail problems: Several cases of distorted
Weighted Safety Arms were found in both K West and in
K East.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 16 of 117
Eight Question (Phoenix) Analysis of Hoist Drop
Impact (continued)
2) Notifications: There have been other notification
shortfalls in the past. These prompted changes in contract
requirements, including requirements that were
specifically not met in this instance.
3) Previous failures to declare near misses.
4) Other safety device precursors.
5) Other departures from ISMS.
6) Other difficult administrative procedures.
7) Other problems caused by not using SAT.
o Barriers that did not succeed:
See items 3, 4, and 5 below.
o Remaining barriers:
1) Good fortune (There were no personnel in the fall
trajectory of the hoist).
o Campaign issues:
1) Personnel safety hazards
2) Notifications.
3) Near Miss calls.
4) Promptness and effectiveness of Critique Meetings.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 17 of 117
Eight Question (Phoenix) Analysis of Hoist Drop
Influences on Consequences
3. Vulnerability (What
set us up for this
event?)
o Elevated monorail.
o Hoist rides on monorail.
o Hoist pendant, chain and chain bucket hang down
obstructing activities below.
o Monorail 21 is tilted toward the south end.
o Monorail 21 and hoist have low friction.
o Weighted Safety Arms prevent hoist dropping off end of
monorail when FTC not present.
o Single failure vulnerability.
o Weighted Safety Arm bent slightly causing interference
with Monorail 21:
1) Probably caused by repeated collisions of the
hoist/trolley assembly with the WSA and,
2) The impact of the FTC when coupling with the
monorail.
o WSA in the up position potentially caused by long tools
striking the WSA.
o Adverse impact of tool strikes not recognized.
o RCT not trained on hoist vulnerability to dropping off end
of monorail.
o RCT not trained to avoid relying on safety devices, e.g.,
WSAs.
o Practice of relying on Weighted Safety Arms to stop hoist.
o Practice of moving hoists without safety checkout.
o Practice of moving FTCs while coupling to monorails.
o ISMS not applied to routine RCT tasks.
o Collisions between hoist and Weighted Safety Arms not
considered as precursors.
o Collisions between FTCs and Locking Arms not
considered as precursors.
o PMs don’t find Weighted Safety Arm problems.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 18 of 117
Eight Question (Phoenix) Analysis of Hoist Drop
Influences on Consequences (continued)
o Previous history of notification shortfalls.
o Previous history of “near miss” declaration shortfalls.
o Non-User-Friendly guidance on notification.
4. Trigger (What put the
ball in motion?)
o Radiological Control Technician moved hoist.
5. Exacerbation (What
made the
consequences as bad
as they were?)
o RCT did not keep positive control of the hoist trolley.
o Monorail elevation.
o Failure to notify Facility Representative.
o Previous FH performance issues (notification
deficiencies).
o Lack of questioning attitudes of managers.
o Imprudent references to “intentional act” and “sabotage”.
6. Mitigation (What kept
the consequences from
being a lot worse?)
o No personnel in fall trajectory.
o Prompt FH response to RL letter.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 19 of 117
Eight Question (Phoenix) Analysis of Hoist Drop
Close Out
7. Lessons to be
Learned (What skills,
rules, and knowledge
should be added or
reinforced?)
o Notifying Facility Representative when Facility Manager is
notified of non-reportable “bad news” is not optional.
o Routinely using and relying on Weighted Safety Arms to
stop moving hoists is a bad practice.
o Routinely coupling moving FTCs with monorails is a bad
practice.
o Using safety devices as operational controls is a bad idea.
o Interviews and interview-assisted personal statements
should be obtained before the involved personnel leave
the site at shift end.
o Inflammatory words, such as “sabotage”, should be used
prudently and only when necessary.
o Professionals hate to find out bad news about their areas
of responsibility from their superiors (e. g., SNF DOE-FR
notified by DOE-RL FR Manager).
o Criminal investigations and Root Cause investigations
have different objectives and should not be combined.
o Every event should be scrutinized to determine the extent
to which it was a near miss to something much more
serious.
o Managers who receive bad news reports should exhibit a
questioning attitude toward the information received. In
particular, the caller should be queried about potential
circumstances that would make the bad news more
significant.
8. Corrective Actions
(What conditions and
behaviors should be
changed?)
o Interim compensatory measures:
1) Implemented a Production Pause (The Production
Pause was lifted following completion of actions 2, 3, and
4 below).
2) The Crane Inspection Team completed an evaluation
of the monorail system and adjusted other WSAs into
their proper position and verified proper operation.
3) Implemented a Long Term Order requiring a shiftly
check to verify WSAs are in the down position; to verify
WSAs are in the down position while performing
operations; and, to prohibit the hanging of tools off of the
WSAs.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 20 of 117
Eight Question (Phoenix) Analysis of Hoist Drop
Close Out (continued)
4) Conducted management briefings on the event to
SNFP employees.
5) FH VP of Regulatory Compliance issued an e-mail to
all Project Vice Presidents to disseminate to their Facility
Managers the expectations for compliance with the
requirement to notify the FR whenever the FM is notified
of “non-reportable” bad news.
6) Provide stop-gap training on near miss identification
and FR notification expectations for all personnel involved
in the notification process.
o Corrective actions for symptoms and effects: (Complete)
1) “Straightened” the Weighted Safety Arm on Monorail
#21 and verified proper operation.
2) Replaced the cracked casing on the Hoist HOI-440 and
verified proper operation.
o Corrective actions for causes:
1) Change monorail hoist system operating procedures to
require that persons moving hoists must maintain positive
control of the hoist, and must not cause collisions
between the hoist and the WSAs.
2) Change monorail hoist system operating procedures to
require that before beginning evolutions involving hoists,
all monorail system safety features must be checked.
3) Change the FTC operating procedures to require that
persons moving FTCs do not cause collisions with the
monorail system (Locking Arms).
4) Rewrite HNF-PRO-060 to meet appropriate procedure
standards, including human factors.
5) Revise HNF-PRO-060 to require that every event be
suspected to be a near miss for something more serious
(e.g., would people normally work in an area that was in
the line of fire?)
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 21 of 117
Eight Question (Phoenix) Analysis of Hoist Drop
Close Out (continued)
6) Revise training policy to require all training for new
requirements meets the intent of the SAT.
o Corrective actions for generic implications:
1) Examine other PM procedures for weaknesses similar
to the ones found in the monorail system procedures.
2) Check all PM procedures to assure that they require
returning the system to a reference condition after the
evolution.
3) Share this report with other facilities.
o Corrective actions for self-assessment deficiencies:
1) Change the text of the monorail system PM instructions
to clearly specify what needs to be checked and what the
acceptance criteria are.
2) Change the text of the monorail system PM instructions
to clearly specify the safety significance of the items being
checked.
o Disposition of Extraneous Conditions Adverse to Quality:
1) Management and the Bargaining Unit should agree on
the union’s role in investigations so that union leadership
does not obstruct.
2) Interviews of involved personnel should be conducted
before the end of the shift.
3) Experienced interviewers should assist personnel in
preparing personal statements.
4) Personal statements should be dated and signed, and
completed before the end of the shift.
5) Personnel who have not been trained in evidence
preservation should be barred from the scene of an
accident.
6) Any physical work on evidence should be done under
the work control process and the work control instructions
should include measures to prevent spoliation.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 22 of 117
Eight Question (Phoenix) Analysis of Hoist Drop
Close Out (continued)
7) The basis for both imposition and lifting of work
interruptions (Production Pause) should be documented.
8) Private deals to abrogate contractual requirements
must be discouraged (prompt notifications from FMs and
FRs).
9) The adherence to the intent of ISMS and SAT should
be transparent in work instructions and other documents.
One should be able to pick-up one of these documents
and tell that ISMS and/or SAT has been applied.
10) Personnel in high hazard environments should not
take action without ensuring that they posses the
qualifications/training. It is not acceptable for a person to
assume that the action they are about to take is trivial.
11) Criminal investigations should take precedence over
organizational learning investigations (critiques). If law
enforcement personnel are present, critiques should be
suspended.
12) FH Senior Management needs to ensure that bad
news that will be communicated to DOE-RL Senior
Management is first reported to the project Facility
Representative.
13) All investigative activities should result in written
documentation.
14) Subsequent chartered investigation teams should be
provided with all reports of previous investigative
activities.
15) Unintended and unanalyzed impacts between
equipment should be treated as abnormal events and
should be stopped pending engineering approval (e.g.,
impacts of long tools against overhead components.)
16) The effectiveness of corrective actions should be
checked some time after implementation.
o Follow-up plans:
1) Audit this facility and all facilities with similar
vulnerabilities to confirm that they accommodate the
lessons to be learned and corrective actions of this
incident.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 23 of 117
Missed Opportunity Matrix
This Matrix is a summary of the most significant situations in which ordinary people could have prevented the event or
made its consequences less severe by doing ordinary actions ordinarily well. The matrix is not intended to capture brilliant
insights or out-of-the-box thinking.
Missed Opportunity Matrix
Who Situation Opportunity (action) Expected Result Impact on
Consequences/Remarks
EP Dissemination of
CRD requirements
Train all personnel
whose behaviors
needed to change, in
accordance with SAT
Personnel needing to
change their previous
behavior relating to prompt
notification to FR would
have done so
All the personnel involved in
notifying FRs of non-reportable
bad news would have been
trained to the CRD requirements.
It is more likely that notifications
would have been made.
EP During or after
dissemination of
CRD requirements
Trainees’ knowledge
should have been
checked both during
and after the training.
The behaviors of the target
population would have
been checked to ensure
prompt notifications were
being made and the
training would have been
modified if necessary
All the personnel involved in
notifying FRs of non-reportable
bad news would have been
properly trained to the CRD
requirements.
It is more likely that notifications
would have been made.
EP Dissemination of
CRD requirements in
PRO-060
Plan the training in
order to accomplish a
specified result
Personnel needing to
change their previous
behavior relating to prompt
notification to FR would
have done so
All the personnel involved in
notifying FRs of non-reportable
bad news would have been
trained to the CRD requirements.
It is more likely that notifications
would have been made.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 24 of 117
Missed Opportunity Matrix
Who Situation Opportunity (action) Expected Result Impact on
Consequences/Remarks
FM Dissemination of
CRD requirements in
PRO-060
Disseminate the
information provided in
the Facility Manager’s
Forum related to
prompt notification to
FR when FM is notified
of “bad news”
Personnel needing to
change their previous
behavior relating to prompt
notification to FR would
have done so
All the personnel involved in
notifying FRs of non-reportable
bad news would have been
informed of the CRD
requirements.
It is more likely that notifications
would have been made.
EP Procedure Review
Process
Obtain peer review of
PRO-060 after CRD
requirements were
incorporated
Human factor issues would
have been identified
Personnel needing to
change their previous
behavior relating to prompt
notification to FR would
have done so
Procedure would have been more
user friendly
It is more likely that notifications
would have been made.
EP Dissemination of
CRD requirements
in PRO-060
Provided supplemental
information/continuing
training to personnel
who had been
previously trained
when CRD requirement
was added (not all
personnel received
training to the new
requirement).
Personnel needing to
change their previous
behavior relating to prompt
notification to FR would
have done so
All the personnel involved in
notifying FRs of non-reportable
bad news would have been
informed of the CRD
requirements.
It is more likely that notifications
would have been made.
EP Procedure change
process
Verification and
validation of
administrative
procedures
Human factor issues would
have been identified
Procedure would have been more
user friendly
It is more likely that notifications
would have been made.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 25 of 117
Missed Opportunity Matrix
Who Situation Opportunity (action) Expected Result Impact on
Consequences/Remarks
FM Dissemination of
PRO-060 for prompt
notifications to FR
Provided information
from the Facility
Manager’s Forum on
prompt notifications to
the FR to their project
personnel.
Personnel needing to
change their previous
behavior relating to prompt
notification to FR would
have done so
All the personnel involved in
notifying FRs of non-reportable
bad news would have been
informed of the CRD
requirements.
It is more likely that notifications
would have been made.
QA
Operations
Engineering
Safety
Performance of
periodic audits,
assessments, and
JHAs for K West
Basin Operations,
including the use of
hoists
Audits, assessments,
and JHAs should have
noticed the single point
failure vulnerability.
The event is aborted by
identifying the problem and
establishing additional
controls.
No event, however, the prompt
notification vulnerability remains in
place.
An example of a control would be
to hang caution tags on the hoists
when they are on slanted
monorails to say: CAUTION: Keep
positive control while moving hoist.
Monorail is tilted and hoist will
continue to travel on its own.
QA
Operations
Engineering
Safety
Performance of
periodic audits,
assessments, and
JHAs for K West
Basin Operations,
including the use of
hoists
Audits, assessments,
and JHAs should have
noticed the routine
repeated use of the
safety device to stop a
hoist.
The event is aborted by
identifying the problem and
establishing additional
controls.
No event
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 26 of 117
Missed Opportunity Matrix
Who Situation Opportunity (action) Expected Result Impact on
Consequences/Remarks
Design
Authority
Thinking about
failure modes and
safety vulnerabilities.
Identify safety
vulnerabilities related
to the single point
failure and the
challenging of a safety
device (Weighted
Safety Arms).
The event is aborted by
identifying the problem and
establishing additional
controls.
No event.
Millwright
Supervisor
Preventative
Maintenance
Discrepancies (bent or
sticking Weighted
Safety Arms) should
have been found.
The discrepant condition of
the Weighted Safety Arm
would have been identified
and corrected.
No event
Operators Thinking about
impact in the
overhead from use of
long tools
Recognize that long
tools are impacting
overhead hoist system
equipment
Areas would be inspected
for damage and repairs
initiated if necessary
Potentially prevented event
Comment: The long tool may
have bent the Weighted Safety
Arm, or caused it to stick in the up
position. Evidence is inconclusive.
Operators Routine movement
of the hoist
Pre-use hoisting and
rigging checks should
identify safety
anomalies including the
Weighted Safety Arm
on Monorail 21 being in
the up position.
The pre-use checks would
have aborted the event by
identifying the Weighted
Safety Arm in the up
position.
No event
Operators Routine movement
of the FTC
Recognize that FTCs
are impacting Locking
Arms
FTCs would be stopped
before coupling with the
Locking Arm on the
monorail.
Potentially prevented event
Comment: The FTC impact may
have bent the Weighted Safety
Arm.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 27 of 117
Missed Opportunity Matrix
Who Situation Opportunity (action) Expected Result Impact on
Consequences/Remarks
SOM Compensatory
actions from hoist
fall.
Implement controls to
ensure other hoists
don’t fall off monorails
before resuming
normal operations.
Controls would have been
implemented.
Mitigation of RL concerns for
worker safety described in Letter
of Concern to FH.
SOM Found out that hoist
fell off monorail and
impacted an area
where workers
sometimes are
present.
SOM should have
recognized that this
was a near miss
personnel injury.
Reportability would have
been clearly established
and SOM would have
made notifications to FM
and FR
The prompt notification issue
would not have occurred.
FM Found out that hoist
fell off monorail.
FM should have asked
questions to reveal that
this was a near miss
personnel injury.
Reportability would have
been clearly established
and SOM would have
made notifications to FR
The prompt notification issue
would not have occurred.
MCO
Production
Manager
Found out that hoist
fell off monorail.
MCO Production
Manager should have
asked questions to
reveal that this was a
near miss personnel
injury.
Reportability would have
been clearly established
and SOM would have
made notifications to FR
The prompt notification issue
would not have occurred.
K West
Management
Chain (SOM &
FM)
Notification process FR should have been
notified in accordance
with SNF Notification
Guidelines and HNF-
PRO-060 for events
meeting the threshold
for notification of the
FM.
The FM or SOM would
have promptly notified the
FR.
The prompt notification issue
would not have occurred.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 28 of 117
Missed Opportunity Matrix
Who Situation Opportunity (action) Expected Result Impact on
Consequences/Remarks
K West
Management
Chain (SOM &
FM)
Notification process FM and SOM should
have been aware of RL
concern for receiving
prompt notification.
The FM or SOM would
have promptly notified the
FR.
The prompt notification issue
would not have occurred.
VP S&H Escalating customer
dissatisfaction with
prompt notification of
non-reportable bad
news
Find out about
escalating
dissatisfaction with
prompt notification of
non-reportable bad
news
Assure that EP understood
the purpose of the changes
to PRO-060 to
accommodate CRD
requirements
Change would have been better,
training would have been better
and the notification issue would
not have happened
Collective
Feedback
Community
Escalating customer
dissatisfaction with
prompt notification of
non-reportable bad
news
Take instances of
customer
dissatisfaction with
prompt notifications,
record them, analyze
them, implement
corrective actions, and
disseminate lessons to
be learned
Notifications would have
been done as required by
the CRD
The prompt notification issue
would not have occurred
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 29 of 117
Influence-Consequence Matrices
Explanation of Influence-Consequence Matrices
See one of the two matrices on the next two pages. These matrices display important influences on the consequences
and how the removal of a single influence would have impacted the consequences.
In the Influence-Consequence Matrix for the Hoist Drop, for example, the upper part of the left hand column lists important
influences on the consequences of the hoist drop. The lower part of the left hand column lists the types of consequences.
The next column indicates that in the actual case all of the influences were present and the actual consequences resulted.
The remaining columns relate to hypothetical cases in which, respectively, certain influences are assumed to have been
negated. At the bottom of each of these hypothetical case columns one sees the consequences that would have resulted
from the hypothetical case.
One sees readily from that matrix that there were seven individual influences (causes) that, if it had not been there the
hoist would not have dropped.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 30 of 117
Hoist Drop
Influences Actual
Case
Case 1
WSA
straight
Case 2
WSA
down
Case 3 MR
level
Case 4 RCT
pushes
hoist N
Case 5 RCT
holds onto
hoist
Case 6 Pre-
use check of
WSA
Case 7
PM ID’s bent
WSA
Case 8
Person
under end of
MR
WSA is bent Yes No Yes Yes Yes Yes Yes Yes Yes
WSA is up Yes No, WSA
would
drop
No Yes Yes Yes Yes Yes Yes
M 21 is tilted to
S
Yes Yes Yes No Yes Yes Yes Yes Yes
RCT pushes
hoist to S
Yes Yes Yes Yes No Yes Yes Yes Yes
RCT does not
maintain +Ctrl
Yes Yes Yes Yes Yes No Yes Yes Yes
No pre-use
check of WSA
Yes Yes Yes Yes Yes Yes No Yes Yes
PM does not ID
bent WSA
Yes Yes Yes Yes Yes Yes Yes No Yes
No person
under end of
MR
Yes Yes Yes Yes Yes Yes Yes Yes No
Consequences
Hoist Dropped,
casing
broke
None None Probably
None
None None None None Dropped,
casing not
broken
Personnel None None None None None None None None Injury or
death
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 31 of 117
Non-Notification
Influences Actual
Case
Case 1
Previous
notification
success
Case 2 No
CRD
Case 3
Mgmt
chooses
effective
wording
Case 4
PRO-060
is human
factored
Case 5
PRO-060
training
per SAT
Case 6
Realistic
“near miss”
sensitivity
Case 7
Questioning
attitude by
FM
Case 8
Questioning
attitude by
MCO Mgr
RL unhappy w/
notifications
Yes No Yes Yes Yes Yes Yes Yes Yes
CRD on
notifications
Yes Yes No Yes Yes Yes Yes Yes Yes
PRO-060 uses
CRD wording
Yes Yes Yes No Yes Yes Yes Yes Yes
PRO-060 LTA
human factored
Yes Yes Yes Yes No Yes Yes Yes Yes
Ineffective PRO-
060 training
Yes Yes Yes Yes Yes No Yes Yes Yes
Inadequate “near
miss” sensitivity
Yes Yes Yes Yes Yes Yes No Yes Yes
Inadequate
challenge by FM
Yes Yes Yes Yes Yes Yes Yes No Yes
Inadequate
challenge by
MCO Mgr
Yes Yes Yes Yes Yes Yes Yes Yes No
Consequences
FR not promptly
notified per CRD
Yes FR not
promptly
notified per
CRD
FR not
promptly
notified
per CRD
FR
promptly
notified
per CRD
FR
promptly
notified
per CRD
FR
promptly
notified
per CRD
FR promptly
notified per
“near miss”
requirement
FR promptly
notified per
CRD
FR promptly
notified per
CRD
Letter of concern
from RL
Yes Probably no
letter
Letter of
concern
from RL
Probably
no letter
Probably
no letter
Probably
no letter
Probably no
letter
Probably no
letter
Probably no
letter
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 32 of 117
Barrier Analysis Matrix
The Barrier Analysis Matrix is an implementation of the Threat-Barrier-Target Model,
which has been a mainstay of the DOE approach to Root Cause Analysis for many
decades. The matrix below evaluates the principal barriers in terms of their
effectiveness during this episode only. Failed, missing, and ineffective barriers are often
causes of adverse events. The Barrier Analysis Matrix does not provide analysis of the
causes of the problem barrier conditions.
Barrier Analysis Matrix ©
Barrier Target
Protected
Threat Effectiveness
in this case
Significance
Note: In Barrier Analysis a 'barrier' is a device, measure, entity, or the like that has the
effect of reducing the probability or consequences of a 'threat' to a 'target'. Devices that
could have or are intended to have the same effect are also called 'barriers'.
Note: Barriers that did work can be retained and reinforced. Barriers that did not exist can
be deployed. Barriers that existed, but did not work, can be strengthened.
SNF Project
Notification
Guideline
Compliance
with
notification
requirements
Notifications
not made
Not Effective:
Guideline did not
ensure the FR
would be notified
when the FM is
notified of bad
news.
Missed
Opportunity:
Notification to
FR not made
FH sensitivity
to RL
requirement
that prompt
notification was
required for
non-reportable
bad news
Customer (RL)
satisfaction
Customer
(RL) finds out
they have not
been notified
as required
Not Effective:
Despite a history
of customer
dissatisfaction in
this area, FH did
not take effective
steps to rectify the
situation
Missed
Opportunity:
FH not sensitive
to RL
dissatisfaction
with lack of
prompt
notification.
FH Feedback
Systems (i.e.,
LL, RCA)
RL satisfaction
with prompt
notification
Lack of
awareness of
problems
Not Effective:
These processes
were not triggered
because there
was no
documentation of
customer
dissatisfaction –
required to initiate
feedback
processes
Missed
Opportunity:
The feedback
system did not
recognize an
emerging
escalating
problem
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 33 of 117
Barrier Analysis Matrix ©
Barrier Target
Protected
Threat Effectiveness
in this case
Significance
HNF-PRO-060 Compliance
with prompt
notification
requirement
for non-
reportable bad
news.
Non-
compliance
with prompt
notification
requirement
Not Effective:
Human Factors
not considered.
May have
prevented non-
notification
Review
process for
change to
HNF-PRO-060
Compliance
with prompt
notification
requirement
for non-
reportable bad
news.
Non-
compliance
with prompt
notification
requirement
Not Effective:
Procedure change
was not reviewed.
Missed
Opportunity :
CRD
requirements in
PRO-060 were
not implemented
Review cycle
could have
corrected human
factors
deficiency.
Procedure
verification and
validation of
HNF-PRO-060
Compliance
with CRD
Procedural
defects that
impede user
understanding
of
requirements
Not Effective:
Did not happen,
not required.
Missed
Opportunity:
V&V is always
available for
complex
processes.
Failed Barrier:
Systematic
Approach to
Training
(A post
training
knowledge
check is a
barrier against
continued
ineffective
knowledge.)
Training to
PRO-060
Ineffective
training
Not used. Missed
Opportunity:
Target
population was
not identified and
effectiveness of
training was not
verified.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 34 of 117
Barrier Analysis Matrix ©
Barrier Target
Protected
Threat Effectiveness
in this case
Significance
PRO-060
Training
Compliance
with
requirement to
notify RL of
non-reportable
bad news.
Non-
compliance
with
requirement
Not Effective:
Training to notify
RL of non-
reportable bad
news was not
performed. (SOM
& FM not trained
to revision.)
Missed
Opportunity:
Personnel were
not trained to the
change.
FM
Communication
of Facility Mgr
Forum
information
Compliance
with
requirement to
notify RL of
non-reportable
bad news.
Non-
compliance
with
requirement
Not Effective:
Not consistently
applied across
site
Missed
Opportunity:
Not all personnel
received
communication
Audits,
assessments,
and JHAs
Weighted
Safety Arm
(safety device)
Unrecognized
use of a
safety device
as an
operational
control
Not Effective:
If use of the safety
device as an
operational
control was
recognized,
additional controls
would have been
implemented.
Missed
Opportunity:
Did not
recognize the
use of Weighted
Safety Arms as
operational
controls.
Audits,
assessments,
and JHAs
Weighted
Safety Arm
(safety device)
Failure to
control for
condition that
sloping
monorails
could cause
hoist to strike
Weighted
Safety Arms.
Not Effective:
If the sloped
monorails were
appropriately
identified (even if
not re-leveled),
then additional
controls could
have been
implemented.
Missed
Opportunity:
An example of a
control would be
to hang caution
tags on the
hoists when they
are on slanted
monorails to say:
CAUTION: Keep
positive control
while moving
hoist. Monorail is
tilted and hoist
will continue to
travel on its own.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 35 of 117
Barrier Analysis Matrix ©
Barrier Target
Protected
Threat Effectiveness
in this case
Significance
Audits,
assessments,
and JHAs
Weighted
Safety Arm
(safety device)
Unrecognized
impact on the
locking arms
from moving
FTC during
coupling to
the monorail
Not Effective:
If damage caused
by the moving
FTC was
recognized,
additional controls
would have been
implemented.
Missed
Opportunity:
Did not
recognize
damage was
being caused by
FTC.
Preventive
Maintenance
procedure
Functional
Weighted
Safety Arm
Non-
functional
Weighted
Safety Arm
Not Effective:
Did not find the
bent/ sticking
Weighted Safety
Arm.
Missed
Opportunity:
Did not
incorporate
procedural steps
to ensure bent/
sticking latches
were identified.
Millwright Functional
Weighted
Safety Arm
Non-
functional
Weighted
Safety Arm
Not Effective:
Did not find the
bent/ sticking
Weighted Safety
Arm.
Missed
Opportunity:
Skill of the craft
could have
identified the
bent/ sticking
Weighted Safety
Arm.
Procedure OP-
14-002
“Perform Pre-
Use Test on
Hoist”
The Hoist. Damage to
Hoist.
Not Effective:
Did not require the
performance of a
pre-use test for
relocation of the
hoist AND, if a
pre-use test had
been required for
this activity, there
was no
requirement to
ensure the
Weighted Safety
Arm was in the
down position.
Missed
Opportunity:
The operating
procedure could
have required
the operator to
ensure the
Weighted Safety
Arm was in the
down position.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 36 of 117
Barrier Analysis Matrix ©
Barrier Target
Protected
Threat Effectiveness
in this case
Significance
Procedure OP-
07-113W
“Position and
Secure
Irradiated Fuel
at 105-KW”
The Hoist. Damage to
Hoist.
Not Effective:
Not intended to be
effective.
However, neither
this procedure nor
OP-14-002 would
have prevented
the hoist from
falling from the
monorail.
Missed
Opportunity:
The procedure
could have
required the
operator to
ensure the
Weighted Safety
Arm was in the
down position.
FTC operating
procedures
Safe operation Damage to
the Locking
Arms
(subsequently
damaging the
Weighted
Safety Arm -
bending it)
Not Effective:
Did not ensure
that FTCs were
not moving while
coupling to the
monorail
Missed
Opportunity:
The procedure
could have
required that the
operator ensure
that FTCs were
stopped prior to
coupling to the
monorail
Training for
hoist/monorail
system
operation
Safe operation Repeated use
of Weighted
Safety Arms
as operational
controls (to
stop the
hoist).
Not Effective:
Weighted Safety
Arms are used as
operational
controls.
Missed
Opportunity:
Train people to
keep positive
control of hoist
when moving it.
Training for
FTC operation
Safe operation Repeated
practice of
coupling
FTCs to
monorails
while FTCs
are still
moving
Not Effective:
Did not ensure
that FTCs were
not moving while
coupling to the
monorail
Missed
Opportunity:
The training
could have
required that the
operator ensure
that FTCs were
stopped prior to
coupling to the
monorail
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 37 of 117
Barrier Analysis Matrix ©
Barrier Target
Protected
Threat Effectiveness
in this case
Significance
SOM, FM,
MCO Mgr, and
Deputy Director
Questioning
Attitude with
respect to near
misses
Compliance
with near miss
reporting
requirements
and correction
of unsafe
conditions
Unidentified
near miss
situations
Not Effective:
Conversations
between SOM on
one hand and FM,
MCO Mgr, and
Deputy Director
respectively did
not surface the
near miss
Failed Barrier:
Questioning
attitude was not
exhibited.
Missed
Opportunity:
People at this
level often
exhibit
questioning
attitudes
Exacerbating
Factor:
If this barrier had
been successful,
the whole
episode would
have been much
less significant.
Operators Safe operation Continued
unsafe
operations
Success:
Workers stopped
work and notified
management
Without this, the
opportunity to
evaluate and
correct the
unsafe condition
would have been
missed.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 38 of 117
Discussion of Design Vulnerabilities
At the time of this hoist fall event, the K Basin Fuel Storage Monorail System was old.
The design was probably current at the time of construction. Comments on the design
vulnerabilities should be taken in that context. They are intended not as criticisms of the
design, but as possibilities for learning.
Discussion of Process Vulnerabilities
By process vulnerabilities, we mean vulnerabilities associated with the way evolutions
are conducted, i.e., the process for moving hoists. In this case one single adverse
contingency, a mis-positioned WSA, together with process issues, allowed the hoist
drop to occur. This discussion applies to the design the way it was at the time of the
hoist drop event.
Procedures
The procedure for operating hoists contained precautions and notes associated with the
upper limit switch on the hoist. This is appropriate since upper limit switches should not
be relied upon to stop the “UP” travel of the hoist. Conversely, similar steps were not
included to caution operators on not utilizing the WSA as a normal means for stopping
the trolley. Similarly, steps were not included to verify the WSA position prior to moving
the trolley or to ensure positive control of the hoist/trolley assembly is maintained during
movement to avoid challenging the WSA.
Procedures that direct the operation of FTCs do not contain steps to ensure that FTCs
are stopped prior to attempts to engaging the FTC to a monorail. No caution
statements or other steps warn operators of the damage that can occur to the Locking
Arm and WSA should an FTC be moving while engaging it to a monorail.
The procedure for performing preventative maintenance does not ensure that
deficiencies associated with the WSAs are identified. The procedure does not specify
any acceptance criteria and does not require recording “as found” conditions, actions
taken, and “as left” conditions.
The procedure used for defining and declaring Near Miss events does not ensure that
Near Miss events are recognized and reported. This problem exists not only here at
FH, but across the DOE Complex. Also, this same procedure is used to ensure prompt
notification of reportable and non-reportable occurrences to the DOE FR. However, the
steps in the procedure are not well placed to ensure that when FMs are notified of non-
reportable events (bad news), the FR is promptly notified of the same. The wording is
essentially the same as the CRD, but it does not need to be. Simple words that clearly
state that whenever the FM is notified of non-reportable “bad news”, the FR shall be
promptly notified of the same bad news are all that is required.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 39 of 117
Training
The training on the new CRD requirement for prompt notifications (as codified in the
Occurrence Reporting procedure) was not adequate to ensure compliance with the
CRD and customer expectations. While both the procedure and training did contain this
information, the recently inserted requirement was not clear to most personnel who
received this training. Many believed that the information was merely more of the same
(notify the FR of abnormal events), without recognizing the significance of the change
(FR shall be promptly notified anytime the FM is notified of “bad news”).
In addition to the training content, the SAT did not appear to be used. No knowledge
check was used to ensure the objectives of the training were met and no follow-up
reviews were conducted to ensure the new requirement was being implemented. Also,
while most of the applicable personnel were originally identified that needed this
training, it does not appear that actions were taken to ensure they were scheduled,
attended, or that it was added to their Training Matrix as required training to perform
their assigned duties.
Hoisting and Rigging training is provided to operators who use hoists. Specifically, the
monorail hoist training contains information regarding the checking of upper limit
switches and the need to ensure that operators due not use upper limit switches as a
normal means to stop the “UP” travel of the hoist. No information could be found
regarding the need to check the position of WSAs and to ensure positive control is
maintained when moving the hoist/trolley assembly so that it does not collide with
WSAs. Similarly, it would appear the training for operating FTCs does not warn against
moving FTCs while engaging them to monorails. This practice has been witnessed in
the past and is believed to be a probable cause for the damage being done to WSAs.
Safety Vulnerability Audits
Safety audits and assessments have failed to recognize “normal” operations as
potential event precursors. The practice of allowing hoist/trolley assemblies to collide
with WSAs, and the practice of engaging FTCs to monorails while they are still moving,
was known by many within the SNF Project. The fact that these practices have been
known, but allowed to continue points to a less than adequate sensitivity towards single
vulnerability safety devices such as the WSAs.
Discussion of Process Strength
As soon as the scrap processing crew personnel were aware that the noise was caused
by the fallen hoist, they stopped work and investigated the scene. The personnel then
notified the SOM, who sent other personnel to investigate the fallen hoist. The use of
“Stop Work” by the crew upon discovery of an abnormal event is encouraging. The
prompt notifications made to the SOM and attempts to secure the scene were also
encouraging.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 40 of 117
Discussion of Influences
Although the definitive cause(s) for the bent WSA is not known, the WSA was likely bent
as a result of the hoist/trolley assembly colliding with the WSA and/or the FTC colliding
with the Locking Arm while it was moving and engaging to the monorail. The influence
the bent WSA had on this event is that it stuck when moved to the up position.
The definitive cause for the WSA being moved to the up position is unknown. However,
one possible scenario is that long tools may have contacted the WSA, moving it to the
up position. It is not clear to what extent hanging tools from the WSAs might contribute
to it being moved to the up position. The influence of the WSA being in the up position
is that it no longer provided that “rail stop” function it was designed to provide. Thus,
when the hoist/trolley assembly was pushed to the south, it continued to roll until it
passed the bent WSA in the up position and fell to the grating below.
The cause for monorail 21 being tilted to the south is likely the result of a combination of
aging/old equipment whose preventative maintenance program does not inspect and/or
correct for this condition. The influence this tilted monorail had on this event is that
once the hoist/trolley assembly was pushed to the south, it continued to roll down grade
until it reached the end of the monorail, passing the bent WSA stuck in the up position,
and fell to the grating below.
The cause for the RCT to push the hoist to the south is not significant. However, it is
important that if RCTs and other non-qualified personnel are permitted to move
hoists/trolley assemblies out of their way, they need to understand that they must have
positive control and have verified the position of WSAs. The influence the RCT had on
this event is the creation of the momentum necessary to cause the hoist/trolley
assembly to roll south on the monorail until it fell to the grating below.
The fact that no pre-use check of WSA positions is required had an influence on this
event. Had personnel been trained, and the procedure directed the performance of this
check prior to movement of the hoist/trolley assembly, the WSA would have been
discovered in the up position and the event would have been avoided.
The cause for the less than adequate preventative maintenance (PM) for WSAs
appears to be due to the lack of recognition that this equipment performs a vital safety
function. The influence this had on this event is that the PM (as currently written) did
not ensure that the bent WSA (safety device) was identified and corrected.
Although the area where the hoist fell is a normal traffic area, no personnel were in the
vicinity of the end of the monorail when the hoist fell. This had a beneficial influence on
this event since the result was no personnel were injured (Near Miss).
The cause for the RL concern regarding prompt notifications (or the lack there of) is a
result of a lack of sensitivity (past and present) to this issue by FH. The RCA Team
believes that this condition alone likely influenced the customer to issue a formal letter
of concern.
The CRD wording incorporated into HNF-PRO-060 had a significant influence in the
prompt notification issue reflected in this report. The fact that the wording was “overly
wordy” likely contributed to a lack of recognition that this was a change from past
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 41 of 117
practice (FH has always been required to notify the FR of abnormal events). Also, the
fact that the change was not human factored when incorporated into HNF-PRO-060
was significant because personnel following the procedure would not reach this step
since the section it is listed in deals with “Reportable Occurrences”.
Training to the new requirement (HNF-PRO-060) for prompt notification to the FR
whenever the FM is notified was ineffective. The influence this had on personnel is a
lack of sensitivity to a historical concern from RL and an inability to recognize the need
to promptly notify the FR whenever the FM is notified of “bad news”. This training also
failed to correct a long standing problem of identifying Near Miss events. Not
surprisingly, this problem still exists throughout the DOE Complex and FH has yet to
satisfy the customer’s concern regarding a perceived lack of sensitivity to this issue.
The inability of management and others to apply a “questioning attitude” to some events
may influence others to not recognize them as near misses. Despite the design,
process, administrative, and operational issues, the application of a “questioning
attitude” alone would have prevented this event or made it less consequential at many
junctures.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 42 of 117
Corrective Action Recommendations
The Corrective Action Table addresses the principal influences on the consequences of
this episode. It not only includes recommended corrective actions, but also provides an
assessment of the expected impact of the corrective action, if implemented. The table
also addresses the expected impact on the episode being investigated if the corrective
action had been in place before the event. For each corrective action recommended,
the team provides its advice on the priority of the corrective action.
Corrective Action Table
Consequence
Influence
Corrective
Action
Expected
Impact
Impact on
Current Event
if pre-
implemented
Remarks
Not keeping
positive control
of the hoist
when moving it.
Procedure
change:
positive control
of hoist when
moving it.
High Priority
Hoist does not
hit WSA and
does not fall
off a monorail
when there is
no WSA.
None on this
event (since
RCT does not
follow
operating
procedures.)
Need corrective
action related to
RCT
performance.
RCT moved
hoist without
proper
precautions.
Procedure
change: all
activities in high
hazard areas to
be done by
qualified
individuals.
High Priority
People moving
hoists and
doing other
potentially
hazardous
activities will
act safely.
RCT either
would have
been properly
qualified or
would have
asked for a
properly
qualified
individual to
move the hoist.
Maintaining
positive control
would be one of
the proper
precautions.
Tilted monorail. None: Residual
risk
None. None. Previous
corrective
actions would
make tilted
monorail non-
significant.
Need USQE.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 43 of 117
Corrective Action Table
Consequence
Influence
Corrective
Action
Expected
Impact
Impact on
Current Event
if pre-
implemented
Remarks
Distorted WSA
hardware.
PM instruction
change:
Effective
checking and
condition
recording of
WSA hardware.
Medium Priority
WSA hardware
problems
would be
identified.
WSA hardware
problems
would have
been identified
a month before
the event, and
the event
would not have
happened.
PM leaves a
time window for
adverse
conditions to
remain
undetected.
This barrier is
not 100%
effective.
Distorted WSA
hardware.
Procedure
change: Pre-
use operational
checks of WSA
hardware on
monorails to be
used.
High Priority
WSA mis-
positioning
would be
identified and
corrected
before moving
hoists.
None on this
event (since
RCT does not
follow
operating
procedures.)
Need corrective
action related to
RCT
performance.
Distorted WSA
hardware.
Procedure
change:
Operational
controls to
ensure FTCs
are not moving
when coupling
with monorails
High Priority
FTC does not
impact Locking
Arm
Locking Arm
and
subsequent
WSA damage
may not have
occurred
Reduces
potential for
damaging the
locking arms
and subsequent
WSAs
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 44 of 117
Corrective Action Table
Consequence
Influence
Corrective
Action
Expected
Impact
Impact on
Current Event
if pre-
implemented
Remarks
Numerous
conditions
adverse to
safety
associated with
operation and
material
condition of the
FSMS.
Apply ISMS to
FSMS
operation and
maintenance.
Low Priority
Operational
and material
hazards will be
identified and
controlled.
WSA distortion
issues would
have been
identified.
Safety device
precursor
issues would
have been
identified.
Positive control
issues would
have been
identified, and
so forth.
There was no
evidence that
FSMS operation
and
maintenance
had received the
benefit of ISMS.
SOM does not
recognize near
miss.
FH Senior
Mgmt establish
and enforce
their
expectations for
near miss
recognition by
FH personnel.
High Priority
Reduced
likelihood of
mis-
identification of
near miss
situations.
SOM would
have
recognized
near miss.
Near miss
would have
been reported.
No opportunity
to
misunderstand
CRD
requirements.
Without Senior
Mgmt
reinforcement of
good near miss
declaration
performance,
this corrective
action will be
useless.
Inadequate
challenge of
SOM telephone
notification
results in
situation not
being fully
described to
FM.
Questioning
attitude
workshop
training for all
mgmt personnel
whose duties
include
receiving
notifications
from Shift
Managers.
High Priority
Shift
Management
personnel will
receive
collegial
assistance
from the senior
managers to
whom they
make
notifications.
FM would have
led SOM
through the
proper logic to
identify that
only luck stood
between the
hoist drop
situation and
an injury.
All new
managers in this
situation must
be trained
before taking on
duties involving
receiving
notifications.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 45 of 117
Corrective Action Table
Consequence
Influence
Corrective
Action
Expected
Impact
Impact on
Current Event
if pre-
implemented
Remarks
Inadequate
challenge of
SOM telephone
notification
results in
situation not
being fully
described to
MCO
Production Mgr.
Questioning
attitude
workshop
training for all
mgmt personnel
whose duties
include
receiving
notifications
from Shift
Managers.
High Priority
Shift
Management
personnel will
receive
collegial
assistance
from the senior
managers to
whom they
make
notifications.
MCO
Production Mgr
would have led
SOM through
the proper
logic to identify
that only luck
stood between
the hoist drop
situation and
an injury.
This is a repeat
corrective
action; it only
needs to be
done once.
HNF-PRO-060
contained
human factors
defects.
Subject HNF-
PRO-060 to
validation and
verification that
includes
consideration of
human factors
effectiveness.
Medium Priority
On-shift
personnel
involved in
occurrence
response will
be able to pick
up the
procedure and
succeed.
SOM would
have
understood
that “non-
reportable” bad
news given to
the FM should
be given to the
FR. No
notification
issue would
have arisen.
HNF-PRO-060
is probably not
the only
administrative
procedure that
is a human
factors problem.
HNF-PRO-060
training was
ineffective in
that it did not
result in the
desired
performance.
Retrain
personnel on
this procedure
meeting the
intent of the
Systematic
Approach to
Training.
High Priority
On-shift
personnel
involved in
occurrence
response will
be able to pick
up the
procedure and
succeed.
SOM would
have
understood
that “non-
reportable” bad
news given to
the FM should
be given to the
FR. No
notification
issue would
have arisen.
SAT is just good
business.
Training that
does not meet
the intent of
SAT is not
effective.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 46 of 117
Corrective Action Table
Consequence
Influence
Corrective
Action
Expected
Impact
Impact on
Current Event
if pre-
implemented
Remarks
There was no
post-training
effectiveness
check on HNF-
PRO-060
training.
Conduct post-
training
effectiveness
checks on
future HNF-
PRO-060
training.
High Priority
Any significant
performance
issues caused
by HNF-PRO-
060 training
will be
detected.
Significant
performance
issues would
have been
identified and
corrected.
SOM would
have
understood
that “non-
reportable” bad
news given to
the FM should
be given to the
FR. No
notification
issue would
have arisen.
This would be
included in
meeting the
intent of SAT.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 47 of 117
Analysis of Expected Effectiveness of Recommended
Corrective Action
The RCA Team has concluded that if FH implements the corrective actions marked
“high priority”, FH will not have another event like the hoist drop event and will not have
another notification shortfall. It is important that the effectiveness of these corrective
actions be periodically confirmed.
Conclusions
The RCA Team completed all four objectives of its charter.
In the context of high hazard industry events, this one is of rather low consequence.
However, its significance is high because it is a result of behaviors and conditions that
could be ingredients of much more consequential events.
The team believes that the implementation of the recommended corrective actions and
the dissemination of the lessons to be learned will likely result in the saving of lives,
pain, assets, and careers at Fluor Hanford.
The three main areas for improvement are:
1. Conduct of Operations and Maintenance.
2. Handling of Adverse Event Information
3. Rapid Investigative Response
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 48 of 117
Attachments
Comparative TimeLine ©......................................................................................... 49
Why Staircase Trees............................................................................................... 73
Extent of Condition and Cause Matrix..................................................................... 83
Cause Codes .......................................................................................................... 92
Documents Reviewed ........................................................................................... 103
Personnel Contacted............................................................................................. 106
Analysts Comments .............................................................................................. 108
Structural Analysis Report..................................................................................... 111
Calculations........................................................................................................... 113
Unanswered Questions......................................................................................... 117
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 49 of 117
Comparative TimeLine©
The Comparative TimeLine©
is a convenient method of capturing the behaviors and conditions that made up the episode.
The resources required to carefully construct a Comparative TimeLine pay for themselves many times over in making it
easier to do the Why Staircase Trees, the Barrier Analysis Matrix, the Missed Opportunity Matrix, and other tools.
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
Earlier FH management chose not
to notify RL of certain “non-
reportable” bad news that
was reported to Facility
Managers (FMs).
FH management should
have determined RL needs
and complied.
RL becomes dissatisfied. Set-up Factor:
This sets the stage for the
notification consequence.
About
2003_09
Paul Golan (DOE-HQ)
established metrics that
were drivers for RL
notifications to HQ.
OK RL identifies what types of
things it wants to be notified
of. These included “non-
reportable” bad news
usually reported to FMs.
Set-up Factor:
Creates additional need for
Facility Representatives
(FRs) to get the types of
information they wanted
anyway.
Prior to
2003_11_03
RL drafted Contracts
Requirements Document
(CRD) asking for
notification to FR of bad
news reported to FM.
OK FH has to change PRO-60
to implement this
requirement.
Set-up Factor:
Creates formal requirement
for notifying the FR of non-
reportable bad news.
Prior to
2003_07
A revision of HNF-PRO-060
not including the CRD was
in the review cycle. It had
been reviewed by many
reviewers.
OK Created opportunity to
change PRO-60 after it had
been reviewed.
The post-review addition of
the CRD provisions meant
that some early reviewers
thought they knew what
was in PRO-60, but didn't.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 50 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
Prior to
2003_11_03
Shift Managers,
Occurrence Reporting
personnel, and others were
trained on the Revised
PRO-060 (with CRD
provisions). The K West
Basin Operations Shift
Manager (SOM) was not
trained.
The SOM should have
received this training.
The SOM was not aware of
FR notification
requirements for non-
reportable bad news.
Set-up factor:
Creates a condition for non-
compliance with CRD.
2003_11_03 CRD provisions for non-
reportable bad news were
inappropriately placed in a
section on reportable
occurrences.
CRD provisions for non-
reportable bad news should
have been appropriately
placed in a section on non-
reportable occurrences that
are reported to FM’s,
placed in a table, or
otherwise presented in a
human factored way.
Increased the probability of
non-compliance.
Set-up factor:
Creates a condition for non-
compliance with CRD.
2003_11_03 HNF-PRO-060 Revision 7
issued with CRD
provisions.
Revised PRO-60 should
have gone back through
the review cycle.
Reviewers did not have the
opportunity to comment on
the human factors
deficiencies in how CRD
provisions were
incorporated.
Failed barrier:
Review cycle could have
corrected human factors
deficiency.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 51 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
After
2003_11_03
Director of Emergency
Preparedness briefs Facility
Managers’ Forum on CRD
provisions in PRO-060.
Only the FMs received this
CRD training.
The appropriate target
audience should have been
trained on the CRD
provisions in PRO-060 in
accordance with the
Systematic Approach to
Training (SAT). The K
West Basin FM did not
receive this training.
Not all the personnel
involved in notifying FRs of
non-reportable bad news
are trained to do so.
Missed Opportunity:
All personnel involved in
notifying FRs could have
been trained to do so.
Failed Barrier:
- SAT is a barrier against
ineffective training
Set-up factor:
Creates a condition for non-
compliance with CRD.
After
2003_11_03
The Director of EP does not
check to see that the
training was effective, i.e.,
that personnel involved in
notifying FR’s of non-
reportable bad news know
they are supposed to do so.
In accordance with SAT,
trainees’ knowledge should
have been checked.
The training deficiency is
not noticed by FH.
Missed Opportunity:
A check of the training
would have found that it
was ineffective - Leaves
set-up factors in place.
Failed barrier:
A post training knowledge
check is a barrier against
continued ineffective
knowledge.
Set-up factor:
Creates a condition for non-
compliance with CRD.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 52 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
After
2003_11_03
The FMs who attended the
Forum did not tell facility
personnel about the
requirement to notify the
FR about non-reportable
bad news when the FM is
notified.
The FMs should have
informed facility personnel
about the requirement to
notify the FR
Not all the personnel
involved in notifying FRs of
non-reportable bad news
are informed of the
requirement to do so.
Missed Opportunity:
All personnel involved in
notifying FRs could have
been informed to do so.
Failed Barrier:
- Communications LTA –
The FMs did not
communicate the new CRD
requirement to facility
personnel AND the PRO-
060 author did not
communicate the new CRD
requirement to all of the
procedure users.
After
2003_11_03
Non-reportable bad news is
reported to FMs, but not to
FRs.
Non-reportable bad news
reported to FMs should
also have been reported to
FRs.
FH Management not
sensitive to Increased RL
dissatisfaction
Set-up Factor:
Increases level of FR
dissatisfaction.
After
2003_11_03
FR complaint about not
being told about non-
reportable bad news
reported to FMs.
OK FH management not
sensitive to RL
dissatisfaction
Set-up Factor:
Problem continues
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 53 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
After
2003_11_03
FH management does not
reinforce expectation for
CRD compliance, that FRs
be notified when FMs are
notified.
FH management should
have noted that previous
training was LTA and fixed
it.
Conditions for non-
compliance continue.
Missed Opportunity:
FH management could
have aborted the
notification portion of this
event by correcting a
known condition adverse to
business.
Failed barrier:
Management failed to
correct the non-compliant
condition (via training
improvement and
reinforcement of
expectations)
Set-up Factor:
Leaves the previous set-up
factors in place.
The Fuel Storage Monorail System (FSMS) is designed such that a single barrier prevents a hoist from dropping off the monorail. This
single barrier is the Weighted Safety Arm (safety device). Training does not sensitize users of this single failure vulnerability. The
combination of the design and the way it was used is an example of using safety protection for operational control.
Earlier Workers routinely rely on
Weighted Safety Arms to
stop moving hoists during
normal operations (passive
safety devices used as
operational controls).
Workers should not have
been using WSAs as hoist
stopping devices.
Repeated hoist drop
precursors (the Weighted
Safety Arm safety device
was repeatedly
challenged).
Set-up Factor:
If the hoist is moved to the
end of the Monorail and the
Weighted Safety Arm is up
or breaks, the hoist will fall.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 54 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
Earlier Audits, assessments, and
Job Hazards Analyses
(JHAs) of the FSMS do not
notice the single failure
vulnerability of the
Weighted Safety Arms.
Audits, assessments, and
JHAs should have noticed
the single failure
vulnerability and required
that procedures be
changed to either
frequently verify the
condition of the Weighted
Safety Arms, or prohibit
challenging them.
Set-up Factor is not
corrected.
Missed Opportunity:
Any of these safety audits
could have aborted the
event by establishing
additional controls.
Failed Barrier:
Audits, assessments, and
JHAs are barriers that
guard against safety
vulnerabilities.
ISMS infraction.
Earlier Audits, assessment, and
JHAs of the FSMS do not
notice the routine repeated
use of the WSA (safety
device) to stop a hoist.
Audits, assessments, and
JHAs should have noticed
the routine repeated use of
the safety device to stop a
hoist.
The recognition of the use
of WSAs to stop hoists
should have resulted in PM
and Operating procedures
that ensure their position
and operability.
The precursor continues. Missed Opportunity:
Any of these could have
aborted the event by
identifying and correcting
the unsafe behavior.
Failed Barrier:
Audits, assessments, and
JHAs are barriers that
guard against safety
vulnerabilities.
ISMS infraction.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 55 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
2003_02_03 100K monorail inspection
procedure (annual
inspection) SP-14-005
changed to add rail stops
(Weighted Safety Arms) to
Step 4.1.2 to ensure they
were checked for
looseness and alignment.
OK Missed Opportunity:
The procedure change
acknowledges the need for
Millwrights to check
Weighted Safety Arms, but
many Weighted Safety
Arms were not in proper
alignment.
2003_06_09 The Crane Inspection
Team was chartered as a
work place improvement
initiative for the K Basins.
The charter was to evaluate
reported issues and make
recommendations for
resolution.
OK
The FSMS was first built in 1954, and has become worn out over the years, requiring extensive repair to restore “normal” operation.
Although some FTCs have been replaced with new ones, original FTCs are worn to the point that alignment is not possible with many of
the monorails. Additional damage results from the increased forces necessary for the workers to align FTCs with monorails during hoist
transport of fuel loads exceeding 700 pounds. Worker frustration resulting from difficult FSMS operations with worn-out equipment may
have contributed to equipment abuse.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 56 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
2004_02_19 FSMS Maintenance
completed the annual
preventive maintenance
inspection on the Weighted
Safety Arms in the K West
Basin. No discrepancies
were noted in the work
package.
Discrepancies (bent or
sticking Weighted Safety
Arms) should have been
found.
No problem was found with
the Weighted Safety Arm
on the south end of
Monorail 21. The bent
WSA was not identified and
corrected.
Missed Opportunity:
The bent or sticking
condition of Weighted
Safety Arms -
FSMS Maintenance does
not provide adequate
documentation of any
repairs in the work
packages to provide data
for trending.
Failed Barrier:
No problems with Weighted
Safety Arms were identified
by performing the PM.
Set-up Factor:
Problem continues.
2004_02_19 FSMS preventative
maintenance was
performed to a procedure
that did not require that an
FTC be installed when
inspecting WSA
functionality.
The PM procedure should
have required that an FTC
be engaged to the
monorail.
No problem was found with
the Weighted Safety Arm
on the south end of
Monorail 21. The bent
WSA was not identified and
corrected.
Missed Opportunity:
The bent or sticking
condition of Weighted
Safety Arms.
Failed Barrier:
Because the FTC was not
engaged to the monorail,
no problems with Weighted
Safety Arms were identified
by performing the PM.
Set-up Factor:
Problem continues.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 57 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
Before
2004_03_10
Personnel had been
working in the basin pool
under the south end of
Monorail 21 using long
tools.
OK Created opportunity for
personnel to strike
Weighted Safety Arm with
long tool during work.
Set-up Factor:
Possible inadvertent
contact with Weighted
Safety Arm causing it to
travel beyond its normal
range of motion, and
allowing it to become stuck.
Missed Opportunity:
Long tools were known to
strike or become stuck in
the overhead during use.
Before
2004_03_10
Personnel had been
working in the basin pool
under the south end of
Monorail 21.
OK Created opportunity for
personnel to be injured if
the hoist falls on them.
Exacerbating Set-up
Factor:
This converts a potential
equipment event into a
personnel safety event.
Personnel working in the basin use long poles to retrieve empty fuel canisters. The canisters are hooked to a chain on the hoist, so the
hoist can lift the canister in the water, and the canister can be transported along the monorail. The hoists roll by manual operation, but
lift by means of an electric motor. The monorail system is approximately seven feet above the grating over the water. The grating is
approximately 22 feet above the bottom of the pool. In use, long poles are reported to often strike into the overhead.
Before
2004_03_10
The Weighted Safety Arm
was bent.
The Weighted Safety Arm
should not have been bent.
The bent Weighted Safety
Arm became stuck in the
up position.
Set-up Factor:
If the Weighted Safety Arm
had not been bent, it would
have been in the down
position.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 58 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
2004_03_10 The south-end Weighted
Safety Arm on Monorail 21
is in the up position.
The Weighted Safety Arm
should have been in the
down position.
The Weighted Safety Arm
could not perform its
intended function. If the
hoist travels to the end of
the monorail, it will fall off.
Set-up Factor:
The Weighted Safety Arm
in the up position will allow
a hoist traveling to the end
of the monorail to fall off.
Missing Barrier:
The Weighted Safety Arm
in the down position
(barrier) was missing.
Before
2004_03_10
Pre-use hoisting and
rigging checks do not
identify the Weighted
Safety Arm on Monorail 21
in the up position.
Pre-use hoisting and
rigging checks should
identify safety anomalies
including the Weighted
Safety Arm on Monorail 21
being in the up position.
Set-up Factor allowed to
continue.
Missed Opportunity:
The pre-use checks could
have aborted the event by
identifying the Weighted
Safety Arm in the up
position.
Missing Barrier:
The pre-use checks did not
require checks of the
Weighted Safety Arms.
2004_03_10_
2317
Radiological Control
Technician (RCT) who is
part of a waste processing
crew needs to move hoist.
OK The RCT moves the Hoist. Triggering Factor.
2004_03_10_
2317
(before)
Monorail 21 was tilted
downward to the south. (A
number of monorails in the
K West Basin were known
for many years to have “a
long glide path”.)
Monorail 21 should have
been level.
Created vulnerability to
continued hoist motion
once the hoist is started.
Set-up Factor.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 59 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
The Weighted Safety Arms were verified in the down position three weeks preceding this event by personnel working at the south end of
Monorail 21.
2004_03_10_
2317
RCT, following normal
practice, does not request a
Qualified Operator to move
hoist.
RCT should have
requested Qualified
Operator to move the hoist.
A non-qualified operator
moved the hoist.
Extraneous Condition
Adverse to Quality
2004_03_10_
2317
Procedure OP-14-002
“Perform Pre-UseTest on
Hoist” did not require pre-
use testing when relocating
hoist.
The procedure should have
required visual inspection
to verify the Weighted
Safety Arms (safety device)
were in the down position.
The Weighted Safety Arm
was not visually inspected.
Set-up Factor
Missing Barrier:
The procedure did not
require a pre-use check
before the hoist was
relocated.
Procedure OP-14-002 “Perform Pre-UseTest on Hoist” precautions and limitations stated the following: “Limit switches are safety
devices for protection of personnel and equipment. Limit switches shall not be used as a controller to stop the upward travel of the
hoist.” Section 4.2 includes the following note: “The up limit switch is a back-up; its use should be avoided during normal operation.”
The pre-use test procedure does not recognize the Weighted Safety Arms as safety devices and does not call out pre-use inspection.
The team could not find any safety principle that allowed treating mechanical safety devices less rigorously than electrical safety
devices.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 60 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
2004_03_10_
2317
RCT pushes hoist to south. A Qualified Operator should
have moved it.
(Note that if the hoist had
been moved to the north
this event would not have
happened.)
Hoist begins moving to the
south. (We do not know
whether the Qualified
Operator would have
behaved differently. E.g., a
Qualified Operator may
have intuitively moved the
hoist northward or may
have noticed the Weighted
Safety Arm in the up
position, or may have
moved it south more
carefully.)
Final Triggering Factor.
2004_03_10_
2317
The hoist rolls southward
and travels off the end of
the monorail.
OK (given the existing set-
up factors)
Hoist is in free-fall. Natural outcome of prior
behaviors and conditions.
2004_03_10_
2317
No personnel or sensitive
equipment are in vicinity of
the hoist trajectory.
OK: Fortuitous condition No personnel or equipment
consequences (except to
hoist casing).
Mitigating Factor. This
non-robust barrier
prevented this near miss
from being a consequential
Safety Event.
2004_03_10_
2318
The five involved personnel
(1 RCT and 4 Nuclear
Chemical Operators
(NCOs)) went to the fallen
hoist and confirmed the
Weighted Safety Arm was
in the down position. The
immediate inspection found
no obvious problems with
the hoist or monorail.
OK Inexplicable condition
discovered.
Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004
Page 61 of 117
When What did happen What should have
happened
Immediate Result
(Consequence)
Significance (Impact on
Final Consequences)
2004_03_10_
2330
An NCO notified the K
West Basin SOM that a
hoist had fallen off the
south end of Monorail 21
onto the grating; no
personnel were in the area
of the fallen hoist.
OK Notification sequence starts
2004_03_10_
2330
SOM does not declare that
the hoist drop was a near
miss for personnel injury.
Because the hoist dropped
in an area where people
work, the drop should have
been recognized as a near
miss.
The hoist drop is not
treated as a near miss and
near miss notifications are
not made.
Missed Opportunity:
First Failed Barrier to CRD
non-compliance.
2004_03_10_
2330
Discussion between the
personnel and the SOM led
to determination that, other
than the hoist pre-use
operational check, no
specific check of the
Weighted Safety Arms is
required; also, that no
Flexible Transfer Cranes
(connecting to Weighted
Safety Arms on the
monorails) had been in use
along the south part of the
basin.
OK
2004_03_10_
2330
Interviews of involved
personnel were not
conducted.
Interviews should have
been obtained as soon as
possible after the event.
Extraneous Condition
Adverse to Quality
Extraneous Condition
Adverse to Quality:
Real-time recording of the
facts at the event scene did
not occur.
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail
Doe hanford hoist drop from monorail

More Related Content

What's hot

Draft Bill to Extend SAFETEA-LU for Six Months
Draft Bill to Extend SAFETEA-LU for Six MonthsDraft Bill to Extend SAFETEA-LU for Six Months
Draft Bill to Extend SAFETEA-LU for Six Months
Ports-To-Plains Blog
 
Failure to Follow Procedures
Failure to Follow Procedures   Failure to Follow Procedures
Failure to Follow Procedures
FAA Safety Team Central Florida
 
Safe cable excavation___mod_h&s_leaflet_33
Safe cable excavation___mod_h&s_leaflet_33Safe cable excavation___mod_h&s_leaflet_33
Safe cable excavation___mod_h&s_leaflet_33
Moses Opuoru
 
Npc force weekly 11 15 july 2011
Npc force weekly 11 15 july 2011Npc force weekly 11 15 july 2011
Npc force weekly 11 15 july 2011
Master Chief Petty Officer of the Navy
 
Crash during approach to landing of maryland state police aerospatiale sa36...
Crash during approach to landing of maryland state police   aerospatiale sa36...Crash during approach to landing of maryland state police   aerospatiale sa36...
Crash during approach to landing of maryland state police aerospatiale sa36...
Jeferson Espindola
 
Eidws 103 organization
Eidws 103 organizationEidws 103 organization
Eidws 103 organization
IT2Alcorn
 

What's hot (6)

Draft Bill to Extend SAFETEA-LU for Six Months
Draft Bill to Extend SAFETEA-LU for Six MonthsDraft Bill to Extend SAFETEA-LU for Six Months
Draft Bill to Extend SAFETEA-LU for Six Months
 
Failure to Follow Procedures
Failure to Follow Procedures   Failure to Follow Procedures
Failure to Follow Procedures
 
Safe cable excavation___mod_h&s_leaflet_33
Safe cable excavation___mod_h&s_leaflet_33Safe cable excavation___mod_h&s_leaflet_33
Safe cable excavation___mod_h&s_leaflet_33
 
Npc force weekly 11 15 july 2011
Npc force weekly 11 15 july 2011Npc force weekly 11 15 july 2011
Npc force weekly 11 15 july 2011
 
Crash during approach to landing of maryland state police aerospatiale sa36...
Crash during approach to landing of maryland state police   aerospatiale sa36...Crash during approach to landing of maryland state police   aerospatiale sa36...
Crash during approach to landing of maryland state police aerospatiale sa36...
 
Eidws 103 organization
Eidws 103 organizationEidws 103 organization
Eidws 103 organization
 

Similar to Doe hanford hoist drop from monorail

Homko Colgan Flight 3407 HFACS Safety Analysis
Homko Colgan Flight 3407  HFACS Safety AnalysisHomko Colgan Flight 3407  HFACS Safety Analysis
Homko Colgan Flight 3407 HFACS Safety Analysis
Chris Homko MS, ATP
 
Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)
Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)
Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)
Steph Cliche
 
Http _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_report
Http  _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_reportHttp  _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_report
Http _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_report
OCIMF OVID
 
Strictly Financials 2014: Chasing Chesapeake by Brian Grow
Strictly Financials 2014: Chasing Chesapeake by Brian GrowStrictly Financials 2014: Chasing Chesapeake by Brian Grow
Strictly Financials 2014: Chasing Chesapeake by Brian Grow
Reynolds Center for Business Journalism
 
Geotechnical engineering procedures for foundation
Geotechnical engineering procedures for foundationGeotechnical engineering procedures for foundation
Geotechnical engineering procedures for foundation
Goyo Cisneros García
 
Reserve Force weekly 11 july 2011
Reserve Force weekly 11 july 2011Reserve Force weekly 11 july 2011
Reserve Force weekly 11 july 2011
Master Chief Petty Officer of the Navy
 
06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto
06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto
06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto
artba
 
Hoist drop from monorail investigation
Hoist drop from monorail investigationHoist drop from monorail investigation
Hoist drop from monorail investigation
William R. Corcoran, Ph.D., P.E.
 

Similar to Doe hanford hoist drop from monorail (8)

Homko Colgan Flight 3407 HFACS Safety Analysis
Homko Colgan Flight 3407  HFACS Safety AnalysisHomko Colgan Flight 3407  HFACS Safety Analysis
Homko Colgan Flight 3407 HFACS Safety Analysis
 
Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)
Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)
Mv 22 ci-final_redacted_copy_for_public_release_without_enclosures_(14_aug_12)
 
Http _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_report
Http  _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_reportHttp  _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_report
Http _www.maib.gov.uk_cms_resources.cfm_file=_sbs_typhoon_report
 
Strictly Financials 2014: Chasing Chesapeake by Brian Grow
Strictly Financials 2014: Chasing Chesapeake by Brian GrowStrictly Financials 2014: Chasing Chesapeake by Brian Grow
Strictly Financials 2014: Chasing Chesapeake by Brian Grow
 
Geotechnical engineering procedures for foundation
Geotechnical engineering procedures for foundationGeotechnical engineering procedures for foundation
Geotechnical engineering procedures for foundation
 
Reserve Force weekly 11 july 2011
Reserve Force weekly 11 july 2011Reserve Force weekly 11 july 2011
Reserve Force weekly 11 july 2011
 
06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto
06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto
06/03/11: Amicus Brief Opposing EPA Wetlands Permit Veto
 
Hoist drop from monorail investigation
Hoist drop from monorail investigationHoist drop from monorail investigation
Hoist drop from monorail investigation
 

More from William R. Corcoran, Ph.D., P.E.

Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017
Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017
Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017
William R. Corcoran, Ph.D., P.E.
 
Hprct 2011 barrier analysis_upd 2017.08.01
Hprct 2011 barrier analysis_upd 2017.08.01Hprct 2011 barrier analysis_upd 2017.08.01
Hprct 2011 barrier analysis_upd 2017.08.01
William R. Corcoran, Ph.D., P.E.
 
Treatment of concerned nrc employee 2016 sept-27 transcript
Treatment of concerned nrc employee 2016 sept-27 transcriptTreatment of concerned nrc employee 2016 sept-27 transcript
Treatment of concerned nrc employee 2016 sept-27 transcript
William R. Corcoran, Ph.D., P.E.
 
Gas Line Near Indian Point Nuclear Power Plant
Gas Line Near Indian Point Nuclear Power PlantGas Line Near Indian Point Nuclear Power Plant
Gas Line Near Indian Point Nuclear Power Plant
William R. Corcoran, Ph.D., P.E.
 
Five types of factor building blocks with case study 2014.05.05.0730
Five types of factor building blocks with case study 2014.05.05.0730Five types of factor building blocks with case study 2014.05.05.0730
Five types of factor building blocks with case study 2014.05.05.0730
William R. Corcoran, Ph.D., P.E.
 
E atomizer explosion 2013.10.26
E atomizer explosion 2013.10.26E atomizer explosion 2013.10.26
E atomizer explosion 2013.10.26
William R. Corcoran, Ph.D., P.E.
 
A Foundation For Root Cause Analysis 2011.05.25
A Foundation For Root Cause Analysis 2011.05.25A Foundation For Root Cause Analysis 2011.05.25
A Foundation For Root Cause Analysis 2011.05.25
William R. Corcoran, Ph.D., P.E.
 
Foreign Material Mgt 2010 02 27 0901
Foreign Material Mgt  2010 02 27 0901Foreign Material Mgt  2010 02 27 0901
Foreign Material Mgt 2010 02 27 0901
William R. Corcoran, Ph.D., P.E.
 
Accountability 2010
Accountability 2010Accountability 2010

More from William R. Corcoran, Ph.D., P.E. (9)

Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017
Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017
Comparative TimeLine for LANL Arc Flash-Working Draft 9/14/2017
 
Hprct 2011 barrier analysis_upd 2017.08.01
Hprct 2011 barrier analysis_upd 2017.08.01Hprct 2011 barrier analysis_upd 2017.08.01
Hprct 2011 barrier analysis_upd 2017.08.01
 
Treatment of concerned nrc employee 2016 sept-27 transcript
Treatment of concerned nrc employee 2016 sept-27 transcriptTreatment of concerned nrc employee 2016 sept-27 transcript
Treatment of concerned nrc employee 2016 sept-27 transcript
 
Gas Line Near Indian Point Nuclear Power Plant
Gas Line Near Indian Point Nuclear Power PlantGas Line Near Indian Point Nuclear Power Plant
Gas Line Near Indian Point Nuclear Power Plant
 
Five types of factor building blocks with case study 2014.05.05.0730
Five types of factor building blocks with case study 2014.05.05.0730Five types of factor building blocks with case study 2014.05.05.0730
Five types of factor building blocks with case study 2014.05.05.0730
 
E atomizer explosion 2013.10.26
E atomizer explosion 2013.10.26E atomizer explosion 2013.10.26
E atomizer explosion 2013.10.26
 
A Foundation For Root Cause Analysis 2011.05.25
A Foundation For Root Cause Analysis 2011.05.25A Foundation For Root Cause Analysis 2011.05.25
A Foundation For Root Cause Analysis 2011.05.25
 
Foreign Material Mgt 2010 02 27 0901
Foreign Material Mgt  2010 02 27 0901Foreign Material Mgt  2010 02 27 0901
Foreign Material Mgt 2010 02 27 0901
 
Accountability 2010
Accountability 2010Accountability 2010
Accountability 2010
 

Recently uploaded

Open Channel Flow: fluid flow with a free surface
Open Channel Flow: fluid flow with a free surfaceOpen Channel Flow: fluid flow with a free surface
Open Channel Flow: fluid flow with a free surface
Indrajeet sahu
 
Northrop Grumman - Aerospace Structures Overvi.pdf
Northrop Grumman - Aerospace Structures Overvi.pdfNorthrop Grumman - Aerospace Structures Overvi.pdf
Northrop Grumman - Aerospace Structures Overvi.pdf
takipo7507
 
SELENIUM CONF -PALLAVI SHARMA - 2024.pdf
SELENIUM CONF -PALLAVI SHARMA - 2024.pdfSELENIUM CONF -PALLAVI SHARMA - 2024.pdf
SELENIUM CONF -PALLAVI SHARMA - 2024.pdf
Pallavi Sharma
 
Supermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdfSupermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdf
Kamal Acharya
 
Applications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdfApplications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdf
Atif Razi
 
A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...
A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...
A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...
DharmaBanothu
 
一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理
一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理
一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理
sydezfe
 
DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...
DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...
DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...
OKORIE1
 
Lateral load-resisting systems in buildings.pptx
Lateral load-resisting systems in buildings.pptxLateral load-resisting systems in buildings.pptx
Lateral load-resisting systems in buildings.pptx
DebendraDevKhanal1
 
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call GirlCall Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
sapna sharmap11
 
Determination of Equivalent Circuit parameters and performance characteristic...
Determination of Equivalent Circuit parameters and performance characteristic...Determination of Equivalent Circuit parameters and performance characteristic...
Determination of Equivalent Circuit parameters and performance characteristic...
pvpriya2
 
Accident detection system project report.pdf
Accident detection system project report.pdfAccident detection system project report.pdf
Accident detection system project report.pdf
Kamal Acharya
 
Blood finder application project report (1).pdf
Blood finder application project report (1).pdfBlood finder application project report (1).pdf
Blood finder application project report (1).pdf
Kamal Acharya
 
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICSUNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
vmspraneeth
 
An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...
An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...
An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...
DharmaBanothu
 
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptxSENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
b0754201
 
Object Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOADObject Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOAD
PreethaV16
 
OOPS_Lab_Manual - programs using C++ programming language
OOPS_Lab_Manual - programs using C++ programming languageOOPS_Lab_Manual - programs using C++ programming language
OOPS_Lab_Manual - programs using C++ programming language
PreethaV16
 
Introduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.pptIntroduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.ppt
Dwarkadas J Sanghvi College of Engineering
 
Advancements in Automobile Engineering for Sustainable Development.pdf
Advancements in Automobile Engineering for Sustainable Development.pdfAdvancements in Automobile Engineering for Sustainable Development.pdf
Advancements in Automobile Engineering for Sustainable Development.pdf
JaveedKhan59
 

Recently uploaded (20)

Open Channel Flow: fluid flow with a free surface
Open Channel Flow: fluid flow with a free surfaceOpen Channel Flow: fluid flow with a free surface
Open Channel Flow: fluid flow with a free surface
 
Northrop Grumman - Aerospace Structures Overvi.pdf
Northrop Grumman - Aerospace Structures Overvi.pdfNorthrop Grumman - Aerospace Structures Overvi.pdf
Northrop Grumman - Aerospace Structures Overvi.pdf
 
SELENIUM CONF -PALLAVI SHARMA - 2024.pdf
SELENIUM CONF -PALLAVI SHARMA - 2024.pdfSELENIUM CONF -PALLAVI SHARMA - 2024.pdf
SELENIUM CONF -PALLAVI SHARMA - 2024.pdf
 
Supermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdfSupermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdf
 
Applications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdfApplications of artificial Intelligence in Mechanical Engineering.pdf
Applications of artificial Intelligence in Mechanical Engineering.pdf
 
A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...
A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...
A high-Speed Communication System is based on the Design of a Bi-NoC Router, ...
 
一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理
一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理
一比一原版(uoft毕业证书)加拿大多伦多大学毕业证如何办理
 
DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...
DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...
DESIGN AND MANUFACTURE OF CEILING BOARD USING SAWDUST AND WASTE CARTON MATERI...
 
Lateral load-resisting systems in buildings.pptx
Lateral load-resisting systems in buildings.pptxLateral load-resisting systems in buildings.pptx
Lateral load-resisting systems in buildings.pptx
 
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call GirlCall Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
Call Girls Goa (india) ☎️ +91-7426014248 Goa Call Girl
 
Determination of Equivalent Circuit parameters and performance characteristic...
Determination of Equivalent Circuit parameters and performance characteristic...Determination of Equivalent Circuit parameters and performance characteristic...
Determination of Equivalent Circuit parameters and performance characteristic...
 
Accident detection system project report.pdf
Accident detection system project report.pdfAccident detection system project report.pdf
Accident detection system project report.pdf
 
Blood finder application project report (1).pdf
Blood finder application project report (1).pdfBlood finder application project report (1).pdf
Blood finder application project report (1).pdf
 
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICSUNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
 
An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...
An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...
An In-Depth Exploration of Natural Language Processing: Evolution, Applicatio...
 
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptxSENTIMENT ANALYSIS ON PPT AND Project template_.pptx
SENTIMENT ANALYSIS ON PPT AND Project template_.pptx
 
Object Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOADObject Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOAD
 
OOPS_Lab_Manual - programs using C++ programming language
OOPS_Lab_Manual - programs using C++ programming languageOOPS_Lab_Manual - programs using C++ programming language
OOPS_Lab_Manual - programs using C++ programming language
 
Introduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.pptIntroduction to Computer Networks & OSI MODEL.ppt
Introduction to Computer Networks & OSI MODEL.ppt
 
Advancements in Automobile Engineering for Sustainable Development.pdf
Advancements in Automobile Engineering for Sustainable Development.pdfAdvancements in Automobile Engineering for Sustainable Development.pdf
Advancements in Automobile Engineering for Sustainable Development.pdf
 

Doe hanford hoist drop from monorail

  • 1. Root Cause Analysis Report Version as of May 6, 2004 Hoist Drop from Monorail 21 at K West Spent Fuel Pool March 10, 2004 Sponsor: D. M. Busche Root Cause Analysis Team W. J. Leonard, Team Leader W. R. Corcoran, Senior Advisor J. M. Lukes M. E. Poole
  • 2. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 2 of 117 Table of Contents Executive Summary ........................................................................................................ 4 Glossary.......................................................................................................................... 5 Narrative.......................................................................................................................... 7 Eight Question (Phoenix) Analysis ................................................................................ 15 Missed Opportunity Matrix............................................................................................. 23 Influence-Consequence Matrices.................................................................................. 29 Hoist Drop .................................................................................................................. 30 Non-Notification.......................................................................................................... 31 Barrier Analysis Matrix .................................................................................................. 32 Discussion of Design Vulnerabilities.............................................................................. 38 Discussion of Process Vulnerabilities............................................................................ 38 Procedures................................................................................................................. 38 Training ...................................................................................................................... 39 Safety Vulnerability Audits.......................................................................................... 39 Discussion of Process Strength .................................................................................... 39 Discussion of Influences................................................................................................ 40 Corrective Action Recommendations ............................................................................ 42 Analysis of Expected Effectiveness of Recommended Corrective Action ..................... 47 Conclusions................................................................................................................... 47 Attachments .................................................................................................................. 48 Comparative TimeLine ©......................................................................................... 49 Why Staircase Trees............................................................................................... 73 Extent of Condition and Cause Matrix..................................................................... 83 Cause Codes .......................................................................................................... 92 Documents Reviewed ........................................................................................... 103 Personnel Contacted............................................................................................. 106 Analysts Comments .............................................................................................. 108 Structural Analysis Report..................................................................................... 111 Calculations........................................................................................................... 113 Unanswered Questions......................................................................................... 117
  • 3. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 3 of 117 Table of Figures Figure 1 - Fuel Storage Monorail System........................................................................ 8 Figure 2 - Locking Arm & Weighted Safety Arm.............................................................. 9 Figure 3 - Flexible Transfer Crane................................................................................... 9 Figure 4 - Hoist Trolley Stopped Against Weighted Safety Arm.................................... 10 Figure 5 - Weighted Safety Arms In “UP” Position ........................................................ 11 Figure 6 - Hoist/Trolley Assembly Lying on Grating ...................................................... 11 Figure 7 - WSA Clearance ............................................................................................ 12
  • 4. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 4 of 117 Executive Summary On March 10, 2004 at about 11:17 P.M. a monorail hoist fell from its monorail in the K- West Basin Spent Fuel Pool and came to rest on a grating over the pool. There were no personnel injuries, no radiological releases, and no significant equipment damage. It was, however, a near miss because if workers had been in the fall trajectory of the hoist they could have been injured, perhaps severely. The Facility Manager and the Multi-Canister Overpack (MCO) Production Manager were promptly notified, but the Department of Energy Facility Representative (DOE-FR) was not promptly notified. This episode received press coverage and was the subject of a letter from the DOE Richland Operations Office (RL) Manager to the Fluor Hanford (FH) President. The hoist fell because it was moved by a Radiological Control Technician (RCT) to facilitate his regular duties and because the monorail hoist stop (Weighted Safety Arm) that should have stopped the hoist was stuck in the up position. The causes of the mispositioned Weighted Safety Arm are not fully known, but are believed to include distortion of the Weighted Safety Arm or shaft by some kind of impact combined with the failure to perform checks that would have identified the improper Weighted Safety Arm conditions. The prompt notification to the DOE-FR was not done because the personnel involved incorrectly applied procedural requirements and failed to recognize the “near miss” nature of the event shortly after it occurred. In the opinion of the team, the letter from RL would not have been written had prompt notification of the event occurred. Both RL and FH management have shown keen interest in this event and its investigation. Recommended corrective actions for the primary event include enhanced preventative maintenance and enhanced operating requirements. Recommended corrective actions for the reporting irregularity include clarification of the procedure and re-emphasis of managements’ expectation for strict adherence to the contractual and procedural requirement to notify the DOE-FR of any adverse events or conditions that are worthy of being reported to the FH Facility Manager.
  • 5. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 5 of 117 Glossary Item Description / Meaning Contracts Requirements Document (CRD) Contractual requirement with which Fluor Hanford must comply. Specifically, the requirement that the DOE Facility Representative must be promptly notified of non-reportable bad news when the Facility Manager is notified. (CRD M 231.1-2 (Rev. 1), “Occurrence Reporting and Processing of Operations Information”.) Crane Inspection Team (CIT) Team formed by Spent Nuclear Fuel Project to evaluate crane issues and make recommendations. Also called the Crane Committee. (Letter 03-SNF-JKM-024, dated June 9, 2003, “Establishment of the Spent Nuclear Fuel Crane Inspection Team”) Fuel Storage Monorail System (FSMS) Extensive system over the K Basins for transporting spent nuclear fuel stored under water. Components of the FSMS are Hoists, WSAs, and FTCs. (See Figure 1) Flexible Transfer Crane (FTC) Double rail trolley circling the perimeter of the fuel storage pool. Used for transporting Spent Nuclear Fuel under water from one parallel monorail to another. (See Figure 3) Weighted Safety Arm (WSA) Safety Device for stopping errant hoist trolleys. Also called Receiver Latch, Latch, and Rail Stop and Stop. WSAs are mounted in parallel on a pivot shaft and hang from the top of the monorail. When the FTC is latched with a monorail, the Locking Arms automatically align the FTC and the WSAs rotate up so a hoist can be rolled onto or off of the FTC. (See Figures 2, 4, and 5) Locking Arm An extension of the WSA to align and secure an FTC to the monorail. The Locking Arm is brazed to the top of the WSA. If an FTC is not fully stopped before it is engaged with the monorail, the Locking Arm may be bent outward from impact. This results in inward bending of the WSA which can then become stuck in the up position against the monorail. (See Figures 2 and 7) Non-Reportable Event An event or condition that does not require an occurrence report in accordance with HNF-PRO-060, “Occurrence Reporting and Processing of Operations Information”. RL has repeatedly indicated that it wants to be notified of non-reportable events and conditions that are the subjects of notifications to the Facility Manager.
  • 6. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 6 of 117 Item Description / Meaning Stand-Down Described in HNF-PRO-060 Appendix C “Occurrence Reporting Categories and Criteria”, Group 4 “Facility Status”, Subgroup B “Operations” (7) SC-4: “A facility or site stand-down resulting from safety reasons reportable as an occurrence or occurrences. NOTE: This is a secondary reporting criterion, and does not require a separate occurrence report.” Compare with the next higher level criterion, (6) SC-4*, ”A facility or operations shutdown (i.e., a change of operational mode or curtailment of work or process) directed by management for safety reasons.” The latter is “reportable” (requires an occurrence report). Production Pause A work stoppage not considered a stand-down, not defined in HNF-PRO-060.
  • 7. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 7 of 117 Narrative This event is the conjunction of two parallel developments, one having to do with notifications and the other having to do with equipment. The ultimate consequence was a letter of concern from DOE-RL to Fluor Hanford regarding delayed notification of events and other issues. Over the last few years, RL has become increasingly dissatisfied with the promptness of notification of low-level “non-reportable” events that are sufficiently significant to require informing facility management. This dissatisfaction eventually manifested itself by RL imposing a change to the contract with FH. The Contract Requirements Document (CRD) was implemented by FH in a change to its reporting process procedure, HNF- PRO-060, “Reporting Occurrences and Processing Operations Information”. The need for the change to HNF-PRO-060 was identified after the procedure was through the review cycle. The need for the change was urgent and the time pressure was sufficient that the CRD related addition was not sent back through the review cycle. Thus, the reviewers were not afforded the opportunity to identify certain human factors issues that would later turn out to be important. The human factors issues involved are: 1) The use of the exact words from the CRD, and 2) The location of the change in a portion of the procedure called “Reportable Occurrences”. The wordy descriptions and location of the requirement in HNF-PRO-060 made it highly likely that a person not specifically trained in the intent of the procedure change would not perceive the basic simplicity of it. The basic simplicity of the intent is to have the Facility Representative (FR) promptly notified of all non-reportable bad news that is reported to the Facility Manager. The FH Emergency Preparedness (EP) department had the responsibility to make and implement the changes to HNF-PRO-060. There was no process to ensure that all personnel who needed to change their notification behaviors would be trained to the new requirements. In addition, there was no process to check that the new behaviors would, in fact, take place. Thus, the Shift Operations Manager (SOM) who was on duty when the hoist dropped had not been trained to the new requirements. There was, however, a presentation by the EP Director to the Facility Managers Forum covering the new requirements. Unfortunately, the person who was the K West Basin Facility Manager (FM) on the night of the event was not in attendance because he was not the K West Basin FM at the time of this presentation. In addition, the EP Director did not cause checks to be made to ensure compliance with the new contract requirements as listed in HNF-PRO-060. Meanwhile, FH personnel had difficulties in recognizing and labeling “near miss” (personnel injury) situations. Personnel are reluctant to declare near misses. This reluctance is manifested in prolonged discussions as to whether or not a particular event was a near miss.
  • 8. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 8 of 117 The K West Basin Fuel Storage Monorail System dates from the mid-1950s (Figure 1). Recently the equipment has been used more frequently to support facility closure. The use and misuse of this equipment has resulted in material deficiencies including tilted monorails and distorted components. Figure 1 - Fuel Storage Monorail System The monorail Latch Assembly is made up of two key parts, the Locking Arm (Figure 2) that engages a Flexible Transfer Crane (FTC) to the monorail (Figure 3), and the Weighted Safety Arm (Figure 2) that acts as a rail stop when the FTC is not coupled. “Normal” use of FTCs has included engaging the locking mechanism to the monorails while the FTCs are still moving. This action causes the monorail’s Latch Assembly Locking Arm to be impacted, likely bending the Locking Arm such that the Weighted Safety Arm is bent in the opposite direction. This results in the WSAs contacting and sticking in the up position on the monorail upper flange
  • 9. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 9 of 117 Figure 2 - Locking Arms & Weighted Safety Arms Figure 3 - Flexible Transfer Crane (Photo taken in passageway at south end of Monorail 21, about where the hoist landed, looking east) The Weighted Safety Arms (WSAs) are engineered safety devices for preventing a hoist/trolley assembly from falling off the end of a monorail (Figure 4). “Normal” use of the hoist trolleys has included allowing the hoists to collide with the WSAs. The Root Cause Analysis (RCA) Team considers that this is a use of a safety device for operational purposes. There are two disadvantages to this practice. First, the collisions Locking Arms Weighted Safety Arms
  • 10. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 10 of 117 could damage the hoist trolley assembly, the WSAs, or the WSA shaft. (Note: For the rest of this report, when we refer to bending to the WSA, please understand that the same effects could have been caused by a bent shaft.) Second, if, for some reason, the WSA is missing or stuck in the up position, the hoist trolley would not collide, but rather, continue off the end of the monorail and fall in an area where personnel could be. Figure 4 - Hoist Trolley Stopped Against Weighted Safety Arm The annual Preventive Maintenance (PM) of the K West Basin monorail system was completed in February, 2004, and did not result in the recording of any safety discrepancies. Unfortunately, the records are not sufficient to provide adequate detail. It is not clear whether discrepant conditions existed and actions were taken to correct those conditions or whether no discrepant conditions existed. A recording of “No Discrepancies” was made on the work record of the PM. A variety of job classifications are needed in the K West Basin. Some, such as Radiological Control Technicians (RCTs), do not receive hoisting and rigging training. Many people, nevertheless, move hoists in order to do their assigned jobs unless they are told to stop and get a qualified person. On the night of March 10th , 2004, an RCT moved a hoist on Monorail 21. Monorail 21 was tilted down toward the south. The RCT did not do any pre-use checks for this seemingly trivial evolution. The RCT elected to push the hoist to the south (in the direction of downward tilt). For unknown reasons, the WSA at the south end of Monorail 21 was stuck in the up position and was therefore unavailable to stop the traveling hoist (Figure 5).
  • 11. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 11 of 117 Figure 5 - Weighted Safety Arms Stuck In “UP” Position The hoist fell off the end of the monorail and landed on a grating in a spot that is sometimes occupied by workers. Fortunately, at the time of the impact, there were no workers and no equipment in the trajectory of the hoist (Figure 6). Figure 6 - Hoist/Trolley Assembly Lying on Grating The hoist casing was damaged. No other physical consequences resulted. The personnel in the area, the RCT and four Nuclear Chemical Operators (NCOs), heard the sound of the impact and verified the source. They stopped work and informed the SOM.
  • 12. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 12 of 117 The SOM sent an Operating Engineer (OE) and an NCO to investigate the fallen hoist. They found that the WSA at the south end of Monorail 21 was inexplicably in the down position. They also found that when the WSA was pushed up, could become stuck in the up position. The WSA was left in the up position for further evaluation on dayshift. The OE cordoned off the area. The hoist drop into the area sometimes occupied by people was not considered to be a near miss personnel injury event. Figure 7 - WSA Clearance The scrap processing crew received permission from the SOM to re-enter the basin to complete process work. The SOM reviewed HNF-PRO-060 for reportability, but found no guidance that prompted him to notify the FR or to advise the FM that the FR should be notified. The SOM notified the FM about the event and the near miss aspect of the event was discussed (determined not to be a near miss), but notification to the FR was not. The SOM then notified the Multi-Canister Overpack (MCO) Production Manager. Again, neither notification to the FR nor the near miss was discussed. The next morning, March 11, 2004, at the K West Basin Plan of the Day management meeting, the hoist fall was discussed and determination was made for the Crane Inspection Team to investigate. The Crane Inspection Team investigated K West and found 9 damaged WSAs. Action was taken to correct the identified discrepancies. There was no overt evidence that the Crane Inspection Team (CIT) realized that their activities were affecting root cause analysis evidence. This could have been particularly significant since the possibility of sabotage would later be considered and the WSAs were potentially criminal evidence. The CIT did take photographs of some of the damaged WSAs. They did not, however, record “as found” dimensions of the distorted WSAs prior to making adjustments (straightening). Normal Clearance: described by Structural Engineer as “almost zero.” Wear caused by engagement with moving FTC.
  • 13. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 13 of 117 Meanwhile, the FR called the FM and discussed a number of production issues. During this conversation, the FM mentioned that the hoist fell off the monorail. Later that morning, following the CIT investigation of the stuck WSA, the Fuel Storage Monorail System (FSMS) Design Authority and the Engineering Manager met with the SNF Project Deputy Director. No certain cause for the WSAs to become bent was determined. Additionally, no certain cause for the WSA having been moved up was determined. However, one of the people who had been involved in the investigation disclosed to the Deputy Director that the WSAs may have been damaged intentionally. Following this meeting the CIT inspected the K East Basin and confirmed (based on a list provided by operators) that several WSAs were capable of sticking in the up position. The Crane Inspection Team straightened the bent WSAs, breaking one in the attempt. The Deputy Director notified the SNF Vice President of the event, stating that the CIT thought the WSAs were bent on purpose. The SNF Vice President then notified the FH Vice President. Subsequently, the FH President and the DOE-RL Manager were notified. The RL Manager called the Facility Representative Manager, who notified the K West Basin FR. The FR had not been aware that the hoist fall was a result of a bent WSA stuck in the up position or that it was a suspected act of sabotage. At about 2:00 in the afternoon, the FH Executive Vice President directed the implementation of a “Production Pause”. Also, the SNF Vice President notified site security and security notified local law enforcement. The only evidence obtained by the RCA Team was a seven line hand written, undated personal statement. Some interviews were conducted prior to the critique meeting; however, there does not appear to be any formal record of these interviews. A Critique Meeting was scheduled for 6:30 that evening. The event and corrective actions were discussed, but no probable causes were determined. Security and law enforcement personnel were present at the meeting. The hoist fall event was determined reportable as a Near Miss Occurrence following the Critique Meeting. The next day, March 12, 2004 the Tri-City Herald printed an article titled “Hanford Officials Investigate K Basin Accident”. The newspaper article stated that work was stopped throughout Hanford’s K Basins after an accident lead to an investigation that found mechanical problems at both the K East and K West Basins. Discussion of possible sabotage was mentioned, however, the article went on to state that the evidence was inconclusive. The same day, the DOE-RL Manager sent the letter of concern to the FH President stating that RL was “extremely concerned about the degradation of hoisting safety equipment and subsequent hoist drop off the monorail to the basin grating, which was initially identified as a purposeful act”. They went on to state that they were “particularly dissatisfied with the response actions from the time the hoist fell on Wednesday night, March 10, 2004, until RL was notified of issues on Thursday afternoon, March 11, 2004”.
  • 14. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 14 of 117 On March 15, 2004 at about 8:00 in the morning, the SNF Deputy Director informed the FH Executive Vice President that corrective actions had been completed. The Production Pause was terminated based on a restart plan that had been approved by the Executive Vice President. On March 18, 2004 the FH VP of Regulatory Compliance appointed an “Independent Review of Recent Near Miss at SNF”, headed by the Manager of Assessments. The review team was convened to gather information about the event for a response to the RL letter of concern. No formal report was issued from this review. On April 2, 2004, The Tri-City Herald printed an article titled “K Basins accident concerns DOE”. The newspaper article described the RL letter and the FH response to the letter. The article indicated that FH now believes that the problem was not caused by worker sabotage, but was caused by wear and tear on the equipment at the end of its life cycle. On April 5, 2004, a Root Cause Analysis Team was chartered to investigate two primary issues of concern: 1) The near miss event involving the falling hoist, and 2) The delayed notification to RL personnel. The items to be included were: • Determine the facts associated with the event; • Analyze the causes for the event; • Identify the potential for similar events to occur at the SNF Project and rest of the site, and; • Provide recommendations for corrective action to the SNF Project and the rest of the site.
  • 15. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 15 of 117 Eight Question (Phoenix) Analysis This analysis provides summary answers to the Eight Questions for Insight and is a brief high-level summary of the entire episode. Eight Question (Phoenix) Analysis of Hoist Drop Impact 1. Consequences (Tangible and intangible adverse effects) o Actual: 1) Adverse letter from RL to FH. 2) Adverse press coverage. 3) Costs of a Production Pause of several days. 4) On-site investigation Costs. 5) Costs incurred in interfacing with RL over this incident. 6) Reduced employee moral due to accusations of wrongdoing. o Expected: 1) Increased RL scrutiny of FH activities and associated costs. 2) Probable withholding of fees to FH. 3) Costs of repairing/replacing distorted Weighted Safety Arms. o Potential Consequences: 1) Severe personnel injuries and an extensive production pause or shut down. 2. Significance (What does this mean for the future of the facility?) o Precursor to: 1) More serious personnel, economic or customer satisfaction consequences. (The behaviors and conditions involved in this one would, if not corrected, be capable of causing a much more serious event.) 2) Potential breakdown of the Occurrence Reporting element of the Safety Management Program. o Temporal and Spatial Extent: 1) Monorail problems: Several cases of distorted Weighted Safety Arms were found in both K West and in K East.
  • 16. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 16 of 117 Eight Question (Phoenix) Analysis of Hoist Drop Impact (continued) 2) Notifications: There have been other notification shortfalls in the past. These prompted changes in contract requirements, including requirements that were specifically not met in this instance. 3) Previous failures to declare near misses. 4) Other safety device precursors. 5) Other departures from ISMS. 6) Other difficult administrative procedures. 7) Other problems caused by not using SAT. o Barriers that did not succeed: See items 3, 4, and 5 below. o Remaining barriers: 1) Good fortune (There were no personnel in the fall trajectory of the hoist). o Campaign issues: 1) Personnel safety hazards 2) Notifications. 3) Near Miss calls. 4) Promptness and effectiveness of Critique Meetings.
  • 17. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 17 of 117 Eight Question (Phoenix) Analysis of Hoist Drop Influences on Consequences 3. Vulnerability (What set us up for this event?) o Elevated monorail. o Hoist rides on monorail. o Hoist pendant, chain and chain bucket hang down obstructing activities below. o Monorail 21 is tilted toward the south end. o Monorail 21 and hoist have low friction. o Weighted Safety Arms prevent hoist dropping off end of monorail when FTC not present. o Single failure vulnerability. o Weighted Safety Arm bent slightly causing interference with Monorail 21: 1) Probably caused by repeated collisions of the hoist/trolley assembly with the WSA and, 2) The impact of the FTC when coupling with the monorail. o WSA in the up position potentially caused by long tools striking the WSA. o Adverse impact of tool strikes not recognized. o RCT not trained on hoist vulnerability to dropping off end of monorail. o RCT not trained to avoid relying on safety devices, e.g., WSAs. o Practice of relying on Weighted Safety Arms to stop hoist. o Practice of moving hoists without safety checkout. o Practice of moving FTCs while coupling to monorails. o ISMS not applied to routine RCT tasks. o Collisions between hoist and Weighted Safety Arms not considered as precursors. o Collisions between FTCs and Locking Arms not considered as precursors. o PMs don’t find Weighted Safety Arm problems.
  • 18. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 18 of 117 Eight Question (Phoenix) Analysis of Hoist Drop Influences on Consequences (continued) o Previous history of notification shortfalls. o Previous history of “near miss” declaration shortfalls. o Non-User-Friendly guidance on notification. 4. Trigger (What put the ball in motion?) o Radiological Control Technician moved hoist. 5. Exacerbation (What made the consequences as bad as they were?) o RCT did not keep positive control of the hoist trolley. o Monorail elevation. o Failure to notify Facility Representative. o Previous FH performance issues (notification deficiencies). o Lack of questioning attitudes of managers. o Imprudent references to “intentional act” and “sabotage”. 6. Mitigation (What kept the consequences from being a lot worse?) o No personnel in fall trajectory. o Prompt FH response to RL letter.
  • 19. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 19 of 117 Eight Question (Phoenix) Analysis of Hoist Drop Close Out 7. Lessons to be Learned (What skills, rules, and knowledge should be added or reinforced?) o Notifying Facility Representative when Facility Manager is notified of non-reportable “bad news” is not optional. o Routinely using and relying on Weighted Safety Arms to stop moving hoists is a bad practice. o Routinely coupling moving FTCs with monorails is a bad practice. o Using safety devices as operational controls is a bad idea. o Interviews and interview-assisted personal statements should be obtained before the involved personnel leave the site at shift end. o Inflammatory words, such as “sabotage”, should be used prudently and only when necessary. o Professionals hate to find out bad news about their areas of responsibility from their superiors (e. g., SNF DOE-FR notified by DOE-RL FR Manager). o Criminal investigations and Root Cause investigations have different objectives and should not be combined. o Every event should be scrutinized to determine the extent to which it was a near miss to something much more serious. o Managers who receive bad news reports should exhibit a questioning attitude toward the information received. In particular, the caller should be queried about potential circumstances that would make the bad news more significant. 8. Corrective Actions (What conditions and behaviors should be changed?) o Interim compensatory measures: 1) Implemented a Production Pause (The Production Pause was lifted following completion of actions 2, 3, and 4 below). 2) The Crane Inspection Team completed an evaluation of the monorail system and adjusted other WSAs into their proper position and verified proper operation. 3) Implemented a Long Term Order requiring a shiftly check to verify WSAs are in the down position; to verify WSAs are in the down position while performing operations; and, to prohibit the hanging of tools off of the WSAs.
  • 20. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 20 of 117 Eight Question (Phoenix) Analysis of Hoist Drop Close Out (continued) 4) Conducted management briefings on the event to SNFP employees. 5) FH VP of Regulatory Compliance issued an e-mail to all Project Vice Presidents to disseminate to their Facility Managers the expectations for compliance with the requirement to notify the FR whenever the FM is notified of “non-reportable” bad news. 6) Provide stop-gap training on near miss identification and FR notification expectations for all personnel involved in the notification process. o Corrective actions for symptoms and effects: (Complete) 1) “Straightened” the Weighted Safety Arm on Monorail #21 and verified proper operation. 2) Replaced the cracked casing on the Hoist HOI-440 and verified proper operation. o Corrective actions for causes: 1) Change monorail hoist system operating procedures to require that persons moving hoists must maintain positive control of the hoist, and must not cause collisions between the hoist and the WSAs. 2) Change monorail hoist system operating procedures to require that before beginning evolutions involving hoists, all monorail system safety features must be checked. 3) Change the FTC operating procedures to require that persons moving FTCs do not cause collisions with the monorail system (Locking Arms). 4) Rewrite HNF-PRO-060 to meet appropriate procedure standards, including human factors. 5) Revise HNF-PRO-060 to require that every event be suspected to be a near miss for something more serious (e.g., would people normally work in an area that was in the line of fire?)
  • 21. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 21 of 117 Eight Question (Phoenix) Analysis of Hoist Drop Close Out (continued) 6) Revise training policy to require all training for new requirements meets the intent of the SAT. o Corrective actions for generic implications: 1) Examine other PM procedures for weaknesses similar to the ones found in the monorail system procedures. 2) Check all PM procedures to assure that they require returning the system to a reference condition after the evolution. 3) Share this report with other facilities. o Corrective actions for self-assessment deficiencies: 1) Change the text of the monorail system PM instructions to clearly specify what needs to be checked and what the acceptance criteria are. 2) Change the text of the monorail system PM instructions to clearly specify the safety significance of the items being checked. o Disposition of Extraneous Conditions Adverse to Quality: 1) Management and the Bargaining Unit should agree on the union’s role in investigations so that union leadership does not obstruct. 2) Interviews of involved personnel should be conducted before the end of the shift. 3) Experienced interviewers should assist personnel in preparing personal statements. 4) Personal statements should be dated and signed, and completed before the end of the shift. 5) Personnel who have not been trained in evidence preservation should be barred from the scene of an accident. 6) Any physical work on evidence should be done under the work control process and the work control instructions should include measures to prevent spoliation.
  • 22. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 22 of 117 Eight Question (Phoenix) Analysis of Hoist Drop Close Out (continued) 7) The basis for both imposition and lifting of work interruptions (Production Pause) should be documented. 8) Private deals to abrogate contractual requirements must be discouraged (prompt notifications from FMs and FRs). 9) The adherence to the intent of ISMS and SAT should be transparent in work instructions and other documents. One should be able to pick-up one of these documents and tell that ISMS and/or SAT has been applied. 10) Personnel in high hazard environments should not take action without ensuring that they posses the qualifications/training. It is not acceptable for a person to assume that the action they are about to take is trivial. 11) Criminal investigations should take precedence over organizational learning investigations (critiques). If law enforcement personnel are present, critiques should be suspended. 12) FH Senior Management needs to ensure that bad news that will be communicated to DOE-RL Senior Management is first reported to the project Facility Representative. 13) All investigative activities should result in written documentation. 14) Subsequent chartered investigation teams should be provided with all reports of previous investigative activities. 15) Unintended and unanalyzed impacts between equipment should be treated as abnormal events and should be stopped pending engineering approval (e.g., impacts of long tools against overhead components.) 16) The effectiveness of corrective actions should be checked some time after implementation. o Follow-up plans: 1) Audit this facility and all facilities with similar vulnerabilities to confirm that they accommodate the lessons to be learned and corrective actions of this incident.
  • 23. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 23 of 117 Missed Opportunity Matrix This Matrix is a summary of the most significant situations in which ordinary people could have prevented the event or made its consequences less severe by doing ordinary actions ordinarily well. The matrix is not intended to capture brilliant insights or out-of-the-box thinking. Missed Opportunity Matrix Who Situation Opportunity (action) Expected Result Impact on Consequences/Remarks EP Dissemination of CRD requirements Train all personnel whose behaviors needed to change, in accordance with SAT Personnel needing to change their previous behavior relating to prompt notification to FR would have done so All the personnel involved in notifying FRs of non-reportable bad news would have been trained to the CRD requirements. It is more likely that notifications would have been made. EP During or after dissemination of CRD requirements Trainees’ knowledge should have been checked both during and after the training. The behaviors of the target population would have been checked to ensure prompt notifications were being made and the training would have been modified if necessary All the personnel involved in notifying FRs of non-reportable bad news would have been properly trained to the CRD requirements. It is more likely that notifications would have been made. EP Dissemination of CRD requirements in PRO-060 Plan the training in order to accomplish a specified result Personnel needing to change their previous behavior relating to prompt notification to FR would have done so All the personnel involved in notifying FRs of non-reportable bad news would have been trained to the CRD requirements. It is more likely that notifications would have been made.
  • 24. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 24 of 117 Missed Opportunity Matrix Who Situation Opportunity (action) Expected Result Impact on Consequences/Remarks FM Dissemination of CRD requirements in PRO-060 Disseminate the information provided in the Facility Manager’s Forum related to prompt notification to FR when FM is notified of “bad news” Personnel needing to change their previous behavior relating to prompt notification to FR would have done so All the personnel involved in notifying FRs of non-reportable bad news would have been informed of the CRD requirements. It is more likely that notifications would have been made. EP Procedure Review Process Obtain peer review of PRO-060 after CRD requirements were incorporated Human factor issues would have been identified Personnel needing to change their previous behavior relating to prompt notification to FR would have done so Procedure would have been more user friendly It is more likely that notifications would have been made. EP Dissemination of CRD requirements in PRO-060 Provided supplemental information/continuing training to personnel who had been previously trained when CRD requirement was added (not all personnel received training to the new requirement). Personnel needing to change their previous behavior relating to prompt notification to FR would have done so All the personnel involved in notifying FRs of non-reportable bad news would have been informed of the CRD requirements. It is more likely that notifications would have been made. EP Procedure change process Verification and validation of administrative procedures Human factor issues would have been identified Procedure would have been more user friendly It is more likely that notifications would have been made.
  • 25. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 25 of 117 Missed Opportunity Matrix Who Situation Opportunity (action) Expected Result Impact on Consequences/Remarks FM Dissemination of PRO-060 for prompt notifications to FR Provided information from the Facility Manager’s Forum on prompt notifications to the FR to their project personnel. Personnel needing to change their previous behavior relating to prompt notification to FR would have done so All the personnel involved in notifying FRs of non-reportable bad news would have been informed of the CRD requirements. It is more likely that notifications would have been made. QA Operations Engineering Safety Performance of periodic audits, assessments, and JHAs for K West Basin Operations, including the use of hoists Audits, assessments, and JHAs should have noticed the single point failure vulnerability. The event is aborted by identifying the problem and establishing additional controls. No event, however, the prompt notification vulnerability remains in place. An example of a control would be to hang caution tags on the hoists when they are on slanted monorails to say: CAUTION: Keep positive control while moving hoist. Monorail is tilted and hoist will continue to travel on its own. QA Operations Engineering Safety Performance of periodic audits, assessments, and JHAs for K West Basin Operations, including the use of hoists Audits, assessments, and JHAs should have noticed the routine repeated use of the safety device to stop a hoist. The event is aborted by identifying the problem and establishing additional controls. No event
  • 26. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 26 of 117 Missed Opportunity Matrix Who Situation Opportunity (action) Expected Result Impact on Consequences/Remarks Design Authority Thinking about failure modes and safety vulnerabilities. Identify safety vulnerabilities related to the single point failure and the challenging of a safety device (Weighted Safety Arms). The event is aborted by identifying the problem and establishing additional controls. No event. Millwright Supervisor Preventative Maintenance Discrepancies (bent or sticking Weighted Safety Arms) should have been found. The discrepant condition of the Weighted Safety Arm would have been identified and corrected. No event Operators Thinking about impact in the overhead from use of long tools Recognize that long tools are impacting overhead hoist system equipment Areas would be inspected for damage and repairs initiated if necessary Potentially prevented event Comment: The long tool may have bent the Weighted Safety Arm, or caused it to stick in the up position. Evidence is inconclusive. Operators Routine movement of the hoist Pre-use hoisting and rigging checks should identify safety anomalies including the Weighted Safety Arm on Monorail 21 being in the up position. The pre-use checks would have aborted the event by identifying the Weighted Safety Arm in the up position. No event Operators Routine movement of the FTC Recognize that FTCs are impacting Locking Arms FTCs would be stopped before coupling with the Locking Arm on the monorail. Potentially prevented event Comment: The FTC impact may have bent the Weighted Safety Arm.
  • 27. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 27 of 117 Missed Opportunity Matrix Who Situation Opportunity (action) Expected Result Impact on Consequences/Remarks SOM Compensatory actions from hoist fall. Implement controls to ensure other hoists don’t fall off monorails before resuming normal operations. Controls would have been implemented. Mitigation of RL concerns for worker safety described in Letter of Concern to FH. SOM Found out that hoist fell off monorail and impacted an area where workers sometimes are present. SOM should have recognized that this was a near miss personnel injury. Reportability would have been clearly established and SOM would have made notifications to FM and FR The prompt notification issue would not have occurred. FM Found out that hoist fell off monorail. FM should have asked questions to reveal that this was a near miss personnel injury. Reportability would have been clearly established and SOM would have made notifications to FR The prompt notification issue would not have occurred. MCO Production Manager Found out that hoist fell off monorail. MCO Production Manager should have asked questions to reveal that this was a near miss personnel injury. Reportability would have been clearly established and SOM would have made notifications to FR The prompt notification issue would not have occurred. K West Management Chain (SOM & FM) Notification process FR should have been notified in accordance with SNF Notification Guidelines and HNF- PRO-060 for events meeting the threshold for notification of the FM. The FM or SOM would have promptly notified the FR. The prompt notification issue would not have occurred.
  • 28. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 28 of 117 Missed Opportunity Matrix Who Situation Opportunity (action) Expected Result Impact on Consequences/Remarks K West Management Chain (SOM & FM) Notification process FM and SOM should have been aware of RL concern for receiving prompt notification. The FM or SOM would have promptly notified the FR. The prompt notification issue would not have occurred. VP S&H Escalating customer dissatisfaction with prompt notification of non-reportable bad news Find out about escalating dissatisfaction with prompt notification of non-reportable bad news Assure that EP understood the purpose of the changes to PRO-060 to accommodate CRD requirements Change would have been better, training would have been better and the notification issue would not have happened Collective Feedback Community Escalating customer dissatisfaction with prompt notification of non-reportable bad news Take instances of customer dissatisfaction with prompt notifications, record them, analyze them, implement corrective actions, and disseminate lessons to be learned Notifications would have been done as required by the CRD The prompt notification issue would not have occurred
  • 29. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 29 of 117 Influence-Consequence Matrices Explanation of Influence-Consequence Matrices See one of the two matrices on the next two pages. These matrices display important influences on the consequences and how the removal of a single influence would have impacted the consequences. In the Influence-Consequence Matrix for the Hoist Drop, for example, the upper part of the left hand column lists important influences on the consequences of the hoist drop. The lower part of the left hand column lists the types of consequences. The next column indicates that in the actual case all of the influences were present and the actual consequences resulted. The remaining columns relate to hypothetical cases in which, respectively, certain influences are assumed to have been negated. At the bottom of each of these hypothetical case columns one sees the consequences that would have resulted from the hypothetical case. One sees readily from that matrix that there were seven individual influences (causes) that, if it had not been there the hoist would not have dropped.
  • 30. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 30 of 117 Hoist Drop Influences Actual Case Case 1 WSA straight Case 2 WSA down Case 3 MR level Case 4 RCT pushes hoist N Case 5 RCT holds onto hoist Case 6 Pre- use check of WSA Case 7 PM ID’s bent WSA Case 8 Person under end of MR WSA is bent Yes No Yes Yes Yes Yes Yes Yes Yes WSA is up Yes No, WSA would drop No Yes Yes Yes Yes Yes Yes M 21 is tilted to S Yes Yes Yes No Yes Yes Yes Yes Yes RCT pushes hoist to S Yes Yes Yes Yes No Yes Yes Yes Yes RCT does not maintain +Ctrl Yes Yes Yes Yes Yes No Yes Yes Yes No pre-use check of WSA Yes Yes Yes Yes Yes Yes No Yes Yes PM does not ID bent WSA Yes Yes Yes Yes Yes Yes Yes No Yes No person under end of MR Yes Yes Yes Yes Yes Yes Yes Yes No Consequences Hoist Dropped, casing broke None None Probably None None None None None Dropped, casing not broken Personnel None None None None None None None None Injury or death
  • 31. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 31 of 117 Non-Notification Influences Actual Case Case 1 Previous notification success Case 2 No CRD Case 3 Mgmt chooses effective wording Case 4 PRO-060 is human factored Case 5 PRO-060 training per SAT Case 6 Realistic “near miss” sensitivity Case 7 Questioning attitude by FM Case 8 Questioning attitude by MCO Mgr RL unhappy w/ notifications Yes No Yes Yes Yes Yes Yes Yes Yes CRD on notifications Yes Yes No Yes Yes Yes Yes Yes Yes PRO-060 uses CRD wording Yes Yes Yes No Yes Yes Yes Yes Yes PRO-060 LTA human factored Yes Yes Yes Yes No Yes Yes Yes Yes Ineffective PRO- 060 training Yes Yes Yes Yes Yes No Yes Yes Yes Inadequate “near miss” sensitivity Yes Yes Yes Yes Yes Yes No Yes Yes Inadequate challenge by FM Yes Yes Yes Yes Yes Yes Yes No Yes Inadequate challenge by MCO Mgr Yes Yes Yes Yes Yes Yes Yes Yes No Consequences FR not promptly notified per CRD Yes FR not promptly notified per CRD FR not promptly notified per CRD FR promptly notified per CRD FR promptly notified per CRD FR promptly notified per CRD FR promptly notified per “near miss” requirement FR promptly notified per CRD FR promptly notified per CRD Letter of concern from RL Yes Probably no letter Letter of concern from RL Probably no letter Probably no letter Probably no letter Probably no letter Probably no letter Probably no letter
  • 32. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 32 of 117 Barrier Analysis Matrix The Barrier Analysis Matrix is an implementation of the Threat-Barrier-Target Model, which has been a mainstay of the DOE approach to Root Cause Analysis for many decades. The matrix below evaluates the principal barriers in terms of their effectiveness during this episode only. Failed, missing, and ineffective barriers are often causes of adverse events. The Barrier Analysis Matrix does not provide analysis of the causes of the problem barrier conditions. Barrier Analysis Matrix © Barrier Target Protected Threat Effectiveness in this case Significance Note: In Barrier Analysis a 'barrier' is a device, measure, entity, or the like that has the effect of reducing the probability or consequences of a 'threat' to a 'target'. Devices that could have or are intended to have the same effect are also called 'barriers'. Note: Barriers that did work can be retained and reinforced. Barriers that did not exist can be deployed. Barriers that existed, but did not work, can be strengthened. SNF Project Notification Guideline Compliance with notification requirements Notifications not made Not Effective: Guideline did not ensure the FR would be notified when the FM is notified of bad news. Missed Opportunity: Notification to FR not made FH sensitivity to RL requirement that prompt notification was required for non-reportable bad news Customer (RL) satisfaction Customer (RL) finds out they have not been notified as required Not Effective: Despite a history of customer dissatisfaction in this area, FH did not take effective steps to rectify the situation Missed Opportunity: FH not sensitive to RL dissatisfaction with lack of prompt notification. FH Feedback Systems (i.e., LL, RCA) RL satisfaction with prompt notification Lack of awareness of problems Not Effective: These processes were not triggered because there was no documentation of customer dissatisfaction – required to initiate feedback processes Missed Opportunity: The feedback system did not recognize an emerging escalating problem
  • 33. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 33 of 117 Barrier Analysis Matrix © Barrier Target Protected Threat Effectiveness in this case Significance HNF-PRO-060 Compliance with prompt notification requirement for non- reportable bad news. Non- compliance with prompt notification requirement Not Effective: Human Factors not considered. May have prevented non- notification Review process for change to HNF-PRO-060 Compliance with prompt notification requirement for non- reportable bad news. Non- compliance with prompt notification requirement Not Effective: Procedure change was not reviewed. Missed Opportunity : CRD requirements in PRO-060 were not implemented Review cycle could have corrected human factors deficiency. Procedure verification and validation of HNF-PRO-060 Compliance with CRD Procedural defects that impede user understanding of requirements Not Effective: Did not happen, not required. Missed Opportunity: V&V is always available for complex processes. Failed Barrier: Systematic Approach to Training (A post training knowledge check is a barrier against continued ineffective knowledge.) Training to PRO-060 Ineffective training Not used. Missed Opportunity: Target population was not identified and effectiveness of training was not verified.
  • 34. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 34 of 117 Barrier Analysis Matrix © Barrier Target Protected Threat Effectiveness in this case Significance PRO-060 Training Compliance with requirement to notify RL of non-reportable bad news. Non- compliance with requirement Not Effective: Training to notify RL of non- reportable bad news was not performed. (SOM & FM not trained to revision.) Missed Opportunity: Personnel were not trained to the change. FM Communication of Facility Mgr Forum information Compliance with requirement to notify RL of non-reportable bad news. Non- compliance with requirement Not Effective: Not consistently applied across site Missed Opportunity: Not all personnel received communication Audits, assessments, and JHAs Weighted Safety Arm (safety device) Unrecognized use of a safety device as an operational control Not Effective: If use of the safety device as an operational control was recognized, additional controls would have been implemented. Missed Opportunity: Did not recognize the use of Weighted Safety Arms as operational controls. Audits, assessments, and JHAs Weighted Safety Arm (safety device) Failure to control for condition that sloping monorails could cause hoist to strike Weighted Safety Arms. Not Effective: If the sloped monorails were appropriately identified (even if not re-leveled), then additional controls could have been implemented. Missed Opportunity: An example of a control would be to hang caution tags on the hoists when they are on slanted monorails to say: CAUTION: Keep positive control while moving hoist. Monorail is tilted and hoist will continue to travel on its own.
  • 35. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 35 of 117 Barrier Analysis Matrix © Barrier Target Protected Threat Effectiveness in this case Significance Audits, assessments, and JHAs Weighted Safety Arm (safety device) Unrecognized impact on the locking arms from moving FTC during coupling to the monorail Not Effective: If damage caused by the moving FTC was recognized, additional controls would have been implemented. Missed Opportunity: Did not recognize damage was being caused by FTC. Preventive Maintenance procedure Functional Weighted Safety Arm Non- functional Weighted Safety Arm Not Effective: Did not find the bent/ sticking Weighted Safety Arm. Missed Opportunity: Did not incorporate procedural steps to ensure bent/ sticking latches were identified. Millwright Functional Weighted Safety Arm Non- functional Weighted Safety Arm Not Effective: Did not find the bent/ sticking Weighted Safety Arm. Missed Opportunity: Skill of the craft could have identified the bent/ sticking Weighted Safety Arm. Procedure OP- 14-002 “Perform Pre- Use Test on Hoist” The Hoist. Damage to Hoist. Not Effective: Did not require the performance of a pre-use test for relocation of the hoist AND, if a pre-use test had been required for this activity, there was no requirement to ensure the Weighted Safety Arm was in the down position. Missed Opportunity: The operating procedure could have required the operator to ensure the Weighted Safety Arm was in the down position.
  • 36. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 36 of 117 Barrier Analysis Matrix © Barrier Target Protected Threat Effectiveness in this case Significance Procedure OP- 07-113W “Position and Secure Irradiated Fuel at 105-KW” The Hoist. Damage to Hoist. Not Effective: Not intended to be effective. However, neither this procedure nor OP-14-002 would have prevented the hoist from falling from the monorail. Missed Opportunity: The procedure could have required the operator to ensure the Weighted Safety Arm was in the down position. FTC operating procedures Safe operation Damage to the Locking Arms (subsequently damaging the Weighted Safety Arm - bending it) Not Effective: Did not ensure that FTCs were not moving while coupling to the monorail Missed Opportunity: The procedure could have required that the operator ensure that FTCs were stopped prior to coupling to the monorail Training for hoist/monorail system operation Safe operation Repeated use of Weighted Safety Arms as operational controls (to stop the hoist). Not Effective: Weighted Safety Arms are used as operational controls. Missed Opportunity: Train people to keep positive control of hoist when moving it. Training for FTC operation Safe operation Repeated practice of coupling FTCs to monorails while FTCs are still moving Not Effective: Did not ensure that FTCs were not moving while coupling to the monorail Missed Opportunity: The training could have required that the operator ensure that FTCs were stopped prior to coupling to the monorail
  • 37. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 37 of 117 Barrier Analysis Matrix © Barrier Target Protected Threat Effectiveness in this case Significance SOM, FM, MCO Mgr, and Deputy Director Questioning Attitude with respect to near misses Compliance with near miss reporting requirements and correction of unsafe conditions Unidentified near miss situations Not Effective: Conversations between SOM on one hand and FM, MCO Mgr, and Deputy Director respectively did not surface the near miss Failed Barrier: Questioning attitude was not exhibited. Missed Opportunity: People at this level often exhibit questioning attitudes Exacerbating Factor: If this barrier had been successful, the whole episode would have been much less significant. Operators Safe operation Continued unsafe operations Success: Workers stopped work and notified management Without this, the opportunity to evaluate and correct the unsafe condition would have been missed.
  • 38. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 38 of 117 Discussion of Design Vulnerabilities At the time of this hoist fall event, the K Basin Fuel Storage Monorail System was old. The design was probably current at the time of construction. Comments on the design vulnerabilities should be taken in that context. They are intended not as criticisms of the design, but as possibilities for learning. Discussion of Process Vulnerabilities By process vulnerabilities, we mean vulnerabilities associated with the way evolutions are conducted, i.e., the process for moving hoists. In this case one single adverse contingency, a mis-positioned WSA, together with process issues, allowed the hoist drop to occur. This discussion applies to the design the way it was at the time of the hoist drop event. Procedures The procedure for operating hoists contained precautions and notes associated with the upper limit switch on the hoist. This is appropriate since upper limit switches should not be relied upon to stop the “UP” travel of the hoist. Conversely, similar steps were not included to caution operators on not utilizing the WSA as a normal means for stopping the trolley. Similarly, steps were not included to verify the WSA position prior to moving the trolley or to ensure positive control of the hoist/trolley assembly is maintained during movement to avoid challenging the WSA. Procedures that direct the operation of FTCs do not contain steps to ensure that FTCs are stopped prior to attempts to engaging the FTC to a monorail. No caution statements or other steps warn operators of the damage that can occur to the Locking Arm and WSA should an FTC be moving while engaging it to a monorail. The procedure for performing preventative maintenance does not ensure that deficiencies associated with the WSAs are identified. The procedure does not specify any acceptance criteria and does not require recording “as found” conditions, actions taken, and “as left” conditions. The procedure used for defining and declaring Near Miss events does not ensure that Near Miss events are recognized and reported. This problem exists not only here at FH, but across the DOE Complex. Also, this same procedure is used to ensure prompt notification of reportable and non-reportable occurrences to the DOE FR. However, the steps in the procedure are not well placed to ensure that when FMs are notified of non- reportable events (bad news), the FR is promptly notified of the same. The wording is essentially the same as the CRD, but it does not need to be. Simple words that clearly state that whenever the FM is notified of non-reportable “bad news”, the FR shall be promptly notified of the same bad news are all that is required.
  • 39. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 39 of 117 Training The training on the new CRD requirement for prompt notifications (as codified in the Occurrence Reporting procedure) was not adequate to ensure compliance with the CRD and customer expectations. While both the procedure and training did contain this information, the recently inserted requirement was not clear to most personnel who received this training. Many believed that the information was merely more of the same (notify the FR of abnormal events), without recognizing the significance of the change (FR shall be promptly notified anytime the FM is notified of “bad news”). In addition to the training content, the SAT did not appear to be used. No knowledge check was used to ensure the objectives of the training were met and no follow-up reviews were conducted to ensure the new requirement was being implemented. Also, while most of the applicable personnel were originally identified that needed this training, it does not appear that actions were taken to ensure they were scheduled, attended, or that it was added to their Training Matrix as required training to perform their assigned duties. Hoisting and Rigging training is provided to operators who use hoists. Specifically, the monorail hoist training contains information regarding the checking of upper limit switches and the need to ensure that operators due not use upper limit switches as a normal means to stop the “UP” travel of the hoist. No information could be found regarding the need to check the position of WSAs and to ensure positive control is maintained when moving the hoist/trolley assembly so that it does not collide with WSAs. Similarly, it would appear the training for operating FTCs does not warn against moving FTCs while engaging them to monorails. This practice has been witnessed in the past and is believed to be a probable cause for the damage being done to WSAs. Safety Vulnerability Audits Safety audits and assessments have failed to recognize “normal” operations as potential event precursors. The practice of allowing hoist/trolley assemblies to collide with WSAs, and the practice of engaging FTCs to monorails while they are still moving, was known by many within the SNF Project. The fact that these practices have been known, but allowed to continue points to a less than adequate sensitivity towards single vulnerability safety devices such as the WSAs. Discussion of Process Strength As soon as the scrap processing crew personnel were aware that the noise was caused by the fallen hoist, they stopped work and investigated the scene. The personnel then notified the SOM, who sent other personnel to investigate the fallen hoist. The use of “Stop Work” by the crew upon discovery of an abnormal event is encouraging. The prompt notifications made to the SOM and attempts to secure the scene were also encouraging.
  • 40. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 40 of 117 Discussion of Influences Although the definitive cause(s) for the bent WSA is not known, the WSA was likely bent as a result of the hoist/trolley assembly colliding with the WSA and/or the FTC colliding with the Locking Arm while it was moving and engaging to the monorail. The influence the bent WSA had on this event is that it stuck when moved to the up position. The definitive cause for the WSA being moved to the up position is unknown. However, one possible scenario is that long tools may have contacted the WSA, moving it to the up position. It is not clear to what extent hanging tools from the WSAs might contribute to it being moved to the up position. The influence of the WSA being in the up position is that it no longer provided that “rail stop” function it was designed to provide. Thus, when the hoist/trolley assembly was pushed to the south, it continued to roll until it passed the bent WSA in the up position and fell to the grating below. The cause for monorail 21 being tilted to the south is likely the result of a combination of aging/old equipment whose preventative maintenance program does not inspect and/or correct for this condition. The influence this tilted monorail had on this event is that once the hoist/trolley assembly was pushed to the south, it continued to roll down grade until it reached the end of the monorail, passing the bent WSA stuck in the up position, and fell to the grating below. The cause for the RCT to push the hoist to the south is not significant. However, it is important that if RCTs and other non-qualified personnel are permitted to move hoists/trolley assemblies out of their way, they need to understand that they must have positive control and have verified the position of WSAs. The influence the RCT had on this event is the creation of the momentum necessary to cause the hoist/trolley assembly to roll south on the monorail until it fell to the grating below. The fact that no pre-use check of WSA positions is required had an influence on this event. Had personnel been trained, and the procedure directed the performance of this check prior to movement of the hoist/trolley assembly, the WSA would have been discovered in the up position and the event would have been avoided. The cause for the less than adequate preventative maintenance (PM) for WSAs appears to be due to the lack of recognition that this equipment performs a vital safety function. The influence this had on this event is that the PM (as currently written) did not ensure that the bent WSA (safety device) was identified and corrected. Although the area where the hoist fell is a normal traffic area, no personnel were in the vicinity of the end of the monorail when the hoist fell. This had a beneficial influence on this event since the result was no personnel were injured (Near Miss). The cause for the RL concern regarding prompt notifications (or the lack there of) is a result of a lack of sensitivity (past and present) to this issue by FH. The RCA Team believes that this condition alone likely influenced the customer to issue a formal letter of concern. The CRD wording incorporated into HNF-PRO-060 had a significant influence in the prompt notification issue reflected in this report. The fact that the wording was “overly wordy” likely contributed to a lack of recognition that this was a change from past
  • 41. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 41 of 117 practice (FH has always been required to notify the FR of abnormal events). Also, the fact that the change was not human factored when incorporated into HNF-PRO-060 was significant because personnel following the procedure would not reach this step since the section it is listed in deals with “Reportable Occurrences”. Training to the new requirement (HNF-PRO-060) for prompt notification to the FR whenever the FM is notified was ineffective. The influence this had on personnel is a lack of sensitivity to a historical concern from RL and an inability to recognize the need to promptly notify the FR whenever the FM is notified of “bad news”. This training also failed to correct a long standing problem of identifying Near Miss events. Not surprisingly, this problem still exists throughout the DOE Complex and FH has yet to satisfy the customer’s concern regarding a perceived lack of sensitivity to this issue. The inability of management and others to apply a “questioning attitude” to some events may influence others to not recognize them as near misses. Despite the design, process, administrative, and operational issues, the application of a “questioning attitude” alone would have prevented this event or made it less consequential at many junctures.
  • 42. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 42 of 117 Corrective Action Recommendations The Corrective Action Table addresses the principal influences on the consequences of this episode. It not only includes recommended corrective actions, but also provides an assessment of the expected impact of the corrective action, if implemented. The table also addresses the expected impact on the episode being investigated if the corrective action had been in place before the event. For each corrective action recommended, the team provides its advice on the priority of the corrective action. Corrective Action Table Consequence Influence Corrective Action Expected Impact Impact on Current Event if pre- implemented Remarks Not keeping positive control of the hoist when moving it. Procedure change: positive control of hoist when moving it. High Priority Hoist does not hit WSA and does not fall off a monorail when there is no WSA. None on this event (since RCT does not follow operating procedures.) Need corrective action related to RCT performance. RCT moved hoist without proper precautions. Procedure change: all activities in high hazard areas to be done by qualified individuals. High Priority People moving hoists and doing other potentially hazardous activities will act safely. RCT either would have been properly qualified or would have asked for a properly qualified individual to move the hoist. Maintaining positive control would be one of the proper precautions. Tilted monorail. None: Residual risk None. None. Previous corrective actions would make tilted monorail non- significant. Need USQE.
  • 43. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 43 of 117 Corrective Action Table Consequence Influence Corrective Action Expected Impact Impact on Current Event if pre- implemented Remarks Distorted WSA hardware. PM instruction change: Effective checking and condition recording of WSA hardware. Medium Priority WSA hardware problems would be identified. WSA hardware problems would have been identified a month before the event, and the event would not have happened. PM leaves a time window for adverse conditions to remain undetected. This barrier is not 100% effective. Distorted WSA hardware. Procedure change: Pre- use operational checks of WSA hardware on monorails to be used. High Priority WSA mis- positioning would be identified and corrected before moving hoists. None on this event (since RCT does not follow operating procedures.) Need corrective action related to RCT performance. Distorted WSA hardware. Procedure change: Operational controls to ensure FTCs are not moving when coupling with monorails High Priority FTC does not impact Locking Arm Locking Arm and subsequent WSA damage may not have occurred Reduces potential for damaging the locking arms and subsequent WSAs
  • 44. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 44 of 117 Corrective Action Table Consequence Influence Corrective Action Expected Impact Impact on Current Event if pre- implemented Remarks Numerous conditions adverse to safety associated with operation and material condition of the FSMS. Apply ISMS to FSMS operation and maintenance. Low Priority Operational and material hazards will be identified and controlled. WSA distortion issues would have been identified. Safety device precursor issues would have been identified. Positive control issues would have been identified, and so forth. There was no evidence that FSMS operation and maintenance had received the benefit of ISMS. SOM does not recognize near miss. FH Senior Mgmt establish and enforce their expectations for near miss recognition by FH personnel. High Priority Reduced likelihood of mis- identification of near miss situations. SOM would have recognized near miss. Near miss would have been reported. No opportunity to misunderstand CRD requirements. Without Senior Mgmt reinforcement of good near miss declaration performance, this corrective action will be useless. Inadequate challenge of SOM telephone notification results in situation not being fully described to FM. Questioning attitude workshop training for all mgmt personnel whose duties include receiving notifications from Shift Managers. High Priority Shift Management personnel will receive collegial assistance from the senior managers to whom they make notifications. FM would have led SOM through the proper logic to identify that only luck stood between the hoist drop situation and an injury. All new managers in this situation must be trained before taking on duties involving receiving notifications.
  • 45. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 45 of 117 Corrective Action Table Consequence Influence Corrective Action Expected Impact Impact on Current Event if pre- implemented Remarks Inadequate challenge of SOM telephone notification results in situation not being fully described to MCO Production Mgr. Questioning attitude workshop training for all mgmt personnel whose duties include receiving notifications from Shift Managers. High Priority Shift Management personnel will receive collegial assistance from the senior managers to whom they make notifications. MCO Production Mgr would have led SOM through the proper logic to identify that only luck stood between the hoist drop situation and an injury. This is a repeat corrective action; it only needs to be done once. HNF-PRO-060 contained human factors defects. Subject HNF- PRO-060 to validation and verification that includes consideration of human factors effectiveness. Medium Priority On-shift personnel involved in occurrence response will be able to pick up the procedure and succeed. SOM would have understood that “non- reportable” bad news given to the FM should be given to the FR. No notification issue would have arisen. HNF-PRO-060 is probably not the only administrative procedure that is a human factors problem. HNF-PRO-060 training was ineffective in that it did not result in the desired performance. Retrain personnel on this procedure meeting the intent of the Systematic Approach to Training. High Priority On-shift personnel involved in occurrence response will be able to pick up the procedure and succeed. SOM would have understood that “non- reportable” bad news given to the FM should be given to the FR. No notification issue would have arisen. SAT is just good business. Training that does not meet the intent of SAT is not effective.
  • 46. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 46 of 117 Corrective Action Table Consequence Influence Corrective Action Expected Impact Impact on Current Event if pre- implemented Remarks There was no post-training effectiveness check on HNF- PRO-060 training. Conduct post- training effectiveness checks on future HNF- PRO-060 training. High Priority Any significant performance issues caused by HNF-PRO- 060 training will be detected. Significant performance issues would have been identified and corrected. SOM would have understood that “non- reportable” bad news given to the FM should be given to the FR. No notification issue would have arisen. This would be included in meeting the intent of SAT.
  • 47. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 47 of 117 Analysis of Expected Effectiveness of Recommended Corrective Action The RCA Team has concluded that if FH implements the corrective actions marked “high priority”, FH will not have another event like the hoist drop event and will not have another notification shortfall. It is important that the effectiveness of these corrective actions be periodically confirmed. Conclusions The RCA Team completed all four objectives of its charter. In the context of high hazard industry events, this one is of rather low consequence. However, its significance is high because it is a result of behaviors and conditions that could be ingredients of much more consequential events. The team believes that the implementation of the recommended corrective actions and the dissemination of the lessons to be learned will likely result in the saving of lives, pain, assets, and careers at Fluor Hanford. The three main areas for improvement are: 1. Conduct of Operations and Maintenance. 2. Handling of Adverse Event Information 3. Rapid Investigative Response
  • 48. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 48 of 117 Attachments Comparative TimeLine ©......................................................................................... 49 Why Staircase Trees............................................................................................... 73 Extent of Condition and Cause Matrix..................................................................... 83 Cause Codes .......................................................................................................... 92 Documents Reviewed ........................................................................................... 103 Personnel Contacted............................................................................................. 106 Analysts Comments .............................................................................................. 108 Structural Analysis Report..................................................................................... 111 Calculations........................................................................................................... 113 Unanswered Questions......................................................................................... 117
  • 49. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 49 of 117 Comparative TimeLine© The Comparative TimeLine© is a convenient method of capturing the behaviors and conditions that made up the episode. The resources required to carefully construct a Comparative TimeLine pay for themselves many times over in making it easier to do the Why Staircase Trees, the Barrier Analysis Matrix, the Missed Opportunity Matrix, and other tools. When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) Earlier FH management chose not to notify RL of certain “non- reportable” bad news that was reported to Facility Managers (FMs). FH management should have determined RL needs and complied. RL becomes dissatisfied. Set-up Factor: This sets the stage for the notification consequence. About 2003_09 Paul Golan (DOE-HQ) established metrics that were drivers for RL notifications to HQ. OK RL identifies what types of things it wants to be notified of. These included “non- reportable” bad news usually reported to FMs. Set-up Factor: Creates additional need for Facility Representatives (FRs) to get the types of information they wanted anyway. Prior to 2003_11_03 RL drafted Contracts Requirements Document (CRD) asking for notification to FR of bad news reported to FM. OK FH has to change PRO-60 to implement this requirement. Set-up Factor: Creates formal requirement for notifying the FR of non- reportable bad news. Prior to 2003_07 A revision of HNF-PRO-060 not including the CRD was in the review cycle. It had been reviewed by many reviewers. OK Created opportunity to change PRO-60 after it had been reviewed. The post-review addition of the CRD provisions meant that some early reviewers thought they knew what was in PRO-60, but didn't.
  • 50. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 50 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) Prior to 2003_11_03 Shift Managers, Occurrence Reporting personnel, and others were trained on the Revised PRO-060 (with CRD provisions). The K West Basin Operations Shift Manager (SOM) was not trained. The SOM should have received this training. The SOM was not aware of FR notification requirements for non- reportable bad news. Set-up factor: Creates a condition for non- compliance with CRD. 2003_11_03 CRD provisions for non- reportable bad news were inappropriately placed in a section on reportable occurrences. CRD provisions for non- reportable bad news should have been appropriately placed in a section on non- reportable occurrences that are reported to FM’s, placed in a table, or otherwise presented in a human factored way. Increased the probability of non-compliance. Set-up factor: Creates a condition for non- compliance with CRD. 2003_11_03 HNF-PRO-060 Revision 7 issued with CRD provisions. Revised PRO-60 should have gone back through the review cycle. Reviewers did not have the opportunity to comment on the human factors deficiencies in how CRD provisions were incorporated. Failed barrier: Review cycle could have corrected human factors deficiency.
  • 51. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 51 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) After 2003_11_03 Director of Emergency Preparedness briefs Facility Managers’ Forum on CRD provisions in PRO-060. Only the FMs received this CRD training. The appropriate target audience should have been trained on the CRD provisions in PRO-060 in accordance with the Systematic Approach to Training (SAT). The K West Basin FM did not receive this training. Not all the personnel involved in notifying FRs of non-reportable bad news are trained to do so. Missed Opportunity: All personnel involved in notifying FRs could have been trained to do so. Failed Barrier: - SAT is a barrier against ineffective training Set-up factor: Creates a condition for non- compliance with CRD. After 2003_11_03 The Director of EP does not check to see that the training was effective, i.e., that personnel involved in notifying FR’s of non- reportable bad news know they are supposed to do so. In accordance with SAT, trainees’ knowledge should have been checked. The training deficiency is not noticed by FH. Missed Opportunity: A check of the training would have found that it was ineffective - Leaves set-up factors in place. Failed barrier: A post training knowledge check is a barrier against continued ineffective knowledge. Set-up factor: Creates a condition for non- compliance with CRD.
  • 52. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 52 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) After 2003_11_03 The FMs who attended the Forum did not tell facility personnel about the requirement to notify the FR about non-reportable bad news when the FM is notified. The FMs should have informed facility personnel about the requirement to notify the FR Not all the personnel involved in notifying FRs of non-reportable bad news are informed of the requirement to do so. Missed Opportunity: All personnel involved in notifying FRs could have been informed to do so. Failed Barrier: - Communications LTA – The FMs did not communicate the new CRD requirement to facility personnel AND the PRO- 060 author did not communicate the new CRD requirement to all of the procedure users. After 2003_11_03 Non-reportable bad news is reported to FMs, but not to FRs. Non-reportable bad news reported to FMs should also have been reported to FRs. FH Management not sensitive to Increased RL dissatisfaction Set-up Factor: Increases level of FR dissatisfaction. After 2003_11_03 FR complaint about not being told about non- reportable bad news reported to FMs. OK FH management not sensitive to RL dissatisfaction Set-up Factor: Problem continues
  • 53. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 53 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) After 2003_11_03 FH management does not reinforce expectation for CRD compliance, that FRs be notified when FMs are notified. FH management should have noted that previous training was LTA and fixed it. Conditions for non- compliance continue. Missed Opportunity: FH management could have aborted the notification portion of this event by correcting a known condition adverse to business. Failed barrier: Management failed to correct the non-compliant condition (via training improvement and reinforcement of expectations) Set-up Factor: Leaves the previous set-up factors in place. The Fuel Storage Monorail System (FSMS) is designed such that a single barrier prevents a hoist from dropping off the monorail. This single barrier is the Weighted Safety Arm (safety device). Training does not sensitize users of this single failure vulnerability. The combination of the design and the way it was used is an example of using safety protection for operational control. Earlier Workers routinely rely on Weighted Safety Arms to stop moving hoists during normal operations (passive safety devices used as operational controls). Workers should not have been using WSAs as hoist stopping devices. Repeated hoist drop precursors (the Weighted Safety Arm safety device was repeatedly challenged). Set-up Factor: If the hoist is moved to the end of the Monorail and the Weighted Safety Arm is up or breaks, the hoist will fall.
  • 54. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 54 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) Earlier Audits, assessments, and Job Hazards Analyses (JHAs) of the FSMS do not notice the single failure vulnerability of the Weighted Safety Arms. Audits, assessments, and JHAs should have noticed the single failure vulnerability and required that procedures be changed to either frequently verify the condition of the Weighted Safety Arms, or prohibit challenging them. Set-up Factor is not corrected. Missed Opportunity: Any of these safety audits could have aborted the event by establishing additional controls. Failed Barrier: Audits, assessments, and JHAs are barriers that guard against safety vulnerabilities. ISMS infraction. Earlier Audits, assessment, and JHAs of the FSMS do not notice the routine repeated use of the WSA (safety device) to stop a hoist. Audits, assessments, and JHAs should have noticed the routine repeated use of the safety device to stop a hoist. The recognition of the use of WSAs to stop hoists should have resulted in PM and Operating procedures that ensure their position and operability. The precursor continues. Missed Opportunity: Any of these could have aborted the event by identifying and correcting the unsafe behavior. Failed Barrier: Audits, assessments, and JHAs are barriers that guard against safety vulnerabilities. ISMS infraction.
  • 55. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 55 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) 2003_02_03 100K monorail inspection procedure (annual inspection) SP-14-005 changed to add rail stops (Weighted Safety Arms) to Step 4.1.2 to ensure they were checked for looseness and alignment. OK Missed Opportunity: The procedure change acknowledges the need for Millwrights to check Weighted Safety Arms, but many Weighted Safety Arms were not in proper alignment. 2003_06_09 The Crane Inspection Team was chartered as a work place improvement initiative for the K Basins. The charter was to evaluate reported issues and make recommendations for resolution. OK The FSMS was first built in 1954, and has become worn out over the years, requiring extensive repair to restore “normal” operation. Although some FTCs have been replaced with new ones, original FTCs are worn to the point that alignment is not possible with many of the monorails. Additional damage results from the increased forces necessary for the workers to align FTCs with monorails during hoist transport of fuel loads exceeding 700 pounds. Worker frustration resulting from difficult FSMS operations with worn-out equipment may have contributed to equipment abuse.
  • 56. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 56 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) 2004_02_19 FSMS Maintenance completed the annual preventive maintenance inspection on the Weighted Safety Arms in the K West Basin. No discrepancies were noted in the work package. Discrepancies (bent or sticking Weighted Safety Arms) should have been found. No problem was found with the Weighted Safety Arm on the south end of Monorail 21. The bent WSA was not identified and corrected. Missed Opportunity: The bent or sticking condition of Weighted Safety Arms - FSMS Maintenance does not provide adequate documentation of any repairs in the work packages to provide data for trending. Failed Barrier: No problems with Weighted Safety Arms were identified by performing the PM. Set-up Factor: Problem continues. 2004_02_19 FSMS preventative maintenance was performed to a procedure that did not require that an FTC be installed when inspecting WSA functionality. The PM procedure should have required that an FTC be engaged to the monorail. No problem was found with the Weighted Safety Arm on the south end of Monorail 21. The bent WSA was not identified and corrected. Missed Opportunity: The bent or sticking condition of Weighted Safety Arms. Failed Barrier: Because the FTC was not engaged to the monorail, no problems with Weighted Safety Arms were identified by performing the PM. Set-up Factor: Problem continues.
  • 57. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 57 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) Before 2004_03_10 Personnel had been working in the basin pool under the south end of Monorail 21 using long tools. OK Created opportunity for personnel to strike Weighted Safety Arm with long tool during work. Set-up Factor: Possible inadvertent contact with Weighted Safety Arm causing it to travel beyond its normal range of motion, and allowing it to become stuck. Missed Opportunity: Long tools were known to strike or become stuck in the overhead during use. Before 2004_03_10 Personnel had been working in the basin pool under the south end of Monorail 21. OK Created opportunity for personnel to be injured if the hoist falls on them. Exacerbating Set-up Factor: This converts a potential equipment event into a personnel safety event. Personnel working in the basin use long poles to retrieve empty fuel canisters. The canisters are hooked to a chain on the hoist, so the hoist can lift the canister in the water, and the canister can be transported along the monorail. The hoists roll by manual operation, but lift by means of an electric motor. The monorail system is approximately seven feet above the grating over the water. The grating is approximately 22 feet above the bottom of the pool. In use, long poles are reported to often strike into the overhead. Before 2004_03_10 The Weighted Safety Arm was bent. The Weighted Safety Arm should not have been bent. The bent Weighted Safety Arm became stuck in the up position. Set-up Factor: If the Weighted Safety Arm had not been bent, it would have been in the down position.
  • 58. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 58 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) 2004_03_10 The south-end Weighted Safety Arm on Monorail 21 is in the up position. The Weighted Safety Arm should have been in the down position. The Weighted Safety Arm could not perform its intended function. If the hoist travels to the end of the monorail, it will fall off. Set-up Factor: The Weighted Safety Arm in the up position will allow a hoist traveling to the end of the monorail to fall off. Missing Barrier: The Weighted Safety Arm in the down position (barrier) was missing. Before 2004_03_10 Pre-use hoisting and rigging checks do not identify the Weighted Safety Arm on Monorail 21 in the up position. Pre-use hoisting and rigging checks should identify safety anomalies including the Weighted Safety Arm on Monorail 21 being in the up position. Set-up Factor allowed to continue. Missed Opportunity: The pre-use checks could have aborted the event by identifying the Weighted Safety Arm in the up position. Missing Barrier: The pre-use checks did not require checks of the Weighted Safety Arms. 2004_03_10_ 2317 Radiological Control Technician (RCT) who is part of a waste processing crew needs to move hoist. OK The RCT moves the Hoist. Triggering Factor. 2004_03_10_ 2317 (before) Monorail 21 was tilted downward to the south. (A number of monorails in the K West Basin were known for many years to have “a long glide path”.) Monorail 21 should have been level. Created vulnerability to continued hoist motion once the hoist is started. Set-up Factor.
  • 59. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 59 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) The Weighted Safety Arms were verified in the down position three weeks preceding this event by personnel working at the south end of Monorail 21. 2004_03_10_ 2317 RCT, following normal practice, does not request a Qualified Operator to move hoist. RCT should have requested Qualified Operator to move the hoist. A non-qualified operator moved the hoist. Extraneous Condition Adverse to Quality 2004_03_10_ 2317 Procedure OP-14-002 “Perform Pre-UseTest on Hoist” did not require pre- use testing when relocating hoist. The procedure should have required visual inspection to verify the Weighted Safety Arms (safety device) were in the down position. The Weighted Safety Arm was not visually inspected. Set-up Factor Missing Barrier: The procedure did not require a pre-use check before the hoist was relocated. Procedure OP-14-002 “Perform Pre-UseTest on Hoist” precautions and limitations stated the following: “Limit switches are safety devices for protection of personnel and equipment. Limit switches shall not be used as a controller to stop the upward travel of the hoist.” Section 4.2 includes the following note: “The up limit switch is a back-up; its use should be avoided during normal operation.” The pre-use test procedure does not recognize the Weighted Safety Arms as safety devices and does not call out pre-use inspection. The team could not find any safety principle that allowed treating mechanical safety devices less rigorously than electrical safety devices.
  • 60. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 60 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) 2004_03_10_ 2317 RCT pushes hoist to south. A Qualified Operator should have moved it. (Note that if the hoist had been moved to the north this event would not have happened.) Hoist begins moving to the south. (We do not know whether the Qualified Operator would have behaved differently. E.g., a Qualified Operator may have intuitively moved the hoist northward or may have noticed the Weighted Safety Arm in the up position, or may have moved it south more carefully.) Final Triggering Factor. 2004_03_10_ 2317 The hoist rolls southward and travels off the end of the monorail. OK (given the existing set- up factors) Hoist is in free-fall. Natural outcome of prior behaviors and conditions. 2004_03_10_ 2317 No personnel or sensitive equipment are in vicinity of the hoist trajectory. OK: Fortuitous condition No personnel or equipment consequences (except to hoist casing). Mitigating Factor. This non-robust barrier prevented this near miss from being a consequential Safety Event. 2004_03_10_ 2318 The five involved personnel (1 RCT and 4 Nuclear Chemical Operators (NCOs)) went to the fallen hoist and confirmed the Weighted Safety Arm was in the down position. The immediate inspection found no obvious problems with the hoist or monorail. OK Inexplicable condition discovered.
  • 61. Root Cause Analysis: Hoist Drop at K West Spent Fuel Pool on March 10, 2004 Page 61 of 117 When What did happen What should have happened Immediate Result (Consequence) Significance (Impact on Final Consequences) 2004_03_10_ 2330 An NCO notified the K West Basin SOM that a hoist had fallen off the south end of Monorail 21 onto the grating; no personnel were in the area of the fallen hoist. OK Notification sequence starts 2004_03_10_ 2330 SOM does not declare that the hoist drop was a near miss for personnel injury. Because the hoist dropped in an area where people work, the drop should have been recognized as a near miss. The hoist drop is not treated as a near miss and near miss notifications are not made. Missed Opportunity: First Failed Barrier to CRD non-compliance. 2004_03_10_ 2330 Discussion between the personnel and the SOM led to determination that, other than the hoist pre-use operational check, no specific check of the Weighted Safety Arms is required; also, that no Flexible Transfer Cranes (connecting to Weighted Safety Arms on the monorails) had been in use along the south part of the basin. OK 2004_03_10_ 2330 Interviews of involved personnel were not conducted. Interviews should have been obtained as soon as possible after the event. Extraneous Condition Adverse to Quality Extraneous Condition Adverse to Quality: Real-time recording of the facts at the event scene did not occur.