Root cause analysis master plan

+
18.0 Root Cause Analysis
Root Cause Analysis is the method of problem solving that
identifies the root causes of failures or problems. A root cause is
the source of a problem and its resulting symptom, that once
removed, corrects or prevents an undesirable outcome from
recurring.
Failure
Is Trying to Tell Us Something
Performance–Based Project Management®, Copyright © Glen B. Alleman, 2002 ― 2016 842

+ The Notion of Root Cause Analysis
Symptom:The result or outcome of the problem.
An observation
§ We have late additions to the release that break the software
§ We have “core” defects that should been caught long before
production release
§ We make changes to software,stored procedures,or the
database only to discover it was a mistake.
§ We make promises to the customer before assessing the
impact on our resources or the technical difficulty
Problem:
§ Test coverage insufficient the detect latent bugs in software
§ We commit before understanding the consequences
Root Cause:
§ No software structure to determine test coverage or change
impacts on baseline.
§ No detailed understanding of our capacity for work and
productivity of our technical staff
18. Root Cause
Performance–Based Project Management®, Copyright © Glen B. Alleman, 2002 ― 2016
843

+ Four Root Causes for Program Cost
and Schedule Overruns
844
Unrealistic Performance Expectations,
missing Measures of Effectiveness (MOE)
and Measures of Performance (MOP).
Unrealistic Cost and Schedule estimates
based on inadequate risk adjusted growth
models.
Inadequate assessment of risk and
unmitigated exposure to these risks
without proper handling plans.
Unanticipated technical issues without
alternative plans and solutions to maintain
effectiveness.
Unanticipated
Cost and
Schedule
Growth
TheLensofPerformance
Assessment
“Borrowed” with permission from Mr. Gary
Bliss, Director, Performance Assessment and
Root Cause Analysis (PARCA), Office of
Assistant Secretary of Defense for Acquisition,
Technology and Logistics.
18. Root Cause

+ Our Path to Better Root Cause
Analysis
n Principles of Root Cause Analysis
n Understanding the weaknesses in our current method
n Introduction the Apollo Method
n Steps to applying Apollo
n Transition from the current method to Apollo
845
18. Root Cause
ISO/IEC 17025:2005 (4.11.2) ‒ The procedure for corrective action
shall start with an investigation to determine the root cause(s) of the
problem.

+ Evaluation of Significance
n Can the nonconformity
recur or does it raise
doubt about compliance
of the laboratory’s
operations with its own
policies and procedures?
(ISO/IEC 17025, Clause
4.9.2)
n What criteria is utilized to
determine significance?
(ISO/IEC 17025, Clause
4.9.1.b)
846
18. Root Cause

+ A Simple Perspective of Problem
Solving
n Every Problem in our lives
has three basic elements
connected through
causality.
n Each Effect, has at least two
causes:
n An Action
n A Condition
847
Effect
Condition
Cause
Action
Cause
This Cause and Effect relationship is the core principle of
effective problem solving
18. Root Cause

+
18.1 Beyond the
Conventional
Wisdom of
Problem
Solving
Original Quote from George Bernard
Shaw
In Apollo Root Cause Analysis: Effective
Solutions to Everyday Problems,Every
Time, Dean L. Gano
The common approach to problem
solving is to categorize causes or
identify causal factors and look for
root causes within the categories.
Categorization schemes do not reveal
the cause and effect relationships
needed to find effective solutions.
It is the effective solution we are after.
18. Root Cause
848
Ignorance is a most wonderful thing.
It facilitates magic.
It allows the masses to be led.
It provides answers when there are none.
It allows happiness in the presence of
danger.
All this, while the pursuit of knowledge
can only destroy the illusion. It is any
wonder mankind chooses ignorance?

+ The Persistent Problem …
Increasing use of larger and more complex systems
potentially results in greater number of problems,
evidenced by the symptoms of unstable software, latent
defects, unexpected performance issues.
Many of these problems and their symptoms, have a serious
impact on business operations than ever before. In many
case these problems are also more difficult to solve.
— A Management System for the Information Business,
Edward Van Schaik
18. Root Cause
We can not solve problems by using the same kind of
thinking we used when we created them.
— Albert Einstein
849

+ … and the Persistent Unsuccessful
Solution to recurring problems.
In every human endeavor, a critical component to success is
the ability to solve problems.
Unfortunately, we often set ourselves up to fail with our
problem–solving strategies and our inherent prejudices.
We typically rely on what we believe to be common sense,
storytelling, and categorizing to resolve our problems.
Conventional wisdom has us believe that problem solving is
inherent to the subject at hand.
— Dean L. Gano,The Apollo Method
18. Root Cause
850

+ Some sources of Guidance for Root
Cause Analysis†
§2.4.5.1 Event Management – is the process that monitors all events
that occur through the IT infrastructure to allow for normal operation
and also to detect and escalate exception conditions.
§2.4.5.2 Incident Management – concentrates on restoring the
service to users as quickly as possible, in order to minimize business
impact.
§2.4.5.3 Problem Management – involves root–cause analysis to
determine and resolve the cause of events and incidents, proactive
activities to detect and prevent future problems/incidents and a
Known Error sub process to allow quicker diagnosis and resolution if
further incidents do occur.
† §2.4.5. Processes within Service Operation, ITIL V3 Service Operation
18. Root Cause
851

+ Root Cause Analysis
n A structured approach to investigating and analyzing of significant
adverse events or system deficiencies and their required
improvement.
n Root Cause Analysis provides information and tools to be
incorporated into risk management, quality management,
independent verification and validation and improvement
procedures in order to:
n PREVENT future occurrence of adverse events that cause or can cause
harm to individuals; and,
n CORRECT practices that have led to identified deficiencies
18. Root Cause
852

+
n The cause of a system defect
or failure needs to be
identified so the cause can be
eliminated in the future.
n The symptom and problem are
recurring
n There are repeated failures
ascribed to human error,
process errors, technical
failures.
n To review every incident or
system failure
n Only symptoms, failures, and
incidents that impact business
functions, Service Level
Agreements, maintenance,
operations, security, and
integrity of the system in the
future.
n When the reason for the failure
is obvious
n If it is obvious – fix it.
n Use RCA When … n Do Not Use RCA …
Do’s and Don’ts of Root Cause
Analysis
18. Root Cause
853

+ Root Cause
n Direct causes often result from another set of causes, which could be
called intermediate causes, and these may be the result of still other
causes.
n When a chain of cause and effects is followed from a known end-
state, back to an origin or starting point, root causes are found.
n The process used to find root causes is called root cause analysis ---
systematic problem solving.
n A root cause is an initiating cause of a causal chain which leads to an
outcome or effect of interest.
18. Root Cause
854

+
Discovering Root Cause starts with …
18. Root Cause
855

+ The Five Whys
Five Whys involves holding meetings immediately – in our case the Friday
immediately following the release.
The problems can be anything – development mistakes, infrastructure
performance issues, process failures, or even internal missed schedules.
Any time something unexpected happens, we start the root cause analysis
with the 5Ways in the domain from the chart above.
Process – how do the process not catch the problem?
Tools – where in the path did the tool not catch the problem?
Training – what training was missing that created the problem?
Environment(s) – how did the environment fail to protect the baseline or
the production system?
Communications – what communications created the problem?
Management – how did management fail in its actions?
The 5 Whys NEVER seek blame. But are exclusively focused on
discovering the root cause that creates the problem, evidenced by the
symptom
18. Root Cause
856

+ Five Steps in the Five Whys
n Invite all affected parties
n Select the meeting leader
(someone training in RCA)
n Ask Why 5 times for each
topic area
n Assign responsibilities for
collected actual factual data
n Publish the results
Start this process with the planned release of 29 January.
18. Root Cause
857

+ Why Root Cause Analysis is Hard
n Many problems are poorly defined.
n A systematic approach is not used to classify problem and cause.
n Investigations are stopped prematurely – moving on the next problem.
n Decisions are based on guesses, hunches or assumptions.
n An inadequate level of detail is used to get to the real root cause.
n Interim containment fixes are sometimes allowed to become
"permanent."
n The skills, knowledge and experience needed to uncover the root
cause are not available.
n Lack of organizational will to address the bigger issues
n Fear of being blamed
n I really don’t have time for this,we have bigger problems to solve
18. Root Cause
858

+ Testing Answers from the
Five Whys
n What evidence is there that this cause exists?
n Is it concrete?
n Is it measurable?
n What evidence is there that this cause could lead to the observed effect?
n Are we merely asserting causation?
n What evidence is there that this cause actually contributed to the problem?
n Even given that it exists and could lead to this problem, how do we know it wasn't
actually something else?
n Is anything else needed, along with this cause, for the stated effect to occur?
n Is it self-sufficient?
n Is something needed to help it along?
n Can anything else, besides this cause, lead to the stated effect?
n Are there alternative explanations that fit better?
n What other risks are there?
18. Root Cause
859

+
18.2 Understanding
the Weakness
of our Current
Approach to
Root Cause
Analysis†
n A common Root Cause Analysis
approach is to start with a narrative of
what happened and how we think that
undesirable state was achieved.
n This is a storytelling approach.
Storytelling uses linear language and
linear thinking
n Stories start in the past, while causal
relationships start in the present
n Stories are linear while causal
relationships follow the branches of
an infinite set
n Stories use inference to communicate
meaning, while problems are known
by sensed causal relationships
The overriding theme of traditional
Root Cause Analysis is the focus on
the Root Cause. We can eliminate the
problem if we eliminate the Root
Cause.
This assumes the causal relationships
are linear and that problems come
from a single source.
18. Root Cause
† Apollo Root Cause Analysis: Effective Solution's to Everyday
Problems, Dean L. GanoPerformance–Based Project Management®, Copyright © Glen B. Alleman, 2002 ― 2016
860

+
Stories seldom identify causes because they are busy setting the stage for
who was where and when some action occurred.
A story is a sequence of events starting in the past, leading to the
consequences disguised as a root cause
Root Cause Analysis is Not about Story Telling
861

+ Core Failure of Story Telling and
the Filling Out of Forms
n Stories rely on experience and judgment of the authors to connect
the causes of the problem.The mapping between Event, Cause, and
Effect not provided in the story narrative.
n Story telling can be used to document the investigation and
describe the corrective actions. But stories are poor in providing the
analytical connections between cause and effect.
n Measures of the effectiveness of corrective actions can not be
provided by narratives.Traceability between Effect, Action, and
Condition can not be provided by the narrative.
n It is a false premise that analysis of a problem, its causes, effects,
and solutions can be reduced to filling out a form and checking
boxes.
18. Root Cause
862

+ Story Telling Is Not Good Root Cause
Analysis Approach
n Story telling describes an event by relating people (who), places
(where), and things (what) in a linear time frame (when).
n When using storytelling to analyze an event (system, outage for
example), the causes identified in the report are difficult to follow
and hinder our ability to understand the relationships between all
the causes and provide a critique of the analysis.
n The investigators may well understand all the causal relationships,
but because they are not presented causally it is difficult to know
these relationships.
n Peer reviews will result in more questions because of the missing
connections between Primary, Intermediate Effects, and their
Causes.
18. Root Cause
863

+ Problem with the Story Telling Approach
to Root Cause Analysis
n Stories start with the past – we saw this happen and something else
happened after that,and then something else happened…
n Causal relationships leading to the Root Cause start with the present
and work backwards to the causes – both Activities and Conditions of
this cause.
n Stories are linear – they come from the minds of the story tellers,
usually as a linear time line.
n The linear understanding of an event in a time sequence from past to
present, ignores the cause–and–effect principle.
n Since we do not understand the branched causes, we use our own
understanding of cause rather than the actual causal connections.
n Stories use inference to communicate causes.
n Causal relationships require clear evidence of the existence of each
cause and its connection to the effects.
18. Root Cause
864

+ Root Cause Analysis is the
Event, the Cause, and the Resulting Effect
n We need a structured approach to investigating and analyzing
significant adverse events or system deficiencies and their required
improvement – not based on Story Telling.
n We need an approach that provides information and tools to be
incorporated into risk management, quality management,
independent verification and validation and improvement
procedures in order to:
n PREVENT future occurrence of adverse events that cause or can cause
undesired performance of our systems.
n CORRECT practices that have led to identified deficiencies.
n This approach separates story telling from the Primary Effect, and
the cause–effect chain leading to the Primary Effect.
18. Root Cause
865

+ A Better Approach to Root Cause Analysis for
Primary Effect, Cause, and Effect
n Direct causes often result from another set of causes – the
intermediate causes – and these may be the result of still other
causes.
n This chain of cause and effect needs to be revealed in a way that
clearly points to the corrective actions.
n When a chain of cause and their effects is followed from a known
end–state (time now), back to an origin or starting point, root causes
are revealed and corrective actions can be applied.
n A root cause is an initiating cause of the causal chain which leads to
an outcome or effect of interest.
18. Root Cause
866

+ Why Root Cause Analysis is Hard
n The problem is poorly defined.
n Systematic approach is not used to classify problem and cause.
n Investigations stopped prematurely – moving on to next problem.
n Decisions based on guesses, hunches or assumptions.
n Inadequate level of detail used to get to the Primary Effect.
n Interim containment fixes sometimes allowed to become
"permanent.”
n Skills, knowledge, and experience needed to uncover the root
cause not available.
n Lack of organizational will to address bigger issues.
n Fear of being blamed.
n I really don’t have time for this,we have bigger problems to solve.
18. Root Cause
867

+ The Problems with Categorical
Thinking
n We need to put order to the things we perceive.
n This is a natural process, but creates laziness in our thinking
processes.
n The notion of good and bad is categorical thinking at its base level.
n Categorical thinking creates the believe that once categorized, we
can establish relationships, and act on the according to other
perceived solutions.
n Filling out root cause forms or assigning elements to Fishbone
charts reinforces the perception we can put the rot causes in
categories (boxes) and assign solutions.
18. Root Cause
868

+ The Real Problem with Categorical
Thinking
n When interacting with others, we assume there is a single reality
and therefore their categories are like ours.
n They are not.
n We assign value that establishes our basis of understanding and
prejudices.
n If this is not recognized there is danger these prejudices set us up
for failure when trying to produce an effective solution to the root
cause.
18. Root Cause
869

+ Testing Answers from the Five
Whys Question Stream
n What evidence is there that this cause exists?
n Is it concrete?
n Is it measurable?
n What evidence is there that this cause could lead to the observed effects?
n Are we merely asserting causation without evidence?
n What evidence is there that this cause actually contributed to the Primary
Effect?
n Even given that it exists and could lead to this problem, how do we know it wasn't actually
something else?
n Is anything else needed, along with this cause, for the stated effect to occur?
n Is it self–sufficient?
n Is something needed to help it along?
n Can anything else, besides this cause, lead to the stated effect?
n Are there alternative explanations that fit better?
n What other risks are there?
18. Root Cause
870

To navigate the path to the actual
Root Cause, we need to connect
the Action and Condition
causes to the primary Effect and
all the Intermediate Effects in
a single picture revealing the
corrective actions that prevent
the Primary Effect in the future.
18. Root Cause
871

+
18.3 The Apollo
Method
Nothing happens without a cause.
Every time we ask WHY we must find
at least two causes – the Action and
the Condition in which that action
causes the effect.
18. Root Cause
872

+
n Causes are never part of a Linear Chain found in standard Fishbone
diagram or narrative approach.
n Look for causes to create the effect.Two causes are needed for each
Effect.
n Conditions – may exist prior to the Effect. Or conditions may be in motion or
active during the Effect. Conditions are the causes often ignored or beyond our
knowledge.
n Actions – momentary causes that bring conditions together to cause an effect.
Actions are causes most easily recognized.
n Connect all causes (Actions and Conditions) with a Caused By phrase to
either an action or a condition.
n Support each Cause with evidence or an answered question.
For each Primary Effect we need ask why that Effect
occurred.
Principles of the Cause and Effect
Map
18. Root Cause
873

+ Five Steps to using the Five Whys
n Invite all affected parties to
contribute to the map.
n Select the leader (someone
trained in RCA)
n Ask Why 5 times for each
topic area in the Cause and
Effect map.
n Assign responsibilities for
collected actual factual data.
n Publish the map.
Start this process using Apollo now.
18. Root Cause
874

+ The Apollo Principles
n Cause and Effect are the same thing
n If we look closely at cause and effect, we see that a “cause” and an
“effect” are the same thing.
n A single thing may be both a cause and an effect.
n They differ only by how we perceive them in time.
n Each effect has at least two causes in the form of actions and
conditions.
n This is the most important and overlooked principle of causation.
n Unlike storytelling used to capture the Fishbone style charts, which
focuses on linear action causes, reality demands that each effect have at
least one action cause and one or more conditional causes.
18. Root Cause
875

+ Cause and Effect are the Same
Thing
n The cause of one thing becomes the effect when you ask why.
n The cause of the “Injury” was a “Fall”, and when you ask why “Fall”,
it changes to an effect and the cause is “Slipped.”
n This relationship continues as long as we continue to ask why.
Effects Cause
Injury Caused by Fall
Fall Caused by Slipped
Slipped Caused by Wet Surface
Wet Surface Caused by Leaky Faucet
Leaky Faucet Caused By Seal Failure
Seal Failure Caused by Not Maintained
18. Root Cause
876

+ Each Effect Has At Least Two Causes in
the form of Actions and Conditions
n Primary Effect – is any effect
we want to prevent
n Action – momentary causes that
bring condition together to
cause an effect
n Conditions – the fundamental
causal element of all that
happens. It is made up of an
effect and its immediate causes
that represent a single causal
relationship.
n As a minimum, the causes in this
set consist of an action and one or
more conditions.
n Causal sets, like causes, cannot
exist alone.
n They are part of a continuum of
causes with no beginning or end,
which leads us to the next
principle:
n Causes and Effects are Part of
an Infinite Continuum of
Causes.
18. Root Cause
877

+ Causes and Effects are Part of an
Infinite Continuum of Causes
n Causes are not linear.
n They branch out into at
least two causes each
time we ask why of an
effect and if we ask why
of each of those causes
we find an ever
expanding set of causes
as shown in the example
to the right.
18. Root Cause
878

+ An Effect Exists Only if Its Causes Exist
in the Same Space and Time Frame
n Cause-and-effect
relationships exist with
or without the human
understanding
n We perceive them
relative to time and
space.
n Every causal
relationship is made up
of conditional causes
with a history of
existence over time
combining with an
action cause in some
defined time frame and
existing in the same
space to create an
effect.
18. Root Cause
879

+
18.4 Four Phases of
the Apollo
Method
1. Define the Problem
2. Create the Cause and Effect Chart
3. Identify effective solutions
4. Implement the best solutionsThese four phases are the basis of
discovering the corrective actions for
the undesirable Effects we see in our
development, testing, and
deployment efforts.
18. Root Cause
880

+
1. Define the problem
• What is the problem?
• When did it happen?
• Where did it happen?
• What is the significance of
the problem?
2. Create the Cause and Effect
chart
• For the primary Effect, ask
Why did this happen
• Look for causes in Actions
and Conditions
• Connect all the causes with
Caused By for the next
cause and its effect
• Support causes with
evidence or an open
Question
3. Identify effective solutions
must
• Prevent recurrence
• Be within our control
• Meet our goals and
objectives
4. Implement the best solutions
• Measure the effectiveness of
these solutions in units
defined in the Action and
Condition causes
Four Phases of the Apollo Method
18. Root Cause
881

+ No Fishbone Charts or Narratives
Allowed in the Apollo Method
18. Root Cause
882

+ The Flaw of our Linear Thinking
Process
n Like a string of dominos, asking why in the conventional Five Whys
method assumes, A caused B,B caused C,and C caused D.
n At the end of this chain we believe the Root Cause of the
undesirable outcome can be found.
n In the traditional Fishbone approach we are looking for the event
that caused the Effect.
n Instead we need to find the Actions and Conditions that ALLOWED
the event to happen.
n These Actions and Conditions are the actual Root Cause.
18. Root Cause
883

+ The Principles of the Apollo
Method
n Cause and Effect are the same thing.
n If we look closely at cause and effect, we see that a “cause” and an
“effect” are the same thing.
n A single thing may be both a cause and an effect.
n They differ only by how we perceive them in time.
n Each Effect has at least two causes in the form of Actions and
Conditions.
n This is the most important and overlooked principle of causation.
n Unlike storytelling used to capture the Fishbone style charts, which
focuses on linear action causes, reality demands that each effect have at
least one action cause and one or more conditional causes.
n Causes and Effects are part of a continuum of causes.
n An Effect exists only if its causes exist in the same space and time
frame.
18. Root Cause
884

+ Cause and Effect are the Same
Thing
n The cause of one thing becomes the effect when you connect
caused by.
n The cause of the “Injury” was a “Fall”, and when you ask why “Fall”,
it changes to an effect and the cause is “Slip.”
n This relationship continues as long as we continue to ask why.
Effect Caused by Action or Condition
Injury Caused by Fall
Fall Caused by Slip
Slip Caused by Wet Surface
Wet Surface Caused by Leaky Faucet
Leaky Faucet Caused By Seal Failure
Seal Failure Caused by Seal Not Maintained
18. Root Cause
885

+ An Effect Exists Only if Its Causes Exist
in the same Space and Time Frame
n Cause–and–effect
relationships exist with or
without the human
understanding.
n We perceive them relative
to time and space.
n Every causal relationship
is made up of Conditional
causes with a history of
existence over time,
combining with an Action
cause in some defined
time frame and existing in
the same space to create
an effect.
18. Root Cause
886

+ The Apollo Method structures our
current information collection method
to identify solutions to the Root Cause
Every time-series entry in our current
narrative method is an ACTION cause.
By focusing on ACTION’s and not
associated CONDITION causes, we
leave out important causes that can be
acted on to provide an effective
SOLUTION.
18. Root Cause
887

+ Example of a Naïve RCA
18. Root Cause
888
Why?
Action Cause?
Condition Cause?
Without following the Why’s to the terminal node with Action and Condition,
the actual Root Cause is still buried in the narrative, waiting to reoccur

+ Actual System Outage Root Cause
Analysis using Apollo Method
889
Each stopping condition points to the Root cause.
Until these are found only symptoms are
discovered, not the Root cause
18. Root Cause

+ Missing Elements of Success
n Linear thought process
n A caused B, B caused C, C caused D.
n No causal chain from Primary Effect to related Cause and Effect
n No Actions and Conditions to connect to the Effect
n Under what conditions was the Effect observed?
n What actions triggered the Effect?
n No evidence for each Action and Condition
n Stopping too soon, before actual Root Cause found
Every Effect is caused by momentary Action coming
together with existing Condition in the same time
and space.
18. Root Cause
890

+
18.5 Seven Steps to
Discovery
1. Define the Problem.
2. Determine the Causal Relationships.
3. Provide a Graphical Representation of
Cause and Effect that is not linear
thinking.
4. Provide Evidence for each Cause and
Effect.
5. Determine if each Cause is Sufficient and
Necessary.
6. Identify Effective Solutions THIS IS
WHAT WE’RE AFTER.
§ Finding the cause is needed.
§ Preventing the effect is needed.
§ But installing an effective solution is
the desired business outcome.
7. Implement And Track the Effective
Solutions.
Effective problem solving and
Strategies for business success that
move away from blame finding and
linear thinking of Fish Bone diagrams.
And move toward finding the
interconnected factors where Cause
and Effect are intertwined.
These Seven Steps expand on the
Four phases, to further detail the
process of arriving at the Root Cause
18. Root Cause
891

+ Seven Steps of the Apollo Method
1. Define the Problem.
2. Determine the Causal Relationships.
3. Provide a Graphical Representation of Cause and Effect that is not
linear thinking.
4. Provide Evidence for each Cause and Effect.
5. Determine if each Cause is Sufficient and Necessary.
6. Identify Effective Solutions
§ Finding the cause is needed.
§ Preventing the effect is needed.
§ But installing an effective solution is the desired business outcome.
7. Implement And Track the Effective Solutions.
18. Root Cause
892
This is what we’re after in Root Cause Analysis

+ An Effective Solution …
n Prevents recurrence of the Primary Effect.
n Assures corrections and prevents actions within our control.
n Meets our goals and objectives, including a solution that …
n Does not cause unacceptable problems.
n Prevents similar occurrences.
n Provides reasonable value for the cost.
18. Root Cause
893

+
A Recent Example of Cause and Effect Analysis
18. Root Cause
894

+
18.6 An Ugly Truth
About Root
Causes
The truth?
You can’t handle the
Truth!
18. Root Cause
895

+ In the end it usually comes down to the
People.The People are the Root Cause
This does NOT and CAN NOT mean a Blame Game.
If we ask properly people will see their role in the failure of the process.
Human beings cause problems, not tools, processes, systems, or culture.
18. Root Cause
896

+ This is our core problem†
n Getting the staff engaged in Preventing problems not just Correcting
problems means
n If I do this what will happen?
n Is this the right thing to do at this time?
n Is this really what the customer wanted me to do?
n Did I consider the impact of my actions?
n If I don’t have time to be careful,then what damage will result from my
actions?
n Did I consult with someone who knows more than I do about what the
solution should look like?
n Am I being as careful as I should be,when I make a change?
n Am I being pressured to do something I know isn’t the right thing to do?
If the people change, the culture will change
† IEEE 13th Annual Workshop on Human Performance / Root Cause / Trending / Operating Experience / Self Assessment, August 26-31,
2007, Monterey Marriott, Monterey, CA
18. Root Cause
897

+
18.7 Effective
Problem
Solving
Culture
n Critical Elements
n Infrastructure
n Management Buy-In and Support
n Crating a Program Champion
n Dedicated Incident Investigators
n Integration
n Implementation Strategies
n Deployment
Every organization is different, so
before implementing this plan you
should determine who the players
are and what level of training they
need.
http://www.realitycharting.com/training/problemsolvingculture/plan
18. Root Cause
898

+
n Exposed to the principles of causation to understand that “stuff” does
not just happen.
n Should know that we can find effective solutions to event-based
problems by using RealityCharting® and RC Simplified™.
n Must understand that different perspectives are a key to effective
solutions and easily accommodated when you use the Reality
Charting process.
n Must know their role in defining problems and finding effective
solutions to prevent recurrence.
n Must know that management is behind this initiative.
The Work Force should
Critical Elements
18. Root Cause
899

+
n Top-level management support.
n A Program Champion.
n Dedicated Incident Investigation Facilitators.
n Incorporation of the Reality Charting process into existing
procedures and protocol.
n Involve every employee in this effective problem-solving initiative.
n Utilize RealityCharting®, the RC Learning Center™, and RC
Simplified™ throughout the organization.
The Plan musty implement
Infrastructure
18. Root Cause
900

+
n Show the Effective Problem-Solving processes to top-level managers
so they know the principles of effective problem-solving.
n Show RealityCharting® Overview processes to all managers so they
know what the software does and why it is so effective.
n Read the RealityCharting – Seven Steps to Effective Problem Solving
and Personal Success and work the interactive exercises in this book.
n Provide a manager’s workshop on Managing Effective Problem-
Solving. Discuss application to Defect Elimination, Continuous
Improvement programs
n Six Sigma, Lean, Proactive Maintenance, Chronic (Systemic) issues.
Top level support of the RCA process is done
Management Buy-In and Support
18. Root Cause
901

+
n Qualifications
n An experienced incident investigator and facilitator.
n Training
n Familiar with RCA concepts
n Familiar with Apollo method
n User of reality Charting®
n Role
n The Go To persons for all things RCA
n Effectively facilitate incident investigations
The Program Champion is the Cheer Leader for the RCA
Improvement initiative
Create a Program Champion
18. Root Cause
902

+ Dedicated Incident Investigator
n Identify stakeholders ‒ who have to solve problems as part of their
daily work scope.
n System or Process Engineers
n Supervisors or Team Leaders
n Some Managers
n Training ‒ online training takes ½ the time classroom training does
n Mentoring ‒ program champion and investigators mentor
practitioners and support them to develop skills for conducting Root
Cause Analyses
18. Root Cause
903

+ Integration
n Entry Points ‒ Identify problem-solving entry points by reviewing
the work processes.
n Problems can occur anywhere and evaluation includes what the actions of
any employee should be if they identify a problem.
n This may include Non-Conformance Reports, Corrective Action Reports,
and Customer Feedback Forms.
n Minor Problems ‒ If problem is below the threshold criteria for
performing a full-blown RCA, use Simplified process to document
the problem.
18. Root Cause
904

+ Implementation Strategy
n Initial Findings ‒ If the initial problem analysis provides effective
solutions, implement them according to existing procedures and
approval protocol.
n Send the final analysis to the Program Champion for review and approval and
put into the organizations tracking and trending system and/or print, or
otherwise transmit the final copy to stakeholders who may need it.
n The Program Champion will review to determine if the problem exceeds the
threshold criteria for a formal investigation and respond accordingly.
n This review provides the Program Champion oversight of how well the
program is working and when and where to adjust it.
n Incomplete Analysis ‒ If the event-analysis does not find effective
solutions, then send the initial analysis to Program Champion
n The Program Champion will determine if the problem exceeds the threshold
criteria for a formal investigation and respond accordingly.
n The Program Champion determines if analysis can be finished or if a team is
needed to work the issue.
n If a team is required, Program Champion will gather team and work with them
to finalize the analysis.
18. Root Cause
905

+ Deployment
18. Root Cause
906

+
18.7 Root Cause
Analysis in
Agile Software
Development
n Do just enough Root Cause Analysis
sessions
n Have a knowledgeable facilitator
n Communicate the corrective actions
n Do Agile Root Cause Analysis
Root Cause Analysis is used in
software development to build a
shared understanding of a problem
to determine the first or “root”
causes.
Knowing these causes identifies
effective improvement actions to
prevent similar problems in the
future.
Root Cause Analysis in Agile stops
problems that have been inhibiting
the team for too long.
18. Root Cause
907

+ Do Just Enough RCA
n For any problem to be investigated, the loss or potential for loss
must be significant to the business or the project
n The definition of significant is held by those paying never by those
providing the solution
n The loss needs to monetized or defined in some unit of measure
meaningful to the decision makers
n There must exist a significant probability that similar problems will
occur in the future if no corrective action is taken
18. Root Cause
908

+ Knowledgeable Facilitator
n The team itself cannot produce a credible Root Cause Analysis
n Facilitation of the process is needed by a separate party
n This can be:
n Quality Assurance
n Systems Architecture
n Program Management
n Release Management
n The facilitator must follow a process ‒ the Apollo Method is one
shown here
n Tools are needed to capture the information and avoid the narrative
biases also shown here
18. Root Cause
909

+ Communicate the Corrective
Actions
n Finding the Root Cause is just the start.
n Taking corrective actions is next, but that starts with communicating
to all stakeholders what those actions are.
n The communication must show
n The problem
n The Root Cause
n The Corrective Actions
n The expected outcomes in quantifiable measures of effectiveness and
performance
18. Root Cause
910

Root Causes Have Root Causes
It’s not a matter of asking 5 Whys
You must keep asking until you reach the end of the chain or action and condition
18. Root Cause

“Kto Nie Pamięta
Historie Skazany
Jest Na Jej Ponowne
Preżycie” ‒ George
Santayana
“Those Who Do Not
Remember the Past
are Condemned to
Repeat it” ‒ George
Santayana
18. Root Cause

+ Root Cause
Analysis
n Events have two contributors ‒ the condition and the action. Both must be found
before root cause can be determined for the primary effect.
n There is an endless chain of cause and effect, stopping too early is a common
failure mode of Root Cause Analysis
n Formal Root Cause Analysis processes and tools provide information not
available with the narrative approach
n Without determining the Root Cause and suggested solution just address the
symptoms.This approach does not remove the cause and allows the symptoms
to recur in a loop of fix, break, fix.
913
18. Root Cause

Root cause analysis master plan

More Related Content

What's hot

Viewers also liked

Similar to Root cause analysis master plan

More from Glen Alleman

Recently uploaded

Root cause analysis master plan