1. Root Cause Analysis in the
Department of Defense
February 10th, 2011
David J. Nicholls
David.Nicholls@osd.mil
0
2. Acronyms
SME – subject matter expert
SEP – systems engineering plan
DAES – Defense Acquisition Executive Summary
PARCA – Performance Assessments and Root Cause
Analyses
WSARA – Weapon System Acquisition Reform Act
MGB – Milestone B
1
3. PARCA Introduction
PARCA’s institutional role
– Provides USD(AT&L) execution-phase situational
awareness of major defense acquisition programs
– Performs forensics for troubled programs
– Reports activities annually to the four defense committees
Three functions of PARCA office
– “Performance Assessment”; function in statute
– “Root Cause Analysis”; function in statute
– “EVMS”; function not in statute
2
4. PARCA Does NOT . . .
Forecast program requirements
– Funding requirements
– Total Life Cycle Cost
Evaluate alternative means to execute
– Acquisition strategies
– Contracting terms / incentives
Compare alternative means to achieve an end
– Evaluations of alternate approaches
– Cost-effectiveness
Necessary for independence
3
6. Root Cause Analysis Functions
Statutory duties defined in WSARA 09
– Conduct root cause analyses for major defense
acquisition programs
As part of the Nunn-McCurdy breach certification process
When requested by designated officials.
– Issue policies, procedures, and guidance governing
the conduct of root cause analyses.
Identification of lessons learned for the
benefit of acquisition community
5
7. Analytical Framework
Unrealistic
performance
expectations
Unrealistic cost M
a
or schedule Inadequate risk n
estimates assessment a Cost,
g schedule and
Any other e
matters Unanticipated m performance
e impact
technological or n
manufacturing issues t
Quantity
change Funding instability
or inadequacy
Problems will occur: why they occur and our response
are crucial issues for root cause analysis
6
8. RCA Operations
PARCA deliberately Analytical Steps
limited in organic
Initial Investigation
resources
Depends on other Fact Finding
OSD organizations for
Analysis
functional expertise
Depends on FFRDCs Products
and academia for
analytical support
7
9. Initial Investigation
Define the cost and/or
schedule growth
Gather documentation
– Program content and
guidance
– Acquisition documents
– General background
Program chronology
Change analysis
Prepare for fact finding
Define the problem
8
10. Fact Finding
Interviews
– Program Office and Contractors
– Organizations providing functional
support to program
Analysis of performance
reporting data
– EVMS
– Cost reporting
– Performance metrics
Program documentation,
studies, reports
Comprehend perspectives
9
11. Analysis
Exact analytical methods
are case dependent
Blend of quantitative and
Product
qualitative (but fact based)
methodologies
Programmatic
Decision-maker concerns
– Actionability
Acquisition System
– Exogenous vs endogenous
– Effectiveness
Develop and evaluate hypotheses
10
12. Products
One-page memorandum for Nunn-McCurdy
certification process
– Describes breach and its root causes
– Supporting document in the program certification package
sent to Congress
– Briefing summarizing results
Lessons for acquisition community
– Reports
– Analytical tools
Contribution for PARCA’s annual report to
Congress
11
14. Root Cause Analyses in 2010
Performed six root cause analyses as
part of Nunn-McCurdy breach process
– F-35 – RMS
– DDG-1000 – Apache Block III
– ATIRCM/CMWS – WGS
Seven root cause analyses are ongoing
– Some “on request”
– Variety of programs
Expect about 20 root cause analyses per year
13
15. PreliminaryTrends
Inception Issues
Unrealistic cost or schedule estimates X X X X X X
Immature technology, excessive
manufacturing, integration risk X
Unrealistic performance expectations X
Execution issues
Changes in procurement quantity X X X X X X
Inadequate funding/ funding instability
Unanticipated design, engineering,
manufacturing or technology issues X X X
Other X X
Poor Performance X X X X
14
16. Observations
Inception Issues
Unrealistic cost or schedule estimates X X X X X X
Immature technology, excessive
manufacturing, integration risk X
Unrealistic performance expectations X
Most likely problem “to be discovered” by program
is an unrealistic cost or schedule estimate
Other factors are apparently not common or
perhaps need better tools to detect
Problems to be discovered in execution
15
17. Observations
Execution issues
Changes in procurement quantity X X X X X X
Inadequate funding/ funding instability
Unanticipated design, engineering,
manufacturing or technology issues X X X
Other X X
Quantity changes a common factor but usually
accompanied by (or caused by) other factors
Programs likely to encounter the unexpected
Suprisingly, funding has not been a major factor
Quantity change as a cause must be questioned
16
18. Assessing Management
An important evaluation
– Assessed to be a significant factor in
several cases
– Area for “actionable” root causes M
a
A difficult area to evaluate n
a
– Quantitative impact of poor mgt? g Cost, and
e schedule
– Likely contentious m
impact
e
Approach is to evaluate pieces n
t
– Defining system and interfaces
– Appropriate contracting strategy
– Maintaining situational awareness
Key evaluation that includes contractor, PMO, and governance
17
19. Defining Systems & Interfaces
Complex process which demands SME input
Poorly developed or defined Insufficient or incomplete SEP
requirements Lack of a well developed Integrated
Undefined dependencies Master Schedule (IMS)
Insufficient interoperability Program Office defaults
planning control/responsibility to contractor
Requirements instability Reliance on schedule over
Lack of stakeholder involvement performance
Poorly developed or under Poor integration test planning
specified MS exit/entrance criteria Lack of integrated planning
Two key questions
– Does Government know what it wants/able to define it ?
– Is schedule based upon engineering realities?
18
20. Effective Contracting Strategy
Two elements
– Blocking and tackling
– Incentive strategy effectiveness
Incentive evaluation
– Aligned with program goals
and challenges
– Demanding yet achievable
– Sufficient to motivate
– No perverse effects
– Consistent with overall
incentives
– Signal sent and received
19
21. Situational Awareness
Tailored metrics
– Relevant metrics e.g. identification AND tracking of “bets”
– Effective metrics e.g. avoiding integration effects
Earned Value Management
– Compliance
– Effectiveness PA Situational
awareness
Integrated with PARCA’s
program assessments Tailored RCA
– Semi-annual, statutory reviews after concerns
critical Nunn-McCurdy breach
– As part of DAES process
20
23. Ongoing Development
Analytical methodologies
– Impact of funding constraints in
development
– Evaluation of incentive strategies
– Systems engineering assessment
– EV metrics and data presentation
Relationships
– SME sources for specific issues
– Knowledge-sharing of analytical approaches e.g. UK MoD
– Effective use of root cause analysis results
– Effective mechanisms for sharing lessons for benefit of
acquisition community
22
24. Policies, Procedures,
and Guidance
State of play
–WSARA 09 requires PARCA to address
– Virtually no current guidance for root cause analysis
– Other organizations have done some root cause analysis
– Key questions exist e.g. is it enough to say that estimate was
unrealistic (WSARA 09 category) or should we ask why?
Three different purposes
– Documentation of PARCA processes
– Service conduct of root cause analyses
– Program Office conduct of root cause analyses
Documentation of organizational lessons
23
