1. Quantitative Schedule Risk Analysis
of DoD Operational Environment
Captain James ‘Spool’ Nichols
Justin Hornback
Kelly Moses
NASA PM Challenge
Risk, Safety and Mission Assurance Track
February 9 & 10, 2010
Used with Permission
2. The Naval Aviation Enterprise (NAE) Carrier Readiness Team (CRT)
required a holistic view for assessing strategic plans for future
aircraft carrier availability
Problem: The NAE CRT required a holistic understanding of the risks associated with the looming aircraft
carrier availability gap and how best to handle these risks.
– Reduction in number of aircraft carriers from 11 to 10, planned retirement of USS Enterprise in 2012
– Demand for aircraft carrier deployment unchanged
Several important questions needed to be addressed:
– How does risk impact aircraft carrier operational availability (Ao)?
– What are the cost and schedule impacts of risk?
– How should mitigation dollars be prioritized against high-impact risks?
– Is historical data useful for future planning?
– How does one carrier in the enterprise (portfolio) impact the others?
Previous attempts to address this problem were largely qualitative in nature and lacked a rigorous analytical
framework and incorporated no uncertainty or risk.
Applied the operational RISC-IQ™ methodology to this problem to address these challenges.
Constrained Resources and Emphasis on Efficiency Makes
Understanding Risks Essential to Carrying Out Strategic Objectives
1
3. Over the next 10-15 years, naval aviation faces unparalleled risks
and challenges associated with planning to ensure a continued
projection of power
Constrained/unpredictable budgets
Dynamic scheduling and events
Increasing age of existing F/A-18 aircraft
Potential F/A-18 service life extensions
Composition of squadrons within an Air Wing
Type/model/series transitions
– E/A-18G
– H-60R and H-60S
New program startups/delays
– JSF
– E-2D
– Extensive training and re-training requirements
Linchpin to Resourcing Combatant Commander Demand is CVN
Availability
2
4. The Schedule Risk Analysis Methodology utilized has a strong
foundation in acquisition project risk analysis
Risk Identification
Historical Data Analyses
Foundation
Process
Stakeholder Collaboration
Risk Mapping
Risk Modeling
Analytics
Translation to Availability Metrics
Sensitivity Analyses
Results
Recommendations
3
5. Each step of the NAE Risk Methodology can be further broken
down and explained in more detail
Risk ID &
Foundational
Aircraft Carrier
Operations
Efforts
(Initial Data Gathering &
Formulation; Risks Identified
from a Variety of Sources)
Historical Data
Analyses
(Data Gathered,
Trends & Outliers Analyzed)
Definitions Definitions
Constraints Stakeholder
Risk Modeling Constraints
Survey Results
Objectives
Assumptions Stakeholder
Risk Modeling
Collaboration Objectives
Assumptions
Survey
(Key Inputs Discussed & 1 2 3 4
Reviewed)
DRAFT FINAL
Cost Schedule Cost Schedule
Risk Risk Mapping
Risk A
(Discrete Risks Mapped to Cost Risk B
Elements & Schedule Tasks)
4
6. NAE Risk Methodology (cont’d)
Cost Risk Schedule
600 Risk 1
FY08$M
500 0.8
400
Modeling
Cumulative Probability
300 0.6
200
100 0.4
0
2009 2011 2013 2015
2010 2012 2014 2016 (Simulations Run to Evaluate 0.2
Risk Impacts on Cost & 0
34 32 30 28 26 24 22 20
Schedule) -0.2
Months Meeting Availability Objective
Translation to Ao Metrics
Availability Metrics
(Simulation Results Compiled,
Aggregate Ao Metrics Calculated)
Sensitivity Analyses
Risk Modeling
Risk Modeling
(Simulation Data Analyzed,
Excursions Run, “What If” Cases Cost/
Developed) Schedule
Drivers
Recommendations Strategic Focus Areas:
- A, B, C Maintenance
Schedules During Months D,
E, F -
(Results Compiled & System Dependency G
Recommendations Formulated) - Bottleneck Point H
- Costs of I, J, K
- Stakeholder L
5
7. Defining the Metric(s) to be assessed proved to be a very
important element to the success of the analysis
The underlying process is that of quantitative schedule risk analysis.
However, when analyzing an operational environment and/or a portfolio of systems, it is not
always a simple schedule slip that best communicates to the client the risk posture of the
project or program.
Developing the appropriate metrics is critical.
The FRP notional 32 month cycle rules depicted visually below, were applied to the
development of and incorporation of 90 day asset events and 30 day asset events into the
MS Project model.
90 Day Asset
30 Day Asset
6
8. Risk Modeling Methodology Step One: Determine the
probability of any one month attaining metric
Calculate probability of schedule slip for each program/system per each time period
being measured.
Operating rule was based on 15th of the month
Across portfolio of carriers, probability of 30 or 90 day asset was assessed
For the example below:
– P (x ≤ July 15, 2014) = 3%
– P (x ≤ Aug 15, 2014) = 42%
– P (x ≤ Sept 15, 2014) = 79%
– P (x ≤ Oct 15, 2014) = 96%
– P (x ≤ Nov 15, 2014) = 97%
– P (x ≤ Dec 15, 2014) = 99%
7
9. Risk Modeling Methodology Step Two: Schedules were overlaid
upon another to create the “S” curve from the Monte Carlo
simulations Likelihood (Percentage)
Months achieving desired
readiness level (48 max)
=
Multiple Scenario generation
Slope of S curve relates the amount
of risk associated with a scenario
8
10. Risk Modeling Methodology Step Three: Assess combined
probability of each month, one at a time
Translate to desired metric (in this case, Operational Availability) and combine probabilities for each
time period for the portfolio of systems
An assessment of each of the 48 months during the gap is conducted as follows:
– Review all events that have probabilistic curves for slips in their end date
– Taking a particular month as an example (see snapshot to the right):
– The baseline case is 5+1
– There are two events with potential slips during this month
– Event A can also slip from a 30 day asset to a 90 day asset with a probability of 4% (96%
chance of not slipping)
– Event B can slip from a 30 day asset to a 90 day asset with a probability of 65% (35% chance
of not slipping)
– Using the theorem of total probability and the Venn Diagram below, the resulting scenarios
and associated probabilities for one month are shown to the right
– NOTE: There APPEARS to be a slight rounding error in this example: .34+.64+.03 = 1.01 without
looking at three or four significant digits
A = 96%
A=
B = 65%
4%
B = 35%
9
11. Risk Modeling Methodology Step Four: Development of S curve
Determine how many months have a probability of not achieving the desired metric and record those
probabilities
This analysis is completed for each of the 48 months in the gap
For the metric under consideration, in this case, 6+X, the number of months with a probability less than
100% (eight months below) of achieving 6+x are assessed and tabulated as follows:
Month Probability
The next step again involves the theorem of total probability combined with combinations and
permutations. Order for this particular instance does not matter since we are not concerned with how
many months meeting or not meeting Ao are of interest
10
12. Risk Modeling Methodology Step Five: Exact calculation of S
curve
Probability of occurrence of each
individual scenario of exactly a certain
number of months meeting the desired
metric
Equal to 1.00 minus the sum of the other
calculated probabilities.
Probability of occurrence of each month
that could fall below 6+x probabilistically
Cumulative Distribution Function “S-
Curve” raw data
Compliment probability: Probability of
nonoccurrence of each month that could
fall below 6+x probabilistically
11
13. Four sensitivity analyses were performed after the baseline case
was developed and communicated to the client
Alternative 1: Compression of Basic Phase Training
– Aggressive reduction in training during basic phase
– Maximum gain of 6+x months is 2. Probability of 2 month gain is less than 5%
Alternative 2: Compression of Basic Phase Training and White Space
– Aggressive reduction in training and white space during basic phase
– Maximum gain of 6+x months is 5 months. Probability of 5 month gain is less than 1%
Alternative 3: Extension of Planned Incremental Availability (PIA) Basic Phase
by 3 weeks
– Three month decrease in Ao prior to risk analysis
– There are months during the gap years where basic phase can be extended with little to no
impact on Ao either deterministically or risk-based
Alternative 4: Extension of Basic Phase for All Maintenance Events by 10%
– One month decrease in Ao prior to risk analysis
– It is possible extension of basic phase in appropriate months will alleviate risk of aggressive
Strike Fighter Squadron (VFA) transition timelines (allows for slack in the schedule)
12
15. Key Challenges
Client’s first application of this methodology
Politically charged environment
– Potential conflict with previous Congressional Briefings
Diverse stakeholders
– World-wide implications
Initial resistance due to client’s unfamiliarity with the process
– Report Card of performance
Client used to stand-alone (i.e., stoplight chart) risk management
that did not reveal the range of potential outcomes
– Optimistic Schedules
Complex environment with intricate interdependencies
– Cyclical critical path
– Resource constraints, industrial complex
14
16. Fall 2009 Aircraft Carrier Maintenance Stack-Up used as a test of
the RISC-IQ™ methodology
September 2009, four carriers in carrier maintenance at Northrop Grumman Shipbuilding, Newport News
(NGSB NN) creating work capacity risks across all four carriers.
– 36% of US aircraft carrier fleet
– CVN 65 USS Enterprise
– CVN 70 USS Carl Vinson
– CVN 71 USS Theodore Roosevelt
– CVN 77 USS George H. W. Bush
2004 CV/CVN Maintenance Availability Schedule projected 1 carrier at NGSB – September 2009
– CVN 65 EDSRA
Risk Analysis of 2004 CV/CVN Maintenance Schedule projects potential of 3 carriers - September 2009
– CVN 65 EDSRA
– CVN 70 PSA/SRA (50%)
– CVN 77 PSA/SRA (80%)
(% likelihood/potential of CVN maintenance event occurring at NGSB NN - September 2009)
CVN 71 RCOH was not projected due to the 2004 scheduled RCOH start date of 11/2009. This date was
moved up to 9/2/2009 during CVN 71 RCOH planning in 2007.
15
17. Resulting Recommendations
Mitigation strategy and associated resources should focus on identified brittle schedule
months and identified high risk maintenance events/carrier cycles rather than be
implemented in a sweeping fashion across the entire gap to enhance cost and schedule
effectiveness
Attention to high likelihood of consecutive Ao = 4+x should be increased
Investigate potential to gain Ao through specific maintenance schedule adjustments
Consider extending Basic Phase in select months where there is no impact to Ao >6+x
Consider use of COCOM presence vice Ao as a readiness metric
Continue communication/coordination enabling risk-based decisions across the Naval
Aviation Enterprise
16
18. Applying a risk analysis provided information required to make
informed decisions
This process was beneficial in that it:
– Quantified the intuition of industry stakeholders
– Specified availability of assets
– Provided a forum for discussion and gathering subject matter expert/stakeholder input
– Uncovered historical data, analyzed trends, and used the data to validate outputs
– Highlighted complex interdependencies and constraints with carrier operations and maintenance across
the enterprise
– Examined the root cause of schedule divergences
– Allowed the client to build confidence in their ‘go forward’ plan
– Built a foundation from which further analyses can be conducted
– Expansion outside of Aircraft Carriers
– Generated a portfolio of risk models
– Each risk model represented a unique compilation of data
which created…
Quantifiable and Defensible Results
17
