Patterson.dan
- 1. Using the Risk Register in Integrated
Cost/Schedule Risk Analysis
with Monte Carlo Simulation
David T. Hulett, Ph.D.
Hulett & Associates, LLC
Dr. Dan Patterson, PMP
Acumen
NASA PM Challenge
Daytona Beach, FL
February 24-25, 2009
© 2009 Hulett & Associates,
AcumenPM
- 2. Agenda
• Explain “Risk Factors” approach
• Apply Risk Factors to schedule and cost risk
• Apply Risk Factors to simple space vehicle
development schedule as an example
© 2009 Hulett & Associates,
AcumenPM
- 3. Limitations with the Traditional
3-point Estimate of Activity Duration
• Typical schedule risk analysis starts with the
activity that is impacted by risks
– Estimates the 3-point estimate for optimistic, most
likely and pessimistic duration
– Implies the risk is 100% likely with uncertain impact
• Which risks cause the most overall schedule
risk? Cannot say directly, but indirectly:
– Sensitivity to activity durations
– Criticality of activity durations
© 2009 Hulett & Associates,
AcumenPM
- 4. Some Problems with Traditional
Approach
• Can tell which activities are crucial, but not
directly which risks are driving
• Makes poor use of the Risk Register that
is usually available
• Cannot decompose the overall schedule
risk into its components BY RISK
– Ability to assign the risk to its specific risk
drivers helps with communication of risk
causes and risk mitigation
© 2009 Hulett & Associates,
AcumenPM
- 5. We Propose a Different Approach:
Start with the Risks Themselves
• Drive the schedule risk by the risks already
prioritized in the Risk Register
• For each risk, specify:
– Probability it will occur (NEW)
– Impact on time if it does (multiplicative factors, allows it to
affect different duration activities)
– Assign to the activities it will affect
• Starting with the risks themselves gives us benefits
– Links qualitative analysis to the quantitative analysis
– Estimates the impact of specific risks for prioritized
mitigation purposes
– Correlations between activities modeled automatically
© 2009 Hulett & Associates,
AcumenPM
- 6. Simple Example of Risk Register Risks
• Use the Risk Factors module in Pertmaster 8
• Collect probability and impact data on risks
• Map the risks to activities
© 2009 Hulett & Associates,
AcumenPM
- 7. Risk Factors Mechanics (1)
• The risk factor is assigned to one or several
activities, affecting their durations by a
multiplicative factor
– E.g., the factor may be .90 for optimistic, 1.0 for most
likely and 1.25 for pessimistic
– These factors multiply the schedule durations of the
activities to which they are assigned
• Risks can be assigned to one or more activities
• Activity durations can be influenced by one or
more risks
© 2009 Hulett & Associates,
AcumenPM
- 8. Risk Factors Mechanics (2)
• Risk Factors are assigned a probability of
occurring on any iteration
– When the risk occurs, the factor used is chosen at
random from the 3-point estimate and operates on all
activities to which it is assigned
– When not occurring on an iteration the risk factor
takes the value 1.0, a neutral value
• When an activity is influenced by more than one
risk, their factors are multiplied together, if they
happen, on any iteration
© 2009 Hulett & Associates,
AcumenPM
- 9. Risk Factor
Probability is 100%, Factor can be + or -
0 0 1 0 - C o n s tr u c tio n : D u r a tio n
1 0 0% 11 5
2 50
9 5 % 1 11
Here the 9 0 % 1 09
8 5 % 1 08
For the
Ranges are 2 00
8 0 % 1 06
7 5 % 1 05
examples we
based on 7 0 % 1 04 use an activity
6 5 % 1 04
deviations + with 100 days
Cumulative Frequency
6 0 % 1 03
1 50
and – from 5 5 % 1 02
in the
Hits
5 0 % 1 01
the Plan. 4 5 % 1 01
4 0 % 1 00
schedule
1 00
Probability 35% 99
30% 99
is 100% 25% 98
50 20% 97
15% 96
10% 95
5 % 94
0 0 % 90
90 95 10 0 1 05 1 10 11 5
D istrib u tio n (sta rt o f in te rva l)
© 2009 Hulett & Associates,
AcumenPM
- 10. Assigning a Probability Less than 100%
0010 - Construction : Duration 0040 - Technology Design : Duration
100% 115 100% 130
Spike 95% 107 95% 123
contains
90% 103 1200 90% 120
Spike
85% 101 85% 118
70% of
2000
80% 100 80% 116 contains
the
75% 100
70% 100
1000 75% 114
70% 113
40% of
probability
65% 100 65% 111
the
Cumulative Frequency
Cumulative Frequency
1500
60% 100 800 60% 110
55% 100 55% 109 probability
Hits
Hits
50% 100 50% 107
45% 100 600 45% 105
1000 40% 100
40% 100
35% 100 35% 100
30% 100 400 30% 100
25% 100 25% 100
500 20% 100 20% 100
15% 100 200 15% 100
10% 99 10% 100
5% 97 5% 100
0 0% 100
0 0% 91
100 110 120 130
95 100 105 110 115
Distribution (start of interval) Distribution (start of interval)
© 2009 Hulett & Associates,
AcumenPM
- 11. Assigning
More than One Risk to an Activity
• If more than one risk is acting on an
activity, the resulting ranges are the
multiplication of the factors
• The activity duration range is derived from
the risk factors that affect the risk
– Model how the activity duration range is
generated
– Focus on the causes of activity duration
ranges
© 2009 Hulett & Associates,
AcumenPM
- 12. Two Risks affect One Activity
using Factors that Occur 100%
0040 - T echnology Design : Duration
100% 144
95% 130
90% 127
140
85% 124
80% 123
120 75% 121
70% 119
Range 100
65% 118
Cumulative Frequency
60% 117
from 90 to 55% 116
Hits
150 days, 80 50% 115
45% 114
Peak about 60
40% 113
35% 111
113 days 30% 110
40 25% 109
20% 108
15% 106
20
10% 104
5% 102
0 0% 93
100 110 120 130 140
Distribution (sta rt of inte rva l)
© 2009 Hulett & Associates,
AcumenPM
- 13. Two Risks with Less than 100%
Probability Affecting one Activity
0040 - T echnology Design : Duration
1100 100% 144
95% 123
The spike at 1000 90% 119
85% 116
100 days 900
80% 113
75% 111
represents (1) 800
70% 110
the likelihood 700
65% 108
Cumulative Frequency
60% 106
that neither risk 600 55% 104
Hits
50% 102
occurs and (2) 500 45% 101
the chance that 400
40% 100
35% 100
100 days is 300
30% 100
25% 100
picked when 200
20% 100
one or both 100
15% 100
10% 99
occur 5% 97
0 0% 91
100 110 120 130 140
Distribution (sta rt of inte rva l)
© 2009 Hulett & Associates,
AcumenPM
- 14. Risk Factors Model How Correlation Occurs
Coefficients are Calculated (1)
Risk #1
P = 50%, Factors
.95, 1.05, 1.15
Activity A Activity B
Activities A and B
Correlation is Calculated
to be 100%
© 2009 Hulett & Associates,
AcumenPM
- 15. Risk Factors Model How Correlation Occurs
Coefficients are Calculated (2)
Risk #2 Risk #1 Risk #3
P = 25%, Factors P = 50%, Factors P = 45%, Factors
.8, .95, 1.05 .95, 1.05, 1.15 1.0, 1.1, 1.2
Activity A Activity B
Activities A and B
Correlation is Calculated
to be 48%
Correlation is modeled as it is caused in the project
Correlation coefficients are generated, not guessed
© 2009 Hulett & Associates,
AcumenPM
- 16. Sensitivity to the Risk Factors
The tornado
diagram
focuses on
risks, not
activities
© 2009 Hulett & Associates,
AcumenPM
- 17. Simple 2-Stage Space Vehicle Schedule
Software used: Pertmaster v. 8
© 2009 Hulett & Associates,
AcumenPM
- 18. Case Study: Simple Space Vehicle
Development Schedule
• 87 month schedule
• 11 work activities linked, 3 major milestones
• Beginning 3 March 2008
• PDR on 11 September 2009
• CDR on 3 June 2011
• Delivery to launch site 12 June 2015
© 2009 Hulett & Associates,
AcumenPM
- 19. Two Types of Risk
• Background risk based on typical general
risk, estimating error
– Used Quick Risk of -5% and +10%
• Discrete risks derived from Risk Register
– Summarized from detailed Risk Register
– These have a probability of occurring and an
impact on specific activities if they do
– Parallel to their Risk Register information
– Risk Register is used in data collection
© 2009 Hulett & Associates,
AcumenPM
- 20. Standard 3-point Range Representing
Schedule Estimating Error
Background
risk:
Optimistic -5%
Pessimistic
+10%
© 2009 Hulett & Associates,
AcumenPM
- 21. Results with
Schedule Estimating Error Only
Spacecraft Program
Entire Plan : Finish Date
100% 30/Dec/15
95% 27/Oct/15
400
90% 15/Oct/15
85% 07/Oct/15
80% 30/Sep/15
Deterministic:
75% 24/Sep/15 12JUN15 is <1%
300 70% 18/Sep/15
Cumulative Frequency
65% 14/Sep/15
60% 09/Sep/15
P-80 is 30SEP15,
55% 04/Sep/15 about 3.5 months
Hits
200
50% 01/Sep/15
45% 26/Aug/15
later than planned
40% 21/Aug/15
35% 18/Aug/15 Spread from P-5 to
30% 12/Aug/15
P-95 is 5JUL15 to
25% 07/Aug/15
100
20% 31/Jul/15 27OCT15 for 3.7
15% 24/Jul/15
10% 17/Jul/15
months
5% 06/Jul/15
0 0% 14/May/15
31/May/15 08/Sep/15 17/Dec/15
Distribution (start of interval)
© 2009 Hulett & Associates,
AcumenPM
- 22. Risk Analysis on Space Vehicle Project
Risk Factors are from Risk Register
Time Impact Range Cost Impact Range
Probability Minimum Most Likely Maximum Minimum Most Likely Maximum
Requirements have not been
30% 95% 105% 120%
decided
Several alternative designs
60% 95% 100% 115%
considered
New designs not yet proven 40% 96% 103% 112%
Fabricaton requires new
50% 96% 105% 115%
materials
Lost know‐how since last full
30% 95% 100% 105%
spacecraft
Funding from Congress is
70% 90% 105% 115%
problematic
Schedule for testing is
100% 100% 120% 130%
aggressive
Cost Risk is based on immature
100% 95% 105% 110%
data
© 2009 Hulett & Associates,
AcumenPM
- 23. Mapping Risks to Activities (1)
FS
Requirements FS Test FS
Risk Preliminary FS Final Design
Definition Fabrication Engine
Design
Requirements Not
X
Complete
Alternative Designs
X
Possible
Designs Not Proven X
New Materials in
X
Fabrication
Lost Know-How X
Funding Problematic X X X X
Testing Schedule
X
Aggressive
Cost Estimate is
based on Immature X X X X X
Data
© 2009 Hulett & Associates,
AcumenPM
- 24. Mapping Risks to Activities (2)
US
US Final Integratio
Risk Preliminary US Fabrication US Test Integration
Design n Testing
Design
Requirements Not
Complete
Alternative Designs
X
Possible
Designs Not
X
Proven
New Materials in
X
Fabrication
Lost Know-How X X
Funding
X X X X X X
Problematic
Testing Schedule
X X
Aggressive
Cost Estimate is
based on Immature X X X X X X
Data
© 2009 Hulett & Associates,
AcumenPM
- 25. Results Adding
Risk Factors to the Background Risk
Spacecraft for PMChallenge 2009
Entire Plan : Finish Date
100% 03/May/17
350
95% 05/Oct/16
90% 11/Aug/16
Baseline 12JUN 15 300 85% 01/Jul/16
80% 26/May/16
is only 3% likely 75% 02/May/16
250
The 80th percentile
70% 06/Apr/16
65% 15/Mar/16
(P-80) is 26MAY16, 60% 24/Feb/16
Cumulative Frequency
200
11.5 months later 55% 04/Feb/16
Hits
50% 15/Jan/16
Spread P-5 to P-95 150
45% 30/Dec/15
40% 16/Dec/15
is 13JUL15 to 35% 01/Dec/15
5OCT16, for 15.5 100
30% 16/Nov/15
months 25% 02/Nov/15
20% 15/Oct/15
15% 25/Sep/15
50
10% 27/Aug/15
5% 13/Jul/15
0 0% 21/Jan/15
20/Feb/15 08/Sep/15 26/Mar/16 12/Oct/16 30/Apr/17
Distribution (start of interval)
© 2009 Hulett & Associates,
AcumenPM
- 26. Activity Tornado Chart
from All-In Simulation
Spacecraft for PMChallenge 2009
Duration Sensitivity
00025 - US Fabrication 81% Risky Activities:
80%
Fabrication,
00011 - FS Fabrication
Integration, Final
00028 - Integration 78%
Design, Preliminary
00009 - FS Final Design 76% Design, Testing
76%
This is the typical
00023 - US Final Design
sensitivity analysis of
00021 - US Preliminary Design 69%
3-point estimating
00007 - FS Preliminary Design 69%
00012 - Test FS Engine 63%
00029 - Integration Testing 62%
00026 - US Test 61%
© 2009 Hulett & Associates,
AcumenPM
- 27. Risk Factor Tornado
from All-In Simulation
Driving Schedule Risk Factors
6 - Funding from Congress is problematic
4 - Fabricaton requires new materials
3 - New designs not yet proven
The main RISK,
7 - Schedule for testing is aggressive
however, is funding
from Congress, which
5 - Lost know-how since last full spacecraft affects all activities. This
is the main risk to
2 - Several alternative designs considered
mitigate, if possible
1 - Requirements have not been decided
0.00 0.20 0.40 0.60 0.80
Correlation
© 2009 Hulett & Associates,
AcumenPM
- 28. Contribution of
Each Risk to the Time Contingency (1)
Explain the Contingency to the P-80
P-80 Date Contribution of Risk
All Risks In 26-May-16 Days Saved % of Contingency
Specific Risks Taken Out in Order
Funding Risk 22-Jan-16 125 36%
Testing Schedule is Aggressive 1-Dec-15 52 15%
Design Requires New Materials 28-Oct-15 34 10%
New Design Risk 15-Oct-15 13 4%
Alternative Design Risk 6-Oct-15 9 3%
Requirements Risk 1-Oct-15 5 1%
Lost Know How Risk 30-Sep-15 1 0%
Background Schedule Estimating Risks
Background Risk 12-Jun-15 110 32%
Total Contingency 349 100%
© 2009 Hulett & Associates,
AcumenPM
- 29. Contribution of
Each Risk to the Time Contingency (2)
g g
100%
90%
26/May /16
01/Dec /15
12/J un/15
01/O c t/15
30/Sep/15
06/O c t/15
16/O c t/15
28/O c t/15
20/J an/16
80%
70%
Cum ulative P robability
60%
50%
40%
30%
20%
10%
0%
06/ Apr/ 15 15/ Jul/ 15 23/ Oct/ 15 31/ Jan/ 16 10/ May/ 16 18/ Aug/ 16 26/ Nov/ 16 06/ Mar/ 17
© 2009 Hulett & Associates,
AcumenPM
- 31. Cost Risk Results
Spacecraft for PMChallenge 2009
Entire Plan : Cost
100% $9,283,008,684
95% $8,429,356,166
650
90% $8,234,937,806
Baseline $6.86 billion 600
85% $8,088,750,149
is only 5% likely 550
80% $7,976,966,619
75% $7,890,603,866
500
The 80th percentile 70% $7,813,239,105
450 65% $7,749,200,097
(P-80) is $7.98 B for 60% $7,687,432,085
Cumulative Frequency
a $1.1 B contingency Hits
400
55% $7,630,117,222
350 50% $7,568,332,545
Spread P-5 to P-95 300
45% $7,514,273,968
is $6.86 to $8.43, for 250
40% $7,452,396,460
35% $7,391,997,593
$1.57 B 200
30% $7,331,963,666
25% $7,258,107,546
150
20% $7,188,049,098
100 15% $7,101,752,271
10% $7,007,130,725
50
5% $6,861,086,752
0 0% $6,189,493,454
$7,000,000,000 $8,000,000,000 $9,000,000,000
Distribution (start of interval)
© 2009 Hulett & Associates,
AcumenPM
- 32. Contribution of
Each Risk to the Cost Contingency
Contributions of Individual Risks to Cost Risk
All Risks $ millions
Cost Risk is based on immature data 287
Funding from Congress is problematic 277
Fabricaton requires new materials 168
Schedule Estimate inaccurate 110
Schedule for testing is aggressive 63
New designs not yet proven 31
Lost know‐how since last full spacecraft 2
Several alternative designs considered 1
Requirements have not been decided 0
© 2009 Hulett & Associates,
AcumenPM
- 33. Scatter Diagram of Time and Cost
Spacecraft for PMChallenge 2009
Deterministic Point Inside both limits Outside both limits
3%
There is a 94% $9,200,000,000
1% 94%
chance of $9,000,000,000
overrunning both $8,800,000,000
cost and $8,600,000,000
schedule $8,400,000,000
Notice the slope $8,200,000,000
of the scatter – $8,000,000,000
Entire Plan: Cost
time drives cost $7,800,000,000
in this model $7,600,000,000
$7,400,000,000
$7,200,000,000
$7,000,000,000
5%
$6,800,000,000 $6,861,500,000
$6,600,000,000
$6,400,000,000
2% 12/ Jun/ 15 3%
$6,200,000,000
20/Feb/15 31/May/15 08/Sep/15 17/Dec/15 26/Mar/16 04/Jul/16 12/Oct/16 20/Jan/17 30/Apr/17
Entire Plan: Finish
© 2009 Hulett & Associates,
AcumenPM
- 34. Probabilistic Cash Flow
Resource Flow for Cost
Filter: Entire Plan
Mean P20 P80
This monthly 9,000,000,000
probabilistic cash
8,000,000,000
flow can be
compared to the Deterministic Cost: $6,861,500,000 7,000,000,000
budget to adjust the 6,000,000,000
spending patterns
Cu m u lative
5,000,000,000
when considering
D eterm inis tic F inis h: 12/J un/15
risk 4,000,000,000
3,000,000,000
2,000,000,000
1,000,000,000
0
26/ Jul/ 08 11/ Feb/ 09 30/ Aug/ 09 18/ Mar/ 10 04/ Oct/ 10 22/ Apr/ 11 08/ Nov/ 11 26/ May/ 12 12/ Dec/ 12 30/ Jun/ 13 16/ Jan/ 14 04/ Aug/ 14 20/ Feb/ 15 08/ Sep/ 15 26/ Mar/ 16 12/ Oct/ 16 30/ Apr/ 17
Time
© 2009 Hulett & Associates,
AcumenPM
- 36. Summary (1)
• The focus is on the risks, not their impact
• Risks “explain” the need for a contingency
• Management appreciates this focus on
risks
• Risk interviews are conducted at 10,000
foot level, where people typically think of
risk
• Interviews go faster, stick to the substance
© 2009 Hulett & Associates,
AcumenPM
- 37. Summary (2)
• Use Risk Register for quantitative analysis
• Specific risks can be quantified and assigned to
schedule activities
– Quantification is probability and impact
– A risk can affect several activities
– An activity can be affected by several risks
• Risk Factors can be combined with other more
traditional approaches such as 3-point estimates
for background risk or probabilistic branching
© 2009 Hulett & Associates,
AcumenPM
- 38. Summary (3)
• Schedule uncertainty creates cost
uncertainty
• Analysis of cost simultaneously with time
requires inserting the budget/resources
into the schedule and simulating both
together
• More accurate cost risk analysis and full
appreciation of the role of schedule risk in
creating cost risk
© 2009 Hulett & Associates,
AcumenPM
- 39. Using the Risk Register in Integrated
Cost/Schedule Risk Analysis
with Monte Carlo Simulation
David T. Hulett, Ph.D.
Hulett & Associates, LLC
Dr. Dan Patterson, PMP
Acumenpm
NASA PM Challenge
Daytona Beach, FL
February 24-25, 2009
© 2009 Hulett & Associates,
AcumenPM