Increasing the probability of program success at Oak Ridge Y-12 Facility

1. MANAGING COST,
SCHEDULE, AND
TECHNICAL PERFORMANCE
RISK AT Y-12
Increasing the Probability of Program Success
1
Niwot Ridge
Consulting

2. Starting with DOE Guidance
2

3. The DOE Risk Management Lifecycle
3

4. DOE G 413.3-7A Risk Process
4
DOE G 413.3-7A, page 9

5. Connecting the Dots Between the Elements
of the Performance Measurement Baseline
5
Risk
SOW
Cost
WBS
Schedule
TPM
PMB

6. There are two types of Uncertainty
Uncertainty about the
functional and performance
aspects of the program’s
technology that impacts the
produceability of the product
or creates delays in the
schedule
Uncertainty about the
duration and cost of the
activities that deliver the
functional and performance
elements of the program
independent of the technical
risk
6
Technical Programmatic

7. Risk Assessment and Management
Techniques Vary with Maturity†
7
Add a Risk Factor or Percentage to the critical paths
A “bottom line” Monte Carlo or Range analysis
Detailed Monte Carlo for each WBS element
Expert Opinions in a Database with assessment
Detailed Bayesian Network Analysis
Increasing Detail and Difficulty
IncreasingPrecisionandValue
¨ There are several approaches to building a Risk Tolerant
Performance Measurement Baseline
¤ First recognize where you are on the curve
¤ Then recognize there is value in moving further up the curve
† Ron Coleman Litton TASC, 33rd
ADoDDCAS, Williamsburg, VA

8. Risk is Different from Uncertainty
Knowing this Difference is Critical to Success
¨ Cost estimating methodology
risk resulting from improper
models of cost
¨ Cost factors such as inflation,
labor rates, labor rate burdens,
etc
¨ Configuration risk (variation in
the technical inputs)
¨ Schedule and technical risk
coupling
¨ Correlation between risk
distributions
¨ Requirements change impacts
¨ Budget Perturbations
¨ Re–work, and re–test
phenomena
¨ Contractual arrangements
(contract type, prime/sub
relationships, etc)
¨ Potential for disaster (labor
troubles, shuttle loss, satellite
“falls over”, war, hurricanes,
etc.)
¨ Probability that if a discrete
event occurs it will invoke a
project delay
8
Risk stems from known probability
distributions
Uncertainty stems from unknown
probability distributions

9. Schedule Risk Management …
¨ Seeks to anticipate and address uncertainties that threaten the
goals and timetables of a project
¨ Recognizes unmitigated risks lead rapidly to delays in delivery
dates and budget overages that undermine confidence in the
schedule and in the project manager
¨ Is process oriented, guided by DOE G 413.3-7
¨ Accepts a certain level of risk, regular and rigorous risk
analysis and risk management techniques serve to defuse
problems before they arise
¨ Defines an Integrated Master Plan that reflects the
development phases and the hierarchical structure of the
system.
: Risk Based Planning

10. A sample Risk Management System at
Johnson Space Flight Center
10

11. Connecting the Dots, Again
11
Risk
SOW
Cost
WBS
Schedule
TPM
PMB
Named
Deliverables
defined in the WBS
BCWS at the Work
Package, rolled to the
Control Account
TPMs attached to each
critical deliverables in the
WBS and identified in
each Work Package in the
IMS, used to assess
maturity in the IMP
The Products and
Processes that produce
them in a “well structured”
decomposition in the WBS
Schedule contains
all the Work
Packages, BCWS,
Risk mitigation
plans, and rolls to
the Integrated
Master Plan to
measure
increasing maturity
Technical and Programmatic
Risks Connected to the WBS
and IMS

12. The Basics of Probabilistic Schedule12

13. All Schedule Activities Are Random
Variables, with Probability Distributions
13

14. Cost does not have a linear relationship with
schedule.
Basic Principles of Probabilistic Cost14

15. Keys to Cost Estimating Success
15
¨ Start with guidance on
cost estimating.
¨ Tailor the guidance to fit
the problem domain.
¨ Verify the processes work
and add value.
¨ Improve the fidelity of
estimates with feedback.
¨ Adjust estimating
parameters to match
actuals.

16. Basic Principles with Probabilistic Cost
Estimating Relationships (CER)
16
¨ Cost estimates involve many CERs
¤ Each of these CERs has uncertainty (standard error)
¤ CER input variables have uncertainty (technical uncertainty)
¨ Combine CER uncertainty with technical uncertainty for many
CERs in an estimate
¤ Usually cannot be done arithmetically; must use simulation to roll
up costs derived from Monte Carlo samples
n Add and multiply probability distributions rather than numbers
n Statistically combining many uncertain, or randomly varying, numbers
¤ Monte Carlo simulation
n Take random sample from each CER and input parameter, add and
multiply as necessary, then record total system cost as a single sample
n Repeat the procedure thousands of times to develop a frequency
histogram of the total system cost samples
n This becomes the probability distribution of total system cost

17. The Cost Probability Distributions as a
function of the weighted cost drivers
17
$
Cost Driver (Weight)
Cost = a + bXc
Cost
Estimate
Historical data point
Cost estimating relationship
Standard percent error boundsTechnical Uncertainty
Combined Cost Modeling
and Technical Uncertainty
Cost Modeling Uncertainty

18. Basic Principles of connecting cost
models with the IMS involve three steps
18
¨ Step 1: Define “likely–to–be” program
¤ Using deterministic inputs from the Independent Technical
Assessment (ITA)
¨ Step 2: Quantify the probability distributions describing the
modeling uncertainty of all CERs, cost factors, and other
estimating methods
¤ Specifically, the type of distribution (normal, triangular,
lognormal, beta, etc.)
¤ The mean and variance of the distribution
¨ Step 3: Quantify the correlation between all WBS elements
that are estimated using CERs and other methods
¤ If unknown, assess whether No correlation, Mild correlation, or
High correlation, for example:
n None: r = 0, Mild: r = ±0.2, High: r = ± 0.6
¤ Correlation affects the overall cost variance

19. Basic Principles
19
¨ Step 4: Set up and run the cost estimate in a Monte Carlo
framework (e.g., Crystal Ball, @RISK), resulting in a
“baseline” estimate
¤ This will provide a probability distribution of the cost based on
cost estimating model uncertainty only
¤ Report the MEAN as the baseline expected cost
¨ Step 5: Now incorporate technical uncertainty and discrete
risks
¤ Step 5a: Set up a new estimate which also contains any “discrete
risk” events that are to be guarded against
n Quantify appropriate modeling uncertainties and correlations, as in
Steps 2 and 3, for these discrete risks
¤ Step 5b: Define the probability distributions for all CER input
variables
n Also may need to quantify correlation between CER input variables

20. Basic Principles of connecting cost
models with the IMS involve three steps
20
¨ Step 6: Re–run the Monte Carlo simulation with random CER
input variables and discrete risk events, resulting in a final
“risk–adjusted” estimate
¤ Results in a new risk–adjusted cost probability distribution.
¤ Wider and shifted to the right Baseline vs. Risk-Adjusted Estimates
0 50 100 150 200 250 300 350
FY$M
Likelihood

21. Baseline versus Risk Adjusted Cost Estimates
Almost Always Shows an Increase In Cost
21
Baseline vs. Risk-Adjusted Estimates
0 50 100 150 200 250 300 350
FY$M
Likelihood

22. S-Curve for Cost Modeling
22
Cumulative Distribution Function
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
$60 $80 $100 $120 $140 $160 $180 $200
FY00$M
CumulativeProbability

23. Schedule Management
23

24. Risk Management
24

25. Connecting the Dots
25