Probabilistic Schedule and Cost Analysis

Programmatic Risk Analysis 1/186
Probabilistic Schedule and
Cost Analysis
Our goal is to develop a resource loaded, risk
tolerant, Integrated Master Schedule (IMS), derived
from the Integrated Master Plan (IMP) that clearly
shows the increasing maturity of the program’s
deliverables, through vertical and horizontal
traceability to the program’s requirements.
Prepared for NNJ05111915R, by GB Alleman, December 2005

Source Material Disclaimer
§ All the materials in this briefing originated from publicly
available sources, nothing here should be construed as
proprietary or unique to an individual vendor or manufacturer.
§ All materials are used for informational and educational
purposes ONLY and are not be reused outside the teaching
process.
§ The Fair Use Copyright law provides use for research and
teaching.
§ All references to materials used in this briefing are provided in
the “resources” section.
§ No other reuse of this briefing should be performed outside of
the specific learning objectives of “Probabilistic Schedule and
Cost Analysis,” education on proposal and execution.
: Introduction

Programmatic Risk Management Involves
Danger and Opportunity
§ The Chinese symbol for risk represents
– Danger
– Opportunity
§ Taken together this suggests that risk is a strategic
combination of vulnerability and opportunity.
§ Viewed in this light, programmatic risk management is a tool
for managing risk that enables the program to take advantage
of value enhancing opportunities.
§ A missed strategic opportunity can result in a greater loss of
(potential) value than an unfortunate incident or adverse
change master schedule.
§ Many programs address risk in “silos.”
§ Programs cannot take this approach – an integrated technical
and programmatic risk management strategy must be used.
: Introduction

Dealing With Our “Learning Opportunity” In
The Presence Of The “Danger” Of Being Late
The task of space management is rocket science. It is
terribly complicated. Launchers explode and spacecraft
disappear. No one wants to fail. Good enough is not good
enough for mechanisms within which thousands of
components must work in tandem for a mission to
succeed.
– “Bureaucracy and the Space Program,” in Sadeh, ed.,
Space Politics and Policy, 2003.
§ When we add programmatic risk issues and their management to the
technical risk issues, the integrated programmatic solution must
track and support the integrated technical solution
§ This briefing provides the background for the use of Monte Carlo
simulation in the construction of a risk tolerant IMS
: Introduction
§ The material presented here is an “in depth” look at Monte Carlo which is more than just a
tool, it is the basis of a process
§ Because of this, a deeper understanding of the process is needed to properly apply Monte
Carlo to build a risk tolerant IMS
§ So be prepared, this is not one of the overview tours, this will be work with the equivalent
reward – a competitive IMS that can hold its own in front of the customer and the
competition

This Briefing is an Overview of
Programmatic Risk, but not a Handbook of
how to do it
§ Why the IMS must include a programmatic and schedule risk
management strategy
– Programmatic risk and technical risk are linked
§ Techniques for managing uncertainty in the IMS
– Specific tasks for risk management
§ PERT and Monte Carlo methods
– Why are we using Risk+ and not focusing on PERT
§ Models of a risk tolerant schedule
– Examples of Risk+
– Examples of risk mitigation plans
: Introduction

One View of Project Management
§ Technocrats manage projects
– Driven by the adherence to the
profession of engineering and the
profession of managing engineering
projects
§ This approach is very useful when
we have reliable, predictable and
operational projects
– Single point failures are managed
within an “engineering” culture with
built–in redundancy
§ The problem with this approach is:
– Hierarchical structures try to ensure organizational control and accountability
– Complex systems are prone to failure
– Budget and schedule issues and to the technical complexity
– Interactive failure modes abound for both technical and programmatic risk situations
§ The role of a Risk Tolerant IMS is to make visible these problems and provide the
opportunity for their solution
: Introduction

What Kind of Risk are We Talking About in
the IMS?
§ People are not generally
good at analyzing risk
§ Most risk analysis is
qualitative in nature
– Skills, time and resources
are in short supply to
undertake a quantitative
risk analysis
– The result is usually a
subjective assessment of
the risk
§ However quantitative
analysis is needed for high
programmatic risks
§ A resource loaded IMP/IMS containing accurate predecessors and
successor relationships is the starting point for quantitative risk
assessment
: Introduction

Integrating Risk Management into the IMS
is a Multi–step Process
§ Programmatic risk assessment must define in the IMS both the
technical risks, their mitigations as well as the programmatic steps
take during these mitigations.
§ These steps include:
– Connection of
technical and
programmatic risks
– Branching
probabilities for
alternative paths
taken during
mitigation
– Resources needed
for these alternative
paths
– Impacts on critical
milestones resulting
from the occurrence
of a risk
: Introduction

I think you should be more explicit here in step two
Executive
Overview
There is a large amount of material in this briefing. So this overview
provides a “summary” of the concepts and the recommended approach
to developing a credible IMS based on probabilistic risk analysis.
But in the end the details must be “owned” before we can say we have a
handle on Probabilistic Schedule and Cost Analysis.
: Executive Overview

We Usually Want To Know What Our
Motivation In Order To Hold Our Interest
§ Noel Coward’s motivational speech
– If you must have motivation, think of your pay check
on Friday
§ Our motivation starts with DID 81650, which says:
– The IMS shall be used to verify attainability of contract
objectives, to evaluate progress toward meeting program
objectives, and to integrate the program schedule
activities with all related components.
§ The construction of an IMS in this manner is necessary but not
sufficient to show that the contract objectives can be attained.
– An IMS that is tolerant of both technical and programmatic risks is the next
step
– Such an IMS contains both the known risk mitigation processes as well as
the unknown risk mitigation opportunities
§ The BIG QUESTION is what are the units of measure of risk tolerance?
Noel Coward 1899 – 1973

Programmatic Risk Management in One Slide
Starts with Abandoning PERT and CPM
§ The Critical Path Method (CPM) does not provide a realistic view of
programmatic risk.
– Task durations are random variables drawn from an underling probability
distribution
– Near Critical Paths biases the completion time of the program in ways not
show by the CPM
– Correlations between multiple paths bias the risk analysis as well
§ PERT estimates do not address the underlying probabilistic nature of
activity networks. PERT assumes…
– Statistical independence of each activity – this is almost never the case
– Symmetrical probability distributions that allow the addition of the “most
likely” (Mode) value of each activity to produce a total project duration
§ Monte Carlo must be used and requires understanding of ...
– The probability distribution functions of the activity network
– The branching probabilities for alternative paths created by risk mitigation
– The influence of correlations between network activities and their impacts
on risk

Merging Technical and Programmatic Risk
is the Core to Building a Risk Tolerant IMS
§ There are two types of “uncertainty” on a
program
– Technical – uncertainty about the functional
and performance aspects of the program’s
technology that impacts the produceability of
the product or creates delays in the schedule
– Programmatic – uncertainty about the
duration and cost of the activities that deliver
the functional and performance elements of
the program independent of the technical risk
§ We’re interested in connecting the two in the schedule and cost
model(s)
– When the technical uncertainty arises what is the impact on the schedule
and cost?
– When the schedule or cost uncertainty arises what is the impact on the
functional and performance aspects?
So much for our strategy of winning
through technical dominance

The Meaning of Uncertainty in the Context
of an Integrated Master Schedule
§ Uncertainty in plain English is about the “lack of certainty”
– Uncertainty is about the “variability” in the performance measures
like cost, duration, or quality
– Uncertainty is about the “ambiguity” associated with a lack of this
clarity
§ Discovering the known and unknown sources of bias and
ignorance helps define much effort it is worth to clarify the
uncertainty
– This is the underlying process driving uncertainty
§ As well, uncertainty arises from the basic processes of work
– This is Deming uncertainty
– It is the statistical “noise” built into the work process
§ Both of these sources of uncertainty impact cost and schedule
– Trying to control the “noise” adds little value
– Trying to control the “lack of certainty” arising from ambiguity and
lack of clarity does have value

Schedules Are Networks Of Random Variables
not Collections of Deterministic Statements
§ Task completion durations are random variables not just
dates:
– These random variables have underlying probability
distributions
– These distributions can not be “added” to arrive at a project
completion date
– Trying to force the work into a fixed duration does not
increase the likelihood of completion
§ The PERT approach to estimating project duration
contains several faulty assumptions:
– The assumed independence is rarely the case
– Uniform distribution of completion times can not be
confirmed
– The 3–Point estimates have built in optimistic bias
§ Monte Carlo Simulation provides more accurate
estimates of project completion times:
– But only if the network topology is “well formed”
– And if the interactions of the underlying probability
distributions are understood
Attempting to make the dryer
“dry faster” is a loosing
proposition.
The dryer’s capacity is
constrained by its mechanics.
Much like the capacity of the
design team is constrained by
availability and technical
productivity.

Building a Credible IMS Means Managing
the Numbers in Meaningful Ways
§ First step – build the numbers correctly
– Build an IMS in “layers” of detail
– Understand the probability distributions of each
layer
– Identify schedule margin opportunities within
the IMS
– Assess the margin’s impact on the probabilistic
outcome
§ Continuous process improvement – build the IMS “many times”
– Gather historical and “expert” opinions of task duration
– Build probability distribution functions from this data
– Improve the Monte Carlo model using this data
§ Answer the question – many times, possibly continuously
– “What are the units of measure for credibility?”
– “What is the coupling and cohesion of the tasks in the IMS?”

The Right Effort Produces Results, But
Interpreting the Results is Sometimes Difficult
§ Proposal Improvements
– Credible schedule based on probabilistic model
– Traceable data to probability distributions for each
class of task
– Verifiable forecast of risk areas and project duration
§ Executable program Improvements
– Model of “hot spots” in the IMS
– Continuous assessment of schedule and cost risk
– Increased visibility of probabilistic methods
§ Improvement in our understanding of probabilistic planning
– “What is the critical path” requires more than a red line in a power point
chart
– No point values are allowed, only statistically qualified estimates
– Stochastic network models of schedule and cost are the minimum
deliverables for proposal and execution

The One Slide Describing Our Search for
“Actionable Outcomes”
§ Building a risk tolerant IMS
– Explicit technical risk mitigation must be embedded in the IMS
– Explicit schedule margin must be embedded in the IMS
» Margin values identified through Monte Carlo simulations
» Margin assigned in front gating events
– Technical risks connected to ARM in some form
– Cost and Schedule risks connected to i–MAP (impact mapping from
Woods Analysis)
§ Assessing the Risk Tolerant IMS – what does risk tolerant mean?
– Weekly status, monthly Earned Value, forecast of risk impacts
– Weekly Monte Carlo assessment of confidence intervals and their
historical changes – are we getting better or worse?
– Performance forecast based on likelihood outcomes from Monte
Carlo simulations, not just “adding up the numbers”

Programmatic Risk Analysis 18/186: Executive Overview
The Page Intentionally Left Blank

Risk Based Planning
The difference between failure
and success is the difference
between doing something almost
right and doing something right.
— Benjamin Franklin
: Risk Based Planning
Murray didn’t feel the first pangs of real panic
until he pulled the emergency cord.

In the Risk Management Business, there is
Simply Too Much Information for Our Needs

Risk Assessment and Management
Techniques Vary with Maturity †
Add a Risk Factor or Percentage to the critical paths
A “bottom line” Monte Carlo or Range analysis
Detailed Monte Carlo for each WBS element
Expert Opinions in a Database with assessment
Detailed Bayesian Network Analysis
Increasing Detail and Difficulty
IncreasingPrecisionandValue
§ There are several approaches to building a Risk Tolerant IMS
– First recognize that we’re in the early stage of this effort
– There is likely value in moving further up the curve
† Ron Coleman, Litton TASC, 33rd ADoDDCAS, Williamsburg, VA

Risk is Different from Uncertainty
Knowing this Difference is Critical to Success
§ Reducible risk stems from known probability
distributions
– An Estimating methodology risk resulting
from improper models of cost + schedule
– Cost factors such as inflation, labor rates,
labor rate burdens, etc
– Configuration risk (variation in the technical
inputs)
– Unknown Cost, Schedule and Technical risk
coupling
§ Irreducible risk stems from known statistical processes
– Requirements change impacts
– Budget Perturbations
– Re–work, and re–test phenomena
– Contractual arrangements (contract type, prime/sub relationships, etc)
– Potential for disaster (labor troubles, shuttle loss, satellite “falls over”, war,
hurricanes, etc.)
– Probability that if a discrete event occurs it will invoke a project delay

There are Two Types of Uncertainty
Encountered in a Risk Tolerant IMS
§ Static uncertainty is natural variation and foreseen risks
– Uncertainty about the value of a parameter
§ Dynamic uncertainty is unforeseen uncertainty and “chaos”
– Stochastic changes in the underlying environment
– System time delays, interactions between the network elements, positive
and negative feedback loops
– Internal dependencies
Stochastic behavior of forecasted completion dates
Low Work
Quality
Poor Work
Conditions
External
Scope
Changes
Unintended
Changes
Upstream Hidden Change

The Multiple Sources of Schedule Uncertainty
and Sorting Them Out is the Role of Planning
§ Unknown interactions drive
uncertainty
§ Dynamic uncertainty can be
addressed by flexibility in the IMS
– On ramps
– Off ramps
– Alternative paths
– Schedule “crashing” opportunities
§ Modeling of this dynamic
uncertainty requires simulation
rather than static PERT based path
assessment
– Changes in critical path are
dependent on time and state of the
network
– The result is a stochastic network

Schedule Risk Management is…
§ Schedule risk management seeks to anticipate and address
uncertainties that threaten the goals and timetables of a project
§ Unmitigated risks lead rapidly to delays in delivery dates and
budget overages that undermine confidence in the schedule
and in the project manager
§ Schedule risk management is process oriented
§ While any project accepts a certain level of risk, regular and
rigorous risk analysis and risk management techniques serve
to defuse problems before they arise
§ Integrated Planning reflects the development phases and the
hierarchical architecture of the system

What’s our Goal as Planners?
§ Construct an IMS that has integrity and credibility to show …
§ External assessors who may consider that the schedule …
– Reflects the total scope of work
– Is fully integrated
» Internally (task/milestone interdependencies)
» Externally (other NASA facilities, contractor schedules, vendor deliveries,
etc.)
– Has an established baseline
– Is reasonable or even feasible at proposal submission
– Does not provide for “What–if” analysis
– Is capable of providing for multiple and varying Critical Path identification
or slack for all tasks and milestones
§ … and how the schedule may…
– reflect an accurate model of planned implementation
– reflect an accurate or complete status
– provide the correct basis for resource planning

Some “Unpleasant” Questions Can Easily Occur If
We Don’t Pay Attention To The Details
§ What is the degree of risk in our
baseline? How do we measure this
risk?
§ What are the branching
probabilities for the critical path in
the IMS? How are they derived?
§ How many “near critical” paths
will become critical as the program
proceeds? What drives these?
§ Have the “risk drivers” been identified and mitigations put in place
through explicit tasks in the IMS to deal with each identified risk? If
so, how are they shown in the IMS?
§ Do we understand the underlying task completion probability
distributions? How are they derived?
§ How do these probability distributions change as the program
proceeds? What is the analytical basis for this?
Remember Edsel Ford’s dream of the future?

Risk Management at NASA
§ Risk includes undesirable consequences (harm) and
probability of occurrence of this harm
§ For the IMS, this harm is the failure to …
– Identify risk mitigation tasks
– Provide sufficient schedule margin at the right places
§ Risk consists of three elements:
– What can go wrong? – define a set of scenarios
– How likely is it? – an evaluation of the probabilities
– What are the consequences? – an evaluation of the consequences
§ The identification of these risk scenarios is the most important result
of our Risk Assessment process for the IMS:
– What are the failure scenarios for the program?
– What are the mitigation strategies for these failure modes?
– What resources are needed for each mitigation?

NASA IRMA Tool is Used at Johnson Space
Center

NASA Risk Summary Card and
Programmatic Risk

Implementing Programmatic Risk
Assessment is a Straight Forward Process
Initiating Event
Selection
Scenario
Development
Scenario Logic
Modeling
Scenario
Frequency
Modeling
Consequence
Modeling
Risk Integration
§ Some simple steps to identifying risk opportunities in the IMS
– Scenario based planning – “what if this happens?”
– Event impact planning – “what inhibits success?”
§ Both must focus on the consequences in order to identify the
mitigations

Continuous Risk Management (CRM) is the
Basis of Programmatic Risk Management
§ NASA Guidance
– OMB A–11
– NASA NPG
7120.5A
– NASA–SP–610S
– NASA NPR 8000.4
§ DoD Guidance
– DAU “Risk Management
Guide for DoD Acquisition
– Air Force, “Acquisition
Risk Management”
– Air Force “SMC Systems
Engineering Primer and
Handbook”
CRM Activity IMS Representation
Identify Risk items with IMP/IMS #’s, CA/WP & resource assignments
Analyze Risk management responsibilities assigned
Plan Mitigation plans with durations and resource assignments
Track Status reported from Risk Management to IMS
Control Risk tasks reporting in weekly status process
Communicate IMS status reporting

Design v. Risk Evaluation are Two Sides
of the Same Coin – Risk Tolerance
§ The IMS for the “planned” program
can be considered the “reference
mission” plan
– Meets the SOW and DRD deliverable
plan with deterministic tasks
– Critical paths defined
– Explicit schedule margin assigned per
PMP and risk identification
§ Missing elements from the design evaluation that must be in
the risk evaluation
– Near critical paths impacting durations
– Individual task risk assignment
– Risk distribution curve for classes of activities
– Cumulative risk probability distribution skewing
– Branching probabilities not defined
– Correlation between risk paths and off risk paths
– Dynamic interactions that drive risk

Embedding Risk Management in the
Integrated Master Schedule
§ The IMS should show the coupling between technical risk and
programmatic risk
– Technical risk activities in the IMS connected to Active Risk
Manager
– Programmatic risk visible in the IMS
§ Technical estimates of task durations developed from subject
matter experts
– Past performance
– Basis of estimate
– Expert judgments
– Parametric estimates
– Dynamic models
§ Programmatic estimates developed through the win strategies
– Convey the risk buy down and win theme support through the risk
mitigation activities
All these estimates must be
calibrated before being
accepted into the IMS.
Without this effort, these
numbers are just as
unreliable as raw estimates

Why Probabilistic Risk Analysis is Often
Opposed by Management and IPT Leads
§ Many people do not understand
the underlying statistics
– Education, practice, guidance
§ Many planners lack the formal
probability and statistics
training
– Education, practice, guidance
§ Most planners perform
deterministic analysis of
schedules and cost
– Risk is hard work
§ The fact the probabilistic risk analysis is built on uncertainty is seen
as weakness in the planning process, not a strength
– Why can’t you know how long it will take or how much it costs?
§ People tend to think that the “lack of data” is a reason not to perform
probabilistic schedule risk analysis
– The exact opposite is true

Programmatic Risk Analysis

Managing
Uncertainty in
the IMS
A ship on the beach is a
lighthouse to the sea.
— Dutch Proverb
: Managing Uncertainty in the IMS

3 Troubles With Deterministic Schedule
Estimating in a Traditional Manner
1. Single point estimates can be
accurate
a) Without stating the distribution
statistics, the number has no
frame of reference
b) The median temperature in
Cody Wyoming is a balmy 78º
c) Don’t be there in February in
your shirt sleeves
2. There is no standard definition for the term “best” estimate
a) A hoped for best
b) A planned for best
c) The actual best derived from the underlying statistical model
3. Rollup of the “most likelies” is not the same as Most Likely Total
Duration
a) They are probability distributions not integers
b) Probability distribution function (pdf) convolution is needed
The flaw with using averages alone
Average depth is 3 feet

No Single Point Estimate Can Be Accurate,
Since It Ignores The Underlying Statistics
§ Schedule durations are always vague
in the beginning
– Existing technical capability often falls
short of project needs
– Firm requirements, especially
software requirements cannot be
described in a simple list
– “Normal” schedule slippage of
varying lengths result from integration
problems and test failures
– Various other anticipated and
unforeseen events resulting from the
natural variation (Deming variation)
§ “Point” estimates can not be correct because…
– Task point estimates of activity durations are not correct
– Project point estimate is the sum of these “incorrect” activity estimates
§ “Actual” project duration will fall within some range around the point estimate
– The best that can be done is to understand the uncertainty
1

The Traditional Roll Up Approach Starts with
the “Best” estimates for the Most Likely
Durations for Tasks
§ Build a network of the project’s activities
§ Determine the “best” estimate of the duration for each activity
in the network
§ Compare the activities’ “best” estimates to find the critical path
§ Sum all the “best estimate” durations of activities on the
critical path
§ Define this sum of tasks to be “best” estimate of the project’s
schedule duration
§ This will almost always be optimistically wrong
§ Or it is pessimistically wrong
§ Either way – it is wrong
1

The Problem is the Term “Best” Has No
Standard Definition
§ For each activity the “best” estimate is …
– The “most likely” duration – the mode of the distribution of
durations? (Mode is the number that appears most often)
– It’s 50th percentile duration – the median of the distribution?
(Median is the number in the middle of all the numbers)
– It’s expected duration – the mean of the distribution? (Mean is the
average of all the numbers)
§ These definitions lead to values that are almost always different
from each other
§ Rolling up the “best” estimate of completion is almost never
one of these.
2

Durations Are Probability Samples not
Single Point Values
§ We know this because…
– “Best” estimate is not the only possible estimate, so other
estimates must be considered “worse”
– Common use of the phrase “most likely duration” assumes that
other possible durations are “less likely”
– “Mean,” “median,” and “mode” are statistical terms characteristic
of probability distributions
§ This implies activity distributions have probability distributions
– They are random variables drawn from the probability distribution
function (pdf)
§ “Actual” project duration is an uncertain quality that can be
modeled as a sum of random variables
– The pdf may be known or unknown
2

Durations are Educated “Guesses,” but Rarely
Have underlying Probability Distributions
§ Define the problem
§ Identify the prediction variable
§ Build the prediction model
– Develop a list of relevant factors
– Consider the effects
– Collect data
– Plot each factor independently
– Develop a prediction model using linear regression
– Understand the model
– Check the model
§ Make guesses with the model
§ Take care with the results
2

One Way Of Producing A Guess Is With A
“Twenty Questions” Game
§ The 20 questions approach
– Ask an engineer how long it will
take to do a task and the
answer might be “I can’t say.”
§ Planner
– Will to take a year?
§ Engineer
– Oh, of course not
§ Planner
– Will it take a day?
§ Engineer
– Oh, of course not
§ Planner
– How about 6 months?
§ Engineer
– Could be, but that’s too long
§ Planner
– How about 2 months?
§ Engineer
– No, that’s too short
§ Planner
– How about 3 ½ months?
§ Engineer
– Yea, that could work
§ In 5 question a first order
estimate can be found to with
20%
2

Putting Guesses into a Schedule Requires
Us to Sort Out Fact from Fiction
§ Rank all the tasks
– 1 = scope known, duration known
– 2 = scope known, duration unknown
– 5 = scope unknown, duration unknown
§ Have a “planning session” where no one leaves the room until
all Rank–5 tasks are turned into Rank–2 tasks
– The reduction in rank comes for “information” about the
probability distribution which underlying the random variable
representing the duration
§ Focus on “randomness” of the estimate is critical to success
– Adjustments for confidence must become part of our vocabulary
– This approach turns “guessing” into statistical estimating
2

Risk Drivers That Impact Uncertainty Must
be Identified Before They are Used
§ Risk drivers include (but may not be limited to)
– Beyond the state–of–the–art development
– Unusual production requirements, either time or technique
– Cost constraints, derived without consideration of technical or
programmatic processes
– Software development issues
– Multiple interface management
– Subcontractor and supplier viability, variability or plain olde
“ability”
– System integration and testing impacts from coupling and
cohesion of the tasks in the IMS
– Unforeseen events
2

Estimating Accomplishment Criteria starts
with defining the what “done” look like
§ When building the IMS, the first round should
– Confirm the IMP (PE/SA/AC) structure overlays the topology of the
program
– Ask for a duration estimate for each Accomplishment Criteria
which represent
» Exit criteria
» Deliverables for the maturing program
» Incremental progress along the path to maturity
– Get this estimate as a “single” task with a duration
§ Assign SA’s to an IPT Lead
– Develop IPT processes that support the SA
– Identify AC’s from process deliverables or maturity improvements
– Link vertical path to Program Events and horizontal paths across
IPTs
2

Modeling Duration Probability starts with
experts but must include statistical estimates
§ Compile duration estimates from different sources and rank the
estimates:
– Subcontractors or IPTs estimate
– Project manager’s estimate
– “Independent” estimate
– Risk–impacted estimate
§ Associate confidence levels with ranges between estimates,
using information available from different situations and at
different stages in the project development cycle
– These can not be looked up in a book
– Are not directly derived from historical data
– Must be subjective, knowledge based consensus of technical
experts in a particular WBS
– Should be a standard part of the risk mitigation plan
3

Statistics at a Glance as a starting point.
But more details are needed
§ Probability distribution – A function
that describes the probabilities of
possible outcomes in a "sample
space.”
§ Random variable – variable a
function of the result of a statistical
experiment in which each outcome
has a definite probability of
occurrence.
§ Determinism – a theory that
phenomena are causally determined
by preceding events or natural
laws.
§ Standard deviation (sigma value) –
An index that characterizes the
dispersion among the values in a
population.
§ Bias – The expected deviation of
the expected value of a statistical
estimate from the quantity it
estimates.
§ Correlation – A measure of the joint
impact of two variables upon each
other that reflects the simultaneous
variation of quantities.
§ Percentile – A value on a scale of
100 indicating the percent of a
distribution that is equal to or below
it.
§ Monte Carlo sampling – A modeling
technique that employs random
sampling to simulate a population
being studied.
3

Siren Song of the Central Limit Theorem must
be avoided in any robust estimating process
§ The probability distribution of the project’s total duration is obtained by
statistically summing distributions of all activities along the schedule
network critical path
§ Central Limit Theorem – if the number of critical path activities is
“large,” the probability distribution of the total duration is
“approximately” Gaussian.
§ Another theorem (not related to Mr. Gauss)
– the sum of the means of activity duration
equals the mean of the total duration
– But because the Gaussian distribution is
symmetric, the total duration distribution
results in mean = medium = mode
– Therefore,
» Sum of the activity duration medians < total
duration median
» Sum of the activity modes < total duration
mode
3

The Central Limit Theorem must be
understood in order to be useful
§ The Central Limit Theory (CLT) consists of three statements
– The mean of the sampling distribution of means equals the mean
of the population from which the samples were drawn
– The variance of the sampling distribution of means is equal to the
variance of the population from which the samples were drawn
divided by the size of the samples
– If the original population is distributed normally, the sampling
distribution of means will also be normal. If the original population
is not normally distributed, the sampling distribution of means will
increasingly approximate a normal distribution as sample size
increases
§ PERT assumes the duration and variance along the critical path
are normally distributed and therefore the total duration follows
the CLT
§ Departures from normal distribution and near critical path tasks
becoming critical are usually ignored – to the peril of planning
3

Mr. Gauss’s Distribution is found in many text
book examples, too bad is not applicable
§ The Gaussian distribution can be proved (by the Central Limit
Theorem) in the situation that each measurement is the result of
a large amount of small, independent error sources. These
errors have to be of the same magnitude, and as often positive
as negative.
§ When a physical item is
measured and systematic
errors are eliminated the
measured values will spread
around the average value.
§ The average value of a
measured value is the “best
value.”
3

Task “Most Likely” ≠ Project “Most Likely,”
Must be Understood by Every Planner
§ PERT assumes
probability
distribution of the
project times is the
same as the tasks
on the critical path
§ Because other paths
can become critical
paths, PERT
consistently
underestimates the
project completion
time
1 + 1 = 3
3

Probability Distribution Function is the
Lifeblood of good planning
§ Probability of
occurrence as a
function of the
number of samples
§ “The number of
times a task
duration appears in
a Monte Carlo
simulation”

Standard Deviation of a Probability
Distribution
§ Which describes
that “spread” of
the random
variables around
the mean
represented by
the distribution
§ Standard deviation describes the width of the probability distribution
function

Underlying Statistics and Confidence

Families of CDF’s Can Look Alike
§ Cumulative Distribution Functions (CDF) look similar for a
variety of Probability Distribution Functions (pdf)
Cumulative Probability
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200%
Bounds on Point Estimate
Beta
Triangular
Uniform
Probability Density
0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200%
Beta
Triangular
Uniform
Cumulative Probability
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200%
Beta
Triangular
Uniform
Normal
LogNormal
Probability Density
0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200%
Beta
Triangular
Uniform
Normal
LogNormal

Programmatic Risk Analysis 58/186: Managing Uncertainty in the IMS

Approaches To
Uncertainty
“It is moronic to predict without first establishing an
error rate for a prediction and keeping track of
one’s past record of accuracy”
— Nassim Nicholas Taleb, Fooled By Randomness
(where he argues people constantly delude
themselves because they do not understand
probability and are programmed to find reasons for
optimism or pessimism where none exist.)
: Approaches to Uncertainty
If you hang them all,
you’re sure to get the
guilty ones.
– Judge Roy Bean,
“The Law West of
the Pecos”

There Is No Such a Thing as Risk
Neutral Decision Making
§ Can we have a discussion of a probabilistic event without a
discussion of associated risk? (confidence in the confidence
interval)
§ The most severe problem with making decisions in the
presence of risk is when the decision maker doesn’t have the
faintest idea about what risks are to be incurred, yet thinks
there is precision about the decision results.
§ Three features of decision making
– Alternatives – which course of action might be taken?
– Uncertainties – what uncontrolled elements exist?
– Outcomes – Alternatives combined with Uncertainties = Outcomes
Natural Gas?
Rotating equipment?
Open flames?
High voltage?

Types of Uncertainty Based Decisions
Strategy
Describes a collection of actions the decision
maker makes
è
Goal
A possible outcome of the strategy is a goal
Decision
Actually performing a task is a decision è
Prioritization
Deciding which task to perform is a
prioritization
Alternative
A set of choices that are allowable è
Option
An option is an alternative that permits a future
decision following the discovery of new
information
Sensitivity of Decision
Knowing “what” to do
è
Sensitivity of Outcome
Knowing “how” it will turn out
Direct values
Outcomes traceable to bookable benefits to
the project
è
Indirect Values
Things the decision maker value but are
unlikely to be visible in the project
Certain equivalent effect
The smallest sum of money for which the
decision maker would be willing to sell rights
to a risky product
è
Expected Net Present Value
The hypothetical average NPV from numerous
independent launches of identical projects

Approaches to Decisions with Uncertainty
§ Decision trees
§ Line of balance
§ PERT
§ Monte Carlo

Decision Trees
§ Decision Trees are tools for helping
choose between several courses of
action.
§ They provide a effective structure
within which to lay out options and
investigate the possible outcomes of
choosing those options.
§ They also help to form a balanced
picture of the risks and rewards
associated with each possible
course of action.

Line Of Balance
§ The line–of–
balance technique
is based on the
underlying
assumption that
the rate of
production for an
activity is uniform
§ Measurements are
compared against
a specific plan, all
collected in one
“balance” chart

PERT As The Starting Point For
Probabilistic Schedule Management
§ PERT is a method to determine how long a project should take
– Which activities are most critical
– Deterministic PERT and Probabilistic PERT are common
§ PERT algorithms
– Activity duration:
– Activity Standard Deviation:
– Activity Variance:
– Total Standard Deviation:
( )4 /6et a m b= + +
( )/6et b as = -
( )
22
/6ev t b as é ù= = -ë û
( ) ( )
2 2
1 2e e eT t ts s s= +

Deterministic PERT Uses The Three Point
Estimates In A Static Manner
§ Durations are defined as three point estimates
– These estimates are very subjective if captured individually by asking…
– “What is the Minimum, Maximum, and Most Likely”
§ Critical path is defined from
these estimates is the algebraic
addition of three point estimates
§ Project duration is based on the
algebraic addition of the times
along the critical path
§ This approach has some serious
problems from the outset
– Durations must be independent
– Most likely is not the same as
the average

Probabilistic PERT Uses The Underlying
Probability Distributions For Each Task
§ Any path could be critical depending on the convolution of the
underlying task completion time probability distribution functions
§ The independence or
dependency of each task
with others in the network,
greatly influences the
outcome of the total project
duration
§ Understanding this
dependence is critical to
assessing the credibility of
the plan as well as the total
completion time of that plan

Statistics V. Probability – We Need Both To
Build A Risk Tolerant Schedule
§ In building a risk tolerant IMS,
we’re interested in the
probability of a successful
outcome
– “What is the probability of
making a desired completion
date?”
§ But the underlying statistics of
the tasks influence this
probability
§ The statistics of the tasks, their arrangement in a network of tasks and
correlation define how this probability based estimated developed.
Statistical thinking will one day be as necessary for efficient citizenship as
the ability to read and write.
– H.G. Wells

The Problem with PERT
§ When the activity durations are random variables, each path of
the project network is a likely candidate to be the critical path.
§ Every activity
duration could
result in a
different longest
path
§ Evaluating the
distribution of
the longest path,
even under very
specific
distributional is
not an easy
problem

Simple Understanding of Statistics
The number of times a specific
occurrence of a parameter occurs in a
population.
The number of time it snows before
September 15th in Colorado indicates
the confidence that it will snow on or
before 9/15 this year

Inputs to the process have statistics too
§ The statistics of the events that impact the project must be understood
as well – this second order impact is critical
§ These events form a
stochastic process
that drives the
network
§ These drivers may
or may not be
random events
§ Correlations
between the events
can create nonlinear
behaviors in the
sensitivity of the
model

The Program is a System, Just like the Spacecraft
§ The programmatic and planning dynamics act as a system
§ The “system response” is the transfer function between input and
output
§ Understanding this transfer
function may appear
beyond our interest
– But it is part of the
stochastic dynamic
response to disruptions in
our plans
– “What if” really means
“what if” at this point in
the response curve of the
system
Inputs
Outputs

The Beta Distribution
§ The Beta distribution is
given as:
§ Where B is the Beta
function
( ) ( )
( ) ( )
( )
( )
1
11
0
, 1 ,
,
, .
B t t dt
B y
ba
a b
a b
a b
b
--
= -
G G
=
G +
=
ò
( )
( )
( )
1 1
1 x x
P x
B
b a
a b
- -
-
=
-

What does the Beta Distribution Provide?
§ Beta can model events that are constrained to take place within
an interval defined by a minimum and maximum value
§ Turns out the equation for task duration
– Is a empirical approximation formulas not derived from the beta
distribution directly
– The theoretical argument showing the relative weight of 4 on the
modal time is based largely on the Beta distribution where
developed by the Pearson & Tukey based on the 5% points and the
median
– Presumably, some experimentation was done in the early days,
and some empirical basis was found for these forms
– At this point, we merely accept the formulas as the "traditional"
way of doing PERT
§ Beta is used directly in Monte Carlo as a model of task duration
( )4 /6et a m b= + +

Another Alternative, the Triangle Distribution
§ The triangle distribution was proposed in the late 90’s as a substitute
of the Beta distribution
– The parameters of the triangle distribution have a one–to–one
correspondence with the PERT parameters
§ The original “fitting” of PERT equation took place in 1959
§ Triangle distribution is
better behaved in
certain instances, but
the “mean” is still
greater than the
“mode” (the most
likely)
§ The result is overly
optimistic durations for
the tasks

Sensitivity of the PDF: Triangle
§ Triangle distribution
– The “minimum” and the “maximum” are as influential on the mean
on the “most likely”
– The “minimum,” “maximum” and the “modal” values capture
limited information about the underlying distribution
– There can be an infinite number of distributions with these same
three values

Sensitivity of the PDF: Beta PERT
§ Manipulating the standard Beta distribution produces the
BetaPERT (Vose, 2000, pp. 275)
– BetaPERT combines the Beta distribution and the PERT formula
for the mean

Programmatic Risk Analysis 78/186: Approaches to Uncertainty

Gathering Risk
and Mitigation
Information
Getting viable estimates of the
relative risk requires effort, patience
and care not to over or under
estimate the risk or the mitigation
activities.
Magic rabbits are not a reliable
source of information.
: Gathering Risk and Mitigation Information

General Flow of Risk Management
§ Schedule and Cost risk management must be performed in a
structured and rigorous manner
§ Each element below must appear in the IMS

Gathering Risk Analysis Information
§ Interviews with CAMs and IPT Leads about risk creates a “bias”
toward optimism or pessimism – but never toward neutrality
– Optimist says – The glass is ½ full
– Pessimist says – The glass is ½ empty
– An engineer says – The glass is twice as big as it needs to be
§ Asking for three point estimates (optimistic, most likely,
pessimistic) is not the way to do this…
– Gain some sense of the risk “ranking” from the owner of the task
or work package
– Look to any historical data
» Durations
» Causes of Over Target Baseline (OTB) and Over Target Schedule
(OTS)

The Dreaded 3–Point Estimates
§ Optimistic Estimate
– The shortest duration
– “It can’t be done in less time than
this”
§ Most Likely Estimate
– The median time (middle most), not
the mean time (the average)
– This builds in a symmetric
distribution of the probability
distribution function
§ Pessimistic Estimate
– The additional time needed if things
go wrong
– It is not the maximum time it would
take
– It is not a “worst case scenario”
estimate

Classes of Project Risk
§ Delay
– Slow decisions
– Access
– Lack of information
– Time difference
§ Dependency
– Interaction
– Interface
difficulties
– Interruption of
service
§ Estimates
– Mis–estimation
– Learning curves
– Overly aggressive
deadlines
Pareto of Risk Causes

Thinking About Risk Classification
§ These classifications can be used to avoid asking the “3 point”
question for each task
§ This information will be maintained in the IMS
§ When updates are made the percentage change can be applied
across all tasks
Classification Uncertainty Overrun
A Routine, been done before Low 0% to 2%
B Routine, but possible difficulties Medium to Low 2% to 5%
C Development, with little technical difficulty Medium 5% to 10%
D Development, but some technical difficulty Medium High 10% to 15%
E Significant effort, technical challenge High 15% to 25%
F No experience in this area Very High 25% to 50%

Steps in Characterizing Uncertainty in
Task Duration Estimating Data
§ Use an “envelope” method to characterize the minimum,
maximum and “most likely”
§ Fit this data to a statistical distribution
§ Use conservative assumptions
§ Apply greater uncertainty to less mature technologies
§ Confirm analysis matches intuition
Remember Sir Francis Bacon’s quote
about beginning with uncertainty and
ending with certainty.
If we start with a what we think is a
valid number we will tend to continue
with that valid number.
When in fact we should speak only in
terms of confidence intervals and
probabilities of success

Some Sobering Observations
§ In 1979, Tversky and Kahneman proposed an alternative to
utility theory. Prospect theory asserts that people make
predictably irrational decisions. [45], [52]
§ The way that a choice of decisions is presented can sway a
person to choose the less rational decision from a set of
options.
§ Once a problem is clearly and reasonably presented, rarely
does a person think outside the bounds of the frame.
§ Source:
– “The Causes of Risk Taking By Project Managers,” Proceedings of
the Project Management Institute Annual Seminars & Symposium
November 1–10, 2001 • Nashville, Tenn
– Tversky, Amos, and Daniel Kahneman. 1981. The Framing of
Decisions and the Psychology of Choice. Science 211 (January
30): 453–458

The Dark Side of PERT
During the modeling process the “most likely” durations
can not be added since they represent the moments of
the underlying probability distribution (mean, mode,
median, variance)
As well, the phenomenon of “merge bias” at the merge
points of parallel task streams move the completion
probability point to the right in unpredictable ways.
The critical path is often referred to as the laugh track of
the project.
: The Dark Side of PERT

Some Useful (and dreadful) History
§ The original paper (Malcolm 1959) states
– The method is “the best that could be done in a real situation
within tight time constraints.”
– The time constraint was One Month
§ The PERT time made the assumption that the standard
deviation was about 1/6 of the range (b–a), resulting in the
PERT formula.
§ It has been shown that the PERT mean and standard deviation
formulas are poor approximations for most Beta distributions
(Keefer 1983 and Keefer 1993).
– Errors up to 40% are possible for the PERT mean
– Errors up to 550% are possible for the PERT standard deviation

Critical Path and Most Likely
§ Critical Path’s are Deterministic
– At least one path exists through the
network
– The critical path is identified by
adding the “single point” estimates
– The critical predicts the completion
date only if everything goes
according to plan (we all know this
of course)
§ Schedule execution is Probabilistic
– There is a likelihood that some durations will comprise a path that
is off the critical path
– The single number for the estimate – the “single point estimate” is
in fact a most likely estimate
– The completion date is not the most likely date, but is a
confidence interval in the probability distribution function
resulting from the convolution of all the distributions along all the
paths to the completion of the project

Some (False) Assumptions of PERT
§ Three point estimates follow the Beta distribution
– Using the simplified algebraic formula
– But this formula has built in biases not revealed in normal use
§ The expected completion time and variance are calculated by
summing the mean and variance of critical path activities
– The central limit theorem suggests that if there are sufficient
activities on the critical path, the activity times are independent
– If the activity times follow a probabilistic distribution with no one
activity time dominating, then the sum of activity times on the
critical path follows approximately the normal distribution
§ The result are derived completion times using the normal
distribution with a built–in optimistic bias

Merge Bias Must Be Addressed Upfront
Before Management Gets Involved
§ The standard
approach to
schedule risk
analysis
§ A “merge point” is
where two tasks
have a common
successor
§ PERT naively
assumes the pdf’s
are identical for all
tasks

The Architecture of Merge Bias Starts with
Parallel Tasks Landing on a Single Node
§ The Cumulative Distribution Function (CDF) is biased by the
merge points
– The CDF is the source of samples for the Monte Carlo simulation
Effect of the Merge Bias on Schedule Risk
0%
20%
40%
60%
80%
100%
10/6 11/25 1/14 3/4
Date
Cumulative%
One
Path
Three
Path

Merge Point Bias Can Be Very Confusing
and Lead to False Optimism
§ It is misleading to consider only variances from single
predecessors for each node in the critical path
– Early start of a node depends on the maximum of finish (or start)
times of predecessors
– This includes ALL the non–critical paths
§ Early Start is a random variable that is the maximum of all the
non–independent random variables in the network
§ This effect is strongest if
– There are more predecessors
– Predecessors have equal or near equal duration
– Low dependency among the predecessors
§ The result is an unrealistic optimism for the expected
completion times, but most importantly the variance in the
completion times

Notional View of Merge Bias
§ Near critical paths bias the
completion time of the
critical path
§ The finish (or start) times for all
paths are random variables with
individual probability distributions
§ These distributions are “joined” to
form a probability distribution for
the completion of the project

Notional Impact of Merge Bias
§ Activities with duration of 2 have s=0.707
§ Activities with duration of 4 have s=1.414

Why Does This Happen?
§ The completion time of a task in the IMS can be considered a
random variable
– This is a mathematical random variable, independent of our ability
to manage to plan.
– The distributions of random variables can be “added” by
convolving their probability distribution functions.
– This is the case independent of the underlying distribution (Beta,
Triangle, Gauss, Poisson)
§ This is a critically important concept
– No matter what our planning fidelity, the underlying processes of
task completion duration are random variables
– These random variables have predictable and unpredictable
behaviors that impact the outcome is unfavorable ways

Conveying the Effect of Merge Bias is
Sometimes Difficult, But it is Always There
§ Most projects have parallel paths, many times at crucial points in the
schedule – PDR, CDR, ATLO
§ “Merge Bias” creates extra risk at these point but extending the
probabilistic completion date
§ This may be the factual case or it may be the result of the statistics
§ Either way the
answer will likely be
unacceptable to
those without the
underlying
knowledge
§ We must both
educate and inform
before proceeding

Conclusion of Merge Bias
§ The Critical Path is pretty much meaningless at the deal level of
the project
– Dynamic completion times must be modeled with Monte Carlo
§ Discussing the critical path requires that you look at your
watch first to see what time it is
– The critical path is not static
– It is highly dependent on but stochastic behaviors of the task
completion time the emerge from the underlying probability
distributions
– It is also dependent on the dynamics of the interactions of the
network nodes
§ USE MONTE CARLO AND TRY TO AVOID SPEAKING ABOUT THE
CRITICAL PATH IN ANY WORDS OTHER THAN VERY HIGH LEVEL A
STATIC PATH THROUGH THE NETWORK

A Quick Look at Monte Carlo Simulation
Georges Louis Leclerc, Comte
de Buffon, asked what was the
probability that the needle would
fall across one of the lines,
marked here in green. That
outcome will occur only if
sinA l q<
: A Quick Look at Monte Carlo Simulation

Monte Carlo Simulation Provides a Solution
to Merge Bias and Other PERT Issues
§ Yes Monte Carlo is named after
the country full of casinos
located on the French Rivera
§ Advantages of Monte Carlo over
PERT is that Monte Carlo…
– Examines all paths not just the
critical path
– Provides an accurate (true)
estimate of completion
» Overall duration distribution
» Confidence interval (accuracy
range)
– Sensitivity analysis of interacting tasks
– Varied activity distribution types – not restricted to Beta
– Schedule logic can include branching – both probabilistic and conditional
– When resource loaded schedules are used – provides integrated cost and
schedule probabilistic model

The Monte Carlo Tour Starts with WWII
History
§ Any method which solves a problem
by generating suitable random
numbers and observing that fraction
of the numbers obeying some
property.
§ The Monte Carlo method provides
approximate solutions to a variety of
mathematical problems by
performing statistical sampling
experiments on a computer.
§ The method applies to problems with no probabilistic content as well
as to those with inherent probabilistic structure.
§ The method is named after the city of Monte Carlo in the principality of
Monaco, because of a roulette, a simple random number generator. The
name and the systematic development of Monte Carlo methods dates
from about 1944 and the Manhattan project.

The Monte Carlo Method Can Be Found in
Many Scientific and Business Domains
§ Monte Carlo is a well developed discipline in
many areas of science, finance and statistics
§ Probability distribution functions (pdf's) – the
physical (or mathematical) system must be
described by a set of pdf's.
§ Random number generator – a source of
random numbers uniformly distributed on the
unit interval must be available.
§ Sampling rule – a prescription for sampling
from the specified pdf's, assuming the
availability of random numbers on the unit
interval, must be given.
§ Scoring (or tallying) – the outcomes must be accumulated into overall
tallies or scores for the quantities of interest.

The Monte Carlo Method is
Computationally Based
§ Tools like Risk+ are just a sample
of the many approaches to
simulating physical system
§ Error estimation – an estimate of
the statistical error (variance) as a
function of the number of trials and
other quantities must be
determined.
§ Variance reduction techniques –
methods for reducing the variance
in the estimated solution to reduce
the computational time for Monte
Carlo simulation
§ Parallelization and vectorization – algorithms to allow Monte Carlo
methods to be implemented efficiently on advanced computer
architectures.

Monte Carlo is Fundamentally a Sampling
Approach
§ The large the number
of “runs” the higher
the fidelity of the
simulation
§ Size does matter
§ The “sample space” of random
numbers is defined by the
probability distribution
function of the task completion
times
§ Knowing this pdf is important
for quality answers to “how
long will it take?”

Using Monte Carlo Starts with the Three
Point Estimates, but Goes Far Beyond
§ Conceptually simple approach
– The normal PERT 3–Point estimate can be used (reused)
– There is no need for special assumption about the underlying
probability distributions of the completion times
§ The Criticality Index is provided from the simulation
– Which tasks are critically impacting the completion time of the
project
§ It is computationally expensive
– But desktop computing can keep up (3GHz with 2GB of memory is
baseline machine for any serious Monte Carlo work)
§ Scalable analysis quality
– Small runs for testing assumptions: £ 1,000 runs
– Larger runs for validation of assumptions: ³ 10,000 runs

Risk+ Quick Overview
Task to “watch”
(Number3)
Most Likely
(Duration3)
Pessimistic
(Duration2)
Optimistic
(Duration1)
Distribution
(Number1)

Risk+ Quick Overview
§ The height of each box
indicates how often the project
complete in a given interval
during the run
§ The S–Curve shows the
cumulative probability of
completing on or before a given
date.
§ The standard deviation of the
completion date and the 95%
confidence interval of the
expected completion date are in
the same units as the “most
likely remaining duration” field
in the schedule
Date: 9/26/2005 2:14:02 PM
Samples: 500
Unique ID: 10
Name: Task 10
Completion Std Deviation: 4.83 days
95% Confidence Interval: 0.42 days
Each bar represents 2 days
Completion Date
Frequency
CumulativeProbability
3/1/062/10/06 3/17/06
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16 Completion Probability Table
Prob ProbDate Date
0.05 2/17/06
0.10 2/21/06
0.15 2/22/06
0.20 2/22/06
0.25 2/23/06
0.30 2/24/06
0.35 2/27/06
0.40 2/27/06
0.45 2/28/06
0.50 3/1/06
0.55 3/1/06
0.60 3/2/06
0.65 3/3/06
0.70 3/3/06
0.75 3/6/06
0.80 3/7/06
0.85 3/8/06
0.90 3/9/06
0.95 3/13/06
1.00 3/17/06
Task to “watch”
80% confidence
that task will
complete by
3/7/06

A Well Formed Risk+ Schedule
§ A good critical path network
– No constraint dates
– Lowest level tasks have predecessors
and successors
– 80% of relationships are finish to start
§ Identify risk tasks
– These are “reporting tasks”
– Identify the preview task to watch during
simulation runs
§ Enter probability distribution profile for each task
– Bulk assignment is an easy way to start
– 1 – 5 ranking is another approach
– Individual risk profile assignments is best but tedious

Analyzing the Risk+ Simulation
§ Risk + will generate one or
more of the following outputs:
– Earliest, expected, and latest
completion date for each
reporting task
– Graphical and tabular
displays of the completion
date distribution for each
reporting task
– The standard deviation and
confidence interval for the
completion date distribution
for each reporting task
– The criticality index
(percentage of time on the
critical path) for each task
– The duration mean and standard deviation for each task
– Minimum, expected, and maximum cost for the total project
– Graphical and tabular displays of cost distribution for the total project
– The standard deviation and confidence interval for cost at the total project
level

Programmatic Risk Analysis 110/186: A Quick Look at Monte Carlo Simulation

Building a Robust IMS
Never undertake a project unless it is
manifestly important and nearly
impossible.
— Edwin Land
: Building a Robust IMS
Edwin Land 1909 – 1991

Schedule Contingency Analysis
§ The schedule contingency
needed to make the plan
credible can be derived from
the Risk+ analysis
§ The schedule contingency is
the amount of time added (or
subtracted) from the baseline
schedule necessary to achieve
the desired probability of an
under run or over run.
§ The schedule contingency can be determined through
– Monte Carlo simulations (Risk+)
– Best judgment from previous experience
– Percentage factors based on historical experience
– Correlation analysis for dependency impacts
Is This Our
Contingency
Plan ?

Schedule Quality and Accuracy
§ Accuracy range
– Similar for each estimate class
§ Consistent with estimate
– Level of project definition
– Purpose
– Preparation effort
§ Monte Carlo simulation
– Analysis of results shows quality attained versus the quality
sought (expected accuracy ranges)
§ Achieving specified accuracy requirements
– Select value at end points of confidence interval
– Calculate percentages from base schedule completion date,
including the contingency

Technical Performance Measures
§ Technical Performance Measures are one method of showing risk
by done
– Specific actions taken in the IMS to move the compliance forward
toward the goal
§ Activities that
assessing the
increasing
compliance to the
technical
performance
measure can be
show in the IMS
– These can be
Accomplishment
Criteria

Elements of a Risk Tolerant Program
CAIV
APB
Schedule
Risk
Cost
Risk
Technical
Risk
• Critical Path
• Monte Carlo
• IMS Maturity
• Systems Engineering
• Simulation & Test
• Design
• TPM
• WBS
• Cost Estimates
• EVMS
CAIV – Cost as an Independent Variable
APB – Allocated Program Baseline
When the three
independent variables of
the program are FIXED in
the beginning.
Building in risk tolerance
now becomes a real
challenge

Inserting Risk Tolerant in the IMS
§ Technical risk management using ARM
– Identification
– Classification
– Triage
§ Programmatic risk management
– Mitigation tasks
– Maturity enhancement
– Technical Performance Measures
§ Tracking and Communication
– Risk ID embedded in the IMS
– Resource assignment
– Cost model cross connections

Moving forward
§ The previous approaches to
the development of an IMS
was to build the IMP and
then the IMS and all the
supporting tasks.
§ The tasks define the work
effort to deliver the ACs and
the product
§ Simple RISK+ metrics run at
the end
§ Explicit risk mitigation provides another view
– Risks identified in ARM
– Mitigation tasks for those risks are made explicit in the IMS
– Branching probabilities for each mitigation initiation events
– Monte Carlo analysis of the main stream and mitigation effort

Integrating Risk and Schedule
§ Probabilistic
completion times
change as the
program matures
§ The efforts that
produce these
improvements
must be traceable
in the IMS
§ The “error bands”
on the events must
include the risk
mitigation activities
as well
§ IMS activities show how the “error band” narrows over time.
– This is the basis of a “programmatic risk tolerant” IMS
– The probabilistic interval becomes more reliable as risk mitigations and
maturity assessments add confidence the to IMS
Baseline
Plan
80%
Mean
Missed
Launch
Period
Launch
Period
Ready
Early
Oct 07
Nov 07
Dec 07
Jan 08
Feb 08
Mar 08
Apr 08
May 08
Jun 08
Plan
Margin
Current Plan
with risks is the
stochastic schedule
CDR
PDR
SRR
FRR
ATLO
20%
Aug 05 Jan 06 Aug 06 Mar 07 Dec 07 Feb 08
Current Plan
with risks is the
deterministic schedule
Risk
Margin

Event Driven Risk Management
§ Water fall risk management
integrated with the IMP or
high level IMS
§ We’re missing this
connection at the moment
§ But making it visible in the
simplest manner is the first
step in building awareness
§ Technical risk management
and programmatic risk
management must be
integrated into the IMS

Steps in the Process of a Risk Tolerant
IMS
1. Identifying the risks
a) ARM and risk trace number
b) Mitigation tasks
c) Duration and resources for the mitigation
2. Integrate risk in the IMS
a) Embedded tasks with IMP/IMS numbers
b) Network tasks
c) Assess Monte Carlo impact on completion dates
3. Evaluate outcome
a) Events and SAs within acceptable confidence interval?
b) If not, risk mitigation activities need improvement
c) If so, baseline mitigation and narrative supporting the strategy
4. Connect Risk Management (ARM) and IMS at the tracking
number level

Branching Probabilities – Simple Approach
§ Plan the risk alternatives that
“might” be needed
– Each mitigation has a Plan B
branch
– Keep alternatives as simple as
possible (maybe one task)
§ Assess probability of the
alternative occurring
§ Assign duration and resource
estimates to both branches
§ Turn off for alternative for a
“success” path assessment
§ Turn off primary for a “failure”
path assessment
30% Probability
of failure
70% Probability
of success
Plan B
Plan A Current Margin Future Margin
80% Confidence for completion
with current margin
Duration of Plan B Plan A + Margin£

Managing Margin in the Risk Tolerant IMS
requires the reuse of unused durations
§ Programmatic Margin is added
between Development, Production
and Integration & Test phases
§ Risk Margin is added to the IMS
where risk alternatives are identified
§ Margin that is not used in the IMS for
risk mitigation will be moved to the
next sequence of risk alternatives
– This enables us to buy back schedule
margin for activities further
downstream
– This enables us to control the ripple
effect of schedule shifts on Margin
activities
5 Days Margin
5 Days Margin
Plan B
Plan A
Plan B
Plan AFirst Identified Risk Alternative in IMS
Second Identified Risk
Alternative in IMS
3 Days Margin Used
Downstream
Activities shifted to
left 2 days
Duration of Plan B < Plan A + Margin
2 days will be added
to this margin task
to bring schedule
back on track

Simulation Considerations of Monte Carlo
need to be understood to use the data
§ Multiple Critical Paths
– Microsoft Project™ lacks the capability to:
» Identify off critical path impacts on the critical path
» Identify effects of branching
– This can be manually compensated for by:
» Examine each activity’s criticality associated wit more than one
critical path
» Path analysis may give substantially different vie of the schedule’s
uncertainties
» Use bar chart to illustrate multiple critical paths
§ Monte Carlo examines ALL paths not just the critical path
– Near critical paths are exposed

Simulation Considerations
§ Schedule logic and constraints
– Simplify logic – model only paths which, by inspection, may have a
significant bearing on the final result
– Correlate similar activities
– No open ends
– Use only finish–to–start relationships with no lags
– Model relationships other than finish–to–start as activities with
base durations equal to the lag value
– Eliminate all date constraints
– Consider using branching for known alternatives

Simulation Considerations
§ Selection of Probability Distributions
– Develop schedule simulation inputs concurrently with the cost
estimate
» Early in process – use same subject matter experts
» Convert confidence intervals into probability duration distributions
– Number of distributions vary depending on software
– Difficult to develop inputs required for distributions
– Beta and Lognormal better than triangular; avoid exclusive use of
Normal distribution

Sensitivity Analysis describes which tasks
drive the completion times
§ Concentrates on inputs most likely to improve quality
(accuracy)
§ Identifies most promising opportunities where additional work
will help to narrow input ranges
§ Methods
– Run multiple simulations
– Use criticality index
– “Tornado” or Pareto graph

Models of the Schedule
All models are lies. Some models are
useful.
– George Box
: Models of the Schedule
Concept generator from Ramon
Lull’s Ars Magna (C. 1300)

Graphical Interpretations provide
information that numbers alone can’t
§ Graphical view of
confidence,
contingency and
target
management
assessment

What Can Confidence Intervals Tell Us
about the validity of the IMS?
§ As the program
proceeds so does
– Increasing
accuracy
– Reduced
schedule risk
– Increasing
visual
confirmation
that success
can be reached Current Estimate Accuracy

Confidence of schedule dates

Sensitivity Analysis
§ The schedule sensitivity of a task measures the closeness with
which change in the task duration matches change in the
project duration over the simulation.
§ This closeness is called correlation
– Correlations can be derived from the sampling processes
§ A task with high schedule sensitivity is more likely to be a
major driver of the project duration than a lower ranked task

Task Criticality Analysis
§ A measure of the frequency that an activity in the project
schedule is critical (Total Float = 0) in a simulation
§ If a task is critical in 500 of the 1,000 iterations of the
simulation, it has a Criticality Index of 0.5
§ The higher the criticality index, the more certain it is that the
task will always be critical in the project

Schedule Cruciality describes how
important each tasks is the risk tolerance
§ Cruciality = Schedule Sensitivity X Criticality
§ Schedule Sensitivity can be statistically misleading:
– A task with high sensitivity may not be on or near the critical path
– Thus a reduction in that task’s duration may have little effect on
the project duration
§ Cruciality sharpens the analytical focus:
– It highlights critical or near–critical activities with high
§ Schedule Sensitivity
– These tasks are most likely to drive project duration

Programmatic Risk Analysis 134/186: Models of the Schedule

Examples Of Monte
Carlo
A simple overview of Risk+ shows how to
produce an estimate of a project
completion date. Interpreting this
information takes thought and practice
and most importantly reading and
rereading the manual and the resources
provided at the end of this presentation
: Examples of Monte Carlo
Sewall Wright’s
probabilistic
network notation
(1921)

The Monte Carlo Process starts with the
PERT 3 point estimates
§ Estimates of the task duration are still needed, just like they are
in PERT
– Three point estimates could be used
– But risk ranking and algorithmic generation of the “spreads” is a
better approach
§ Duration estimates must be parametric rather than numeric
values
– A geometric scale of parametric risk is one approach
§ Branching probabilities need to be defined
– Conditional paths through the schedule can be evaluated using
Monte Carlo tools
– This also demonstrate explicit risk mitigation planning to answer
the question “what if this happens?”

Expert Judgment is required to build a
Risk Management approach
§ Expert judgment is typically the basis of cost and schedule
estimates
– Expert judgment is usually the weakest area of process and
quantification
– Translating from English (SOW) to mathematics (probabilistic risk
model) is usually inconsistent at best and erroneous at worst
§ One approach
– Plan for the “best case” and preclude a self–fulfilling prophesy
– Budget for the “most likely” and recognize risks and uncertainties
– Protect for the “worst case” and acknowledge the conceivable in
the risk mitigation plan
§ The credibility of the “best case” estimates if crucial to the
success of this approach

Guiding the Risk Factor Process requires
careful weighting of each level of risk
§ For tasks marked “Low” a reasonable
approach is to score the maximum
10% greater than the minimum.
§ The “Most Likely” is then scored as a
geometric progression for the
remaining categories with a common
ratio of 1.5
§ Tasks marked “Very High” are bound
at 200% of minimum.
– No viable project manager would like a
task grow to three times the planned
duration without intervention
§ The geometric progress is somewhat
arbitrary but it should be used instead
of a linear progression
Min Most
Likely
Max
Low 1.0 1.04 1.10
Low+ 1.0 1.06 1.15
Moderate 1.0 1.09 1.24
Moderate+ 1.0 1.14 1.36
High 1.0 1.20 1.55
High+ 1.0 1.30 1.85
Very High 1.0 1.46 2.30
Very High+ 1.0 1.68 3.00

Progressive Risk Factors
§ A geometric progression (1.534) of risk can be used
§ The phrases associated with increasing risk have been shown
at the Naval Research Laboratory to correlate with an engineers
“sense” of increasing risk

Risk Factor Attributes
§ The “narrative” for each risk factor
needs to be developed
§ Each description may be dependent
on…
– Discipline
– Program stage
– Complexity
– Historical data
– Current “risk state” of the program
§ This approach is similar to NASA’s
Technology Readiness Level
§ This is currently missing from our
efforts to quantify schedule and cost
risk

Extreme Serial Task Example
§ Task completion
distributions are
“added” with little
effect on the project
completion estimate
Date: 9/26/2005 2:03:28 PM
Samples: 500
Unique ID: 10
Name: Task 10
Each bar represents 1 day
Completion Date
Frequency
2/10/062/1/06 2/23/06
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
Prob ProbDate Date
0.05 2/6/06
0.10 2/7/06
0.15 2/8/06
0.20 2/8/06
0.25 2/9/06
0.30 2/9/06
0.35 2/10/06
0.40 2/10/06
0.45 2/10/06
0.50 2/10/06
0.55 2/13/06
0.60 2/13/06
0.65 2/13/06
0.70 2/14/06
0.75 2/14/06
0.80 2/14/06
0.85 2/15/06
0.90 2/16/06
0.95 2/17/06
1.00 2/23/06

Extreme Parallel Task Example
Date: 9/26/2005 2:14:02 PM
Samples: 500
Unique ID: 10
Name: Task 10
Each bar represents 2 days
Completion Date
Frequency
CumulativeProbability3/1/062/10/06 3/17/06
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.02
0.04
0.06
0.08
0.10
0.12
0.14
Prob ProbDate Date
0.05 2/17/06
0.10 2/21/06
0.15 2/22/06
0.20 2/22/06
0.25 2/23/06
0.30 2/24/06
0.35 2/27/06
0.40 2/27/06
0.45 2/28/06
0.50 3/1/06
0.55 3/1/06
0.60 3/2/06
0.65 3/3/06
0.70 3/3/06
0.75 3/6/06
0.80 3/7/06
0.85 3/8/06
0.90 3/9/06
0.95 3/13/06
1.00 3/17/06
§ Task completion
distributions are
“added” with a large
unfavorable effect on
the project completion
estimate distribution

A “Real World”
Schedule Analysis
One should expect that the
expected can be prevented,
but the unexpected should
have been expected
— Augustine Law XLV
: “Real World” Schedule Analysis

The Baseline Schedule
§ Sample construction project plan

PERT Assessment
Most
Likely
Min Max
PERT Adjusted Duration
PERT Adjusted Date
Original Target Date: 2/8/06

Risk+ Assessment
Most
Likely
Min Max
Date: 10/4/2005 1:58:06 PM
Samples: 23
Unique ID: 143
Name: Construction Schedule Margin
Each bar represents 1 day
Completion Date
Frequency
2/9/062/3/06 2/14/06
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.05
0.10
0.15
0.20
0.25
0.30
Prob ProbDate Date
0.05 2/6/06
0.10 2/6/06
0.15 2/7/06
0.20 2/7/06
0.25 2/7/06
0.30 2/8/06
0.35 2/8/06
0.40 2/9/06
0.45 2/9/06
0.50 2/9/06
0.55 2/9/06
0.60 2/9/06
0.65 2/9/06
0.70 2/10/06
0.75 2/10/06
0.80 2/10/06
0.85 2/10/06
0.90 2/10/06
0.95 2/10/06
1.00 2/14/06
Target Date
80% confidence

What is the Purpose
of Project Risk
Analysis?
What do users want from a project risk
analysis?
How accurate must we be to provide
value to the program?
Can we confirm this accuracy and
integrity to build confidence in the
projected completion date?
: What is the Purpose of Project Risk Analysis?
Risk appears in all aspects of spaceflight

Accuracy
§ Given a specified final cost or project duration, what is the
probability of achieving this cost or duration?
§ Frequentist approach
– Over many different projects, four out of five will cost less or be
completed in less time than the specified cost or duration
§ Bayesian approach
– We would be willing to bet at 4 to 1 odds that the project will be
under the 80% point in cost or duration
§ Accuracy is needed to plan reserves
§ Accuracy is needed when comparing competing proposals

Structured Thinking
§ All estimates will be in error
§ Trying to quantify these errors will result in bounds too wide to
be useful for decision making
§ Risk analysis should be used to
– Think about different aspects of the project
– Try to put numbers against probabilities and impacts
– Discuss with colleagues the different ideas and perceptions
§ Thinking things through carefully results in
– Which elements of the programmatic and technical risk are
represented in the IMS
– The process becomes more valuable than the numbers

Programmatic Risk Analysis 150/186: What is the Purpose of Project Risk Analysis?

Basic Principles of
Probabilistic Cost
Now that the schedule can be produced using
probabilistic methods, it’s time to talk about the
cost.
Cost does not have a linear relationship with
schedule unfortunately
: Basic Principles of Probabilistic Cost

Basic Principles with Probabilistic Cost
Estimating
§ Cost estimates usually involve many CERs
– Each of these CERs has uncertainty (standard error)
– CER input variables have uncertainty (technical uncertainty)
§ Must combine CER uncertainty with technical uncertainty for many
CERs in an estimate
– Usually cannot be done arithmetically; must use simulation to roll up costs
derived from Monte Carlo samples
» Add and multiply probability distributions rather than numbers
» Statistically combining many uncertain, or randomly varying, numbers
– Monte Carlo simulation
» Take random sample from each CER and input parameter, add and multiply
as necessary, then record total system cost as a single sample
» Repeat the procedure thousands of times to develop a frequency histogram
of the total system cost samples
» This becomes the probability distribution of total system cost

The Cost Probability Distributions as a
function of the weighted cost drivers
$
Cost Driver (Weight)
Cost = a + bXc
Cost
Estimate
Historical data point
Cost estimating relationship
Standard percent error boundsTechnical Uncertainty
Combined Cost
Modeling and Technical
Uncertainty
Cost Modeling
Uncertainty

Basic Principles of connected Cost with
the IMS involve three steps
§ Step 1: Define “likely–to–be” program
– Using deterministic inputs from the Independent Technical
Assessment (ITA)
§ Step 2: Quantify the probability distributions describing the
modeling uncertainty of all CERs, cost factors, and other
estimating methods
– Specifically, the type of distribution (normal, triangular, lognormal,
beta, etc.)
– The mean and variance of the distribution
§ Step 3: Quantify the correlation between all WBS elements that
are estimated using CERs and other methods
– If unknown, assess whether No correlation, Mild correlation, or
High correlation, for example:
» None: r = 0, Mild: r = ±0.2, High: r = ± 0.6
– Correlation affects the overall cost variance

Basic Principles
§ Step 4: Set up and run the cost estimate in a Monte Carlo
framework (e.g., Crystal Ball, @RISK), resulting in a “baseline”
estimate
– This will provide a probability distribution of the cost based on
cost estimating model uncertainty only
– Report the MEAN as the baseline expected cost
§ Step 5: Now incorporate technical uncertainty and discrete
risks
– Step 5a: Set up a new estimate which also contains any “discrete
risk” events that are to be guarded against
» Quantify appropriate modeling uncertainties and correlations, as in
Steps 2 and 3, for these discrete risks
– Step 5b: Define the probability distributions for all CER input
variables
» Also may need to quantify correlation between CER input variables

Basic Principles
§ Step 6: Re–run the Monte Carlo simulation with random CER
input variables and discrete risk events, resulting in a final
“risk–adjusted” estimate
– Results in a new risk–adjusted cost probability distribution.
– Wider and shifted to the right
Baseline vs. Risk-Adjusted Estimates
0 50 100 150 200 250 300 350
FY$M
Likelihood

Basic Principles
§ Step 7: Assess “risk dollars”
– Difference between the “risk–adjusted” mean and the “baseline”
mean represents the estimate of “risk dollars”
– Risk dollars can be allocated downward to any level of WBS using
a variety of simple approaches
Risk-Adjusted Estimate
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
Risk-Adjusted
Estimate $346M
Baseline
Estimate $270M
Risk Dollars
Relative to Baseline
Estimate $76M

Basic Principles
§ Step 8: Assess “budget risk”
– Area under the PDF to the right of the budget represents budget
risk
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget Risk
= 51%
Budget Risk
= 51%
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget Risk
= 51%
Budget Risk
= 45%
Note: Assumes budget is set
at the risk-adjusted estimate
expected value.
Note: Assumes budget is set
at the risk-adjusted estimate
expected value.
Budget
= $346M
Budget
= $346M

Basic Principles
§ Step 9: Assess “management reserve”
– Difference between mean of risk–adjusted estimate and budget
represents management reserve
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget
$405M
Budget
$405M
Unencumbered
Margin $59M
Unencumbered
Margin $59M
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget
$405M
Budget
$405M
Unencumbered
Margin $59M
Unencumbered
Margin $59M
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
0 100 200 300 400 500 600 700 800
FY$M
Likelihood
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget
$405M
Budget
$405M
Unencumbered
Margin $59M
Unencumbered
Margin $59M
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget
$405M
Budget
$405M
Unencumbered
Margin $59M
Management
Reserve $59M
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget
$405M
Budget
$405M
Unencumbered
Margin $59M
Unencumbered
Margin $59M
Risk-Adjusted
Estimate
$346M
Risk-Adjusted
Estimate
$346M
Budget
$405M
Budget
$405M
Unencumbered
Margin $59M
Management
Reserve $59M
Note: If budget is set at risk-
adjusted expected value, then
management reserve is ZERO.
Note: If budget is set at risk-
adjusted expected value, then
management reserve is ZERO.

The Risk Adjusted Cost Estimate Connected
To The IMS Is The Basis Of Risk Tolerance
§ In the risk–adjusted cost estimate, we now combine discrete
risk events and the uncertainty of the input distributions with
the uncertainty of the CERs
§ Since the input distributions tend to be right–skewed, the
expected cost tends to be larger than the baseline estimate
§ In addition, the risk–adjusted cost distribution tends to be
wider than the baseline estimate
§ The difference between the expected cost of the risk–adjusted
estimate and the expected cost of the baseline estimate is, by
definition, the amount of RISK dollars included in the risk–
adjusted estimate

Baseline versus Risk Adjusted Cost Estimates
Almost Always Show an Increase In Cost
Baseline vs. Risk-Adjusted Estimates
0 50 100 150 200 250 300 350
FY$M
Likelihood
Baseline:
Mean = $102.6M
Std Dev = $29.8M
Risk–Adjusted:
Mean = $122.6M
Std Dev = $42.8M

The S–Curve for Cost Modeling
Cumulative Distribution Function
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
$60 $80 $100 $120 $140 $160 $180 $200
FY00$M
Baseline Estimate
Mean $102.6M
50th percentile
$114.7M
Risk–adjusted
Estimate Mean
$122.6M
80th percentile
$153.5M

The Real Question Always Returns to…
“But How Much Does It Cost? Really?”
§ This is impossible to answer precisely
§ Decision–makers and cost analysts should always think of a
cost estimate as a probability distribution, NOT as a
deterministic number
§ The best we can provide is the probability distribution – If we
think we can be any more precise, we’re fooling ourselves
§ It is up to the decision–maker to decide where he/she wants to
set the budget
§ The probability distribution provides a quantitative basis for
making this determination
– Low budget = high probability of overrun
– High budget = low probability of overrun

Programmatic Risk Analysis 164/186: Basic Principles of Probabilistic Cost

Summary
At this point there is too much information.
Processing of this information will take time,
patience, and most of all practice with the
tools and the results they produce.
But there are some fundamental conclusions
that can be applied to our problem at hand –
Phase II
: Summary
I’ll be happy to give you innovative
thinking. What are my guidelines?
With our new found knowledge we need
to break out of the habits of the past and
start applying probabilistic cost and
schedule analysis to our program

Conclusions
§ Project schedule status must be assessed in terms of a critical
path through the schedule network
§ Because the actual durations of each task in the network are
uncertain (they are random variables following a probability
distribution function), the project schedule duration must be
modeled statistically
: Summary

Conclusions
§ Quality (accuracy) is measured at the end points of achieved
confidence interval (suggest 80% level)
§ Simulation results depend on:
– Accuracy and care taken with base schedule logic
– Use of subject matter experts to establish inputs
– Selection of appropriate distribution types
– Through analysis of multiple critical paths
– Understanding which activities and paths have the greatest
potential impact
: Summary

Conclusions
§ Cost and schedule estimates are made up of many independent
elements.
– When each element is planned as best case – e.g. a probability of
achievement of 10%
– The probability of achieving best case for a two–element estimate
is 1%
– For three elements, 0.01%
– For many elements, infinitesimal
– In effect, it is zero.
§ In the beginning no attempt should be made to distinguish
between risk and uncertainty
– Risk involves uncertainty but it is indeed more
– For initial purposes it is unimportant
– The effect is combined into one statistical factor called “risk,”
which can be described by a single probability distribution
function
: Summary

Points To Remember When we Build The
Risk Tolerant IMS
§ Good project management is good risk management
§ Risk management is how adults manage projects
§ The only thing we manage is project risk
§ Risks impact objectives
§ Risks come from the decisions we make while trying to achieve
the objectives
§ Risks require a factual condition and have potential negative
consequences that must be mitigated in the schedule
: Summary

Programmatic Risk Analysis 170/186: Basic Principles of Probabilistic Cost

More Opportunities to
Improve our Processes
Here and elsewhere, we shall not obtain the
best insights into things until we actually see
them growing from the beginning.
— Aristotle
: Summary

Improvement Opportunities
§ Statistical dependence of risk related activities
§ Construction of probability distributions from actual project
data
§ Connecting cost and schedule risk into a “program” risk
assessment model
§ Causal models versus regression model
: Summary

The End
This is actually the beginning,
since building a risk tolerant,
credible, robust IMS requires
constant “execution” and
corrects to the plan.
A planning algorithm from
Aristotle’s De Motu Animalium c.
400 BC
: Summary

Resources for
Probabilistic
Risk Analysis
Knowledge is of two kinds;
we know a subject
ourselves, or we know
where we can find
information upon it.
— Samuel Johnson
: Resources

Resources
1. “The Parameters of the Classical PERT: An Assessment of its Success,”
Rafael Herrerias Pleguezuelo,
http://www.cyta.com.ar/biblioteca/bddoc/bdlibros/pert_van/PARAMETROS.PDF
2. “Advanced Quantitative Schedule Risk Analysis,” David T. Hulett, Hulett &
Associates, http://www.projectrisk.com/index.html
3. “Schedule Risk Analysis Simplified,” David T. Hulett, Hulett & Associates,
http://www.projectrisk.com/index.html
4. “Project Risk Management: A Combined Analytical Hierarchy Process and
Decision Tree Approach,” Prasanta Kumar Dey, Cost Engineering, Vol. 44, No.
3, March 2002.
5. “Adding Probability to Your ‘Swiss Army Knife’,” John C. Goodpasture,
Proceedings of the 30th Annual Project Management Institute 1999 Seminars
and Symposium, October, 1999.
6. “Modeling Uncertainty in Project Scheduling,” Patrick Leach, Proceedings of
the 2005 Crystal Ball User Conference
7. “Near Critical Paths Create Violations in the PERT Assumptions of Normality,”
Frank Pokladnik and Robert Hill, University of Houston, Clear Lake,
http://www.sbaer.uca.edu/research/dsi/2003/procs/237–4203.pdf
: Resources

Resources
8. “Teaching SuPERT,” Kenneth R. MacLeod and Paul F. Petersen, Proceedings
of the Decision Sciences 2003 Annual Meeting, Washington DC,
http://www.sbaer.uca.edu/research/dsi/2003/by_track_paper.html
9. “The Beginning of the Monte Carlo Method,” N. Metropolis, Los Alamos
Science, Special Issue, 1987.
http://www.fas.org/sgp/othergov/doe/lanl/pubs/00326866.pdf
10. “Defining a Beta Distribution Function for Construction Simulation,” Javier
Fente, Kraig Knutson, Cliff Schexnayder, Proceedings of the 1999 Winter
Simulation Conference.
11. “The Basics of Monte Carlo Simulation: A Tutorial,” S. Kandaswamy,
Proceedings of the Project Management Institute Annual Seminars &
Symposium, November, 2001.
12. “The Mother of All Guesses: A User Friendly Guide to Statistical Estimation,”
Francois Melese and David Rose, Armed Forces Comptroller, 1998,
http://www.nps.navy.mil/drmi/graphics/StatGuide–web.pdf
13. “Inverse Statistical Estimation via Order Statistics: A Resolution of the Ill–
Posed Inverse problem of PERT Scheduling,” William F. Pickard, Inverse
Problems 20, pp. 1565–1581, 2004
: Resources

Resources
14. “Schedule Risk Analysis: Why It Is Important and How to Do It, “Stephen A.
Book, Proceedings of the Ground Systems Architecture Workshop (GSAW
2002), Aerospace Corporation, March 2002,
http://sunset.usc.edu/GSAW/gsaw2002/s11a/book.pdf
15. “Evaluation of the Risk Analysis and Cost Management (RACM) Model,”
Matthew S. Goldberg, Institute for Defense Analysis, 1998.
http://www.thedacs.com/topics/earnedvalue/racm.pdf
16. “PERT Completion Times Revisited,” Fred E. Williams, School of Management,
University of Michigan–Flint, July 2005,
http://som.umflint.edu/yener/PERT%20Completion%20Revisited.htm
17. “Overcoming Project Risk: Lessons from the PERIL Database,” Tom Hendrick ,
Program Manager, Hewlett Packard, 2003,
http://www.failureproofprojects.com/Risky.pdf
18. “The Heart of Risk Management: Teaching Project Teams to Combat Risk,”
Bruce Chadbourne, 30th Annual Project Management Institute 1999 Seminara
and Symposium, October 1999,
http://www.risksig.com/Articles/pmi1999/rkalt01.pdf
: Resources

20. Project Risk Management Resource List, NASA Headquarters Library,
http://www.hq.nasa.gov/office/hqlibrary/ppm/ppm22.htm#art
21. “Quantify Risk to Manage Cost and Schedule,” Fred Raymond, Acquisition
Quarterly, Spring 1999, http://www.dau.mil/pubs/arq/99arq/raymond.pdf
22. “Continuous Risk Management,” Cost Analysis Symposium, April 2005,
http://www1.jsc.nasa.gov/bu2/conferences/NCAS2005/papers/5C_–
_Cockrell_CRM_v1_0.ppt
23. “A Novel Extension of the Triangular Distribution and its Parameter
Estimation,” J. Rene van Dorp and Samuel Kotz, The Statistician 51(1), pp. 63 –
79, 2002.
http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/TheStatistician20
02.pdf
24. “Distribution of Modeling Dependence Cause by Common Risk Factors,”
J. Rene van Dorp, European Safety and Reliability 2003 Conference
Proceedings, March 2003,
http://www.seas.gwu.edu/~dorpjr/Publications/ConferenceProceedings/Esrel20
03.pdf
Resources
: Resources

Resources
25. “Improved Three Point Approximation To Distribution Functions For
Application In Financial Decision Analysis,” Michele E. Pfund, Jennifer E.
McNeill, John W. Fowler and Gerald T. Mackulak, Department of Industrial
Engineering, Arizona State University, Tempe, Arizona,
http://www.eas.asu.edu/ie/workingpaper/pdf/cdf_estimation_submission.pdf
26. “Analysis Of Resource–constrained Stochastic Project Networks Using
Discrete–event Simulation,” Sucharith Vanguri, Masters Thesis, Mississippi
State University, May 2005, http://sun.library.msstate.edu/ETD–
db/theses/available/etd–04072005–
123743/restricted/SucharithVanguriThesis.pdf
27. “Integrated Cost / Schedule Risk Analysis,” David T. Hulett and Bill Campbell,
Fifth European Project Management Conference, June 2002.
28. “Risk Interrelation Management – Controlling the Snowball Effect,” Olli
Kuismanen, Tuomo Saari and Jussi Vähäkylä, Fifth European Project
Management Conference, June 2002.
29. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth
Century, David Salsburg, W. H. Freeman, 2001
: Resources

Resources
30. “Triangular Approximations for Continuous Random Variables in Risk Analysis,”
David G. Johnson, The Business School, Loughborough University,
Liecestershire.
31. “Statistical Dependence through Common Risk Factors: With Applications in
Uncertainty Analysis,” J. Rene van Dorp, European Journal of Operations
Research, Volume 161(1), pp. 240–255.
32. “Statistical Dependence in the risk analysis for Project Networks Using Monte
Carlo Methods,” J. Rene van Dorp and M. R. Dufy, International Journal of
Production Economics, 58, pp. 17–29, 1999.
http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Prodecon1999.pdf
33. “Risk Analysis for Large Engineering Projects: Modeling Cost Uncertainty for
Ship Production Activities,” M. R. Dufy and J. Rene van Dorp, Journal of
Engineering Valuation and Cost Analysis, Volume 2. pp. 285–301,
http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/EVCA1999.pdf
34. “Risk Based Decision Support techniques for Programs and Projects,” Barney
Roberts and David Frost, Futron Risk Management Center of Excellence,
http://www.futron.com/pdf/RBDSsupporttech.pdf
: Resources

Resources
35. Probabilistic Risk Assessment Procedures Guide for NASA Managers and
Practitioners, Office of Safety and Mission Assurance, April 2002.
http://www.hq.nasa.gov/office/codeq/doctree/praguide.pdf
36. “Project Planning: Improved Approach Incorporating Uncertainty,” Vahid
Khodakarami, Norman Fenton, and Martin Neil, Track 15 EURAM2005:
“Reconciling Uncertainty and Responsibility” European Academy of
Management.
http://www.dcs.qmw.ac.uk/~norman/papers/project_planning_khodakerami.p
df
37. “A Distribution for Modeling Dependence Caused by Common Risk Factors,”
J. Rene van Dorp, European Safety and Reliability 2003 Conference
Proceedings, March 2003.
38. “Probabilistic PERT,” Arthur Nadas, IBM Journal of Research and
Development, 23(3), May 1979, pp. 339–347.
39. “Ranked Nodes: A Simple and effective way to model qualitative in large–
scale Bayesian Networks,” Norman Fenton and Martin Neil, Risk Assessment
and Decision Analysis Research Group, Department of Computer Science,
Queen Mary, University of London, February 21, 2005
: Resources

Resources
40. “Quantify Risk to Manage Cost and Schedule,” Fred Raymond, Acquisition
Review Quarterly, Spring 1999, pp. 147–154
41. “The Causes of Risk Taking by Project Managers,” Michael Wakshull,
Proceedings of the Project Management Institute Annual Seminars &
Symposium, November 2001.
42. “Stochastic Project Duration Analysis Using PERT–Beta Distributions,” Ron
Davis
43. “Triangular Approximation for Continuous Random Variables in Risk
Analysis,” David G. Johnson, Decision Sciences Institute Proceedings 1998.
http://www.sbaer.uca.edu/research/dsi/1998/Pdffiles/Papers/1114.pdf
44. “The Cause of Risk Taking by Managers,” Michael N.Wakshull, Proceedings of
the Project Management Institute Annual Seminars & Symposium November 1–
10, 2001, Nashville Tenn, http://www.risksig.com/Articles/pmi2001/21261.pdf
45. “The Framing of Decisions and the Psychology of Choice,” Tversky, Amos,
and Daniel Kahneman. 1981, Science 211 (January 30): 453–458,
http://www.cs.umu.se/kurser/TDBC12/HT99/Tversky.html
: Resources

Resources
46. “Three Point Approximations for Continuous Random Variables,” Donald
Keefer and Samuel Bodily, Management Science, 29(5), pp. 595 – 609
47. “Better Estimation of PERT Activity Time Parameters,” Donald Keefer and
William Verdini, Management Science, 39(9), pp. 1086 – 1091.
48. “The Benefits of Integrated, Quantitative Risk Management,” Barney B.
Roberts, Futron Corporation, 12th Annual International Symposium of the
International Council on Systems Engineering, July 1–5, 2001,
http://www.futron.com/pdf/benefits_QuantIRM.pdf
49. “Sources of Schedule Risk in Complex Systems Development,” Tyson R.
Browning, INCOSE Systems Engineering Journal, Volume 2, Issue 3, pp. 129 –
142, 14 September 1999,
http://sbufaculty.tcu.edu/tbrowning/Publications/Browning%20(1999)––
SE%20Sch%20Risk%20Drivers.pdf
50. “Sources of Performance Risk in Complex System Development,” Tyson R.
Browning, 9th Annual International Symposium of INCOSE, June 1999,
http://sbufaculty.tcu.edu/tbrowning/Publications/Browning%20(1999)––
INCOSE%20Perf%20Risk%20Drivers.pdf
: Resources

Resources
51. “Experiences in Improving Risk Management Processes Using the Concepts of
the Riskit Method,” Jyrki Konito, Gerhard Getto, and Dieter Landes, ACM
SIGSOFT Software Engineering Notes , Proceedings of the 6th ACM SIGSOFT
international symposium on Foundations of software engineering SIGSOFT
'98/FSE-6, Volume 23 Issue 6, November 1998.
52. “Anchoring and Adjustment in Software Estimation,” Jorge Aranda and Steve
Easterbrook, Proceedings of the 10th European software engineering
conference held jointly with 13th ACM SIGSOFT international symposium on
Foundations of software engineering ESEC/FSE-13
: Resources

Web Resources
§ Decision Sciences Institute, http://www.decisionsciences.org/
§ Tyson R. Browning, http://sbufaculty.tcu.edu/tbrowning/index.htm
§ The Risk Doctor, http://www.risk-doctor.com/
§ Canadian Treasury Board Policy for the Management of Projects,
http://www.tbs-sct.gc.ca/pm-gp/site/home_accueil-eng.aspx
: Resources

Thank You for Your Patience
: Resources

Probabilistic Schedule and Cost Analysis

More Related Content

What's hot

Viewers also liked

Similar to Probabilistic Schedule and Cost Analysis

More from Glen Alleman

Probabilistic Schedule and Cost Analysis