Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Probabilistic Schedule and Cost Analysis


Published on

An overview of the probabilistic risk analysis processes that can be applied to a program. Although it may not appear to be a “simple” overview, this material is the tip of the iceberg of this complex topic.

Just schedule analysis has been addressed in detail here. The cost aspects of forecasting and simulation must be addressed as well to complete the connections between schedule and cost.

Probabilistic cost will be surveyed here, but an in depth review is for a later time.

  • Ripley's Believe It Or Not Investigated Him After His 5th Win...(unreal story inside) ♥♥♥
    Are you sure you want to  Yes  No
    Your message goes here
  • how to lose weight without exercise - best weight loss diet ■■■
    Are you sure you want to  Yes  No
    Your message goes here
  • Can You Spare 1 Minute Per Day? If you have a few spare minutes per day, we have some very exciting news for you! ♣♣♣
    Are you sure you want to  Yes  No
    Your message goes here
  • home remedies to lose belly fat without exercise easily and quickly ●●●
    Are you sure you want to  Yes  No
    Your message goes here

Probabilistic Schedule and Cost Analysis

  1. 1. Programmatic Risk Analysis 1/186 Probabilistic Schedule and Cost Analysis Our goal is to develop a resource loaded, risk tolerant, Integrated Master Schedule (IMS), derived from the Integrated Master Plan (IMP) that clearly shows the increasing maturity of the program’s deliverables, through vertical and horizontal traceability to the program’s requirements. Prepared for NNJ05111915R, by GB Alleman, December 2005
  2. 2. Programmatic Risk Analysis 2/186 Source Material Disclaimer § All the materials in this briefing originated from publicly available sources, nothing here should be construed as proprietary or unique to an individual vendor or manufacturer. § All materials are used for informational and educational purposes ONLY and are not be reused outside the teaching process. § The Fair Use Copyright law provides use for research and teaching. § All references to materials used in this briefing are provided in the “resources” section. § No other reuse of this briefing should be performed outside of the specific learning objectives of “Probabilistic Schedule and Cost Analysis,” education on proposal and execution. : Introduction
  3. 3. Programmatic Risk Analysis 3/186 Programmatic Risk Management Involves Danger and Opportunity § The Chinese symbol for risk represents – Danger – Opportunity § Taken together this suggests that risk is a strategic combination of vulnerability and opportunity. § Viewed in this light, programmatic risk management is a tool for managing risk that enables the program to take advantage of value enhancing opportunities. § A missed strategic opportunity can result in a greater loss of (potential) value than an unfortunate incident or adverse change master schedule. § Many programs address risk in “silos.” § Programs cannot take this approach – an integrated technical and programmatic risk management strategy must be used. : Introduction
  4. 4. Programmatic Risk Analysis 4/186 Dealing With Our “Learning Opportunity” In The Presence Of The “Danger” Of Being Late The task of space management is rocket science. It is terribly complicated. Launchers explode and spacecraft disappear. No one wants to fail. Good enough is not good enough for mechanisms within which thousands of components must work in tandem for a mission to succeed. – “Bureaucracy and the Space Program,” in Sadeh, ed., Space Politics and Policy, 2003. § When we add programmatic risk issues and their management to the technical risk issues, the integrated programmatic solution must track and support the integrated technical solution § This briefing provides the background for the use of Monte Carlo simulation in the construction of a risk tolerant IMS : Introduction § The material presented here is an “in depth” look at Monte Carlo which is more than just a tool, it is the basis of a process § Because of this, a deeper understanding of the process is needed to properly apply Monte Carlo to build a risk tolerant IMS § So be prepared, this is not one of the overview tours, this will be work with the equivalent reward – a competitive IMS that can hold its own in front of the customer and the competition
  5. 5. Programmatic Risk Analysis 5/186 This Briefing is an Overview of Programmatic Risk, but not a Handbook of how to do it § Why the IMS must include a programmatic and schedule risk management strategy – Programmatic risk and technical risk are linked § Techniques for managing uncertainty in the IMS – Specific tasks for risk management § PERT and Monte Carlo methods – Why are we using Risk+ and not focusing on PERT § Models of a risk tolerant schedule – Examples of Risk+ – Examples of risk mitigation plans : Introduction
  6. 6. Programmatic Risk Analysis 6/186 One View of Project Management § Technocrats manage projects – Driven by the adherence to the profession of engineering and the profession of managing engineering projects § This approach is very useful when we have reliable, predictable and operational projects – Single point failures are managed within an “engineering” culture with built–in redundancy § The problem with this approach is: – Hierarchical structures try to ensure organizational control and accountability – Complex systems are prone to failure – Budget and schedule issues and to the technical complexity – Interactive failure modes abound for both technical and programmatic risk situations § The role of a Risk Tolerant IMS is to make visible these problems and provide the opportunity for their solution : Introduction
  7. 7. Programmatic Risk Analysis 7/186 What Kind of Risk are We Talking About in the IMS? § People are not generally good at analyzing risk § Most risk analysis is qualitative in nature – Skills, time and resources are in short supply to undertake a quantitative risk analysis – The result is usually a subjective assessment of the risk § However quantitative analysis is needed for high programmatic risks § A resource loaded IMP/IMS containing accurate predecessors and successor relationships is the starting point for quantitative risk assessment : Introduction
  8. 8. Programmatic Risk Analysis 8/186 Integrating Risk Management into the IMS is a Multi–step Process § Programmatic risk assessment must define in the IMS both the technical risks, their mitigations as well as the programmatic steps take during these mitigations. § These steps include: – Connection of technical and programmatic risks – Branching probabilities for alternative paths taken during mitigation – Resources needed for these alternative paths – Impacts on critical milestones resulting from the occurrence of a risk : Introduction
  9. 9. Programmatic Risk Analysis 9/186 I think you should be more explicit here in step two Executive Overview There is a large amount of material in this briefing. So this overview provides a “summary” of the concepts and the recommended approach to developing a credible IMS based on probabilistic risk analysis. But in the end the details must be “owned” before we can say we have a handle on Probabilistic Schedule and Cost Analysis. : Executive Overview
  10. 10. Programmatic Risk Analysis 10/186 We Usually Want To Know What Our Motivation In Order To Hold Our Interest § Noel Coward’s motivational speech – If you must have motivation, think of your pay check on Friday § Our motivation starts with DID 81650, which says: – The IMS shall be used to verify attainability of contract objectives, to evaluate progress toward meeting program objectives, and to integrate the program schedule activities with all related components. § The construction of an IMS in this manner is necessary but not sufficient to show that the contract objectives can be attained. – An IMS that is tolerant of both technical and programmatic risks is the next step – Such an IMS contains both the known risk mitigation processes as well as the unknown risk mitigation opportunities § The BIG QUESTION is what are the units of measure of risk tolerance? : Executive Overview Noel Coward 1899 – 1973
  11. 11. Programmatic Risk Analysis 11/186 Programmatic Risk Management in One Slide Starts with Abandoning PERT and CPM § The Critical Path Method (CPM) does not provide a realistic view of programmatic risk. – Task durations are random variables drawn from an underling probability distribution – Near Critical Paths biases the completion time of the program in ways not show by the CPM – Correlations between multiple paths bias the risk analysis as well § PERT estimates do not address the underlying probabilistic nature of activity networks. PERT assumes… – Statistical independence of each activity – this is almost never the case – Symmetrical probability distributions that allow the addition of the “most likely” (Mode) value of each activity to produce a total project duration § Monte Carlo must be used and requires understanding of ... – The probability distribution functions of the activity network – The branching probabilities for alternative paths created by risk mitigation – The influence of correlations between network activities and their impacts on risk : Executive Overview
  12. 12. Programmatic Risk Analysis 12/186 Merging Technical and Programmatic Risk is the Core to Building a Risk Tolerant IMS § There are two types of “uncertainty” on a program – Technical – uncertainty about the functional and performance aspects of the program’s technology that impacts the produceability of the product or creates delays in the schedule – Programmatic – uncertainty about the duration and cost of the activities that deliver the functional and performance elements of the program independent of the technical risk § We’re interested in connecting the two in the schedule and cost model(s) – When the technical uncertainty arises what is the impact on the schedule and cost? – When the schedule or cost uncertainty arises what is the impact on the functional and performance aspects? : Executive Overview So much for our strategy of winning through technical dominance
  13. 13. Programmatic Risk Analysis 13/186 The Meaning of Uncertainty in the Context of an Integrated Master Schedule § Uncertainty in plain English is about the “lack of certainty” – Uncertainty is about the “variability” in the performance measures like cost, duration, or quality – Uncertainty is about the “ambiguity” associated with a lack of this clarity § Discovering the known and unknown sources of bias and ignorance helps define much effort it is worth to clarify the uncertainty – This is the underlying process driving uncertainty § As well, uncertainty arises from the basic processes of work – This is Deming uncertainty – It is the statistical “noise” built into the work process § Both of these sources of uncertainty impact cost and schedule – Trying to control the “noise” adds little value – Trying to control the “lack of certainty” arising from ambiguity and lack of clarity does have value : Executive Overview
  14. 14. Programmatic Risk Analysis 14/186 Schedules Are Networks Of Random Variables not Collections of Deterministic Statements § Task completion durations are random variables not just dates: – These random variables have underlying probability distributions – These distributions can not be “added” to arrive at a project completion date – Trying to force the work into a fixed duration does not increase the likelihood of completion § The PERT approach to estimating project duration contains several faulty assumptions: – The assumed independence is rarely the case – Uniform distribution of completion times can not be confirmed – The 3–Point estimates have built in optimistic bias § Monte Carlo Simulation provides more accurate estimates of project completion times: – But only if the network topology is “well formed” – And if the interactions of the underlying probability distributions are understood : Executive Overview Attempting to make the dryer “dry faster” is a loosing proposition. The dryer’s capacity is constrained by its mechanics. Much like the capacity of the design team is constrained by availability and technical productivity.
  15. 15. Programmatic Risk Analysis 15/186 Building a Credible IMS Means Managing the Numbers in Meaningful Ways § First step – build the numbers correctly – Build an IMS in “layers” of detail – Understand the probability distributions of each layer – Identify schedule margin opportunities within the IMS – Assess the margin’s impact on the probabilistic outcome § Continuous process improvement – build the IMS “many times” – Gather historical and “expert” opinions of task duration – Build probability distribution functions from this data – Improve the Monte Carlo model using this data § Answer the question – many times, possibly continuously – “What are the units of measure for credibility?” – “What is the coupling and cohesion of the tasks in the IMS?” : Executive Overview
  16. 16. Programmatic Risk Analysis 16/186 The Right Effort Produces Results, But Interpreting the Results is Sometimes Difficult § Proposal Improvements – Credible schedule based on probabilistic model – Traceable data to probability distributions for each class of task – Verifiable forecast of risk areas and project duration § Executable program Improvements – Model of “hot spots” in the IMS – Continuous assessment of schedule and cost risk – Increased visibility of probabilistic methods § Improvement in our understanding of probabilistic planning – “What is the critical path” requires more than a red line in a power point chart – No point values are allowed, only statistically qualified estimates – Stochastic network models of schedule and cost are the minimum deliverables for proposal and execution : Executive Overview
  17. 17. Programmatic Risk Analysis 17/186 The One Slide Describing Our Search for “Actionable Outcomes” § Building a risk tolerant IMS – Explicit technical risk mitigation must be embedded in the IMS – Explicit schedule margin must be embedded in the IMS » Margin values identified through Monte Carlo simulations » Margin assigned in front gating events – Technical risks connected to ARM in some form – Cost and Schedule risks connected to i–MAP (impact mapping from Woods Analysis) § Assessing the Risk Tolerant IMS – what does risk tolerant mean? – Weekly status, monthly Earned Value, forecast of risk impacts – Weekly Monte Carlo assessment of confidence intervals and their historical changes – are we getting better or worse? – Performance forecast based on likelihood outcomes from Monte Carlo simulations, not just “adding up the numbers” : Executive Overview
  18. 18. Programmatic Risk Analysis 18/186: Executive Overview The Page Intentionally Left Blank
  19. 19. Programmatic Risk Analysis 19/186 Risk Based Planning The difference between failure and success is the difference between doing something almost right and doing something right. — Benjamin Franklin : Risk Based Planning Murray didn’t feel the first pangs of real panic until he pulled the emergency cord.
  20. 20. Programmatic Risk Analysis 20/186 In the Risk Management Business, there is Simply Too Much Information for Our Needs : Risk Based Planning
  21. 21. Programmatic Risk Analysis 21/186 Risk Assessment and Management Techniques Vary with Maturity † Add a Risk Factor or Percentage to the critical paths A “bottom line” Monte Carlo or Range analysis Detailed Monte Carlo for each WBS element Expert Opinions in a Database with assessment Detailed Bayesian Network Analysis Increasing Detail and Difficulty IncreasingPrecisionandValue § There are several approaches to building a Risk Tolerant IMS – First recognize that we’re in the early stage of this effort – There is likely value in moving further up the curve † Ron Coleman, Litton TASC, 33rd ADoDDCAS, Williamsburg, VA : Risk Based Planning
  22. 22. Programmatic Risk Analysis 22/186 Risk is Different from Uncertainty Knowing this Difference is Critical to Success § Reducible risk stems from known probability distributions – An Estimating methodology risk resulting from improper models of cost + schedule – Cost factors such as inflation, labor rates, labor rate burdens, etc – Configuration risk (variation in the technical inputs) – Unknown Cost, Schedule and Technical risk coupling § Irreducible risk stems from known statistical processes – Requirements change impacts – Budget Perturbations – Re–work, and re–test phenomena – Contractual arrangements (contract type, prime/sub relationships, etc) – Potential for disaster (labor troubles, shuttle loss, satellite “falls over”, war, hurricanes, etc.) – Probability that if a discrete event occurs it will invoke a project delay : Risk Based Planning
  23. 23. Programmatic Risk Analysis 23/186 There are Two Types of Uncertainty Encountered in a Risk Tolerant IMS § Static uncertainty is natural variation and foreseen risks – Uncertainty about the value of a parameter § Dynamic uncertainty is unforeseen uncertainty and “chaos” – Stochastic changes in the underlying environment – System time delays, interactions between the network elements, positive and negative feedback loops – Internal dependencies Stochastic behavior of forecasted completion dates Low Work Quality Poor Work Conditions External Scope Changes Unintended Changes Upstream Hidden Change : Risk Based Planning
  24. 24. Programmatic Risk Analysis 24/186 The Multiple Sources of Schedule Uncertainty and Sorting Them Out is the Role of Planning § Unknown interactions drive uncertainty § Dynamic uncertainty can be addressed by flexibility in the IMS – On ramps – Off ramps – Alternative paths – Schedule “crashing” opportunities § Modeling of this dynamic uncertainty requires simulation rather than static PERT based path assessment – Changes in critical path are dependent on time and state of the network – The result is a stochastic network : Risk Based Planning
  25. 25. Programmatic Risk Analysis 25/186 Schedule Risk Management is… § Schedule risk management seeks to anticipate and address uncertainties that threaten the goals and timetables of a project § Unmitigated risks lead rapidly to delays in delivery dates and budget overages that undermine confidence in the schedule and in the project manager § Schedule risk management is process oriented § While any project accepts a certain level of risk, regular and rigorous risk analysis and risk management techniques serve to defuse problems before they arise § Integrated Planning reflects the development phases and the hierarchical architecture of the system : Risk Based Planning
  26. 26. Programmatic Risk Analysis 26/186 What’s our Goal as Planners? § Construct an IMS that has integrity and credibility to show … § External assessors who may consider that the schedule … – Reflects the total scope of work – Is fully integrated » Internally (task/milestone interdependencies) » Externally (other NASA facilities, contractor schedules, vendor deliveries, etc.) – Has an established baseline – Is reasonable or even feasible at proposal submission – Does not provide for “What–if” analysis – Is capable of providing for multiple and varying Critical Path identification or slack for all tasks and milestones § … and how the schedule may… – reflect an accurate model of planned implementation – reflect an accurate or complete status – provide the correct basis for resource planning : Risk Based Planning
  27. 27. Programmatic Risk Analysis 27/186 Some “Unpleasant” Questions Can Easily Occur If We Don’t Pay Attention To The Details § What is the degree of risk in our baseline? How do we measure this risk? § What are the branching probabilities for the critical path in the IMS? How are they derived? § How many “near critical” paths will become critical as the program proceeds? What drives these? § Have the “risk drivers” been identified and mitigations put in place through explicit tasks in the IMS to deal with each identified risk? If so, how are they shown in the IMS? § Do we understand the underlying task completion probability distributions? How are they derived? § How do these probability distributions change as the program proceeds? What is the analytical basis for this? : Risk Based Planning Remember Edsel Ford’s dream of the future?
  28. 28. Programmatic Risk Analysis 28/186 Risk Management at NASA § Risk includes undesirable consequences (harm) and probability of occurrence of this harm § For the IMS, this harm is the failure to … – Identify risk mitigation tasks – Provide sufficient schedule margin at the right places § Risk consists of three elements: – What can go wrong? – define a set of scenarios – How likely is it? – an evaluation of the probabilities – What are the consequences? – an evaluation of the consequences § The identification of these risk scenarios is the most important result of our Risk Assessment process for the IMS: – What are the failure scenarios for the program? – What are the mitigation strategies for these failure modes? – What resources are needed for each mitigation? : Risk Based Planning
  29. 29. Programmatic Risk Analysis 29/186 NASA IRMA Tool is Used at Johnson Space Center : Risk Based Planning
  30. 30. Programmatic Risk Analysis 30/186 NASA Risk Summary Card and Programmatic Risk : Risk Based Planning
  31. 31. Programmatic Risk Analysis 31/186 Implementing Programmatic Risk Assessment is a Straight Forward Process Initiating Event Selection Scenario Development Scenario Logic Modeling Scenario Frequency Modeling Consequence Modeling Risk Integration § Some simple steps to identifying risk opportunities in the IMS – Scenario based planning – “what if this happens?” – Event impact planning – “what inhibits success?” § Both must focus on the consequences in order to identify the mitigations : Risk Based Planning
  32. 32. Programmatic Risk Analysis 32/186 Continuous Risk Management (CRM) is the Basis of Programmatic Risk Management § NASA Guidance – OMB A–11 – NASA NPG 7120.5A – NASA–SP–610S – NASA NPR 8000.4 § DoD Guidance – DAU “Risk Management Guide for DoD Acquisition – Air Force, “Acquisition Risk Management” – Air Force “SMC Systems Engineering Primer and Handbook” CRM Activity IMS Representation Identify Risk items with IMP/IMS #’s, CA/WP & resource assignments Analyze Risk management responsibilities assigned Plan Mitigation plans with durations and resource assignments Track Status reported from Risk Management to IMS Control Risk tasks reporting in weekly status process Communicate IMS status reporting : Risk Based Planning
  33. 33. Programmatic Risk Analysis 33/186 Design v. Risk Evaluation are Two Sides of the Same Coin – Risk Tolerance § The IMS for the “planned” program can be considered the “reference mission” plan – Meets the SOW and DRD deliverable plan with deterministic tasks – Critical paths defined – Explicit schedule margin assigned per PMP and risk identification § Missing elements from the design evaluation that must be in the risk evaluation – Near critical paths impacting durations – Individual task risk assignment – Risk distribution curve for classes of activities – Cumulative risk probability distribution skewing – Branching probabilities not defined – Correlation between risk paths and off risk paths – Dynamic interactions that drive risk : Risk Based Planning
  34. 34. Programmatic Risk Analysis 34/186 Embedding Risk Management in the Integrated Master Schedule § The IMS should show the coupling between technical risk and programmatic risk – Technical risk activities in the IMS connected to Active Risk Manager – Programmatic risk visible in the IMS § Technical estimates of task durations developed from subject matter experts – Past performance – Basis of estimate – Expert judgments – Parametric estimates – Dynamic models § Programmatic estimates developed through the win strategies – Convey the risk buy down and win theme support through the risk mitigation activities : Risk Based Planning All these estimates must be calibrated before being accepted into the IMS. Without this effort, these numbers are just as unreliable as raw estimates
  35. 35. Programmatic Risk Analysis 35/186 Why Probabilistic Risk Analysis is Often Opposed by Management and IPT Leads § Many people do not understand the underlying statistics – Education, practice, guidance § Many planners lack the formal probability and statistics training – Education, practice, guidance § Most planners perform deterministic analysis of schedules and cost – Risk is hard work § The fact the probabilistic risk analysis is built on uncertainty is seen as weakness in the planning process, not a strength – Why can’t you know how long it will take or how much it costs? § People tend to think that the “lack of data” is a reason not to perform probabilistic schedule risk analysis – The exact opposite is true : Risk Based Planning
  36. 36. Programmatic Risk Analysis The Page Intentionally Left Blank
  37. 37. Programmatic Risk Analysis 37/186 Managing Uncertainty in the IMS A ship on the beach is a lighthouse to the sea. — Dutch Proverb : Managing Uncertainty in the IMS
  38. 38. Programmatic Risk Analysis 38/186 3 Troubles With Deterministic Schedule Estimating in a Traditional Manner 1. Single point estimates can be accurate a) Without stating the distribution statistics, the number has no frame of reference b) The median temperature in Cody Wyoming is a balmy 78º c) Don’t be there in February in your shirt sleeves : Managing Uncertainty in the IMS 2. There is no standard definition for the term “best” estimate a) A hoped for best b) A planned for best c) The actual best derived from the underlying statistical model 3. Rollup of the “most likelies” is not the same as Most Likely Total Duration a) They are probability distributions not integers b) Probability distribution function (pdf) convolution is needed The flaw with using averages alone Average depth is 3 feet
  39. 39. Programmatic Risk Analysis 39/186 No Single Point Estimate Can Be Accurate, Since It Ignores The Underlying Statistics § Schedule durations are always vague in the beginning – Existing technical capability often falls short of project needs – Firm requirements, especially software requirements cannot be described in a simple list – “Normal” schedule slippage of varying lengths result from integration problems and test failures – Various other anticipated and unforeseen events resulting from the natural variation (Deming variation) § “Point” estimates can not be correct because… – Task point estimates of activity durations are not correct – Project point estimate is the sum of these “incorrect” activity estimates § “Actual” project duration will fall within some range around the point estimate – The best that can be done is to understand the uncertainty : Managing Uncertainty in the IMS 1
  40. 40. Programmatic Risk Analysis 40/186 The Traditional Roll Up Approach Starts with the “Best” estimates for the Most Likely Durations for Tasks § Build a network of the project’s activities § Determine the “best” estimate of the duration for each activity in the network § Compare the activities’ “best” estimates to find the critical path § Sum all the “best estimate” durations of activities on the critical path § Define this sum of tasks to be “best” estimate of the project’s schedule duration § This will almost always be optimistically wrong § Or it is pessimistically wrong § Either way – it is wrong : Managing Uncertainty in the IMS 1
  41. 41. Programmatic Risk Analysis 41/186 The Problem is the Term “Best” Has No Standard Definition § For each activity the “best” estimate is … – The “most likely” duration – the mode of the distribution of durations? (Mode is the number that appears most often) – It’s 50th percentile duration – the median of the distribution? (Median is the number in the middle of all the numbers) – It’s expected duration – the mean of the distribution? (Mean is the average of all the numbers) § These definitions lead to values that are almost always different from each other § Rolling up the “best” estimate of completion is almost never one of these. : Managing Uncertainty in the IMS 2
  42. 42. Programmatic Risk Analysis 42/186 Durations Are Probability Samples not Single Point Values § We know this because… – “Best” estimate is not the only possible estimate, so other estimates must be considered “worse” – Common use of the phrase “most likely duration” assumes that other possible durations are “less likely” – “Mean,” “median,” and “mode” are statistical terms characteristic of probability distributions § This implies activity distributions have probability distributions – They are random variables drawn from the probability distribution function (pdf) § “Actual” project duration is an uncertain quality that can be modeled as a sum of random variables – The pdf may be known or unknown : Managing Uncertainty in the IMS 2
  43. 43. Programmatic Risk Analysis 43/186 Durations are Educated “Guesses,” but Rarely Have underlying Probability Distributions § Define the problem § Identify the prediction variable § Build the prediction model – Develop a list of relevant factors – Consider the effects – Collect data – Plot each factor independently – Develop a prediction model using linear regression – Understand the model – Check the model § Make guesses with the model § Take care with the results : Managing Uncertainty in the IMS 2
  44. 44. Programmatic Risk Analysis 44/186 One Way Of Producing A Guess Is With A “Twenty Questions” Game § The 20 questions approach – Ask an engineer how long it will take to do a task and the answer might be “I can’t say.” § Planner – Will to take a year? § Engineer – Oh, of course not § Planner – Will it take a day? § Engineer – Oh, of course not § Planner – How about 6 months? § Engineer – Could be, but that’s too long § Planner – How about 2 months? § Engineer – No, that’s too short § Planner – How about 3 ½ months? § Engineer – Yea, that could work § In 5 question a first order estimate can be found to with 20% : Managing Uncertainty in the IMS 2
  45. 45. Programmatic Risk Analysis 45/186 Putting Guesses into a Schedule Requires Us to Sort Out Fact from Fiction § Rank all the tasks – 1 = scope known, duration known – 2 = scope known, duration unknown – 5 = scope unknown, duration unknown § Have a “planning session” where no one leaves the room until all Rank–5 tasks are turned into Rank–2 tasks – The reduction in rank comes for “information” about the probability distribution which underlying the random variable representing the duration § Focus on “randomness” of the estimate is critical to success – Adjustments for confidence must become part of our vocabulary – This approach turns “guessing” into statistical estimating : Managing Uncertainty in the IMS 2
  46. 46. Programmatic Risk Analysis 46/186 Risk Drivers That Impact Uncertainty Must be Identified Before They are Used § Risk drivers include (but may not be limited to) – Beyond the state–of–the–art development – Unusual production requirements, either time or technique – Cost constraints, derived without consideration of technical or programmatic processes – Software development issues – Multiple interface management – Subcontractor and supplier viability, variability or plain olde “ability” – System integration and testing impacts from coupling and cohesion of the tasks in the IMS – Unforeseen events : Managing Uncertainty in the IMS 2
  47. 47. Programmatic Risk Analysis 47/186 Estimating Accomplishment Criteria starts with defining the what “done” look like § When building the IMS, the first round should – Confirm the IMP (PE/SA/AC) structure overlays the topology of the program – Ask for a duration estimate for each Accomplishment Criteria which represent » Exit criteria » Deliverables for the maturing program » Incremental progress along the path to maturity – Get this estimate as a “single” task with a duration § Assign SA’s to an IPT Lead – Develop IPT processes that support the SA – Identify AC’s from process deliverables or maturity improvements – Link vertical path to Program Events and horizontal paths across IPTs : Managing Uncertainty in the IMS 2
  48. 48. Programmatic Risk Analysis 48/186 Modeling Duration Probability starts with experts but must include statistical estimates § Compile duration estimates from different sources and rank the estimates: – Subcontractors or IPTs estimate – Project manager’s estimate – “Independent” estimate – Risk–impacted estimate § Associate confidence levels with ranges between estimates, using information available from different situations and at different stages in the project development cycle – These can not be looked up in a book – Are not directly derived from historical data – Must be subjective, knowledge based consensus of technical experts in a particular WBS – Should be a standard part of the risk mitigation plan : Managing Uncertainty in the IMS 3
  49. 49. Programmatic Risk Analysis 49/186 Statistics at a Glance as a starting point. But more details are needed § Probability distribution – A function that describes the probabilities of possible outcomes in a "sample space.” § Random variable – variable a function of the result of a statistical experiment in which each outcome has a definite probability of occurrence. § Determinism – a theory that phenomena are causally determined by preceding events or natural laws. § Standard deviation (sigma value) – An index that characterizes the dispersion among the values in a population. § Bias – The expected deviation of the expected value of a statistical estimate from the quantity it estimates. § Correlation – A measure of the joint impact of two variables upon each other that reflects the simultaneous variation of quantities. § Percentile – A value on a scale of 100 indicating the percent of a distribution that is equal to or below it. § Monte Carlo sampling – A modeling technique that employs random sampling to simulate a population being studied. : Managing Uncertainty in the IMS 3
  50. 50. Programmatic Risk Analysis 50/186 Siren Song of the Central Limit Theorem must be avoided in any robust estimating process § The probability distribution of the project’s total duration is obtained by statistically summing distributions of all activities along the schedule network critical path § Central Limit Theorem – if the number of critical path activities is “large,” the probability distribution of the total duration is “approximately” Gaussian. § Another theorem (not related to Mr. Gauss) – the sum of the means of activity duration equals the mean of the total duration – But because the Gaussian distribution is symmetric, the total duration distribution results in mean = medium = mode – Therefore, » Sum of the activity duration medians < total duration median » Sum of the activity modes < total duration mode : Managing Uncertainty in the IMS 3
  51. 51. Programmatic Risk Analysis 51/186 The Central Limit Theorem must be understood in order to be useful § The Central Limit Theory (CLT) consists of three statements – The mean of the sampling distribution of means equals the mean of the population from which the samples were drawn – The variance of the sampling distribution of means is equal to the variance of the population from which the samples were drawn divided by the size of the samples – If the original population is distributed normally, the sampling distribution of means will also be normal. If the original population is not normally distributed, the sampling distribution of means will increasingly approximate a normal distribution as sample size increases § PERT assumes the duration and variance along the critical path are normally distributed and therefore the total duration follows the CLT § Departures from normal distribution and near critical path tasks becoming critical are usually ignored – to the peril of planning : Managing Uncertainty in the IMS 3
  52. 52. Programmatic Risk Analysis 52/186 Mr. Gauss’s Distribution is found in many text book examples, too bad is not applicable § The Gaussian distribution can be proved (by the Central Limit Theorem) in the situation that each measurement is the result of a large amount of small, independent error sources. These errors have to be of the same magnitude, and as often positive as negative. § When a physical item is measured and systematic errors are eliminated the measured values will spread around the average value. § The average value of a measured value is the “best value.” : Managing Uncertainty in the IMS 3
  53. 53. Programmatic Risk Analysis 53/186 Task “Most Likely” ≠ Project “Most Likely,” Must be Understood by Every Planner § PERT assumes probability distribution of the project times is the same as the tasks on the critical path § Because other paths can become critical paths, PERT consistently underestimates the project completion time 1 + 1 = 3 : Managing Uncertainty in the IMS 3
  54. 54. Programmatic Risk Analysis 54/186 Probability Distribution Function is the Lifeblood of good planning § Probability of occurrence as a function of the number of samples § “The number of times a task duration appears in a Monte Carlo simulation” : Managing Uncertainty in the IMS
  55. 55. Programmatic Risk Analysis 55/186 Standard Deviation of a Probability Distribution § Which describes that “spread” of the random variables around the mean represented by the distribution § Standard deviation describes the width of the probability distribution function : Managing Uncertainty in the IMS
  56. 56. Programmatic Risk Analysis 56/186 Underlying Statistics and Confidence : Managing Uncertainty in the IMS
  57. 57. Programmatic Risk Analysis 57/186 Families of CDF’s Can Look Alike § Cumulative Distribution Functions (CDF) look similar for a variety of Probability Distribution Functions (pdf) Cumulative Probability 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200% Bounds on Point Estimate Beta Triangular Uniform Probability Density 0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200% Bounds on Point Estimate Beta Triangular Uniform Cumulative Probability 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200% Bounds on Point Estimate Beta Triangular Uniform Normal LogNormal Probability Density 0% 20% 40% 60% 80% 100% 120% 140% 160% 180% 200% Bounds on Point Estimate Beta Triangular Uniform Normal LogNormal : Managing Uncertainty in the IMS
  58. 58. Programmatic Risk Analysis 58/186: Managing Uncertainty in the IMS The Page Intentionally Left Blank
  59. 59. Programmatic Risk Analysis 59/186 Approaches To Uncertainty “It is moronic to predict without first establishing an error rate for a prediction and keeping track of one’s past record of accuracy” — Nassim Nicholas Taleb, Fooled By Randomness (where he argues people constantly delude themselves because they do not understand probability and are programmed to find reasons for optimism or pessimism where none exist.) : Approaches to Uncertainty If you hang them all, you’re sure to get the guilty ones. – Judge Roy Bean, “The Law West of the Pecos”
  60. 60. Programmatic Risk Analysis 60/186 There Is No Such a Thing as Risk Neutral Decision Making § Can we have a discussion of a probabilistic event without a discussion of associated risk? (confidence in the confidence interval) § The most severe problem with making decisions in the presence of risk is when the decision maker doesn’t have the faintest idea about what risks are to be incurred, yet thinks there is precision about the decision results. § Three features of decision making – Alternatives – which course of action might be taken? – Uncertainties – what uncontrolled elements exist? – Outcomes – Alternatives combined with Uncertainties = Outcomes : Approaches to Uncertainty Natural Gas? Rotating equipment? Open flames? High voltage?
  61. 61. Programmatic Risk Analysis 61/186 Types of Uncertainty Based Decisions Strategy Describes a collection of actions the decision maker makes è Goal A possible outcome of the strategy is a goal Decision Actually performing a task is a decision è Prioritization Deciding which task to perform is a prioritization Alternative A set of choices that are allowable è Option An option is an alternative that permits a future decision following the discovery of new information Sensitivity of Decision Knowing “what” to do è Sensitivity of Outcome Knowing “how” it will turn out Direct values Outcomes traceable to bookable benefits to the project è Indirect Values Things the decision maker value but are unlikely to be visible in the project Certain equivalent effect The smallest sum of money for which the decision maker would be willing to sell rights to a risky product è Expected Net Present Value The hypothetical average NPV from numerous independent launches of identical projects : Approaches to Uncertainty
  62. 62. Programmatic Risk Analysis 62/186 Approaches to Decisions with Uncertainty § Decision trees § Line of balance § PERT § Monte Carlo : Approaches to Uncertainty
  63. 63. Programmatic Risk Analysis 63/186 Decision Trees § Decision Trees are tools for helping choose between several courses of action. § They provide a effective structure within which to lay out options and investigate the possible outcomes of choosing those options. § They also help to form a balanced picture of the risks and rewards associated with each possible course of action. : Approaches to Uncertainty
  64. 64. Programmatic Risk Analysis 64/186 Line Of Balance § The line–of– balance technique is based on the underlying assumption that the rate of production for an activity is uniform § Measurements are compared against a specific plan, all collected in one “balance” chart : Approaches to Uncertainty
  65. 65. Programmatic Risk Analysis 65/186 PERT As The Starting Point For Probabilistic Schedule Management § PERT is a method to determine how long a project should take – Which activities are most critical – Deterministic PERT and Probabilistic PERT are common § PERT algorithms – Activity duration: – Activity Standard Deviation: – Activity Variance: – Total Standard Deviation: ( )4 /6et a m b= + + ( )/6et b as = - ( ) 22 /6ev t b as é ù= = -ë û ( ) ( ) 2 2 1 2e e eT t ts s s= + : Approaches to Uncertainty
  66. 66. Programmatic Risk Analysis 66/186 Deterministic PERT Uses The Three Point Estimates In A Static Manner § Durations are defined as three point estimates – These estimates are very subjective if captured individually by asking… – “What is the Minimum, Maximum, and Most Likely” § Critical path is defined from these estimates is the algebraic addition of three point estimates § Project duration is based on the algebraic addition of the times along the critical path § This approach has some serious problems from the outset – Durations must be independent – Most likely is not the same as the average : Approaches to Uncertainty
  67. 67. Programmatic Risk Analysis 67/186 Probabilistic PERT Uses The Underlying Probability Distributions For Each Task § Any path could be critical depending on the convolution of the underlying task completion time probability distribution functions § The independence or dependency of each task with others in the network, greatly influences the outcome of the total project duration § Understanding this dependence is critical to assessing the credibility of the plan as well as the total completion time of that plan : Approaches to Uncertainty
  68. 68. Programmatic Risk Analysis 68/186 Statistics V. Probability – We Need Both To Build A Risk Tolerant Schedule § In building a risk tolerant IMS, we’re interested in the probability of a successful outcome – “What is the probability of making a desired completion date?” § But the underlying statistics of the tasks influence this probability § The statistics of the tasks, their arrangement in a network of tasks and correlation define how this probability based estimated developed. Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write. – H.G. Wells : Approaches to Uncertainty
  69. 69. Programmatic Risk Analysis 69/186 The Problem with PERT § When the activity durations are random variables, each path of the project network is a likely candidate to be the critical path. § Every activity duration could result in a different longest path § Evaluating the distribution of the longest path, even under very specific distributional is not an easy problem : Approaches to Uncertainty
  70. 70. Programmatic Risk Analysis 70/186 Simple Understanding of Statistics The number of times a specific occurrence of a parameter occurs in a population. The number of time it snows before September 15th in Colorado indicates the confidence that it will snow on or before 9/15 this year : Approaches to Uncertainty
  71. 71. Programmatic Risk Analysis 71/186 Inputs to the process have statistics too § The statistics of the events that impact the project must be understood as well – this second order impact is critical § These events form a stochastic process that drives the network § These drivers may or may not be random events § Correlations between the events can create nonlinear behaviors in the sensitivity of the model : Approaches to Uncertainty
  72. 72. Programmatic Risk Analysis 72/186 The Program is a System, Just like the Spacecraft § The programmatic and planning dynamics act as a system § The “system response” is the transfer function between input and output § Understanding this transfer function may appear beyond our interest – But it is part of the stochastic dynamic response to disruptions in our plans – “What if” really means “what if” at this point in the response curve of the system : Approaches to Uncertainty Inputs Outputs
  73. 73. Programmatic Risk Analysis 73/186 The Beta Distribution § The Beta distribution is given as: § Where B is the Beta function ( ) ( ) ( ) ( ) ( ) ( ) 1 11 0 , 1 , , , . B t t dt B y ba a b a b a b b -- = - G G = G + = ò ( ) ( ) ( ) 1 1 1 x x P x B b a a b - - - = - : Approaches to Uncertainty
  74. 74. Programmatic Risk Analysis 74/186 What does the Beta Distribution Provide? § Beta can model events that are constrained to take place within an interval defined by a minimum and maximum value § Turns out the equation for task duration – Is a empirical approximation formulas not derived from the beta distribution directly – The theoretical argument showing the relative weight of 4 on the modal time is based largely on the Beta distribution where developed by the Pearson & Tukey based on the 5% points and the median – Presumably, some experimentation was done in the early days, and some empirical basis was found for these forms – At this point, we merely accept the formulas as the "traditional" way of doing PERT § Beta is used directly in Monte Carlo as a model of task duration ( )4 /6et a m b= + + : Approaches to Uncertainty
  75. 75. Programmatic Risk Analysis 75/186 Another Alternative, the Triangle Distribution § The triangle distribution was proposed in the late 90’s as a substitute of the Beta distribution – The parameters of the triangle distribution have a one–to–one correspondence with the PERT parameters § The original “fitting” of PERT equation took place in 1959 § Triangle distribution is better behaved in certain instances, but the “mean” is still greater than the “mode” (the most likely) § The result is overly optimistic durations for the tasks : Approaches to Uncertainty
  76. 76. Programmatic Risk Analysis 76/186 Sensitivity of the PDF: Triangle § Triangle distribution – The “minimum” and the “maximum” are as influential on the mean on the “most likely” – The “minimum,” “maximum” and the “modal” values capture limited information about the underlying distribution – There can be an infinite number of distributions with these same three values : Approaches to Uncertainty
  77. 77. Programmatic Risk Analysis 77/186 Sensitivity of the PDF: Beta PERT § Manipulating the standard Beta distribution produces the BetaPERT (Vose, 2000, pp. 275) – BetaPERT combines the Beta distribution and the PERT formula for the mean : Approaches to Uncertainty
  78. 78. Programmatic Risk Analysis 78/186: Approaches to Uncertainty The Page Intentionally Left Blank
  79. 79. Programmatic Risk Analysis 79/186 Gathering Risk and Mitigation Information Getting viable estimates of the relative risk requires effort, patience and care not to over or under estimate the risk or the mitigation activities. Magic rabbits are not a reliable source of information. : Gathering Risk and Mitigation Information
  80. 80. Programmatic Risk Analysis 80/186 General Flow of Risk Management § Schedule and Cost risk management must be performed in a structured and rigorous manner § Each element below must appear in the IMS : Gathering Risk and Mitigation Information
  81. 81. Programmatic Risk Analysis 81/186 Gathering Risk Analysis Information § Interviews with CAMs and IPT Leads about risk creates a “bias” toward optimism or pessimism – but never toward neutrality – Optimist says – The glass is ½ full – Pessimist says – The glass is ½ empty – An engineer says – The glass is twice as big as it needs to be § Asking for three point estimates (optimistic, most likely, pessimistic) is not the way to do this… – Gain some sense of the risk “ranking” from the owner of the task or work package – Look to any historical data » Durations » Causes of Over Target Baseline (OTB) and Over Target Schedule (OTS) : Gathering Risk and Mitigation Information
  82. 82. Programmatic Risk Analysis 82/186 The Dreaded 3–Point Estimates § Optimistic Estimate – The shortest duration – “It can’t be done in less time than this” § Most Likely Estimate – The median time (middle most), not the mean time (the average) – This builds in a symmetric distribution of the probability distribution function § Pessimistic Estimate – The additional time needed if things go wrong – It is not the maximum time it would take – It is not a “worst case scenario” estimate : Gathering Risk and Mitigation Information
  83. 83. Programmatic Risk Analysis 83/186 Classes of Project Risk § Delay – Slow decisions – Access – Lack of information – Time difference § Dependency – Interaction – Interface difficulties – Interruption of service § Estimates – Mis–estimation – Learning curves – Overly aggressive deadlines Pareto of Risk Causes : Gathering Risk and Mitigation Information
  84. 84. Programmatic Risk Analysis 84/186 Thinking About Risk Classification § These classifications can be used to avoid asking the “3 point” question for each task § This information will be maintained in the IMS § When updates are made the percentage change can be applied across all tasks Classification Uncertainty Overrun A Routine, been done before Low 0% to 2% B Routine, but possible difficulties Medium to Low 2% to 5% C Development, with little technical difficulty Medium 5% to 10% D Development, but some technical difficulty Medium High 10% to 15% E Significant effort, technical challenge High 15% to 25% F No experience in this area Very High 25% to 50% : Gathering Risk and Mitigation Information
  85. 85. Programmatic Risk Analysis 85/186 Steps in Characterizing Uncertainty in Task Duration Estimating Data § Use an “envelope” method to characterize the minimum, maximum and “most likely” § Fit this data to a statistical distribution § Use conservative assumptions § Apply greater uncertainty to less mature technologies § Confirm analysis matches intuition Remember Sir Francis Bacon’s quote about beginning with uncertainty and ending with certainty. If we start with a what we think is a valid number we will tend to continue with that valid number. When in fact we should speak only in terms of confidence intervals and probabilities of success : Gathering Risk and Mitigation Information
  86. 86. Programmatic Risk Analysis 86/186 Some Sobering Observations § In 1979, Tversky and Kahneman proposed an alternative to utility theory. Prospect theory asserts that people make predictably irrational decisions. [45], [52] § The way that a choice of decisions is presented can sway a person to choose the less rational decision from a set of options. § Once a problem is clearly and reasonably presented, rarely does a person think outside the bounds of the frame. § Source: – “The Causes of Risk Taking By Project Managers,” Proceedings of the Project Management Institute Annual Seminars & Symposium November 1–10, 2001 • Nashville, Tenn – Tversky, Amos, and Daniel Kahneman. 1981. The Framing of Decisions and the Psychology of Choice. Science 211 (January 30): 453–458 : Gathering Risk and Mitigation Information
  87. 87. Programmatic Risk Analysis 87/186 The Dark Side of PERT During the modeling process the “most likely” durations can not be added since they represent the moments of the underlying probability distribution (mean, mode, median, variance) As well, the phenomenon of “merge bias” at the merge points of parallel task streams move the completion probability point to the right in unpredictable ways. The critical path is often referred to as the laugh track of the project. : The Dark Side of PERT
  88. 88. Programmatic Risk Analysis 88/186 Some Useful (and dreadful) History § The original paper (Malcolm 1959) states – The method is “the best that could be done in a real situation within tight time constraints.” – The time constraint was One Month § The PERT time made the assumption that the standard deviation was about 1/6 of the range (b–a), resulting in the PERT formula. § It has been shown that the PERT mean and standard deviation formulas are poor approximations for most Beta distributions (Keefer 1983 and Keefer 1993). – Errors up to 40% are possible for the PERT mean – Errors up to 550% are possible for the PERT standard deviation : The Dark Side of PERT
  89. 89. Programmatic Risk Analysis 89/186 Critical Path and Most Likely § Critical Path’s are Deterministic – At least one path exists through the network – The critical path is identified by adding the “single point” estimates – The critical predicts the completion date only if everything goes according to plan (we all know this of course) § Schedule execution is Probabilistic – There is a likelihood that some durations will comprise a path that is off the critical path – The single number for the estimate – the “single point estimate” is in fact a most likely estimate – The completion date is not the most likely date, but is a confidence interval in the probability distribution function resulting from the convolution of all the distributions along all the paths to the completion of the project : The Dark Side of PERT
  90. 90. Programmatic Risk Analysis 90/186 Some (False) Assumptions of PERT § Three point estimates follow the Beta distribution – Using the simplified algebraic formula – But this formula has built in biases not revealed in normal use § The expected completion time and variance are calculated by summing the mean and variance of critical path activities – The central limit theorem suggests that if there are sufficient activities on the critical path, the activity times are independent – If the activity times follow a probabilistic distribution with no one activity time dominating, then the sum of activity times on the critical path follows approximately the normal distribution § The result are derived completion times using the normal distribution with a built–in optimistic bias : The Dark Side of PERT
  91. 91. Programmatic Risk Analysis 91/186 Merge Bias Must Be Addressed Upfront Before Management Gets Involved § The standard approach to schedule risk analysis § A “merge point” is where two tasks have a common successor § PERT naively assumes the pdf’s are identical for all tasks : The Dark Side of PERT
  92. 92. Programmatic Risk Analysis 92/186 The Architecture of Merge Bias Starts with Parallel Tasks Landing on a Single Node § The Cumulative Distribution Function (CDF) is biased by the merge points – The CDF is the source of samples for the Monte Carlo simulation Effect of the Merge Bias on Schedule Risk 0% 20% 40% 60% 80% 100% 10/6 11/25 1/14 3/4 Date Cumulative% One Path Three Path : The Dark Side of PERT
  93. 93. Programmatic Risk Analysis 93/186 Merge Point Bias Can Be Very Confusing and Lead to False Optimism § It is misleading to consider only variances from single predecessors for each node in the critical path – Early start of a node depends on the maximum of finish (or start) times of predecessors – This includes ALL the non–critical paths § Early Start is a random variable that is the maximum of all the non–independent random variables in the network § This effect is strongest if – There are more predecessors – Predecessors have equal or near equal duration – Low dependency among the predecessors § The result is an unrealistic optimism for the expected completion times, but most importantly the variance in the completion times : The Dark Side of PERT
  94. 94. Programmatic Risk Analysis 94/186 Notional View of Merge Bias § Near critical paths bias the completion time of the critical path § The finish (or start) times for all paths are random variables with individual probability distributions § These distributions are “joined” to form a probability distribution for the completion of the project : The Dark Side of PERT
  95. 95. Programmatic Risk Analysis 95/186 Notional Impact of Merge Bias § Activities with duration of 2 have s=0.707 § Activities with duration of 4 have s=1.414 : The Dark Side of PERT
  96. 96. Programmatic Risk Analysis 96/186 Why Does This Happen? § The completion time of a task in the IMS can be considered a random variable – This is a mathematical random variable, independent of our ability to manage to plan. – The distributions of random variables can be “added” by convolving their probability distribution functions. – This is the case independent of the underlying distribution (Beta, Triangle, Gauss, Poisson) § This is a critically important concept – No matter what our planning fidelity, the underlying processes of task completion duration are random variables – These random variables have predictable and unpredictable behaviors that impact the outcome is unfavorable ways : The Dark Side of PERT
  97. 97. Programmatic Risk Analysis 97/186 Conveying the Effect of Merge Bias is Sometimes Difficult, But it is Always There § Most projects have parallel paths, many times at crucial points in the schedule – PDR, CDR, ATLO § “Merge Bias” creates extra risk at these point but extending the probabilistic completion date § This may be the factual case or it may be the result of the statistics § Either way the answer will likely be unacceptable to those without the underlying knowledge § We must both educate and inform before proceeding : The Dark Side of PERT
  98. 98. Programmatic Risk Analysis 98/186 Conclusion of Merge Bias § The Critical Path is pretty much meaningless at the deal level of the project – Dynamic completion times must be modeled with Monte Carlo § Discussing the critical path requires that you look at your watch first to see what time it is – The critical path is not static – It is highly dependent on but stochastic behaviors of the task completion time the emerge from the underlying probability distributions – It is also dependent on the dynamics of the interactions of the network nodes § USE MONTE CARLO AND TRY TO AVOID SPEAKING ABOUT THE CRITICAL PATH IN ANY WORDS OTHER THAN VERY HIGH LEVEL A STATIC PATH THROUGH THE NETWORK : The Dark Side of PERT
  99. 99. Programmatic Risk Analysis 99/186 A Quick Look at Monte Carlo Simulation Georges Louis Leclerc, Comte de Buffon, asked what was the probability that the needle would fall across one of the lines, marked here in green. That outcome will occur only if sinA l q< : A Quick Look at Monte Carlo Simulation
  100. 100. Programmatic Risk Analysis 100/186 Monte Carlo Simulation Provides a Solution to Merge Bias and Other PERT Issues § Yes Monte Carlo is named after the country full of casinos located on the French Rivera § Advantages of Monte Carlo over PERT is that Monte Carlo… – Examines all paths not just the critical path – Provides an accurate (true) estimate of completion » Overall duration distribution » Confidence interval (accuracy range) – Sensitivity analysis of interacting tasks – Varied activity distribution types – not restricted to Beta – Schedule logic can include branching – both probabilistic and conditional – When resource loaded schedules are used – provides integrated cost and schedule probabilistic model : A Quick Look at Monte Carlo Simulation
  101. 101. Programmatic Risk Analysis 101/186 The Monte Carlo Tour Starts with WWII History § Any method which solves a problem by generating suitable random numbers and observing that fraction of the numbers obeying some property. § The Monte Carlo method provides approximate solutions to a variety of mathematical problems by performing statistical sampling experiments on a computer. § The method applies to problems with no probabilistic content as well as to those with inherent probabilistic structure. § The method is named after the city of Monte Carlo in the principality of Monaco, because of a roulette, a simple random number generator. The name and the systematic development of Monte Carlo methods dates from about 1944 and the Manhattan project. : A Quick Look at Monte Carlo Simulation
  102. 102. Programmatic Risk Analysis 102/186 The Monte Carlo Method Can Be Found in Many Scientific and Business Domains § Monte Carlo is a well developed discipline in many areas of science, finance and statistics § Probability distribution functions (pdf's) – the physical (or mathematical) system must be described by a set of pdf's. § Random number generator – a source of random numbers uniformly distributed on the unit interval must be available. § Sampling rule – a prescription for sampling from the specified pdf's, assuming the availability of random numbers on the unit interval, must be given. § Scoring (or tallying) – the outcomes must be accumulated into overall tallies or scores for the quantities of interest. : A Quick Look at Monte Carlo Simulation
  103. 103. Programmatic Risk Analysis 103/186 The Monte Carlo Method is Computationally Based § Tools like Risk+ are just a sample of the many approaches to simulating physical system § Error estimation – an estimate of the statistical error (variance) as a function of the number of trials and other quantities must be determined. § Variance reduction techniques – methods for reducing the variance in the estimated solution to reduce the computational time for Monte Carlo simulation § Parallelization and vectorization – algorithms to allow Monte Carlo methods to be implemented efficiently on advanced computer architectures. : A Quick Look at Monte Carlo Simulation
  104. 104. Programmatic Risk Analysis 104/186 Monte Carlo is Fundamentally a Sampling Approach § The large the number of “runs” the higher the fidelity of the simulation § Size does matter § The “sample space” of random numbers is defined by the probability distribution function of the task completion times § Knowing this pdf is important for quality answers to “how long will it take?” : A Quick Look at Monte Carlo Simulation
  105. 105. Programmatic Risk Analysis 105/186 Using Monte Carlo Starts with the Three Point Estimates, but Goes Far Beyond § Conceptually simple approach – The normal PERT 3–Point estimate can be used (reused) – There is no need for special assumption about the underlying probability distributions of the completion times § The Criticality Index is provided from the simulation – Which tasks are critically impacting the completion time of the project § It is computationally expensive – But desktop computing can keep up (3GHz with 2GB of memory is baseline machine for any serious Monte Carlo work) § Scalable analysis quality – Small runs for testing assumptions: £ 1,000 runs – Larger runs for validation of assumptions: ³ 10,000 runs : A Quick Look at Monte Carlo Simulation
  106. 106. Programmatic Risk Analysis 106/186 Risk+ Quick Overview Task to “watch” (Number3) Most Likely (Duration3) Pessimistic (Duration2) Optimistic (Duration1) Distribution (Number1) : A Quick Look at Monte Carlo Simulation
  107. 107. Programmatic Risk Analysis 107/186 Risk+ Quick Overview § The height of each box indicates how often the project complete in a given interval during the run § The S–Curve shows the cumulative probability of completing on or before a given date. § The standard deviation of the completion date and the 95% confidence interval of the expected completion date are in the same units as the “most likely remaining duration” field in the schedule Date: 9/26/2005 2:14:02 PM Samples: 500 Unique ID: 10 Name: Task 10 Completion Std Deviation: 4.83 days 95% Confidence Interval: 0.42 days Each bar represents 2 days Completion Date Frequency CumulativeProbability 3/1/062/10/06 3/17/06 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Completion Probability Table Prob ProbDate Date 0.05 2/17/06 0.10 2/21/06 0.15 2/22/06 0.20 2/22/06 0.25 2/23/06 0.30 2/24/06 0.35 2/27/06 0.40 2/27/06 0.45 2/28/06 0.50 3/1/06 0.55 3/1/06 0.60 3/2/06 0.65 3/3/06 0.70 3/3/06 0.75 3/6/06 0.80 3/7/06 0.85 3/8/06 0.90 3/9/06 0.95 3/13/06 1.00 3/17/06 Task to “watch” 80% confidence that task will complete by 3/7/06 : A Quick Look at Monte Carlo Simulation
  108. 108. Programmatic Risk Analysis 108/186 A Well Formed Risk+ Schedule § A good critical path network – No constraint dates – Lowest level tasks have predecessors and successors – 80% of relationships are finish to start § Identify risk tasks – These are “reporting tasks” – Identify the preview task to watch during simulation runs § Enter probability distribution profile for each task – Bulk assignment is an easy way to start – 1 – 5 ranking is another approach – Individual risk profile assignments is best but tedious : A Quick Look at Monte Carlo Simulation
  109. 109. Programmatic Risk Analysis 109/186 Analyzing the Risk+ Simulation § Risk + will generate one or more of the following outputs: – Earliest, expected, and latest completion date for each reporting task – Graphical and tabular displays of the completion date distribution for each reporting task – The standard deviation and confidence interval for the completion date distribution for each reporting task – The criticality index (percentage of time on the critical path) for each task – The duration mean and standard deviation for each task – Minimum, expected, and maximum cost for the total project – Graphical and tabular displays of cost distribution for the total project – The standard deviation and confidence interval for cost at the total project level : A Quick Look at Monte Carlo Simulation
  110. 110. Programmatic Risk Analysis 110/186: A Quick Look at Monte Carlo Simulation The Page Intentionally Left Blank
  111. 111. Programmatic Risk Analysis 111/186 Building a Robust IMS Never undertake a project unless it is manifestly important and nearly impossible. — Edwin Land : Building a Robust IMS Edwin Land 1909 – 1991
  112. 112. Programmatic Risk Analysis 112/186 Schedule Contingency Analysis § The schedule contingency needed to make the plan credible can be derived from the Risk+ analysis § The schedule contingency is the amount of time added (or subtracted) from the baseline schedule necessary to achieve the desired probability of an under run or over run. § The schedule contingency can be determined through – Monte Carlo simulations (Risk+) – Best judgment from previous experience – Percentage factors based on historical experience – Correlation analysis for dependency impacts : Building a Robust IMS Is This Our Contingency Plan ?
  113. 113. Programmatic Risk Analysis 113/186 Schedule Quality and Accuracy § Accuracy range – Similar for each estimate class § Consistent with estimate – Level of project definition – Purpose – Preparation effort § Monte Carlo simulation – Analysis of results shows quality attained versus the quality sought (expected accuracy ranges) § Achieving specified accuracy requirements – Select value at end points of confidence interval – Calculate percentages from base schedule completion date, including the contingency : Building a Robust IMS
  114. 114. Programmatic Risk Analysis 114/186 Technical Performance Measures § Technical Performance Measures are one method of showing risk by done – Specific actions taken in the IMS to move the compliance forward toward the goal § Activities that assessing the increasing compliance to the technical performance measure can be show in the IMS – These can be Accomplishment Criteria : Building a Robust IMS
  115. 115. Programmatic Risk Analysis 115/186 Elements of a Risk Tolerant Program CAIV APB Schedule Risk Cost Risk Technical Risk • Critical Path • Monte Carlo • IMS Maturity • Systems Engineering • Simulation & Test • Design • TPM • WBS • Cost Estimates • EVMS CAIV – Cost as an Independent Variable APB – Allocated Program Baseline When the three independent variables of the program are FIXED in the beginning. Building in risk tolerance now becomes a real challenge : Building a Robust IMS
  116. 116. Programmatic Risk Analysis 116/186 Inserting Risk Tolerant in the IMS § Technical risk management using ARM – Identification – Classification – Triage § Programmatic risk management – Mitigation tasks – Maturity enhancement – Technical Performance Measures § Tracking and Communication – Risk ID embedded in the IMS – Resource assignment – Cost model cross connections : Building a Robust IMS
  117. 117. Programmatic Risk Analysis 117/186 Moving forward § The previous approaches to the development of an IMS was to build the IMP and then the IMS and all the supporting tasks. § The tasks define the work effort to deliver the ACs and the product § Simple RISK+ metrics run at the end § Explicit risk mitigation provides another view – Risks identified in ARM – Mitigation tasks for those risks are made explicit in the IMS – Branching probabilities for each mitigation initiation events – Monte Carlo analysis of the main stream and mitigation effort : Building a Robust IMS
  118. 118. Programmatic Risk Analysis 118/186 Integrating Risk and Schedule § Probabilistic completion times change as the program matures § The efforts that produce these improvements must be traceable in the IMS § The “error bands” on the events must include the risk mitigation activities as well § IMS activities show how the “error band” narrows over time. – This is the basis of a “programmatic risk tolerant” IMS – The probabilistic interval becomes more reliable as risk mitigations and maturity assessments add confidence the to IMS Baseline Plan 80% Mean Missed Launch Period Launch Period Ready Early Oct 07 Nov 07 Dec 07 Jan 08 Feb 08 Mar 08 Apr 08 May 08 Jun 08 Plan Margin Current Plan with risks is the stochastic schedule CDR PDR SRR FRR ATLO 20% Aug 05 Jan 06 Aug 06 Mar 07 Dec 07 Feb 08 Current Plan with risks is the deterministic schedule Risk Margin : Building a Robust IMS
  119. 119. Programmatic Risk Analysis 119/186 Event Driven Risk Management § Water fall risk management integrated with the IMP or high level IMS § We’re missing this connection at the moment § But making it visible in the simplest manner is the first step in building awareness § Technical risk management and programmatic risk management must be integrated into the IMS : Building a Robust IMS
  120. 120. Programmatic Risk Analysis 120/186 Steps in the Process of a Risk Tolerant IMS 1. Identifying the risks a) ARM and risk trace number b) Mitigation tasks c) Duration and resources for the mitigation 2. Integrate risk in the IMS a) Embedded tasks with IMP/IMS numbers b) Network tasks c) Assess Monte Carlo impact on completion dates 3. Evaluate outcome a) Events and SAs within acceptable confidence interval? b) If not, risk mitigation activities need improvement c) If so, baseline mitigation and narrative supporting the strategy 4. Connect Risk Management (ARM) and IMS at the tracking number level : Building a Robust IMS
  121. 121. Programmatic Risk Analysis 121/186 Branching Probabilities – Simple Approach § Plan the risk alternatives that “might” be needed – Each mitigation has a Plan B branch – Keep alternatives as simple as possible (maybe one task) § Assess probability of the alternative occurring § Assign duration and resource estimates to both branches § Turn off for alternative for a “success” path assessment § Turn off primary for a “failure” path assessment 30% Probability of failure 70% Probability of success Plan B Plan A Current Margin Future Margin 80% Confidence for completion with current margin Duration of Plan B Plan A + Margin£ : Building a Robust IMS
  122. 122. Programmatic Risk Analysis 122/186 Managing Margin in the Risk Tolerant IMS requires the reuse of unused durations § Programmatic Margin is added between Development, Production and Integration & Test phases § Risk Margin is added to the IMS where risk alternatives are identified § Margin that is not used in the IMS for risk mitigation will be moved to the next sequence of risk alternatives – This enables us to buy back schedule margin for activities further downstream – This enables us to control the ripple effect of schedule shifts on Margin activities 5 Days Margin 5 Days Margin Plan B Plan A Plan B Plan AFirst Identified Risk Alternative in IMS Second Identified Risk Alternative in IMS 3 Days Margin Used Downstream Activities shifted to left 2 days Duration of Plan B < Plan A + Margin 2 days will be added to this margin task to bring schedule back on track : Building a Robust IMS
  123. 123. Programmatic Risk Analysis 123/186 Simulation Considerations of Monte Carlo need to be understood to use the data § Multiple Critical Paths – Microsoft Project™ lacks the capability to: » Identify off critical path impacts on the critical path » Identify effects of branching – This can be manually compensated for by: » Examine each activity’s criticality associated wit more than one critical path » Path analysis may give substantially different vie of the schedule’s uncertainties » Use bar chart to illustrate multiple critical paths § Monte Carlo examines ALL paths not just the critical path – Near critical paths are exposed : Building a Robust IMS
  124. 124. Programmatic Risk Analysis 124/186 Simulation Considerations § Schedule logic and constraints – Simplify logic – model only paths which, by inspection, may have a significant bearing on the final result – Correlate similar activities – No open ends – Use only finish–to–start relationships with no lags – Model relationships other than finish–to–start as activities with base durations equal to the lag value – Eliminate all date constraints – Consider using branching for known alternatives : Building a Robust IMS
  125. 125. Programmatic Risk Analysis 125/186 Simulation Considerations § Selection of Probability Distributions – Develop schedule simulation inputs concurrently with the cost estimate » Early in process – use same subject matter experts » Convert confidence intervals into probability duration distributions – Number of distributions vary depending on software – Difficult to develop inputs required for distributions – Beta and Lognormal better than triangular; avoid exclusive use of Normal distribution : Building a Robust IMS
  126. 126. Programmatic Risk Analysis 126/186 Sensitivity Analysis describes which tasks drive the completion times § Concentrates on inputs most likely to improve quality (accuracy) § Identifies most promising opportunities where additional work will help to narrow input ranges § Methods – Run multiple simulations – Use criticality index – “Tornado” or Pareto graph : Building a Robust IMS
  127. 127. Programmatic Risk Analysis 127/186 Models of the Schedule All models are lies. Some models are useful. – George Box : Models of the Schedule Concept generator from Ramon Lull’s Ars Magna (C. 1300)
  128. 128. Programmatic Risk Analysis 128/186 Graphical Interpretations provide information that numbers alone can’t § Graphical view of confidence, contingency and target management assessment : Models of the Schedule
  129. 129. Programmatic Risk Analysis 129/186 What Can Confidence Intervals Tell Us about the validity of the IMS? § As the program proceeds so does – Increasing accuracy – Reduced schedule risk – Increasing visual confirmation that success can be reached Current Estimate Accuracy : Models of the Schedule
  130. 130. Programmatic Risk Analysis 130/186 Confidence of schedule dates : Models of the Schedule
  131. 131. Programmatic Risk Analysis 131/186 Sensitivity Analysis § The schedule sensitivity of a task measures the closeness with which change in the task duration matches change in the project duration over the simulation. § This closeness is called correlation – Correlations can be derived from the sampling processes § A task with high schedule sensitivity is more likely to be a major driver of the project duration than a lower ranked task : Models of the Schedule
  132. 132. Programmatic Risk Analysis 132/186 Task Criticality Analysis § A measure of the frequency that an activity in the project schedule is critical (Total Float = 0) in a simulation § If a task is critical in 500 of the 1,000 iterations of the simulation, it has a Criticality Index of 0.5 § The higher the criticality index, the more certain it is that the task will always be critical in the project : Models of the Schedule
  133. 133. Programmatic Risk Analysis 133/186 Schedule Cruciality describes how important each tasks is the risk tolerance § Cruciality = Schedule Sensitivity X Criticality § Schedule Sensitivity can be statistically misleading: – A task with high sensitivity may not be on or near the critical path – Thus a reduction in that task’s duration may have little effect on the project duration § Cruciality sharpens the analytical focus: – It highlights critical or near–critical activities with high § Schedule Sensitivity – These tasks are most likely to drive project duration : Models of the Schedule
  134. 134. Programmatic Risk Analysis 134/186: Models of the Schedule The Page Intentionally Left Blank
  135. 135. Programmatic Risk Analysis 135/186 Examples Of Monte Carlo A simple overview of Risk+ shows how to produce an estimate of a project completion date. Interpreting this information takes thought and practice and most importantly reading and rereading the manual and the resources provided at the end of this presentation : Examples of Monte Carlo Sewall Wright’s probabilistic network notation (1921)
  136. 136. Programmatic Risk Analysis 136/186 The Monte Carlo Process starts with the PERT 3 point estimates § Estimates of the task duration are still needed, just like they are in PERT – Three point estimates could be used – But risk ranking and algorithmic generation of the “spreads” is a better approach § Duration estimates must be parametric rather than numeric values – A geometric scale of parametric risk is one approach § Branching probabilities need to be defined – Conditional paths through the schedule can be evaluated using Monte Carlo tools – This also demonstrate explicit risk mitigation planning to answer the question “what if this happens?” : Examples of Monte Carlo
  137. 137. Programmatic Risk Analysis 137/186 Expert Judgment is required to build a Risk Management approach § Expert judgment is typically the basis of cost and schedule estimates – Expert judgment is usually the weakest area of process and quantification – Translating from English (SOW) to mathematics (probabilistic risk model) is usually inconsistent at best and erroneous at worst § One approach – Plan for the “best case” and preclude a self–fulfilling prophesy – Budget for the “most likely” and recognize risks and uncertainties – Protect for the “worst case” and acknowledge the conceivable in the risk mitigation plan § The credibility of the “best case” estimates if crucial to the success of this approach : Examples of Monte Carlo
  138. 138. Programmatic Risk Analysis 138/186 Guiding the Risk Factor Process requires careful weighting of each level of risk § For tasks marked “Low” a reasonable approach is to score the maximum 10% greater than the minimum. § The “Most Likely” is then scored as a geometric progression for the remaining categories with a common ratio of 1.5 § Tasks marked “Very High” are bound at 200% of minimum. – No viable project manager would like a task grow to three times the planned duration without intervention § The geometric progress is somewhat arbitrary but it should be used instead of a linear progression Min Most Likely Max Low 1.0 1.04 1.10 Low+ 1.0 1.06 1.15 Moderate 1.0 1.09 1.24 Moderate+ 1.0 1.14 1.36 High 1.0 1.20 1.55 High+ 1.0 1.30 1.85 Very High 1.0 1.46 2.30 Very High+ 1.0 1.68 3.00 : Examples of Monte Carlo
  139. 139. Programmatic Risk Analysis 139/186 Progressive Risk Factors § A geometric progression (1.534) of risk can be used § The phrases associated with increasing risk have been shown at the Naval Research Laboratory to correlate with an engineers “sense” of increasing risk : Examples of Monte Carlo
  140. 140. Programmatic Risk Analysis 140/186 Risk Factor Attributes § The “narrative” for each risk factor needs to be developed § Each description may be dependent on… – Discipline – Program stage – Complexity – Historical data – Current “risk state” of the program § This approach is similar to NASA’s Technology Readiness Level § This is currently missing from our efforts to quantify schedule and cost risk : Examples of Monte Carlo
  141. 141. Programmatic Risk Analysis 141/186 Extreme Serial Task Example § Task completion distributions are “added” with little effect on the project completion estimate Date: 9/26/2005 2:03:28 PM Samples: 500 Unique ID: 10 Name: Task 10 Completion Std Deviation: 2.51 days 95% Confidence Interval: 0.22 days Each bar represents 1 day Completion Date Frequency CumulativeProbability 2/10/062/1/06 2/23/06 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 Completion Probability Table Prob ProbDate Date 0.05 2/6/06 0.10 2/7/06 0.15 2/8/06 0.20 2/8/06 0.25 2/9/06 0.30 2/9/06 0.35 2/10/06 0.40 2/10/06 0.45 2/10/06 0.50 2/10/06 0.55 2/13/06 0.60 2/13/06 0.65 2/13/06 0.70 2/14/06 0.75 2/14/06 0.80 2/14/06 0.85 2/15/06 0.90 2/16/06 0.95 2/17/06 1.00 2/23/06 : Examples of Monte Carlo
  142. 142. Programmatic Risk Analysis 142/186 Extreme Parallel Task Example Date: 9/26/2005 2:14:02 PM Samples: 500 Unique ID: 10 Name: Task 10 Completion Std Deviation: 4.83 days 95% Confidence Interval: 0.42 days Each bar represents 2 days Completion Date Frequency CumulativeProbability3/1/062/10/06 3/17/06 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 Completion Probability Table Prob ProbDate Date 0.05 2/17/06 0.10 2/21/06 0.15 2/22/06 0.20 2/22/06 0.25 2/23/06 0.30 2/24/06 0.35 2/27/06 0.40 2/27/06 0.45 2/28/06 0.50 3/1/06 0.55 3/1/06 0.60 3/2/06 0.65 3/3/06 0.70 3/3/06 0.75 3/6/06 0.80 3/7/06 0.85 3/8/06 0.90 3/9/06 0.95 3/13/06 1.00 3/17/06 § Task completion distributions are “added” with a large unfavorable effect on the project completion estimate distribution : Examples of Monte Carlo
  143. 143. Programmatic Risk Analysis 143/186 A “Real World” Schedule Analysis One should expect that the expected can be prevented, but the unexpected should have been expected — Augustine Law XLV : “Real World” Schedule Analysis
  144. 144. Programmatic Risk Analysis 144/186 The Baseline Schedule § Sample construction project plan : “Real World” Schedule Analysis
  145. 145. Programmatic Risk Analysis 145/186 PERT Assessment Most Likely Min Max PERT Adjusted Duration PERT Adjusted Date Original Target Date: 2/8/06 : “Real World” Schedule Analysis
  146. 146. Programmatic Risk Analysis 146/186 Risk+ Assessment Most Likely Min Max Date: 10/4/2005 1:58:06 PM Samples: 23 Unique ID: 143 Name: Construction Schedule Margin Completion Std Deviation: 1.55 days 95% Confidence Interval: 0.63 days Each bar represents 1 day Completion Date Frequency CumulativeProbability 2/9/062/3/06 2/14/06 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Completion Probability Table Prob ProbDate Date 0.05 2/6/06 0.10 2/6/06 0.15 2/7/06 0.20 2/7/06 0.25 2/7/06 0.30 2/8/06 0.35 2/8/06 0.40 2/9/06 0.45 2/9/06 0.50 2/9/06 0.55 2/9/06 0.60 2/9/06 0.65 2/9/06 0.70 2/10/06 0.75 2/10/06 0.80 2/10/06 0.85 2/10/06 0.90 2/10/06 0.95 2/10/06 1.00 2/14/06 Target Date 80% confidence : “Real World” Schedule Analysis
  147. 147. Programmatic Risk Analysis 147/186 What is the Purpose of Project Risk Analysis? What do users want from a project risk analysis? How accurate must we be to provide value to the program? Can we confirm this accuracy and integrity to build confidence in the projected completion date? : What is the Purpose of Project Risk Analysis? Risk appears in all aspects of spaceflight
  148. 148. Programmatic Risk Analysis 148/186 Accuracy § Given a specified final cost or project duration, what is the probability of achieving this cost or duration? § Frequentist approach – Over many different projects, four out of five will cost less or be completed in less time than the specified cost or duration § Bayesian approach – We would be willing to bet at 4 to 1 odds that the project will be under the 80% point in cost or duration § Accuracy is needed to plan reserves § Accuracy is needed when comparing competing proposals : What is the Purpose of Project Risk Analysis?
  149. 149. Programmatic Risk Analysis 149/186 Structured Thinking § All estimates will be in error § Trying to quantify these errors will result in bounds too wide to be useful for decision making § Risk analysis should be used to – Think about different aspects of the project – Try to put numbers against probabilities and impacts – Discuss with colleagues the different ideas and perceptions § Thinking things through carefully results in – Which elements of the programmatic and technical risk are represented in the IMS – The process becomes more valuable than the numbers : What is the Purpose of Project Risk Analysis?
  150. 150. Programmatic Risk Analysis 150/186: What is the Purpose of Project Risk Analysis? The Page Intentionally Left Blank
  151. 151. Programmatic Risk Analysis 151/186 Basic Principles of Probabilistic Cost Now that the schedule can be produced using probabilistic methods, it’s time to talk about the cost. Cost does not have a linear relationship with schedule unfortunately : Basic Principles of Probabilistic Cost
  152. 152. Programmatic Risk Analysis 152/186 Basic Principles with Probabilistic Cost Estimating § Cost estimates usually involve many CERs – Each of these CERs has uncertainty (standard error) – CER input variables have uncertainty (technical uncertainty) § Must combine CER uncertainty with technical uncertainty for many CERs in an estimate – Usually cannot be done arithmetically; must use simulation to roll up costs derived from Monte Carlo samples » Add and multiply probability distributions rather than numbers » Statistically combining many uncertain, or randomly varying, numbers – Monte Carlo simulation » Take random sample from each CER and input parameter, add and multiply as necessary, then record total system cost as a single sample » Repeat the procedure thousands of times to develop a frequency histogram of the total system cost samples » This becomes the probability distribution of total system cost : Basic Principles of Probabilistic Cost
  153. 153. Programmatic Risk Analysis 153/186 The Cost Probability Distributions as a function of the weighted cost drivers $ Cost Driver (Weight) Cost = a + bXc Cost Estimate Historical data point Cost estimating relationship Standard percent error boundsTechnical Uncertainty Combined Cost Modeling and Technical Uncertainty Cost Modeling Uncertainty : Basic Principles of Probabilistic Cost
  154. 154. Programmatic Risk Analysis 154/186 Basic Principles of connected Cost with the IMS involve three steps § Step 1: Define “likely–to–be” program – Using deterministic inputs from the Independent Technical Assessment (ITA) § Step 2: Quantify the probability distributions describing the modeling uncertainty of all CERs, cost factors, and other estimating methods – Specifically, the type of distribution (normal, triangular, lognormal, beta, etc.) – The mean and variance of the distribution § Step 3: Quantify the correlation between all WBS elements that are estimated using CERs and other methods – If unknown, assess whether No correlation, Mild correlation, or High correlation, for example: » None: r = 0, Mild: r = ±0.2, High: r = ± 0.6 – Correlation affects the overall cost variance : Basic Principles of Probabilistic Cost
  155. 155. Programmatic Risk Analysis 155/186 Basic Principles § Step 4: Set up and run the cost estimate in a Monte Carlo framework (e.g., Crystal Ball, @RISK), resulting in a “baseline” estimate – This will provide a probability distribution of the cost based on cost estimating model uncertainty only – Report the MEAN as the baseline expected cost § Step 5: Now incorporate technical uncertainty and discrete risks – Step 5a: Set up a new estimate which also contains any “discrete risk” events that are to be guarded against » Quantify appropriate modeling uncertainties and correlations, as in Steps 2 and 3, for these discrete risks – Step 5b: Define the probability distributions for all CER input variables » Also may need to quantify correlation between CER input variables : Basic Principles of Probabilistic Cost
  156. 156. Programmatic Risk Analysis 156/186 Basic Principles § Step 6: Re–run the Monte Carlo simulation with random CER input variables and discrete risk events, resulting in a final “risk–adjusted” estimate – Results in a new risk–adjusted cost probability distribution. – Wider and shifted to the right Baseline vs. Risk-Adjusted Estimates 0 50 100 150 200 250 300 350 FY$M Likelihood : Basic Principles of Probabilistic Cost
  157. 157. Programmatic Risk Analysis 157/186 Basic Principles § Step 7: Assess “risk dollars” – Difference between the “risk–adjusted” mean and the “baseline” mean represents the estimate of “risk dollars” – Risk dollars can be allocated downward to any level of WBS using a variety of simple approaches Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate $346M Baseline Estimate $270M Risk Dollars Relative to Baseline Estimate $76M : Basic Principles of Probabilistic Cost
  158. 158. Programmatic Risk Analysis 158/186 Basic Principles § Step 8: Assess “budget risk” – Area under the PDF to the right of the budget represents budget risk Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget Risk = 51% Budget Risk = 51% Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget Risk = 51% Budget Risk = 45% Note: Assumes budget is set at the risk-adjusted estimate expected value. Note: Assumes budget is set at the risk-adjusted estimate expected value. Budget = $346M Budget = $346M : Basic Principles of Probabilistic Cost
  159. 159. Programmatic Risk Analysis 159/186 Basic Principles § Step 9: Assess “management reserve” – Difference between mean of risk–adjusted estimate and budget represents management reserve Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget $405M Budget $405M Unencumbered Margin $59M Unencumbered Margin $59M Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget $405M Budget $405M Unencumbered Margin $59M Unencumbered Margin $59M Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate 0 100 200 300 400 500 600 700 800 FY$M Likelihood Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget $405M Budget $405M Unencumbered Margin $59M Unencumbered Margin $59M Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget $405M Budget $405M Unencumbered Margin $59M Management Reserve $59M Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget $405M Budget $405M Unencumbered Margin $59M Unencumbered Margin $59M Risk-Adjusted Estimate $346M Risk-Adjusted Estimate $346M Budget $405M Budget $405M Unencumbered Margin $59M Management Reserve $59M Note: If budget is set at risk- adjusted expected value, then management reserve is ZERO. Note: If budget is set at risk- adjusted expected value, then management reserve is ZERO. : Basic Principles of Probabilistic Cost
  160. 160. Programmatic Risk Analysis 160/186 The Risk Adjusted Cost Estimate Connected To The IMS Is The Basis Of Risk Tolerance § In the risk–adjusted cost estimate, we now combine discrete risk events and the uncertainty of the input distributions with the uncertainty of the CERs § Since the input distributions tend to be right–skewed, the expected cost tends to be larger than the baseline estimate § In addition, the risk–adjusted cost distribution tends to be wider than the baseline estimate § The difference between the expected cost of the risk–adjusted estimate and the expected cost of the baseline estimate is, by definition, the amount of RISK dollars included in the risk– adjusted estimate : Basic Principles of Probabilistic Cost
  161. 161. Programmatic Risk Analysis 161/186 Baseline versus Risk Adjusted Cost Estimates Almost Always Show an Increase In Cost Baseline vs. Risk-Adjusted Estimates 0 50 100 150 200 250 300 350 FY$M Likelihood Baseline: Mean = $102.6M Std Dev = $29.8M Risk–Adjusted: Mean = $122.6M Std Dev = $42.8M : Basic Principles of Probabilistic Cost
  162. 162. Programmatic Risk Analysis 162/186 The S–Curve for Cost Modeling Cumulative Distribution Function 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% $60 $80 $100 $120 $140 $160 $180 $200 FY00$M CumulativeProbability Baseline Estimate Mean $102.6M 50th percentile $114.7M Risk–adjusted Estimate Mean $122.6M 80th percentile $153.5M : Basic Principles of Probabilistic Cost
  163. 163. Programmatic Risk Analysis 163/186 The Real Question Always Returns to… “But How Much Does It Cost? Really?” § This is impossible to answer precisely § Decision–makers and cost analysts should always think of a cost estimate as a probability distribution, NOT as a deterministic number § The best we can provide is the probability distribution – If we think we can be any more precise, we’re fooling ourselves § It is up to the decision–maker to decide where he/she wants to set the budget § The probability distribution provides a quantitative basis for making this determination – Low budget = high probability of overrun – High budget = low probability of overrun : Basic Principles of Probabilistic Cost
  164. 164. Programmatic Risk Analysis 164/186: Basic Principles of Probabilistic Cost The Page Intentionally Left Blank
  165. 165. Programmatic Risk Analysis 165/186 Summary At this point there is too much information. Processing of this information will take time, patience, and most of all practice with the tools and the results they produce. But there are some fundamental conclusions that can be applied to our problem at hand – Phase II : Summary I’ll be happy to give you innovative thinking. What are my guidelines? With our new found knowledge we need to break out of the habits of the past and start applying probabilistic cost and schedule analysis to our program
  166. 166. Programmatic Risk Analysis 166/186 Conclusions § Project schedule status must be assessed in terms of a critical path through the schedule network § Because the actual durations of each task in the network are uncertain (they are random variables following a probability distribution function), the project schedule duration must be modeled statistically : Summary
  167. 167. Programmatic Risk Analysis 167/186 Conclusions § Quality (accuracy) is measured at the end points of achieved confidence interval (suggest 80% level) § Simulation results depend on: – Accuracy and care taken with base schedule logic – Use of subject matter experts to establish inputs – Selection of appropriate distribution types – Through analysis of multiple critical paths – Understanding which activities and paths have the greatest potential impact : Summary