Programmatic risk management workshop (slides)


Published on

An introduction to programmatic risk management.

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The statistical processes needed for probabilistic risk analysis vary according to the level of understanding needed. A top level view is that each task’s duration is a random variable. When these random variables are connected in a network (a schedule), the end date is a random variable as well.There are many more subtleties though in this concept..What kind of random variables?From what distributions are they drawn?How are they independent for each run of the model?What couplings are there between the individual tasks and the end date of the project?What is the confidence interval on this probabilistic end date – that is what is the variance in the variance?Why are the counter intuitive aspects of the modeling process? For example, when a task duration is reduced the probabilities end date gets longer. Why? How can this be?
  • When we use the term uncertainty or risk it means at least 4 things.First let’s sort of “uncertainty”There are two classes of uncertainty in large complex programs.Static uncertainty emerges from the natural variations in the completion times of tasks. This is a Deming uncertainty. is an example of this type of uncertaintyThe dynamic uncertainty is about the unknowns and the unknowable
  • In deterministic PERT the durations are defined as a three point estimate and the PERT formula is used to compute the mean and standard deviation for the program duration as well as the critical path. This is the algorithm used in Microsoft Project when the PERT tool bar is turned on and the three point estimates entered into the appropriate columns.It is billed as probabilistic but in fact the 3–point estimates work against a fixed probability distribution function with no way to adjust its shape, bounds or moments.As well, there is no way to insert the correlations that naturally occur in the IMS.
  • In deterministic PERT the durations are defined as a three point estimate and the PERT formula is used to compute the mean and standard deviation for the program duration as well as the critical path. This is the algorithm used in Microsoft Project when the PERT tool bar is turned on and the three point estimates entered into the appropriate columns.It is billed as probabilistic but in fact the 3–point estimates work against a fixed probability distribution function with no way to adjust its shape, bounds or moments.As well, there is no way to insert the correlations that naturally occur in the IMS.
  • The result is a cumulative distribution and a probability distribution function.Interpreting this result is straight forward.The confidence of each date is shown in the table on the right. This is the probability of completing the task by the date.
  • When we use the term “credible” the first question is “what are the units of measure of credible?”
  • Thinking about schedule contingency is different in a Probabilistic Risk Analysis context. For a simple project, 15% contingency is assumed. But placing the contingency is the first problem. The process is:Run Risk+ and watch the final date.Compare the 80% confidence date against the deterministic date. This difference is the first cut at the needed margin.Assign this duration across the project in front of the critical (high risk) milestones.Rerun Risk+ and add or subtract this margin until the desired confidence date is achieved.
  • The use of “expert judgment” itself needs to be calibrated.The unanswered question on this program and many others is “what does a good risk tolerant IMS actually look like?”The “units of measure” for risk tolerance and the confidence in the probabilistic estimates needs to be established before the estimating and modeling process can be “calibrated”
  • The ranking of risk or the ranking of anything needs to be done in a structured manner.A geometric progression is a very useful approach, since it forces the focus on ranking.
  • The use of branching probabilities is important for the assessment of the “risk tolerance.”
  • Simulating the schedule requires an understanding of the underlying probability distributions. This can range from simple to sophisticated. The simple approach takes the standard out of the box distributions from Risk+ - a Low, Medium, High confidence and a Triangle distribution and applies this to all the tasks.The sophisticated approach looks at groups of tasks – flight software, hardware development, testing, integration, operations, etc. and develops actual probability distributions for these classes of activities.Risk+ provides only Triangle, Normal, and Beta distributions. @Risk for Project provides many others. Either way, the development of the details of the distribution is needed before any assessment can be done.
  • By this time your into information overload. But this is the nature of detailed project schedule analysis. There is just a lot of information. This is why there is usually one person responsible for the modeling of the schedule. It’s a full time job just to get your head around the topic.
  • Using Risk+ produces a probability distribution function (pdf) for the activity being “watched.” The confidence measure of the resulting date distributions needs to be in the 70% to 80% range to have any credibility.Getting the schedule “tuned” for result requires effort and patience. The first runs will be very disappointing and will require more work to figure out what is going on.
  • The confidence intervals produced by the CDF can be assessed over time against targets. These targets can be Technical Performance Measures or any other style of metric that is connected with cost, schedule and technical performance.As the program proceeds, the risks should be reduced as well as the tolerance for risk.
  • This whole topic is about risk management. But in the end risk management is about project management.
  • Managing risk is a project management practice. As a practice, it takes practices. And like all practice, practice make perfect – or as close to perfect as you can get in the project management business.
  • This is a list of books and papers that are the “tip of the iceberg” for programmatic risk management. Look through some of these to start to get a handle on the complexities of the topic. It may appear daunting but with a little work, the topic starts to reveal itself.
  • Programmatic risk management workshop (slides)

    1. 1. Programmatic Risk Management:A “not so simple” introduction to the complex but critical process of building a “credible” schedule<br />Workshop, Lewis & Fowler Team, Denver, Colorado<br />October 6th and October 14th 2008<br />1/69<br />Programmatic Risk Management Work (Handbook)<br />
    2. 2. Agenda<br />2/69<br />
    3. 3. When we say “Risk Management” What do we really mean?<br />3/69<br />
    4. 4. Five Easy Pieces†:The Essentials of Managing Programmatic Risk<br />Managing the risk to cost, schedule, and technical performance is the basis of a successful project management method.<br />† With apologies to Carole Eastman and Bob Rafelson for their 1970 film staring Jack Nicholson<br />Risk in Five Easy Pieces<br />4/69<br />
    5. 5. Hope is Not a Strategy<br />When General Custer was completely surrounded, his chief scout asked, “General what's our strategy?” Custer replied, “The first thing we need to do is make a note to ourselves – never get in this situation again.” <br />Hope is not a strategy!<br />A Strategy is the plan to successfully complete the project<br />If the project’s success factors, the processes that deliver them, the alternatives when they fail, and the measurement of this success are not defined in meaningful ways for both the customer and managers of the project – Hope is the only strategy left.<br />Risk in Five Easy Pieces<br />5/69<br />
    6. 6. No Single Point Estimate can be correct without knowing the variance<br />SinglePoint Estimates use sample data to calculate a single value (a statistic) that serves as a "best guess" for an unknown (fixed or random) population parameter<br />Bayesian Inference is a statistical inference where evidence or observations are used to infer the probability that a hypothesis may be true<br />Identifying underlying statistical behavior of the cost and schedule parameters of the project is the first step in forecasting future behavior<br />Without this information and the model in which it is used any statements about cost, schedule and completion dates are a 50/50guesses<br />When estimating cost and duration for planning purposes using Point Estimates results in the least likely result.<br />A result with a 50/50 chance of being true.<br />Risk in Five Easy Pieces<br />6/69<br />
    7. 7. Without Integrating $, Time, and TPM you’re driving in the rearview mirror<br />Cost ($)<br />Schedule (t)<br />Technical<br />Performance (TPM)<br />Addressing customer satisfaction means incorporating product requirements and planned quality into the Performance Measurement Baseline to assure the true performance of the project is made visible.<br />Risk in Five Easy Pieces<br />7/69<br />
    8. 8. Without a model for risk management, you’re driving in the dark with the headlights turn off<br />The Risk Management process to the right is used by the US DOD and differs from the PMI approach in how the processes areas are arranged.<br />The key is to understand the relationships between these areas.<br />Risk Management means using a proven risk management process, adapting this to the project environment, and using this process for everyday decision making.<br />Risk in Five Easy Pieces<br />8/69<br />
    9. 9. Risk Communication is …<br />An interactive process of exchange of information and opinion among individuals, groups, and institutions; often involving multiple messages about the nature of risk or expressing concerns, opinions, or reactions to risk messages or to legal or institutional arrangements for risk management.<br />Bad news is not wine. It does not improve with age — Colin Powell<br />Risk in Five Easy Pieces<br />9/69<br />
    10. 10. Basic Statistics for Programmatic Risk Management<br />Since all point estimates are wrong, statistical estimates will be needed to construct a credible cost and schedule model<br />Basic Statistics<br />10/69<br />
    11. 11. Uncertainty and Risk are not the same thing – don’t confuse them<br />Uncertainty stems from unknown probability distributions<br />Requirements change impacts<br />Budget Perturbations<br />Re–work, and re–test phenomena<br />Contractual arrangements (contract type, prime/sub relationships, etc)<br />Potential for disaster (labor troubles, shuttle loss, satellite “falls over”, war, hurricanes, etc.)<br />Probability that if a discrete event occurs it will invoke a project delay<br />Risk stems from known probability distributions<br />Cost estimating methodology risk resulting from improper models of cost<br />Cost factors such as inflation, labor rates, labor rate burdens, etc<br />Configuration risk (variation in the technical inputs)<br />Schedule and technical risk coupling<br />Correlation between risk distributions<br />Basic Statistics<br />11/69<br />
    12. 12. There are 2 types of Uncertainty encountered in cost and schedule<br />Static uncertainty is natural variation and foreseen risks<br />Uncertainty about the value of a parameter<br />Dynamic uncertainty is unforeseen uncertainty and “chaos”<br />Stochastic changes in the underlying environment<br />System time delays, interactions between the network elements, positive and negative feedback loops<br />Internal dependencies <br />Basic Statistics<br />12/69<br />
    13. 13. The Multiple Sources of Schedule Uncertainty and Sorting Them Out is the Role of Planning<br />Unknown interactions drive uncertainty<br />Dynamic uncertainty can be addressed by flexibility in the schedule<br />On ramps<br />Off ramps<br />Alternative paths<br />Schedule “crashing” opportunities<br />Modeling of this dynamic uncertainty requires simulation rather than static PERT based path assessment<br />Changes in critical path are dependent on time and state of the network<br />The result is a stochastic network<br />Basic Statistics<br />13/69<br />
    14. 14. Statistics at a Glance<br />Probability distribution – A function that describes the probabilities of possible outcomes in a "sample space.”<br />Random variable – variable a function of the result of a statistical experiment in which each outcome has a definite probability of occurrence. <br />Determinism – a theory that phenomena are causally determined by preceding events or natural laws.<br />Standard deviation (sigma value) – An index that characterizes the dispersion among the values in a population.<br />Bias –The expected deviation of the expected value of a statistical estimate from the quantity it estimates.<br />Correlation – A measure of the joint impact of two variables upon each other that reflects the simultaneous variation of quantities.<br />Percentile – A value on a scale of 100 indicating the percent of a distribution that is equal to or below it.<br />Monte Carlo sampling – A modeling technique that employs random sampling to simulate a population being studied.<br />Basic Statistics<br />14/69<br />
    15. 15. Statistics Versus Probability<br />In building a risk tolerant schedule, we’re interested in the probability of a successful outcome<br />“What is the probability of making a desired completion date?”<br />But the underlying statistics of the tasks influence this probability<br />The statistics of the tasks, their arrangement in a network of tasks and correlation define how this probability based estimated developed.<br />Basic Statistics<br />15/69<br />
    16. 16. Each path and each task along that path has a probability distribution<br /><ul><li>Any path could be critical depending on the convolution of the underlying task completion time probability distribution functions
    17. 17. The independence or dependency of each task with others in the network, greatly influences the outcome of the total project duration
    18. 18. Understanding this dependence is critical to assessing the credibility of the plan as well as the total completion time of that plan</li></ul>Basic Statistics<br />16/69<br />
    19. 19. Probability Distribution Functions are the Life Blood of good planning<br /><ul><li>Probability of occurrence as a function of the number of samples
    20. 20. “The number of times a task duration appears in a Monte Carlo simulation”</li></ul>Basic Statistics<br />17/69<br />
    21. 21. Statistics of a Triangle Distribution<br />50% of all possible values are under this area of the curve. This is the definition of the median<br />Triangle distributions are useful when there is limited information about the characteristics of the random variables are all that is available.<br />This is common in project cost and schedule estimates.<br />Minimum <br />1000 hrs<br />Maximum<br />6830 hrs<br />Mode = 2000 hrs<br />Mean = 3879 hrs<br />Median = 3415 hrs<br />Basic Statistics<br />18/69<br />
    22. 22. Basics of Monte Carlo Simulation<br />Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise. — John W. Tukey, 1962<br />Basics of Monte Carlo<br />19/69<br />
    23. 23. Monte Carlo Simulation<br /><ul><li>Yes Monte Carlo is named after the country full of casinos located on the French Rivera
    24. 24. Advantages of Monte Carlo over PERT is that Monte Carlo…
    25. 25. Examines all paths, not just the critical path
    26. 26. Provides an accurate (true) estimate of completion
    27. 27. Overall duration distribution
    28. 28. Confidence interval (accuracy range)
    29. 29. Sensitivity analysis of interacting tasks
    30. 30. Varied activity distribution types – not restricted to Beta
    31. 31. Schedule logic can include branching – both probabilistic and conditional
    32. 32. When resource loaded schedules are used – provides integrated cost and schedule probabilistic model</li></ul>Basics of Monte Carlo<br />20/69<br />
    33. 33. First let’s be convinced that PERT has limited usefulness<br />The original paper (Malcolm 1959) states<br />The method is “the best that could be done in a real situation within tight time constraints.”<br />The time constraint was One Month<br />The PERT time made the assumption that the standard deviation was about 1/6 of the range (b–a), resulting in the PERT formula.<br />It has been shown that the PERT mean and standard deviation formulas are poor approximations for most Beta distributions (Keefer 1983 and Keefer 1993).<br />Errors up to 40% are possible for the PERT mean<br />Errors up to 550% are possible for the PERT standard deviation<br />Basics of Monte Carlo<br />21/69<br />
    34. 34. Critical Path and Mostly Likelies <br />Critical Path’s are Deterministic<br />At least one path exists through the network<br />The critical path is identified by adding the “single point” estimates<br />The critical predicts the completion date only if everything goes according to plan (we all know this of course)<br /><ul><li>Schedule execution is Probabilistic
    35. 35. There is a likelihood that some durations will comprise a path that is off the critical path
    36. 36. The single number for the estimate – the “single point estimate” is in fact a most likely estimate
    37. 37. The completion date is not the most likely date, but is a confidence interval in the probability distribution function resulting from the convolution of all the distributions along all the paths to the completion of the project</li></ul>Basics of Monte Carlo<br />22/69<br />
    38. 38. Deterministic PERT Uses Three Point Estimates In A Static Manner<br /><ul><li>Durations are defined as three point estimates
    39. 39. These estimates are very subjective if captured individually by asking…
    40. 40. “What is the Minimum, Maximum, and Most Likely”
    41. 41. Critical path is defined from these estimates is the algebraic addition of three point estimates
    42. 42. Project duration is based on the algebraic addition of the times along the critical path
    43. 43. This approach has some serious problems from the outset
    44. 44. Durations must be independent
    45. 45. Most likely is not the same as the average</li></ul>Basics of Monte Carlo<br />23/69<br />
    46. 46. Foundation of Monte Carlo Theory<br />George Louis Leclerc, Comte de Buffon, asked what was the probability that the needle would fall across one of the lines, marked in green. <br />That outcome occurs only if:<br />Basics of Monte Carlo<br />24/69<br />
    47. 47. Mechanics of Risk+ integrated with Microsoft Project<br />Any credible schedule is a credible model of its dynamic behavior. This starts with a Monte Carlo model of the schedule’s network of tasks<br />Mechanics of Risk+<br />25/69<br />
    48. 48. The Simplest Risk+ elements <br />Task to “watch”<br />(Number3)<br />Most Likely<br />(Duration3)<br />Distribution<br />(Number1)<br />Pessimistic<br />(Duration2)<br />Optimistic<br />(Duration1)<br />Mechanics of Risk+<br />26/69<br />
    49. 49. The output of Risk+<br />Task to “watch”<br />80% confidence that task will complete by 3/7/06<br /><ul><li>The height of each box indicates how often the project complete in a given interval during the run
    50. 50. The S–Curve shows the cumulative probability of completing on or before a given date.
    51. 51. The standard deviation of the completion date and the 95% confidence interval of the expected completion date are in the same units as the “most likely remaining duration” field in the schedule</li></ul>Mechanics of Risk+<br />27/69<br />
    52. 52. A Well Formed Risk+ Schedule<br />For Risk+ to provide useful information, the underlying schedule must be well formed on some simple way.<br />Mechanics of Risk+<br />28/69<br />
    53. 53. A Well formed Risk+ Schedule<br /><ul><li>A good critical path network
    54. 54. No constraint dates
    55. 55. Lowest level tasks have predecessors and successors
    56. 56. 80% of relationships are finish to start
    57. 57. Identify risk tasks
    58. 58. These are “reporting tasks”
    59. 59. Identify the preview task to watch during simulation runs
    60. 60. Defining the probability distribution profile for each task
    61. 61. Bulk assignment is an easy way to start
    62. 62. A – F ranking is another approach
    63. 63. Individual risk profile assignments is best but tedious</li></ul>Mechanics of Risk+<br />29/69<br />
    64. 64. Analyzing the Risk+ Simulation<br /><ul><li>Risk+ generates one or more of the following outputs:
    65. 65. Earliest, expected, and latest completion date for each reporting task
    66. 66. Graphical and tabular displays of the completion date distribution for each reporting task
    67. 67. The standard deviation and confidence interval for the completion date distribution for each reporting task
    68. 68. The criticality index (percentage of time on the critical path) for each task
    69. 69. The duration mean and standard deviation for each task
    70. 70. Minimum, expected, and maximum cost for the total project
    71. 71. Graphical and tabular displays of cost distribution for the total project
    72. 72. The standard deviation and confidence interval for cost at the total project level </li></ul>Mechanics of Risk+<br />30/69<br />
    73. 73. Programmatic Risk Ranking<br />The variance in task duration must be defined in some systematic way. Capturing three point values is the least desirable.<br />Programmatic Risk Ranking<br />31/69<br />
    74. 74. Thinking about risk ranking<br />These classifications can be used to avoid asking the “3 point” question for each task<br />This information will be maintained in the IMS<br />When updates are made the percentage change can be applied across all tasks<br />Programmatic Risk Ranking<br />32/69<br />
    75. 75. Steps in characterizing uncertainty<br />Use an “envelope” method to characterize the minimum, maximum and “most likely”<br />Fit this data to a statistical distribution<br />Use conservative assumptions<br />Apply greater uncertainty to less mature technologies<br />Confirm analysis matches intuition <br />Remember Sir Francis Bacon’s quote about beginning with uncertainty and ending with certainty.<br />If we start with a what we think is a valid number we will tend to continue with that valid number.<br />When in fact we should speak only in terms of confidence intervals and probabilities of success.<br />Programmatic Risk Ranking<br />33/69<br />
    76. 76. Sobering observations about 3 point estimates when asking engineers<br />In 1979, Tversky and Kahneman proposed an alternative to Utilitytheory. Prospecttheory asserts that people make predictably irrational decisions. <br />The way that a choice of decisions is presented can sway a person to choose the less rational decision from a set of options.<br />Once a problem is clearly and reasonably presented, rarely does a person think outside the bounds of the frame.<br />Source:<br />“The Causes of Risk Taking By Project Managers,” Proceedings of the Project Management Institute Annual Seminars & Symposium November 1–10, 2001 • Nashville, Tenn<br />Tversky, Amos, and Daniel Kahneman. 1981. The Framing of Decisions and the Psychology of Choice. Science 211 (January 30): 453–458<br />Programmatic Risk Ranking<br />34/69<br />
    77. 77. Building a Credible Schedule<br />A credible schedule contains a well formed network, explicit risk mitigations, proper margin for these risks, and a clear and concise critical path(s). All of this is prologue to analyzing the schedule.<br />Building a Credible Schedule<br />35/69<br />
    78. 78. Is This Our <br />Contingency<br />Plan ?<br />Good schedules have a contingency plans<br /><ul><li>The schedule contingency needed to make the plan credible can be derived from the Risk+ analysis
    79. 79. The schedule contingency is the amount of time added (or subtracted) from the baseline schedule necessary to achieve the desired probability of an under run or over run.
    80. 80. The schedule contingency can be determined through
    81. 81. Monte Carlo simulations (Risk+)
    82. 82. Best judgment from previous experience
    83. 83. Percentage factors based on historical experience
    84. 84. Correlation analysis for dependency impacts</li></ul>Building a Credible Schedule<br />36/69<br />
    85. 85. Schedule quality and accuracy<br /><ul><li>Accuracy range
    86. 86. Similar for each estimate class
    87. 87. Consistent with estimate
    88. 88. Level of project definition
    89. 89. Purpose
    90. 90. Preparation effort
    91. 91. Monte Carlo simulation
    92. 92. Analysis of results shows quality attained versus the quality sought (expected accuracy ranges)
    93. 93. Achieving specified accuracy requirements
    94. 94. Select value at end points of confidence interval
    95. 95. Calculate percentages from base schedule completion date, including the contingency</li></ul>Building a Credible Schedule<br />37/69<br />
    96. 96. Technical Performance Measures<br /><ul><li>Technical Performance Measures are one method of showing risk by done
    97. 97. Specific actions taken in the IMS to move the compliance forward toward the goal
    98. 98. Activities that assessing the increasing compliance to the technical performance measure can be show in the IMS
    99. 99. These can be Accomplishment Criteria</li></ul>Building a Credible Schedule<br />38/69<br />
    100. 100. The Monte Carlo Process starts with the 3 point estimates <br />Estimates of the task duration are still needed, just like they are in PERT<br />Three point estimates could be used<br />But risk ranking and algorithmic generation of the “spreads” is a better approach<br />Duration estimates must be parametric rather than numeric values<br />A geometric scale of parametric risk is one approach<br />Branching probabilities need to be defined<br />Conditional paths through the schedule can be evaluated using Monte Carlo tools<br />This also demonstrate explicit risk mitigation planning to answer the question “what if this happens?”<br />These three point estimates are not the PERT ones.<br />They are derived from the ordinal risk ranking process.<br />This allows them to be “calibrated” for the domain, correlated with the technical risk model.<br />Building a Credible Schedule<br />39/69<br />
    101. 101. Expert Judgment is required to build a Risk Management approach<br />Expert judgment is typically the basis of cost and schedule estimates<br />Expert judgment is usually the weakest area of process and quantification<br />Translating from English (SOW) to mathematics (probabilistic risk model) is usually inconsistent at best and erroneous at worst<br />One approach<br />Plan for the “best case” and preclude a self–fulfilling prophesy<br />Budget for the “most likely” and recognize risks and uncertainties<br />Protect for the “worst case” and acknowledge the conceivable in the risk mitigation plan<br />The credibility of the “best case” estimates if crucial to the success of this approach<br />Building the variance values for the ordinal risk rank is a technical process, requiring engineering judgment.<br />Building a Credible Schedule<br />40/69<br />
    102. 102. Guiding the Risk Factor Process requires careful weighting of each level of risk<br />For tasks marked “Low” a reasonable approach is to score the maximum 10% greater than the minimum.<br />The “Most Likely” is then scored as a geometric progression for the remaining categories with a common ratio of 1.5<br />Tasks marked “Very High” are bound at 200% of minimum.<br />No viable project manager would like a task grow to three times the planned duration without intervention<br />The geometric progress is somewhat arbitrary but it should be used instead of a linear progression<br />Building a Credible Schedule<br />41/69<br />
    103. 103. Assume now we have a well formed schedule – now what?<br />With all the “bone head” elements removed, we can say we have a well formed schedule<br />But the real role of Planning is to forecast the future, provide alternative Plan’s for this forecast and actively engage all the participants in the projects in the Planning Process<br />For the role of PP&C is to move “reporting past performance” to “forecasting future performance” it must break the mold of using static models of cost and schedule<br />Building a Credible Schedule<br />42/69<br />
    104. 104. We’re really after the management of schedule margin as part of planning<br /><ul><li>Plan the risk alternatives that “might” be needed
    105. 105. Each mitigation has a Plan B branch
    106. 106. Keep alternatives as simple as possible (maybe one task)
    107. 107. Assess probability of the alternative occurring
    108. 108. Assign duration and resource estimates to both branches
    109. 109. Turn off for alternative for a “success” path assessment
    110. 110. Turn off primary for a “failure” path assessment</li></ul>Building a Credible Schedule<br />43/69<br />
    111. 111. Successful margin management requires the reuse of unused durations<br /><ul><li>Programmatic Margin is added between Development, Production and Integration & Test phases
    112. 112. Risk Margin is added to the IMS where risk alternatives are identified
    113. 113. Margin that is not used in the IMS for risk mitigation will be moved to the next sequence of risk alternatives
    114. 114. This enables us to buy back schedule margin for activities further downstream
    115. 115. This enables us to control the ripple effect of schedule shifts on Margin activities</li></ul>Building a Credible Schedule<br />44/69<br />
    116. 116. Simulation Considerations<br />Schedule logic and constraints<br />Simplify logic – model only paths which, by inspection, may have a significant bearing on the final result<br />Correlate similar activities<br />No open ends<br />Use only finish–to–start relationships with no lags<br />Model relationships other than finish–to–start as activities with base durations equal to the lag value<br />Eliminate all date constraints<br />Consider using branching for known alternatives<br />Building a Credible Schedule<br />45/69<br />
    117. 117. The contents of the schedule<br />Constraints<br />Lead/Lag<br />Task relationships<br />Durations<br />Network topology<br />Building a Credible Schedule<br />46/69<br />
    118. 118. Simulation Considerations<br />Selection of Probability Distributions<br />Develop schedule simulation inputs concurrently with the cost estimate<br />Early in process – use same subject matter experts<br />Convert confidence intervals into probability duration distributions<br />Number of distributions vary depending on software<br />Difficult to develop inputs required for distributions<br />Beta and Lognormal better than triangular; avoid exclusive use of Normal distribution<br />Building a Credible Schedule<br />47/69<br />
    119. 119. Sensitivity Analysis describes which tasks drive the completion times <br />Concentrates on inputs most likely to improve quality (accuracy)<br />Identifies most promising opportunities where additional work will help to narrow input ranges<br />Methods<br />Run multiple simulations<br />Use criticality index<br />“Tornado” or Pareto graph<br />Building a Credible Schedule<br />48/69<br />
    120. 120. What we get in the end is a Credible Model of the schedule<br />All models are wrong. Some models are useful.<br />– George Box (1919 – )<br />Concept generator from Ramon Lull’s Ars Magna (C. 1300) <br />Building a Credible Schedule<br />49/69<br />
    121. 121. Conclusion<br />At this point there is too much information. Processing this information will take time, patience, and most of all practice with the tools and the results they produce.<br />Conclusion<br />50/69<br />
    122. 122. Conclusions<br />Project schedule status must be assessed in terms of a critical path through the schedule network<br />Because the actual durations of each task in the network are uncertain (they are random variables following a probability distribution function), the project schedule duration must be modeled statistically<br />Conclusion<br />51/69<br />
    123. 123. Conclusions<br />Quality (accuracy) is measured at the end points of achieved confidence interval (suggest 80% level)<br />Simulation results depend on:<br />Accuracy and care taken with base schedule logic<br />Use of subject matter experts to establish inputs<br />Selection of appropriate distribution types<br />Through analysis of multiple critical paths<br />Understanding which activities and paths have the greatest potential impact<br />Conclusion<br />52/69<br />
    124. 124. Conclusions<br />Cost and schedule estimates are made up of many independent elements. <br />When each element is planned as best case – e.g. a probability of achievement of 10% <br />The probability of achieving best case for a two–element estimate is 1% <br />For three elements, 0.01%<br />For many elements, infinitesimal<br />In effect, it is zero.<br />In the beginning no attempt should be made to distinguish between risk and uncertainty<br />Risk involves uncertainty but it is indeed more<br />For initial purposes it is unimportant<br />The effect is combined into one statistical factor called “risk,” which can be described by a single probability distribution function<br />Conclusion<br />53/69<br />
    125. 125. What are we really after in the end?<br /><ul><li>As the program proceeds so does:
    126. 126. Increasing accuracy
    127. 127. Reduced schedule risk
    128. 128. Increasing visual confirmation that success can be reached</li></ul>Current Estimate Accuracy<br />Conclusion<br />54/69<br />
    129. 129. Points to remember<br />Good project management is good risk management<br />Risk management is how adults manage projects<br />The only thing we manage is project risk<br />Risks impact objectives<br />Risks come from the decisions we make while trying to achieve the objectives<br />Risks require a factual condition and have potential negative consequences that must be mitigated in the schedule<br />Conclusion<br />55/69<br />
    130. 130. Usage is needed before understanding is acquired<br />Here and elsewhere, we shall not obtain the best insights into things until we actually see them growing from the beginning.<br />— Aristotle<br />Conclusion<br />56/69<br />
    131. 131. The End<br />This is actually the beginning, since building a risk tolerant, credible, robust schedule requires constant “execution” of the plan.<br />A planning algorithm from Aristotle’s De Motu Animalium c. 400 BC<br />Conclusion<br />57/69<br />
    132. 132. Resources<br />“The Parameters of the Classical PERT: An Assessment of its Success,” Rafael Herrerias Pleguezuelo,<br />“Advanced Quantitative Schedule Risk Analysis,” David T. Hulett, Hulett & Associates,<br />“Schedule Risk Analysis Simplified,” David T. Hulett, Hulett & Associates,<br />“Project Risk Management: A Combined Analytical Hierarchy Process and Decision Tree Approach,” Prasanta Kumar Dey, Cost Engineering, Vol. 44, No. 3, March 2002.<br />“Adding Probability to Your ‘Swiss Army Knife’,” John C. Goodpasture, Proceedings of the 30th Annual Project Management Institute 1999 Seminars and Symposium, October, 1999.<br />“Modeling Uncertainty in Project Scheduling,” Patrick Leach, Proceedings of the 2005 Crystal Ball User Conference<br />“Near Critical Paths Create Violations in the PERT Assumptions of Normality,” Frank Pokladnik and Robert Hill, University of Houston, Clear Lake,–4203.pdf<br />Resources<br />58/69<br />
    133. 133. Resources<br />“Teaching SuPERT,” Kenneth R. MacLeod and Paul F. Petersen, Proceedings of the Decision Sciences 2003 Annual Meeting, Washington DC,<br />“The Beginning of the Monte Carlo Method,” N. Metropolis, Los Alamos Science, Special Issue, 1987.<br />“Defining a Beta Distribution Function for Construction Simulation,” Javier Fente, Kraig Knutson, Cliff Schexnayder, Proceedings of the 1999 Winter Simulation Conference.<br />“The Basics of Monte Carlo Simulation: A Tutorial,” S. Kandaswamy, Proceedings of the Project Management Institute Annual Seminars & Symposium, November, 2001.<br />“The Mother of All Guesses: A User Friendly Guide to Statistical Estimation,” Francois Melese and David Rose, Armed Forces Comptroller, 1998,–web.pdf<br />“Inverse Statistical Estimation via Order Statistics: A Resolution of the Ill–Posed Inverse problem of PERT Scheduling,” William F. Pickard, Inverse Problems 20, pp. 1565–1581, 2004<br />Resources<br />59/69<br />
    134. 134. Resources<br />“Schedule Risk Analysis: Why It Is Important and How to Do It, “Stephen A. Book, Proceedings of the Ground Systems Architecture Workshop (GSAW 2002), Aerospace Corporation, March 2002,<br />“Evaluation of the Risk Analysis and Cost Management (RACM) Model,” Matthew S. Goldberg, Institute for Defense Analysis, 1998.<br />“PERT Completion Times Revisited,” Fred E. Williams, School of Management, University of Michigan–Flint, July 2005,<br />“Overcoming Project Risk: Lessons from the PERIL Database,” Tom Hendrick , Program Manager, Hewlett Packard, 2003,<br />“The Heart of Risk Management: Teaching Project Teams to Combat Risk,” Bruce Chadbourne, 30th Annual Project Management Institute 1999 Seminara and Symposium, October 1999,<br />Resources<br />60/69<br />
    135. 135. Resources<br />Project Risk Management Resource List, NASA Headquarters Library,<br />“Quantify Risk to Manage Cost and Schedule,” Fred Raymond, Acquisition Quarterly, Spring 1999,<br />“Continuous Risk Management,” Cost Analysis Symposium, April 2005,–_Cockrell_CRM_v1_0.ppt<br />“A Novel Extension of the Triangular Distribution and its Parameter Estimation,” J. Rene van Dorp and Samuel Kotz, The Statistician 51(1), pp. 63 – 79, 2002.<br />“Distribution of Modeling Dependence Cause by Common Risk Factors,” J. Rene van Dorp, European Safety and Reliability 2003 Conference Proceedings, March 2003,<br />Resources<br />61/69<br />
    136. 136. Resources<br />“Improved Three Point Approximation To Distribution Functions For Application In Financial Decision Analysis,” Michele E. Pfund, Jennifer E. McNeill, John W. Fowler and Gerald T. Mackulak, Department of Industrial Engineering, Arizona State University, Tempe, Arizona,<br />“Analysis Of Resource–constrained Stochastic Project Networks Using Discrete–event Simulation,” Sucharith Vanguri, Masters Thesis, Mississippi State University, May 2005,–db/theses/available/etd–04072005–123743/restricted/SucharithVanguriThesis.pdf<br />“Integrated Cost / Schedule Risk Analysis,” David T. Hulett and Bill Campbell, Fifth European Project Management Conference, June 2002.<br />“Risk Interrelation Management – Controlling the Snowball Effect,” Olli Kuismanen, Tuomo Saari and Jussi Vähäkylä, Fifth European Project Management Conference, June 2002.<br />The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century, David Salsburg, W. H. Freeman, 2001<br />Resources<br />62/69<br />
    137. 137. Resources<br />“Triangular Approximations for Continuous Random Variables in Risk Analysis,” David G. Johnson, The Business School, Loughborough University, Liecestershire.<br />“Statistical Dependence through Common Risk Factors: With Applications in Uncertainty Analysis,” J. Rene van Dorp, European Journal of Operations Research, Volume 161(1), pp. 240–255.<br />“Statistical Dependence in the risk analysis for Project Networks Using Monte Carlo Methods,” J. Rene van Dorp and M. R. Dufy, International Journal of Production Economics, 58, pp. 17–29, 1999.<br />“Risk Analysis for Large Engineering Projects: Modeling Cost Uncertainty for Ship Production Activities,” M. R. Dufy and J. Rene van Dorp, Journal of Engineering Valuation and Cost Analysis, Volume 2. pp. 285–301,<br />“Risk Based Decision Support techniques for Programs and Projects,” Barney Roberts and David Frost, Futron Risk Management Center of Excellence,<br />Resources<br />63/69<br />
    138. 138. Resources<br />Probabilistic Risk Assessment Procedures Guide for NASA Managers and Practitioners, Office of Safety and Mission Assurance, April 2002.<br />“Project Planning: Improved Approach Incorporating Uncertainty,” Vahid Khodakarami, Norman Fenton, and Martin Neil, Track 15 EURAM2005: “Reconciling Uncertainty and Responsibility” European Academy of Management.<br />“A Distribution for Modeling Dependence Caused by Common Risk Factors,” J. Rene van Dorp, European Safety and Reliability 2003 Conference Proceedings, March 2003.<br />“Probabilistic PERT,” Arthur Nadas, IBM Journal of Research and Development, 23(3), May 1979, pp. 339–347.<br />“Ranked Nodes: A Simple and effective way to model qualitative in large–scale Bayesian Networks,” Norman Fenton and Martin Neil, Risk Assessment and Decision Analysis Research Group, Department of Computer Science, Queen Mary, University of London, February 21, 2005.<br />Resources<br />64/69<br />
    139. 139. Resources<br />“Quantify Risk to Manage Cost and Schedule,” Fred Raymond, Acquisition Review Quarterly, Spring 1999, pp. 147–154 <br />“The Causes of Risk Taking by Project Managers,” Michael Wakshull, Proceedings of the Project Management Institute Annual Seminars & Symposium, November 2001.<br />“Stochastic Project Duration Analysis Using PERT–Beta Distributions,” Ron Davis.<br />“Triangular Approximation for Continuous Random Variables in Risk Analysis,” David G. Johnson, Decision Sciences Institute Proceedings 1998.<br />“The Cause of Risk Taking by Managers,” Michael N.Wakshull, Proceedings of the Project Management Institute Annual Seminars & Symposium November 1–10, 2001, Nashville Tennessee ,<br />“The Framing of Decisions and the Psychology of Choice,” Tversky, Amos, and Daniel Kahneman. 1981, Science 211 (January 30): 453–458,<br />Resources<br />65/69<br />
    140. 140. Resources<br />“Three Point Approximations for Continuous Random Variables,” Donald Keefer and Samuel Bodily, Management Science, 29(5), pp. 595 – 609.<br />“Better Estimation of PERT Activity Time Parameters,” Donald Keefer and William Verdini, Management Science, 39(9), pp. 1086 – 1091.<br />“The Benefits of Integrated, Quantitative Risk Management,” Barney B. Roberts, Futron Corporation, 12th Annual International Symposium of the International Council on Systems Engineering, July 1–5, 2001,<br />“Sources of Schedule Risk in Complex Systems Development,” Tyson R. Browning, INCOSE Systems Engineering Journal, Volume 2, Issue 3, pp. 129 – 142, 14 September 1999,––SE%20Sch%20Risk%20Drivers.pdf<br />“Sources of Performance Risk in Complex System Development,” Tyson R. Browning, 9th Annual International Symposium of INCOSE, June 1999,––INCOSE%20Perf%20Risk%20Drivers.pdf<br />Resources<br />66/69<br />
    141. 141. Resources<br />“Experiences in Improving Risk Management Processes Using the Concepts of the Riskit Method,” Jyrki Konito, Gerhard Getto, and Dieter Landes, ACM SIGSOFT Software Engineering Notes , Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering SIGSOFT '98/FSE-6,  Volume 23 Issue 6, November 1998.<br />“Anchoring and Adjustment in Software Estimation,” Jorge Aranda and Steve Easterbrook, Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering ESEC/FSE-13<br />“The Monte Carlo Method,” W. F. Bauer, Journal of the Society of Industrial Mathematics, Volume 6, Number 4, December 1958,<br />“A Retrospective and Prospective Survey of the Monte Carlo Method,” John H. Molton, SIAM Journal, Volume 12, Number 1, January 1970,<br />Resources<br />67/69<br />
    142. 142. Resources<br />68/69<br />
    143. 143. Lewis & Fowler<br />8310 South Valley Highway<br />Suite 300<br />Englewood, Colorado 80112<br /><br />303.524.1610<br />Deliverables Based Planningsm<br />Integrated Master Plan<br />Integrated Master Schedule<br />Earned Value<br />Risk Management<br />Proposal Support Service<br />Glen B. Alleman, VP, Program Planning and Controls<br /><br />303.437 5226<br />69/69<br />