- 1. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 1/186 This briefing is an overview of the probabilistic risk analysis processes that can be applied to our program. Although it may not appear to be a “simple” overview, this material is the tip of the iceberg of this complex topic. Just schedule analysis has been addressed in detail here. The cost aspects of forecasting and simulation must be addressed as well to complete the connections between schedule and cost. Probabilistic cost will be surveyed here, but an in depth review is for a later time. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 2. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 2/186 An important aspect of education and research in our business domain, is “Fair Use” copyright law. All the material in this briefing is accessible through the internet. Conference proceedings journal articles, company white papers and other public sources form the basis of much of this material and are referenced in the bibliography. Some materials in this briefing make references to other copyrighted materials in the course of research, investigation, and analysis. These references are solely intended for non– commercial use and as such have no intent to infringe on the copyright holder. All attempts have been made to acknowledge the original copyright holder in pursuit of fair use laws as currently defined in the United States. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 3. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 3/186 The concept that risk and the management of risk is a desirable part of our program is not always appreciated or well understood Without risk there can be no opportunities. The plans for the program become static and deterministic. While risk and opportunity are related, the management of risk is not the complement of opportunity. - even if this is a popular notions these days. See the Conrow, AT&L article for detailed discussion of this somewhat controversial topic. The primary opportunity in Programmatic Risk Management is the avoidance of being late and over budget on the planned launch date. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 4. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 4/186 When we use the term “risk tolerant IMS” it means a plan and supporting that can tolerate risks. Technical risks and programmatic risks. These risks are built into the program by its very nature. These risks must be addressed both technically and programmatically. The real challenge though is not how to address them, but how to recognize that they are being addressed in a manner that actually reduces the level of risk as the program proceeds along its path to final maturity. A measure of “increasing maturity” is the reduction of risk made visible to the evaluator of the IMS. The materials here guide us through the process of building a risk tolerant IMS. But putting it to work still requires practice. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 5. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 5/186 The credibility of the Integrated Master Schedule (IMS) is the critical success factor for both our proposal and our execution phase after the win. Without a credible schedule and the related cost credibility, there is a low probability of a win. The effort put into constructing a credible schedule during the proposal phase will pay off (assuming the program structure remains intact) during the execution phase. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 6. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 6/186 The skills of creating and managing a schedule and the associated cost require special understanding. However, the planners are usually the last in a long line of “culprits” for finding the cause of any failure. This is a “no win” situation. People skills, project management skills, and some level of technical skill is needed. But most important is the people skill, since the knowledge of how to assemble a successful IMS resides in the minds of others. Getting this knowledge out and on paper requires interpersonal communication as a primary process, not technical tools and formal processes. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 7. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 7/186 Understanding the difference between qualitative and quantitative risk assessment is important. Our first approach is usually qualitative. But what is needed is quantitative. A specific measure of programmatic risk, is the impact of the mitigations or risk retirement activities and measure of the increasing maturity of the program deliverables in the presence of risk. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 8. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 8/186 Programmatic risk management makes visible the technical risk mitigation steps as well as the alternative programmatic processes in the presence of these risks. Alternative branching in the IMS must be defined to a level of detail that instills confidence that the IMS properly represents a “risk tolerant” plan. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 9. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 9/186 Since there is quite a bit of material here, a quick overview will get us started. The executive overview should leave the reader with a sense of the important topics • There are no point estimates allowed in planning. All estimates must be probabilistic • There are core issues with simple (deterministic) PERT and it is not to be trusted • The use of a probabilistic tool is useful, but understanding how the underlying statistic works is critical to its use in planning and program execution Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 10. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 10/186 When asked “why are we doing this?” many would answer – because our customer wants us to. This would be too simple an answer. The main reason is, most programs are simply too complex not to have a better understanding of how the programmatic and technical risks interact. Not understanding the interaction between these two types of risk that creates the biggest risk. Individually these risk “could” be managed. But when combined they behave in unpredictable and maybe unknowable ways. This is a core feature of any system. See Systems Bible Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 11. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 11/186 If we get only two concepts out of this briefing they should be: • There are multiple critical paths in any executing program. Asking “what is the critical” indicates that the questioner does not understand the probabilistic nature of the program • PERT is a poor estimating metric. It has built in biases which under estimate the total duration of the program. Monte Carlo is a better estimating tool, but it too needs careful adjustment before realistic numbers can be derived. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 12. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 12/186 The DID–MGMT–81650 describes the Integrated Master Schedule. Integrating Programmatic and Technical risk identification and mitigation adds credibility to the IMS and therefore to the overall program. Applying probabilistic risk analysis to the IMS is mandated, but care is needed to interpret the results. These tools aid in the evaluation, but they are not replacements for good program management processes. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 13. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 13/186 The idea that uncertainty and the risk that it produces can be “programmed out” of the schedule is a false hope. Without understanding the principles of Deming, the management and the planning staff will be “chasing their tail,” trying to control the naturally occurring variances in the plan. The first approach is to set the error bands wide enough to not trigger an exception report for these variances. This approach is “good enough” but what is missing is the knowledge of “how wide is wide enough?” for a specific set of tasks or during a specific phase of a program? Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 14. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 14/186 The first step in the process of adding credibility to the IMS is to recognize that all task completion times are random variables. They are not “point” numbers (scalars) but are “estimates” of the completion time drawn from a probability distribution of the underlying population of all completion times possible for the specific task. Modeling schedule durations are random variables does not imply these durations are “random.” It reflects how a duration’s uncertainty is influenced by the underlying probabilistic nature of the activity network. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 15. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 15/186 Building a credible IMS starts with identifying the architecture of the IMP and the supporting tasks in the IMS. Although this is restating the obvious the process to do this is actually quite hard. Adding schedule and cost risk identification and mitigation to the process is the minimal result for a winning proposal. It cannot be emphasized enough – the architecture of the IMS is critical to identifying a risk tolerant schedule. The “rats nest” approach is simply unacceptable to the success of any program. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 16. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 16/186 The goal of introducing probabilistic schedule and cost analysis is to improve the probability of a “win” on the proposal. While winning is important, executing the program is even more important. What ever “credibility” elements were in the proposed IMS need to be carried into the execution schedule. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 17. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 17/186 The use of Monte Carlo for assessing the IMS must be turned into forecasting performance. This is done by identifying the “hot spots” in the IMS through sensitivity analysis, interventions for these “hot spots” and the measure of change resulting from the intervention. The important concept is to connect metrics to measurable benefits to the program. Without this the creation of metrics is just wasted effort. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 18. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 18/186
- 19. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 19/186 Using risk and uncertainty as an integral part of the planning process is a sign of maturity. Making decisions on the this risk information improves maturity. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 20. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 20/186 When we speak of risk management, either technical or programmatic, the term usually has a very localized context. For the planning context risk management must include both technical and programmatic risk. The technical risk aspects come from external sources but are directly represented in the IMS. The programmatic impacts of this technical risk must be explicitly addressed. This is the easy part. The hard part is determining the implicit programmatic risk that is derived from the technical risk and the risks that are derived from the “architecture” of the program itself. This is where the true “risk tolerant” IMS adds value. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 21. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 21/186 There are many approaches to building a risk tolerant IMS. Our current approach is to add risk factors and margin to specific areas of the IMS The current approach to use a Monte Carlo tool to assess where this margin should be placed. There are several other steps along the way. Which steps to take, how much effort to invest and how to recognize the value of this investment are some of the management challenges as well as the technical challenges. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 22. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 22/186 The difference between risk and uncertainty needs to be understood at some level. For the most part the differences are not important in the beginning. But once decisions start to be made about mitigation steps, branching probabilities for failure modes, these differences become more important. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 23. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 23/186 When we use the term uncertainty or risk it means at least 4 things. First let’s sort out “uncertainty” There are two classes of uncertainty in large complex programs. • Static uncertainty emerges from the natural variations in the completion times of tasks. This is a Deming uncertainty. http://webserver.lemoyne.edu/ ~wright/deming.htm is an example of this type of uncertainty • The dynamic uncertainty is about the unknowns and the unknowable Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 24. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 24/186 The static uncertainty in a program can be addressed directly in the plan with mitigation tasks. The dynamic uncertainty arises from the dynamic interactions between the tasks of the plan. This interaction and the outcomes to the end date cannot be modeled with static paradigms. Monte Carlo simulation is an approach to modeling these interactions and their impact on other elements of the plan Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 25. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 25/186 Managing risk in the schedule requires anticipation to identify the risks, but also requires understanding of the source of risk, the impacts of these risks, and the interaction between the risks and the plan. A process is needed to guide the risk management activities. This process must address both the programmatic as well as technical risk. The interaction between programmatic and technical risks must also be managed. These interactions must be considered a “first order” interaction. The common approach is to consider the technical risk as first order and the programmatic risks secondary. The combination becomes a first order interaction. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 26. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 26/186 As planners our goal must be to produce a plan that has credibility and integrity. Credible plans are believable plans Integrity plans are trustworthy plans. Both attributes are needed for a winning proposal and the follow on execution. The successful assessment of the IMS during a proposal or during execution by the customer or DCMA depends on how believable the plan is and how well it can be assessed to confirm this believability Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 27. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 27/186 The assessment of the credibility and integrity of the IMS can take place by asking some questions. These and similar questions shine light on the underlying attributes of the IMS in ways that simple assessments do not. These are not technical assessment, like counting data in the predecessors field, but are architectural questions about the “quality” of the IMS independent of the technical details. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 28. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 28/186 NASA does risk management in a specific way. We need to understand their way as a starting point. Reading the NASA materials is a start, but there is other research available from conferences and vendor web sites that needs to be gathered and read as well. Other government agencies as well as civilian firms have similar risk management approaches. NASA’s approach is a good starting because of manned space flight’s inherent risk. And NASA’s emphasis on Safety and Mission Assurance. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 29. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 29/186 The IRMA tool developed at NASA Johnson Space Center is the basis of risk management for a NASA side. Although this approach is focused on the technical risks the programmatic risks appear in the database. As well there are other risk management systems and paradigms. Active Risk Manager (ARM) is a popular one as well, http://www.strategicthought.com/ Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 30. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 30/186 The NASA Risk Management Summary Card calls out “schedule” impacts in three places. Connecting programmatic and technical risk is a critical success factor for a proposal as well as an execution assessment. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 31. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 31/186 Adding probabilistic schedule and risk analysis to the IMS can be done through a structured process. 1. The initiating event of the risk is identified. 2. The result from this event is described 3. The consequence that flow from the scenario are developed 4. The connections, flows, interactions and correlations between the scenarios are modeled 5. The probability of occurrence for each of these scenarios is developed 6. The model of the probability of occurrence and consequences from the occurrence are combined Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 32. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 32/186 The Continuous Risk Management paradigm found in the technical risk world can be applied to the programmatic risk as well. NASA has adopted Continuous Risk Management (CRM) through several guidelines listed here. The table summarizes how CRM is managed in a structured manner throughout the program life, Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 33. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 33/186 There is a difference between the design evaluation of the IMS and the risk evaluation. The design evaluation describes how the technical activities needed to develop and deploy the product – in this case a manned spacecraft – must come together in the right sequence to make the planned completion date. The risk evaluation defines the probabilistic completion model for each task, the correlations between the tasks and the resulting probabilistic model. This model is a Bayesian Network of all the tasks. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 34. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 34/186 To construct an IMS with integrity and credibility both technical and programmatic risk must be connected. This process starts with the identification of the technical risks in ARM and their mitigations in the IMS. This is the explicit risk approach. Next comes the explicit programmatic risk activities. This can be the well known margin needed in front of major milestones, program events or deliverables. Finally comes the implicit risk mitigation activities that will be needed to differentiate this IMS from any other IMS to start to build confidence that we have a “risk tolerant” IMS. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 35. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 35/186 A pedagogical literature survey from the RAND Corporation supports the notion that probabilistic risk assessment is not seen in a favorable light by management. • It is too complex. • The underlying statistic are not will understood. • “It’s the customers that are asking for this.” • There is little historical data to calibrate the underlying probability distribution functions for task completion times. All of these gaps must be closed in some way in order to call our IMS Risk Tolerant Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 36. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 36/186
- 37. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 37/186 Managing in the presence of uncertainty is the core behavior for any modern program. Trying to control this uncertainty requires two basic understandings: 1. The natural variations in the schedule cannot be sufficiently controlled to remove risk. These are the Deming variations and the foreseen uncertainties 2. The unforeseen uncertainties and the inherent chaos of the program must be dealt with through contingencies Attempting to manage uncertainty is limited to foreseen risk. Managing in the presence of uncertainty deals with unforeseen and chaotic sources of risk Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 38. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 38/186 When estimating the completion times for tasks, there are three primary problems. 1. A number produced by a CAM or an IPT must be a statistical estimate, not a specific duration. 2. The meaning of “best” must be established prior to accepting the statistical estimate 3. The collecting of the “most likely” estimates cannot be added in the sense of adding scalar numbers, since they are probability distributions. 4. The “most likely” is NOT the average completion time, it is the completion time that occurs most often from a large sample of possible completion times. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 39. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 39/186 The first approach to “planning” the program is to ask the CAMs or IPT Leads for each task in their WBS or IMP/IMS area: “how long with this take to do?” The numbers that come back are then entered in the duration field on the schedule. These numbers are not only wrong they are dangerously wrong. They are “point” estimates that live inside a probability distribution. The built in bias from the approach has clinically be shown to be optimistic or pessimistic, but rarely “most likely.” Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 40. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 40/186 The traditional approach is to roll up the single point estimates into a sum of the durations and search for the longest path. This is the Critical Path Method (CPM) for assessing the finish date of the plan. The problem of course is these “numbers” are not actual scalar values. They are samples drawn from probability distributions. Addition is not mathematically possible in the sense of addition, defined over the set of natural numbers (0, 1, 2, … ∞] These probability distributions can be “convolved” into a new probability distribution, but a better approach is Monte Carlo Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 41. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 41/186 When asked “what is the most likely” or the “best guess” duration, the variety of answers removes any chance of getting a reasonable answer. The meaning of “best” is undefined in almost any situation that has not taken explicit steps to bound the answers. Without calibrating the meaning of “best” the planner cannot bound the underlying probability distribution of all the value that are not “best” but could possibly occur in the project Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 42. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 42/186 When we use a term “best” or “most likely” there is an implicit assumption – often not acknowledged – that other values than “best” and “most likely” can occur. This is the probabilistic nature of the duration estimate. A single value cannot exist. The actual shape of the probability distributions is what is needed for generating the “best” estimate. Without this knowledge, the planner is guessing in the dark. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 43. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 43/186 Here are some steps to producing “educated guesses.” This is a model based approach which depends on the maturity of the data that is the basis of the model. While this is a high level description, it needs raw data underneath to make it valid. Without this data the “guess” is of little value. What is missing in most cases is any historical trends for the IMS elements. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 44. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 44/186 Playing the 20 questions game is on approach to calibrating the “guess” for the duration. This approach will get an answer to without 10% to 20% in a few questions. This is a way to start the “conversation” about duration when the participants have convinced themselves that they can’t come up with the answer because there is not enough information. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 45. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 45/186 Another approach is to classify the fidelity of the information. This can be done with a 1, 2, 5 approach. Gathering estimates by asking for durations is the preferred approach. Instead, making a risk adjusted estimate – duration and confidence interval provides a better approach. This approach neutralizes the guessing game by asking a risk question first, then the duration. The classification of risk provides the lower and upper bounds of the task duration. Along with the underlying probability distribution, this forms the basis of probabilistic schedule analysis Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 46. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 46/186 In all cases, uncertainty is the normal mode for information gathering. When we ask a CAM or IPT for an estimate and do not ask for the risk associated with that estimate and the confidence intervals for that number we are simply increasing the risk to the program by absorbing unreliable numbers. This unacknowledged risk is always present . By not making it visible, the program is mortgaging the future without budgeting for the cost of paying off the mortgage. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 47. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 47/186 Starting with a good topology for the IMS is important. Not only because the programmatic activities need to be well defined, but the sensitivity of the risk analysis depends on a “properly formed” IMS. If the logic of the IMS is ill– formed than the results of the risk analysis will also be ill–formed. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 48. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 48/186 There are several elements of the probability model for duration. Not only are the activities from the IPTs and CAMs important, but the subcontractors play on important role. The data from the subcontractors includes: • Durations and the probabilities • The internal connectivity of the activities that produce the external; “milestones” conveyed to the prime contractor. • The other programmatic risk factors for the performance of subcontractor work Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 49. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 49/186 Although the formalities of the probabilistic risk analysis are not needed for this briefing. Here is some background on terminology. If we are to learn to “speak” in probabilistic programmatic risk, these terms should become familiar. This is an almost endless topic, but some understanding of probability and statistics is needed. This of course requires some effort and patience . Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 50. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 50/186 We should not be drawn into the illusion that the Central Limit Theorem is operable for the program. This is the core assumption of PERT and CPM based planning. This requires normally distributed completion time and independence between tasks. Neither can be verified in practice. As such the impact of making these assumptions is “whistling in the dark.” The result is that the program is late before it starts. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 51. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 51/186 The Central Limit Theorem can be useful in many cases. But it needs to be understood where it is not useful. The assumptions of the CLT applied to the PERT problem mask even more problems when naively applied to estimating the duration of a program. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 52. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 52/186 The core of the Central Limit Theorem of the production of a Gaussian probability distribution by assembling a collection of arbitrary probability distributions. The primary assumptions that these distributions are independent provides the basis of the CLT. If the activities represented by the arbitrary distributions are not statistically independent – which is hardly ever the case on a real project – then the assumptions of the Central Limit Theorem are false and the probability distribution of the program completion time is no long Gaussian distributed Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 53. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 53/186 What happens is the rollup of the most likely times of the critical path activities is biased to an optimistic location in the probability distribution of the project completion distribution. This is the fundamental reason PERT is not very useful. This criticism is only partly true. If a probabilistic PERT approach is used or a Bayesian network approach is used, then the deterministic issues are removed. But it is easier to use a Monte Carlo simulator since this avoids gathering all the underlying probabilistic distribution information for an initial estimate of the completion time of the program Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 54. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 54/186 The probability distribution function describes the frequency of occurrence of the events in the underlying statistical process – say the duration of a task completion, the roll of a die, or the time it takes a light bulb to burn out. The ordinate of the graph (the y axis) is normalize to a scale of [0, 1] which represents the probability percentage 0.10 = 10% The abscissa represents the range of values that can be found in the underlying sample population. In this case [0.0, …, 5.0] The mode is the “most likely” value to occur when samples are drawn from the distribution. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 55. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 55/186 The standard deviation is a description of the “spread” of the probability distribution function around the mean. Without understanding the standard deviation ,a point estimate or even a sampled estimate is of little value. The shape of the probability distribution is also important in understanding the confidence in a single number. These “higher order moments” will be discussed later, but for now no estimated number should be used without the standard deviation value being attached. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 56. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 56/186 Looking at the population statistics of a random process is not very useful. Humans have a hard time making any sense from the graphs. The Histogram view can show the frequency of occurrence of the various values – how often a specific value occurs in the total population of value or the sampled population of values, but more insight is needed. The Cumulative Probability Density is a way to show this. The CDF shows the probability that a sampled number drawn from the population of all possible numbers Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 57. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 57/186 Various Probability Distribution Functions (pdf) have similar Cumulative Density Functions (CDF). This is important for several reasons: • The underlying probability distribution function has great influence on how the end point values are weighted. This has impact on the PERT formula • The cumulative distribution is the source of random numbers in Monte Carlo. For a variety of pdf’s, similar CDF’s are generated, neutralizing the differences in the pdf’s. Monte Carlo isolates these underlying differences. This may be good or bad depending in the need. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 58. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 58/186
- 59. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 59/186 Any estimating process must address the probabilistic boundaries of the estimate. Without this, planners and cost estimators are hopelessly under or over estimating duration and associated cost. The real issue is not over or under estimate, but not knowing which one it is or why. This lack of knowledge about the underlying statistical process creates a greater risk. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 60. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 60/186 Making decisions in a risk neutral manner is not advised. We should always talk about risk adjusted decisions, risk adjusted values, and risk adjusted outcomes. The difference between alternatives, uncertainties and outcomes also needs to be understood. They are not interchangeable concepts Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 61. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 61/186 Decision making must address the different types of uncertainty. Understanding how these uncertainties impact the decision is critical to selecting alternatives Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 62. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 62/186 The idea that we can produce “estimates” about the future in the absence of models, historical data, or a methodology for discovering these models or historical data is common in the IMS planning realm. Forecasting the future is sporty business. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 63. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 63/186 The IMS contains “branches” where the path of work makes a change in direction. These braches can be modeled with a decision tree paradigm. The risk management discipline uses this approach. And it is applicable to the construction of the probabilistic branching found in the network of tasks in the IMS. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 64. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 64/186 In the “olde days” the line of balance chart was used to forecast the cost at completion. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 65. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 65/186 PERT and the Critical Path Method are called out as explicit methods to be used in the planning process. The formulas for PERT are simplified models of the underlying complexity of probabilistic networks (Bayesian Networks) As such they have little or no connection to the reality of the IMS Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 66. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 66/186 In deterministic PERT the durations are defined as a three point estimate and the PERT formula is used to compute the mean and standard deviation for the program duration as well as the critical path. This is the algorithm used in Microsoft Project when the PERT tool bar is turned on and the three point estimates entered into the appropriate columns. It is billed as probabilistic but in fact the 3–point estimates work against a fixed probability distribution function with no way to adjust its shape, bounds or moments. As well, there is no way to insert the correlations that naturally occur in the IMS. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 67. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 67/186 The estimates produced by the deterministic three point data can be used to construct a probabilistic PERT if the underlying probability distributions are defined for each task completion time. The development of the probability distributions requires historical data as well as an understanding the behavior of each node in the network (coupling). This is a difficult task without the proper tools and data sets. With the Risk+ tool, individual distribution functions can be assigned to each task. But the “tuning” of each function is difficult. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 68. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 68/186 When we speak of probabilistic risk analysis, we also need to speak of the statistical nature of the activity network. When we speak of a probabilistic activity network (a Bayesian network) we also need to speak in terms of probability. A question that can be asked of the network is – “ what is the probability of completing this task by a certain date?” A second question that can be asked is – “what are the underlying statistics of the activities of the network?” A final question that needs to be asked is “what is the inherent uncertainty in these estimates?” In other words – how good is our ability to guess in the presence of a statistical process? Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 69. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 69/186 Once the activity durations are treated as probability distributions it can be seen that they can not be “added” in the normal sense to produce a program duration. They must be “summed” in the probabilistic sense. This can be done with Monte Carlo or with convolution of the Cumulative Distribution Functions. Again, add to this the correlation issues (one task influencing the outcome of another task), and the simple approach of adding the durations to come up with a total duration falls apart. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 70. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 70/186 Here’s another look at distribution functions. This approach should be the standard vocabulary for discussing the IMS duration estimates. The topological integrity of the IMS is important, but just as important is our understanding of how the activity durations have been developed, their confidence interval and the underlying distribution of the values the durations can take. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 71. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 71/186 A missing element is the statistics of the “events” that occur during the execution of the program. For example if a fixed date is defined in the IMS (this is very usual for things like IBR, PDR, CDR), what is the underlying probability distribution of the confidence of that date. The same is true for subcontractor provided dates, where the details of the deliverables is not visible. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 72. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 72/186 With the input probability distributions, the program schedule can be treated as a “system” with a response function. The “system” is a Bayesian network where the elements of the network are probabilistic and the driving function is probabilistic. The “output” of the system is therefore probabilistic as well. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 73. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 73/186 One probability distribution commonly found in scheduling is the Beta Functions. This is a “tunable” probability distribution function that has been shown to closely match the behavior of task completion durations. The term “closely” needs to be used with care. The deviations between actual completion times and the “model” of completion times needs to be assessed before confidence in the probabilistic results can be useful. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 74. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 74/186 The Beta distribution is used for PERT estimates. This use is many times done with no understanding of the shape or the dynamics of the probability distribution function. Beta is a selection for Risk+ as well, with no obvious way to change the shape of the curve. Some understanding of the impact of the Beta function on the outcome of the PERT formula or the Monte Carlo simulation is needed. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 75. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 75/186 There are many alternatives to Beta. The Triangle distribution is one. The triangle distribution has an intuitive appeal due to its simplicity useful for estimating task durations. But the triangle distribution still has the problem that the most likely value and the expected (mean) value are not the same. So when planning asks for the “most likely” value many people respond with the Mean, which biases to result in the optimistic direction. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 76. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 76/186 The triangle distribution can better describe some statistical processes, but it too needs “tuning” for specific task duration processes. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 77. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 77/186 BetaPERT is currently the vogue in the probabilistic analysis world. The BetaPERT distribution provides a “tunable” curve where the most likely “Mode” is near or identical to the “mean” of the distribution. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 78. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 78/186
- 79. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 79/186 The challenge to building a risk tolerant IMS is the initial capture of the task durations and the sensitivity of the IMS to correlations between tasks. There is a optimism bias created when a CAM is asked “what is the duration?” The answer is usually a “mean” (average) duration rather than the “Mode” (most likely). If the Mean is used in place of the Mode, then the three point estimates are biased to start with without the explicit knowledge of the planning staff. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 80. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 80/186 The general flow of creating a risk tolerant IMS looks like this. The critical aspect is to get the CAMs to identify the embedded risks and the mitigation tasks for those risks. Once this is done, “planning” can then assess if the mitigation processes make sense in terms of supporting the AC’s and SA’s of the IMP/IMS. Constant and continuous feedback is needed for this to work properly. Without this feedback, the IMS is assembled in the absence of the knowledge base and the risk tolerant aspects are lost or become confused with the mainline activities. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 81. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 81/186 The capturing of the risk information is an interactive process. A Kaizen is one way to do this and probably the best. Having the CAM fill out the “most likely” durations and identify the risk mitigations cannot be done without direct contact. Without this direct contact, planners have not chance of intervening in the process and the IMS becomes a collection of tasks rather than an “architected” plan. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 82. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 82/186 The 3 point estimates required by DiD–81650 have a variety of uses. They can be simple values used for PERT calculations. These calculations can be “made up” by the IPT lead and entered into the schedule. A risk adjusted value can be used from the Macro in Risk+. The CAM or IPT Lead states the relative risk in a number between 1 and 5. The macro defines the percentage boundaries for the classified risk. Individual risk ranges can be developed from historical information. This is the best approach, since it represents the past. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 83. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 83/186 There are several classes of programmatic risk. Although the Pareto chart shows that scope change is the most common, delays are also common. These come from the customer side most often as well. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 84. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 84/186 The classification of risk results in a percentile or quartile classification scheme. This is a better approach than asking someone for the minimum and maximum durations. The challenge is to calibrate these ranges in a meaningful manner for the specific program. There can be general classification ranges, but having them set for the specific program is much better. This of course requires that data is kept from past programs, normalized and then made available in a form useful for probabilistic risk analysis. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 85. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 85/186 During the data capturing process where estimates are extracted from the technical experts, there is a natural tendency to accept the numbers at face value. Without qualifying the numbers in some statistical form, this information is absorbed into the IMS or Cost and becomes “fact.” These “facts” then progress through the program and are never challenged for their lack of statistical basis. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 86. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 86/186 The core problem with capturing estimate from human beings is they are biased. Either negatively biased or positively biased. There is plenty of literature on this effect and ways to overcome it. For now we’ll just live with the outcome of the bias Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 87. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 87/186 Let’s take another tour of the problems with PERT. These issues are well documented in the literature, but poorly understood in practice. The poor understanding comes from the difficulty of the explanation – statistical conversations are usually not very interesting; and the natural tendency to look for easy answers to complex problems. The core issue is that without a deep understanding of the errors produced by the PERT equation, the confidence in completion dates and the risk tolerance of the IMS is difficult to build. When the actual numbers come in (ACWP and BCWP) and they don’t match the expectations – is it the original plan or the underlying performance? Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 88. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 88/186 There are several myths about PERT. The first is that is was scientifically thought out in detail. This is not the case. The book The Management of Projects, Peter W. G. Morris provides the background on this development as well as other project management histories. The second historical myth is that PERT is a general purpose approach. In fact it is very specialized and is applicable to a narrow range of activity networks. Those with normally distributed completion times, statistically independent relationships, ones where the critical path does not change and with the “most likely” estimate actually representing the “mode” of the underlying probability distribution function. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 89. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 89/186 When a manger asks “what is the critical path for this program?” there are several thoughts and actions: • In a probabilistic activity network there are many critical paths, which change as a function of time, adjustments to the risk profiles, and the completion of work. • Correlated activities are influenced by off–critical–path activities to place them on the critical path. So the answer to the management question is “it depends on what you mean by critical and path.” The real answer only comes by moving away from the static representations of the IMS to a probabilistic representation – and that requires much more effort. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 90. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 90/186 Once we recognize that the activity network is probabilistic in nature, the first choice (the naïve choice) is to apply the PERT method. While this may be a useful “first” choice it produces results that are overly optimistic and sometimes overly pessimistic. Either way they are wrong from a statistical point of view. They are wrong because the assumptions of PERT are wrong. These assumptions are almost never found to be true in practice. Even if they were true, the probability distribution function used by PERT does not represent any useful activity completion time distribution. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 91. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 91/186 One of the “killer” assumptions in PERT is the lack of understanding of “merge bias.” Merge Bias occurs when two or more activities are joined at a merge point. Usually a milestone or a simple Finish to Start of several tasks. The result is the statistical behavior of the activities prior to this merge point influence the statistics of the following activities in undesirable ways. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 92. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 92/186 Since statistical distributions can not be simply “added” the duration of the downstream activity is not the sum of the duration of all the upstream activities (or the longest activity). Instead it is the statistical sum (convolution) of the probability distribution function (pdf) Without understanding this, the PERT estimate generates an optimistic estimate of the duration, since the PERT formula simply adds the durations to arrive at the total duration. The PERT formula also adds the individual activity variances to arrive at a total project variance. While this provided a simple method to “guess” the total duration it produces a poor model for real analysis of risk. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 93. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 93/186 The PERT approach fails to consider the “random variable” nature of the dates in activity network. As well the correlation between each of these random variables is not considered. The result is the potential for large variances in the completion time estimates – 15% is not uncommon. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 94. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 94/186 The visual impacts of Merge Point bias is show here. This is a small and sample activity network. A “real” network would have different outcomes. It is not important exactly how the merge point bias impacts the final completion date, but that the merge point bias DOES impact the final completion date. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 95. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 95/186 How the activity network is arranged has significant impact on the calculations for PERT. Here are some examples. Notice that the PERT mean (the average) stays the same, while the “real” mean and the variance on that mean change dramatically depending on the arrangement. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 96. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 96/186 The reason for these changes involves how the statistics are “added” in the various configurations. The critical concept is that the PERT calculations are unreliable as a predictor of the completion time in a probabilistic model of the activities. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 97. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 97/186 The effect of the merge bias is shown in the graph. It is unlikely in any real plan that only three parallel paths exists. This number is usually much larger, sometimes in the dozens. All of this discussion is leading to the suggestion that PERT is not viable on any complex program. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 98. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 98/186 Use Monte Carlo, don’t use PERT. The problem of course is that DID 81650 and even the corporate guidelines either require or strongly suggest the use of PERT and CPM. This can be done of course, but don’t use the numbers for any real planning processes. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 99. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 99/186 The use of Monte Carlo simulation is a logical outcome of the problems with PERT. What is missing is the understanding of how Monte Carlo works, what it’s limitations are, where it should not be used and of course how to interpret the outcomes when they don’t meet our expectations. Even though Monte Carlo is a powerful tool it can produce unexpected results. This section is an attempt to give some background on the mathematics and stimulate further interest in applying this tool to the problem of schedule forecasting Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 100. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 100/186 Monte Carlo simulations provide a useful approach to modeling schedule risk. But their value is more than that. Unlike PERT or other deterministic approaches – even though the three point estimates are billed as probabilistic – Monte Carlo examines the schedule network independent of a critical path, topological constraints or other “human induced” problems. It looks at the network as a collection of nodes and arcs, independent of the “meaning” of this information and produces a model of the behavior of these nodes and arcs Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 101. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 101/186 The concept behind Monte Carlo is to sample the possible durations for a task from the population of all durations and apply them to the schedule. The population of possible samples is defined by the Cumulative Probability Density (CDF) function for each task. This in turn is defined by the 3– point estimate for the task, which selects the bounds in the CDF for sampling. Since there is no direct concept of a Critical Path in Monte Carlo, the near critical path tasks are considered in the analysis of the completion time. As well the PERT biases produced by the simple minded PERT formula are avoided as well. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 102. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 102/186 There are several “components” to the Monte Carlo process. So when we speak of Monte Carlo it is both a process and a product – in our current case Risk+ Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 103. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 103/186 Samples drawn from the underlying distribution function can produce an “error estimate” on a completion date. These error estimates are different than the fixed boundaries for PERT, since they represent the actual probabilities distribution error bounds Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 104. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 104/186 The number of sample runs needs to be sufficient to cover all the possibilities in the pdf. This is usually 500. A production run for a Monte Carlo simulation is around 2,000 to 3,000 iterations. As the iteration count increases the fidelity of the simulation increases. But there is a point where more samples don’t add value. This point can be determined by the statistical performance of the variance of these sample space. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 105. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 105/186 Since Monte Carlo does not need to know about the Critical Path, it is conceptually simpler to use. A well formed network is needed and the 3–point estimates need to represent the proper risk assessment. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 106. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 106/186 This is a view of how Risk+ sets up the project file. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 107. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 107/186 The result is a cumulative distribution and a probability distribution function. Interpreting this result is straight forward. The confidence of each date is shown in the table on the right. This is the probability of completing the task by the date. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 108. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 108/186 A good IMS is needed. The risk assessments should be done with a ranking process rather than specific 3–point estimates. This disconnects the personal opinions from the assignment of risk. A 5 level process is one approach, but any odd numbered level ranking is best. The differences between the levels should be geometric not linear. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 109. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 109/186 Risk+ generates lots of information useful for the analysis of the program. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 110. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 110/186
- 111. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 111/186 Constructing a robust IMS means building a “risk tolerant” plan. The robustness of the plan means that it (the plan) can deal with disruptions that occur naturally through the course of execution or un–naturally through external events. In either case the “robustness” of the plan must be visible to the evaluator without any detailed explanation, beyond the IMS narrative. No hand waving explanations of how the plan works. The risk tolerant aspects most be obvious. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 112. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 112/186 Thinking about schedule contingency is different in a PRA context. For a simple project, 15% contingency is assumed. But placing the contingency is the first problem. The process is: • Run Risk+ and watch the final date. • Compare the 80% confidence date against the deterministic date. This difference is the first cut at the needed margin. • Assign this duration across the project in front of the critical (high risk) milestones. • Rerun Risk+ and add or subtract this margin until the desired confidence date is achieved. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 113. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 113/186 More detailed statistics and interpretations of the results an be produced with Risk+. This information can them be used to perform further analysis of the IMS. The analysis is what we’re after, not just the date produced by Risk+. Like the PERT numbers, the Risk+ numbers must be interpreted with the understanding of how they were arrived at. This is one of the purposed of this briefing – to provide knowledge of how to use this approach and what its strengths and weaknesses are. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 114. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 114/186 Incorporating Technical Performance Measures (TPM) with Monte Carlo is a powerful way of showing how risk is reduced and maturity increased in a program. At each step in the program – each Program Event – a target confidence interval for a completion date can be forecast. Along with the technical performance measure, this programmatic performance measure approach results in a “risk tolerant” IMS. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 115. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 115/186 Risk tolerance in the IMS requires more than just the planning processes. It requires the connections to technical and cost. This has been stated before, but it needs to be made not only visible but actionable. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 116. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 116/186 Using a simple process steps, risk tolerance can be developed from the same processes used by the technical risk engineers. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 117. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 117/186 The goal here is to move the integrated risk tolerance – technical, schedule, cost – forward from a dis–integrated plan to an integrated plan Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 118. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 118/186 Read the chart as follows: The upper horizontal band on the plot is “Ready Early”. “Ready On–time” is the middle band that also spans the launch window. “Ready late” is the lower band, which means a 6–month slip to the next launch window and all associated costs that go with that slip. The upper line plotted is the deterministic completion date (i.e. no risk) and the lower line plotted with the 20th and 80th percentile confidence bands on the risk– adjusted completion date. The project’s objective is to continue to invest in risk mitigation actions until the band and the area of highest likelihood is no longer in the “Missed Launch Period” area of the chart. Note the improving trend over time indicating the success of the risk mitigation actions as well some “Accepted” risks passing their exposure window without becoming problems. Taken from [Risk Based Decision Support techniques for Programs and Projects] http://www.futron.com/pdf/RBDSsupport tech.pdf Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 119. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 119/186 As Program Events progress the risk mitigation processes need to progress as well. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 120. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 120/186 Here’s a 4 step progress for installing risk in the IMS and producing a “risk tolerant” plan Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 121. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 121/186 The use of branching probabilities is important for the assessment of the “risk tolerance.” Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 122. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 122/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 123. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 123/186 The use of Risk+ and Monte Carlo replaces the PERT approach to schedule duration probability analysis. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 124. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 124/186 The “goodness” of the IMS is important to the quality of the results Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 125. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 125/186 The distribution to use for a task depends on the underlying risk profile. Triangle is common, but it over biased the risk on the high end. Beta can be used, but the simple Beta distributions in Risk+ may not represent the real risk profile. BetaPERT is the better one, but Risk+ does not support it. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 126. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 126/186 Which tasks drive the sensitivity of a completion date needs to be understood. Not all tasks have the same impact on the outcome. The “tornado” chart is one way of showing this. The Power Law’s behind Pareto’s rule is worth understanding for many reasons, not just schedule and cost modeling. Power Laws occur across a wide variety of domains, from moon crater sizes to the frequency of words in English. http://www.nslij– genetics.org/wli/zipf/ is a good place to look for the impacts of Power Laws on everyday life. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 127. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 127/186 In order to build a model of the schedule we have to start with the schedule. But first we have to start with the model of the schedule. This is the role of the IMP, but the connections between the ACs are needed, not just the list of the IMP elements. From this model the schedule elements can be arranged to follow the strategy of the IMP rather than represent the passage of time and the consumption of resources. From there a model of the risk areas, mitigations, parallel development paths, reevaluation points, and “hot spots” (sensitivity analysis) can be extracted. This information can them be used to assess the robustness of the IMS Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 128. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 128/186 The primary graphic for an IMS evaluation is the cumulative probability of a completion time. This is technically referred to as the Cumulative Density Function (CDF) This is the format most useful for answering the question – how long will this take? Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 129. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 129/186 The confidence intervals produced by the CDF can be assessed over time against targets. These targets can be Technical Performance Measures or any other style of metric that is connected with cost, schedule and technical performance Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 130. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 130/186 Another view is the confidence in the schedule dates as a function of time. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 131. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 131/186 It is important to understand the sensitivity of a completion time to the various “drivers” of this sensitivity. This makes visible the “hot spots” in the IMS that require attention, mitigation, or even re–planning to reduce sensitivity Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 132. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 132/186 In Monte Carlo each task can take on a wide variety of roles. It can be the driver for the total schedule duration at one time, and at another time (in the simulation) have little effect on the outcome. The Criticality of the task is how “important” it is as a function of the number of simulation runs. The higher the criticality of the task, the more important it is to look at the details and determine what mitigations should take place to keep this task lower in the criticality index. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 133. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 133/186 When sensitivity and criticality are combined a sense of the cruciality. Cruciality is defined as “a state of critical urgency.” Although this sounds like a redundancy term, it can be used to focus our attention on those tasks that are both critical and sensitive. It is important to understand the sensitivity aspects, since these can change and drive the schedule in non–obvious ways. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 134. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 134/186
- 135. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 135/186 Let’s look at some examples of Monte Carlo Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 136. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 136/186 The Monte Carlo simulation makes use of the three point estimates generated during the PERT analysis. This numbers represent the upper, lower and most likely durations. This values are then used to draw random numbers from the probability distribution for evaluating the activity network. The branching probabilities can then be added for the alternatives paths and risk mitigation activities. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 137. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 137/186 The use of “expert judgment” itself needs to be calibrated. The unanswered question on this program and many others is “what does a good risk tolerant IMS actually look like?” The “units of measure” for risk tolerance and the confidence in the probabilistic estimates needs to be established before the estimating and modeling process can be “calibrated” Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 138. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 138/186 The ranking of risk or the ranking of anything needs to be done in a structured manner. A geometric progression is a very useful approach, since it forces the focus on ranking. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 139. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 139/186 The “sense” of risk and real risk need to be connected. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 140. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 140/186 Some type of risk ranking needs to be developed for the IMS tasks. One approach is the TRL scale. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 141. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 141/186 When tasks are arranged in series the cumulative probability of completion is show in the table on the right Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 142. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 142/186 When the tasks are arranged in parallel a different completion profile results. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 143. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 143/186 All of this is very interesting in a Power Point presentation – marketecture it's called. Let’s look at a real schedule and start to apply some of the things we’ve learned. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 144. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 144/186 This is a very simple construction plan. The tasks are networked in a way to show how the Risk+ tool works. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 145. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 145/186 The first picture of a completion time is the PERT assessment. The task Construction Schedule Margin (the end of this task is the end of the planned margin) has a target date of 2/8/06 and a forecast PERT date of 3/6/06. This shows there is not enough margin by one month for this task Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 146. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 146/186 The same task, evaluated with Risk+ shows a different completion date. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 147. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 147/186 Project Risk Analysis is part of any good risk management activity. This has been said numerous times and needs to he repeated daily. Both the technical and the programmatic risk aspects of the program need to be shown in the IMS. Any questions, changes, updates, suggestions – anything that touches the IMS or the cost model – needs to be assessed from the point of view of programmatic risk. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 148. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 148/186 The accuracy of the dates and costs in the IMS is a “relative” term. ±20% to start with is pretty good. As the program proceeds accuracy improves but it is always a statistical estimate until after the fact. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 149. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 149/186 If we take a deterministic approach the planning then there will be built in issues. The first is that all estimates must include a confidence interval or they are wrong. The “natural” approach to estimating almost always results in a bound that is too wide as well as being optimistic or pessimistic but hardly never accurate. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 150. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 150/186
- 151. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 151/186 Now that we’ve reached a fairly detailed level of discussion regarding programmatic risk assessment, it’s time to talk about cost risk assessment. The first concept to understand is that cost and schedule are connected. This is obvious. But they are not connected in any linear manner. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 152. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 152/186 The basic principles of cost estimating start with the understanding of the uncertainty in the estimates of cost. These uncertainties must be connected to the technical uncertainties as well as the programmatic and simple cost variances. The arithmetic addition of costs creates a false number of the not only the cost but any variance in this cost. Monte Carlo simulation is one starting point, but like the programmatic simulations, the underlying probability distributions must be understood before the numbers have any real meaning Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 153. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 153/186 The connection between a technical parameter is its cost is not only potentially non–linear it is probabilistically non-linear. Knowing these Cost Element Relationships (CER) is a critical success factor for managing in the presence of uncertainty for both reducible and irreducible risks that are created Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 154. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 154/186 A simple 9 step (not so simple actually) process can be used to build a cost estimate. Starting with the “likely” program in the form of an IMS, the tasks for delivering that program are defined. The underlying probability distributions for the cost of each delivering activity are developed. This is much like the development of the baseline IMS, but the next step is much different. The correlation between each WBS element is developed. These correlations are used to build a model of sensitivity of the cost to changes in the tasks. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 155. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 155/186 At this point the Risk+ tool fails to deliver what is needed. Wither Crystal Ball or @RISK is needed to connect these correlations together. The technical uncertainty of the program is used to drive the cost uncertainty. This is where the technical and programmatic risk assessments joins. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 156. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 156/186 The production of the familiar probability curves for the likelihood of cost is the result. This curve tells is the likelihood that some event will occur. For example the probability of a cost in this example. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 157. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 157/186 The risk margin in dollars is the result needed to make this connection. Without this margin, the budget for the program ‒ as a single point estimate ‒ is not credible. All programs operate in the presence of uncertainty, so no single point estimate can be correct. For reducible uncertainties, specific actions can be take to protect the outcome. For irreducible uncertainties, only margin can protect the outcomes Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 158. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 158/186 Without the cost margin, the result is the risk that the budget is not sufficient to complete the program without a cost overrun. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 159. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 159/186 Management reserve, contingency and other budget protection is then needed. Much much reserve and contingency comes from modeling the program and determining the level of confidence needed to protect the budget from overrun. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 160. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 160/186 With models of both the cost uncertainties (reducible and irreducible) and schedule uncertainties (reducible and irreducible) a Risk Adjusted Integrated Master Schedule can be created. This IMS has Risk Handling activities for the Reducible Risks and Margin for the Irreducible Risks Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 161. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 161/186 With the Risk Handling content in place, a new assessment of the probabilistic outcomes can be shown. Before risk management and after risk management. This is no actionable information to the decision makers. Was the risk reduced enough to start the program, continue the program? Or are further risk reduction activities needed? Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 162. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 162/186 The cumulative cost curve – in this example, can then be used to make decisions using the value at risk Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 163. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 163/186 All programs operate in the presence of uncertainty. Making decisions in the presence of this uncertainty can take place with the proper information. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 164. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 164/186
- 165. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 165/186 We’re near the end now, so your brain is certainly getting full. This is quite a bit of information to absorb, but it needs to be done before we can say we are building “risk tolerant” plans. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 166. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 166/186 When probabilistic schedule analysis is used it does not replace the need for a well formed project network. It only replaces the use of PERT for estimating the completion dates. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 167. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 167/186 The quality of the probabilistic estimates is the foundation of confidence. The next step is to clearly identify where in the IMS risks are being mitigated, the impacts of this mitigation and the overall confidence in the master plan resulting from this mitigation Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 168. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 168/186 The correlations between cost, schedule, and technical risk must be made explicit. A model of how these elements interact is the basis for answering the “what if” questions that occur when the risk item becomes active. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 169. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 169/186 Risk based schedule and cost management is core to programmatic integrity. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 170. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 Prepared for NNJ05111915R, by Glen B. Alleman December 2005 170/186
- 171. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 171/186 Like any good idea it can be improved on forever. These opportunities are much harder to address than the process so far. They require care and effort to build a correlation matrix for the tasks. They require detailed understanding of the underlying statistical processes and the historical data that was used to develop these distributions. For most projects this is beyond the scope of the effort and may be beyond the business interests as well – since the pay back is not clearly defined. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 172. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 172/186 The dependencies between tasks is the basis of the correlation function. This is very important if a true model of the network is to be developed. In the absence of the correlations it is assumed tasks are independent, which of course can not be the case. Building a Program risk assessment requires that cost and schedule be connected as well – correlated. Cost and schedule are not linear, so any simple model of changes in one linearly effecting the other cannot work. Finally the idea of a causal model – a cause and a set of effects provides deeper insight into the risk behaviors of the network. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 173. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 173/186 There is too much information here for a single digestive process. The only way to absorb all this is to start practicing probabilistic schedule and cost analysis and make the knowledge appear in the output information. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 174. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 174/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 175. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 175/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 176. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 176/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 177. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 177/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 178. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 178/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 179. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 179/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 180. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 180/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 181. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 181/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 182. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 182/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 183. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 183/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 184. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 184/186 Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 185. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 185/186 There are nearly unlimited resources on the web. The challenge of course is finding them. Here’s some know starting points. Prepared for NNJ05111915R, by Glen B. Alleman December 2005
- 186. Analysis of Probabilistic Schedule and Cost Last Updated: 7/8/19 186/186 This has been a long journey over hopefully many weeks of discussion and hands on experience with Risk+ and real project schedules. Building a risk tolerant IMS is a “practice” and practices require proficiency. Proficiency comes from “doing the work,” looking at the results and making changes for improvement. This is just the beginning. Prepared for NNJ05111915R, by Glen B. Alleman December 2005