Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Risk-informed Decision Making Presented at the Seventh Annual NASA Project Management Challenge Galveston, Texas February 9-10, 2010 Homayoon Dezfuli, Ph.D. Office of Safety and Mission Assurance NASA Headquarters Gaspare Maggio Technology Risk Management Operations Information Systems Laboratories, Inc.Used with Permission
  2. 2. Acknowledgments• This presentation is based on the material contained in the first draft of NASA Risk-informed Decision Making Handbook, released in October 2009. The authors acknowledge the contribution of the following individuals in the preparation of this handbook: – Chris Everet, Information Systems Laboratories, Inc. – Rod Williams, Information Systems Laboratories, Inc. – Robert Youngblood, Idaho National Laboratory – Curtis Smith, Idaho National Laboratory – Peter Rutledge, Quality Assurance & Risk Management Services, Inc. 2
  3. 3. Background 3
  4. 4. NPR 8000.4A• The latest version of NPR 8000.4A, Agency Risk Management Procedural Requirements, was issued on December 16, 2008 – Accessible from NASA Online Directives System (NODIS) Library – http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=8000&s=4A• This directive evolves NASA’s Risk Management (RM) approach to entail two complementary processes: – Risk-informed Decision Making (RIDM) Emphasizes the proper use of risk analysis in its broadest sense to make risk informed decisions that impact all mission execution domains (e.g., safety, technical, cost, and schedule) for program/projects and mission support organizations for supporting development of baseline performance requirements by selecting performance commitments – Continuous Risk Management (CRM) Focuses on the management of risk associated with implementation of baseline performance requirements RM ≡ RIDM + CRM 4
  5. 5. General Definition of “Risk” per NPR 8000.4A “Potential for performance shortfalls, which may be realized in the future, with respect to achieving explicitly established and stated Performance Requirements”• This definition of “risk” guided the development of some of the RIDM concepts• The performance shortfalls may be related to any one or more of the following mission execution domains – Safety – Technical – Cost – Schedule 5
  6. 6. The RIDM Process as defined in NPR 8000.4A• What is RIDM? – A risk-informed decision-making process that uses a diverse set of performance measures along with Risk-Informed Decision Making (RIDM) other considerations within a deliberative process to inform decision making. (Paragraph A-14) Identification of Alternatives Identify Decision Alternatives (Recognizing Opportunities) in the Context of Objectives A decision-making process relying primarily on a narrow set of model-based risk metrics would be considered “risk-based.” (Note to Paragraph A-14) Risk Analysis of Alternatives Risk Analysis (Integrated Perspective) and Development of the Technical Basis for Deliberation• What does it involve? – Identification of decision alternatives (decision Risk-Informed Alternative Selection context) and considering a sufficient number and Deliberate and Select an Alternative and diversity of Performance Measures Associated Performance Commitments Informed by (not solely based on) Risk Analysis – Risk analysis of decision alternatives (uncertainty analysis of performance associated with the alternative To Requirements Baselining – Selection of a decision alternative informed by (not solely based on) Risk Analysis Results 6
  7. 7. The RIDM Process Begins with NASAStrategic Goals• Within NASA’s organizational hierarchy, high-level objectives (NASA Strategic ss e Goals) flow down in the form oc Pr of progressively more DM RI detailed performance requirements, whose satisfaction assures that objectives are met• RIDM is designed to maintain focus on strategic goals as decisions are made throughout the hierarchy 7
  8. 8. RIDM and CRM Within the NASA Hierarchy• RIDM and CRM operate at each level of the NASA hierarchy, with interfaces for the flowdown of requirements, the elevation of risk issues, and the communication of risk information 8
  9. 9. RIDM Handbook Development 9
  10. 10. RIDM Handbook• OSMA has developed a Special Publication (in draft) to provide implementation guidance NASA/SP-2009-XXXX NASA/SP-2009-XXXX Rev0 Rev0 – It decomposes RIDM into specific process steps, with specific guidance provided for each step NASA NASA – It elaborates on the relationships between RIDM, Risk Informed Decision Making Risk Informed Decision Making requirements development, requirements baselining Handbook Handbook (or rebaselining), and CRM – The present emphasis is on Programs and Projects; however, the process is generally applicable to all activities covered by NPD 7120.4• The development team observed/reviewed a number of NASA decision-making activities for Office of Safety and Mission Assurance NASA Headquarters good practices & lessons learned – Altair Buyback Analysis – Ares I Recovery Assessment T H I S H A NDB OOK H A S NOT B E E N R E V I E W E D F OR E X POR T C ONT R OL R E ST R I C T I ONS; – Ares Launch Order Analysis C ONSUL T Y OUR C E NT E R /F A C I L I T Y /H E A DQUA R T E R S E X POR T C ONT R OL PR OC E DUR E S/A UT H OR I T Y PR I OR T O DI ST R I B UT I ON OF T H I S DOC UM E NT . – Exploration Systems Architecture Study 10
  11. 11. RIDM Process Themes• The importance of close ties between the selected alternative and the requirements derived from it – The RIDM process should promote the generation of achievable requirements (e.g., mean value results from high-level analyses should not become requirements) – As alternatives are modified, derived requirements should be rebaselined to follow suit• The importance of maintaining a focus on high-level objectives, for decisions made at all levels of the NASA hierarchy• The importance of considering multiple objectives across all mission execution domains (safety, technical, cost, schedule)• The importance of a documented decision 11
  12. 12. Information Flow in RIDM Coordinated by Risk Manager 12
  13. 13. RIDM Process Steps 13
  14. 14. RIDM ProcessPart 1 – Identification of Alternatives 14
  15. 15. RIDM Process – Part 1Step 1 – Receive Objectives & Understand Stakeholder Expectations • The goal of Step 1 is the development of unambiguous objectives, reflecting stakeholder expectations. • Typical inputs to Step 1 include: – Upper Level Requirements and Expectations: The needs, wants, desires, capabilities, constraints, external interfaces, etc., that are being flowed down from a higher level (e.g., program, project, etc.) – Identification of Stakeholders: Individuals or organizations that are materially affected by the outcome of a decision or deliverable but are outside the organization doing the work or making the decision • Typical outputs for capturing stakeholder expectations include the following: – Top-Level Requirements and Expectations: These are the top-level needs, wants, desires, capabilities, constraints, external interfaces, etc., for the product(s) to be developed – Top-Level Conceptual Boundaries and Functional Milestones: How the selected alternative will be operated to meet expectations. It describes the alternative’s characteristics from an operational perspective 15
  16. 16. RIDM Process – Part 1Step 2 – Derive Performance Measures from Objectives• In general, it can be difficult to assess decision alternatives against multifaceted and/or qualitative top-level objectives• To deal with this situation, objectives are decomposed, using an objectives hierarchy, into a set of lower-level performance objectives that any attractive alternative should have• A performance measure is then developed for each performance objective, as the quantity that measures the extent to which a decision alternative meets the performance objective Notional Objectives Hierarchy 16
  17. 17. RIDM Process – Part 1Step 2 – Derive Performance Measures from Objectives• Imposed Constraints – Performance objectives whose performance measures must remain within defined limits for every feasible alternative, give rise to imposed constraints that reflect those limits – Imposed constraints propagate through the objectives hierarchy – Imposed constraints include the success criteria for the undertaking, outside of which the top-level objectives are not achieved • Example: If an objective is to put a satellite of a certain mass into a certain orbit, then the ability to loft that mass into that orbit is an imposed constraint, and any proposed solution that is incapable of doing so is infeasible 17
  18. 18. RIDM Process – Part 1Step 3 – Compile Feasible Alternatives• Structuring the set of alternatives – Trade trees – Initially, the trade tree contains a number of high-level classes of decision alternatives representing different strategies – The tree is developed in greater detail by determining option categories for each strategy – Defined to the level required to quantify performance measures• As the tree is developed, alternatives may be pruned – Criteria • Infeasibility (e.g., does not meet imposed constraints) • Inferiority to other alternatives – Methods • Bounding analysis using point estimates • Expert opinion / deliberation 18
  19. 19. RIDM Process ReviewPart 1 – Identification of Alternatives 19
  20. 20. RIDM ProcessPart 2 – Risk Analysis of Alternatives 20
  21. 21. RIDM Process – Part 2Step 4 – Set Framework & Choose Analysis Methodologies (1)• Goal: to develop a risk analysis framework that integrates domain-specific performance assessments and quantifies the performance measures – Risk Analysis - probabilistic modeling of performance Uncertain Conditions Probabilistically - Determined Outcomes Funding Environment Operating Environment Risk Analysis of an Alternative Performance Measure 1 Limited • Safety Risk Data … • Technical Risk Technology • Cost Risk Development • Schedule Risk Design, Test & Production Processes Etc. Performance Measure n * Performance measures depicted for a single alternative• The challenge is to establish a transparent framework that: – Operates on a common set of performance parameters for each alternative – Consistently addresses uncertainties across mission execution domains and across alternatives – Preserves correlations between performance measures 21
  22. 22. RIDM Process – Part 2Step 4 – Set Framework & Choose Analysis Methodologies (2) • Setting the risk analysis framework (alternative specific) 22
  23. 23. RIDM Process – Part 2Step 4 – Set Framework & Choose Analysis Methodologies (3) • Choosing the analysis methodologies – Detailed domain-specific analysis guidance is available in domain- specific guidance documents like the NASA Cost Estimating Handbook, the NASA Systems Engineering Handbook, and the NASA Probabilistic Risk Assessment Procedures Guide – Depending on project scale, life cycle phase, etc., different levels of analysis are appropriate. The rigor of analysis should be enough to: • Assess compliance with imposed constraints • Distinguish between alternatives – Iteration is to be expected as part of the analysis process, as analyses are refined and additional issues are raised during deliberations 23
  24. 24. RIDM Process – Part 2Step 4 – Set Framework & Choose Analysis Methodologies (4) • Choosing the analysis methodologies – “Consumer Guide” chart • The rigor of the analysis should be sufficient to support robust decision-making (i.e., the decision maker is confident that the selected alternative is best, given the state of knowledge) 24
  25. 25. RIDM Process – Part 2Step 5 – Quantify Performance Measures (1)• Once the risk analysis framework is established and risk analysis methodologies determined, performance measures can be quantified• Since performance measures are typically not independent, correlation between performance measures should be preserved – For example, cost and schedule tend to be highly correlated. High costs tend to be associated with slipped schedules• One way to preserve correlations is to conduct analysis within a common Monte Carlo “shell” – For each iteration of the Monte Carlo shell, a common set of performance parameters is sampled and propagated through the entire suite of analyses, to produce the performance measures for that iteration 25
  26. 26. RIDM Process – Part 2Step 5 – Quantify Performance Measures (2)• Quantification via probabilistic modeling of performance 26
  27. 27. RIDM Process – Part 2Step 6 – Develop Risk-Normalized Candidate PerformanceCommitments (1)• Performance measure pdfs constitute the fundamental risk analysis results• However, there are practical difficulties comparing performance measures whose values are expressed as pdfs: – Overlapping pdfs – Relationships between pdfs and imposed constraints – Relationships between pdfs and derived requirements• To simplify the problem, one might use mean values to compare alternatives, but this approach can: – Produce values that are disproportionately influenced by the tail ends of the pdfs – Introduce significant probabilities of falling short of imposed constraints, even when the mean values meet imposed constraints – Lead to derived requirements that are not achievable• What is needed is a technique for selecting alternatives, that is informed by an understanding of each alternative’s chances of not meeting performance expectations 27
  28. 28. RIDM Process – Part 2Step 6 – Develop Risk-Normalized Candidate PerformanceCommitments (2)• Performance Commitments – A performance commitment is a performance measure value set at a specified percentile of the performance measure’s pdf – Performance commitments help to anchor the decision-maker’s perspective to specific performance expectations for each alternative – For a given performance measure, the performance commitment is set at the same percentile for all decision alternatives – Performance commitments support a risk-normalized comparison of decision alternatives, at a level of risk tolerance determined by the decision maker 28
  29. 29. RIDM Process – Part 2Step 6 – Develop Risk-Normalized Candidate PerformanceCommitments (3) Candidate Performance Commitments facilitate comparison of performance across alternatives, subject to the decision-maker’s risk tolerance for each Performance Measure Risk of not meeting specified Alternative performance A Alternative B Alternative C Payload Capability Imposed Constraint 29
  30. 30. RIDM Process – Part 2Step 6 – Develop Risk-Normalized Candidate PerformanceCommitments (4)• Developing Performance Commitments: – The inputs to performance commitment development are: • The performance measure pdfs for each decision alternative • An ordering of the performance measures • A risk tolerance for each performance measure, expressed as a percentile value – For each alternative, performance commitments are developed by sequentially determining the value of each performance measure that matches the decision maker’s risk tolerance for that performance measure, conditional on meeting previously-defined performance commitments. • This value becomes the performance commitment for the current performance measure – The process is repeated until performance commitments have been developed for all performance measures 30
  31. 31. RIDM Process – Part 2Step 6 – Develop Risk-Normalized Candidate PerformanceCommitments (5) 31
  32. 32. RIDM Process – Part 2Step 7 – Develop the Technical Basis for Deliberation• The Technical Basis for Deliberation (TBfD) contains the information needed to risk-inform the selection of a decision alternative• The TBfD contains: – A statement of the top-level objectives and imposed constraints – The objectives hierarchy and performance measures – A summary description of the compiled decision alternatives, indicating pruned alternatives – A summary of the risk analysis framework and models – Scenario descriptions – Marginal performance measure pdfs and a summary of significant correlations – A tabulation of risk with respect to imposed constraints – Identification of significant risk drivers with respect to imposed constraints – Candidate performance measure risk tolerances 32
  33. 33. RIDM Process ReviewPart 2 – Risk Analysis of Alternatives 33
  34. 34. RIDM ProcessPart 3 – Risk-Informed Alternative Selection 34
  35. 35. RIDM Process – Part 3Step 8 – Deliberate (1)• In Step 8, Deliberate, relevant stakeholders, risk analysts, and decision makers deliberate the merits and drawbacks of each alternative, given information in the TBfD• This step is iterative, and may involve additional risk analysis and/or information gathering• The decision maker, or his proxy, may also invoke deliberation as an intermediate step to cull the alternatives going forward (i.e., downselection) Deliberation: Any process for communication and for raising and collectively considering issues. In deliberation, people discuss, ponder, exchange observations and views, reflect upon information and judgments concerning matters of mutual interest, and attempt to persuade each other. Deliberations about risk often include discussions of the role, subjects, methods, and results of risk analysis. 35
  36. 36. RIDM Process – Part 3Step 8 – Deliberate (2)• Step 8, Deliberate, is structured in terms of: – Generate candidate performance commitments • Establish risk tolerances on the performance measures • Order the performance measures – Assess the credibility of the estimation methods – Identify contending alternatives • Infeasibility • Dominance • Inferior performance in key areas – Additional uncertainty considerations • The potential for exceptionally high or poor performance• Deliberation is iterative 36
  37. 37. RIDM Process – Part 3Step 8 – Deliberate (3)• Generate Candidate Performance Commitments -- Candidate performance commitments are generated by the deliberators for the purpose of deliberation and down-selection prior to finalization by the decision maker. This is done by: – Establishing risk tolerances on the performance measures: • Relationship to imposed constraints – Low risk tolerances on performance measures that have imposed constraints assure a high likelihood of program/project success • High-priority objectives – Low risk tolerances are appropriate for objectives that have high priority, but for which imposed constraints have not been set Note: The lack of an imposed constraint on a performance measure does not necessarily mean that the objective is of less importance; it may just mean that there is no well defined threshold that defines success • Low-priority objectives and/or “stretch goals” – Higher risk tolerances may be appropriate for objectives that are not crucial to program/project success 37
  38. 38. RIDM Process – Part 3Step 9 – Select an Alternative (1)• In Step 9, Select an Alternative, the deliberators present the contending alternatives to the decision maker, along with supporting information. The decision maker selects an alternative and documents his/her rationale• In addition to information in the TBfD, information produced during deliberation should also be summarized and forwarded to the decision-maker. This includes: – Risk tolerances and performance commitments – These are key pieces of information for the decision-maker. They strongly influence requirements development and the corresponding program/project risk that is to be accepted going forward. – Pros and cons of each contending alternative – An itemized table of the pros and cons of each alternative is recommended for the contending alternatives. This format has a long history of use, and is capable of expressing qualitative and contentious issues 38
  39. 39. RIDM Process – Part 3Step 9 – Select an Alternative (2)• Information forwarded to the decision-maker should also include: – Risk lists – Each alternative will have different contributors to its performance commitment risks. Correspondingly, a risk list can be compiled for every contending alternative, which identifies the major uncertainties that contribute to risk – Analysis credibility matrix – Communicates the credibility of the risk analysis methods and results 39
  40. 40. RIDM Process – Part 3Step 10 – Document the Decision Rationale• The Risk-Informed Selection Report (RISR) is a record of the risk- informed decision, and documents the decision rationale. The RISR contains: – The TBfD – From deliberation: • Assessment of the credibility of the risk analysis • Identification of the contending decision alternatives • Pros and cons of each contending alternative • Any briefing material presented by the deliberators to the decision-maker – From the decision-maker: • Identification of the selected alternative • The finalized risk tolerance for each performance measure, along with the corresponding performance commitments for the selected alternative • Comparison of the selected alternative to the non-selected contending alternatives, summarizing the relative pros and cons, and the reasons why the selected alternative is preferred • Assessment of the robustness of the decision 40
  41. 41. RIDM Process ReviewPart 3 – Risk-Informed Alternative Selection 41
  42. 42. Summary - 1• Risk-Informed Decision Making (RIDM) attempts to respond to some of the primary issues that have derailed programs in the past: – the “mismatch” between stakeholder expectations and the “true” resources required to address the risks to achieve those expectations, – the miscomprehension of the risk that a decision-maker is accepting when making commitments to stakeholders, and – the miscommunication in considering the respective risks associated with competing alternatives• A multi-step process has been developed to take advantage of existing systems engineering practices while also introducing risk analysis and systematic deliberative techniques into the decision-making process 42
  43. 43. Summary - 2• OSMA has developed a Special Publication (in draft) and associated training material to provide implementation guidance – Comments and suggestions from an agency-wide review cycle are being compiled and will be used in revising the draft for final publication – To download a copy and participate in the process go to: https://secureworkgroups.grc.nasa.gov/armwg (PBMA registration and site access approval required)• Future steps include: – Revision of how CRM should be conducted to be consistent with 8000.4A and take advantage of the information provided by the RIDM process – Better integration of the RIDM and CRM processes with the ultimate goal of a completely integrated and seamless Risk Management process 43