1. Risk-informed Decision Making
Presented at the Seventh Annual NASA Project
Management Challenge
Galveston, Texas
February 9-10, 2010
Homayoon Dezfuli, Ph.D.
Office of Safety and Mission Assurance
NASA Headquarters
Gaspare Maggio
Technology Risk Management Operations
Information Systems Laboratories, Inc.
Used with Permission
2. Acknowledgments
• This presentation is based on the material contained in the first draft of NASA
Risk-informed Decision Making Handbook, released in October 2009. The
authors acknowledge the contribution of the following individuals in the
preparation of this handbook:
– Chris Everet, Information Systems Laboratories, Inc.
– Rod Williams, Information Systems Laboratories, Inc.
– Robert Youngblood, Idaho National Laboratory
– Curtis Smith, Idaho National Laboratory
– Peter Rutledge, Quality Assurance & Risk Management Services, Inc.
2
4. NPR 8000.4A
• The latest version of NPR 8000.4A, Agency Risk Management Procedural
Requirements, was issued on December 16, 2008
– Accessible from NASA Online Directives System (NODIS) Library
– http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=8000&s=4A
• This directive evolves NASA’s Risk Management (RM) approach to entail two
complementary processes:
– Risk-informed Decision Making (RIDM)
Emphasizes the proper use of risk analysis in its broadest sense to make risk informed
decisions that impact all mission execution domains (e.g., safety, technical, cost, and
schedule) for program/projects and mission support organizations for supporting
development of baseline performance requirements by selecting performance commitments
– Continuous Risk Management (CRM)
Focuses on the management of risk associated with implementation of baseline
performance requirements
RM ≡ RIDM + CRM
4
5. General Definition of “Risk” per NPR 8000.4A
“Potential for performance shortfalls, which may be realized in
the future, with respect to achieving explicitly established and
stated Performance Requirements”
• This definition of “risk” guided the development of some of the
RIDM concepts
• The performance shortfalls may be related to any one or more
of the following mission execution domains
– Safety
– Technical
– Cost
– Schedule
5
6. The RIDM Process as defined in NPR
8000.4A
• What is RIDM?
– A risk-informed decision-making process that uses a
diverse set of performance measures along with Risk-Informed Decision Making (RIDM)
other considerations within a deliberative process to
inform decision making. (Paragraph A-14) Identification of Alternatives
Identify Decision Alternatives (Recognizing
Opportunities) in the Context of Objectives
A decision-making process relying primarily on a
narrow set of model-based risk metrics would be
considered “risk-based.” (Note to Paragraph A-14) Risk Analysis of Alternatives
Risk Analysis (Integrated Perspective) and
Development of the Technical Basis for
Deliberation
• What does it involve?
– Identification of decision alternatives (decision
Risk-Informed Alternative Selection
context) and considering a sufficient number and Deliberate and Select an Alternative and
diversity of Performance Measures Associated Performance Commitments
Informed by (not solely based on) Risk
Analysis
– Risk analysis of decision alternatives (uncertainty
analysis of performance associated with the
alternative
To Requirements Baselining
– Selection of a decision alternative informed by (not
solely based on) Risk Analysis Results
6
7. The RIDM Process Begins with NASA
Strategic Goals
• Within NASA’s organizational
hierarchy, high-level
objectives (NASA Strategic
ss
e
Goals) flow down in the form
oc
Pr
of progressively more
DM
RI
detailed performance
requirements, whose
satisfaction assures that
objectives are met
• RIDM is designed to maintain
focus on strategic goals as
decisions are made
throughout the hierarchy
7
8. RIDM and CRM Within the NASA Hierarchy
• RIDM and CRM operate at each level of the NASA hierarchy, with
interfaces for the flowdown of requirements, the elevation of risk
issues, and the communication of risk information
8
10. RIDM Handbook
• OSMA has developed a Special Publication (in
draft) to provide implementation guidance NASA/SP-2009-XXXX
NASA/SP-2009-XXXX
Rev0
Rev0
– It decomposes RIDM into specific process steps, with
specific guidance provided for each step
NASA
NASA
– It elaborates on the relationships between RIDM, Risk Informed Decision Making
Risk Informed Decision Making
requirements development, requirements baselining Handbook
Handbook
(or rebaselining), and CRM
– The present emphasis is on Programs and Projects;
however, the process is generally applicable to all
activities covered by NPD 7120.4
• The development team observed/reviewed a
number of NASA decision-making activities for Office of Safety and Mission Assurance
NASA Headquarters
good practices & lessons learned
– Altair Buyback Analysis
– Ares I Recovery Assessment
T H I S H A NDB OOK H A S NOT B E E N R E V I E W E D F OR E X POR T C ONT R OL R E ST R I C T I ONS;
– Ares Launch Order Analysis C ONSUL T Y OUR C E NT E R /F A C I L I T Y /H E A DQUA R T E R S E X POR T C ONT R OL
PR OC E DUR E S/A UT H OR I T Y PR I OR T O DI ST R I B UT I ON OF T H I S DOC UM E NT .
– Exploration Systems Architecture Study
10
11. RIDM Process Themes
• The importance of close ties between the selected alternative
and the requirements derived from it
– The RIDM process should promote the generation of achievable
requirements (e.g., mean value results from high-level analyses should
not become requirements)
– As alternatives are modified, derived requirements should be
rebaselined to follow suit
• The importance of maintaining a focus on high-level
objectives, for decisions made at all levels of the NASA
hierarchy
• The importance of considering multiple objectives across all
mission execution domains (safety, technical, cost, schedule)
• The importance of a documented decision
11
15. RIDM Process – Part 1
Step 1 – Receive Objectives & Understand Stakeholder Expectations
• The goal of Step 1 is the development of unambiguous objectives,
reflecting stakeholder expectations.
• Typical inputs to Step 1 include:
– Upper Level Requirements and Expectations: The needs, wants, desires,
capabilities, constraints, external interfaces, etc., that are being flowed down from a
higher level (e.g., program, project, etc.)
– Identification of Stakeholders: Individuals or organizations that are materially
affected by the outcome of a decision or deliverable but are outside the organization
doing the work or making the decision
• Typical outputs for capturing stakeholder expectations include the
following:
– Top-Level Requirements and Expectations: These are the top-level needs, wants,
desires, capabilities, constraints, external interfaces, etc., for the product(s) to be
developed
– Top-Level Conceptual Boundaries and Functional Milestones: How the selected
alternative will be operated to meet expectations. It describes the alternative’s
characteristics from an operational perspective
15
16. RIDM Process – Part 1
Step 2 – Derive Performance Measures from Objectives
• In general, it can be difficult to assess decision alternatives against multifaceted and/or
qualitative top-level objectives
• To deal with this situation, objectives are decomposed, using an objectives hierarchy, into
a set of lower-level performance objectives that any attractive alternative should have
• A performance measure is then developed for each performance objective, as the quantity
that measures the extent to which a decision alternative meets the performance objective
Notional Objectives Hierarchy
16
17. RIDM Process – Part 1
Step 2 – Derive Performance Measures from Objectives
• Imposed Constraints
– Performance objectives whose performance measures must remain within
defined limits for every feasible alternative, give rise to imposed
constraints that reflect those limits
– Imposed constraints propagate through the objectives hierarchy
– Imposed constraints include the success criteria for the undertaking,
outside of which the top-level objectives are not achieved
• Example: If an objective is to put a satellite of a certain mass into a
certain orbit, then the ability to loft that mass into that orbit is an
imposed constraint, and any proposed solution that is incapable of
doing so is infeasible
17
18. RIDM Process – Part 1
Step 3 – Compile Feasible Alternatives
• Structuring the set of alternatives – Trade trees
– Initially, the trade tree contains a number of high-level classes of decision
alternatives representing different strategies
– The tree is developed in greater detail by determining option categories for each
strategy
– Defined to the level required to quantify performance measures
• As the tree is developed,
alternatives may be pruned
– Criteria
• Infeasibility (e.g., does not
meet imposed constraints)
• Inferiority to other alternatives
– Methods
• Bounding analysis using point estimates
• Expert opinion / deliberation
18
21. RIDM Process – Part 2
Step 4 – Set Framework & Choose Analysis Methodologies (1)
• Goal: to develop a risk analysis framework that integrates domain-specific
performance assessments and quantifies the performance measures
– Risk Analysis - probabilistic modeling of performance
Uncertain Conditions Probabilistically - Determined
Outcomes
Funding
Environment
Operating
Environment
Risk Analysis
of an Alternative
Performance Measure 1
Limited
• Safety Risk
Data
…
• Technical Risk
Technology • Cost Risk
Development • Schedule Risk
Design, Test &
Production
Processes
Etc.
Performance Measure n
* Performance measures depicted for a single alternative
• The challenge is to establish a transparent framework that:
– Operates on a common set of performance parameters for each
alternative
– Consistently addresses uncertainties across mission execution domains
and across alternatives
– Preserves correlations between performance measures 21
22. RIDM Process – Part 2
Step 4 – Set Framework & Choose Analysis Methodologies (2)
• Setting the risk analysis framework (alternative specific)
22
23. RIDM Process – Part 2
Step 4 – Set Framework & Choose Analysis Methodologies (3)
• Choosing the analysis methodologies
– Detailed domain-specific analysis guidance is available in domain-
specific guidance documents like the NASA Cost Estimating
Handbook, the NASA Systems Engineering Handbook, and the
NASA Probabilistic Risk Assessment Procedures Guide
– Depending on project scale, life cycle phase, etc., different levels of
analysis are appropriate. The rigor of analysis should be enough to:
• Assess compliance with imposed constraints
• Distinguish between alternatives
– Iteration is to be expected as part of the analysis process, as analyses
are refined and additional issues are raised during deliberations
23
24. RIDM Process – Part 2
Step 4 – Set Framework & Choose Analysis Methodologies (4)
• Choosing the analysis methodologies – “Consumer Guide”
chart
• The rigor of the
analysis should be
sufficient to
support robust
decision-making
(i.e., the decision
maker is confident
that the selected
alternative is best,
given the state of
knowledge)
24
25. RIDM Process – Part 2
Step 5 – Quantify Performance Measures (1)
• Once the risk analysis framework is established and risk
analysis methodologies determined, performance measures
can be quantified
• Since performance measures are typically not independent,
correlation between performance measures should be
preserved
– For example, cost and schedule tend to be highly correlated. High
costs tend to be associated with slipped schedules
• One way to preserve correlations is to conduct analysis within
a common Monte Carlo “shell”
– For each iteration of the Monte Carlo shell, a common set of
performance parameters is sampled and propagated through the entire
suite of analyses, to produce the performance measures for that
iteration
25
26. RIDM Process – Part 2
Step 5 – Quantify Performance Measures (2)
• Quantification via probabilistic modeling of performance
26
27. RIDM Process – Part 2
Step 6 – Develop Risk-Normalized Candidate Performance
Commitments (1)
• Performance measure pdfs constitute the fundamental risk analysis
results
• However, there are practical difficulties comparing performance
measures whose values are expressed as pdfs:
– Overlapping pdfs
– Relationships between pdfs and imposed constraints
– Relationships between pdfs and derived requirements
• To simplify the problem, one might use mean values to compare
alternatives, but this approach can:
– Produce values that are disproportionately influenced by the tail ends of the pdfs
– Introduce significant probabilities of falling short of imposed constraints, even
when the mean values meet imposed constraints
– Lead to derived requirements that are not achievable
• What is needed is a technique for selecting alternatives, that is
informed by an understanding of each alternative’s chances of not
meeting performance expectations
27
28. RIDM Process – Part 2
Step 6 – Develop Risk-Normalized Candidate Performance
Commitments (2)
• Performance Commitments
– A performance commitment is a performance measure value set at a
specified percentile of the performance measure’s pdf
– Performance commitments help to anchor the decision-maker’s
perspective to specific performance expectations for each alternative
– For a given performance measure, the performance commitment is set
at the same percentile for all decision alternatives
– Performance commitments support
a risk-normalized comparison of
decision alternatives, at a level of
risk tolerance determined by the
decision maker
28
29. RIDM Process – Part 2
Step 6 – Develop Risk-Normalized Candidate Performance
Commitments (3)
Candidate Performance Commitments facilitate comparison of
performance across alternatives, subject to the decision-maker’s
risk tolerance for each Performance Measure
Risk of not
meeting specified
Alternative performance
A
Alternative
B
Alternative
C
Payload
Capability
Imposed
Constraint
29
30. RIDM Process – Part 2
Step 6 – Develop Risk-Normalized Candidate Performance
Commitments (4)
• Developing Performance Commitments:
– The inputs to performance commitment development are:
• The performance measure pdfs for each decision alternative
• An ordering of the performance measures
• A risk tolerance for each performance measure, expressed as a
percentile value
– For each alternative, performance commitments are developed by
sequentially determining the value of each performance measure
that matches the decision maker’s risk tolerance for that
performance measure, conditional on meeting previously-defined
performance commitments.
• This value becomes the performance commitment for the
current performance measure
– The process is repeated until performance commitments have
been developed for all performance measures
30
31. RIDM Process – Part 2
Step 6 – Develop Risk-Normalized Candidate Performance
Commitments (5)
31
32. RIDM Process – Part 2
Step 7 – Develop the Technical Basis for Deliberation
• The Technical Basis for Deliberation (TBfD) contains the
information needed to risk-inform the selection of a decision
alternative
• The TBfD contains:
– A statement of the top-level objectives and imposed constraints
– The objectives hierarchy and performance measures
– A summary description of the compiled decision alternatives, indicating pruned
alternatives
– A summary of the risk analysis framework and models
– Scenario descriptions
– Marginal performance measure pdfs and a summary of significant correlations
– A tabulation of risk with respect to imposed constraints
– Identification of significant risk drivers with respect to imposed constraints
– Candidate performance measure risk tolerances
32
35. RIDM Process – Part 3
Step 8 – Deliberate (1)
• In Step 8, Deliberate, relevant stakeholders, risk analysts, and decision
makers deliberate the merits and drawbacks of each alternative, given
information in the TBfD
• This step is iterative, and may involve additional risk analysis and/or
information gathering
• The decision maker, or his proxy, may also invoke deliberation as an
intermediate step to cull the alternatives going forward (i.e.,
downselection)
Deliberation: Any process for communication and for raising
and collectively considering issues. In deliberation, people
discuss, ponder, exchange observations and views, reflect upon
information and judgments concerning matters of mutual
interest, and attempt to persuade each other. Deliberations
about risk often include discussions of the role, subjects,
methods, and results of risk analysis.
35
36. RIDM Process – Part 3
Step 8 – Deliberate (2)
• Step 8, Deliberate, is structured in terms of:
– Generate candidate performance commitments
• Establish risk tolerances on the performance measures
• Order the performance measures
– Assess the credibility of the estimation methods
– Identify contending alternatives
• Infeasibility
• Dominance
• Inferior performance in key areas
– Additional uncertainty considerations
• The potential for exceptionally high or poor performance
• Deliberation is iterative
36
37. RIDM Process – Part 3
Step 8 – Deliberate (3)
• Generate Candidate Performance Commitments -- Candidate
performance commitments are generated by the deliberators for the
purpose of deliberation and down-selection prior to finalization by the
decision maker. This is done by:
– Establishing risk tolerances on the performance measures:
• Relationship to imposed constraints – Low risk tolerances on
performance measures that have imposed constraints assure a high
likelihood of program/project success
• High-priority objectives – Low risk tolerances are appropriate for
objectives that have high priority, but for which imposed constraints
have not been set
Note: The lack of an imposed constraint on a performance measure
does not necessarily mean that the objective is of less importance; it
may just mean that there is no well defined threshold that defines
success
• Low-priority objectives and/or “stretch goals” – Higher risk
tolerances may be appropriate for objectives that are not crucial to
program/project success
37
38. RIDM Process – Part 3
Step 9 – Select an Alternative (1)
• In Step 9, Select an Alternative, the deliberators present the contending
alternatives to the decision maker, along with supporting information. The
decision maker selects an alternative and documents his/her rationale
• In addition to information in the TBfD, information produced during deliberation
should also be summarized and forwarded to the decision-maker. This
includes:
– Risk tolerances and performance commitments – These are key pieces of
information for the decision-maker. They strongly influence requirements
development and the corresponding program/project risk that is to be accepted
going forward.
– Pros and cons of each contending alternative – An itemized table of the pros
and cons of each alternative is recommended for the contending alternatives.
This format has a long history of use, and is capable of expressing qualitative and
contentious issues
38
39. RIDM Process – Part 3
Step 9 – Select an Alternative (2)
• Information forwarded to the decision-maker should also include:
– Risk lists – Each alternative will have different contributors to its performance
commitment risks. Correspondingly, a risk list can be compiled for every
contending alternative, which identifies the major uncertainties that contribute to
risk
– Analysis credibility matrix – Communicates the credibility of the risk analysis
methods and results
39
40. RIDM Process – Part 3
Step 10 – Document the Decision Rationale
• The Risk-Informed Selection Report (RISR) is a record of the risk-
informed decision, and documents the decision rationale. The RISR
contains:
– The TBfD
– From deliberation:
• Assessment of the credibility of the risk analysis
• Identification of the contending decision alternatives
• Pros and cons of each contending alternative
• Any briefing material presented by the deliberators to the decision-maker
– From the decision-maker:
• Identification of the selected alternative
• The finalized risk tolerance for each performance measure, along with the
corresponding performance commitments for the selected alternative
• Comparison of the selected alternative to the non-selected contending
alternatives, summarizing the relative pros and cons, and the reasons why
the selected alternative is preferred
• Assessment of the robustness of the decision
40
42. Summary - 1
• Risk-Informed Decision Making (RIDM) attempts to
respond to some of the primary issues that have derailed
programs in the past:
– the “mismatch” between stakeholder expectations and the
“true” resources required to address the risks to achieve
those expectations,
– the miscomprehension of the risk that a decision-maker is
accepting when making commitments to stakeholders, and
– the miscommunication in considering the respective risks
associated with competing alternatives
• A multi-step process has been developed to take
advantage of existing systems engineering practices
while also introducing risk analysis and systematic
deliberative techniques into the decision-making
process
42
43. Summary - 2
• OSMA has developed a Special Publication (in draft) and
associated training material to provide implementation
guidance
– Comments and suggestions from an agency-wide review
cycle are being compiled and will be used in revising the
draft for final publication
– To download a copy and participate in the process go to:
https://secureworkgroups.grc.nasa.gov/armwg (PBMA
registration and site access approval required)
• Future steps include:
– Revision of how CRM should be conducted to be
consistent with 8000.4A and take advantage of the
information provided by the RIDM process
– Better integration of the RIDM and CRM processes with the
ultimate goal of a completely integrated and seamless Risk
Management process
43
