This document outlines the agenda for a presentation on risk-informed decision making (RIDM). The presentation will cover: 1. The inherent riskiness of current uncertain times and the need to evolve risk management approaches to remain relevant. 2. An explanation of what RIDM is and why it is important now, given that continuous risk management (CRM) is already practiced. 3. Examples of when and why to use RIDM in addition to discussing the actual steps involved in conducting RIDM. The presentation aims to demonstrate how RIDM can help risk management practices evolve to address a more dynamic environment with changing mission objectives and resources. RIDM is presented as a complement to

2. Presentation Themes
• Inherent riskiness of uncertain times
• Evolving risk management to stay relevant
• What exactly is Risk Informed Decision-Making (RIDM)?
• Why RIDM now, since we’re already doing Continuous Risk
Management (CRM)?
• RIDM is scary!
• When to use RIDM and why
• Actual steps (really?!)
• Go forth & do good RIDM
2

3. Uncertainty = Reality – Perception?
• We live in uncertain times! Whether at NASA or in industry, it’s no longer
business as usual – belts are tighter!
• Budgets are fluctuating, missions requirements are changing.
• Advocates and naysayers alike point to risk – known unknowns – and scarier
still, unknown unknowns!
• However, simply acknowledging risk ≠ active risk management.
• Since uncertainty is rife with risk, NASA requires a more flexible or agile
approach to risk management to help decision makers adapt.
3

4. Evolving Risk Management
• Risk is a triplet = a scenario with a chance or probability (L) of
experiencing a negative consequence (C).
− Ref. NASA Procedural Requirements (NPR) 8000.4A
• In recent years, NASA Programs, Projects, and Directorates have
employed Continuous Risk Management (CRM) to mitigate risk
across key performance domains:
− Cost (to achieve or regain required performance, cost to mitigate
risk, cost of inaction)
− Schedule (Critical Path)
− Technical
− Safety (and Mission Assurance)
− Core Capabilities (Critical Assets)
4

5. Evolving Risk Management (cont.)
• Why the need for more flexibility?
− CRM presupposes a linear mission or a mission that has clearly defined
end-objectives.
– Today, missions can be less linear (especially for human spaceflight)
and end-objectives can be dynamic as policies change.
– In this environment, decision-makers must be flexible and even
innovative to readily adapt to changes in mission
objectives, funding, and resources.
– To keep pace, we need to be innovative, systemic thinkers in the way
we view risk!
5

6. Evolving Risk Management (cont.)
• To remain relevant, risk management practices must evolve as NASA
adapts/evolves into a 21st century agency:
– For spaceflight, adaptation means a suite of capabilities independent of discrete mission
objectives; i.e., build a launch capability whether the ultimate destination is a near-Earth-
object, the Moon, or Mars.
Convergence
NEO
Lunar
Policy Mission
Direction Objectives Mars
Destination
– As NASA redefines itself, the nature of risk management must shift as well:
• From: Identifying and mitigating risk(s) that prevent a performing organization from
meeting a known requirement.
• To: identifying and mitigating performance shortfalls while helping decision-makers
find the best solution to meet tactical and strategic objectives.
• RIDM can help to evolve Risk Management practices to meet this challenge!
– NPR 8000.4A uses RIDM as a pre-quel to CRM.
– This can help frame the risk discussion from a systemic perspective.
6

7. Evolving Risk Management (cont.)
Risk Management adds RIDM to CRM in an integrated framework to:
• Foster proactive management of threats to performance
• Better inform decision-making using relevant (timely) risk information
• More effectively manage risks by focusing CRM on existing baseline performance
requirements and on those emerging via the RIDM process
• Through a graded approach, adapt to varying levels of risk over time.
Not my
problem! Too funny!
Hey, we Interface is 50% right!
followed Huh?
the spec, I
better get
When should
paid decision makers
assess for risk
outcomes?
Risk Management – Avoiding Undesirable Consequences! 7

8. RIDM is Scary
• RIDM terminology can seem daunting – not to worry, we will translate the
key terms!
• A key component of RIDM per NPR 8000.4A is advocacy for quantitative
risk analyses to add rigor to decision processes and trades for:
– Complex, multi-scenario projects
– Projects with competing stakeholders
– (fill-in your case).
• However desirable, quantitative analyses are not always feasible. Can be
costly and may not be feasible for activities that are smaller, lower dollar, or
in the operational phase.
– Graded approach allows you to match the rigor of analysis commensurate
with the criticality and type of decision to be made.
• This presentation discusses RIDM in practical ways and outlines a RIDM
case study to illustrate the how’s & why’s – let’s get started!
8

9. Generic Risk Model
Controls Controls/Response
Risk Consequence
Activity Event
Consequence
Consequence
Preventive Measures Reduce Severity Mitigate or Accept
(Based on Risk Scenarios, Requirements, Plans, Historical & Current Performance, Resources…)
RIDM can help to clarify To develop or
objectives: ? ? revise: To meet:
● Activities and Products? Activity Requirements Performance
(“Shall requirements”) Capabilities objectives, o
● Support functions? ? Operations ver time
(Insight/Oversight) Processes
9

10. Risk Model with RIDM
(NPR 8000.4A)
10

11. Got Risk?
RIDM Process
• ID Scenario
• ID Stakeholders
• Performance
Objectives
• Performance
Measures
• Confirm
Constraints
• Analyze
• Down Select
• Risk Inform
Decision
Do you know where your alligators are today?
11

12. Key RIDM Terms
RIDM
Key
#
Pt
Term "RIDM-Speak" Handbook Translation
reference
Determine depth of analysis needed to Rigor: Match level of analysis to
Section
reduce uncertainty. Not all risk analysis significance of decision, project
1 √ Graded Approach 3.2.1.3,
can/should be quantitative. Use qualitative p. 51
scale, likelihood, and/or level of
analysis, as appropriate. impact.
Performance Derived via an objectives hierarchy; General Mandated or delegated goals that
2 √
Objective objectives at each level of an organization. term organizations must meet at each level.
Metric to capture the extent to which a
Performance system, process, or activity meets a Generic
3 √ Figures of merit (FOM)
Measure performance objective. Same measure term
can be used for all alternatives.
Alternative-independent level of Match performance objective to
Sec 1.5,
Performance performance: Using accepted measures, risk tolerance of decision-maker (to
4 √ Part 3, pp.
Commitment compare across alternatives, and consider 19-22
compare across alternatives at
the risk tolerance of the decision-maker. same level of confidence).
Evolving document of ground rules and p.17;
Technical Basis assumptions used to structure a risk- App.D: Basis of Estimate (BOE) that
5 √ for Deliberation informed decision. Specifies information Content evolves with each iteration of
(TBfD) needed to characterize uncertainty for each Guide, pp. risk analysis
decision alternative. 108-109
Ref: NASA Risk Management Handbook NHBK_2011_3422
12

13. Iterative Risk Decision Process
Mission: Mission Support:
● Technical Objectives ● Capability
● Tactical Operations ● Capacity / Infrastructure
● Schedule / Discrete Milestones ● Institutional Goals
● Logistics ● Supply Chain
● Strategic Objectives ● Strategic Objectives
Systems Engineering Handbook
13

14. RECAP
• Evolving Risk Management = RIDM + CRM
− Uncertainty and evolving Agency direction elevate the need for responsive Risk
Management.
− CRM is still a successful tool in the Risk Management tool box.
− RIDM complements CRM by enhancing the risk analyses in a graded approach
commensurate with decisions to be made.
− RIDM and CRM used together throughout your iterative decision process can
ensure timely risk information.
• Risk-informed decisions = an adaptive process:
− Factoring timely risk information into key decisions adaptation to dynamic
environments.
− CRM alone enables response to a potential risk.
− Adding RIDM iteratively can help with early insight into individual risks as well as
overall performance risks over time = flexibility.
14

15. Perspective: RIDM-to-CRM
CRM-Centric RIDM-Centric
• Start with a • Start with a
Performance Performance
Shortfall (Risk) Objective
• Use mini-RIDMs • Frame the
to evaluate/make decision space
handling (graded
decisions approach)
• Use CRM to
• Assess
analyze risk of
qualitatively or
each alternative
quantitatively
• Down-select
• Develop/ alternatives
implement
• Track/monitor
mitigation via CRM
Systems Engineering Handbook
“Two Sides of the Same Coin” 15

16. Deciphering RIDM
1a. Identify Performance
Objectives
1b. Derive Performance
Measures
2. Identify
Feasible Alternatives
3. Set the Framework (TBfD)*
4. Analyze Alternatives &
Document
5. Risk-Normalize
Commitments
6a. Deliberate & Decide
6b. Select an Alternative &
Document
16

17. Illustrated Example*
• Decision:
1. Select best location for Short Stack Hypergolic Servicing.
2. Base decision on performance objectives, performance
measures, stakeholder considerations, and domain risks of each feasible
alternative.
• Alternatives:
1. Existing facilities (e.g., Pad)
2. New facility
3. Other?
CARGO HAZARDOUS
SERVICING FACILITY
Map of KSC Facilities and Launch Pads
*Loosely based on an actual risk scenario 17

18. Step 1a. Refine the Decision
Identification of Alternatives
1a. Identify
Performance • Concise statement of a performance shortfall.
Objectives
1b. Derive • Establish common understanding of performance
Performance Measures constraints.
2. Identify • Document assumptions and scope of decision.
Feasible Alternatives
Risk Analysis of Alternatives
3. Set the Framework
(TBfD)*
Risk-Informed Alternative Selection
4. Analyze
Alternatives &
Document
5. Risk-Normalize
Commitments
6a. Deliberate &
Decide
6b. Select an
Alternative &
Document Scenario for Off-line Servicing & Assembly
18

19. 1b. Criteria for Success
Identification of Alternatives
1a. Identify
Performance
• Identify all Stakeholders and their expectations.
Objectives • Define Performance Objectives.
1b. Derive
Performance • Characterize Performance Measures.
Measures
2. Identify “Show me the money!” is a cool phrase.
Feasible Alternatives For RM, it’s not just about the $$, we must consider all domains!
Risk Analysis of Alternatives
3. Set the Framework
(TBfD)*
Risk-Informed Alternative Selection
4. Analyze
Alternatives &
Document
5. Risk-Normalize
Commitments Program Ground
Ops
6a. Deliberate &
Decide
HQ ? KSC
6b. Select an
Alternative & EPA Public
Document etc.
19

20. 2. Feasible Alternatives
Identification of Alternatives
1a. Identify
Performance • Initially consider a range of solutions – even wild-
Objectives
eyed ideas okay at this point!
1b. Derive
Performance
Measures
• Always include the “Do-Nothing” option.
2. Identify • Use Graded Approach to determine appropriate
Feasible Alternatives
level of rigor needed to refine your list of feasible
Risk Analysis of Alternatives
3. Set the Framework
alternatives.
(TBfD)*
Risk-Informed Alternative Selection
4. Analyze
Alternatives & VAB (existing facility) Stop-n-Go Model (new alternative)
Document
5. Risk-Normalize
Commitments MPPF Pad (existing process)
(existing facility)
6a. Deliberate &
Decide
Vehicle Integration Building (new facility)
6b. Select an
Alternative &
Document
20

21. 3. Set the Framework
Identification of Alternatives
1a. Identify
Performance
• Select “Key” Decision Parameters.
Objectives
– Note: Parameters must apply to ALL alternatives.
1b. Derive
Performance
Measures • Determine reasonable Performance Measures and
2. Identify criteria needed for a decision-maker to commit to
Feasible Alternatives an alternative (minimum threshold).
Risk Analysis of Alternatives
3. Set the Framework
(TBfD)*
• Establish weighted priority (basis) for making a
decision.
Risk-Informed Alternative Selection
4. Analyze
Alternatives &
Document
Launch Availability
5. Risk-Normalize Contingency & Recovery
Commitments “Green Pad”
Life Cycle Cost
6a. Deliberate &
Decide Safety
6b. Select an Flexibility Veh. Orientation
Alternative &
Document
21

22. 4. Analyze Alternatives
Identification of Alternatives
1a. Identify
Performance • Conduct analysis of each alternative’s key
Objectives performance measures.
1b. Derive
Performance
Measures
• Prune “unsuitable” alternatives expeditiously
based on key parameters:
2. Identify
Feasible Alternatives – Reduces analysis cost / effort.
Risk Analysis of Alternatives
– Clarifies options for decision makers.
3. Set the Framework
(TBfD)*
Risk-Informed Alternative Selection
4. Analyze
Alternatives & VAB (existing facility) Stop-n-Go Model (new alternative)
Document
5. Risk-Normalize MPPF
Commitments
Pad (existing process)
(existing facility)
6a. Deliberate &
Decide
Vehicle Integration Building (new facility)
6b. Select an
Alternative &
Document
22

23. Additional RIDM Terminology for Step 5
RIDM
Key
#
Pt
Term "RIDM-Speak" Handbook Translation
reference
Create means to compare “like” perspectives
across alternatives by first establishing a Normalize performance to assess
“common break-point of uncertainty” for each across a range of options
analysis.
• For any measure, allowable uncertainty is
based on a number of factors
For ex. Worst-case threshold
• Set standards used to compare alternatives
at the same level of uncertainty; allocated
Risk Normalized resources represent some point value in an Section Avoid co-mingling analysis uncertainty
6 √ Performance uncertainty range (.01 – .99) for each 3.3.1, with the uncertainty of an alternative;
Commitment alternative. pp.73-78 each option produces its own PDF and
CDF.
• A common “confidence” level would be used
to arrive at a “normalized” value of
Apply the same confidence level (i.e.,
uncertainty for each alternative
risk tolerance) to each alternative for a
• Additional uncertainty considerations depend criterion (criteria can have different risk
on risk tolerance of decision-makers. If risk- tolerance levels)
neutral, an option with uncertain
performance may be preferred if it performs
at the “mean value” of a performance Risk-neutral not likely today!
measure pdf.
Ref: NASA Risk Management Handbook NHBK_2011_3422
23

24. Step 5. Risk Normalize Commitments
Identification of Alternatives
1a. Identify • Consider risk tolerance of the decision-maker and all
Performance
Objectives key stakeholders for each performance measure.
1b. Derive
Performance Measures
2. Identify
Feasible Alternatives
Risk Analysis of Alternatives
3. Set the Framework
(TBfD)* • Determine degree of certainty required for each
performance measure for comparison.
Risk-Informed Alternative Selection
4. Analyze
Alternatives & • Each performance measure should yield a probability
Document
distribution function (pdf) for each alternative.
5. Risk-Normalize
Commitments
Note:
6a. Deliberate &
Decide Pdf – Cdf applies for quantitative analyses; similarly for qualitative
assessments, one should apply the same level of confidence when
6b. Select an evaluating alternatives (constructed scales).
Alternative &
Document
24

25. Risk Normalized Commitments
• Each alternative yields its own PDF and corresponding cumulative
distribution function (CDF).
NOTES:
1. A common “uncertainty” level would be used to arrive at a “normalized” value of
uncertainty for each alternative.
2. Assumed 20% Risk Tolerance = 80% Certainty (use 80% value of CDF)
25

26. Example of Analysis Documented on TBfD Matrix
Step 1 Step 1b Step 2 Step 3
High- Level Objectives Performance Objectives Measures Pad VAB MPPF VIB S&G Weight
Meets CARD  Yes Yes Yes Yes Yes 0%
Launch Availability
Optimal: Exceeds L.A. reqmts No Yes Yes Yes No 7%
Construction/ refurbishmt ($M)  2.5 5 4 20 6 8%
Life Cycle Cost
Operations & Maintenance ($M) 4 2.5 4 1 1 10%
Discovery 1 - Low Risk 3 1 1 1 2 6%
Verification 2 - Mild Risk 2 1 1 1 1 8%
Contingency Capabilities
Repair 3 - Moderate Risk 1 2 1 1 3 3%
Flexibility and Recovery
4 - Potentially Unreliable
Off-nominal (de-servicing) 5 - Unreliable 3 2 3 1 1 2%
Minimize Pad Time***(Clean Pad) Quan (Hr) 106 1 1 1 1 11%
Processing Efficiency
De-couple critical path Quan (Hr) 96 12 24 12 12 2%
Quantity Distance Impact Quan (hrs contact) 3 16 20 20 3 10%
Hypergol Contamination Quan (H/L) 3.2 2.8 2.2 1.6 0.8 10%
Safety
Adequate “clear” Y/N Yes Yes Yes Yes Yes 0%
Minimize Wet Time Quan (Hr) 8 106 106 106 16 6%
Scalability (flight rate) Flights/Yr 52 52 70 100 100 10%
Schedule
- Flexibility Re-Cycle Time (Hr) 48 48 12 12 0 7%
Tally 85 113 161 178 140 100%
Note:
Conceptual only – data & values fabricated for illustrative purposes only.
26

27. Step 6a. Decide & Deliberate
Identification of Alternatives
1a. Identify
Performance • Apply analysis results to TBfD and make
Objectives
recommendation to Decision-Maker.
1b. Derive
Performance • Reconsider TBfD if no clear leader and report
Measures
results to Decision-Maker.
2. Identify
Feasible Alternatives • Reconsider requirements / constraints if no
Risk Analysis of Alternatives
acceptable alternatives.
3. Set the Framework
(TBfD)*
Risk-Informed Alternative Selection
4. Analyze
Alternatives &
Document
5. Risk-Normalize
Commitments
Analysis Findings (for example):
− Recommend New Facility
6a. Deliberate & − Document “Green Pad” Requirement
Decide
− Work to Minimize Wet Time
6b. Select an
Alternative &
Document
27

28. Recap: Risk Management = RIDM + CRM
Identification of Alternatives RIDM: Inform decision-making through
use of quantitative and qualitative risk info
1a. Identify 1b. Derive 2. Identify to establish baseline performance
Performance Performance Feasible requirements for mission support
Objectives Measures Alternatives organizations, programs and projects.
Ref. RIDM Handbook Fig. 11.
RIDM Process Steps (p.30)
CRM:
Risk Analysis of Alternatives Manage risk associated with implementing
baseline performance requirements using a
3. Set the 4. Analyze 5. Risk- systematic and iterative process to
Framework Alternatives & Normalize identify, analyze, plan, track, control, comm
TBfD:* (TBfD)* Document Commitments unicate and document risks.
Technical Basis
for Deliberation;
also known as
(BOE) Basis of
Estimate Risk-Informed Alternative Selection
6a. 6b. Select an Build
Deliberate & Alternative & Performance
Decide Document Requirements
Iterative
28

29. Concluding Remarks
• Uncertainty, evolving objectives, scarce
resources – require an evolution in the
way we approach risk management.
• The key is coupling RIDM with CRM to
evolve Risk Management practices to be
more pro-active vs. simply reactive.
• Aim: Employ an integrative, systemic
approach to yield relevant and timely Always
information for missions and mission consider
support. mission
support
• Doing so keeps NASA Risk Management risks!
practices fresh and relevant to enable risk-
informed decisions.
• Go forth & do good RIDM (and CRM)!
29

30. JSCAC3 Sharon L. Thomas
Sharon.L.Thomas@nasa.gov
(281) 244-7668
JSCNA Scott L. Field, PMP
Scott.Field@nasa.gov
(281) 335-2440
30