NASA uses two complementary processes for risk management: risk-informed decision making (RIDM) and continuous risk management (CRM). RIDM emphasizes using risk analysis to make risk-informed decisions across dimensions like safety, cost, and schedule. CRM manages risks associated with implementation and uses risk statements to document risks across multiple dimensions. Current risk analysis methods often fail to provide a complete risk picture by only considering risks one dimension at a time. MRisk addresses this by analyzing risks across all dimensions simultaneously using anchor points and Mahalanobis distance, providing a more objective and accurate assessment of total project risk.

1. Multidimensional RISK
RISK INTEGRATED WITH SCHEDULE, COST, PERFORMANCE, AND
ANYTHING ELSE YOU CAN THINK OF
NASA PM Challenge
Feb 2011
0

2. NASA Uses Two Complementary Processes For Risk Management
 Risk-Informed Decision Making (RIDM)
– Emphasizes the proper use of risk analysis to make risk-informed decisions that impact
all risk dimensions including safety, technical, cost, schedule, etc…
– Acknowledges the role that subject matter experts (SMEs) play in decisions. Emphasizes
that the cumulative wisdom provided of SMEs is essential for integrating technical and
nontechnical factors to produce sound decisions due to the availability of technical data
and the complexity of missions
– Source: NASA/SP-2010-576 NASA Risk-Informed Decision Making Handbook
 Continuous Risk Management (CRM)
– To manage those risks associated with the performance levels that drove selection of a
particular alternative (from RIDM)
– A systematic and iterative process that efficiently identifies, analyzes, plans, tracks,
controls, and communicates and documents risks associated with implementation of
designs, plans, and processes
– Source: NPR 8000.4A Agency Risk Management Procedural Requirements
1

3. RIDM Selects Alternatives & CRM Addresses The Implementation
Of Alternatives
Risk-Informed Decision Making (RIDM) Continuous Risk Management (CRM)
Identification of Alternatives
Identify Decision Alternatives (Recognizing Opportunities) in the
Context of Objectives
Risk Analysis of Alternatives
Risk Analysis (Integrated Perspective) and Development of the
Technical Basis for Deliberation
Risk-Informed Alternative Selection
Deliberate and Select an Alternative and Associated Performance
Commitments Informed by (not solely based on) Risk Analysis
* Source: NASA/SP-2010-576 NASA Risk-Informed Decision Making Handbook
2

4. CRM Uses The Identify Step To Document Risks In The Form of
Risk Statements
Risk Statements have 3 distinct elements
1. Scenario
– A sequence of credible events that specifies the evolution of a system or process from a given
state to a future state. In the context of risk management, scenarios are used to identify the
ways in which a system or process in its current state can evolve to an undesirable state
2. Likelihood
– Probability of occurrence
3. Consequence
– The possible negative outcomes of the current conditions that are creating uncertainty
there is a
Given SCENARIO LIKELIHOOD CONSEQUENCE will occur
that,
3

5. Risk Is Typically Measured As Likelihood Times Consequence
Consequence
Likelihood
RISKS Estimation of
Estimation of Definitions Definitions the impact to the
the likelihood program if the
that the risk risk event
event will occur occurs
Likelihood x Consequence
Quantitative Risk Score
4.79 10.73 29.30 48.94
4

6. The Identify Step of CRM Documents Risk In Multiple Dimensions
To Get A Complete Risk Picture
People
Safety Environment Schedule
On-Orbit operations risk
Configuration
Management
Technical
Cost
5

7. Managers Use Customized Criteria To Bin Risks Into A Risk Matrix
RISK MATRIX
Likelihood Rating 5
Level Probability
LIKELIHOOD
4
5 Very Likely Expected to happen
3
Could happen. Controls have significant limitations or
4 Likely uncertainties. 2
Could happen. Controls exist, with some limitations or
3 Possible uncertainties.
1
Not expected to happen. Controls have minor limitations 1 2 3 4 5
2 Unlikely or uncertainties.
CONSEQUENCES
Highly Extremely remote possibility that it will happen. Strong
1 Unlikely controls in place.
CONSEQUENCE 1 Very Low 2 Low 3 Moderate 4 High 5 Very High
Technical Negligible or no impact to Minor impact to achievement Some impact to achievement of Moderate impact to Major impact to achievement of
achievement of Subsonic of Subsonic Transport System Subsonic Transport System achievement of Subsonic Subsonic Transport System
Transport System Level Level Metrics, Technical Level Metrics, Technical Transport System Level Metrics, Level Metrics, Technical
Metrics, Technical Deliverables, Technology Deliverables, Technology Technical Deliverables, Deliverables, Technology
Deliverables, Technology Maturation, or KPP Goals Maturation, or KPP Goals Technology Maturation, or KPP Maturation, or KPP Goals
Maturation, or KPP Goals Goals
Schedule Level 1 Milestone(s): Level 1 Milestone(s):
Level 2 Milestone(s): Level 2 Milestone(s): Level 1 Milestone(s):
≤1 month impact > 1 month impact
< 1 month impact ≥ 1 month impact > 2 month impact
Level 2 Milestone(s): ≤ 2 Level 2 Milestone(s):
month impact > 2 month impact
Level 3,4 Milestone(s): ≤ 1 Level 3,4 Milestone(s): ≤ 2 Level 2 Milestone(s):
Level 3,4 Milestone(s): Level 3,4 Milestone(s):
month impact month impact ≥ 3 month impact
≤ 3 month impact >3 month impact
Cost Between 0% and 5% Between 5% and 10% Between 10% and 15% increase Between 15% and 20% increase Greater than 20% increase over
increase over allocated increase over allocated budget over allocated budget (Sub- over allocated budget (Sub- that allocated budget (Sub-
budget (Sub-Project, (Sub-Project, Element or Task Project, Element or Task level) Project, Element or Task level) Project, Element or Task level)
Element or Task level) level)
Safety Negligible or no impact Could cause the need for only May cause minor injury or May cause severe injury or May cause death or permanently
minor first aid treatment occupational illness or minor occupational illness or major disabling injury or destruction of
property damage property damage property
6 6

8. The Risk Matrix Provides The Framework For CRM Risk Analysis
Effective analysis makes it possible to
RISK MATRIX move total project risk from red to
5 green
But how do you know you are focused
LIKELIHOOD
4
on the right project risks?
3 Focusing on the wrong risks may keep
total project risk in the red?
2
1
1 2 3 4 5
CONSEQUENCES
7

9. Current Risk Matrix Development Methods Often Fail To Give A
Complete Risk Picture
Notional Representation Of Risks In Three Dimensions
Cost Risk Schedule Risk Performance Risk
5 5 5
LIKELIHOOD
LIKELIHOOD
LIKELIHOOD
4 4 4
3 3 3
2 2 2
1 1 1
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
CONSEQUENCES CONSEQUENCES CONSEQUENCES
=pt1
=pt2
 Why are we looking at only one dimension at a time? =pt3
 Should we call pt3(3,3,3) a Cost Risk, a Schedule Risk, or a Performance Risk?
 Is pt2(1,4,1) more risky than the other points just because it has a high schedule severity?
 Is pt1(3,2,3) just as risky as pt3(3,3,3)?
 What if we have risk across four dimensions? Or five? Or Six?
 How do we know we are focusing on the right risks?
8

10. MRisk Makes Use Of Anchor Points And Multidimensional-Distance
Measure To Determine Total Risk
Cost Risk Schedule Risk Performance Risk
5 5 5
LIKELIHOOD
LIKELIHOOD
LIKELIHOOD
4 4 4
3 3 3
2 2 2
1 1 1
1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
min (1,1,1) max (5,5,5)
CONSEQUENCES CONSEQUENCES CONSEQUENCES
dmin
d =
 The anchor points (1,1,1) and (5,5,5) come from our definition dmin + dmax
of the consequence scale
 Distance for each point is defined by the distance of that point
from the minimum over the sum of the distance from the
minimum and the maximum
Consequence Scale
 The distance value explains the precise consequence for each 100
90
80
risk regardless of the number of dimensions 70
60
 The greater the distance the greater the consequence and vice 50
40
versa 30
20
 This procedure is scalable to infinite dimensions of 10
0
consequence, i.e. (1,1,…,1n) (5,5,…,5n) L M1 M2 M3
Low Medium
H1 H2
High
H3 C1 C2 C3
Critical
9

11. Anchor Points & Mahalanobis Distance Make Risk Analysis
Objective & Logically Consistent
 The anchor points make it possible for us to know relative risk
– Anchor points allow us to make the distinction between a (3,3,3) and a (4,2,2)
– A cost consequence of 3, schedule consequence of 3, and safety consequence of 3 has a
distinct distance away from no consequence (1,1,1) and disaster (5,5,5)
 Mahalanobis Distance keeps decision makers consistent in their thinking. By calculating risk
based on the relationship between costs, schedule, safety, etc… MRisk identifies when
violations of known relationships occur in the risk ranking process
– For example, cost and schedule have a known relationship in the PM world
Schedule Cost
Scope
10

12. MRisk Provides A Complete Risk Picture
 MRisk addresses several shortcomings in the current methods
1. MRisk deals with all of the dimensions of Risk simultaneously to provide a complete risk
picture
2. MRisk makes risk analysis objective and consistent with SME judgment
3. MRisk provides more advanced statistical algorithms to Risk Management without changing
the current processes or products
RISK MATRIX
5
LIKELIHOOD
Schedule
4
Cost
3
2
1
Scope
1 2 3 4 5
CONSEQUENCES
11

13. Mahalanobis Distance Mapping Tells Us How Far Each Risk Is
From All Of The Other Risks, Thus Highlighting Outliers
Notional Data Set From Risk Scoring
5
Point Sched Cost Perf …Dimn
LIKELIHOOD
4
1 4 3 3 0 3
2
2 2 1 5 3
1
…
…
…
…
…
1 2 3 4 5 Outlier
m 2 4 1 4 CONSEQUENCES Typical Point
 Using traditional distance measures the outlier point in the above
dmin = (x-xmin)S-1(x-xmin)
scenario could be masked by its proximity to the other points
-1
 Mahalanobis distance highlights the point as an outlier because of its dmax = (x-xmax)S (x-xmax)
relative distance away from the group where
 Mahalanobis distance accounts for the relationship of each risk to S-1 is the Inv(Covariance Matrix)
another and highlights the risks that are uncorrelated, thus detecting xmin = [1,1,1] xmax = [5,5,5]
extreme risks more efficiently
12

14. Mahalanobis Distance Is Based On The Interdependencies Of
Dimensions
 Consider two, random variables X and Y that consist of risk observations for some project or
program
 Those observations will have a variance and covariance
 Any set of random variables will have a Variance-Covariance matrix
Obs1 Obs1
Obs2 Obs2
Obs3 Obs3
Obsn Obsn
13

15. Two Dimensional Mahalanobis Distance Example
 Consider again our two, random variables X and Y that consist of risk observations for some
project or program with the derived variance-covariance matrix
 The distance between two points in XY-plane depends on the inverse of the variance-
covariance matrix
 It’s simple to expand this case to Schedule Risks(X) vs Cost Risks(Y) vs Technical Risks(Z) or
any other type of risk comparison
5
4
Y0=(2,4)
3
Y
2 X0=(2,1)
1
1 2 3 4 5
X
14

16. Legacy Methods By Contrast Assume Independence Of The
Dimensions Of Risk
 Consider again our two, random variables X and Y that consist of risk observations for some
project or program with the derived variance-covariance matrix
 In Euclidean Measure the distance between two points in XY-plane depends on the inverse of
the Identity matrix
5
4
Y0=(2,4)
3
Y
2 X0=(2,1)
1
1 2 3 4 5
X
15

17. The MRisk Metric Calculates Distance While Accounting For The
Point To Point Relationship
 Mahalanobis D2 is a multidimensional version of
a z-score. It measures the distance of a case
from the centroid (multidimensional mean) of a
distribution, given the covariance
(multidimensional variance) of the distribution.
 A case is a multivariate outlier if the probability
associated with its D2 is 0.001 or less. D2
follows a chi-square distribution with degrees of
freedom equal to the number of variables
included in the calculation.
 Mahalanobis' distance identifies observations
which lie far away from the center of the data
cloud, giving less weight to variables with large
variances or to groups of highly correlated
variables (Joliffe, 1986).
 This distance has advantages to other distance
measures like the Euclidean distance which
ignores the covariance structure and thus treats
all variables equally
16

18. Case In Point: Multiple Risks
 Consider a risk scoring session involving
5 risks
 SMEs vote on the probability and
consequence (1,5) for five events across
three dimensions: Performance, Cost, &
Schedule
 The score for each event is recorded in
the table below
Event Prob Perf Cost Sched
1 2 1 2 1
2 4 4 4 3
3 4 3 4 5
4 3 5 1 3
5 4 4 2 1
17

19. Using MRisk All The Events Fit Onto One Scale
Event Prob Perf Cost Sched Event Prob dmin dmax d dscaled
1 2 1 2 1 1 2 0.75 14.00 0.05 1.20
2 4 4 4 3 2 4 3.69 1.59 0.70 3.79
3 4 3 4 5 3 4 2.85 4.37 0.39 2.58
4 3 5 1 3 4 3 2.02 9.99 0.17 1.67
5 4 4 2 1 5 4 2.46 7.00 0.26 2.04
 MRisk answers the question regarding
Event 1 Event 3 Event 2 highest project risk
 A (4,4,3) is more consequential than a
Event 4 Event 5 (3,4,5) or a (5,1,3)
 The current methods would have us
focus our attention on the (3,4,5) and
max (5,5,5) the (5,1,3) despite the fact that they
min (1,1,1)
are not the most consequential
18

20. MRisk Provides A Clear Picture Of The Risk Profile Regardless Of
The Number Of Dimensions Involved
Event Prob Perf Cost Sched dscaled
5
1 2 1 2 1 1.20
2 4 4 4 3 3.79
3 4 3 4 5 2.58
4
LIKELIHOOD
4 3 5 1 3 1.67
5 4 4 2 1 2.04 3
2
1
1 2 3 4 5
CONSEQUENCES
19

21. Traditional Multivariate Methods Like Euclidean Distance May Not
Be As Clear Because They Don’t Consider Relationships
Event Prob Perf Cost Sched Escaled
5
1 2 1 2 1 1.10
2 4 4 4 3 4.14
3 4 3 4 5 4.41
4
LIKELIHOOD
4 3 5 1 3 3.00
5 4 4 2 1 2.11 3
2
Lumping on
severity despite
differences 1
1 2 3 4 5
Possible collusion
of extreme risks
CONSEQUENCES
20

22. MRisk Deals With Several Shortcomings In Risk Analysis
 Just because we cannot visualize risk in multiple dimensions doesn’t mean it’s not there. We
all realize that Risk Management is a multi-dimensional problem that requires a multi-
dimensional solution.
 MRisk does not seek to change Risk Management from its current practices and procedures. It
just revolutionizes Risk Analysis.
 MRisk does not require any change to current data collection techniques for implementation
 MRisk takes the data from the current risk methods and allows for interpretation of risks
through a multidimensional lens
 The use of Mahalanobis Distance as a measure of consequence takes into account the
relationships that risk events have across dimensions, i.e. cost, schedule, etc…
– Since we know cost relates to schedule, schedule relates to performance, performance
relates to safety, etc… MRisk is most appropriate for measuring risk as it emphasizes the
relationships among risks to calculate distance
21