The document outlines the management of risk in performance measurement baselines for government and industry programs, identifying four main sources of issues that lead to cost and schedule overruns. It distinguishes between epistemic uncertainty, which stems from a lack of knowledge, and aleatory uncertainty, which arises from natural variances in processes, emphasizing the need for effective risk management strategies. Furthermore, it discusses frameworks for identifying, assessing, and mitigating risks, noting that a thorough understanding of both types of uncertainties is essential for project success.

1. MANAGEMENT OF RISK IN THE PERFORMANCE MEASUREMENT BASELINE
Many large government and industry programs are plagued by cost and schedule overruns and
technical shortfalls. [2] [3] Sources of these issues can be summarized in four notions by Mr.
Bliss. [1]
1. Unrealistic performance expectations with missing Measures of Effectiveness (MOE) and
Measures of Performance (MOP). [6]
2. Unrealistic cost and schedule estimates based on inadequate risk adjusted growth
models.
3. Inadequate assessment of risk and unmitigated exposure to these risks without proper
handling plans.
4. Unanticipated technical issues with alternative plans and solutions to maintain
effectiveness.
All project work operates in the presence of uncertainty. Cost uncertainty, schedule
uncertainty, technical uncertainty that creates these risks.
Risk management is how adults manage projects ‒ Tim Lister
Knowledge of project risk provides information needed to make decisions in the presence of
uncertainty, where predicting future outcomes is part of the project management process.
When we talk about risk we need to have definitions that are shared across the domain. Risk in
federal acquisition programs may not be the same definition as risk in the financial investment
domain.
For projects subject to EIA-748-C, there can be uncertainty about the parameters of a system,
for example ‒ the probability that the propulsion system will not start when asked to. There can
be uncertainty about the incompleteness of our knowledge. This is missing knowledge about
the naturally occurring variances in the underlying processes of the project.
When we say uncertainty, we speak about a future state of an external system that is not fixed
or determined. Uncertainty is related to three aspects of our program management domain:
§ The external world – the activities of the program
§ Our knowledge of this world – the planned and actual behaviors of the program
§ Our perception of this world – the data and information we receive about these behaviors
This paper shows how to normalize the many definitions we encounter in our Earned Value
Management domain.
RISK IS A CONSEQUENCE OF UNCERTAINTY
Uncertainty is the precursor to Risk
Traditional risk analysis focuses on Event Based uncertainty. The probability that something
undesirable will happen in the future. This uncertainty creates a risk that the occurrence will
negatively impact the project. This type of uncertainty is called Epistemic uncertainty or
reducible uncertainty.

2. Epistemic uncertainty comes from the Greek Epistemology ‒ the theory of knowledge.
Epistemology is the investigation of what distinguishes justified belief from opinion. Epistemic
uncertainties are reducible because the source of uncertainty can be reduced with more
information.
The second source of risk comes from Aleatory uncertainty. This is the naturally occurring
variances project’s products and processes. Aleatory uncertainties are irreducible because the
source of the uncertainty is natural randomness.
Managing in the presence of both epistemic and aleatory uncertainty, requires we model of the
relationships between the probabilistic Epistemic uncertainties and the statistical ‒ naturally
occurring ‒ variances produced by the Aleatory uncertainty. Both are present on all projects.
Before proceeding to manage in the presence of these uncertainties, let’s quickly review some
definitions.
§ Aleatory uncertainty ‒ comes from natural variance or the randomness process. Flipping a
coin and predicting either HEADS or TAILS is aleatory uncertainty. This uncertainty is a
random process, it is part of the natural processes of what we are observing.
In Aleatory uncertainty, the observed process is understood but is behaving randomly. This is
randomness in the possible outcomes of the process. Task durations are an example of
aleatory uncertainties. These random processes have an understood variability and a
propagated through the project model ‒ the Integrated Master Schedule.
Aleatory risk is not a lack of knowledge. It is a naturally occurring process variance. We cannot
buy more information to reduce the risk. We have to provide margin to protect the project
from the risk produced by aleatory uncertainty. Schedule Margin to cover the naturally
occurring variances in how long it takes to do the work. Cost Margin to cover the naturally
occurring variances in the price of something we are consuming in our project.
§ Epistemic uncertainty ‒ comes from our lack of knowledge. This lack of knowledge comes
from inadequate understanding of the underlying processes, incomplete knowledge of the
phenomena, or imprecise evaluation of the related characteristics.
Epistemic uncertainty is about the distribution of the possible probability of a future
outcome. For example the probability that a light bulb will burn out, which can be modeled by
a Poisson distribution. The longer you wait the higher the probability the light will fail when
turned on.
It is important to differentiate between two types of uncertainty that create risk on the project,
because they are mitigated in two completely different ways. Epistemic uncertainty and the risk
can be overcome with explicit projects actions. We can fill our gap in the missing knowledge
that causes the uncertainty which creates the risk. We can’t remove the naturally occurring
variances from a process. We can mitigate aleatory uncertainty an it’s resulting risk with margin
developed from historical data using probabilistic models. This is usually done using Monte
Carlo simulations of the underlying processes.
Sources of Project Aleatory Uncertainty
Another source of uncertainty and Risk resulting from uncertainty is the natural variations in
work durations, cost of that work or cost of materials, and technical performance. These

3. natural variations are irreducible, they are Aleatory in nature. Aleatory variability is the natural
randomness in a process.
Sources of Project Epistemic Uncertainty
FINDING AND MANAGING RISK ON THE PROGRAM
Developing an understanding of program risks ‒ reducible and irreducible ‒ is necessary but far
from sufficient to increase the probability of program success.
With the uncertainties classified ‒ aleatory (irreducible) and epistemic (reducible) we need a
process to identify the risks and guidance of where to look for the risk. The naïve notion that
risk is everywhere doesn’t help guide us in finding risks everywhere. There are many guides for
risk management ‒ DOD, DOE, NASA, DHS, ITIL, ISO, NIST, Software Engineering Institute, and
many books and papers. These are shown in the References section.
This paper uses the Software Engineering Mission Risk Diagnostic (MRD) Method Description
[11] Each step in Figure 1 can be traced to the artifacts of the WBS, Integrated Master Plan,
Integrated Master Schedule, and Risk Register when building the Risk Adjusted Performance
Measurement Baseline.
Figure 1 ‒ there are many frameworks for managing risk. The Software Engineering Institute’s Continuous Risk Management
framework has been adopted by several federal agencies as well as commercial risk management frameworks. It contains all
the elements needed to successfully identify, classify, assess, mitigate, monitor, and report risks in the Performance
Measurement Baseline.

4. The remainder of this section will show how to identify risks in the various source documents,
how to categorize those risk into reducible or irreducible, how to place those risks in a Risk
Register, assign Work of Baseline for the Reducible risks and margin for Irreducible risks.

5. Identifying Risk in the Work Breakdown Structure
The Work Breakdown Structure is Paramount – Mr. Gordan Kranz
The starting point for identifying risk is WBS. With the product-oriented family tree composed
of hardware, software, services, data, and facilities, there is possibility of risk in each of the
deliverables. The WBS is family tree results from systems engineering efforts during the
acquisition of a defense materiel item. [8]
With the WBS dictionary shows the hierarchical relationship of the elements and describes each
WBS element and the resources and processes required to produce it. It also provides basic
technical characteristics for the WBS elements and provides a link to the detailed technical
definition documents. The WBS dictionary is routinely revised to incorporate changes and must
reflect the current status of the program throughout the program’s life. [8]
Using the WBS Dictionary, risks are identified for each deliverable and the processes that
produce these deliverables
Identifying Risks in the Integrated Master Plan
Managing in the presence of uncertainty starts with the Integrated Master Plan. The IMP is an
event-based plan consisting of a hierarchy of program events, with each event being supported
by specific accomplishments, and each accomplishment associated with specific criteria to be
satisfied for its completion. [21] The IMP provides the assessment points that measure the
maturity of the product or service against the planned maturity. These measures are in units of
Effectiveness and Performance. This is the only real measure of progress – not the passage of
time or consumption of money.
The IMP starts at proposal, with Program Events, Significant Accomplishments, and
Accomplishment Criteria.
Figure 2 ‒ the Integrated Master Plan provides vertical traceability from the work performed in the Integrated Master Schedule
to the assessment of increasing maturity of the program’s deliverables and the Measures of Effectiveness, Measures of
Performance, and Key Performance Parameters.

6. During program execution, the IMP plays a critical role in definition the measures of increasing
maturity by assessing the increasing maturity of
§ Measures of Effectiveness ‒ which are Operational measures of success closely related to the
achievements of the mission or operational objectives.
§ Measures of Performance ‒ that characterize physical or functional attributes relating to the
system operation, measured or estimated under specific conditions.
§ Key Performance Parameters ‒ that Represent the capabilities and characteristics so
significant that failure to meet them can be cause for reevaluation, reassessing, or
termination of the program.
Identifying Risk in the Integrated Master Schedule
A risk adjusted Integrated Master Schedule (IMS) ‒ from a risk Adjusted Integrated Master Plan
(IMP) is necessary, but far from sufficient. Executing the IMS and assessing increasing maturity
of the deliverables is needed as well. The IMS is an integrated, master schedule containing the
networked, detailed tasks necessary to support the events, accomplishments, and criteria of
the IMP. The IMS should be a logical network-based schedule, based on sound technical
planning, that is directly traceable to the contractor’s cost and schedule reporting instrument
used to address variances. [11]
The mitigation for the irreducible uncertainties requires margin. This margin is shown in the
IMS, in accordance with DID-81861 § 3.6.7.3.
«put Rick’s margin paper here»
BUILDING THE RISK ADJUSTED PERFORMANCE MEASUREMENT BASELINE
Defining Work to Mitigate Reducible Risk
Defining Margin to Mitigate Irreducible Risk
EXECUTING A RISK ADJUSTED PERFORMANCE MEASUREMENT BASELINE
Statusing the Mitigation of Reducible Risk
Statusing the Mitigation of Irreducible Risk
«Rick’s risk burn down charts and words»

7. REFERENCES
Knowledge is of two. We know a subject ourselves, or we know where we can
find information upon it. – Samuel Johnson, 1709 – 1784
[1] Observations from AT&L/PARCA's Root Cause Analyses, Mr. Gary Blisss, March 21st, 2012,
http://www.dtic.mil/ndia/2012annual_psr/BLISS.pdf
[2] Annual Growth of Contract Costs for Major Programs in Development and Early
Production, Dan Davis and Philip S. Anton, Acquisition Policy Analysis Center Performance
Assessments and Root-Cause Analyses Office of the Under Secretary of Defense for
Acquisition, Technology, and Logistics (AT&L) U.S. Department of Defense, March 21,
2016, http://www.acq.osd.mil/fo/docs/Growth-of-Contracted-Costs-21Mar2016.pdf
[3] “Engineered Resilience for Complex Systems as a Predictor for Cost Overruns,” Blake
Roberts, Thomas Mazzuchi, and Shahram Sarkani, Systems Engineering, Vol. 19, No. 2,
2016
[4] “From Nobel Prize to Project Management: Getting Risks Right,” Bent Flyvbjerg, Aalborg
University, Denmark, Project Management Journal, vol. 37, no. 3, August 2006, pp. 5-15
[5] “Picturing the Uncertain World: How to Understand, Communicate, and Control
Uncertainty through Graphical Display,” Howard Wainer, Princeton University Press, 2009
[6] Technical Measurement: A Collaborative Project of PSM, INCOSE, and Industry,
https://acc.dau.mil/adl/en-
US/144086/file/27912/INCOSE%20Technical%20Measurement%20Guide%202005.pdf
[7] Forecasting and Simulating Software Development Projects: Effective Modeling of Kanban
& Scrum Projects using Monte-carlo Simulation, Troy Magennis, Focused Objectives,
https://www.amazon.com/Forecasting-Simulating-Software-Development-
Projects/dp/1466454830
[8] MIL-STD-881-C, https://acc.dau.mil/adl/en-US/482538/file/61223/MIL-
STD%20881C%203%20Oct%2011.pdf
[9] “Management of Risk and its Integration with ITIL,” Herve Doornbos
[10] Integrated Master Plan and Integrated Master Schedule Implementation and Use Guide,
http://www.acq.osd.mil/se/docs/IMP_IMS_Guide_v9.pdf
[11] “Mission Risk Diagnostics (MRD) Method Description,” Christopher Alberts and Audrey
Dorofee, February 2012, CMU/SEI-2012-TN-005,
http://resources.sei.cmu.edu/asset_files/TechnicalNote/2012_004_001_15431.pdf
[12] NASA Risk Management Handbook, NASA/SP-2011-3422
[13] The NASA Risk Management Page, with current and archive NASA risk management
guidance http://www.hq.nasa.gov/office/codeq/risk/index.htm
[14] DOE G 413.3-7A, Risk Management Guide, https://www.directives.doe.gov/directives-
documents/400-series/0413.3-EGuide-07a
[15] Risk Management Fundamentals, https://www.dhs.gov/xlibrary/assets/rma-risk-
management-fundamentals.pdf
[16] EVM’s Potential for Enabling Effective Integrated Cost-Risk Management by David R.
Graham, http://www.iceaaonline.com/ready/wp-content/uploads/2016/06/MS03-paper-
EVM-Potential-Enabling.pdf

8. [17] Advances Cost Risk Analysis, Greg Hogan and Christian Smart, ICEAA June 2105, San Diego
California.
[18] COBIT 5 for Risk, http://www.isaca.org/cobit/pages/risk-product-page.aspx
[19] ISO 31000 Risk Management, http://www.iso.org/iso/home/standards/iso31000.htm
[20] “DOD Risk Management Guide V7,” http://www.acq.osd.mil/se/docs/DoD-Risk-Mgt-
Guide-v7-interim-Dec2014.pdf
[21] “Department of Defense Risk, Issue, and Opportunity Management Guide for Defense
Acquisition Programs,” Office of the Deputy Assistant Secretary of Defense for Systems
Engineering, http://www.acq.osd.mil/se/docs/RIO-Guide-Jun2015.pdf
[22] “Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide,”
http://www.acq.osd.mil/se/docs/IMP_IMS_Guide_v9.pdf
[23] “Defense Acquisition Guidebook (DAG),” https://dag.dau.mil/Pages/Default.aspx
[24] Project Risk Management: Process, Techniques, and Insights, 2nd Edition, Chris Chapman
and Stephen Ward, John Wiley & Sons, 2003.
[25] Technical Risk Management, Jack V. Michaels, 1996, Prentice-Hall
[26] Effective Opportunity Management of Projects: Exploiting Positive Risk, David Hillson,
2004, Taylor & Francis
[27] “Joint Agency Cost Schedule Risk and Uncertainty Handbook,” 12 March 2014,
https://www.ncca.navy.mil/tools/csruh/JA_CSRUH_16Sep2014.pdf
[28] Effective Risk Management: Some Keys to Success, 2nd Edition, Edmund Conrow, AIAA
Press, 2003.
[29] “Integrating Risk Management with Earned Value Management,”
http://www.ndia.org/Divisions/Divisions/Procurement/Documents/Content/ContentGrou
ps/Divisions1/Procurement/Integrating_RM_with_EVM.pdf
[30] “Combining Earned Value Management and Risk Management to Create Synergy,” Dr.
David Hillson, http://risk-doctor.com/pdf-files/cev-a1004.pdf
[31] “Interfacing Risk and Earned Value Management,” Association for Project Management,
2008. https://pbcdata.files.wordpress.com/2011/03/interfacing-risk-and-earned-value-
management.pdf
[32] DI-MGMT-81861 Integrated Program Management Report (IPMR)
[33] “Department of Defense Earned Value Management Interpretation Guide,” OUSD AT&L
(PARCA), February 18, 2015.
[34] “Taxonomy-Based Risk Identification,” Technical Report, CMU/SEI-93-TR-6, ESC-TR-93-
183, http://www.sei.cmu.edu/reports/93tr006.pdf
[35] “From Nobel Prize to Project Management: Getting Risks Right,” Bent Flyvbjerg, University
of Oxford, Said Business School, April 1, 2006,
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2238013
[36] “A Guide to Managing Programs Using Predictive Measures,”
http://www.ndia.org/Divisions/Divisions/IPMD/Documents/WorkingGroups/Predictive_M
easures_Guide_IPMD_Review_Copy.pdf
[37] Cost Risk and Uncertainty Analysis Guidebook, https://acc.dau.mil/adl/en-
US/316093/file/46243/AF_Cost_Risk_and_Uncertainty_Guidebook_Jul07.pdf

9. [38] “LANL’s Development of Schedule Contingency Based on Probabilistic Risk Results,”
Proceedings of the Project Management Institute Annual Seminars & Symposium
November 1–10, 2001, Nashville, Tennessee
[39] “Integration Of Risk And Opportunity Thinking In Projects,” Kalle Kähkönen, VTT Building
and Transport, Finland, Presented at the Fourth European Project Management
Conference, PMI Europe 2001, London UK, 6-7, June 2001.
[40] “Probabilistic Schedule Reserve Allocation,”
http://www.nasa.gov/sites/default/files/files/Probabilistic_Schedule_Allocation_2013_TA
GGED.pdf
[41] “Depicting Schedule Margin in Integrated Master Schedules,”
http://www.ndia.org/Divisions/Divisions/IPMD/Documents/WhitePapers/NDIASchedule
MarginWhitePaperFinal-2010%282%29.pdf
[42] “Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide,” The
Office of the Secretary of Defense (OSD) Acquisition, Technology, & Logistics (AT&L),
www.acq.osd.mil/se/docs/IMP_IMS_Guide_v9.pdf
[43] “Prediction of project outcomes: The application of statistical methods to Earned Value
Management and Earned Schedule Performance Indexed,” Walt Lipke, Ofer Zwikal, Kym
Henderson, and Frank Anbari, International Journal of Project Management, 27 (2009),
400-407.
[44] Integrating Risk Management with Earned Value Management, NDIA,
http://www.ndia.org/Divisions/Divisions/Procurement/Documents/Content/ContentGrou
ps/Divisions1/Procurement/Integrating_RM_with_EVM.pdf
[45] “Technical Performance Measurement, Earned Value, and Risk Management: An
Integrated Diagnostic Tool for Program Management,” Commander N. D. Pisano, SC, USN,
Program Executive Office for Air ASW, Assault, and Special Mission Programs (PEO(A)),
www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA404646
[46] “Monitoring Risk Response Actions for Effective Project Risk Management,” Edouard
Kujawski and Diana Angelis, Systems Engineering, Vol. 13, No. 4, 2010
[47] “Transforming project risk management into project uncertainty management,” Stephen
Ward and Chris Chapman, International Journal of Project Management, 21 (2003) 97–
105, http://web.nchu.edu.tw/pweb/users/arborfish/lesson/10490.pdf
[48] GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing
Capital Program Costs, March 2009, GAO-09-3SP, www.gao.gov/assets/600/591240.pdf
[49] NNSA Cost Estimating Guide, 50.005, http://www.efcog.org/wg/pm_ce/docs/HQ-406529-
v1-Signed_NNSA_Cost_Estimating_Guide_50_005.pdf
[50] “A Guide to Managing Programs Using Predictive Measures,” September 17, 2014,
National Defense Industry Association, Integrated Program Management Division,
http://www.ndia.org/Divisions/Divisions/IPMD/Documents/ComplementsANSI/NDIA_IPM
D_Predictive_Measures_Guide_Sept172014(a).pdf
[51] “Dicing with the unknown,” Tony O’Hagan, Significance, September 2004.
http://www.stat.columbia.edu/~gelman/stuff_for_blog/ohagan.pdf
[52]