EAI-748-C asks us to objectively assess accomplishments at the work performance level. As well §3.8 of 748-C tells us Earned Value is a direct measurement of the quantity of work accomplished. The quality and technical content of work is controlled by other processes. To provide visibility to integrated cost, schedule, and technical performance, we need more than CPI and SPI. We need measures of increasing technical performance.

1. 5745 1
Assessing Project Performance Beyond CPI/SPI
Informing Programmatic Performance (BCWP) with Technical
Performance to Produce a Credible Estimate At Completion
Glen B. Alleman, Thomas J. Coonce, and Rick A. Price
INTRODUCTION
EAI-748-C asks us to “objectively assess accomplishments at the work performance
level.” As well §3.8 of 748-C tells us “Earned Value is a direct measurement of the
quantity of work accomplished. The quality and technical content of work performed is
controlled by other processes.”
Without connecting the technical and quality measures to Earned Value, CPI and SPI
provide little in the way of assuring the delivered products will actually perform as
needed. To provide visibility to integrated cost, schedule, and technical performance,
we need more than CPI and SPI. We need measures of the increasing technical maturity
of the project’s deliverables in units of measure meaningful to the decision makers.
Those units include Effectiveness, Performance, all the …ilities, and risk reduction. 1
TOP LEVEL PROCESSFOR PROGRAM SUCCESS
The elements in Figure 1 are the basis of a credible Performance Measurement Baseline
(PMB) with Table 1 describing the management processes needed to increase the
probability of program success using these elements.
Table 1 – Steps to building a risk adjustedPerformance Measurement Baseline with Management Reserve.
Step Outcome
WBS  Using SOW, SOO, ConOps, and other program documents, developWBS for deliverables and work
processes that produce the deliverables.
 Develop WBS Dictionarydescribing the criteria for the successful deliveryof the outcomes.
IMP  Develop Integrated Master Plan, showing how each deliverable in the WBS will move through the
maturation process at each Program Event.
 Define Measures of Effectiveness (MOE) at each Accomplishment (SA)
 Define Measures of Performance (MOP) at each Criteria (AC)
Risk  For each keydeliverable in the WBS, identifyreducible risks, probabilityof occurrence, handling
plans performed on Baseline, and residual risks.
 Place this risk reductionwork in the Integrated Master Schedule (IMS).
 For risks without handling plans, place budget in Management Reserve (MR) identified for that risk.
IMS  Arrange Work Packages and their Tasks in a logical network.
 Define the exit criteria for each Work Package to assess the plannedTechnical Performance of the
deliverable.
 Define exit criteria for each Work Package using Physical Percent Complete mechanism to inform
BCWP with TPM, MOP, MOE, and Risk Reduction assessments .
Margin  For irreducible risks in the IMS, use Reference Classes for Monte Carlo Simulation of Most Likely
duration to calculate schedule margin.
 Assign schedule margin infront of key deliverables.
Baseline  Using risk adjustedIMS, containing reducible risk and explicit schedule margin tasks, establish the
Performance Measurement Baseline and Management Reserve .
1 The term …ilities refers to maintainability, reliability, serviceability, operability, andtestability. [4]

2. 5745 2
THE SITUATION
We’re working ACAT1/ACAT2 programs for the Department of Defense using JCIDS
(Joint Capabilities and Development Systems) the governance paradigm defined in DODI
5000.02.
The JCIDS Capabilities Based Planning paradigm tells us to define what Done looks like,
so we need to start by measuring progress toward delivered Capabilities. Requirements
elicitation is part of the process, but programs shouldn’t start with requirements, but
with an assessment of the Capabilities Gaps. While this approach may seem overly
formal, the notion of defining what capabilities are needed for success is the basis of
defining what Done looks like. We’ll use this definition, so we’ll recognize it when it
arrives.
With the definition of Done, we can define the processes for incrementally assessing of
our deliverables along the path to Done. Earned Value Management is one tool to
perform these measurements of progress to plan. But like it says in EAI-718-C, Earned
Value – left to itself – is a measure of quantity. We need measures of quality and many
other …ilities that describe the effectiveness and performance of the product in order to
inform our Earned Value measures of the Estimate to Complete (ETC) and Estimate at
Completion (EAC).
Forecasting unanticipated EAC growth is where Earned Value Management most
contributes to increasing the probability of program success.
We need to integrate other measures with the Earned Value assessment process. We
can start this by recognizing that the Budgeted Cost of Work Performed (BCWP) is a
measure of the efficacy of our dollar.
Earned Value is actually the measure of “earned budget.” Did we “earn” our planned
budget? Did we get our money’s worth? We only know the answer if we measure
“Value” in units other than money. This can be effectiveness of the solution, the
performance of the solution, the technical performance of the products? The fulfillment
of the mission. The maintainability, reliability, serviceability, and other …ilities are
assessments of Value earned, not described by CPI and SPI from the simple calculation
of BCWP.
THE IMBALANCE
Without the connection between the technical plan and programmatic plan – cost and
schedule – the program performance assessment cannot be credible. Of course BCWP
represents the Earned Value, but the Gold Card and other guidance doesn’t explicitly
state how to calculate this number. BCWP is a variable without directions for its
creation. BCWS and ACWP have clear direct on how to assign values to them, but not
BCWP.
The common advice is to determine percent done and then multiply that with BCWS to
produce BCWP. But percent done of what outcome? What units of measure are use to
calculate percent complete?

3. 5745 3
It’s this gap between just assigning a value to BCWP and informing BCWP with tangible
evidence of progress to plan, in units of measure meaningful to the decision makers,
that is the goal of this paper.
RESTORING THE BALANCE
If we define the expected Effectiveness, Technical Performance, …illities, or level of risk
at a specific time in the program, then measure the actual Effectiveness, Technical
Performance, …illities, or Risk, and compare it to the planned values of these measures,
we can determine the physical percent complete of any deliverable in the WBS.
Using the steps in Table 1, the elements of Figure 1 can be developed. The connection
between each element assures the Technical Plan and the Programmatic Plan are
integrated with units of technical performance used to inform BCWP.
Figure 1 – Programmatic elements needed to increase the probability of program success. Connecting the dots
between Measures of Effectiveness, Measures of Performance, and Technical Performance Measures needed to
inform BCWP provides leading indicators of program performance needed to take corrective actions to Keep The
Program GREEN.
The remainder of this paper describes the steps needed to establish a credible PMB
containing event-based risk reduction activities and the schedule margin for irreducible
risk, needed to assure a high probability of success for cost, schedule, and technical
performance.

4. 5745 4
THE SOLUTION
Starting with the processes in Table 1, using the elements in Figure 1, assures the
Technical Plan and the Programmatic Plan are integrated from day one. The culture of
integrating Systems Engineering – the source of MOE, MOP, TPM, the …ilities, and risk –
with Earned Value Management must be guided manually. It is not a natural affinity.
One speaks in technical terms of machinery, software systems, bent metal, and
electronic devices. The other speaks in terms of Dollars.
The solution to making this connection naturally is through policy guidance and working
examples. The step-by-step process of establishing a credible PMB is shown in the
remainder of this paper.
Measuresof TechnicalProgress Startin the IMP
The Program Events, the Significant Accomplishments, and the Accomplishment Criteria
form the elements of the IMP. Measures of Effectiveness (MOE) for the program are
derived from the JCIDS process and are reflected in the IMP’s Significant
Accomplishments (SA). The Measures of Performance (MOP) are derived from the MOEs
and are reflected in the IMP’s Accomplishment Criteria (AC).
These measures assess the physical percent complete of the deliverable used to inform
Earned Value (BCWP) for reporting programmatic performance. With this Physical
Percent Complete measure, the EVM indices can then reflect the cost, schedule, and
technical performance of the program.
 Measures of Effectiveness (MOE) – are operational measures of success closely
related to the achievements of the mission or operational objectives evaluated in
the operational environment, under a specific set of conditions.
 Measures of Performance (MOP) – characterize physical or functional attributes
relating to the system operation, measured or estimated under specific conditions.
 Key Performance Parameters (KPP) – represent capabilities and characteristics
significant that failure to meet them can be cause for reevaluation, reassessing, or
termination of the program.
 Technical Performance Measures (TPM) – are attributes that determine how well a
system or system element is satisfying or expected to satisfy a technical
requirement or goal.
Continuous verification of actual versus anticipated achievement of a selected technical
parameter confirms progress and identifies variances that might jeopardize meeting a
higher-level end product requirement. Assessed values falling outside established
tolerances indicate the need for management attention and corrective action.
A well thought out TPM program provides early warning of technical problems, supports
assessments of the extent to which operational requirements will be met, and assesses
the impacts of proposed changes made to lower-level elements in the system hierarchy

5. 5745 5
on system performance. 2 With this estimate, the programmatic performance of the
program can be informed in way not available with CPI and SPI alone.
Technical Performance Measurement (TPM), as defined in industry standard, EIA-632,
involves estimating the future value of a key technical performance parameter of the
higher-level end product under development, based on current assessments of products
lower in the system structure. Continuous verification of actual versus anticipated
achievement for selected technical parameters confirms progress and identifies
variances that might jeopardize meeting a higher-level end product requirement.
Assessed values falling outside established tolerances indicate the need for
management attention and corrective action.
Risk ManagementStartsAt The IMP
Throughout all product definition processes, technical and programmatic risks
assessment is performed. These risks are placed in the Risk Register with uncertainties.
Uncertainty comes in two forms: 3
 Aleatory – These are Naturally occurring Variances in the underlying processes of
the program. These are variances in work duration, cost, technical performance. We
can state the probability range of these variances as a stochastic variability from the
natural randomness of the process and are characterized by a probability density
function (PDF) for their range and frequency, and therefore are irreducible.
 Epistemic – These are Event based uncertainties, where there is a probability that
something will happen in the future. We can state this probability of an event, and
do something about reducing this probability of occurrence. These are (subjective or
probabilistic) uncertainties are event-based probabilities, are knowledge-based, and
are reducible by further gathering of knowledge.
Structure of the IMP
Figure 2 shows the structure of the Integrated Master Plan (IMP), of the Program
Events, Significant Accomplishments, and Accomplishment Criteria. This structure builds
the assessment of the maturity of the deliverables to assure that the technical
performance of the deliverables meet the planned technical, performance, and
effectiveness needs to fulfill the capabilities needed for the mission or business goals.
With this structure, Earned Value Management measures can be connected to the Work
Packages defined in the Integrated Master Schedule (IMS) to assess technical
performance in ways not available with CPI and SPI. The Program Manager now has
leading indicators of the program success through the MOEs defined by the Significant
Accomplishments and MOPs defined by the Accomplishment Criteria, each assessed for
compliance with plan at the Program Event.
2 https://dap.dau.mil/acquipedia/Pages/ArticleDetails.aspx?aid=7c1d9528-4a9e-4c3a-8f9e-6e0ff93b6ccb
3 “On Numerical Representation of Aleatory and Epistemic Uncertainty,” Hans Schjær-Jacobsen,
Proceedings of 9th International Conference of Structural Dynamics, EURODYN 2014.

6. 5745 6
This vertical connectivity, described in the IMP, is supported by the horizontal
traceability of Work Packages in the Integrated Master Schedule.
During actual program execution, the IMP and IMS provide visibility to the program’s
performance for both the government and the contractor. When integrated with the
Earned Value Management System (EVMS), the IMP and IMS enable the program’s
management to:
 Identify and assess actual progress versus the planned progress.
 Monitor the program critical path and help develop workarounds to problem areas
 Assess program maturity.
 Assess the status of risk management activities based on the inclusion of the
program risk mitigation activities in the IMP and IMS.
 Assess the progress on selected Key Performance Parameters (KPPs) and Technical
Performance Measures (TPMs).
 Provide an objective, quantitative basis for the contractor’s performance
assessment rating and award fee.
 Help develop and support “what-if” exercises, and to identify and assess candidate
problem workarounds.
 Provide better insight into potential follow-on efforts that were not part of the
original contract award.
IMP and IMS Relationship
The IMP is an event-based plan demonstrating the maturation of the development of
the product as it progresses through a disciplined systems engineering process. The IMP
events are not tied to calendar dates. Each event is completed when its supporting
Accomplishments are completed and when this is evidenced by the satisfaction of the
Criteria supporting each of those accomplishments. 4
The IMP is usually placed on contract and becomes the baseline execution plan for the
program or project. Although detailed, the IMP is a top-level document in comparison
with the IMS.
The IMS flows directly from the IMP and supplements it with additional levels of detail.
The IMS incorporates all of the IMP Events, Accomplishments, and Criteria through
Work Packages and detailed Tasks to support the IMP Criteria. This network of
integrated tasks creates and calendar-based schedule that is the IMS defined to a level
of detail for day-to-day execution of the program.
4 Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide, Version 0.9, August
15, 2009,OUSD (AT&L) Defense Systems, Systems Engineering, Enterprise Development (OUSD(AT&L)
DS/SE/ED).

7. 5745 7
Figure 2 – The Integrated Master Plan defines the increasing maturity of each key deliverable assessed at Program
Events. Significant Accomplishments are entry criteria for the maturity assessment. Accomplishment Criteria are
measure substantiating the maturitylevel of the work products producedbyWork Packagesinthe IMS.
ASSEMBLING THE PARTS NEEDED TO CONNECT THE DOTS
In order to inform Earned Value (BCWP) we need information about the performance of
the program beyond just cost and schedule performance. We need information about
how the program is progressing toward delivering the needed capabilities.
TECHNICAL MEASURESUSED ARE TO INFORM EARNED VALUE
Technical Performance
Measuresof Effectiveness
Measuresof Performance
Key Performance Parameters
TechnicalPerformance Measures
RISK REDUCTIONMEASURESARE USED TO INFORM EARNED VALUE
All programs have risk. Assessing how these risks are being reduced is a measure of the
programs probability of success. If the risks are not being reduced as planned, to the
level they are planned for, on the date they are planned to be reduced, then the
program is accumulating a risk debt. This debt lowers the probability of success.
The naturally occurring uncertainties in cost, schedule, and technical performance can
be modeled in a Monte Carlo Simulation tool. The event-based uncertainties require
they be captured in the Risk Register, modeled for their impacts, with defined handling
strategies, modeled for the effectiveness of these handling efforts, and the residual
risks, and their impacts assessed for both the original risk and the residual risk on the
program.

8. 5745 8
The management of the naturally occurring uncertainties in cost, schedule, and
technical performance as well as the event-based uncertainty and the resulting risk are
critical success factors to informing Earned Value.
Definitions of Uncertainty and Risk Needed Before Proceeding
In program performance management, risk drives the probability of program success.
Risk is not that actual source of this probability of success. All risk comes from
uncertainty. 5 In this paper we use specific definitions of risk and uncertainty applied to
assessing impacts on program performance.
 Uncertainty is present when probabilities cannot be quantified in a rigorous or valid
manner, but can described as intervals within a probability distribution function
(PDF)
 Risk is present when the uncertainty of the outcome can be quantified in terms of
probabilities or a range of possible values
This distinction is important for modeling the future performance of cost, schedule,
and technical outcomes of a program using the Risk Register and Monte Carlo
Simulation tools. As well it is important to distinguish between two type of
uncertainty that create risk to the program.
 Aleatory (stochastic) variability's are the natural randomness of the process and are characterizedbya
probabilitydensityfunction (PDF) for their range andfrequency, andtherefore irreducible.
 Epistemic (subjective or probabilistic) uncertainties are event-basedprobabilities, are knowledge-based,
and are reducible byfurther gatheringof knowledge.
Separating these classes helps in design of assessment calculations and in
presentation of results for the integrated program risk assessment.
Sourcesof Uncertainty
There are sources of uncertainty on all programs 6
 Lack of precision about the underlying process – we simply can tell what is going in
with the underlying process.
 Lack of accuracy about the possible values in the uncertainty probability
distributions – this may come from too few samples of past performance.
 Undiscovered Biases used in defining the range of possible outcomes of project
processes – this is a naturally occurring process when people are asked to assess
risk.
 Natural variability from uncontrolled processes – this is the Aleatory uncertainty of
the underlying processes. The only protection from Lavatory uncertainty is margin,
since it cannot be reduced or eliminated.
5 “Epistemic Uncertainty in the Calculation of Margins,”Laura Swiler,50th AIAA/ASME/ASCE/AHS
Structural Dynamics and MaterialsConference, 4-7 May, 2009,PalmSprings,California.
6 Towards a Contingency Theory of Enterprise Risk Management, Harvard Business School, Anette Mikes
and Robert Kaplan, WorkingPaper 13-063 October 17, 2013

9. 5745 9
 Undefined probability distributions for project processes and technology – with
some form of probability distribution function the assessment of the risk and its
impact cannot be modeled without making assumptions about the underlying
behaviors.
 Unknowability of the range of the probability distributions – is the probability
distributions are truly unknowable, alternative modeling processes are needed.
Bayesian risk modeling is one approach. 7
 Absence of information about the probability distributions – like the unknowability
of the probability distributions, alternative approaches are needed.
All Risk Comes From Uncertainty
Uncertainty creates risk. Uncertainty comes in
two types – reducible and irreducible. We need
to handle both on the program. But both
reducible and irreducible uncertainty starts with
a statistical process that is either naturally
occurring – irreducible, or event-based reducible.
Uncertainty is present when probabilities cannot
be quantified in a rigorus manner, but can only
be described as intervals on probability
distribution function. Risk is present when the
uncertainty of the outcome can be quantified in
terms of probabilties or range of possible value.
The distinction between statistics and probability
is important for modeling the performance of cost, schedule, and technical outcomes of
the program. And most important in defining the measures used to asses the program’s
actual performance against the planned performance as it is impacted by uncertainty
and the resulting risk.
Risks that result from the irreducible and reducible uncertainties are recorded in the
Risk Register. The contents of the Risk Register starts with a short description of the risk,
its probability of occurrence and its impact. Each risk then needs a handling strategy.
Each of these handling strategies can be represented in the IMS or in Management
Reserve.
For reducible risks the Performance Measurement Baseline can contain work to reduce
the probability of occurrence of the risk or reduce the impact of the risk on the program
should the risk become an issue. This work is on baseline, has budget, and is contained
in a Work Package.
The risk can also be handled through indirect means. Management Reserve can be
assigned for each risk outside the PMB. Should the risk occur, this Management Reserve
can be used to perform work to address the impact from the risk.
7 Bayesian Inferencefor NASA Probabilistic Risk and Reliability Analysis,NASA/SP-2009-569
Figure 3 – Both statistics and probability
impact the success ofthe program, through
the irreducible uncertaintyof statistical
process andprobabilityof anevent-based risk
occurring.

10. 5745 10
Both explicit risk retirement activities and Management Reserve are necessary for the
successful completion of any program. Management Reserve is budget set aside of use
by management to address the Known Unknowns on the program. Explicit risk
retirement activities have budget on baseline for reduction of risk.
Using Risk Reduction as a Measure of Program Performance
With the reducible risk, specific work is performed on baseline to reduce the probability
of occurrence of the risk. 8 With Irreducible risk, margin is needed to protect the
delivery date of key deliverables. A measure of the performance of the program is the
Margin Burn-down Plan shown in Figure 5. If the actual risk reduction does not follow
the risk burn-down plan, this is a leading indicator of future difficulties in the program’s
performance and an indicator of impact on cost and schedule.
Figure 4 – risk reductionactivities are guided bythe RiskRegister’s contents andthe WBS inthe IMS. Workto reduce
risk is planned and executed On Baseline throughWorkPackages. The riskreductionlowers the probabilityof
occurrence and therefore lowers the probabilityof impact ofthe namedrisk as shownin Figure 5.
8 NASA Risk Informed Decision MakingHandbook,NASA/SP-2010-576

11. 5745 11
Figure 5 – the plannedrisk reduction inthe Center Of Gravitygoing out of bounds for a flight vehicle as a functionof
time inthe IntegratedMaster Schedule. Baseline workis performed to reduce this risk and therefore reduce the
impact of the risk. Markingthe reductions to the plannedlevel onthe planned dayinforms the BCWPof the Earned
Value numbers. Reducingthe risklate must reduce the BCWPfor activities relatedto the risk.
This risk reduction plan is traced back to the Risk Register through the Risk ID and the
Work Breakdown Structure number.
Developmentof Schedule Margin
 Assure all work in the WBS performed in the proper sequence in a well formed
network
 Identify all reducible risks from Risk Register
 Add work for reducible risks with budget to the PMB to form the deterministic IMS
 Identify variances in work durations with a probability distribution function shape,
Upper and Lower limits

12. 5745 12
Schedule Margin Burn-Down asa Measure of Program Performance
A similar reduction plan can be used for schedule margin.
Figure 6 – Schedule MarginBurn-DownPlandescribes the plannedschedule marginversus of the actualschedule
marginthat protects the date of a keydeliverable. This schedule marginplanis a leading indicator of the risk to the
deliverydate and the success of the program.

13. 5745 13
SIX STEPS TO BUILDING A CREDIBLE INTEGRATED MASTER PLAN
The Integrated Master Plan (IMP) is strategy for the successful delivery of outcomes of
the program. Strategies are hypotheses that need to be tested. The IMP’s Events, and
their Criteria and Accomplishments are the testing points for the hypothesis that the
program is proceeding as planned. Both technically as planned and programmatically as
planned.
1. Identify Program Events
 Program Events are maturity assessment points in the program
 They define what levels of maturity for the products and services are needed before
proceeding to the next maturity assessment point
 The entry criteria for each Event defines the units of measure for the successful
completion of the Event
 Confirm the end to end description of the increasing maturity of the program’s
deliverables
 Establish of RFP or Contract target dates for each Event.
 Socialize the language of speaking in “Events” rather than time and efforts
2. Identify SignificantAccomplishments
 The Significant Accomplishments are the “road map” to the increasing maturity of
the program
 The “Value Stream Map” resulting from the flow of SA’s describes how the products
or services move through the maturation process while reducing risk
 The SA map is the path to “done”
3. Identify AccomplishmentCriteria
 The definition of “done” emerges in the form of deliverables rather than measures
of cost and passage of time.
 These deliverables come from Work Packages, whose outcomes can be assessed
against the Technical Performance Measures (TPM) to assure compliance with the
MOP.
 At each Program Event, the increasing maturity of the deliverables is defined
through the Measures of Effectiveness (MoE) and Measures of Performance (MoP).
4. Identify Work Packages to Complete the AccomplishmentCriteria
 The work identified that produces a measurable outcome.
 This work defined in each Work Package
 The Accomplishment Criteria (AC) state explicitly what “done” looks like for this
effort
 With “done” stated, Measures of Performance and Measures of Effectiveness can be
assessed with the products or services produced by the Work Package.

14. 5745 14
5. Sequence The Work PackagesIn A LogicalNetwork.
 Work Packages partition work efforts into “bounded” scope
 Interdependencies constrained to Work Package boundaries prevents “spaghetti
code” style schedule flow
 Visibility of the Increasing Flow of Maturity starting to emerge from the flow of
Accomplishment Criteria (AC)
6. Adjust Sequence Of WPs, PPs, & SLPPsTo Mitigate Reducible Risk
 Both the maturity assessment criteria and the work needed to reach that level of
maturity are described in a single location
 Risks are integrated with the IMP and IMS at their appropriate levels
 Risks to Effectiveness – risk to JROC KPPs
 Risks to Performance – risk to program KPPs and TPMs
 Leading and Lagging indicator data provide through each measure to forecast future
performance.
EIGHT STEPS TO BUILDING THE INTEGRATED MASTER SCHEDULE
A Risk Informed PMB represented by the resource loaded Integrated Master Schedule
(IMS) means that both Irreducible (Aleatory Uncertainty) and reducible (Epistemic
Uncertainty) risk mitigations are embedded in the IMS. For non-mitigated risks,
Management Reserve (MR) must be in place outside the PMB to cover risks that are not
being mitigated in the IMS.
While DCMA would object, this Management Reserve needs to be assigned to specific
risks or classes of risk to assure that sufficient MR is available and use is pre-defined.
Assemble a Credible Description of Whatis Being Delivered
 Use MIL-STD-881C as the framework for defining the structure of the deliverables
 Assure only deliverables and processes that produce deliverables are in the WBS
 Using the Integrated Master Plan, develop the Integrated Master Schedule showing
the work to be performed that increases the maturity of each deliverable assessed
in the IMP with Measures of Effectiveness (MOE) and Measures of Performance
(MOP)
 Assign Technical Performance Measures to each key deliverable produced by a Work
Package in the IMS.

15. 5745 15
Figure 7 – vertical andhorizontaltraceabilitybetween the Integrated Master Plan(IMP) and the IntegratedMaster
Schedule (IMS). The IMPstates describes how the increasing maturityof the deliverables willbe assessedwith
Measures of Effectiveness(MOE) and Measures of Performance (MOP). The IMS describesthe work neededto
produce that increasingmaturitythroughthe assessment of TechnicalPerformance Measures (TPM).
Identify the Reducible Risks
 Variances in duration and cost are applied to the Most Likely values for the work
activities
 Apply these variances in the IMS
 Model the outcomes using a Monte Carlo Simulation tool
 The result is a model of the confidence of completing on or before a date and at or
below a cost
Put These Risks in the Risk Register
Develop Risk RetirementPlans for the Reducible Risks
Assess the Irreducible Risks
Use Monte Carlo Simulation to Determine Schedule Margin
«Describe how schedule margin is development using
reference classes and Monte Carlo simulation of the
deterministic schedule to produce the needed confidence
level for the probabilistic schedule. The different between
the deterministic schedule – with it’s event-based risk
reduction activities – the and probabilistic schedule at
some acceptable level of confidence – 80% e.g. – is the

16. 5745 16
schedule margin. Show the step-by-step process for doing
this»
Assign Schedule Margin to ProtectKey Deliverables
Using schedule margin to protect against schedule risk – created by the natural
uncertainties in the work durations – is the appropriate method to enable on-time
contractual end item deliveries. Up for debate are where you place the margin and how
you manage its use.
Fundamentally, there are 2 approaches to consider:
 Holding all margin at the end (of a program or end item deliverable); or
 Visually distributing margin at strategic junctures along critical paths where there is
known schedule risk.
Some believe holding it all at the end, forcing the earliest possible baseline, is the best
means of ensuring on-time completion. This is effective in short duration or production
efforts where the primary schedule risk is not driven by technical complexity or
development risk. However, the same objective can be achieved when a disciplined
process is followed for control and consumption of distributed margin. Paramount to
this approach is accelerating downstream efforts when margin is NOT consumed. Most
schedule risk in a development program is encountered when program elements are
integrated and tested. So even when using distributed margin, margin is often kept at
the end of the deliverable schedule to help protect against risk when all paths come
together during final integration and test. This approach enables on-time end item
delivery with realistic cost and schedule baselines that provide accurate forecasts and
decisions based on current status, remaining efforts and related schedule risks.
Valid reasons for distributing schedule margin earlier in the schedule include:
 Protecting use of critical shared resources so that being a few weeks late doesn’t
turn into a several month schedule impact. An example in space programs is use of a
thermal vacuum chamber shared across multiple programs at very critical times in
their schedules. If a program is unable to enter the chamber at their scheduled time
the ultimate delay may be an exponential factor of their original delay.
 Protecting highly visible milestones that are difficult and highly undesirable to
change like a Critical Design Review (CDR)
 Establishing realistic performance baselines accounting for schedule risk at key
points provides more valid data to make program decisions
 Realistic baselines (vs. one planned as early as possible) are more cost effective with
regards to our subcontractors; valid integration dates are less likely to lead to forced
early procurements that incur expediting costs and/or rework due to
immature/changing requirements
 Placing margin where you believe it will be needed and consumed provides the most
realistic schedule baseline possible for succeeding efforts and enables more
accurate resource planning (for us, our customer, & our suppliers)

17. 5745 17
But extreme care and discipline must be used in deciding where distributed margin is
placed, in providing rules on how it is consumed, and for what to do when it is NOT used
where it is initially allocated.
When a distributed schedule margin “task” is reached in the schedule and the risk it was
planned for is realized, the margin is converted to the appropriate task(s) and included
in the baseline. This realized risk represents new scope to the affected CAMs and can
be funded from management reserve as appropriate.
If the risk is not realized and schedule margin is not consumed you must also be
prepared to accelerate succeeding efforts where possible. The margin task is zeroed out
and the remaining margin is moved down stream in the form of increased forecast
durations for subsequent margin tasks or as increased total float. The determination of
which method to use should be a risk based decision. Succeeding tasks are accelerated
and accurately depicted ahead of schedule (positive schedule variance) as tasks are
completed ahead of the baseline.
The point is to plan schedule margin strategically where it ultimately enables on time
end item delivery. It’s an important way Planning brings value to our programs.
Summary of Key Points:
 Schedule margin (reserve) is different from slack (float) as acknowledged in the
“GAO Schedule Assessment Guide”. Margin is preplanned and consumed for known
schedule risk and float is the calculated difference between early and late dates. In
many ways margin is much like management reserve and float is similar to
underruns/overruns.
 Schedule margin is placed where there is known schedule risk. It is never consumed
because of poor schedule performance. (managed like MR)
 You don’t budget schedule margin. If the risk the margin was designed to protect
against comes to fruition, identify the new tasks and budget the new tasks (or
extend the duration of existing tasks). If the risk is not realized, zero out the margin
and accelerate succeeding tasks
 Allocating margin for known risks at key points makes it “off limits” for crutching
poor schedule performance. This helps force immediate recovery action to stay on
schedule instead of degrading margin as if it was float.
 Inclusion of margin (either interim or at the end of contractual deliverables) does
not affect contractual period of performance.
 Inclusion of interim schedule margin for known schedule risk provides the most
realistic baseline and accurate resource planning (including our customer). If not
consumed the effort is accurately “ahead of schedule”. It’s risk-based planning.
 Provides ability to meet interim key milestones via probabilistic scheduling
 Integrates EVM & Risk Management
 Makes best use of SRA data

18. 5745 18
CONNECTING THE DOTS TO THE IPMR SUBMITTAL
BUILDING A RISKADJUSTED PMB
Identifying the Reducible Uncertainties
Identifying the Irreducible Uncertainties
Identify Schedule Margin
Identify Cost Margin
INFORMING BCWPWITH MEASURES
MeasurementCompliance Processes

19. 5745 19
REFERENCES
1. ProbabilisticRiskAssessmentProceduresGuide forNASA ManagersandPractitioners,
NASA/SP-2011-3421, 2nd
Edition,December2011.
2. NASA RiskInformedDecisionMaking Handbook,NASA/SP-2010-576,April 2010.
3. EAI-748-C
4. The Forgotten“-ilities”,JamesD.WillisandDr.StevenDam, SPEC Innovations.
http://www.dtic.mil/ndia/2011system/13166_WillisWednesday.pdf
5. GAO CostEstimatingandAssessmentGuideBestPracticesforDevelopingandManaging
Capital ProgramCosts,GAO-09-3SP
GAO Cost
Estimating
and Assessment
Guide
Best Practices for Developing and
Managing
1. Capital Program Costs

20. 5745 20
Introduction ........................................................................................................................ 1
Top Level Process for Program Success.......................................................................... 1
The Situation ................................................................................................................... 2
The Imbalance................................................................................................................. 2
Restoring The Balance..................................................................................................... 3
The Solution .................................................................................................................... 4
Measures of Technical Progress Start with the IMP ................................................... 4
Risk Management Starts At The IMP .......................................................................... 5
Structure of the IMP ................................................................................................... 5
IMP and IMS Relationship........................................................................................... 6
Assembling the parts needed to connect the dots............................................................. 7
Technical Measures Used Are To Inform Earned Value ................................................. 7
Measures of Effectiveness.......................................................................................... 7
Measures of Performance .......................................................................................... 7
Key Performance Parameters ..................................................................................... 7
Technical Performance Measures............................................................................... 7
Risk Reduction Measures Are Used to Inform Earned Value ......................................... 7
Definitions of Uncertainty and Risk Needed Before Proceeding ................................ 8
Sources of Uncertainty................................................................................................ 8
All Risk Comes From Uncertainty................................................................................ 9
Using Risk Reduction as a Measure of Program Performance ................................. 10
Schedule Margin Burn-Down as a Measure of Program Performance .................... 11
Six Steps to building a credible integrated master plan ............................................... 13
1. Identify Program Events........................................................................................ 13
2. Identify Significant Accomplishments................................................................... 13
3. Identify Accomplishment Criteria ......................................................................... 13
4. Identify Work Packages to Complete the Accomplishment Criteria .................... 13
5. Sequence The Work Packages In A Logical Network. ........................................... 14
6. Adjust Sequence Of WPs, PPs, & SLPPs To Mitigate Reducible Risk .................... 14
Eight Steps to Building the Integrated Master Schedule.............................................. 14
Assemble a Credible Description of What is Being Delivered .................................. 14
Identify the Reducible Risks...................................................................................... 15

21. 5745 21
Put These Risks in the Risk Register.......................................................................... 15
Develop Risk Retirement Plans for the Reducible Risks........................................... 15
Assess the Irreducible Risks...................................................................................... 15
Use Monte Carlo Simulation to Determine Schedule Margin .................................. 15
Assign Schedule Margin to Protect Key Deliverables ............................................... 16
Connecting the Dots to the ipMr submittal...................................................................... 18
Building a Risk Adjusted PMB ....................................................................................... 18
Identifying the Reducible Uncertainties ................................................................... 18
Identifying the Irreducible Uncertainties.................................................................. 18
Identify Schedule Margin.......................................................................................... 18
Identify Cost Margin.................................................................................................. 18
Informing BCWP with Measures................................................................................... 18
Measurement Compliance Processes....................................................................... 18
References......................................................................................................................... 19