Objectively measuring Physical Percent Complete to Planis a critical success factor for all project work

1. Integrated Performance Management 2014
Using Technical Progress to Inform
Earned Value Performance
Glen B. Alleman
Niwot Ridge, L.L.C.
+1 303 241 9633
glen.alleman@niwotridge.com
Thomas J. Coonce
Institute for Defense Analyses
+1 703 362 2568
tom.coonce@ida.org
Rick A. Price
Lockheed Martin
+1 303 971 1826
rick.a.price@lmco.com
1

2. 2
EAI-748-C Tells Us To …
 Objectively assess accomplishments at the
work performance level.
And at the same time, says in §3.8
 Earned Value is a direct measurement of the
quantity of work accomplished.
 The quality and technical content of the work
performed is controlled by other processes.
GA

3. 3
It’s Going To Be Hard to Tell When
We’ll Arrive at our Destination …
 If we only measure our progress by CPI and
SPI, whose units of measure are dollars.
 We need to connect technical and quality
measures with these EV measures.
GA

4. 4
4
Technical Performance Informs
Earned Value
DI-MGMT-81861 §1.2.1 says …
Integrate cost and schedule performance with objective technical measures of performance
Objective Technical Measures
Cost Performance Schedule Performance
Conventional Earned Value
+
=
Basis of Estimate developed from
scope of work and historical
performance, and risk adjusted
for all classes of cost, to produce
a resource loaded PMB with MR
established in the EVMS.
The ETC and EAC updated as the
program progresses using
objective technical measures of
performance.
Cost
Measures of Effectiveness
(MOE), Measures of
Performance (MOP), Technical
Performance Measures (TPM),
Key Performance Parameters
(KPP), and, other …ilities of the
technical aspects of the program
performance inform the Physical
Percent Complete used to Inform
BCWP.
Objective Technical Measures
Deterministic IMS derived from
the IMP / Statement of Work
(SOW), showing how all
deliverables defined in the WBS
will be produced and reducible
risks mitigated with Retirement
Plans, and Irreducible risks
mitigated with schedule marring
and Management Reserve, to
establish the Credible PMB.
Schedule
GA

5. 5
5
What Sources Can Be Used To For
Objective Technical Measures of
Performance?
Risk
SOW
Cost
WBS
IMP/IMS
TPM
PMB
GA

6. 6
6
How Are These Sources Connected?
Risk
Management
SOW
SOO
ConOps
WBS
Techncial and Operational
Requirements
CWBS &
CWBS Dictionary
Integrated Master Plan
(IMP)
Integrated Master Schedule
(IMS)
Earned Value Management
System (EVMS)
Performance Measurement Baseline (PBM)
Measures of
Effectiveness (MOE)
Measures of
Performance (MOP)
Measures of Progress
(Physical % Complete)
JROC
Key Performance Parameters
(KPP)
Program Specific
Key Performance Parameters
(KPP)
Technical Performance
Measures (TPM)
CWBS
6
Objective Technical Measures of Performance provide information for proactive
management processes needed to keep the program GREENTC

7. 7
7
6 Steps To These Connections
Step Outcome
❶
Define
WBS
 With SOW, SOO, ConOps, WBS, and other program documents, develop
CWBS of system deliverables and work processes to produce the program
outcomes.
 Develop CWBS Dictionary describing scope of work and Criteria for the
successful delivery of these outcomes.
❷
Build IMP
 Develop Integrated Master Plan (IMP), showing how each system element in
the CWBS moves through the maturation process at each Program Event.
 Define Measures of Effectiveness (MOE) for each Accomplishment.
 Define Measures of Performance (MOP) for each Criteria.
❸
Identify
Reducible
Risk
 For each key system element in the CWBS, identify reducible risks,
probability of occurrence, mitigation plan, and residual risk in the Risk
Register.
 Risk mitigation activities placed in IMS and PMB to assure probability of
occurrence and probability of impact reduced.
 For risks without mitigation plans, place budget for risk in Management
Reserve (MR) to be used to handle risk when it becomes an Issue.TC

8. 8
8
Step Outcome
❹
Build the
IMS
 Arrange Work Packages and Tasks in a logical network of increasing maturity
of the deliverables.
 Define exit criteria for each Work Package to assess planned Physical
Percent Complete to inform BCWP using TPM, MOP, MOE, and Risk
Reduction activities in support of Accomplishments in the IMS.
❺
Adjust for
Irreducible
Risks
 For irreducible risks in the IMS, use Reference Classes for Monte Carlo
Simulation anchored with Most Likely duration to calculate needed schedule
margin.
 Assign schedule margin tasks in the IMS, to protect the key system
elements, per DI-MGMT-81861 guidance.
❻
Establish
PMB
 Using risk adjusted IMS, calculate needed Management Reserve (MR) to
account for the latent risks in the Risk Register.
 With deterministic IMS and its embedded Schedule Margin and
Management Reserve for latent risk, determine the resulting confidence
level of the PMB.
6 Steps To These Connections
TC

9. 9
9
A Reminder of our Starting Point
1.2.1 Integrate cost and schedule
performance data with
objective technical measures
of performance
DI-MGMT-81861†
†Integrated Program Management Report (IPMR), DI-MGMT-81861, OUSD
(AT&L) PARCA, 20 Jun 2012
TC

10. 10
10
❶ DEFINE WORK BREAKDOWN
STRUCTURE
WORK BREAKDOWN STRUCTURE IS PARAMOUNT
The Work Breakdown Structure is the starting point for
developing all other elements needed for the
Performance Measurement Baseline.
With the WBS Dictionary, technical measures are
developed to provide assessment of Physical Percent
Complete of each deliverable.
GA

11. 11
Work Breakdown Structure
 Mutually Exclusive – no subcategory should represent
any other subcategory ("no overlaps"). In the WBS this
means the deliverables are unique so we can assign
cost to them and determine who is going to develop
them.
 Collectively Exhaustive - the set of all subcategories,
taken together, should fully characterize the larger
category of which the data are part ("no gaps"). The
WBS represents the "all in” work. If it's not in the WBS,
it's going to get done.
Mutually Exclusive and Collectively Exhaustive
GA

12. 12
Building a Good WBS
 Know the components of the system that are ME
(Mutually Exclusive).
 Children of these components are CE (Collectively
Exhaustive).
– Parts can be shared in the WBS, only if the work for
those parts can be shared
– But they need different WBS numbers
– So they are ME, while being CE
 WBS collects costs and answers, "what does this
part, sub-assembly, assembly, system cost?”
GA

13. 13
The Work Breakdown Structure is
Start of Risk Management
 Terminal nodes of the WBS are where the
Deliverables live.
 Each must be assessed for the possible risk
that will imped their success in meeting:
– Measures of Effectiveness,
– Measures of Performance,
– Technical Performance Measures,
– Key Performance Parameters,
– All their …ilities.
GA

14. 14
14
❷ BUILD INTEGRATED MASTER PLAN
BUILDING THE IMP IS A SYSTEMS
ENGINEERING ACTIVITY
The IMP represents the Programmatic Architecture.
Poor, weak, or unstructured Programmatic Architecture
reduces visibility to the product’s cost and schedule
performance connected with Technical Performance
Measures.
RP

15. 15
15
The Integrated Master Plan describes where the program is going, the various
paths taken to reach destination, and the progress or performance
assessment points along the way to assure it is on the right path.
These assessment points measures the “maturity” of the product or service
against the planned maturity. This is the only real measure of progress – not
the passage of time or consumption of money.
The Integrated Master Plan (IMP) Is The Strategy For
The Successful Completion Of The Project
RP

16. 16
Structure of the Integrated Master
Plan (IMP)
 Vertical and Horizontal traceability assures each deliverable
is assessed for increasing maturity against its planned
maturity
IMS
IMP
Describes how program
capabilities will be
delivered and
how these
capabilities will
be recognized
as ready for
delivery
Supplemental Schedules (CAM Notebook)
Work Packages and Tasks
Criteria
Accomplishment
Events
or
Milestones
RP

17. 17
17
Wright Flyer
Objective Technical
Measures
The Army Contract No. 486 Says
Program
Performance
Measure
The flying machine must be designed to carry two people having a combined weight of no more than 350
pounds,
MOP
Also sufficient fuel for a flight of 125 miles MOP
The flying machine should be designed to have a speed of at least 40 miles per hour in still air for at least
125 miles
MOP
The flying machine should be designed so that it may be quickly and easily assembled and taken apart
and packed into an Army wagon.
KPP
It should be capable of being assembled and put in operating condition within one hour KPP
Before acceptance, a trial endurance flight will be required of at least one hour during which time, MOP
The flying machine must remain continuously in the air without landing. MOE
It shall return to the starting point and land without any damage that would prevent it immediately
starting upon another flight.
MOE
During this flight of one hour, it must be steered in all directions without difficulty and at all times under
perfect control and equilibrium.
MOE
It should be sufficiently simple in its construction and operation to permit an intelligent man to become
proficient in its use within a reasonable length of time.
KPP
TC

18. 18
18
❸ IDENTIFY REDUCIBLE RISKS
REDUCIBLE RISKS IDENTIFIED AND PLACED IN THE
RISK REGISTER
The Risk Register contains the probability of occurrence,
the probabilistic impact of the risk on cost, schedule, and
technical performance, cost of mitigating the risk, residual
probability of the risk after mitigation, and impact of the
risks after mitigation.
GA

19. 19
19
Reducible Risk Is Created By
Lack of Knowledge (Epistemology)
Uncertainty
Reducible Irreducible
Natural
Variability
Ambiguity
Known but
un-mitigateable
Probabilistic
Events
Probabilistic
Impacts
Periods of
Exposure
The probability of an
event that we can do
something about,
reducing this
probability through
explicit actions.
Statistical range of
natural randomness
characterized by a
historical data, and
therefore irreducible
Epistemic Aleatory
GA/TC

20. 20
Identifying Reducible Risks
 Identify pre-mitigation
– Probability of
occurrence
– Probability of impact
– Cost of mitigation
 Identify post-mitigation
– Probability of occurrence
– Probability of impact
– Residual risk after
mitigation
TC

21. 21
Reducible Risks Must Have
Retirement Plans in the IMS
 Buying down reducible risk is a measure of
performance for the program
RP
Risk ID# 12 If the Power Distribution Unit fails vibration test, redesign will be necessary

22. 22
22
❹ BUILD INTEGRATED MASTER
SCHEDULE
INTEGRATED MASTER SCHEDULE (IMS)
DERIVED FROM
INTEGRATED MASTER PLAN (IMP) CRITERIA
IMS Shows The Order In Which The Work Packages Must
Be Performed To Assure The Criteria Are Completed Within
The Define Measures Of Performance, Key Performance
Parameters, And The Technical Performance Measures.
GA

23. 23
The Integrated Master Schedule
 Is a horizontal sequence of work activities that
produce products with increasing maturity of
technical performance
GA

24. 24
24
Assess the maturity
of a Capability at a point in
time.
Requirements that
enable planned Capabilities.
Exit Criteria for Work Packages
That deliver the Requirements.
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
package
 Integrated Master Plan (IMP) is the strategy for the successful
completion of the program, measured as “increasing maturity” of
the Performance Measures (MOE, MOP, TPM, KPP).
 Without a plan we cannot say what “done” looks like.
 Integrated Master Schedule (IMS) defines the Work Packages and
Planning Packages needed to implement the Plan, retire risks,
manage resources, and sequence the detailed work activities.
 Both the Plan and the Schedule are essential for success.
The structure of a
Performance-Based
Program
CRITERIA
ACCOMPLISHMENTS
EVENTS
MOP
MOE
Subsystem
System
GA

25. 25
25
❺ ADJUST FOR IRREDUCIBLE RISKS
IRREDUCIBLE UNCERTAINTY
CREATES
IRREDUCIBLE TECHNICAL AND PROGRAMMATIC
RISK
This Naturally Occurring Uncertainty Impacts Schedule
And Cost In Irreducible Ways. It Can’t Be Bought Down.
Irreducible Risks Can Only Be Mitigated With Margin.
GA

26. 26
26
Irreducible Risk Driven By
Natural Variability
Uncertainty
Reducible Irreducible
Natural
Variability
Ambiguity
Known but
un-mitigateable
Probabilistic
Events
Probabilistic
Impacts
Periods of
Exposure
The probability of an
event that we can do
something about
reducing this
probability through
explicit actions.
Statistical range of
natural randomness
characterized by a
historical data and
therefore irreducible
Epistemic Aleatory
GA

27. 27
27
Identifying Irreducible Risks
Starts With Reference Class Forecasting
GA

28. 28
Applying Reference Classes to Work Activities
to Construct Network of Random Processes
 The independence or
dependency of each task
with others in the network,
greatly influences the
outcome of the total project
duration.
 Understanding these
dependencies is critical to
assessing the credibility of
the IMS as well as the total
completion time.
 Any path could be critical depending on the probability distributions
of the underlying task completion probability functions.
We must know something about the
probability distributions of the work efforts
GA

29. 29
29
Irreducible Risks Are Mitigated With
Margin For Both Cost and Schedule
 The contract delivery date is 8/31/08
 The IMS without schedule margin has
a confidence of 80% on or before
8/14/08
 The deterministic date is 8/4/08
 10 days of margin protect contractual
schedule with 80% confidence
8/4/1908
Deterministic
Date
GA/RP
8/14/1908
80% On or
Before
8/31/1908
Deterministic
Date

30. 30
30
Schedule Margin Burn Down is a
Measure of Program Performance
RP

31. 31
31
But, Beware the Black Swan
There is no Plan B for Black Swans – Must Replan 31
Natalie Portman, as Nina Sayers, The Swan Queen
Black Swan – An event or occurrence that
deviates beyond what is normally expected of a
situation and that would be extremely difficult
to predict. This term was popularized by Nassim
Nicholas Taleb, a finance professor and former
Wall Street trader.

32. 32
32
❻ ESTABLISH THE PMB
EACH ELEMENT OF THE IMP AND IMS ARE
NEEDED FOR A CREDIBLE PERFORMANCE
MEASUREMENT BASELINE
The Credible PMB Requires That Each Element Itself Be
Credible With The Relationships Properly Established,
Maintained, And Updated.

33. 3333
Connecting the Dots in the PMB
WBS KPPs TPMs
EVM
ETC
EAC
Irreducible
uncertainty
in reference
classes Reducible
uncertainty held
in Risk Register
Schedule Margin in
DI-MGMT-81861
Cost Contingency †
Risk retirement in PMB
Management Reserve
covers unmitigated risk
MOE
MOP
Physical %
Complete
PE
SA
AC
JROC KPP
IMP
IMS WP
33GA

34. 34
34
EXECUTING THE PERFORMANCE
MEASUREMENT BASELINE
A well known example of integrating the elements of a credible PMB needed to
successfully complete a program can be found in the Wright Brothers, 23
December 1907, Army Contract, No. 486
http://www.wright-brothers.org/History_Wing/Wright_Story/Showing_the_World/Back_in_Air/Signal_Corps_Spec.htm
TC

35. 35
35
Executing the
Performance Measurement Baseline
Take management action for any variances to assure on–time, on–
budget and on–specification of all deliverables produced by the Work
Packages.
Maintain Performance Management Baseline (PMB) throughout project
duration for Earned Value measures in Objective units of measure at the
work performance level, meaningful to the decision makes
Perform the Work described in the network of Work Packages held in the
IMS, in the planned order, with the planned budget with the planned
outcomes, meeting the planned performance measures.
Accumulate and Report Performance Data – BCWP – and other
increasing maturing assessment processes based on Physical Percent
Complete described in the MOEs, MOPs, TPMs, and KPPs.
Analyze Performance Data derived from the Earned Value metrics –
informed by Technical Performance – and make any adjustments to the
network of Work Packages to keep program GREEN.
TC

36. 36
Connecting Technical Performance
(Weight) with EV Performance
 Planned weight reduction of Wright Flyer
connected with Earned Value performance by
planned date is a TPM
Earned Value
Performance
Weight Target
in or out of bounds
Informs
Weight out
of bounds
BCWP should
follow weight
TC

37. 37
37
Summary of Steps To Create The
Credible PMB
6 Steps To Success
❶ Define WBS ❹ Build the IMS
❷ Build IMP ❺ Adjust for Irreducible Risks
❸ Identify Reducible Risk ❻ Establish the PMB
1.2.1 Integrate cost and schedule performance
data with objective technical measures of
performance
DI-MGMT-81861†
† Integrated Program Management Report (IPMR), DI-MGMT-81861, OUSD (AT&L) PARCA, 20 Jun 2012
And One Last Reminder
TC

38. 38
References
 IMP/IMS Preparation and Use Guide,
http://www.acq.osd.mil/se/docs/IMP_IMS_Guide_v9.pdf
 GAO Cost Estimating and Assessment Guide,
http://www.gao.gov/new.items/d093sp.pdf
 GAO Schedule Assessment Guide,
http://www.gao.gov/assets/600/591240.pdf
 Air Force Integrated Master Schedule Assessment Guide,
http://afacpo.com/AQDocs/AF_IMS_%20Assessment_%20Process_%20V3.0.p
df
 The Integrated Project Management Handbook, Dayton Aerospace, Inc.
 Scheduling Guide for Program Managers
https://www.acquisition.gov/sevensteps/library/DAUschedulinguide.pdf
 Integrated Master Plan Analysis, The PMAG Approach, Col. Mun H. Kwon,
USAF, http://www.dau.mil/pubscats/ATL%20Docs/Jan-Feb/kwon_jan-
feb10.pdf

39. 39
References
 AFMC Pamphlet 63-5, Integrated Master Plan and Schedule Guide, (11
Nov 2004), http://www.everyspec.com/USAF/USAF-
AFMC/AFMC_PAM_63-5_4210/
 MSFC-HDBK-3173, Project Management and System Engineering
Handbook, https://standards.nasa.gov/documents/detail/3315842
 Project Planning and Control Handbook, MSFC-HDBK-3684,
https://standards.nasa.gov/documents/viewdoc/3315842/3315842
 NASA Scheduling Management Handbook,
http://www.nasa.gov/pdf/420297main_NASA-SP-2010-3403.pdf

40. 4040
The End
40