Program Success depends on a small number of "essential" Views of progress to plan

1. Title: Integrated Program Management Essential
Views
Thomas J. Coonce
Institute for Defense Analyses
tcoonce@ida.org (703) 575-6634
Date: March 27, 2015
EVM World 2015
1

2. Learning Objectives
• Participants will learn about the value of a proposed set of
enhanced metrics that a government program manager
“ought to have” to help manage and control contracted
efforts
2

3. TLO #1
Agenda
• Motivation for proposed Integrated Program Management
metrics/views
• Background
• Twenty-six metrics are proposed along with rational and
value to the government PM
• Audience feedback
3

4. TLO #1
Motivation for This Effort
• Motivation for proposed Integrated Program Management
metrics/views
• Background
• Twenty-six metrics are proposed along with rational and
value to the government PM
• Audience feedback
4

5. TLO #1
Background
• When contractually required, DoD acquisition contractors are
required to provide IPMRs electronically per Data Item Description DI-
MGMT-81861
• Government stakeholders must acquire “reader/viewers” of the data
to understand status and help control
• Several vendors provide multiple metrics/views of these data
– Some data more useful than others
– Different users have different interests
• Easy for the government PM to be overwhelmed
– Key question is which metric/views are considered “Essential” to
accomplish the stated goal of helping the material developer to
“keep the program green”?
5

6. TLO #1
Background (Concluded)
6
• To answer this question, IDA:
– Synthesized existing research on management and control metrics
• Earned Value Project Management (Fleming and Koppelman)
• National Defense Industrial Association’s Predictive Measures Guide
• Earned Schedule (Walt Lipke)
• Integrated Project Management and Control (Mario Vanhouke, University
of Ghent)
• Leading IPMR software vendors (Encore, Steelray, Deltek, and SNA
Software, AzTech International)
• Interfacing Risk and Earned Value Management (Association for Project
Management)
• Numerous subject matter experts
– Distilled candidate metrics and views to 26 that it deemed “essential”
– Created a notional UAV program to demonstrate the metrics/views
The Director of PARCA/EV is aware of these proposed views, but has not
approved them. He desires community feedback.

7. TLO #1
Proposed Essential Views Overview
7
• To answer this question, IDA:
– Synthesized existing research on management and control metrics
• Earned Value Project Management (Fleming and Koppelman)
• National Defense Industrial Association’s Predictive Measures Guide
• Earned Schedule (Walt Lipke)
• Integrated Project Management and Control (Mario Vanhouke, University
of Ghent)
• Leading IPMR software vendors (Encore, Steelray, Deltek, and SNA
Software, AzTech International)
• Interfacing Risk and Earned Value Management (Association for Project
Management)
• Numerous subject matter experts
– Distilled candidate metrics and views to 26 that it deemed “essential”
– Created a notional UAV program to demonstrate the metrics/views

8. TLO #1Proposed Essential Views Overview (Concluded)
8
• Continues with monthly and quarterly data that provides
status and facilitates control
– New quarterly data that provides an integrated picture of
technical, deliverables, staffing, cost, schedule, risk status,
and a forward look at risks that could jeopardize contract
objectives
– Monthly data consists of traditional EMV status metrics
augmented with a few new ones

9. TLO #1Background on Notional UAV Program
9
• Tactical Situational Awareness System (TSAS). DoD desires
– An avionics subsystem that consists of three sensors: EO/IR,
Synthetic Aperture Radar (SAR), and Full Motion Video
– COTS solution if possible
– Five EMD units
– To spend $20 - $25M and complete development less than 48
months with a contract award of 3/17/2014
• Technical approach drives the plan. Contractor:
– Proposes to procure FMV and SAR COTS; build EO/IR in house.
Develop processing software in house
– Procure/modify/build 5 EMD units and 5 ground and land systems
– Derives weight TPM such that sensors must weigh no more than
100 pounds each

10. TLO #1
Background on Notional UAV Program
(Concluded)
10
• Risks modify the plan. Contractor identifies:
1. Insufficient qualified Systems Engineers (SEs) during PDR/CDR
phase
2. Parallel sensor designs could result in under sizing the computer
3. EO/IR technical requirements have never been met before and
may take longer than estimated and/or require different
components
4. Full Motion Video COTS may not provide required resolution
5. Never developed image processing algorithms as complex as
this – may take longer than estimated/expected
6. SAR supplier’s may not meet reliability requirements
7. Much uncertainty during the integration, assembly, test and
check out. May take longer than estimated
8. Funding uncertainty at the start of FY 2016 may stall the
contract effort

11. TLO #1Integrated Baseline Review (IBR) Metrics/Views
11
1. Key Technical Performance Measures plan(s)
2. Deliverables plan
3. Summary level of the Integrated Master Schedule (IMS) and proposed
spend plan
4. Labor FTE utilization plan
5. Schedule health checks
6. Risk Register and mitigation actions
7. Computation of initial Management Reserves
8. Risk burn down plan
9. Computation of Schedule Margin (SM), placement in the IMS, and SM burn
down plan
These metrics provide the PM evidence that the PM has a sound plan at the outset

12. TLO #1Quarterly Metrics/Views
12
10. TPM plan vs. estimated actuals vs. cost and schedule performance metrics
(CPI, SPI, and SPI(t))
11. Deliverables plan vs. actuals vs. CPI, SPI, and SPI(t)
12. FTE plan vs. actuals
13. Cumulative BCWS, BCWP, ACWP against IBR spend plan with Earned
Schedule and Status dates, percent spent, percent complete, and percent
scheduled – An enhanced Gold Card
14. Risk burn down plan vs. actual
15. Schedule margin burn down plan vs. actuals
16. Management Reserve usage and balance
17. Changes to the Baseline (new scope, use of MR or Undistributed Budget)
These metrics provide the PM with an integrated picture of how well the contractor is
meeting the initial technical, product deliverables, staffing , cost, schedule, and risk
management plans.

13. TLO #1Quarterly Metrics/Views (Concluded)
13
18. DCMA schedule heath checks on the “go-forward” IMS
19. Updated Risk Register
20. Cost confidence level of meeting contractor-stated Best
Case, Worst Case, and Most Likely estimates at
completion
21. Schedule confidence level of meeting contractor stated
completion date
22. Schedule sensitivity indices
These metrics/views provide the PM with information about expected
future problems so that pre-emptive correction actions can be taken

14. TLO #1Monthly Metrics/Views
14
• Cumulative BCWS, BCWP, ACWP against IBR spend plan with Earned
Schedule and Status dates, percent spent, percent complete, and
percent scheduled (Enhanced Gold Card – same as #12)
23. Tornado (or Galaxy) chart that shows the relative percentage of
Budget at Complete to total for any level of WBS
24. Monthly and cumulative charts of CV, SV, CPI, SPI, and SPI(t) for any
level of WBS element or OBS
25. Current Execution Index at the top level of the IMS
26. Sources and uses of MR and Undistributed Budget
These metrics provide the PM with information on how the contractor
is meeting the cost and schedule targets and provides leading
indicators of potential future problems

15. DEMONSTRATION OF
SELECTED ESSENTIAL VIEWS
15

16. TLO #1TSAS Sensor Weight Plan
16
1
IBR SFR PDR CDR TRR
TPMs such as weight are not included in the IPMRs or in the IMS, but they should be shown to
the PM periodically because it is one of the key indicators on whether end user requirements are
being met. Reported cost and schedule performance should be in alignment with TPMs.

17. TLO #1TSAS Deliverables Plan
17
2
IBR SFR PDR CDR TRR
Deliverables and dates are flagged in the Integrated Master Schedule. Cost
and schedule progress should also align with required deliverables

18. TLO #1TSAS Initial Deterministic IMS Before IBR
18
3
This plan has not been adjusted for risks. Once done, the costs and
durations may differ.

19. TLO #1Initial TSAS Deterministic Spend Plan
19
3
Based upon TSAS initial resource-loaded Integrated Master Schedule (IMS)

20. TLO #1TSAS Staffing Plan at IBR
20
4
Full Time Equivalents (FTEs) obtained from the “work’ field in the Resource-Loaded IMS

21. TLO #1
The Contractor’s IMS is the Implementation Plan.
Schedule Health Checks Ensures It is Good Quality
21
5
These
schedule
health metrics
were computed
on the TSAS
IMS using
Steelray
Project
Analyzer.
There are
several
vendors who
provide similar
products and
outputs

22. TLO #1TSAS Schedule Health Checks (Concluded)
22
5

23. TLO #1TSAS Reducible Risks
23
6
Risk Register from RiskyProject. Other software products are able to produce similar outputs

24. TSAS Mitigation Strategy for One Risk
24
6

25. TLO #1Management Reserve Calculation Principles
25
• Management reserve (MR) is held for growth within the currently
authorized work scope, for rate changes, and for other program
unknowns. MR is not used to offset accumulated overruns or
underruns and it is not a contingency budget than can be used for
new work or eliminated from the contract price during subsequent
negotiations. The management reserve budget is not included as part
of the Performance Measurement Baseline (PMB). Source:
ACQuipedia
• Operational Definition: Management Reserves (MR) covers in-scope
known reducible risks that were not mitigated. It is for in-scope work
that may or may not materialize.

26. TLO #1Process to Calculate Management Reserves
26
Calculate Contract Budget Base (CBB) from Price and Fee: CBB = Price/(1+fee
percent)
BACi (from initial resource-loaded IMS)
DO WHILE P80 Cost > CBB
Run Monte Carlo Simulation for reducible risks in the Risk Register
Revise Cost Plan (BACi )
END DO
Final cost plan = BACf
MR = CBB - BACf

27. TSAS Management Reserve Calculation*
27
7
* Management Reserves is
based on Monte Carlo
Simulation with Unmitigated
Reducible Risks included.
See following page.

28. TSAS Monte Carlo Simulation for Unmitigated
Reducible Risks Only
28
7

29. TSAS Risk Burn Down Plan
29
8

30. TSAS IMS Prior to Adjustment for Irreducible
Uncertainty
30
This is the revised TSAS IMS after adjustments for
reducible uncertainty that remain in the Risk
Register after mitigation actions.

31. Schedule Margin Calculation Steps
31
Create Bottom-up IMS Plan (P) after adjusting for reducible risks
DO WHILE P80 Date > Need Date (contractually required date)
Run Monte Carlo Simulation for Irreducible Uncertainty
Revise IMS Plan (P)
END DO
Pf = PMB Final Finish Date
Schedule Margin = Need Date Duration (P80 duration) – Pf Duration
WHERE
Irreducible Duration Uncertainty is determined from historical data

32. TSAS Monte Carlo Simulation Results for Irreducible
Duration Uncertainty
32
Created using RiskyProject. Indicates 80% confidence level date = March 11, 2018. Need date = March 17, 2018
9

33. TSAS Monte Carlo Simulation Results for Irreducible
Duration Uncertainty (Concluded)
33
9
P80 date (March 11, 2018) => P80 duration = 1006 days. Pf duration = 859 days. SM = 147 days

34. Schedule Margin Calculation
34
9
Schedule Duration Uncertainty on all tasks of L = .95 ML and H = 1.20 ML.
Inputs must be based on historical data and placed on activities known to vary.

35. Placement of Schedule Margin in TSAS IMS
35
9
Schedule Margin is placed at “strategic points” in the IMS to protect key events,
deliverables or milestones.

36. TSAS Schedule Margin Burn Down Plan
36
9

37. The TSAS Plan at IBR
37
BAC
CBB
Management Reserves

38. TSAS 1QTR2015 QUARTERLY
PROGRESS METRICS AND VIEWS
38

39. TSAS Weights vs. C/S Performance
(as of 1QTR2015)
39
10
13
The Story:
Contractor has
just passed IBR
(10/1/2014) and
engineers have
yet to estimate
the weight of the
sensor
subsystems.
Negative cost
and schedule
performance due
to fact that
contractor missed
four key
deliverables
during the
quarter. See next
slide.

40. TSAS Planned vs. Actual Deliverables vs. C/S Performance
(as of 1QTR2015)
40
11
13
The Story:
Contractor failed
to deliver key
SFR phase
deliverables due
to inadequate
qualified System
Engineers.
Contractor used
additional senior
engineers to
produce the
deliverables, but
were not
completed by end
of quarter. BCWP
scored as 0, 50%
or 100%

41. TSAS Planned vs. Actual FTEs
(as of 1QTR2015)
41
Contractor needed 3 FTE Systems Engineers during 1Qtr2015, but only had 2
on board. This chart helps tell the story of the poor cost and schedule
performance for the period.
12

42. TSAS Risk Status
(as of 1QTR2015)
42
14
• No risks have manifested
themselves
• No use of MR

43. TSAS 2QTR2015 QUARTERLY
PROGRESS METRICS AND VIEWS
43

44. TSAS Weights vs. C/S Performance
(as of 2QTR2015)
44
10
13
The Story:
Contractor just passed
SFR (2/23/2015) and
engineers have
estimated the weight of
the sensor subsystems.
They are above the
Upper Control Limit so
BCWP < BCWS. Also,
contractor does not have
requisite SEs on board
and used more Senior
Engineers than planned
to develop estimates and
deliverables.

45. TSAS Planned vs. Actual Deliverables vs. C/S Performance
(as of 2QTR2015)
45
11
13
The Story:
Contractor was
able to deliver all
promised
documents on time
and thus improved
its cost and
schedule
performance.
However,
performance is not
perfect due to the
estimated weights
being above plan.
(see prior charts).

46. TSAS Planned vs. Actual FTEs
(as of 2QTR2015)
46
The Story:
Contractor needed
4.5 FTE Systems
Engineers during the
2Qtr2015, but only
had 2 on board. The
contractor used more
senior engineers to
estimate the weight
and produce the
deliverables. This
chart helps tell the
story of the negative
cost and schedule
performance for the
period.
12

47. TSAS Risk Status
(as of 2QTR2015)
47
14
• Lack of qualified SEs has impacted
deliverables and reliable estimate weights
• No use of MR as lack of SE has yet to impact
the EO/IR subsystem.

48. Schedule Margin Use
(as of 2QTR2015)
48
15
During this period, several SFR documents required more time and all the 15 days of
schedule margin were used to protect the SFR milestone date.

49. TSAS 4QTR2015 QUARTERLY
PROGRESS METRICS AND VIEWS
49

50. TSAS Weights vs. C/S Performance
(as of 4QTR2015)
50
10
13
The Story:
Contractor is working
through Preliminary
design process.
Estimated weight are
above UCL, so Limit so
BCWP < BCWS. Also,
contractor does not have
requisite SEs on board
and used more Senior
Engineers than planned
to develop estimates and
deliverables. C/S
performance improved,
due to more deliverables
being met. See next
slide

51. TSAS Planned vs. Actual Deliverables vs. C/S Performance
(as of 4QTR2015)
51
10
13
The Story:
Contractor is working
through Preliminary
design process.
Contractor misses
delivering the senor
subsystem preliminary
design documents.
BCWP < BCWS as a
result.

52. TSAS Planned vs. Actual FTEs
(as of 4QTR2015)
52
12
The Story:
Contractor misses
delivering the sensor
subsystem preliminary
design documents because
requisite qualified SE
required SE are still not on
board as planned. BCWP
< BCWS, and negative cost
variances continue as more
senior engineers are used
in lieu of SEs. 13

53. TSAS Risk Status
(as of 4QTR2015)
53
14
• FMV resolution risk resolved, but not until
late in 4QTR2015
• Mitigation strategy to add 1.5 FTE to resolve
SAR reliability issues worked as planned
• No MR used

54. TLO #1
Summary
• Proposed an enhanced set of key, or essential, program
management metrics that a government program
manager ought to have as a minimum to proactively
manage and help control contracted efforts
• Demonstrated selected metrics with a notional UAV
program
• Received suggestions for improvements
54

55. Questions???
55