The document discusses confidence level estimating and budgeting policies. It provides a recap of the joint confidence level (JCL) policy, which requires programs to be baselined at a 70% JCL and projects to be baselined/budgeted at a JCL that supports the program's approved level. It discusses the status of JCL calculations for various programs and projects, issues learned from implementing JCLs, and actions taken to address lessons learned to improve the process.

1. Confidence Level Estimating and Budgeting
February 2011
Tom Coonce

2. Topics
• A brief review of confidence level
estimating, budget-setting policy and
rationale
• Status of program and project Joint
Cost and Schedule Confidence Level
(JCL) calculations
• JCL issues and lessons learned
• Actions completed or planned to
address lessons learned
2

3. Recap of JCL Policy and Rationale
• NPD 1000.5
– Programs are to be baselined at a 70% joint confidence level
– Projects are to be baselined/budgeted at JCL that supports
the program approved JCL
– Projects are to be funded at no less than a 50% JCL
– JCLs are to be developed and maintained through launch
– Program and project proposed cost and schedule baselines
will be assessed by an independent review team
• Policy Rationale
– Improve cost and schedule performance
– Improve expectations with stakeholders
– Improve project planning
– Improve understanding of project plans
3

4. JCL – Defined by a Product, Refers to a
Process
• JCL = Joint Confidence Level
– Identifies probability that a given project or
program’s cost will be equal or less then
the targeted cost AND the schedule will be
equal or less then the targeted schedule
date
– Per NPD 1000.5 used at KDP-C to set
Program/Project Budget Baseline
• JCL is More than the Scatter Plot
– It’s a systematic process of integrating cost, schedule, and risk
– Provides a cohesive and holistic picture of the project or program’s ability to
achieve cost and schedule goals and to help the determination of reserves
(schedule and cost)
– Provides key Decision Support information
– Does the project have enough funds?
– Can the project meet the schedule?
– What are areas of risk toward successful execution of the project?
– What risk mitigation strategies provide the best project benefit?
– What are project phasing (fund) needs? 4

5. Acceptable JCL Methods
• Resource-Loaded Schedule with Monte-Carlo simulation, with
three different techniques
– Bottoms-up Resource Loaded Schedule or BURLS technique is
based on resource or cost-loaded schedule that focuses on the
project risk lists (known-known risks) and uncertainties (known and
unknown unknowns) to set the cost and schedule possibilities
– Cost Estimate/Program Assessment (CEPA) technique is based on
a top-level resource or cost loaded schedule that focuses on project
performance as well as project risks to set cost and schedule
possibilities
– Project Risk Evaluation Process or PREP technique is based on
the projects’ top-level resource or cost loaded schedule that
focuses on the project’s risk list as well as the views of an
Independent panel of experts to set cost and schedule possibilities
• Parametric
– Uses cost and schedule estimating relationships, data dispersion,
and relationships between cost and schedule to determine the
overall cost and schedule target and associated JCL 5

6. An Example of JCL Output
Vertical bar indicates
x-axis (Schedule)
Confidence level
Points represent
cost/Schedule result
pair from each iteration
of the simulation
Lower quadrant
Frontier line identifies
identifies iterations
cost/schedule surface
satisfying cost and
that meets Target JCL
schedule targets ~
the JCL
Horizontal bar
indicates Y-axis
(Cost) confidence
level
6

7. Management Councils Revisions the
Policies and Expectations
• Eliminated the requirement at KDP-B and D, but stipulated that KDP-Cs
JCL must be based on probabilistic resource loaded schedules (May
2009 SMC)
– Directed pilot projects be completed for projects at KDP-B
• Eliminated the requirement to maintain the JCL during implementation
(November 2009 PMC)
– Project are not required to develop a JCL at any other time during
implementation if it is “on plan”
– Projects are to be monitored as part of the Baseline Performance Review
(BPR) process
– If a project is re-baselined, scored red in the BPR, or if BPR leadership
directs, it must re-compute a JCL
– The agency is to use existing metrics and/or establish new factors for
determining when a project turns Red
• Eliminated JCL requirement for loosely coupled programs, retained the
requirement for tightly coupled programs, and required MDs to submit
program impact assessment whenever a new project enters the program
(June 2010 PMC)
– Eliminate JCL language from NPD 1000.5; reconstitute and update in NPD 7120.5E 7

8. Management Councils Revisions the
Policies and Expectations (Continued)
• Include cost and schedule range
estimates and associated probabilities
for all projects at KDP-B with LCC
>$250M (June and Aug 2010 PMCs)
– Coordinate policy language and place into
7120.5E
• Projects must develop, explain and
defend plans and associated probability
calculations at KDP-B and C (June
2010 PMC)
8

9. Final Proposed 7120.5E Language
(After the 4 August 2010 PMC)
Under 2.5.4.2 Confidence Level Estimating and Budgeting
2.5.4.2.4 Projects with an expected life cycle cost in excess of $250
million are required to develop cost and schedule estimates and
associated confidence levels for meeting those estimates.
(1) At KDP B, a range of cost and schedule estimates and
associated confidence levels of meeting those estimates will be
provided at KDP-B for projects and single-project programs (see
the footnote for paragraph 2.5.4.2.2). This policy applies to both
competed (AO) and directed (non-AO) missions. The decision
authority will use this information to guide the formulation
activities and to establish the target cost and schedule ranges
and resources phasing until KDP-C estimates are developed..
(2) At KDP C, projects are budgeted to a 70 percent joint cost and
schedule confidence level or as approved by the decision
9
authority.

10. So, What is Required to Meet the
KDP-B Programmatic Requirements?
• For non-AO missions, projects must provide:
– A “high” development (thru launch) cost estimate and
associated probability to achieve that cost
– A “low” development cost estimate and associate probability
to meeting that cost
– Cost estimates must include all cost elements, including
those for which the project managers is not directly
responsible, e.g. the launch vehicle and external partners
contributions
– High and low schedule durations (thru launch) and
associated probabilities of meeting those durations
– A high level schedule of activities associated with the low
and high life cycle cost estimates
– High and low life cycle cost estimates
– Year-by-year required resources to meet both the low and
high estimates 10

11. Programmatic Expectations at KDP-B
Non-AO Missions (Concluded)
– Basis of Estimates. Examples include:
• How the cost and schedules were derived
• How risks (known and unknown were addressed in the
estimates)
• How the year-by-year resources were determined
• For AO missions, the mission directorate, or
designated entity, will provide the same information
and calculations specified for the Non-AO missions to
the decision authority
• Expect to set budget at higher cost range number,
but set development cost and schedule estimates at
lower numbers
– UFE held by program or mission directorate
– AO mission development and life cycle costs based on
winning bid 11

12. Status of Program and Project JCLs
• During past two years, the following JCLs
were completed and presented at PMCs
– LDCM, MMS, MSL, SOFIA, GPM*, MAVEN,
LADEE, NuStar, and IRIS,
• Following JCLs are still in process
– JWST
• Pilot projects at KDP-B
– CxP (cost loaded schedule)
– SMAP (parametric)
– MAVEN (parametric)
* GPM used the CEPA technique; remainder used BURLS 12

13. JCL Implementation Issues
• Most projects had not assembled multiple schedules into a top-level
electronically integrated schedule. Required extensive time and effort
• Most projects had not quantified their discrete risks nor were they linked to their
cost or schedule estimates. Required extensive time and effort
• Few projects had a basis for their cost estimates. Most were based on assigned
budgets. Costs were usually not separated into fixed and variable
• Most projects had lower level schedule of activities detail than costs
• Project personnel were not staffed or trained to perform the analysis or use the
tools
• Use of low cost COTS scheduling and simulation software were inadequate to
perform the task
– Microsoft Project with @Risk took too long to perform computations
• Projects and Standing Review Boards had relatively narrow opinions about
discrete risks (Known-knowns)
• Consultants were less than successful in convincing projects or SRB members
to add other uncertainty (due to known and unknown-unknown risks)
In spite of these issues, most projects developed improved plans and
explained them better than without the JCL requirement 13

14. JCL Lessons Learned
• Improve project control functions
– Ensure projects develop credible cost and schedule basis of
estimates
– Ensure projects maintain an integrated master schedule that
reflects costs and schedule risks, i.e., maintain resource-loaded
schedules
– Ensure projects continuously quantify and monitor risks and their
impact on their plans
• Provide projects with historical cost and schedule
uncertainly data
– To assembly level if possible, but to the subsystem level at a
minimum
• Provide projects with launch vehicle price and schedule
uncertainty data
• Provide projects with tools that will permit required
turnaround times 14

15. Action Completed and Planned
• Completed
– Created near term optimization techniques for MS Project/@Risk
• Schedule Shorting Tool
• Risk Scenario & Automation Tool (R-SAT)
• In-Process
– Developing rapid stand alone simulation tools to be used in
conjunction with MS Project (replaces @Risk)
• DICE
• JACS
– Collecting cost and schedule uncertainties data
– Collecting LV price and schedule uncertainty data
– Brainstorming program control revitalization plan
• To Be Done:
– Deploy new simulations tools
– Disseminate LV uncertainty data
– Disseminate cost and schedule uncertainty data and educate
project staffs on their use 15
– Decide on program control improvements and implement

16. Backup

17. Detailed Program and Project Status
on JCLs
17

18. Detailed Program and Project Status
on JCLs (Concluded)
18