1. Conceptual Design Optimism,
Cost and Schedule Growth Effects
Presented at the
2010 NASA Program Management Challenge
9-10 February 2010, Houston, Texas
Claude Freaner, Bob Bitten, Debra Emmons
Used with permission
1
3. Background
• Results from a recent study* of 10 NASA missions over the past decade
indicated that the average cost and schedule growth of these missions, over
and above programmatic reserves, was 76% and 26%, respectively,
measured from beginning of Phase B
• One potential causative factor postulated was the inherent optimism in initial
concept designs due to competitive pressures
• Inherent optimism can translate to the underestimation of the technical
specifications such as mass, power, data rate, and the complexity of a system
• Underestimation of these resources can lead to the underestimation of the
final cost of the mission since most cost models use some form of system
resources as a predictor of mission cost
• To compound problems, the desire to launch a system as early as possible, in
order to obtain science quickly, can lead to a success oriented schedule that
may be shorter than historical comparisons would indicate
• This combination of underestimated resources providing an optimistic cost
estimate basis combined with a success oriented schedule can contribute to
the observed history of cost and schedule growth
* “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and
Schedule Growth”, Freaner C., Bitten R., Bearden D., and Emmons D., May 2008
3
4. Study Approach
• For a set of 20 missions in the study, the mass, power, cost,
schedule and other parameters were identified at the beginning of
the Preliminary Design phase (NASA Phase B) of a mission*
• These values were then compared to values presented at the
Preliminary Design Review (PDR), Critical Design Review (CDR)
and at the time of launch to understand the growth over time of each
of these resources
• The resource growth is then compared to industry guidelines to
understand if these guidelines would have adequately predicted the
growth for the mission data set studied
• In addition, resource growth of different mission types and
correlation of cost growth in relation to different parameters is also
shown
* All parametric and programmatic data were obtained from NASA CADRe data"
4
6. Database Description:
20 Missions Represent a Wide Range of Recent NASA Missions
Key Launch Acquisition Number of
Planetary? Program Science Type Center(s) Year Type Instruments Comments
Advanced land imaging technology
EO-1 NMP Earth Science GSFC 2000 Competed 5
demonstrator
Collect samples of solar wind particles at
GENESIS X Discovery Planetary Science JPL 2001 Competed 4
L1 point and return them to Earth
• 5 Directed vs. GRACE ESSP Earth Science JPL 2002 Competed 6 Earth Gravity Measurement
15 Competed Spitzer
Physics of
the Cosmos
Astrophysics JPL 2003 Directed 4
IR space telescope, the last of the Great
Observatories
missions GALEX Explorers Astrophysics JPL/CalTech 2003 Competed 1 UV space telescope
SWIFT Explorers Astrophysics GSFC 2004 Competed 4 Gamma Ray burst detector
MESSENGER X Discovery Planetary Science APL 2004 Competed 7 Investigate Mercury
• 7 Planetary missions MRO X MEP Planetary Science JPL 2005 Directed 7 Investigate history of water on Mars
vs. 13 Earth or near- Deep Impact X Discovery Planetary Science JPL 2005 Competed 3 Comet impactor
Earth Orbiters Cloudsat ESSP Earth Science JPL 2006 Competed 1 Radar observation of clouds
2 spacecraft looking at solar dynamics -
STEREO STP Heliospheric Science GSFC/APL 2006 Directed 4
Earth leading and trailing orbits
CALIPSO ESSP Earth Science LARC 2006 Competed 3 Aerosols
• 7 Planetary Science New
X
New
Planetary Science APL 2006 Competed 7 Investigate Pluto
Horizons Frontiers
vs. 5 Astrophysics DAWN X Discovery Planetary Science JPL 2007 Competed 2 Investigate Ceres and Vesta protoplanets
vs. 5 Earth Science AIM Explorers Heliospheric Science LASP 2007 Competed 3 Aeronomy of Ice in Mesosphere
vs. 3 Heliophysics Fermi
(GLAST)
Physics of
the Cosmos
Astrophysics GSFC 2008 Directed 2 Gamma Ray Telescope
Interaction between solar wind and
missions IBEX Explorers Heliospheric Science GSFC 2008 Competed 2
interstellar medium
Kepler Discovery Astrophysics JPL 2009 Competed 1 Search for Earth-sized exoplanets
Robotic ESMD/Planetary
LRO X GSFC 2009 Directed 7 Origin of the Moon
Lunar Science
Carbon Dioxide Investigation. Mission
OCO ESSP Earth Science JPL 2009 Competed 1
failed due to launch vehicle failure
6
7. Updated 20 Mission Results Are
Similar to Initial 10 Mission Paper* Results
80% 76%
10 Mission Study
70%
20 Mission Study
56%
Average Percent Growth
60%
50% 43% 42% 41%
40% 37% 36% 38%
30%
20%
10%
0%
Mass Power 1 Cost Schedule
1 2 2
Note:
1) As measured from Current Best Estimate, not including reserves
2) As measured from baseline estimate, including reserves
* “An Assessment of the Inherent Optimism in Early Conceptual Designs and its Effect on Cost and
Schedule Growth”, Freaner C., Bitten R., Bearden D., and Emmons D., May 2008
7
9. Satellite Mass Growth Relative to CBE (%)
M
is
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
si
on
M #1
is
si
on
M #2
is
si
on
M #3
is
si
on
M #4
is
si
on
M #5
is
si
on
M #6
is
si
on
M #7
is
si
on
M #8
is
si
on
M #9
is
si
on
M #1
is 0
si
on
9
M #1
is 1
si
Average
on
M #1
is 2
si
on
M #1
is 3
si
Planetary Missions
on
M #1
is 4
Earth Orbiting Missions
si
on
M #1
is 5
si
on
Mass Growth Exceeds Typical Guidance:
M #1
is 6
si
on
M #1
is 7
si
on
M #1
is 8
si
on
M #1
Summary of Mass Growth from Start of Phase B
is 9
si
on
13 out of 20 Missions Exceed 30% Growth from Current Best Estimate
#2
0
A
ve
ra
ge
37%
10. Summary of Satellite Mass Growth Over Time
90%
Mission #1
Mission #2
80% Mission #3
Mission #4
70% Mission #5
Mission #6
Satellite Mass Growth over CBE (%)
Mission #7
60% Mission #8
Mission #9
Mission #10
50%
Mission #11
Mission #12
40% Mission #13
Mission #14 37%
Mission #15
30%
Mission #16
Mission #17
28%
20% Mission #18
Mission #19 19%
Mission #20
10% Average
0%
0%
ATP PDR CDR Launch
-10%
Majority of Mass Growth Occurs Primarily Prior to PDR
10
11. Satellite Power Growth Relative to CBE (%)
M
is
-20%
0%
20%
40%
60%
80%
100%
120%
140%
si
on
M #1
is
si
on
M #2
is
si
on
M #3
is
si
on
M #4
is
si
on
M #5
is
si
on
M #6
is
si
on
M #7
is
si
on
M #8
is
si
on
M #9
is
si
on
M #1
is 0
si
on
M #1
11
is 1
si
on
Average
M #1
is 2
si
on
M #1
is 3
si
on
Planetary Missions
M #1
is 4
si
on
Earth Orbiting Missions
M #1
is 5
si
on
M #1
is 6
Power Growth Exceeds Typical Guidance:
si
on
M #1
is 7
si
on
M #1
is 8
si
on
M #1
is 9
si
Summary of Power Growth from Start of Phase B
on
8 out of 19 Missions Exceed 30% Growth from Current Best Estimate
#2
0
A
ve
ra
ge
41%
12. Summary of Power Growth Over Time
140%
Mission #1
Mission #2
Mission #3
120%
Mission #4
Mission #5
Mission #6
Satellite Power Growth over CBE (%)
100%
Mission #7
Mission #8
Mission #9
80% Mission #10
Mission #11
Mission #12
60% Mission #13
Mission #14
Mission #15
41%
36%
40% Mission #16
Mission #17
Mission #18 19%
20% Mission #19
Mission #20
Average 0%
0%
ATP PDR CDR Launch
-20%
Majority of Power Growth Occurs Primarily Prior to CDR
12
13. Development Cost Growth over Baseline (%)
M
is
0%
20%
40%
60%
80%
100%
120%
140%
160%
si
on
M #1
is
si
on
M #2
is
si
on
M #3
is
si
on
M #4
is
si
on
M #5
is
si
on
M #6
is
si
on
M #7
is
si
on
M #8
is
si
on
M #9
is
si
on
M #1
is
si 0
on
M #1
is
si 1
on
M
13
#1
is
si 2
on
Average
M #1
is
si 3
on
M #1
is
si 4
on
M #1
Planetary Missions
is
si 5
on
M #1
Absolute Growth in $M
Earth Orbiting Missions
is
si 6
on
M #1
is
si 7
Development Cost Growth is Significant:
on
M #1
is
si 8
on
M #1
is
si 9
on
Average Cost Growth, Over & Above Reserves, is 56%
#2
0
A
ve
ra
Summary of Cost Growth from Start of Phase B
ge
56%
$-
$40
$80
$120
$160
$200
$240
$280
$320
$360
$400
Development Cost Growth over Baseline ($M)
14. Summary of Cost Growth Over Time
150%
Mission Development Cost Growth over Baseline with Reserves (%)
Mission #1
Mission #2
Mission #3
125%
Mission #4
Mission #5
Mission #6
100% Mission #7
Mission #8
Mission #9
Mission #10
75% Mission #11
Mission #12
Mission #13
Mission #14
56%
50%
Mission #15
Mission #16
Mission #17
25% Mission #18
Mission #19 14% 21%
Mission #20
Average
0%
0%
ATP PDR CDR Launch
-25%
Cost Growth Occurs Primarily After PDR:
More than Half of Growth (14% to 56%) is Realized After PDR
14
15. Development Schedule Growth over Baseline
(%)
M
is
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
si
on
M #1
is
si
on
M #2
is
si
on
M #3
is
si
on
M #4
is
si
on
M #5
is
si
on
M #6
is
si
Average
on
M #7
is
si
on
M #8
is
si
on
M
Planetary Missions
is #9
si
on
M
Earth Orbiting Missions
#1
is 0
si
on
M
Absolute Growth in Months
#1
is 1
si
on
15
M #1
is 2
si
on
M #1
is 3
si
on
M #1
is 4
si
on
M #1
is 5
si
on
M #1
is 6
Schedule Growth is Significant:
si
on
M #1
is 7
si
on
M #1
is 8
si
on
M #1
is 9
si
on
Average Schedule Growth is on the order of 18 Months
#2
A 0
ve
ra
ge
38%
Summary of Schedule Growth from Start of Phase B
0
5
10
15
20
25
30
35
40
45
50
Development Schedule Growth over Baseline
(Months)
16. Summary of Schedule Growth Over Time
100%
Mission #1
Mission #2
Mission #3
Mission Development Schedule Growth over Plan (%)
Mission #4
80%
Mission #5
Mission #6
Mission #7
Mission #8
60% Mission #9
Mission #10 18 months
Mission #11 5 months 8 months
Mission #12
40% Mission #13
Mission #14
38%
Mission #15
Mission #16
Mission #17
20%
Mission #18 17%
Mission #19
Mission #20 10%
Average
0%
ATP PDR CDR Launch
-20%
Schedule Growth Occurs Primarily After PDR:
More than Half of Growth (13 of 18 Months) is Realized After PDR
16
18. Cost & Schedule Still Show Significant Growth from
Baseline Established at PDR
60%
56%
From Phase B Start
50% From PDR
Average Percent Growth
41%
40% 37% 37% 38%
30% 27%
19%
20% 15%
10%
0%
2
Mass Power Cost Schedule
1 1 2
Over Half of Uncertainty for Mass & Power is Retired by PDR
while 2/3 of Cost & Schedule Uncertainty Remain
Note:
1) As measured from Current Best Estimate, not including reserves
2) As measured from baseline estimate, including reserves
18
19. Data Shows that Programmatic Baseline Maturity
Lags Technical Design Maturity
Programmatic Realization
60% Not Until Launch
Mass
50% Power Technical Design
Average Percent Growth
Mature by CDR
Cost
40% Schedule
30%
20%
10%
0%
Phase B Start PDR CDR Launch
19
20. Comparison of Competed vs. Directed Missions:
No Significant Difference in Results
70%
Average Percent Growth from Phase B Start
Directed (5)
60% 58%
Competed (15)
51%
50%
42% 40%
40%
40% 36%
29% 31%
30%
20%
10%
0%
Mass Power 1 Cost Schedule
1 2 2
Growth in All Resources are Similar Between Competed & Directed Missions
Note:
1) As measured from Current Best Estimate, not including reserves
2) As measured from baseline estimate, including reserves
20
21. Planetary vs. Earth Orbiting Missions:
Resource Growth is Similar for Mass & Power
80% 73%
Average Percent Growth from Phase B Start
Planetary (7)
70%
Earth Orbiting (13)
60%
50% 47%
40% 40%
40% 35% 37%
30% 25%
22%
20%
10%
0%
Mass Power 1 Cost Schedule 2
1 2
Cost & Schedule Growth for Planetary Missions is Significantly Less
Note:
1) As measured from Current Best Estimate, not including reserves
2) As measured from baseline estimate, including reserves 21
22. Correlation Between Cost and Schedule Growth Indicates that
Schedule Growth Has Strong Influence on Cost Growth
140%
Development Cost Growth from Phase B Start
120%
% Cost Growth = 1.4775 * % Schedule Growth
100%
R2 = 0.6166
80%
60%
40%
20%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Schedule Growth from Phase B Start
22
23. Payload Mass and Cost Growth Significantly Greater than
Spacecraft Mass & Cost Growth
120%
Average Percent Growth from Phase B Start
Payload 101%
100% Spacecraft
80%
60%
60%
44%
40% 33%
20%
0%
Mass Cost
1 1
Data Indicated Payload Resource has Greater Uncertainty than Spacecraft
Note:
1) As measured from Current Best Estimate, not including reserves
23
24. Cumulative Cost Distribution Shows Wide Distribution of
Payload Cost Growth vs. Spacecraft Cost Growth
100%
90%
Cumulative Distribution for All Data Points
80%
70%
Payload Cost Growth has
60% Much Greater Variance
than Spacecraft Growth
50%
Spacecraft Cost Growth
40%
Payload Cost Growth
30%
20%
10%
0%
-25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250%
Cost Growth from Phase B Start over Initial Estimate without Cost Reserve
24
25. Instrument Payload Mass/Cost Growth Data Over Time Also
Indicates Payload Cost Lags Payload Maturity
120%
Percent Growth over CBE w/o reserves
100% Payload Mass 101%
Payload Cost
80%
60%
60%
43%
40% 31% 37%
20% 18%
0%
0% 0%
Phase B Start PDR CDR Launch
Payload Cost Significantly Lags Payload Design Maturity
25
26. Range of Mass Growth at Instrument & Subsystem Level
Can Provide Guidance for Initial KDP-B Cost Estimates
Typical Mass Reserve
Instruments
Design Guidance 30%
Propulsion
Below Mean
Above Mean ADCS
Thermal
Subsystem Min Mean Max Median
Comm
Instrument 11% 60% 202% 51%
Propulsion -73% 10% 69% 18%
ADCS -21% 29% 108% 24%
Thermal -36% 77% 269% 58% C&DH
Comm -55% 5% 76% 7%
C&DH -37% 16% 104% 6%
Power -37% 45% 103% 40% Power
Structure & Mech -19% 57% 142% 60%
Struct & Mech
-75% -50% -25% 0% 25% 50% 75% 100% 125% 150% 175% 200% 225% 250% 275%
Range for Mass Distributions
Range of Mass Growth is Large and Exceeds Typical Industry Guidance
26
28. Potential Considerations
• For Project Managers
– Mass and power reserve guidelines for spacecraft and payload
could be increased to be more consistent with historical
averages
– Schedule growth seems to be a key factor in controlling cost
growth
– More emphasis could be placed on early payload designs as
there is much greater uncertainty in the payload development
than the spacecraft
• For Cost Analysts
– Wider ranges of input parameters could be used to provide more
robust initial cost estimates which can address uncertainty in
early design
28
29. References
1) Freaner C., Bitten R., Bearden D., and Emmons D., “An Assessment of the Inherent Optimism
in Early Conceptual Designs and its Effect on Cost and Schedule Growth”, 2008
SSCAG/SCAF/EACE Joint International Conference, Noordwijk, The Netherlands, 15-16 May
2008.
2) Bitten R., Emmons D., Freaner C., “Using Historical NASA Cost and Schedule Growth to Set
Future Program and Project Reserve Guidelines”, IEEE Aerospace Conference, Big Sky,
Montana, March 3-10, 2007.
3) Bitten R.E., “Determining When A Mission Is "Outside The Box": Guidelines For A Cost-
Constrained Environment”, 6th IAA International Low Cost Planetary Conference, October 11-
13, 2005.
4) Bitten R.E., Bearden D.A., Lao N.Y. and Park, T.H., “The Effect of Schedule Constraints on the
Success of Planetary Missions”, 5th IAA International Conference on Low-Cost Planetary
Missions, 24 September 2003
5) Emmons D., “A Quantitative Approach to Independent Schedule Estimates of Planetary &
Earth-orbiting Missions”, 2008 ISPA-SCEA Joint International Conference, Netherlands, 12-14
May 2008
29
