1. Arthur B. Chmielewski
Project Manager
Jet Propulsion Laboratory
California Institute of Technology
Presentation at the 7th NASA Project Management Challenge
February 9, 2010
Used with permission
Copyright 2010 California Institute of Technology 1

2. Many results presented here are based on the research
conducted by Dr. David Ullman of Robust Decisions Inc. for the
Space Technology 6 mission.
McRisk studies were conducted by Charles Garner, JPL
2

3. Famous business consultants have one thing in
common. They develop an innovative premise,
theory or a method which allows solving
difficult business problems with sales,
personnel, marketing or product development.
Consultants explain their premise in a book
which validates it with dramatic examples from
life of famous corporations.
3

5. We NASA managers, highly technically oriented, feel that the
world of business consulting - marketing, sales or mergers -
does not overlap with ours.
• Can the method of selection of TV commercials for
American Idol help with vacuum testing of a deep space
probe?
• Could iPod sales be related to cost estimating for an
asteroid lander?
Not only is the answer to these questions “yes” but the wisdom
of top business consultants may be critical in the fight against
the biggest aerospace plague:
5

6. 257% 1100%
300%
846%
1000%
1400%
600%
230% 1400% 6

7. # of Reserve
under- • Reserve estimation errors are very
Missions
Studied
estimated high with exception of medium size
Mission type by
missions
Flagship
4 164%
( > $800M)
• Reserve estimation accuracy has not
Medium size
5 19% improved in the last 20 years
(< $300M)
Large Instruments
2 34%
• Reserve errors are not limited to one
(< $150M) organization, center, type of mission
System Technology
Experiments 3 131% • Only 3/38* missions did not exceed
(< $150M) the budget reserve allocation
Technology
Experiments 11 107%**
• Only 1/38 missions used a strict
(<$25M)
process to calculate the amount of
Small Technology
Experiments (<$5M) 13 315%** needed reserve
* 38 missions consisted of: 11 microgravity experiments, 13 In-Step experiments, 14 other NASA missions
** No reserve was planned for most technology experiments
7

8. 100%
90%
Needed Reserve
80%
70%
60%
50%
40%
30%
20%
10%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Data from C. Garner, et al, 2002
R. Metzger, et al, 2003 Planned Reserve 8
R. Schmitz, 1991

9. NASA DOD DOT USA Germany Europe Asia
Aeronautical Engineers Civil Engineers Campaign
Managers Party Planners Pool builders
Engineers overran construction of Burj Al Arab in 2009
Pharaohs needed more slaves to finish the pyramids in
3000 BC
Humans overrun!
9

10.  BUT WHY?
 THESE ARE THE USUAL POST-MORTEM
ANSWERS:
 Poor communication
 Underestimated challenges
 Funding interruption
 Bad things happened
 Unknown unknowns popped up
 Requirements not well understood
 Reserves not adequate
 Initial cost estimates not accurate
 Strategic misrepresentation of cost*
* A real phrase from a DOD cost analysis report
10

11.  We use data bases for cost estimates
 We have cost reviews
 We manage risks
 We use processes
 We use earned value
 We use parametric studies
 We try and try and try and…
…WE STILL OVERRUN!
11

12. Every overrun occurs at the end of the project but
originates years earlier with the first entry into
the Excel preliminary cost estimate spreadsheet.
This first entry is frequently wrong because its
author is under influence of many mostly
subconscious psychological factors.
If we could understand the psychology of cost
estimates, then maybe we could improve their
accuracy!
12

13.  NASA scientists and engineers know well that
psychology does not play any role in their
precise, factual, highly technical cost estimating
based on precise facts.
 That is a great news because according to top
consultants psychology plays a major role in
business at every level. Let’s look at the
psychological effects we are immune to.
13

14. In a simple test 215 people were asked how long it will take to clean
up a stack of dirty dishes. They were given a detailed list of the
dishes.
14

15. One group was asked:
The kitchen needs to be clean. Estimate how many
minutes it will take you?
Another group was asked:
The kitchen needs to be clean in 15 min. Estimate how
many minutes it will take you?
15

16. Anchored - 15 min:
mean = 17 min
std= 5 min Base:
0.09
mean = 32 min
0.08 std= 10 min
0.07
0.06
Probability
0.05
0.04
0.03
0.02
0.01
0
0 10 20 30 40 50 60
Time (min)
16

17. The quality of the questions determines the
quality of the answer. In cost estimating we
allow ourselves very imprecise questions:
Manager: “How long will it take to do thermal
vacuum testing?”
Engineer: “Oh, 5 weeks”
What does “5 weeks” convey?
 The average time is 5 weeks – 50% of the time it will
take longer, 50% shorter
 He is 90% sure can be done in 5 weeks
 Something else
17

18. In the dish washing experiment subjects were
asked either:
• Estimate how many minutes it will take you to
clean up the kitchen?
or
• Estimate how many minutes so that you are 50%
sure you will be finished cleaning up the kitchen?
or
• Estimate how many minutes so that you are 90%
sure you will be finished cleaning up the kitchen?
18

19. 50%: 90%:
mean = 18 min mean = 29 min
std= 6 min std= 8 min
Base:
0.07 mean = 32 min
std= 10 min
0.06
0.05
Probability
0.04
0.03
0.02
0.01
0
0 10 20 30 40 50 60
Time (min)
No one asked if a dishwasher was available
19

20.  If someone gives you something you give him
something back. If you do not follow this rule
you suffer social consequences.
 That is why you get:
 Free address labels
 2 free nights before your timeshare presentation
 “Diet coke case study”
 Cost estimating:
 I will give you a low estimate and you will give me a
job.
20

21.  My second offer is so much better than the first
that it cannot be declined.
 The zoo trip study
 Cost estimating:
 After the initial cost estimate is too high “scrub” the
costs and, voila, get a much lower number
 Identify descopes to feel better about strained cost
estimate
21

22.  There is a desire within human beings to be consistent
with their actions, beliefs and attitudes. It is easier and
quicker to make decisions based on our past conduct.
 Timeshare salesmen ask first if you like vacations
 Wii case study
 In cost estimating the engineer is first asked to give a
quick estimate. When asked later to revise it produces
a similar result.
22

23.  We conclude correctness by watching accepted
practices. This especially applies under conditions of
uncertainty and similarity.
 4/5 dentist recommend this gum for patients who chew gum
 “I like football and I switched to ATT.”
 In cost estimating we frequently hear:
 “Everybody in our section does cost estimating that way.”
 “We normally do not list any programmatic risks.”
 “The last mission had only 1 FTE for electronics.”
23

24.  We like people who are similar to us. Liking
greatly increases compliance. It is
subconscious.
 Tiger Woods sells Buicks
 Student fund raisers
 Car ads with models
 Cost estimating:
 Is the cost estimate affected by who requests it and
how the request is made?
 Is it affected by who will be your boss on the project
and if you like her?
24

25.  Subjects comply with requests from the
perceived authority. An authority possesses a
high level of knowledge and experience and
may benefit the subject.
 Robert Young, MD Sanka commercials
 Cost estimating:
 The manager asks: “Can you see if my $250k bogey
is a good estimate for this radiation test?”
 Would a cost estimate request from the Center
Director produce a different value than from a
colleague?
25

26.  People assign more value to anything when it
is scarcer. This principle also applies to
information.
 Limited, final release of a Disney movie
 Only 50 tickets remain…
 In cost estimating:
 How does instrument development compare for a
competed award vs. internal development
 How does the cost estimate compare when the
engineer is swamped with jobs vs. worried about no
new assignment on the horizon?
26

27.  People get a lot of comfort from detailed plans.
The more detail, the more comfort and feeling
of accomplishment and safety.
 Trouble is that plans rarely work and need to
be updated so frequently that instead of project
guidance they become project drag.
 Costs:
 Elaborate plans can increase costs
 Guiding principles are better than plans
 Simple laws frequently are better than processes
27

28.  Managers tend to be fascinated with how to do
things, how to change things, how to develop
processes. Frequently “hows” take precedence
of “whats”. What is our business, what is our
product, what is our uniqueness.
 Home Depot’s labor cutting initiative
 McDonalds – the largest leasing company
 In costing:
 We focus on following a process
 We miss What is the meaning of each number and
What are the assumptions
28

29.  Companies trust their experienced personnel to
develop similar products.
 Bad experiences are short term, good memories are
long term due to location in different parts of brain
 XBOX 360 vs. Zoom vs. iPod case study
 In developing cost estimates experts assume
that old experiences are applicable
 “Heritage”, “small modification”, “off-the-shelf” are
frequently words of doom
29

30.  Complex human psychology is at play in cost
estimating
 The effect of psychology on cost is rarely
acknowledged and thus pervasive
 Psychological effects are subconscious and
hard to correct
 Psychological effects skew initial cost inputs
which build up with the number of estimators
and subsystems
30

31. 31

32. 1. The atoms of cost estimate must be free of bias,
emotion and common cognitive errors caused
by human psychology
 This is not even acknowledged by estimators
2. Use proper tools for reality check of grass roots
 We are becoming excellent at this
3. Calculate needed budget reserve based on risk
 This is almost never done
32

33.  JPL and Robust Decisions Inc. are developing a
software tool which calculates budget reserve based on
risk.
 The tool removes the effects of human psychology by
asking precise cost questions from a menu of “worry
generators”.
 The tool has been validated by 6 NASA instrument
developments.
 We would welcome the opportunity to use it on your
mission.
33

34. Making Robust Decisions, David G. Ullman
Wired to Care, Dev Patnaik
Work Wanted, James W. Walker, Linda H. Lewis
Influence: Science and Practice, Robert Cialdini
The Three Laws of Performance, Steve Zaffron, Dave Logan
Buyology: Truth and Lies about Why We Buy, Martin Lindstrom, Paco Underhill
The Innovator’s Prescription, Clayton M. Christensen, Jerome H. Grossman, J.
Wong
Prove It before You Promote It, Steve Cuno, Michael Shermer
Harvard Business Review, Jan-Dec 2009
Rethink: A Business Manifesto for Cutting Costs and Boosting Innovation, Ric
Merrifield
34