Psychological Impacts on Judgment        in Cost Estimation              Jordan Garner            UC Davis (JPL Summer Hir...
Special Thanks to• Dr. David Ullman of Robust Decisions and Oregon State  University for his assistance with the web exper...
258%       275%                           1100%100%                                                  1500%               B...
Overruns Start with         Bad Initial Cost Estimates• Bad cost estimates are in every sector of business  world: constru...
Causes of Overruns• Overruns start with flawed initial cost estimates and  inadequate reserves.• However, the post mortem ...
Are Estimates Getting Better?“For the past 70 years, for which data on cost estimation isobservable, no significant improv...
Unaccounted Psychological Effects?• Thesis: Could humans be prone to psychological  factors that make them truly and hones...
Overheard During Cost Estimating:•   “I have a bogey of $400k. Please give me your own estimate.”•   “We will hold 30% res...
Dishwashing Experiment• Participants in the on-line experiment were asked in  different ways to estimate the time needed t...
10
Psychological Effects Tested• 5 psychological effects were tested :   1.   Anchoring   2.   Q&A Mismatch   3.   Decomposit...
All answers were graphed and     analyzed to establish conclusions908070605040                                       upper...
Effect #1: AnchoringThe objective was to test how easily influencedpeople may be by a wrong answer – “the anchor.”The anch...
1. Anchoring Results• The nominal value was 30 min, the anchored case 25 min.• The “best case scenario” estimate (describe...
Effect #2: Q&A MismatchThe purpose was to test if there is a mismatch between the typeof estimate expected and provided.Di...
2. Q&A Mismatch Results• The 50% confidence estimate was 31 min. The nominal  estimate was 30 min. People unconsciously in...
Effect #3: DecompositionThe objective was to test if decomposing the project intosmaller pieces and deeper levels of a WBS...
3. Decomposition Results• Decomposition average was 31 minutes, just one  minute longer than the nominal average (30 min)....
Effect #4: Reserve ComfortThis question tested the realism of “a comfortable” reserve.Respondents were asked:1. I am 90% s...
4. Reserve Comfort Results• The reserve for 90% confidence was 8 min or 28%. The 25-30% seems to be  the magical intuition...
Large Projects Reserve Comparison                 100%                 90%                                                ...
Effect #5: Planning FallacyThe planning fallacy, as defined by Daniel Kahneman and AmosTversk is a tendency to be overly o...
5. Planning Fallacy Results• The following results were obtained:   –   51 min worst case   –   45 min 99% confidence   – ...
Conclusions• To improve the quality of cost estimates it is recommended to  diminish the effects of psychological impact o...
Upcoming SlideShare
Loading in …5
×

Arthur.chmielewski

16,607 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
16,607
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
59
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Quote this, his authority is important
  • Arthur.chmielewski

    1. 1. Psychological Impacts on Judgment in Cost Estimation Jordan Garner UC Davis (JPL Summer Hire) Art B. Chmielewski Jet Propulsion Laboratory California Institute of Technology September 12, 2011 1
    2. 2. Special Thanks to• Dr. David Ullman of Robust Decisions and Oregon State University for his assistance with the web experiment and continued support of this novel research.• Prof. Don Forsyth of the University of Richmond for his expert consultation on socio-psychological effects in decision making. 2
    3. 3. 258% 275% 1100%100% 1500% Bad Cost Estimates Happen 1400% 600% 2200% 3 220% 1400%
    4. 4. Overruns Start with Bad Initial Cost Estimates• Bad cost estimates are in every sector of business world: construction projects, movie business, transportation projects, military programs, aerospace, etc.• Bad cost estimates know no borders, race, sex or century. 4
    5. 5. Causes of Overruns• Overruns start with flawed initial cost estimates and inadequate reserves.• However, the post mortem analyses give less blame to the estimating than to failures in execution such as: – Changes in scope and requirements – Inadequate communication – Government and contractor intervention – Unforeseen technical issues – New technology – Acts of god• Specific technical reasons for overruns seem to be more palatable than poor cost estimates. 5
    6. 6. Are Estimates Getting Better?“For the past 70 years, for which data on cost estimation isobservable, no significant improvements inforecasting, estimating or prediction a project’s cost have everbeen made. This is despite the increase in awareness of pastestimation inaccuracy, new strategies of estimation, the hiring ofmore experts to help the estimation process, inventions solvingpast technical and communication issues.”– Prof. Bent Flyvbjerg, at Oxford Universitys Saïd Business School 6
    7. 7. Unaccounted Psychological Effects?• Thesis: Could humans be prone to psychological factors that make them truly and honestly believe in poor estimates?• We conducted a simple experiment to test and quantitatively measure the power of psychological fallacies on people’s ability to make estimates. 7
    8. 8. Overheard During Cost Estimating:• “I have a bogey of $400k. Please give me your own estimate.”• “We will hold 30% reserve for you.”• “I sent you a WBS cost table. Can you fill it in?”• “We need your best estimate by Friday.”• You have an allocation of $1.3M, can you give me an estimate?Our simple experiment proved that the above common costingphrases guarantee overruns! 8
    9. 9. Dishwashing Experiment• Participants in the on-line experiment were asked in different ways to estimate the time needed to perform a simple task – washing the dishes shown on the next chart. 9
    10. 10. 10
    11. 11. Psychological Effects Tested• 5 psychological effects were tested : 1. Anchoring 2. Q&A Mismatch 3. Decomposition 4. Reserve Comfort 5. Planning Fallacy• Every respondent to the survey was randomly asked one of several questions testing different psychological heuristics or fallacies.• 507 volunteers participated: 142 JPLers, 305 college students and 60 other adults. ~2300 data points were collected. 11
    12. 12. All answers were graphed and analyzed to establish conclusions908070605040 upper Standard Deviation estimate30 lower Standard Deviation20100 Psychological category 12
    13. 13. Effect #1: AnchoringThe objective was to test how easily influencedpeople may be by a wrong answer – “the anchor.”The anchor set asked:Estimate how many minutes it will take you to clean thekitchen. One respondent estimated that it will takeabout 10 minutes to finish cleaning up. He may bewrong of course. 13
    14. 14. 1. Anchoring Results• The nominal value was 30 min, the anchored case 25 min.• The “best case scenario” estimate (described later) was 27 min which was 2 min LONGER than the anchored result.• The result points out that it is very easy to dramatically skew the estimates by asking anchored questions, such as: “We would like you to come in around $6M”, “I have a bogey of $400k for you”, “the last robot arm we built cost $7M”… 14
    15. 15. Effect #2: Q&A MismatchThe purpose was to test if there is a mismatch between the typeof estimate expected and provided.Different participants were asked:• Estimate how many minutes it will take you to clean the whole kitchen.• There is a 50% chance that you will finish this task within __ min• There is a 75% chance that you will finish this task within __ min• There is a 99% chance that you will finish this task within __ min 15
    16. 16. 2. Q&A Mismatch Results• The 50% confidence estimate was 31 min. The nominal estimate was 30 min. People unconsciously interpret the nominal as the 50% case, meaning that you will exceed your estimate in half the cases!• However, when a manager asks for an estimate he/she expects a much more reliable result, possibly in the 80%-90% confidence range. This points out that there is mismatch between the expectation and the answer. 16
    17. 17. Effect #3: DecompositionThe objective was to test if decomposing the project intosmaller pieces and deeper levels of a WBS improved accuracyof the estimate.Estimate decomposition was simulated by asking:1. How many minutes will it take to clean all the plates and the sets of silver?2. How long will it take to clean the sets of coffee cups and saucers?3. How long will it take to clean the bowls?4. Etc. 17
    18. 18. 3. Decomposition Results• Decomposition average was 31 minutes, just one minute longer than the nominal average (30 min). The attempt at becoming more accurate by cutting up the project was not accomplished.• Decomposition, at least in this case, was more time consuming than helpful.• Deep decompositions provide more detail but compound psychological effects. 18
    19. 19. Effect #4: Reserve ComfortThis question tested the realism of “a comfortable” reserve.Respondents were asked:1. I am 90% sure that the time it will actually take to clean the kitchen is within plus or minus __ minutes from my estimate. 19
    20. 20. 4. Reserve Comfort Results• The reserve for 90% confidence was 8 min or 28%. The 25-30% seems to be the magical intuitional comfort level that is used by many industries.• When a manager asks for a reserve he/she means “I want to be very sure that I will not exceed this reserve. I want my reserve to cover almost the worst case.”• However, that is not how it is interpreted by the employee. – The worst case estimate was 51 min and required 70% reserve. – The 99% confidence case averaged 45 min. and needed 50% reserve. Both of these cases are significantly higher than the popular 30%. 20
    21. 21. Large Projects Reserve Comparison 100% 90% The realistic amount ofNeeded Reserve 80% budget reserve required 70% for 18 large projects 60% studied is 52%. 50% 40% 30% 20% 10% Recent aerospace projects 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Planned Reserve 21
    22. 22. Effect #5: Planning FallacyThe planning fallacy, as defined by Daniel Kahneman and AmosTversk is a tendency to be overly optimistic in planning.To asses the extent of optimism we asked:1. In the best case scenario (if everything went as well as possible), how many minutes would it take you to clean the whole kitchen?2. In the worst case scenario (if everything went as poorly as possible), how many minutes it would take you to clean the whole kitchen? 22
    23. 23. 5. Planning Fallacy Results• The following results were obtained: – 51 min worst case – 45 min 99% confidence – 30 min nominal – 27 min best case• These results show how skewed people are toward optimism. The nominal estimate was 10% longer than the best case but 70% shorter than the worst case.• People are so optimistic that it was easy to anchor them down but anchoring up failed. 23
    24. 24. Conclusions• To improve the quality of cost estimates it is recommended to diminish the effects of psychological impact on judgment:  Train the managers not to anchor.  Establish proper Estimation Language which makes the questions compatible with common interpretation.  Deep decompositions do not improve accuracy.  Calculate the reserve based on risk.  Account for optimism by including in the baseline likely, historical and common risks. 24

    ×