Overruns or Underestimates?  "... their judgment was based more on blindwishing than upon any sound prediction; for it ist...
Estimation politics The purposeful shrinking or padding of an estimate to achieve an objective or favor a course of actio...
Loss leader strategy ≠ Target costing ≠ Political                   behaviorLoss leader strategy     Target costing       ...
Price ≠ Budget ≠ Cost ≠ Estimate                         Estimate                            – A range of values within ...
Agenda The cone of uncertainty revisited What is the evidence? Successful projects, price and cost, the consequences of...
The cone of uncertaintyRelative Size                                                                                      ...
Some plausible explanations Cognitive bias   – Overconfidence   – Anchoring   – ……. Individual & administrative behavior...
Cognitive biasOverconfidence                                                    Anchoring   Variability and Calibration of...
Individual & administrative behaviorsParkinson’s Law                                                Student syndrome 0.1  ...
Cognitive bias vs. political behavior as cause ofchronic underestimations                                                 ...
So, what is the evidence?                            © Eduardo Miranda, 2012
"The Montreal Olympics can no more have adeficit, than a man can have a baby"                     Montreal won the bid fo...
Montreal’s Olympic Stadium and Toronto’s CNTower                                     •Completed in 1976                   ...
The C-130 Avionic Modernization Program (AMP)according to GAOGAO-08-467SP Assessments of  Major Weapon Programs           ...
C-130 AMP                        According to the C-130                        AMP PO the amount of                       ...
The EMS Commercialization Project                                    © Eduardo Miranda, 2012
EMS Project Migrate 2,000,000 SLOC of application code from mainframe shared memory to client-server architecture   – For...
Sensitivity chart for EMS                        Unfeasible      Need to rewrite                          CPF        MPT  ...
© Eduardo Miranda, 2012
ERP Implementations                                 Medium Organizations                                $m                ...
The Validation Of A Political Model Of InformationSystems Development Cost Estimating, A. Lederer andJ. Prasad, 1991      ...
Software Management And Cost Estimating Error, A.  Lederer and J. Prasad, 1999     Uses                                   ...
Other studies Any other cost estimation inhibitors? A. Magazinović & J. Pernstål,  Chalmers University, 2008    – The res...
Successful projects & the consequences         of underestimations                                   © Eduardo Miranda, 2012
Budget overruns are not the same as project                  failures Iridium    – 66 LEO satellites designed,      built...
Even if overruns do not equate to projectfailures, underestimations are not free: They affect which projects get done Ho...
Which projects get done?  “We have found it isn’t necessarily the best ones, but those projects for which proponents  best...
Olympic gold                        Atlanta               Sydney                 Athens      Beijing    London      Rio   ...
The Freiman Curve         Budget                           29                    © Eduardo Miranda, 2012
Underestimated projects cost more                                                                                   Fatigu...
Recovery cost                                                   Project Parameters                                        ...
Underestimations lead to system inefficiencies                                                   Number of                ...
What can be done?                    © Eduardo Miranda, 2012
Before You Start Attachment to issues have career consequences. How do you expect your advocacy to affect your goals? Th...
Estimation Perspectives  Political: There are multiple and distinct interests in  organizations, each pursuing its own, oc...
Your options Reduce political influences Compensate for political influences                                        © Ed...
Reducing political influences (The inside view) Visibility    –   What is the work to be done?    –   What risks were con...
Benchmarking & Compensating (The outside view) Rolls Royce’s Lead Time Model Class reference forecasting                ...
Lead Time Model – Rolls Royce Controls   A. Powell, Right on Time: Measuring, Modelling and Managing Time-Constrained Soft...
Class Reference Forecasting                              © Eduardo Miranda, 2012
Uplift curves                                          Project sponsor must                                         presen...
Reference Class Forecasting for DoD Projects – Cost Correction                                             Cost Overruns &...
Reference Class Forecasting for DoD Projects – ScheduleCorrection                                         Schedule Overrun...
Applying Reference Class Forecasting to   software development and information systems   acquisitions                     ...
Summary Prices and targets are not estimates. They should be recognized as such and the cost of following a given strateg...
The estimationproblem isn’tunsolvable, wejust need tounderstand whatis involved andperhaps use it asa competitiveadvantage...
Upcoming SlideShare
Loading in …5
×

Eduardo Miranda (Carnegie Mellon) Overruns or Underestimates? A Political Perspective on Software Cost Estimation

1,580 views

Published on

Primera jornada en España especializada en estimación software

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
1,580
On SlideShare
0
From Embeds
0
Number of Embeds
364
Actions
Shares
0
Downloads
0
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide
  • Ultimately time, resources and money are the three criteria by which society and organizations decide what gets build, how it is built and by whom and in consequence, the importance of having credible and reliable estimates cannot be overstressed. Unrealistically small budgets as well as unnecessarily large ones result in misallocation of resources and wasteful spending.
  • The effort, time and resource estimates for a software project are the range of values within which, an organization believes it is possible for it to achieve the objectives of the project with a defined probability and without jeopardizing its viabilityEstimates, are typically used to:Determine the economic feasibility of a projectEvaluate alternativesEstablish a project budgetAn estimate is merely a prediction of what is most likely to happen. There is no implication that the estimator will attempt to shape events so that the estimate is materializedAn estimate and a budget are two different things
  • What are possible explanations for this?
  • Cognitive bias -> The estimates are biased but we don’t do it on purposeIndividual and organizational behaviors -> The estimates are not bias but we don’t observe the overestimations because the behaviors result in the spending of all the effort associated with themPolitical behaviors -> Strategic intent
  • The more important and bigger the project, the less the cognitive bias can be used as an explanation for bad estimates. The reason I make this point here is so you understand the importance of creating a defensible – self evident estimate
  • We examined software project bids made by 35Norwegian and international software companies. Thebidding companies consisted of large, medium, andsmall development companies operating in Norway.All bids are in NorwegianKroner (1000 NOK is about $140 – October 2004).The mean value of the bids was 220 000 NOK, rangingfrom 21 000 to 560 000 NOK. As far as we haveobserved in other bidding rounds, this range of bids isnot uncommon when the number of bidders is high.
  • Who prepares the estimates? The advocates (Olympic committees)
  • The Freiman curve links the actual cost to the budget selected for the project. Frank Freiman, its inventor, was for many years the head of cost estimating for RCA and the developer of the FAST cost-estimating system. Succinctly stated:The greater the underestimate, the greater the actual expenditure;The greater the overestimate, the greater the actual expenditure;The most realistic estimate results in the most economical project cost.Small budgets (prices) may land a contract or result in a project approval, but they also frequently lead to financial loss and business failure. Initial project plans of staffing, scheduling, machine processing, tooling and materials' forming, etc., are not achievable. Though the project plan is established to realize the underestimated cost, the project mid-point management begins to realize that milestones and schedules are slipping. In response, there is reorganization, replanning, and possibly the addition of personnel and equipment. Delays and reorganization invariably increase costs.The cost to the organization is also high in other ways, including poor morale and the loss of capable and trained staff. Projects that suffer significant cost growth are often projects scheduled, planned, and staffed based on early underestimates, that eventually lead to a detailed project plan that simply cannot be realized. Underestimates threaten an organization's ability to survive.Needlessly high budgets serve an organization as poorly as the underestimate. Rather than resulting in greater profits, as one might hope, overestimates conjure up Parkinson's law: the money is available, it must be spent. Unless there is firm management control, the estimate becomes a self fulfilling prophesy and the organization becomes weak, unable to deliver a good product for a reasonable price.Realistic estimates result in the most economical cost. They remind managers to control the excess resources. Good estimates let the organization's resources work in harmony.The Miranda curveThe point here is everybody can produce an accurate estimate by making it arbitrarily large. However if the budget is to high you might jeopardize the feasibility of the project, in other words your client chooses another provider or you don’t get the funds.Budget choices influence the cost of the projectOn the left part of the curve we see that the effort necessary to maintain the commitments made is greater than if the work had been planned from the beginningOn the right size part of the curve we see the consequence of choosing an unnecessarily high budget. The project becomes too expensive and in consequence it does not get funds or another supplier is selected
  • Eduardo Miranda (Carnegie Mellon) Overruns or Underestimates? A Political Perspective on Software Cost Estimation

    1. 1. Overruns or Underestimates? "... their judgment was based more on blindwishing than upon any sound prediction; for it isthe habit of mankind to entrust to careless hopewhat they long for, and to use sovereign reason to thrust aside what they do not desire." Thucydides (c. 460 BC – c. 395 BC) A Political Perspective on Software Cost Estimation Eduardo Miranda Carnegie Mellon University October 15th, 2012 © Eduardo Miranda, 2012
    2. 2. Estimation politics The purposeful shrinking or padding of an estimate to achieve an objective or favor a course of action not warranted by the unaltered estimate – Shrinking consists in reducing the estimate below what the estimator honestly believes will be required – Padding refers to the practice of increasing the estimate beyond what the estimator honestly believes will be required to develop a system An adjustment to compensate for what you think others are or will be doing © Eduardo Miranda, 2012
    3. 3. Loss leader strategy ≠ Target costing ≠ Political behaviorLoss leader strategy Target costing © Eduardo Miranda, 2012
    4. 4. Price ≠ Budget ≠ Cost ≠ Estimate  Estimate – A range of values within which an organization believes it is possible for it to achieve the objectives of the project with a defined probability  Price – What the seller or developer will get for doing the work. Depending on the strategic intent of the seller it could be higher than the cost estimate, with the difference being the expected profit or mark-up or they could be lower, a loss, with the hope of entering a new market, acquire a new technology or recoup it in follow-up work  Budget – A definite amount in terms of effort, schedule, and resources, hopefully chosen from the range of possible values defined by the estimate, to which the budgetee or executor of the project commits to  Margin – The desired profit  Cost – The amount of money or resources that has been used to produce or acquire something, and hence is not available for use anymore  Profit (or loss) – The difference between the price and the cost © Eduardo Miranda, 2012
    5. 5. Agenda The cone of uncertainty revisited What is the evidence? Successful projects, price and cost, the consequences of overruns What can be done? Conclusion © Eduardo Miranda, 2012
    6. 6. The cone of uncertaintyRelative Size Barry Boehm,Range Software Engineering Economics, Prentice Hall, 1981 Todd Little, Schedule Estimation and Uncertainty Surrounding the Cone of Uncertainty, IEEE Software, May/June 2006 Plans and Product Detailed Development Feasibility Requirements Design Design and Test Phases and Milestones © Eduardo Miranda, 2012
    7. 7. Some plausible explanations Cognitive bias – Overconfidence – Anchoring – ……. Individual & administrative behaviors – Student syndrome – Parkinson’s law Political behavior – Shrinking – Padding All of the above © Eduardo Miranda, 2012
    8. 8. Cognitive biasOverconfidence Anchoring Variability and Calibration of Expert Judgment in Software Anchoring and Adjustment in Software Estimation, J. Aranda, 2005 Estimation during the Bid Phase of a Project – An Exploratory Survey, Pedro Faria, University of Coimbra, 2011 © Eduardo Miranda, 2012
    9. 9. Individual & administrative behaviorsParkinson’s Law Student syndrome 0.1 1.80.09 1.60.08 1.40.07 1.20.06 10.05 0.8 Normal0.04 0.60.03 Parkinsons Law 0.40.020.01 0.2 0 0 10 15 20 25 30 Task duration © Eduardo Miranda, 2012
    10. 10. Cognitive bias vs. political behavior as cause ofchronic underestimations (Politics) (Cognitive bias) B. Flyvbjerg, From Nobel Prize To Project Management: Getting Risks Right, 2006 © Eduardo Miranda, 2012
    11. 11. So, what is the evidence? © Eduardo Miranda, 2012
    12. 12. "The Montreal Olympics can no more have adeficit, than a man can have a baby"  Montreal won the bid for the 1976 Olympic Games in May 1970, the estimated budget was $120 M  On Jan. 29, 1973 J. Drapeau, Montreal’s mayor announced that the games would cost $310 million  The 1972 Munich Olympics had cost around $600 million © Eduardo Miranda, 2012
    13. 13. Montreal’s Olympic Stadium and Toronto’s CNTower •Completed in 1976 •Budget 63 million •40,500 cubic meters of concrete •Headcount peak 1,500 •Cost per cubic meter of concrete•Original budget 130 million. $1,555•Final construction 770 millions•Total cost including repairs andinterest: 1.5 billion•Planned completion 1976•Completed 1987•400 000 cubic meters of concrete•Headcount at peak of construction10,000•Cost per cubic meter of concrete$1,925 © Eduardo Miranda, 2012
    14. 14. The C-130 Avionic Modernization Program (AMP)according to GAOGAO-08-467SP Assessments of Major Weapon Programs © Eduardo Miranda, 2012
    15. 15. C-130 AMP According to the C-130 AMP PO the amount of wiring and the number of harnesses and brackets needed for the installation had been underestimated by 400 percent GAO-08-467SPAssessments of Major Weapon Programs © Eduardo Miranda, 2012
    16. 16. The EMS Commercialization Project © Eduardo Miranda, 2012
    17. 17. EMS Project Migrate 2,000,000 SLOC of application code from mainframe shared memory to client-server architecture – Fortran – Assembly – Hard real-time – Custom interfaces to SCADA system – Complicated GUI – Safety critical application – Two organizations that have never worked together and with very different cultures (public service, business) located 600 km away – Productizing the system – Deployment in a foreign country – Generating documentation Constraints – 8 months from bid to promised date + 2 months ready on-site – 1.5 M budget (hardware excluded) © Eduardo Miranda, 2012
    18. 18. Sensitivity chart for EMS Unfeasible Need to rewrite CPF MPT Feasible © Eduardo Miranda, 2012
    19. 19. © Eduardo Miranda, 2012
    20. 20. ERP Implementations Medium Organizations $m Hardware –Application, Web, and database servers including 0.8+ storage Software ERP application Suite License –HR, Financials, Distribution $3.2+ –1,000 seats Implementation 9 months to complete pilot site including process engineering, apps configuration, and testing $9.3+ –30 external consultants @ $1,200 a day –30 internal staffers @ $100,000 Deployment –3 external consultants at 9 sites for 3 months –9 internal staffers at each site for 6 months –5 days of user training at an average burdened user $7.5+ salary of $50,000 –3 full-time training staff at an average burdened salary of $100,000 Total 20.8+ © Eduardo Miranda, 2012
    21. 21. The Validation Of A Political Model Of InformationSystems Development Cost Estimating, A. Lederer andJ. Prasad, 1991 © Eduardo Miranda, 2012
    22. 22. Software Management And Cost Estimating Error, A. Lederer and J. Prasad, 1999 Uses Practices Methods Result Justify using cost- Multiple benefit analysis (+) phasedStaff projects (+) estimatesMonitor project (+) User (+) management Schedule Structured signs-off estimate projects Estimate (+) (+)Select projects prepared Accuracy of proposal stage (-) (+) estimates is part Project of IS performance overruns Evaluate review estimators (+) Evaluate Estimates and developers actuals areCharge users (+) audited Accuracy of Informal estimates is part Software of users management and cost estimating performance error, A. Lederer & J. Prasad, review 1999 © Eduardo Miranda, 2012
    23. 23. Other studies Any other cost estimation inhibitors? A. Magazinović & J. Pernstål, Chalmers University, 2008 – The results of this study were used for validation of the results of the Lederer and Prasad study that was conducted in the form of a questionnaire, answered by 112 software professionals with a response rate of 28% – Pressures from manager, users or others to increase or reduce the estimate was fully validated by the issues found in both cases. – Removal of padding was partly validated by an issue found in the VCC case. Better sure than safe? Over-confidence in judgment based software development effort prediction intervals, M. Jørgensen et al, University of Oslo, 2002 – Hidden agendas: Software professionals have goals other than just a high correspondence between confidence level and hit rate. In particular, the desire to be evaluated as a skilled software developer may be an important agenda that leads to overly narrow effort Pis. – Project managers favor narrow intervals and high confidence: Study D indicates that most software developers preferred effort PIs that were much too narrow or based on a much too high confidence level, even when they knew that they were much too narrow When planners lie with numbers, M. Wachs, 1989 – Planning, however, is not just analytical. We work in the fishbowl of politics and public-policy making. We serve as staff to politicians, consultants to government bodies, and representatives of private landowners and real estate developers. These roles are usually associated with clearly articulated interests. Our agencies, employers, and clients favor particular policies or programs for reasons that may be derived more directly from ideology, political commitments, or economic self-interest than from the results of analytical studies. – I have experienced this conflict between planning as science and planning as advocacy in my own consulting, and have accumulated dozens of case studies from alumni who return to the university to talk about their anxieties and conflicts as professionals. © Eduardo Miranda, 2012
    24. 24. Successful projects & the consequences of underestimations © Eduardo Miranda, 2012
    25. 25. Budget overruns are not the same as project failures Iridium – 66 LEO satellites designed, built, launched, and operated successfully – Completed on schedule – Completed below budget -$5B – Bankruptcy – Entire system was sold for $25M in 2000  Titanic – Six months late – $100M budget, final cost $200M – Financial and creative blockbuster – Revenue of $1.8B © Eduardo Miranda, 2012
    26. 26. Even if overruns do not equate to projectfailures, underestimations are not free: They affect which projects get done How much we end up paying for them; and Might throw the organization into a vicious firefighting cycle © Eduardo Miranda, 2012
    27. 27. Which projects get done? “We have found it isn’t necessarily the best ones, but those projects for which proponents best succeed in conjuring a fantasy world ofunderestimated costs, overestimated revenues, undervalued environmental impacts and overvalued regional development effects” Machiavellian Megaprojects, Bent Flyvbjerg, 2005 © Eduardo Miranda, 2012
    28. 28. Olympic gold Atlanta Sydney Athens Beijing London Rio Year 1996 2000 2004 2012 2012 2016 AUS$ Euros Original US$ 800 US$ 15 US$ 2.4 US$14 1.7 1.5 budget millions billions billions billions billions billions Euros US$ Actual US$1.8 AUS$ 6 US $43* 12 15+ cost billions billions billions billions billions* Estimated. The actual cost has not been revealed by the Chinese government © Eduardo Miranda, 2012
    29. 29. The Freiman Curve Budget 29 © Eduardo Miranda, 2012
    30. 30. Underestimated projects cost more Fatigue Overtime (Eoa) Learning Effort contributed by bringing in additional personnel (Ea)FTE Fatigue Effort contributed thru overtime (Eob) Communication Overhead (Eco) Budgeted Effort (Eb) Ramp-up Coaching preparation Td Ta Tl Tb © Eduardo Miranda, 2012
    31. 31. Recovery cost Project Parameters Tb = 12 months Ta = 1 month Tl = 1 month FTEb = 20 people Teams = 4 Lag = 2 months Ci = 0.025 Cr = 0.1 Cc = 0.1 Cd = 0.1 Co = 0.0 Co = 0.0 2 b t b t 4a t c u, t FTEa t, u 2a t © Eduardo Miranda, 2012
    32. 32. Underestimations lead to system inefficiencies Number of + - + worked Fatigue Productivity hours - PM + intervention + + Number + - QA of Rework + activities defects + - Rate of - Project progress workload + + Other Resources + projects’ allocated to delays the project +/- - - + - Add - Organization Multitasking + resources al resource - + availability +Underestimation Management intervention + Scope + “Band + Organization Aid” cuts workload projects © Eduardo Miranda, 2012
    33. 33. What can be done? © Eduardo Miranda, 2012
    34. 34. Before You Start Attachment to issues have career consequences. How do you expect your advocacy to affect your goals? The introduction of an estimation or screening process will impose limits in otherwise discretionary decisions, and although people vary to the extent that they seek power, they rarely relinquish it voluntarily. There is not unequivocal substantive solution to the problem. Other parties have views and solutions of their own and they are not likely to support you until they recognize their own views in it. Not matter how good the process, it cannot succeed on its own. It depends on others who can amend, delay, obstruct and even block its deployment and operation. © Eduardo Miranda, 2012
    35. 35. Estimation Perspectives Political: There are multiple and distinct interests in organizations, each pursuing its own, occasionally parochial, objectives Cognitive: Our predisposition to judge future events in a more positive light than is warranted by actual experience Epistemological: What do we know? How? When? Models: Measurement errors, assumption errors, and scope errors © Eduardo Miranda, 2012
    36. 36. Your options Reduce political influences Compensate for political influences © Eduardo Miranda, 2012
    37. 37. Reducing political influences (The inside view) Visibility – What is the work to be done? – What risks were considered and which ones were excluded? – Worst, most likely and best case estimates for effort, cost and schedule – Safe estimates (x) for effort, cost and schedule – Staffing curves – Correlations between work elements Sanity checks – A final check that you should do whenever you solve a problem. It’s an informal test to see whether your answer makes sense in the context of the problem. It’s not detailed enough to verify that an answer is correct, but it is good at detecting logic or assumption errors which produce answers that cannot possibly be right Reliability – Given the same outputs two persons should arrive at more or less the same result for the same reason Traceability – It should be possible to understand how the outputs are related to its inputs. Not black magic. Advocates / Non-advocates estimates © Eduardo Miranda, 2012
    38. 38. Benchmarking & Compensating (The outside view) Rolls Royce’s Lead Time Model Class reference forecasting © Eduardo Miranda, 2012
    39. 39. Lead Time Model – Rolls Royce Controls A. Powell, Right on Time: Measuring, Modelling and Managing Time-Constrained Software Development , PhD. Thesis, University of York, 2001 © Eduardo Miranda, 2012
    40. 40. Class Reference Forecasting © Eduardo Miranda, 2012
    41. 41. Uplift curves Project sponsor must present justification for the uplift not to be applied The British Department for Transport Procedures for Dealing with Optimism Bias in Transport Planning Guidance Document June 2004 © Eduardo Miranda, 2012
    42. 42. Reference Class Forecasting for DoD Projects – Cost Correction Cost Overruns & Underruns For 45 DoD Projects According to  C130 – AMP GAO 2008 14 100.00% 90.00% proposed budget = 12 80.00% 3.9B 10 70.00%  Acceptable chance Frequency 60.00% 8 6 50.00% of cost overrun = 40.00% 30.00% 10% 4 2 20.00%  Required uplift = 0 10.00% 0.00% 190% -25 0 25 50 75 100 125 150 175 200 225 250 275 300 More  Uplifted budget = Percentage of Original Budget Frequency Cumulative % 11.3B Required Cost Uplift  Latest estimate with 350 half of original units 300 to be delivered = Original Budget Adjustment 250 5.3B 200  Latest estimate with 150 original number of 100 units to be delivered 50 = 12.5B 0 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Acceptable Chance of Cost Overrun Distribution Trendline © Eduardo Miranda, 2012
    43. 43. Reference Class Forecasting for DoD Projects – ScheduleCorrection Schedule Overruns & Underruns for 45 DoD Projects According to GAO 2008  C130 – AMP 14 100.00% proposed schedule 12 90.00% = 48 months 80.00% 10 70.00%  Acceptable chance Frequency 8 60.00% 50.00% of cost overrun = 6 40.00% 10% 4 30.00% 2 20.00%  Required uplift = 10.00% 0 0.00% 90% -10 0 10 20 30 40 50 60 70 80 90 100 More Percentage of Original Schedule  Uplifted schedule = Frequency Cumulative % 91 months Required Schedule Uplift  Latest estimate = 84 160 months 140Original Schedule Adjustment 120 100 80 60 40 20 0 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Acceptable Chance of Schedule Overrun Distribution Trendline © Eduardo Miranda, 2012
    44. 44. Applying Reference Class Forecasting to software development and information systems acquisitions Collect Pastcompleted project data Perfromance project data base Data A1 Select reference projects Referencequery Projects A2 Generate schedule reference distribution Class Reference Distributions cost reference distribution A3 cost uplift Calculate schedule uplift Upliftacceptable cost overrun chance Valuesacceptable delay chance A4 adjusted cost adjusted schedule Adjust Estimatesestimated costestimated schedule A5 © Eduardo Miranda, 2012
    45. 45. Summary Prices and targets are not estimates. They should be recognized as such and the cost of following a given strategy acknowledged Underestimations cause inefficiencies Political and cognitive biases distort the inputs used by models and determine the fate of its outputs When an estimate can only be explained by delusion or deception, it is safer to assume the latter © Eduardo Miranda, 2012
    46. 46. The estimationproblem isn’tunsolvable, wejust need tounderstand whatis involved andperhaps use it asa competitiveadvantage © Eduardo Miranda, 2012

    ×