Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Presentation on 'Why cant people estimate' event, 23rd June 2016


Published on

APM presentation given by David Simms in Coventry on 23rd June 2016.

Published in: Education
  • My brother found Custom Writing Service ⇒ ⇐ and ordered a couple of works. Their customer service is outstanding, never left a query unanswered.
    Are you sure you want to  Yes  No
    Your message goes here
  • To get professional research papers you must go for experts like ⇒ ⇐
    Are you sure you want to  Yes  No
    Your message goes here
  • Could you use an extra $1750 a week? I'm guessing you could right? If you would like to see how you could make this type of money, right from the comfort of your own home, you absolutely need to check out this short free video. ◆◆◆
    Are you sure you want to  Yes  No
    Your message goes here
  • Can you earn $7000 a month from home? Are you feeling trapped by your life? Stuck in a dead-end job you hate, but too scared to call it quits, because after all, the rent's due on the first of the month, right? Are you ready to change your life for the better? ♣♣♣
    Are you sure you want to  Yes  No
    Your message goes here
  • How to Manifest Anything with the Law of Attraction ▲▲▲
    Are you sure you want to  Yes  No
    Your message goes here

Presentation on 'Why cant people estimate' event, 23rd June 2016

  1. 1. Why Can’t People Estimate: Estimation Bias and Mitigation With SEER Introduction David Simms Galorath International © 2016 Copyright Galorath Incorporated
  2. 2. Key Points for today . . . Estimates can be better, reducing bias & strategic mis-estimation… Parametrics can help. © 2016 Copyright Galorath Incorporated 2 Tempering with an “outside view” can mitigate some bias Without care estimates are usually biased (even with experts)
  3. 3. Some of the most important business decisions about a project are made at the time of minimum knowledge and maximum uncertainty.
  4. 4. ESTIMATION & PLANNING: An Complete Estimate Defined • An estimate is the most knowledgeable statement you can make at a particular point in time regarding: • Effort / Cost • Schedule • Staffing • Risk • Reliability • Estimates more precise with progress • A WELL FORMED ESTIMATE IS A DISTRIBUTION 4
  5. 5. Human Nature: Humans Are Optimists Harvard Business Review explains this Nobel Prize Winning Phenomenon: • Humans seem hardwired to be optimists • Routinely exaggerate benefits and discount costs Delusions of Success: How Optimism Undermines Executives' Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003) 5 Solution - Temper with “outside view”: Past Measurement Results, traditional forecasting, risk analysis and statistical parametrics can help Don’t remove optimism, but balance optimism and realism
  6. 6. Why Estimate?
  7. 7. Estimation Methods 1 of 2 Model Category Description Advantages Limitations Guessing Off the cuff estimates Quick Can obtain any answer desired No Basis or substantiation No Process Usually Wrong Analogy Compare project with past similar projects. Estimates are based on actual experience. Truly similar projects must exist Expert Judgment Consult with one or more experts. Little or no historical data is needed; good for new or unique projects. Experts tend to be biased; knowledge level is sometimes questionable; may not be consistent. Top Down Estimation A hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component. Provides an estimate linked to requirements and allows common libraries to size lower level components. Need valid requirements. Difficult to track architecture; engineering bias may lead to underestimation.
  8. 8. Estimation Methods 2 of 2 Model Category Description Advantages Limitations Bottoms Up Estimation Divide the problem into the lowest items. Estimate each item… sum the parts. Complete WBS can be verified. The whole is generally bigger than the sum of the parts. Costs occur in items that are not considered in the WBS. Design To Cost Uses expert judgment to determine how much functionality can be provided for given budget. Easy to get under stakeholder number. Little or no engineering basis. Simple CER’s Equation with one or more unknowns that provides cost / schedule estimate. Some basis in data. Simple relationships may not tell the whole story. Historical data may not tell the whole story. Comprehensive Parametric Models Perform overall estimate using design parameters and mathematical algorithms. Models are usually fast and easy to use, and useful early in a program; they are also objective and repeatable. Models can be inaccurate if not properly calibrated and validated; historical data may not be relevant to new programs; optimism in parameters may lead to underestimation. 8
  9. 9. Improving Estimation Is Important • Estimation is critical for defining success for all types of projects • Many organizations treat estimation as a second rate activity and do not invest in the process • Everyone estimates….but the process and the quality or success rate varies greatly • Estimation and measurement go hand in hand – BOTH contribute to improvement • Having a repeatable estimation process is critical to both estimating AND to successful projects
  10. 10. Cognitive Bias: How Fair Are We (Source •Human beings exhibit inherent errors in thinking •Perception has more to do with our desires—with how we want to view ourselves—than with reality." Behavioral economist Dan Ariely Cognitive bias: Tendency to make systematic decisions based on PERCEPTIONS rather than evidence •Our brains using shortcuts (heuristics) that sometimes provide irrational conclusions Researchers theorize in the past, biases helped survival •from deciding how to handle our money •to relating to other people •to how we form memories“Bias affects everything: © 2016 Copyright Galorath Incorporated 10
  11. 11. The Planning Fallacy (Kahneman & Tversky, 1979) •Manifesting bias rather than confusion •Judgment errors made by experts and laypeople alike •Errors continue when estimators aware of their nature Judgment errors are systematic & predictable, not random •Underestimate costs, schedule, risks •Overestimate benefits of the same actions Optimistic due to overconfidence ignoring uncertainty •“inside view” focusing on components rather than outcomes of similar completed actions •FACT: Typically past more similar assumed •even ventures may appear entirely different Root cause: Each new venture viewed as unique © 2016 Copyright Galorath Incorporated 11
  12. 12. Reference Class Forecasting Attempt to force the outside view and eliminate optimism and misrepresentation Choose relevant “reference class” completed analogous projects Compute probability distribution Compare range of new projects to completed projects Provide an “outside view” focus on outcomes of analogous projects © 2014 Copyright Galorath Incorporated 12 Best predictor of performance is actual performance of implemented comparable projects (Nobel Prize Economics 2002 Predicts outcome of planned action based on actual outcomes in a reference class: similar actions to that being forecast.
  13. 13. Dealing With the “Problem of Assumptions” • Assumptions are essential Write them down but… Wrong assumptions can drive an estimate to uselessness • Use an assumption verification process © 2016 Copyright Galorath Incorporated 13 1. Identify assumptions 2. Rank order assumptions based on estimate impact 3. Identify high ranking assumptions that are risky 4. Clarify high ranking, high risk assumptions & quantify what happens if those assumptions change 5. Adjust range of SEER inputs to describe the uncertainty in assumption
  14. 14. We should always listen to the experts . . . . Shouldn’t we? © 2016 Copyright Galorath Incorporated
  15. 15. Trouble Starts By Bias or Strategic Mis- Estimation Ignoring Iron Triangle • Typical Trouble: Mandated features needed within specific time by given resources • At least one must vary otherwise quality suffers and system may enter impossible zone! Quality Resources Schedule Scope (features, functionality) Sometimes strategic mis-estimation is used to get projects started or to win Some customers think price to win is strategic mis- estimation (it is not)
  16. 16. Plans Based on Assumptions About Average Conditions Usually Go Wrong • Executive desire to “Show me the number” forces averages • When average is used to represent an uncertain quantity…. • it ends up distorting the results • Because it ignores the impact of the inevitable variations • Flaw may be fixed by tools such as Monte Carlo Analysis • Adds a distribution & probability to “the number” 16
  17. 17. Adding Reality to Estimates – Example – 2 (Source SEI) Step Best Expected Worst 1 27 30 75 2 45 50 125 3 72 80 200 4 45 50 125 5 81 90 225 6 23 25 63 7 32 35 88 8 41 45 113 9 63 70 175 10 23 25 63 500 What would you forecast the schedule duration to be now?
  18. 18. Monte Carlo Analysis To Quantify and Mitigate Bias 90% confidence, project under 817 days Almost guaranteed to miss the 500 days 50% confidence, project will be under 731 days
  19. 19. Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR) • Case Study: Levi Strauss • $5M ERP deployment contracted • Risks seemed small • Difficulty interfacing with customer’s systems • Had to shut down production • Unable to fill orders for 3 weeks • $192.5M charge against earnings on a $5M IT project failure “IT projects touch so many aspects of organization they pose a new singular risk”
  20. 20. Causes of Project Failure Source: POST Report on UK Government IT Projects Lack of a clear link between the project and the organisation’s key strategic priorities, including agreed measures of success. 1. Lack of clear senior management and ministerial ownership and leadership2. Lack of effective engagement with Stakeholders3. Lack of skills and proven approach to project management and risk management 4. Lack of understanding of and contact with the supply industry at senior levels within the organisation 5. Evaluation of proposals driven by initial price rather than long-term value for money (especially securing the delivery of business benefits) 6. Too little attention to breaking development and implementation into manageable steps 7. Inadequate resources and skill to deliver the total delivery portfolio8. Source: POST Report on UK Government IT Projects
  21. 21. Fallacy of Silent Evidence What about what we don’t know? How confident would you feel if the Silent Evidence was visible?
  22. 22. What is Parametric Estimating? • Parametric Approach: • Employs cost estimating relationships • Uses mathematical equations • Allows estimation based on measurement of key parameters that influence the relationship • Based on technical or physical characteristics of the product and production environment • SEER’s Parameters Cover 5 Key Areas • People • Process • Technology • Platform (Environment) • Size 22
  23. 23. Draw Out Range By Obtaining 3 Estimates • Optimistic value (sopt) • Most likely value (sm) • Pessimistic value (spess) • Expected value (EV) © 2016 Copyright Galorath Incorporated 23 6 )4( pessmopt sss EV  
  24. 24. Sophisticated Schedule Modeling: Essential For Viable Project Plans For a given Size, Complexity and Technology Work Expands To Fill Time (Work expands due to lack of pressure) Minimum Time To Complete (Effort reduces to reduce schedule) Optimal Effort for staff (Lower Effort for Longer Schedule) Effort Increase due to Longer Schedule Calendar Time EffortMonths Reasonable Solution Range Software Solution
  25. 25. Range vs. Point Estimates (Source US Army) RangeofRisk&Uncertainty Estimating Accuracy Trumpet Technical and Program Maturity ROM-30%to+75% Parametric-10%to+20% Analogy-15%to+30% Engineering -5%to+15% Actual -3%to +10% Target Cost Point estimate is most likely within range estimate with higher potential for cost increase Range estimate provides a degree of risk and uncertainty +75% -30% BA Materiel Solution Analysis Systems Acquisition Sustainment Technology Development Production & Deployment Operations & Support Engineering and Manufacturing Development C Pre-Systems Acquisition
  26. 26. What can a parametric model tell you? 26 What is likely to happen Feel lucky? Firm Fixed Price? Understand the risk before you commit!
  27. 27. Difference Between Bottoms Up and Parametric • “Bottom-up” or Manual Approach (Detailed Work Measurement) • Measures everything directly and exactly • Little or no analysis of alternatives • Predictive value is limited at best • Parametrics • Measures or estimates key parameter values and ranges • Understand (mathematically) their effect on time, effort • Once relationships are known, parameter values can be changed and the effects evaluated © 2016 Copyright Galorath Incorporated
  28. 28. Example: Parametric Estimate Compared With History
  29. 29. SEER MAJOR Components • SEER-SEM – software/application development, maintenance, integration and testing • SEER-H – system, hardware and electronics development, production and support • SEER-IT – IT infrastructure, services and operations and systems engineering • SEER-MFG – hardware manufacturing and assembly • SEER-SYS – Systems Engineering • SEER-Cost IQ Case based reasoning estimation © 2016 Copyright Galorath Incorporated 29
  30. 30. Multiple Estimation Methods Used • Knowledge based parametrics • Built-in estimating relationships, ready to use • Can be tuned and calibrated to improve accuracy • Expert judgment – utilize SME inputs for a task • Analogy – estimate based on past examples • Top down – estimate total cost and allocate • Bottom up – estimate each detailed task or process step © 2016 Copyright Galorath Incorporated 30
  31. 31. SEER Enterprise Costing Solution: Parts thru Total Ownership Cost SEER-MFG SEER-SEM SEER-IT Detailed SEER-H Estimate: • Development • Production • Operation & Support • System Level Costing Detailed Estimate Parametric Estimate Software/IT Estimate Actual Procurement Cost Data SEER-H © 2016 Copyright Galorath Incorporated 31
  32. 32. SEER – Risk Driven Estimates The Engine For Project Evaluation Least, likely, and most inputs provide a range of cost and schedule outcomes Confidence (probability) can be set and displayed for any estimated item • SEER predicts outcomes • SEER uses inputs to develop probability distributions • The result is a probabilistic estimate • SEER will predict a likely range of outcomes • Monte Carlo provides project-level assessments of risk © 2016 Copyright Galorath Incorporated 32 BiasMitigation:Risk
  33. 33. Knowledge Bases • SEER-H uses knowledge bases to derive and preset initial parameter settings based on industry experience • Choose knowledge bases for: • Application (Aircraft Stabilizer, Analog Display, Electro-Mechanical Control, etc.) • Platform (Air-Manned, Air- Unmanned, Groung Mobile, etc.) • Acquisition Category (Build to Print, Buy & Integrate, Make, Modification, etc.) • Standard (Commercial, Military, Space, etc.) • Operations & Support (Remove & Replace, Repair, Scheduled Maintenance, etc.) • Users may add their own knowledge bases if desired BiasMitigation:CompletedActuals
  34. 34. © 2016 Copyright Galorath Incorporated 34 SEER for Software Example: Top 10 Impacts & Sensitivity Analysis
  35. 35. Estimates Are Validated Against Data (Yours and/or Other) SEER-SEM Estimate Your Data Regression Trendline Galorath Benchmark Trendline Your History Data
  36. 36. 36 Project Monitoring & Control “When Performance is Measured Performance Improves” • Adds Performance Measurement (Earned Value) methods to SEER-SEM • Accepts progress & expenditure inputs • Provides cost, schedule, and time variances • Provides cost, schedule, and time indices • Performance-based cost & schedule Estimate at Completion • Displays health and status indicators © 2016 Copyright Galorath Incorporated
  37. 37. SEER-SEM Total Cost of Ownership Why Should We Care: Impacts ROI & development decisions can impact maintenance costs. 37 Staff Vs Maintenance Rigor 0 500 1000 1500 2000 2500 3000 3500 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 Time staffhourspermonth develop Rigor vhi+ Rigor nom Rigor vlo BiasMitigation:ForgottenCosts
  38. 38. Searchable Cost Catalogs • Catalogs used for actual cost metrics for common tasks and parts • Configurable to the customers requirements • Refresh estimates based on the latest metrics © 2016 Copyright Galorath Incorporated 38 BiasMitigation:Analogies
  39. 39. Ply Cost Estimator – CATIA V5BiasMitigation:viablecosttrades
  40. 40. Cost IQ - Obtaining Cost Directly From Limited Requirements A Case Based Reasoning system that can transform high level requirements and specifications into a cost modeling workup within a sophisticated cost estimating model. This approach can provide even concept planners with an advanced understanding of the cost of potential designs. © 2016 Copyright Galorath Incorporated 40 BiasMitigation:Referenceclasses
  41. 41. • Scenarios can be generated proactively or incidentally • Deliberately defining a standard estimation WBS pattern • Turning an exemplar estimate into a WBS pattern Scenarios • Pre-configured WBS patterns may be stored as a “scenario” • Scenarios may be configured with options for the user to select which elements to include • The WBS structure will be loaded with the user’s tailoring © 2016 Copyright Galorath Incorporated 41 BiasMitigation:CompletedActuals
  42. 42. SEER-DB • Store all estimates in a secure repository • Secure access to individuals or user groups • Supports estimate revisions and locking © 2016 Copyright Galorath Incorporated 43 BiasMitigation:ActualHistory
  43. 43. SEER – In Summary • SEER Parametric cost estimation software provides estimates of Effort, Cost, Schedule For Information Technology projects & Program’s • Gives a consistent framework for estimating and planning projects • Helps you understand why projects cost what they do © 2016 Copyright Galorath Incorporated 44
  44. 44. Key Points for today . . . Estimates can be better, reducing bias & strategic mis-estimation… Parametrics can help. © 2016 Copyright Galorath Incorporated 45 Tempering with an “outside view” can mitigate some bias Without care estimates are usually biased (even with experts)
  45. 45. At Galorath – We Do Estimation Galorath solutions: Tools and consulting Galorath works with industry leaders to help adopt and improve organizational estimation best practices • Over 30 years in business • Hundreds of customers in government & industry; many Fortune 500 • Headquartered in El Segundo, CA; International office in Andover, U.K. with staff in the U.S. and Europe • Professional services organization provides consulting and training