Download to read offline
Uploaded byJohnnie Fox
2015 drupalcampcebu estimation_jrf
The presentation discusses the challenges of software estimation, emphasizing the significant discrepancies in estimates among clients, executives, and developers. It highlights the importance of using accurate estimation techniques and recognizing external factors that can affect project timelines and budgets. The document advocates for a structured approach to estimation that minimizes reliance on judgment and optimizes counting, computing, and calibration.
More Related Content
Similar to 2015 drupalcampcebu estimation_jrf
2015 drupalcampcebu estimation_jrf
- 1. A Science, Notan Art? ESTIMATION Drupal Camp Cebu 7 November 2015
- 2. Johnnie Fox 2 • Directorof Technology/Operations • Promet Source: • Drupal Development & Support • Chicago, IL.
- 3. “The typical softwareorganization is not struggling to improve its estimates from +-10% to +-5% accuracy. The typical software organization is struggling to avoid estimates that are incorrect by 100% or more.” -- Steve McConnell What is this presentation about? 3 What is the presentation NOT about? • Fundamentals (overlooked) • Big Risk factors • Selling your estimate (Team) • 20%-100% over budget • Specific estimates (content type) • Magical formulas • Fine tuning error from 20% to 5% • Selling your estimate (client)
- 4. 4 Agenda What is theProblem with Software Estimation?1 2 3 Avoiding Risk Estimation Techniques
- 5. What is theProblem with Software Estimation? 5
- 6. The greatest discrepancybetween what the client thinks, an executive thinks, and a developer thinks is often the definition of ESTIMATE 6
- 7. 7 Estimates, Targets, andCommitments 1 Estimates Prediction • A Prediction of how long a project will take or how much it will cost 2 Targets Statement/Desire • A Statement of a desirable business objective 3 Commitments Promise • A Promise to deliver defined functionality at a specific level of quality and by a certain date
- 8. 8 Estimates, Targets, andCommitments 2 3 Target Commitment Estimate 1 Cut From Phase
- 9. Your best developeris not necessarily your best estimator 9
- 10. 10 The Solution Architect About yourunicorn: • Knows the most about Drupal • Experience in Ecommerce Migration Integration Site building & Theming • Understands business value • Can magically debug anything Things that may be missing: How fast can a DEVELOPER do the work? How much time will be spent training, instructing, reviewing, and doing oversight? Avoiding unfounded optimism
- 11. What is thedefinition of a “GOOD” estimate? 11
- 12. 90% confidence 12 After all,the process is called Estimation, not Exactimation. - Phillip Armour What is the surface temperature of the Sun? (Give a range in Fahrenheit) Make your range wide enough that you feel you are 90% confident that your range includes the answer 10,000 Degrees F (9,941 F)
- 13. When given theoption of a shorter average schedule with higher variability or a longer average schedule with lower variability, 8 out of 10 executives will choose the second option 13
- 14. 90% confidence 14 How confidentis 90% confident? The average number of correct answers was 2.8. Only 2% of quiz takers scored 8 or more correctly. Most peoples intuitive sense of 90% confidence is in fact closer to 30% confidence.
- 15. Software is notdeveloped in a vacuum 15
- 16. 16 EXTERNAL FACTORS AREOFTEN UNACCOUNTED FOR LESS EXPERIENCED STAFF STAFF NOT READY REQUIREMENTS REMOVED STAFF DIVERTED TO SUPPORT OLD PROJECT UNSTABLE FUNCTIONALITY REMOVED SERVER NOT READY STAFF DIVERTED TO SUPPORT TRADE SHOW REQUIREMENTS ADDED
- 17. 17 Events that happenduring the project nearly always invalidate the assumptions that were used to estimate the project in the first place. 1 Functional Assumptions 2 Staffing Assumptions 3 Priority Assumptions 4 Input Assumptions
- 18. 18 Project Dynamics The activities ofa late or compressed project create destructive “late- project” dynamics that make the project even WORSE than nominal. More Status Meetings Frequent Re- estimation Apologizing to key customers Preparing interim releases More discussion on priority & requirements Fixing problems that arise from quick and dirty workarounds Going back to the contract
- 19.
- 20.
- 21. “Software does nothave a neutral estimation problem. The industry data shows clearly that the software industry has an underestimation problem.” -- Steve McConnell 21
- 22.
- 23. Major Areas ofRisk 23 • What kind of content type/field/thing • Scope w/o specs • Specific definitions • Setup Costs • Management Costs • Other (non-dev) Costs • 90% confidence • Best Estimator • Terminology • External Factors • Lack of end-user involvement or production owner • Timeline • Poor wires/design
- 24.
- 25. Consider your project’scurrent position within the cone of uncertainty, this will help you gauge the level of accuracy in your estimate. 25
- 26. 26 The Cone of Uncertainty InitialProduct Concept “I want a website” Product Definition “I know the scope of services” Requirements Complete “I know the specs of the project” User Interface Design Complete “I have wireframes” Detailed Design Complete “I have full documented specs and design files”
- 27. Leverage the powerof well defined products to move your way through the cone of uncertainty 27
- 28. 28 Products Training Workshops Drupal Concept Educationas an entry into the product Discovery Process Discovery Technical and UI/UX planning for a large scale development projects Evaluation of the inner workings of a current Drupal site, for upgrade and migration planning Audits
- 29. Know when youare in a chaos project and how to reset 29
- 30. Chaos Projects How doesBAD go to TERRIBLE? 30 External Factors Re-estimation or Planning Poor Project Dynamics Pure Chaos “Chaos projects introduce variability for each following step. The answer is not to continue to re-estimate but instead address each issue through project control.”
- 31. NEVER, let yourestimator forget (or your sales team remove) these important activities 31
- 32. Omitted Activities 32 Management • Standups, Contract review, Training, QA review • Engagement Management, Staffing, Governance Maintenance • Updates, Upgrades, Deployments, Performance • Configuration, Automation, Reviews Setup • Setup, Installation, Downloads, Builds • Ramp-up, Kickoff, Onboarding, Accounting Non-software • Vacation, Holiday, Sick, Drupalcon • Outages, Troubleshooting, Workstations
- 33.
- 34. Estimating “Things” 34 Lets assumethat you are all expert estimators and that all the process and risk areas we have spoken today you know PERFECTLY how to avoid. How many people are in this room right now?
- 35. 35 APPROACHES TO ESTIMATING“THINGS” The Hobbyist “I’m guessing 75 people” The Factor Guy “15 rows 5 per row” The Percentage Guy “150 person capacity and its half full” The Data Guy “The last 3 sessions have had XX” Judgement Count Compute Calibration
- 36. Count, Compute, Judge 36 Countif at all possible. Compute when you can’t count. Use Judgement alone only as a last resort. COUNT (& Calibrate) Find something to count that has a direct correlation to what you are building1 COMPUTE Multiply by a statistically meaningful (proven) average2 JUDGE Only make assumptions as a final option (document)3
- 37. • Click toedit Master text styles Items to Consider • Size of Project 5-25 people, 25-50, etc. • Sequential vs. Scrum • Stage of project 37
- 38. 38 Common Items inthese areas 1 2 3Count/Calibrate Compute Judge Site Specs Content types Taxonomies Menus Views Wireframes Designs Migration “features” Non-build items QA Deployment Project Management Training Governance Engagement Management Adjustment Factors Risk Multipliers Contingency Unknown
- 39. 39 Count & Calibrate 1) Determinean item that’s highly correlated with the size of the software 2) COUNT how many you have 3) CALIBRATE your count with data from - Industry Data - Historical Data - Project Data Factor Factor Count Calibration Gross Estimate # Content Types 8 4 32 How many hours for feature “X”?
- 40. 40 Compute 1) Determine adefined value to base your computation 2) Determine a multiplier that is a statistically meaningful average 3) COMPUTE subtotal for your line item Development Subtotal 1200 Hours Subtotal PM % PM SubTotal 1200 20% 240 How many hours for Project Management?
- 41. 41 Judgement 1) JUDGE therating of a specific multiplier (factor) 2) Apply multipliers based on rating of your estimate 3) Determine factors influence on the total project 4) COMPUTE Site Build PM % Gross Estimate Rating Influence Net Estimate 1200 240 1440 Low = 1.25 1.1 1980 Site Building Definition 2.0 1.25 1.0 .95 .85 .75 1.1
- 42. Overcoming Judgement 42 Judgement isthe most hazardous kind of estimation due to subjectivity and unfounded optimism. Magnitude of Relative Error MRE = (Actual Result - Estimated) / (Actual) Expected Case EC = (Best Case + (3 x Most Likely Case) + (2 x Worst Case)) / 6
- 43. Wideband Delphi Method •Coordinator presents each expert with a specification and an estimation form. • Coordinator calls a group meeting in which the experts discuss estimation issues with the coordinator and each other. • Experts fill out forms anonymously. • Coordinator prepares and distributes a summary of the estimates • Coordinator calls a group meeting, specifically focusing on having the experts discuss points where their estimates vary widely • Experts fill out forms, again anonymously, and steps 4 to 6 are iterated for as many rounds as appropriate. 43
- 44. Subtitle for slide Guidefor Standardized Estimation Procedure • Focus on counting and computing when possible, rather than using judgment - create tools to facilitate this and make it easy • Encourage multiple estimation approaches and comparison of results • Define toll gates and encourage re-estimation over the course of a project • Contains a clear description of an estimate’s inaccuracy • Defines when an estimate can be used as the basis for a project budget • Defines when an estimate can be used as the basis for internal and external commitments • Calls for archiving estimation data and reviewing effectiveness of the procedure • McConnell, Steve (2006-02-22). Software Estimation: Demystifying the Black Art (Developer Best Practices) (Kindle Locations 5142-5149). Pearson Education. Kindle Edition. 44
- 45. To Learn More... 45 SoftwareEstimation: Demystifying the Black Art Steve McConnell
- 46. Thank You 46 CHICAGO, ILLINOIS 1802W. Berteau Ave. / Suite 209 / Chicago IL, 60613 773-525-8255 Twitter @johnniefox or @prometsource Email: johnnie@prometsource
- 47.
- 48. 48 Estimation Techniques Calibration andHistorical Data T-Shirt Sizing Wideband Delphi Technique
Editor's Notes
- #5 Problems: Common misconceptions or thought provoking statements Avoiding Risk: Lessons learned Techniques: Very high overview of where to get started
- #7 This may be the number one issue that organizations have when internally discussing project costs
- #8 -- This may be the number one issue that organizations have when internally discussing project costs -- understanding the differences in these terms can greatly reduce internal tension when it comes to pricing a project and setting developer expectations
- #9 “5 months! we need this ready in 3 months” “Here is what we can deliver in 3 months” “well i have to have this one piece you didnt include. Lets add that and settle on 4 months”
- #10 -- In fact, no one is going to be good at estimating something NEW
- #11 “well be more effective on this project than the last time we did this” “A lot of things went wrong last project when it took us longer to do this item” “We started the project slowly due to a learning curve, but now will have a faster velocity with the same team” -- In fact, no one is going to be good at estimating something NEW -- The best estimator is often the one with access to the most historical data and comparables
- #13 -- Activity time! year Alexander the Great was born Total volume of great lakes 5,493 Year and month of registration of 231557 - Feb 2008
- #14 -- Even more shocking! -- Why dont we ask? Why do we continue to give handouts before they are asked for -- Because we dont have these reasons we just discussed to arm ourselves in those conversations!
- #15 -- This test was given in many industries, many question types, many people -- The results are staggering -- 10 questions means you should have 9 right -- Of the people with 7 or 8 right they claimed afterwards that their ranges are too wide, thats how they did well -- They felt BAD about their estimate -- We are conditioned to believe that estimates expressed as narrow ranges are more accurate or appealing than estimates with wider ranges. We believe wider ranges make us appear ignorant or incompetent.
- #17 -- We are laughing because we KNOW these happen on almost all projects -- Even if you do not pass all these costs onto the client, they MUST be accounted for 1Staff not ready when planned 2Less experienced staff than planned 3requirements removed 4staff divered to support old project 5staff diverted to support trade show 6more requirements added 7unstable funcality removed 8Server hardware not setup to support pre-prod testing
- #18 -- There isnt a 3rd party data lookup to support functionality. There isnt exact department definition of all content. There are events that start at 15 min after or 45 min after an hour. -- SA got put on another project that closed first -- supporting past project is higher priority -- Wireframes delivered at a different time. Client didnt complete their rebranding in time for design phase
- #19 Use example of a project that has an issue, that starts to delay a project Project Dynamics aren’t considered or are ignored --
- #21 -- if you cant see this --- showing that most projects are delivered late, very few on time, and non ahead of schedule --- shows that only 20% of projects are on time AND on budget, 50% are late or over, 30% FAIL The Seattle Mariners’ new baseball stadium was estimated in 1995 to cost $ 250 million. It was finally completed in 1999 at a cost of $ 517 million— an estimation error of more than 100% (Withers 1999). The most massive cost overrun in recent times was probably Boston’s Big Dig highway construction project. Originally estimated to cost $ 2.6 billion, costs eventually totaled about $ 15 billion— an estimation error of more than 400% (Associated Press 2003). Of course, the software world has its own dramatic estimation problems. The Irish Personnel, Payroll and Related Systems (PPARS) system was cancelled after it overran its € 8.8 million system by € 140 million (The Irish Times 2005). The FBI’s Virtual Case File (VCF) project was shelved in March 2005 after costing $ 170 million and delivering only one-tenth of its planned capability (Arnone 2005). The software contractor for VCF complained that the FBI went through 5 different CIOs and 10 different project managers, not to mention 36 contract changes (Knorr 2005). McConnell, Steve (2006-02-22). Software Estimation: Demystifying the Black Art (Developer Best Practices) (Kindle Locations 1191-1199). Pearson Education. Kindle Edition. http://www.dispatch.com/content/stories/local/2014/12/06/daunting-drilling.html Project to bore tunnel under Columbus faces $29.5 million cost overrun The $26 million custom machine, like something out of science fiction, was built to drill through mostly dry bedrock. But city engineers’ estimate of water levels didn’t account for millions of gallons of water that travel underground, below the Scioto River. Instead of grinding through dry rock, Marsha was choking on a water/rock/mud mixture called slurry, which is kind of like wet concrete.
- #22 -- Let this sink in -- All the data out there CLEARLY supports that we are drastically underestimating our projects **** this doesnt mean we have to price differently -- we need this info for expectation setting, true costing, and predicting how things are going to go -- ideally we would also learn to pass on some of these costs to the client
- #24 -- discussed #1 -- Estimation process is a large portion of the risk of estimation. (that is why we spent the most time just talking about it) -- Would like to move into a few of the other areas in less detail and discuss some lessons learned
- #25 -- The COST of underestimation is exponentially worse than the COST of overestimation (parkinsons Law -- is a predicable loss)
- #26 This is about Project Info or project specs
- #27 This is about Project Info or project specs
- #28 This is also about project info
- #29 this is also about project info -- this is a great way to NARROW your cone of uncertainty through paid services/products -- get paid to lower your estimation risk
- #30 Project chaos -- discussed partially earlier regarding project dynamics and external factors
- #31 Project chaos -- discussed partially earlier regarding project dynamics and external factors -- “we are past the hard part now our velocity will increase” -- Understanding how to “pause and restart” is key to keeping your estimates meaningful -- AVOID THE SPIRAL “how the heck did we get here” at the end of your project
- #32 Omitted activities is a very common mistake -- even worse is when they are REMOVED -- Dont reduce your estimators estimate, we just proved before that it is OFTEN already too optimistic
- #33 Omitted activities is a very common mistake -- even worse is when they are REMOVED -- Dont reduce your estimators estimate, we just proved before that it is OFTEN already too optimistic -- doesnt mean you have to have the client pay for these all
- #43 72% of software projects were reported to be done by “expert opinion” also known as judgement
- #44 https://en.wikipedia.org/wiki/Wideband_delphi
- #47 EVALUATIONS
- #49 Calibraion - discuss the example of similar project T-Shirt Sizing