Software project estimation - the differences between expert estimates and parametric estimates. The results of a study on the accuracy (effort, schedule and cost) of the two types of estimates is part of the presentation

1. Software Estimation and
Performance Measurement
Harold van Heeringen
Sogeti Sizing, Estimating & Control (SEC)
Sogeti Nederland B.V.
Stockholm, April 18 2013@haroldveendam

2. 2
Topics
• Software projects
• Software estimation
• Expert vs. Parametric Estimates
• Challenge for parametric estimates
• E&PM process
• Accuracy of the different estimation types
• The Swedish software industry?

3. 3
Software projects
• Software project industry : low maturity
− Low estimation maturity
− No or little formal estimation processes
− No or little use of historical data
• Lots of schedule and cost overruns
− Standish Chaos reports: Most projects fail or are at least
unsuccessful
• Low customer satisfaction rates
− In Europe: only slightly higher than the financial sector

4. 4
Software project estimation
• Most of the projects are estimated by ‘experts’
− Bottom up, task by task effort estimation
• Usually very optimistic (>30%)
− Experts estimate, but other people (juniors) do the job
− Forgotten activities (e.g. testscript reviews)
− No feedback loop with past projects: experts don’t learn from
past estimates and actuals
− No scenario’s: duration, team size, etc.
− Not objective, transparent, verifiable or repeatable
• Not defendable!
− ‘Easy’ to push back by stakeholders
• No risk assessment (distribution of the
estimate)

5. 5
Expert Estimation
Probability
Effort
50/50 median result
90% . . . . . . . . . . . . . . . . 24 hours
75% . . . . . . . . . . . . . . . . 22 hours
50% . . . . . . . . . . . . . . . . 20 hours
10% . . . . . . . . . . . . . . . . 18 hours
0% . . . . . . . . . . . . . . . . 16 hours
. . . . . . . . 14 hours
. . . . . . . . 12 hours
. . . . . . . . 10 hours
. . . . . . . . 8 hours
. . . . . . . . 6 hours
. . . . . . . . 4 hours
. . . . . . . . 2 hours
Expert Estimate: 1st possibility of success
0%
100%
Task– Code and Unit test of module XYZ
Expert estimate
Realistic estimate

6. 6
Project Estimates
• Two types of project estimation:
− Expert estimation
− Parametric estimation
• Expert
− Knowledge and experience of experts
− Assign effort hours to tasks (bottom-up)
− Subjective, but always applicable
• Parametric
− Size measurement, historical data and tooling
− Size measurement methods: NESMA FPA, COSMIC, IFPUG
− Objective, but well documented specifications required

7. 7
Comparing the two types
• Expert Estimates
− Bottom-up estimation
− Usually optimistic (up to 30% under estimation is common)
− Forgotten activities
− Hard to defend
− The expert is not going to do all the work
− The expert may not be an expert on the new project
− Are the real experts available?
• Parametric Estimates
− Top-down estimation
− Estimating & Performance Measurement process needed
− Effort = size * Productivity
◦ Size is objectively measureable (COSMIC, FPA)
◦ Productivity from historical data (organization / ISBSG)
◦ scenario’s through tools (QSM / SEER-SEM / ISBSG)

8. 8
Eestimate & Performance Measurement
PLAN
Estimate
AdministrateEvaluate
Adjust &
Report
Size measurement: FPA
Historical data
Estimation tools
Finetune Estimation model
Analyse productivity,
Report productivity
Start: Estimate request
Start: Project completed
Results:
- Parametric Estimation
- Expert Estimation
Result:
-Management report,
-Adjusted model
Result:
-Growing project DB,
-Performance measurement
-Updated expert knowledge
ACT
CHECK DO
Start: Project start
Continuous data collection
• effort hours registration
• defect registration
• change measurement
• project characteristics
Result: Project data
Data collection and
administration
• Collect project data
• Measure size
• Benchmark the project
Start: Periodically
Expert Estimate

9. 9
Challenge – ‘sell parametric estimates’
• Project/Bid management still believe experts
more than parametrics
− ‘More detail must mean more accurate, right’?
− ‘This project is very different from past projects’
− ‘I see that we don’t get any function points for module XYZ,
but we have to do a lot of work for that!’
− ‘I think the project is quite easy and I think that the
parametric estimate overestimates the effort’
− But what about: team size, duration, forgotten activities,
past performance.
• How can we convince project management ??

10. 10
Software equation (Putnam)
Size/productivity
= Effort 1/3 * duration4/3
Effort
Duration
Plan A: 6 months, 4.500 hours
Plan B: 7 months, 2.400 hours

11. 11
Project at different durations
Effort(hours)
Duration
Plan A
Duration: 6 months
Effort: 4.500 hours
Max. team size: 5,8 fte
MTTD: 1,764 days
Plan B
Duration: 7 months
Effort: 2.400 hours
Max. team size: 2,7 fte
MTTD: 2,816 days
Size and productivity being constant
Which duration have the experts in mind??

12. 12
Sogeti SEC
• Sizing, Estimating & Control
− Certified (COSMIC) Function Point Analysts
− Metrics consultants
• Responsible for metrics part of a quotation.
− Size: FPA/COSMIC
− Estimation: SEER-SEM / QSM / Sogeti tool / ISBSG
− Product: Methodical Estimation Report (scenario’s)
− Pricing: EUR/FP
− Quality: Defects/FP
• Centers of Excellence: MS.Net, Java, Oracle,
mobile, drupal, sharepoint, BI, etc.

13. 13
Assignment 2005
• Build estimation instrument
− Gain time and effort in estimating bids
− Accurate enough to depend and rely on
− Flexible:
◦ Estimate onshore / offshore and hybrids
◦ Calculate different test strategies
◦ Take into account complexity
◦ Implement Deming cycle (PDCA)
• Give scenario’s for duration !!!

14. 14
First version Estimating Wizard (2005)
• Try to grasp the duration / effort tradeoff in a
model
− Tuned with experience data
Hour/FP: Average Complexity
Duration in months 3½ 4 4½ 5 5½ 6 6½ 7 7½ 8
0-250 FP 10,1 8,9 8,1 7,7 6,9
250-500 FP 9,1 8,0 7,3 6,9 6,2
500-750 FP 8,6 7,6 6,9 6,5 5,9
750-1000 FP 8,3 7,3 6,6 6,3 5,6
1000 -1250 FP 8,1 7,1 6,5 6,2 5,5
1250 - 1500 FP 7,9 6,9 6,3 6,0 5,4

15. 15
Estimating Wizard 2013
Input
Functional design parameters
Functional Design Yes
Overlap Yes, manually 5
Language English
Availability key users Low
Location On site
Build and test parameters
Development tool Java
Onshore Offshore
Construction 30% 70%
Translation FD required No
System test approach TMap Heavy
System test strategy
Tools/methodologies 5 - Average
Complexity 6
Development team 5 - Average
Reuse 0 - None 5%
General parameters
Size 367 FP
Start date 01-05-13
Project control 2 hrs/week
Risk surcharge (%) 10 %
Warranty (%) 4 %
Organization type Banking
Quality documentation 6
Non functional req.
Scenario interval 2,0 weeks
m20v20120224
Scripting and design NL, execution in India
Average (0)

16. 16
EW - output
• Functional Design – no schedule scenarios
Functional design phase
Duration in weeks 17,6
Design complete 4-05-11
Total effort 1.975
Effort per FP 2,53
Effort cost € 208.531
Additional cost € 14.815
Totaal cost € 223.346
Cost per FP € 286
Average team size 2,80
Data altered due to company security reasons

17. 17
EW - output
• Main build – 7 schedule scenarios
Build and test phase
Duration in weeks 20,0 22,0 24,0 26,0 28,0 30,0 32,0
Start phase 18-02-11 18-02-11 18-02-11 18-02-11 18-02-11 18-02-11 18-02-11
Effort 9.794 6.690 4.723 3.429 2.550 1.935 1.495
Effort per FP 28,06 19,17 13,53 9,83 7,31 5,54 4,28
Effort cost € 550.720 € 376.152 € 265.590 € 192.826 € 143.360 € 108.787 € 84.036
Additional cost € 56.446 € 41.090 € 31.364 € 24.963 € 20.611 € 17.570 € 15.393
Totaal cost € 607.167 € 417.242 € 296.954 € 217.789 € 163.972 € 126.357 € 99.428
Cost per FP € 1.740 € 1.196 € 851 € 624 € 470 € 362 € 285
Average team size 12,24 7,60 4,92 3,30 2,28 1,61 1,17
Risk and warranty
Risk hours 834 586 429 325 255 205 170
Risk cost € 54.301 € 39.107 € 29.484 € 23.150 € 18.845 € 15.836 € 13.682
Warranty hours 209 146 107 81 64 51 43
Warranty cost € 13.575 € 9.777 € 7.371 € 5.788 € 4.711 € 3.959 € 3.420
Total
Duration in weeks 27,0 29,0 31,0 33,0 35,0 37,0 39,0
Delivery for acceptance 8-07-11 22-07-11 5-08-11 19-08-11 2-09-11 16-09-11 30-09-11
Total effort 11.470 8.055 5.892 4.469 3.501 2.825 2.340
Effort per FP 32,87 23,08 16,88 12,80 10,03 8,09 6,71
Totaal cost € 746.634 € 537.717 € 405.400 € 318.319 € 259.120 € 217.744 € 188.122
Cost per FP € 2.139 € 1.541 € 1.162 € 912 € 742 € 624 € 539
Average team size 10,64 6,96 4,76 3,39 2,50 1,91 1,50
Average hourly rate € 65 € 67 € 69 € 71 € 74 € 77 € 80
Data altered due to company security reasons

18. 18
Calibrating the model
• Collect historical data
• Use ISBSG data

19. 19
Calibrating the model
• Collect historical data
• Use ISBSG data

20. 20
Estimation accuracy
• Study of 10 completed projects, Different sizes,
programming languages, etc
− Expert estimate
− Estimating Wizard estimate
− Estimates vs. Actual results
• Calculate
− Effort accuracy (Effort estimate / Actual effort)
− Duration Accuracy (Duration Estimate / Actual Duration)
− Cost Accuracy (Cost Estimate / Actual Cost)

21. 21
Results
• Calculate
− Effort accuracy (Effort estimate / Actual effort)
− Duration Accuracy (Duration Estimate / Actual Duration)
− Cost Accuracy (Cost Estimate / Actual Cost)
Expert Expert Expert Expert Est. Wizard Est. Wizard Est. Wizard Est. Wizard
Project Size (FP)
Effort
Accuracy
Duration
Accuracy
Cost
Accuracy Time Spent
Effort
Accuracy
Duration
Accuracy
Cost
Accuracy
Time
spent
Project 1 277 0,675 1,545 0,467 30 0,477 1,204 0,501 17
Project 2 359 0,579 0,951 1,139 35 0,707 0,775 1,170 26
Project 3 347 0,589 0,142 0,615 40 1,067 0,996 1,283 14
Project 4 1.178 0,414 0,557 0,312 60 0,774 0,590 0,862 55
Project 5 951 1,430 0,997 0,946 34 1,067 0,877 1,718 24
Project 6 295 0,763 0,857 0,619 26 0,881 1,200 0,845 6
Project 7 790 0,717 0,850 0,976 34 0,926 0,865 1,132 27
Project 8 350 1,258 0,800 1,309 28 1,203 1,096 1,318 20
Project 9 746 0,586 0,296 0,545 34 0,826 0,385 1,953 22
Project 10 2.293 0,766 0,421 0,797 40 0,931 0,632 1,058 14

22. 22
Estimation Accuracy results
• Average time spent
− Expert: 36,1 hours
− EW (including FPA / COSMIC measurement): 22,5 hours
Expert Estimate Est. Wizard Estimate
Effort Accuracy
Average 0,778 0,886
St.Dev. 0,319 0,207
Median 0,696 0,904
Duration Accuracy
Average 0,742 0,862
St.Dev. 0,405 0,272
Median 0,825 0,871
Cost Accuracy
Average 0,772 1,184
St.Dev. 0,316 0,423
Median 0,708 1,151

23. 23
Results
• Expert estimates take a lot of time too!
− on average 60% more than Parametric Estimates
− More than 1 expert has to read through all the
documentation
− Discussions and agreements take time
• Parametric estimates are more acurate!
− Effort and duration estimates on average still optimistic, but
less optimistic than expert estimates
− Cost estimate pessimistic, but still closer to actuals than
Expert estimate
• Experts might win the project, but the result will
be overruns!

24. 24
Cost of high and low estimates
Non-linear extra costs
- Planning errors
- Team enlargement more expensive, not faster
- Extra management attention / overhead
- Stress: More defects, lower maintainability !!
Linear extra costs
Extra hours will be used

25. 25
Conclusions of the study
• Estimating wizard
− Higher effort estimation accuracy
− Higher duration estimation accuracy
− Higher cost estimation accuracy (although >1)
− Less hours spent than expert estimates
− Standard WBS, historical data collection and parameter
calibration improve the maturity of the process
• Next steps
− Analyze why costs are overestimated
− Try to identify the projects where EW estimate is enough
− Use the results to convince project management and bid
management to use parametric estimating more

26. 26
Do we have time for some other thoughts?
• What about the maturity in the Swedish market?

27. 27
ISBSG ‘Country Analysis Report’
• ISBSG repository (>6000 projects)

28. 28
ISBSG ‘Country Analysis Report’

29. 29
Where are the Swedish projects??
• Please submit project data to ISBSG !
• Independent, anonymous
• Free benchmark report
• Flyer ISBSG available
www.isbsg.org

30. 30
Thank you for your attention !!
@haroldveendam
President ISBSG (International Software Benchmarking Standards Group (www.isbsg.org))
Board member NESMA (Netherlands Software Metrics Association (www.nesma.nl))
IAC member COSMIC (www.cosmicon.com)
Harold.van.heeringen@sogeti.nl
Harold van Heeringen
Senior Consultant Software Metrics /Software Cost Engineer
Sogeti Sizing, Estimating & Control (SEC)