0
Thesis Defense Daniel Baker West Virginia University “ A Hybrid Approach to Expert and Model  Based Effort Estimation” Dec...
Introduction <ul><li>Research since May 2006 </li></ul><ul><li>WVU and NASA JPL </li></ul><ul><li>Key Thesis Points </li><...
What is Software Effort Estimation? <ul><li>Approximation of the amount of work required to develop a software project </l...
Why is effort estimation important? <ul><li>An accurate estimate is needed to budget new projects </li></ul><ul><li>Bad es...
How is software effort estimation done? <ul><li>Expert Judgment </li></ul><ul><ul><li>Estimate is largely based on unrecov...
A Hybrid Approach <ul><li>Recent studies have suggested new research in combining expert judgment and prediction models (S...
How to Add Expert Qualities? <ul><li>Increase the involvement of the cost analyst with the model:  “white box”, not “black...
Which Model-based Methods? <ul><li>COCOMO (Boehm 2000) </li></ul><ul><ul><li>Clearly defined algorithm </li></ul></ul><ul>...
Method Evaluation <ul><li>The model-based methods should be validated with historical data </li></ul><ul><li>Notoriously u...
Laboratory Studies <ul><li>Large portion of this thesis was experimenting with new uses of COCOMO: </li></ul><ul><ul><li>F...
Industrial Evaluation <ul><li>Methods researched were implemented in a tool, 2CEE, and evaluated at JPL </li></ul><ul><li>...
Uncertainty Representation <ul><li>Kitchenham's* sources of uncertainty: Measurement, Model, Assumption, and Scope Error <...
Unexpected Results <ul><li>2CEE’s data visualization provided insights into the calibration of COCOMO </li></ul><ul><li>Th...
Summary of Achievements <ul><li>Automated 7 of Jorgensen's 12 Expert Judgment Best Practices in the 2CEE tool. </li></ul><...
Related Work as a Coauthor <ul><li>O. Jalali, T. Menzies, D. Baker, and J. Hihn. Column pruning beats stratification in ef...
Future Work <ul><li>Implement More of Jorgensen's Best Practices </li></ul><ul><li>Include More Models in 2CEE </li></ul><...
Concluding Key Points <ul><li>Human / Machine Integration </li></ul><ul><li>“ White Box”, not “Black Box” </li></ul><ul><l...
Questions?
Upcoming SlideShare
Loading in...5
×

A Hybrid Approach to Expert and Model Based Effort Estimation

3,977

Published on

Masters thesis, CSEE, WVU

Published in: Economy & Finance, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
3,977
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
120
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "A Hybrid Approach to Expert and Model Based Effort Estimation "

  1. 1. Thesis Defense Daniel Baker West Virginia University “ A Hybrid Approach to Expert and Model Based Effort Estimation” December 3rd, 2007
  2. 2. Introduction <ul><li>Research since May 2006 </li></ul><ul><li>WVU and NASA JPL </li></ul><ul><li>Key Thesis Points </li></ul><ul><ul><li>Human / Machine Integration </li></ul></ul><ul><ul><li>“ White Box”, not “Black Box” </li></ul></ul><ul><ul><li>Embrace Uncertainty </li></ul></ul><ul><ul><li>What not to do </li></ul></ul><ul><ul><li>Use proper method evaluation </li></ul></ul><ul><ul><li>Research => Industrial Application / Evaluation </li></ul></ul>
  3. 3. What is Software Effort Estimation? <ul><li>Approximation of the amount of work required to develop a software project </li></ul><ul><li>Made with a unit of effort such as work-months </li></ul><ul><li>Effort can be converted to cost </li></ul><ul><li>Effort is useful across time periods, geographic locations, industry. </li></ul>
  4. 4. Why is effort estimation important? <ul><li>An accurate estimate is needed to budget new projects </li></ul><ul><li>Bad estimates can lead to projects being canceled </li></ul><ul><li>“ To gain control over its finances, NASA last week scuttled a new checkout and launch control system (CLCS) for the space shuttle. A recent assessment of the CLCS, which the space agency originally estimated would cost $ 206 million to field, estimated that costs would swell to between $ 488 million and $ 533 million by the time the project was completed.” – June 11 2003, Computer News </li></ul>
  5. 5. How is software effort estimation done? <ul><li>Expert Judgment </li></ul><ul><ul><li>Estimate is largely based on unrecoverable human intuition </li></ul></ul><ul><ul><li>Used by most of industry </li></ul></ul><ul><li>Model Estimation </li></ul><ul><ul><li>Estimate is made as some function of historical data </li></ul></ul><ul><ul><li>Topic of most of research </li></ul></ul><ul><li>Jorgensen reviewed 15 studies using both methods and found that, ”of the fifteen studies, we categorize five to be in favor of expert estimation, five to find no difference, and five to be in favor of model-based estimation.” </li></ul><ul><li>- M. Jorgensen. A review of studies on expert estimation of software development effort.. Journal of Systems and Software , 70(1-2):37–60, 2004. </li></ul>
  6. 6. A Hybrid Approach <ul><li>Recent studies have suggested new research in combining expert judgment and prediction models (Shepperd, Meli). </li></ul><ul><li>“ a conceptual framework of integration and a set of operational rules to follow (Meli)” </li></ul><ul><li>Two ways to do this: full combination “from scratch”, or add aspects of one to the other based on existing methods </li></ul><ul><li>Decided to add expert-judgment qualities to existing model prediction (Don't reinvent the wheel) </li></ul><ul><li>M. Shepperd. Software project economics: A roadmap. In International Conference on Software Engineering 2007: Future of Software Engineering , 2007. </li></ul><ul><li>Roberto Meli. Human factors and analytical models in software estimation: An integration perspective. In Proceedings of the ESCOM-SCOPE 2000 , pages 33–40, Munich, Germany, 2000. Shaker Publishing. </li></ul>
  7. 7. How to Add Expert Qualities? <ul><li>Increase the involvement of the cost analyst with the model: “white box”, not “black box” </li></ul><ul><li>Automate Jorgensen's Expert Judgment Best Practices: </li></ul><ul><ul><li>1. evaluate estimation accuracy, but avoid high evaluation pressure </li></ul></ul><ul><ul><li>2. avoid conflicting estimation goals </li></ul></ul><ul><ul><li>3. ask the estimators to justify and criticize their estimates </li></ul></ul><ul><ul><li>4. avoid irrelevant and unreliable estimation information </li></ul></ul><ul><ul><li>5. use documented data from previous development tasks </li></ul></ul><ul><ul><li>6. find estimation experts with relevant domain background </li></ul></ul><ul><ul><li>7. estimate top-down and bottom-up, independently of each other </li></ul></ul><ul><ul><li>8. use estimation checklists </li></ul></ul><ul><ul><li>9. combine estimates from different experts and estimation strategies </li></ul></ul><ul><ul><li>10. assess the uncertainty of the estimate </li></ul></ul><ul><ul><li>11. provide feedback on estimation accuracy </li></ul></ul><ul><ul><li>12. provide estimation training opportunities </li></ul></ul><ul><li>M. Jorgensen. A review of studies on expert estimation of software development effort. Journal of Systems and Software , 70(1-2):37–60, 2004. </li></ul>
  8. 8. Which Model-based Methods? <ul><li>COCOMO (Boehm 2000) </li></ul><ul><ul><li>Clearly defined algorithm </li></ul></ul><ul><ul><ul><li>Uses historical data to calibrate a linear model relating code size to effort (also uses “cost drivers”) </li></ul></ul></ul><ul><ul><li>Publicly available data </li></ul></ul><ul><ul><li>Used by the client: NASA JPL </li></ul></ul><ul><ul><li>Solid performer, heavily researched </li></ul></ul><ul><li>Feature and Record Selection techniques may improve performance, create simpler, more stable models, and “avoid irrelevant and unreliable estimation information.” </li></ul><ul><li>Boosting and Bagging </li></ul><ul><li>Barry Boehm, Ellis Horowitz, Ray Madachy, Donald Reifer, Bradford K. Clark, Bert Steece, A. Winsor Brown, Sunita Chulani, and Chris Abts. Software Cost Estimation with Cocomo II . Prentice Hall, 2000. </li></ul>
  9. 9. Method Evaluation <ul><li>The model-based methods should be validated with historical data </li></ul><ul><li>Notoriously unstable conclusions of SW effort predictors </li></ul><ul><li>Source: a small number of outliers </li></ul><ul><li>Solution: evaluate using nonparametric techniques </li></ul><ul><li>Median MRE is a more consistent measure </li></ul>
  10. 10. Laboratory Studies <ul><li>Large portion of this thesis was experimenting with new uses of COCOMO: </li></ul><ul><ul><li>Feature Subset Selection (FSS) </li></ul></ul><ul><ul><ul><li>Exhaustive Search (COCOMOST) </li></ul></ul></ul><ul><ul><ul><li>Backward Elimination Search (dBFS) </li></ul></ul></ul><ul><ul><ul><li>Near Linear Time Search (COCOMIN) </li></ul></ul></ul><ul><ul><li>Bagging </li></ul></ul><ul><ul><li>Boosting </li></ul></ul><ul><ul><ul><li>by Subsampling </li></ul></ul></ul><ul><ul><ul><li>by Oversampling </li></ul></ul></ul><ul><ul><ul><li>by Adaboost (Distributive Sampling) </li></ul></ul></ul><ul><li>End result </li></ul><ul><ul><li>FSS is sometimes useful; simple methods are as good as complicated ones </li></ul></ul><ul><ul><li>Bagging and Boosting add complexity without significant improvement (subsampling might be useful) </li></ul></ul>
  11. 11. Industrial Evaluation <ul><li>Methods researched were implemented in a tool, 2CEE, and evaluated at JPL </li></ul><ul><li>Calibrated with most recent NASA data </li></ul><ul><li>Test records represented those estimated at JPL </li></ul><ul><li>Newest methods => good improvements in median error </li></ul>
  12. 12. Uncertainty Representation <ul><li>Kitchenham's* sources of uncertainty: Measurement, Model, Assumption, and Scope Error </li></ul><ul><li>Cross Validation and Representing the Inputs Uncertainty Covers Each Source except Scope Error </li></ul><ul><li>Benefits: </li></ul><ul><ul><li>Risk management </li></ul></ul><ul><ul><li>Budgeting </li></ul></ul><ul><ul><li>Estimates can be made early with high uncertainty, and again later with more confidence </li></ul></ul>*Barbara Kitchenham and Stephen Linkman. Estimates, uncertainty, and risk. IEEE Softw. , 14(3):69–74, 1997.
  13. 13. Unexpected Results <ul><li>2CEE’s data visualization provided insights into the calibration of COCOMO </li></ul><ul><li>The linear COCOMO coefficient a, and the exponential coefficient b are supposed to vary: </li></ul><ul><li>2.5 <= a <= 2.94 </li></ul><ul><li>0.91 <= b < 1.01 </li></ul><ul><li>However calibration with NASA data shows much different numbers with higher variance </li></ul>
  14. 14. Summary of Achievements <ul><li>Automated 7 of Jorgensen's 12 Expert Judgment Best Practices in the 2CEE tool. </li></ul><ul><li>Experimented with a lot of methods for SW effort estimation and evaluated non-parametrically: </li></ul><ul><ul><li>Bagging/boosting were mostly wasted CPU cycles when applied to COCOMO </li></ul></ul><ul><ul><li>Feature selection was sometimes useful, and simpler selectors are as good as complex </li></ul></ul><ul><li>Represented more sources of estimation uncertainty in a tool than previously seen. </li></ul><ul><li>Methods were applied and evaluated in an industrial application at JPL </li></ul><ul><ul><li>Median MRE was greatly reduced </li></ul></ul><ul><li>Reported more unstable / different COCOMO calibrations than previously reported </li></ul>
  15. 15. Related Work as a Coauthor <ul><li>O. Jalali, T. Menzies, D. Baker, and J. Hihn. Column pruning beats stratification in effort estimation. In Proceedings, PROMISE workshop, Workshop on Predictor Models in Software Engineering , 2007. </li></ul><ul><li>T. Menzies, O. Elrawas, D. Baker, J. Hihn, and K. Lum. On the value of stochastic abduction (if you fix everything, you lose fixes for everything else). In International Workshop on Living with Uncertainty (an ASE’07 co-located event) , 2007. Available from http://menzies.us/pdf/07fix.pdf . </li></ul><ul><li>Tim Menzies, Omid Jalali, Jairus Hihn, Dan Baker, and Karen Lum. Software effort estimation and conclusion stability, 2007. </li></ul>
  16. 16. Future Work <ul><li>Implement More of Jorgensen's Best Practices </li></ul><ul><li>Include More Models in 2CEE </li></ul><ul><li>Interactive Feature Exploration </li></ul><ul><li>Tradeoff Analysis </li></ul><ul><li>Approach a hybrid combination of expert and model either from scratch or from the experts point of view (by adding model aspects to standard expert judgment) </li></ul><ul><li>Research into the COCOMO calibration coefficient variance </li></ul>
  17. 17. Concluding Key Points <ul><li>Human / Machine Integration </li></ul><ul><li>“ White Box”, not “Black Box” </li></ul><ul><li>Embrace Uncertainty </li></ul><ul><li>What not to do </li></ul><ul><li>Use proper method evaluation </li></ul><ul><li>Research => Industrial Application / Evaluation </li></ul>
  18. 18. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×