Using Correlation and Accuracy for Identifying Good Estimators http:// nas.cl.uh.edu/boetticher/publications.html The 4 th...
Research vs. Reality according to Jörgensen <ul><li>TSE ’07: 300+ software est. papers, </li></ul><ul><li>76 journals, 15+...
Statement of Problem http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in Softwar...
Statement of Problem <ul><li>How to build human-based estimation models that are accurate, intuitive, and easy to understa...
PROMISE 2008 versus 2007 The 4 th  International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl...
The Approach http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in Software Engine...
Feedback to Users http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in Software E...
Experiments: Data http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in Software E...
Experiments: Tools, Configuration http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Mode...
Results: Correlation Only http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in So...
Results: Scale Factor Only http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in S...
Results: Correlation & Scale Factor http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Mo...
Discussion - 1 http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in Software Engi...
Discussion - 2 http://nas.cl.uh.edu/boetticher/publications.html The 4 th  International Predictor Models in Software Engi...
Conclusions <ul><li>Good accuracy rates, </li></ul><ul><li>especially after attribute reduction </li></ul><ul><li>Correlat...
http://nas.cl.uh.edu/boetticher/publications.html Thank You   ! The 4 th  International Predictor Models in Software Engin...
References <ul><li>Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Sy...
Upcoming SlideShare
Loading in …5
×

Boetticher Presentation Promise 2008v2

706 views

Published on

Using Correlation and Accuracy for Identifying Good Estimates

Published in: Business, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
706
On SlideShare
0
From Embeds
0
Number of Embeds
78
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Boetticher Presentation Promise 2008v2

  1. 1. Using Correlation and Accuracy for Identifying Good Estimators http:// nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop Gary D. Boetticher Nazim Lokhandwala Univ. of Houston - Clear Lake, Houston, TX, USA [email_address] [email_address] 63 62 61
  2. 2. Research vs. Reality according to Jörgensen <ul><li>TSE ’07: 300+ software est. papers, </li></ul><ul><li>76 journals, 15+ Years </li></ul>http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop JSS ’04: Compendium of expert estimation studies -89 89-99 00-04 Total Algorithm 48 137 70 255 ML 1 32 41 74 Human 3 22 21 46 Misc. 7 19 26 52 68% Algorithm 20% ML 12% Human Paper Human Hihn 91 89% Heemstra 91 62% Paynter 96 86% Jørgensen 97 84% Hill 00 100% Kitchenham 02 72% 82% Human 18% Formal
  3. 3. Statement of Problem http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop ((Log (TechGradCourses + (TechGradCourses ^ ((Log TotWShops)/(Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (((ProcIndExp + (Log (Sin MgmtGradCourses)))/(Sin SWPMExp)) + (Sin ((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Sin SWPMExp)))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((Log SWProjEstExp) / (((Log (ProcIndExp + (Log (TechGradCourses ^ ((Log SWProjEstExp) / (Log SWProjEstExp)))))) - 3) / (ProcIndExp + (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (TechGradCourses ^ (Log SWProjEstExp))))) / (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp)))))))))))))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) + ((Log SWProjEstExp) / (Log SWProjEstExp)))))) / (Log (Log (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp))))))))))))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Log ((((Log TotLangExp) / (Log SWProjEstExp)) / (Log SWProjEstExp)) / (Sin SWPMExp))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))))))) + (((((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + ((TechGradCourses ^ (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))) / (Sin SWPMExp))))))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (Sin SWPMExp))) Some Background 2006 http://www.starwarscrawl.com/?id=232
  4. 4. Statement of Problem <ul><li>How to build human-based estimation models that are accurate, intuitive, and easy to understand? </li></ul>http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop TechUGCourses < 45.5 | Hardware Proj Mgmt Exp < 6 | | No Of Hardware Proj Estimated < 4.5 | | | No Of Hardware Proj Estimated < 3 | | | | TechUGCourses < 23 | | | | | Hardware Proj Mgmt Exp < 0.75 | | | | | | TechUGCourses < 18 | | | | | | | Hardware Proj Mgmt Exp < 0.13 | | | | | | | | TechUGCourses < 0.5 | | | | | | | | | TechUGCourses < -1 : F (1/0) | | | | | | | | | TechUGCourses >= -1 | | | | | | | | | | Degree < 3.5 : A (4/0) | | | | | | | | | | Degree >= 3.5 : A (5/2) | | | | | | | | TechUGCourses >= 0.5 | | | | | | | | | TechUGCourses < 5.5 | | | | | | | | | | Degree < 3.5 : F (5/0) | | | | | | | | | | Degree >= 3.5 | | | | | | | | | | | TechUGCrses < 2 : A (1/0) | | | | | | | | | | | TechUGCrses >= 2 : F (1/0) | | | | | | | | | TechUGCrses >= 5.5 | | | | | | | | | | Degree < 3.5 | | | | | | | | | | | TechUGCrs < 10.5 : A (3/0) | | | | | | | | | | | TechUGCrses >= 10.5 | | | | | | | | | | | | TechUGCrs<12.5 : F (3/0) | | | | | | | | | | | | TechUGCrses >= 12.5 | | | | | | | | | | | | | TechUGCrs<16: A (2/0) | | | | | | | | | | | | | TechUGCrs>15 : A (2/1) | | | | | | | | | | Degree >= 3.5 : F (1/0) | | | | | | | HardProjMgmt Exp >= 0.13 : A (2/0) | | | | | | TechUGCourses >= 18 : A (2/0) | | | | | Hard Proj Mgmt Exp >= 0.75 : F (1/0) | | | | TechUGCourses >= 23 : F (5/0) | | | No Of Hardware Proj Est >= 3 : F (1/0) | | No Of Hardware Proj Est >= 4.5 : A (5/0) | Hardware Proj Mgmt Exp >= 6 : F (4/0) TechUGCrses >= 45.5 : A (2/0) Some Background 2007
  5. 5. PROMISE 2008 versus 2007 The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html <ul><li>Sample set: 178 Samples </li></ul><ul><li>One learner  Accuracy and Intuitive Results </li></ul><ul><li>Attribute reduction Analysis. </li></ul><ul><li>Relatively Simple models. </li></ul>
  6. 6. The Approach http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop <ul><li>Personal Demographics </li></ul><ul><ul><li>Age, Gender, Nationality, etc. </li></ul></ul><ul><li>Academic </li></ul><ul><ul><li>Courses Undergrad/Grad: </li></ul></ul><ul><ul><ul><li>CS, HW, SE, Proj. Mgmt, MIS </li></ul></ul></ul><ul><ul><li>Workshops/Conferences: </li></ul></ul><ul><ul><li> CS, HW, SE, Proj. Mgmt, MIS </li></ul></ul><ul><li>Work </li></ul><ul><ul><li>Programming: Ada, ASP, Assembly, C, C++, </li></ul></ul><ul><ul><li>COBOL, DBMS, FORTRAN, Java, PASCAL, </li></ul></ul><ul><ul><li>Perl, PHP, SAP, TCL, VB, Other </li></ul></ul><ul><ul><li>Work Experience (HW/SW) </li></ul></ul><ul><ul><li>Project Management Exp. (HW/SW) </li></ul></ul><ul><ul><li># Projects Estimated (HW/SW) </li></ul></ul><ul><ul><li>Average Project Size </li></ul></ul><ul><li>Domain Experience </li></ul><ul><ul><li>Procurement Industry Experience </li></ul></ul>Estimate 28 Components Scale Factor And Correlation Apply Machine Learners Buyer Admin Buyer 1 Buyer n ... Buyer Software Distribution Server Supplier 1 Supplier 2 Supplier n : Supplier Software
  7. 7. Feedback to Users http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop How user compares to other respondents User’s Estimates Actual Estimates
  8. 8. Experiments: Data http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop Original Data set Experiment 1 Experiment 2 Experiment 3 82.8 -29.4 0.008 29X Correlation S c a l e Correlation S c a l e Correlation S c a l e Correlation S c a l e
  9. 9. Experiments: Tools, Configuration http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop <ul><li>Outliers Removed </li></ul><ul><li>WEKA Toolset </li></ul><ul><li>C4.5 (J48) </li></ul><ul><li>1000 Trials </li></ul><ul><li>10-Fold Cross Validation </li></ul>
  10. 10. Results: Correlation Only http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop 2-Class Problem: 10 Best (A), 10 Worst (F) 1000 Trials, Accuracy = 41.6% Attribute Reduction using WRAPPER 1000 Trials, Accuracy = 78.6%
  11. 11. Results: Scale Factor Only http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop 1000 Trials, Accuracy = 65.0% Attribute Reduction using WRAPPER 1000 Trials, Accuracy = 78.2% 2-Class Problem: 10 Best (A), 10 Worst (F)
  12. 12. Results: Correlation & Scale Factor http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop 1000 Trials, Accuracy = 82.2% Attribute Reduction using WRAPPER 1000 Trials, Accuracy = 93.3% 2-Class Problem: 10 Best (A), 10 Worst (F)
  13. 13. Discussion - 1 http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop How well does the decision tree from the third experiment apply to all the respondents minus outliers? Best Estimators Poorest Estimators Average Correlation 0.4173 0.3686 Average Scale Factor 2.6198 2.7419
  14. 14. Discussion - 2 http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop <ul><li>Scope of effort </li></ul><ul><li>Amortization of effort </li></ul><ul><li>Reuse can skew estimates (esp. Design for Reuse) </li></ul><ul><li>Respondent’s estimates = Boetticher’s estimates </li></ul>Challenges in component effort estimation
  15. 15. Conclusions <ul><li>Good accuracy rates, </li></ul><ul><li>especially after attribute reduction </li></ul><ul><li>Correlation + Scale Factor  Intuitive Model </li></ul><ul><li>Bridges expert and model groups </li></ul>http://nas.cl.uh.edu/boetticher/publications.html The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop
  16. 16. http://nas.cl.uh.edu/boetticher/publications.html Thank You ! The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop
  17. 17. References <ul><li>Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Systems and Software, 2004. </li></ul><ul><li>J ø rgensen, Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33, 1, January, 2007, Pp. 33-53. </li></ul>The 4 th International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html

×