Assessing the Reliability of a Human Estimator http:// nas.cl.uh.edu/boetticher/publications.html The 3 rd  International ...
Current Configuration of PROMISE Repository <ul><li>Defect Prediction – 18 </li></ul>http://nas.cl.uh.edu/boetticher/publi...
Research vs. Reality according to Jörgensen <ul><li>TSE ’07: 300+ software est. papers, </li></ul><ul><li>76 journals, 15+...
Research vs. Reality How to resolve? http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor M...
Statement of Problem <ul><li>How do human demographics affect human-based estimation? </li></ul>Can predictive models be c...
PROMISE 2006 <ul><li>Addressed the problem using Genetic Programs and non-linear regression (up to 5 th  order) models </l...
PROMISE 2007 The 3 rd  International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boet...
Strategy <ul><li>Create a Web-based survey </li></ul><ul><ul><ul><li>Users    demographics </li></ul></ul></ul><ul><ul><u...
The Survey  (2001 -2005) http:// nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html http://nas.cl.uh.edu/boetticher/publ...
Ecommerce: Competitive Procurement Buyer Admin Buyer 1 Buyer n ... Buyer Software Distribution Server Supplier 1 Supplier ...
Sample Estimation Screenshots   http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Models...
Feedback to Users http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Models in Software E...
User Demographics - 1 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Models in Softwa...
User Demographics - 2 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Models in Softwa...
Data preprocessing & Experiments http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Model...
Results: Under vs. Best http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Models in Soft...
Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Mo...
Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Mo...
Results: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Models in Softw...
Experiment: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd  International Predictor Models in So...
Experiment: Best vs. Over TechUGCourses < 45.5 | Hardware Proj Mgmt Exp < 6 | | No Of Hardware Proj Estimated < 4.5 | | | ...
Conclusions <ul><li>Very Good accuracy rates, </li></ul><ul><li>especially after attribute reduction </li></ul><ul><li>Bri...
http://nas.cl.uh.edu/boetticher/publications.html Questions? The 3 rd  International Predictor Models in Software Engineer...
http://nas.cl.uh.edu/boetticher/publications.html Thank You   ! The 3 rd  International Predictor Models in Software Engin...
References <ul><li>Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Sy...
Upcoming SlideShare
Loading in …5
×

Assessing the Reliability of a Human Estimator

982 views

Published on

Gary Boetticher and Nzim Lokhandwala

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
982
On SlideShare
0
From Embeds
0
Number of Embeds
46
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Assessing the Reliability of a Human Estimator

    1. 1. Assessing the Reliability of a Human Estimator http:// nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Gary D. Boetticher Nazim Lokhandwala Univ. of Houston - Clear Lake, Houston, TX, USA [email_address] [email_address]
    2. 2. Current Configuration of PROMISE Repository <ul><li>Defect Prediction – 18 </li></ul>http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Others - 9 Effort Estimation - 9
    3. 3. Research vs. Reality according to Jörgensen <ul><li>TSE ’07: 300+ software est. papers, </li></ul><ul><li>76 journals, 15+ Years </li></ul>http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop JSS ’04: Compendium of expert estimation studies 52 26 19 7 Misc. 46 21 22 3 Human 74 41 32 1 ML 255 70 137 48 Algorithm Total 00-04 89-99 -89 68% Algorithm 20% ML 12% Human 72% Kitchenham 02 100% Hill 00 84% Jørgensen 97 86% Paynter 96 62% Heemstra 91 89% Hihn 91 Human Paper 82% Human 18% Formal
    4. 4. Research vs. Reality How to resolve? http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop <ul><li>Researchers coerce/entice/exhort/nudge practitioners </li></ul><ul><li>Practitioners ignore researchers </li></ul><ul><li>Researchers meet practitioners where they are </li></ul>COCOMO
    5. 5. Statement of Problem <ul><li>How do human demographics affect human-based estimation? </li></ul>Can predictive models be constructed using human demographics? http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    6. 6. PROMISE 2006 <ul><li>Addressed the problem using Genetic Programs and non-linear regression (up to 5 th order) models </li></ul><ul><li>Produced some accurate(77 – 93%) models, GP solutions got lengthy: </li></ul>The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html (( MgmtGCourses ^ ((( Log ((( TotLangExp / ( TotLangExp / ( TechGCourses * HWPMExp ))) - ( TechGCourses * HWPMExp )) - (( Sin ( MgmtGCourses ^ ( Sin (( TechGCourses * HWPMExp ) - ( MgmtGCourses ^ ((( Log ( HWPMExp ^ ( TotLangExp / ( TechGCourses * HWPMExp )))) - ( Abs ( Log (( TotLangExp / ( TechGCourses * HWPMExp )) - (( Sin (( Sin ( Abs ( TechUGCourses / MgmtGCourses ))) - ( TotLangExp / ( MgmtGCourses ^ ((( Log ((( TotLangExp / ( HWPMExp / SWProjEstExp )) - ( Sin ( TotLangExp / ( TotLangExp / (( MgmtGCourses ^ (( Log ( TechGCourses * HWPMExp )) - ( Sin ( Abs ( Log (( HWPMExp / SWProjEstExp ) - ( TechGCourses * HWPMExp ))))))) + (( Sin ( TechGCourses * HWPMExp )) - ( Sin ( TechUGCourses / MgmtGCourses )))))))) - ( Sin ( TechUGCourses / MgmtGCourses )))) - ( TechGCourses * HWPMExp )) - ( Sin ( TechUGCourses / MgmtGCourses ))))))) - ( HWPMExp / SWProjEstExp )))))) - ( Sin ( TechUGCourses / MgmtGCourses )))))))) - (( Sin ( Abs ( Log (( TotLangExp / ( TechGCourses * HWPMExp )) - (( Sin (( Sin ( Abs ( Log ( HWPMExp ^ ( TotLangExp / ( TechGCourses * HWPMExp )))))) - ( TechGCourses * HWPMExp ))) - ( HWPMExp / SWProjEstExp )))))) - ( Sin ( TechUGCourses / MgmtGCourses )))))) - ( TotLangExp / ( TechGCourses * HWPMExp ))) - ( Sin ( TechUGCourses / MgmtGCourses )))) + ( TotLangExp / ( TechGCourses * HWPMExp))) <ul><li>So for 2007… </li></ul>
    7. 7. PROMISE 2007 The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html <ul><li>Larger sample set. </li></ul><ul><ul><li>2006 PROMISE  122 samples </li></ul></ul><ul><ul><li>2007 PROMISE  178 samples </li></ul></ul><ul><li>Many learners. </li></ul><ul><ul><li>51 classifiers, 4142 experimental trials </li></ul></ul><ul><li>Attribute analysis. </li></ul><ul><li>Simpler models. </li></ul><ul><ul><li>Focus is on classifiers  Human readable models </li></ul></ul>
    8. 8. Strategy <ul><li>Create a Web-based survey </li></ul><ul><ul><ul><li>Users  demographics </li></ul></ul></ul><ul><ul><ul><li>Users  Estimate software components </li></ul></ul></ul><ul><ul><ul><li>Feedback  Users </li></ul></ul></ul><ul><li>Build models: demographics  estimates </li></ul>http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    9. 9. The Survey (2001 -2005) http:// nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop <ul><li>Demographics </li></ul><ul><li>Personal </li></ul><ul><li>Academic Background </li></ul><ul><li>Work Experience </li></ul><ul><li>Domain Experience </li></ul>
    10. 10. Ecommerce: Competitive Procurement Buyer Admin Buyer 1 Buyer n ... Buyer Software Distribution Server Supplier 1 Supplier 2 Supplier n : Supplier Software http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    11. 11. Sample Estimation Screenshots http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    12. 12. Feedback to Users http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    13. 13. User Demographics - 1 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop <ul><li>Average age: 31.43 </li></ul><ul><li>148 males, 30 females </li></ul><ul><li>1% Ph.D., 24% Master, 72% Bach., 5% High School </li></ul><ul><li>25 countries: </li></ul><ul><ul><li>42% India, 32% U.S., 6% Romania, 4% Vietnam. </li></ul></ul>
    14. 14. User Demographics - 2 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 5.3856 28 3.6629 Process Industry 4.4391 25 1.4382 Procurement & Billing Domain Experience 5.3856 28 3.6692 Software Projects 4.4390 25 1.4382 Hardware Projects No. of Projects estimated 2.4757 15 1.6967 Software Project Manager 3.0633 25 1.0169 Hardware Project Manager Years of Experience as a Std. Dev. Max. Ave. Years
    15. 15. Data preprocessing & Experiments http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 178 Samples WEKA: 51 Classifiers, 4 seeds, 10-fold Attribute Reduction: 2 configs. Remove outliers: Estimate > 10 * Actual or Estimate < 0.1*Actual 163 <ul><li>Extract: </li></ul><ul><ul><li>25 Worst under -estimators </li></ul></ul><ul><ul><li>25 Best estimators </li></ul></ul><ul><ul><li>25 Worst over -estimators </li></ul></ul>
    16. 16. Results: Under vs. Best http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Ave. Accuracy 48.22% 64% VFI 64% ThresSel 64% Logistic 68% J48 76% PART Accuracy Classifier Y Y Y Total Lang Exp. Y Y Y Total Workshops Y Total Conferences Y Tech Undergrad Courses Y Y Software Proj. Mgmt Exp. Y Y Level of College Y Y Y Y # of Hardware Proj. Est. Y Y Y Mgmt Undergrad Crses Y Mgmt Grad. Courses Y Y Y Y Hardware Project Management Exp. Y Y Y Domain Exp. VFI Thresh. PART Logistic J48 Demographic Evaluator Classifier 68% Logistic/ Logistic 70% VFI / VFI 74% PART/J48 74% J48/J48 74% LogitBoost/J48 74% Bagging/J48 76% ThresholdSel/ ThresholdSel 78% ADTree/Part Accuracy Class./Eval.
    17. 17. Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Y Y Total Lang Experience Y Y Total Workshops Y Y Total Conferences Y Y Tech Undergrad Crses Y Y Soft. Proj. Mgmt Exp. Y Y Y Level of College Y # of Software Proj. Est. Y Y Y Y # of Hardware Proj. Est. Y Y Y Mgmt Undergrad Crses Y Y Y Mgmt Grad. Courses Y Y Y Y Hardware Proj. Mgmt Exp. Y Y Y Y Domain Experience VFI Thresh PART Logistic J48 Demographic Evaluator Classifier 68% Logistic / Logistic 70% VFI / VFI 74% PART / J48 74% ADTree / J48 74% PART/ PART 74% J48/ PART 76% ADTree/ ThreshSel Accuracy Class / Eval
    18. 18. Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Domain Exp <= 3 | No Of Hardware Proj Estimated <= 4 | | Hardware Proj Mgmt Exp <= 1 | | | MgmtUGCourses <= 0: BEST (23.0/8.0) | | | MgmtUGCourses > 0: UNDER (13.0/1.0) | | Hard. Proj Mgmt Exp > 1: BEST (5.0) | No Of Hard. Proj Est. > 4: UNDER (5.0) Domain Exp > 3: BEST (4.0) J48 Rule: 74% Accuracy BEST UNDER <-- classified as 21 4 | BEST  9 16 | UNDER
    19. 19. Results: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Ave. Accuracy 42.86% 60% Ridor 60% ThresholdSel 60% RandComm 62% Decorate 66% RndTree Accuracy Classifier Y Y Total Lang Experience Y Total Workshops Y Y Total Conferences Y Tech Undergrad Courses Y Y Soft. Proj. Mgmt Exp. Y # of Software Proj. Est. Y Mgmt Undergrad Crses Y Y Y Mgmt Grad. Courses Y Y Y Hard. Proj Mgmt Exp. Threshold Selector Ridor Rnd Comm Demographic 62% ADTree / ThresholdSel 66% ThresholdSel / ThreshSel 72% Rand. Comm./ RandComm 80% IB1 / Ridor Accuracy Class/ Eval
    20. 20. Experiment: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 62% Ridor Ridor 62% ThresholdSel Ridor 64% Ridor ThresholdSel 66% ThresholdSel NNge 72% Decorate PART 72% Decorate NNge 72% Decorate RndComm 74% Decorate RandomForest 74% Decorate IBk 74% Decorate IB1 80% RndComm RandomTree 80% RndComm RndComm Accuracy Evaluator Classifier Y Y Y Total Lang Experience Y Total Workshops Y Tech Undergrad Courses Y Y Tech Grad Courses Y Software Proj. Mgmt Exp. Y Y Procurement Industry Exp Y Level of College Y # of Hardware Proj. Est. Y Y Mgmt Undergrad Courses Y Mgmt Grad. Courses Y Y Y Hard. Proj Mgmt Exp Y Domain Experience Thresh Ridor Rand Comm. Decorate Demographic
    21. 21. Experiment: Best vs. Over TechUGCourses < 45.5 | Hardware Proj Mgmt Exp < 6 | | No Of Hardware Proj Estimated < 4.5 | | | No Of Hardware Proj Estimated < 3 | | | | TechUGCourses < 23 | | | | | Hardware Proj Mgmt Exp < 0.75 | | | | | | TechUGCourses < 18 | | | | | | | Hardware Proj Mgmt Exp < 0.13 | | | | | | | | TechUGCourses < 0.5 | | | | | | | | | TechUGCourses < -1 : F (1/0) | | | | | | | | | TechUGCourses >= -1 | | | | | | | | | | Degree < 3.5 : A (4/0) | | | | | | | | | | Degree >= 3.5 : A (5/2) | | | | | | | | TechUGCourses >= 0.5 | | | | | | | | | TechUGCourses < 5.5 | | | | | | | | | | Degree < 3.5 : F (5/0) | | | | | | | | | | Degree >= 3.5 | | | | | | | | | | | TechUGCrses < 2 : A (1/0) | | | | | | | | | | | TechUGCrses >= 2 : F (1/0) | | | | | | | | | TechUGCrses >= 5.5 | | | | | | | | | | Degree < 3.5 | | | | | | | | | | | TechUGCrs < 10.5 : A (3/0) | | | | | | | | | | | TechUGCrses >= 10.5 | | | | | | | | | | | | TechUGCrs<12.5 : F (3/0) | | | | | | | | | | | | TechUGCrses >= 12.5 | | | | | | | | | | | | | TechUGCrs<16: A (2/0) | | | | | | | | | | | | | TechUGCrs>15 : A (2/1) | | | | | | | | | | Degree >= 3.5 : F (1/0) | | | | | | | HardProjMgmt Exp >= 0.13 : A (2/0) | | | | | | TechUGCourses >= 18 : A (2/0) | | | | | Hard Proj Mgmt Exp >= 0.75 : F (1/0) | | | | TechUGCourses >= 23 : F (5/0) | | | No Of Hardware Proj Est >= 3 : F (1/0) | | No Of Hardware Proj Est >= 4.5 : A (5/0) | Hardware Proj Mgmt Exp >= 6 : F (4/0) TechUGCrses >= 45.5 : A (2/0) The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop BEST OVER <-- classified as 23 2 | BEST  8 17 | OVER
    22. 22. Conclusions <ul><li>Very Good accuracy rates, </li></ul><ul><li>especially after attribute reduction </li></ul><ul><li>Bridges expert and model groups </li></ul>http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    23. 23. http://nas.cl.uh.edu/boetticher/publications.html Questions? The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    24. 24. http://nas.cl.uh.edu/boetticher/publications.html Thank You ! The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
    25. 25. References <ul><li>Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Systems and Software, 2004. </li></ul><ul><li>J ø rgensen, Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33, 1, January, 2007, Pp. 33-53. </li></ul>The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html

    ×