Assessing the Reliability of a Human Estimator

Assessing the Reliability of a Human Estimator http:// nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Gary D. Boetticher Nazim Lokhandwala Univ. of Houston - Clear Lake, Houston, TX, USA [email_address] [email_address]

Current Configuration of PROMISE Repository
Defect Prediction – 18
http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Others - 9 Effort Estimation - 9

Research vs. Reality according to Jörgensen
TSE ’07: 300+ software est. papers,
76 journals, 15+ Years
http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop JSS ’04: Compendium of expert estimation studies 52 26 19 7 Misc. 46 21 22 3 Human 74 41 32 1 ML 255 70 137 48 Algorithm Total 00-04 89-99 -89 68% Algorithm 20% ML 12% Human 72% Kitchenham 02 100% Hill 00 84% Jørgensen 97 86% Paynter 96 62% Heemstra 91 89% Hihn 91 Human Paper 82% Human 18% Formal

Research vs. Reality How to resolve? http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
Researchers coerce/entice/exhort/nudge practitioners
Practitioners ignore researchers
Researchers meet practitioners where they are
COCOMO

Statement of Problem
How do human demographics affect human-based estimation?
Can predictive models be constructed using human demographics? http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

PROMISE 2006
Addressed the problem using Genetic Programs and non-linear regression (up to 5 th order) models
Produced some accurate(77 – 93%) models, GP solutions got lengthy:
The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html (( MgmtGCourses ^ ((( Log ((( TotLangExp / ( TotLangExp / ( TechGCourses * HWPMExp ))) - ( TechGCourses * HWPMExp )) - (( Sin ( MgmtGCourses ^ ( Sin (( TechGCourses * HWPMExp ) - ( MgmtGCourses ^ ((( Log ( HWPMExp ^ ( TotLangExp / ( TechGCourses * HWPMExp )))) - ( Abs ( Log (( TotLangExp / ( TechGCourses * HWPMExp )) - (( Sin (( Sin ( Abs ( TechUGCourses / MgmtGCourses ))) - ( TotLangExp / ( MgmtGCourses ^ ((( Log ((( TotLangExp / ( HWPMExp / SWProjEstExp )) - ( Sin ( TotLangExp / ( TotLangExp / (( MgmtGCourses ^ (( Log ( TechGCourses * HWPMExp )) - ( Sin ( Abs ( Log (( HWPMExp / SWProjEstExp ) - ( TechGCourses * HWPMExp ))))))) + (( Sin ( TechGCourses * HWPMExp )) - ( Sin ( TechUGCourses / MgmtGCourses )))))))) - ( Sin ( TechUGCourses / MgmtGCourses )))) - ( TechGCourses * HWPMExp )) - ( Sin ( TechUGCourses / MgmtGCourses ))))))) - ( HWPMExp / SWProjEstExp )))))) - ( Sin ( TechUGCourses / MgmtGCourses )))))))) - (( Sin ( Abs ( Log (( TotLangExp / ( TechGCourses * HWPMExp )) - (( Sin (( Sin ( Abs ( Log ( HWPMExp ^ ( TotLangExp / ( TechGCourses * HWPMExp )))))) - ( TechGCourses * HWPMExp ))) - ( HWPMExp / SWProjEstExp )))))) - ( Sin ( TechUGCourses / MgmtGCourses )))))) - ( TotLangExp / ( TechGCourses * HWPMExp ))) - ( Sin ( TechUGCourses / MgmtGCourses )))) + ( TotLangExp / ( TechGCourses * HWPMExp)))
So for 2007…

PROMISE 2007 The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html
Larger sample set.
2006 PROMISE  122 samples
2007 PROMISE  178 samples
Many learners.
51 classifiers, 4142 experimental trials
Attribute analysis.
Simpler models.
Focus is on classifiers  Human readable models

Strategy
Create a Web-based survey
Users  demographics
Users  Estimate software components
Feedback  Users
Build models: demographics  estimates
http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

The Survey (2001 -2005) http:// nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
Demographics
Personal
Academic Background
Work Experience
Domain Experience

Ecommerce: Competitive Procurement Buyer Admin Buyer 1 Buyer n ... Buyer Software Distribution Server Supplier 1 Supplier 2 Supplier n : Supplier Software http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

Sample Estimation Screenshots http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

Feedback to Users http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

User Demographics - 1 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop
Average age: 31.43
148 males, 30 females
1% Ph.D., 24% Master, 72% Bach., 5% High School
25 countries:
42% India, 32% U.S., 6% Romania, 4% Vietnam.

User Demographics - 2 http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 5.3856 28 3.6629 Process Industry 4.4391 25 1.4382 Procurement & Billing Domain Experience 5.3856 28 3.6692 Software Projects 4.4390 25 1.4382 Hardware Projects No. of Projects estimated 2.4757 15 1.6967 Software Project Manager 3.0633 25 1.0169 Hardware Project Manager Years of Experience as a Std. Dev. Max. Ave. Years

Data preprocessing & Experiments http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 178 Samples WEKA: 51 Classifiers, 4 seeds, 10-fold Attribute Reduction: 2 configs. Remove outliers: Estimate > 10 * Actual or Estimate < 0.1*Actual 163
Extract:
25 Worst under -estimators
25 Best estimators
25 Worst over -estimators

Results: Under vs. Best http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Ave. Accuracy 48.22% 64% VFI 64% ThresSel 64% Logistic 68% J48 76% PART Accuracy Classifier Y Y Y Total Lang Exp. Y Y Y Total Workshops Y Total Conferences Y Tech Undergrad Courses Y Y Software Proj. Mgmt Exp. Y Y Level of College Y Y Y Y # of Hardware Proj. Est. Y Y Y Mgmt Undergrad Crses Y Mgmt Grad. Courses Y Y Y Y Hardware Project Management Exp. Y Y Y Domain Exp. VFI Thresh. PART Logistic J48 Demographic Evaluator Classifier 68% Logistic/ Logistic 70% VFI / VFI 74% PART/J48 74% J48/J48 74% LogitBoost/J48 74% Bagging/J48 76% ThresholdSel/ ThresholdSel 78% ADTree/Part Accuracy Class./Eval.

Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Y Y Total Lang Experience Y Y Total Workshops Y Y Total Conferences Y Y Tech Undergrad Crses Y Y Soft. Proj. Mgmt Exp. Y Y Y Level of College Y # of Software Proj. Est. Y Y Y Y # of Hardware Proj. Est. Y Y Y Mgmt Undergrad Crses Y Y Y Mgmt Grad. Courses Y Y Y Y Hardware Proj. Mgmt Exp. Y Y Y Y Domain Experience VFI Thresh PART Logistic J48 Demographic Evaluator Classifier 68% Logistic / Logistic 70% VFI / VFI 74% PART / J48 74% ADTree / J48 74% PART/ PART 74% J48/ PART 76% ADTree/ ThreshSel Accuracy Class / Eval

Under vs. Best: Attribute Reduction http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Domain Exp <= 3 | No Of Hardware Proj Estimated <= 4 | | Hardware Proj Mgmt Exp <= 1 | | | MgmtUGCourses <= 0: BEST (23.0/8.0) | | | MgmtUGCourses > 0: UNDER (13.0/1.0) | | Hard. Proj Mgmt Exp > 1: BEST (5.0) | No Of Hard. Proj Est. > 4: UNDER (5.0) Domain Exp > 3: BEST (4.0) J48 Rule: 74% Accuracy BEST UNDER <-- classified as 21 4 | BEST 9 16 | UNDER

Results: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop Ave. Accuracy 42.86% 60% Ridor 60% ThresholdSel 60% RandComm 62% Decorate 66% RndTree Accuracy Classifier Y Y Total Lang Experience Y Total Workshops Y Y Total Conferences Y Tech Undergrad Courses Y Y Soft. Proj. Mgmt Exp. Y # of Software Proj. Est. Y Mgmt Undergrad Crses Y Y Y Mgmt Grad. Courses Y Y Y Hard. Proj Mgmt Exp. Threshold Selector Ridor Rnd Comm Demographic 62% ADTree / ThresholdSel 66% ThresholdSel / ThreshSel 72% Rand. Comm./ RandComm 80% IB1 / Ridor Accuracy Class/ Eval

Experiment: Best vs. Over http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop 62% Ridor Ridor 62% ThresholdSel Ridor 64% Ridor ThresholdSel 66% ThresholdSel NNge 72% Decorate PART 72% Decorate NNge 72% Decorate RndComm 74% Decorate RandomForest 74% Decorate IBk 74% Decorate IB1 80% RndComm RandomTree 80% RndComm RndComm Accuracy Evaluator Classifier Y Y Y Total Lang Experience Y Total Workshops Y Tech Undergrad Courses Y Y Tech Grad Courses Y Software Proj. Mgmt Exp. Y Y Procurement Industry Exp Y Level of College Y # of Hardware Proj. Est. Y Y Mgmt Undergrad Courses Y Mgmt Grad. Courses Y Y Y Hard. Proj Mgmt Exp Y Domain Experience Thresh Ridor Rand Comm. Decorate Demographic

Conclusions
Very Good accuracy rates,
especially after attribute reduction
Bridges expert and model groups
http://nas.cl.uh.edu/boetticher/publications.html The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

http://nas.cl.uh.edu/boetticher/publications.html Questions? The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

http://nas.cl.uh.edu/boetticher/publications.html Thank You ! The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop

References
Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Systems and Software, 2004.
J ø rgensen, Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33, 1, January, 2007, Pp. 33-53.
The 3 rd International Predictor Models in Software Engineering (PROMISE) Workshop http://nas.cl.uh.edu/boetticher/publications.html

Assessing the Reliability of a Human Estimator

by promise07 on May 20, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Assessing the Reliability of a Human Estimator — Presentation Transcript