Presentation shigeru sasao_pairedcomparison_2

209 views
122 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
209
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Presentation shigeru sasao_pairedcomparison_2

  1. 1. PAIRED COMPARISON: A USER PERSPECTIVE Shigeru Sasao Master of Software Engineering Carnegie Mellon University
  2. 2. AGENDA Introduction to Paired Comparison  Experiment comparing Ad-hoc, Planning Poker, and Paired Comparison  Observation of Usage in a Project  Points of Improvement  New Version Using Incomplete Cyclic Design  Conclusion  2
  3. 3. INTRODUCTION TO PAIRED COMPARISON We want to estimate the size of the following objects.  We know that size of C is 10 units big.  B A C D 3
  4. 4. B A C  D Estimate the relative sizes for all pairs:  I think A is 3 times as large as D.  I think A is half the size of B.  I think B is 6 times as large as D. ... 4
  5. 5.   A B C D A 1  0.5 2 3 B  2  1 0.5 6 C  0.5  2  1 2 D  0.3  0.17  0.5 I think A is 3 times bigger than D…  1 This is the “Judgment Matrix”.  Principal diagonal is always 1.  The other shaded regions are reciprocals of the un-shaded region.  Does not have to be perfectly consistent, since it is an estimate.  5
  6. 6.  From the Judgment Matrix, we can calculate the estimates of the object size: 1. Calculate the geometric mean of each row. 2. Obtain the “ratio scale”. 3. Calculate the estimated size. 6
  7. 7.  For our example, 1. Calculate the geometric mean of each row. A B C D 2. 1.32 1.57 1.19 0.41 Obtain the “ratio scale”. • Sum the geometric means 1.32 + 1.57 + 1.19 + 0.41 = 4.49 • Divide geometric mean of the row by the sum of the geometric mean. A B C D 0.29 0.35 0.26 0.09 7
  8. 8. 3. Calculate the estimated size. • We know from the beginning that C has size of 10 units. • So, the reference value is C with ratio scale entry of 0.26. A 0.29/0.26 * 10 = 11.15 units B 0.35/0.26 * 10 = 13.46 units C 0.26/0.26 * 10 = 10.00 units D 0.09/0.26 * 10 = 3.46 units 8
  9. 9. INTRODUCTION TO PAIRED COMPARISON  What did we do?  We knew the size of 1 object (reference size).  We compared pairs of relative sizes, and entered it into a judgment matrix.  Calculated the estimates for the objects.  The method reduces individual judgment errors by requiring multiple pair-wise comparisons of relative values. 9
  10. 10. EXPERIMENT COMPARING AD-HOC, PLANNING POKER, AND PAIRED COMPARISON  Comparison was made between ad-hoc methods, planning poker and paired comparison.  Study conducted among students from the Master of Software Engineering program at Carnegie Mellon.  Students have strong technical background with two to three years of industry experience. 10
  11. 11. EXPERIMENT SETUP Students divided into three groups, with each group using either ad-hoc, planning poker, or paired comparison.  Estimations conducted in pairs.  Five pairs per estimation technique.  Students were asked to estimate the size (Lines of Code) of different data structures (stack, queue, binary tree, etc).  Students were told that the size of “linked list (a)” was 40 LOC.  11
  12. 12. EXPERIMENT RESULT 12
  13. 13. EXPERIMENT RESULT Planning poker and paired comparison show a vast improvement in precision over ad-hoc methods.  This result is consistent with previous reports by Miranda.  13 Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000.
  14. 14. EXPERIMENT RESULT  Comparison of standard deviation between planning poker and paired comparison: Data Structure Stack Queue Binary Tree String Manipulation Linked List (b) Balanced Tree Hash Table Std. Dev. (Planning Poker) 9.8 11.4 25.5 31.6 42.4 56.0 33.1 Std. Dev. (Paired Comparison) 2.4 4.9 13.3 8.6 5.5 21.4 31.5 14
  15. 15. EXPERIMENT RESULT  Paired comparison produced more consistent estimations among estimators as compared to adhoc and planning poker.  Earlier studies by both Miranda and Shepperd show that paired comparison produces more reliable estimations than ad-hoc estimation methods. 15
  16. 16. OBSERVATION OF USAGE IN A PROJECT In the Master of Software Engineering program, a team of four to five members work for an external customer.  16 months with a total resource of 4608 to 5760 person hours.  Observed the usage of paired comparison by a MSE team.  The project was to produce an integrated development environment to support global distributed teams.  Estimation of effort for architectural components prior to the detailed design/implementation stage.  16
  17. 17. OBSERVATION OF USAGE IN A PROJECT Five team members and observers gathered in a conference for the estimation session.  A list of 12 components was prepared beforehand.  Pair-wise comparison between all 12 components, for a total of 66 comparisons.  For each comparison, team discussed size, complexity, number of state transitions to agree on a relative size.  Values entered into a spreadsheet tool.  17
  18. 18. OBSERVATION OF USAGE IN A PROJECT Data Elem, Mgr Traceability controller 20 Import UI Trac. View Cont Data UI Traceability UI Traceability Server ADT Def Transf. Chain Import Mgr Traceability Conf RTC Bridge  2.0 9.5 4.1 4.5 7.0 1.8 2.5 0.5 0.2 2.0 4.3 2.2 0.2 1.8 0.8 5.5 2.0 0.4 1.5 0.5 0.8 5.6 2.2 0.4 1.6 0.6 0.7 1.1 7.0 5.5 0.7 2.2 0.8 2.5 2.0 2.2 9.0 6.5 0.8 3.5 2.8 3.8 3.5 3.7 2.5 8.0 6.0 0.9 3.0 2.4 3.5 3.2 3.4 2.8 1.6 RTC Bridge Traceability Conf Import Mgr Transf. Chain ADT Def Traceability Server Traceability UI Data UI Trac. View Cont Artifact Name Import UI Reference Value Traceability controller Snapshot of the resulting judgment matrix: Data Elem, Mgr  1.5 0.5 0.1 0.3 0.4 0.4 0.4 0.4 0.3 0.1 0.1 Ratio Scale 0.256 0.135 0.020 0.076 0.050 0.067 0.067 0.067 0.038 0.021 0.020 0.182 Estimated Value 258.7 136.5 20.0 76.7 50.8 67.4 67.9 67.5 38.4 21.0 20.4 183.7 Total of 66 comparisons in 54 minutes. 18
  19. 19. POINTS OF IMPROVEMENT 1. Estimators became noticeably tired during the 54 minute estimation session. 2. As estimators became tired, they started extrapolating the new relative size values from old ones that have already been determined.  “Since A was 3 times the size of C and C was 2 times the size of D, A must be 6 times the size of D…” 19
  20. 20. NEW VERSION USING INCOMPLETE CYCLIC DESIGN To consider the estimators’ stamina, we need to reduce the number of comparison.  Use incomplete cyclic designs to reduce comparisons.   Maintains balance and connectedness. 20
  21. 21. NEW VERSION USING INCOMPLETE CYCLIC DESIGN Control the number of comparisons by the replication factor.  Example for a 10 component comparison:  Replication Factor 10 8 6 4 2 Number of Comparisons 45 40 30 20 10 21
  22. 22. NEW VERSION USING INCOMPLETE CYCLIC DESIGN  Full comparison of 8 components (28 comparisons) A B C D E F G H A B C D E F G H  Replication factor of 4 (16 comparisons) A A B C D E F G H B C D E F G H 22
  23. 23. NEW VERSION USING INCOMPLETE CYCLIC DESIGN  The reduction of comparison is made possible through:  Incomplete cyclic design (ICD)  Imputing missing values as the geometric mean of comparisons made  Studies have shown that very low replication factors still produce reliable estimates, although precision may diminish as number of comparisons decrease. 23
  24. 24. NEW VERSION USING CYCLIC DESIGN  Developed a tool which uses the new version with cyclic design: Replication factor Randomized comparison order 24
  25. 25. INTEGRATION WITH COCOMO  Use paired comparison to estimate LOC size as input into COCOMO. 25
  26. 26. CONCLUSION Paired comparison produces more consistent estimations than ad-hoc methods and planning poker.  Can be used to directly estimate effort, or to estimate LOC size for input into parametric models such as COCOMO.  New version using incomplete cyclic designs reduce the number of comparison.  If you are interested in the tool, please contact me.  26
  27. 27. REFERENCES          [Aguaron et al. 2003] Aguaron, Juan, Moreno-Jimenez, Jose Maria, The Geometric Consistency Index: Approximated Thresholds, European Journal of Operational Research, 147 (2003), Pages 137-145. [Crawford, Williams 1985] Crawford, Gordon, Williams, Cindy, The Analysis of Subjective Judgment Matrices, Rand Corporation, 1985. [Miranda 2000] Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000. [Miranda 2001] Miranda, Eduardo, Improving Subjective Estimates Using Paired Comparison, IEEE Software, 2001. [Miranda et al. 2009] Miranda, Eduardo, Bourque, Pierre, Abran, Alain, Sizing User Stories Using Paired Comparisons, Information and Software Technology Volume 51, Issue 9, September 2009, Pages 1327-1337, Butterworth-Heinemann, 2009. [PlanningPoker] Cohn, Mike, Planning Poker, www.planningpoker.com, Mountain Goat Software. [Shepperd et al. 2001] Shepperd, Martin, Cartwright, Michelle, Predicting with Sparse Data, IEEE Transactions on Software Engineering, VOL. 27, NO. 11, 2001. [Spencer 1982] Spencer, Ian, Incomplete Experimental Designs for Multidimensional Scaling, Chapter 3. In R.G. Golledge and J.N. Rayner (Eds.), Proximity and Preference: Problems in the Multidimensional Analysis of Large Data Sets, University of Minnesota Press, 1982. [Spencer 1983] Spencer, Ian, Monte Carlo Simulation Studies, Applied Psychological Measurement Vol. 7, No. 4, 1983. 27

×