Multi-criteria Decision Analysis for Customization of Estimation by Analogy Method

1,753 views

Published on

Multi-criteria Decision Analysis for Customization of Estimation by Analogy Method - PROMISE 2008

Published in: Economy & Finance, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,753
On SlideShare
0
From Embeds
0
Number of Embeds
80
Actions
Shares
0
Downloads
60
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Slide 11: 4 Heuristics as a header for the lower Introduce a simplified formulae exlaing how the coefficients were calculated. - Be prepared for the following questions: (1) Are there alternatives to using RSA to determine the importance of the attributes? (2) What is the overall effort of the method(s) (3) Wham means the name AQUA? (4) When do you recommend apply the method? (and when better not?) (5) Needs the learning be done after each new prediction (data point)??
  • Multi-criteria Decision Analysis for Customization of Estimation by Analogy Method

    1. 1. Multi-criteria Decision Analysis for Customization of Estimation by Analogy Method AQUA + Jingzhou Li Guenther Ruhe University of Calgary, Canada PROMISE’08, May 13, 2008
    2. 2. /14 Why this Paper? <ul><li>Practitioners need better advice on how and when </li></ul><ul><li>to use methodologies </li></ul><ul><li>Universal, project-independent methodologies </li></ul><ul><li>are characterized as “weak” in the field of problem solving </li></ul><ul><li>(Robert Glass) </li></ul><ul><li>EBA is no exception in that respect! </li></ul><ul><li>But: How to figure out which variant works best when? </li></ul><ul><li>We do NOT claim to “solve” this problem </li></ul><ul><li>The paper describes an approach to make progress </li></ul><ul><li>on the question of customization </li></ul><ul><li>Approach is: Multi-criteria decision analysis </li></ul>
    3. 3. /14 Prediction accuracy distribution 1. Proposed EBA method AQUA + —Architecture Data set for AQUA + AQUA Existing EBA Predicting Phase2 Effort estimates Objects under estimation Learning Phase1
    4. 4. /14 Effort estimates Prediction accuracy distribution Objects under estimation Attributes & weights Effort estimates Objects under estimation Data set for AQUA + 1. Proposed EBA method AQUA + —Architecture <ul><li>Supports non-quantitative attributes </li></ul><ul><li>Tolerates missing values </li></ul><ul><li>Determines the number of analogies for adaptation by learning </li></ul><ul><li>Proposes new evaluation criteria </li></ul><ul><li>Attribute weighting and selection using RSA </li></ul><ul><li>Four heuristics: H1 to H4 </li></ul>AQUA + AQUA Existing EBA Learning Phase1 Predicting Phase2 Attribute weighting and selection Phase0 Predicting Phase2 Raw historical data Determining attribute types Pre-Phase
    5. 5. 2. Decision-centric process model of EBA /14 Processed Historical Data Raw Historical Data D8. Determining closest analogs D2. Dealing with missing values D1. Impact analysis of missing values D7. Retrieving analogs Objects Under Estimation Effort Estimates D9. Analogy adaptation D11. Comparing EBA methods in general D10. Choosing evaluation criteria D6. Determining similarity measures D3. Object selection D5. Attribute weighting & selection D4. Discretization of attributes
    6. 6. /14 EBA ( DB ) = C ( D , DB , Ch ) Data set type 1 Data set type 2 Data set type k …… Classification according to characteristics of the data sets S i.j for D i ? 3. Customization of EBA — why? D = { D 1 , D 2 , …, D 11 }, D i = { S i.j | solution alternatives of task D i } DB: a historical data set for EBA Ch: a set of characteristics describing DB Customization 1 Customization 2 Customization k
    7. 7. /14 New Data Set Which heuristic should be used? 4. Customization of EBA — how? Empirical knowledge gained from empirical studies.
    8. 8. <ul><li>Decision problem : The selection of attribute weighting heuristic expected to be the best for a data set of given characteristics </li></ul><ul><li>Decision alternatives : Attribute weighting heuristics known from literature. Six heuristics were studied. </li></ul><ul><li>Evaluation method : The alternative heuristics are evaluated by applying them to different data sets for AQUA + . Six publicly available data sets were used. </li></ul><ul><li>Evaluation criteria : MMRE , Pred [6], and Strength [3]. In order to keep the criteria consistent for minimization, MMRE , 1- Pred , and 1- Strength were used. </li></ul><ul><li>Decision objective : Determine solution alternatives (heuristics) such that evaluation criteria MMRE , 1- Pred , and 1- Strength get minimized in a balanced manner. </li></ul>/14 5. Multi-criteria Decision Problem
    9. 9. <ul><li>Definition 1 . MRE ( rk )—Magnitude of Relative Error [6] </li></ul><ul><li>Definition 2 . MMRE(N , T) —Mean Magnitude of Relative Error [6] </li></ul><ul><li>Definition 3 . Pred ( α , N , T )—prediction at level α [6] , α = 0.25 </li></ul><ul><li>N - number of analogs, T – similarity threshold </li></ul><ul><li>Definition 4 . Strength(N , T) </li></ul><ul><li>Support(N , T) is the number of objects in R that can be estimated with a given values of ( N , T ). Strength(N , T) is then defined as the ratio of Support to the total number of objects in R . </li></ul>/14 6. Definition of Decision Criteria Pred(α, N, T)= MMRE(N , T) = MRE ( rk )=
    10. 10. /14 7. Data sets Data Sets #Objects #Attributes %Missing Values %Non-Quantitative Attributes Source USP05-RQ 121 14 2.54 71 Li et al., 2005 USP05-FT 76 14 6.8 71 Li et al., 2005 ISBSG04-2 158 24 27.24 63 ISBSG, 2004 Kem87 15 5 0 40 Kemerer et al., 1987 Mends03 34 6 0 0 Mendes et al., 2003 Desh89 81 10 0.006 20 Shepperd et al., 1997
    11. 11. /14 8. Decision Analysis Using ELECTRE Outranking graph and analysis data for Desh89 (an example) Heuristic MMRE Pred(0.25) H0 0.62 0.44 H1 0.61 0.44 H3 0.6 0.42 H4 0.59 0.42 CfsSubset (Cfs) 0.52 0.4 Wrapper (Wp) 0.66 0.43
    12. 12. <ul><li>Clusters of Pareto frontier of Desh89 with three criteria </li></ul>/14 9. Pareto Analysis Results ID MMRE 1-Pred(0.25) 1-Strength Heuristic Cluster 26 0.23 0.21 0.83 H1 0 8 0.11 0.17 0.93 H0 1 24 0.16 0.14 0.91 H1 1 12 0.27 0.31 0.64 H0 2 13 0.26 0.27 0.73 H0 2 16 0.61 0.56 0 H1 3 28 0.58 0.53 0.05 H1 3 46 0.59 0.58 0 H4 3 58 0.55 0.56 0.04 H4 3 59 0.58 0.54 0.02 H4 3 61 0.52 0.6 0 CfsSubset 3 17 0 0 0.99 H1 4 19 0.08 0 0.95 H1 4 62 0.02 0 0.98 CfsSubset 4
    13. 13. <ul><li>Clusters of Pareto frontier of Desh89 with cirteria 1-Pred(25) and 1-Strength </li></ul>/14 9. Pareto Analysis Results
    14. 14. /14 10. Conclusions and Future Work <ul><li>Future work: </li></ul><ul><li>Use PROMISE data base for benchmarking analysis </li></ul><ul><li>To broaden the scope from EBA method AQUA+ and its weighting attributes heuristics to other classes of decision and prediction problems </li></ul><ul><li>To study more weighting heuristics over additional available data sets </li></ul><ul><li>To investigate other aspects of EBA customization </li></ul>Analysis Method Analysis tool Number of alternatives Number of data points for each alternative Number of criteria Expert preference ELECTRE Outranking relation small Small Multiple Easy to apply Pareto analysis and clustering Pareto frontier and clustering large large Multiple Easy to apply
    15. 15. /14 Discussion and questions?
    16. 16. Major references <ul><li>M. Shepperd, C. Schofield, “Estimating Software Project Effort Using Analogies”, IEEE Transactions on Software Engineering , 23(1997) 736-743. </li></ul><ul><li>G. Ruhe, &quot;Software Engineering Decision Support and Empirical Investigations - A Proposed Marriage&quot;, The Future of Empirical Studies in Software Engineering (A. Jedlitschka, M. Ciolkowski, Eds.), Workshop Serious on Empirical Studies in Software Engineering, Vol. 2, 2003, pp 25-34. </li></ul><ul><li>T. Menzies, Z.H. Chen, J. Hihn, and K. Lum, &quot;Selecting Best Practices for Effort Estimation&quot;, IEEE Transactions on Software Engineering , Vol. 32, No. 11, 2006, pp 1-13. </li></ul><ul><li>R. Glass, &quot;Matching methodology to problem domain&quot;, Communications of the ACM , 47 (5), 19-21. </li></ul><ul><li>J.Z. Li, G. Ruhe, A. Al-Emran, and M.M. Ritcher, &quot;A Flexible Method for Effort Estimation by Analogy&quot;, Empirical Software Engineering , Vol. 12, No. 1, 2007, pp 65-106. </li></ul><ul><li>J.Z. Li, G. Ruhe, &quot;Decision Support Analysis for Software Effort Estimation by Analogy&quot;, Proceedings of ICSE 2007 Workshop on Predictor Models in Software Engineering (PROMISE'07) , USA, May 2007. </li></ul><ul><li>J.Z. Li, A. Ahmed, G. Ruhe, &quot;Impact Analysis of Missing Values on the Prediction Accuracy of Analogy-based Software Estimation Method AQUA&quot;, ESEM’07, Madrid, Spain, September 2007. </li></ul>/14

    ×