Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AN AUTOMATED APPROACH TO ASSIGN SOFTWARE CHANGE REQUESTS (Ph.D. Presentation)

917 views

Published on

Slides of my Ph.D. presentation on March 2014.

Published in: Software
  • Nice ppt, could you share me your presentation please? my email id is 568492181@qq.com thank you very much!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

AN AUTOMATED APPROACH TO ASSIGN SOFTWARE CHANGE REQUESTS (Ph.D. Presentation)

  1. 1. An Automated Approach to Assign Software Change Requests Ph.D. Thesis Yguarat˜a Cerqueira Cavalcanti Centro de Inform´atica – UFPE March 20, 2014
  2. 2. Agenda 1 Introduction 2 Literature Review 3 Survey on CR Assignment 4 Proposal 5 Experiment 6 Conclusions
  3. 3. 1 Introduction 2 Literature Review 3 Survey on CR Assignment 4 Proposal 5 Experiment 6 Conclusions
  4. 4. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Change Management Every software project changes (1st Lehman’s law) user needs defects new functionalities Changes are made during software development or after release (software maintenance and evolution) Changes need to be managed, instead you lose control component versions software versions (different clients) 1/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  5. 5. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Change Requests (CRs) CR describes a defect to be fixed, an adaptive or perfective change, or a new functionality. CRs are stored and managed through CR Repositories. 2/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  6. 6. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions CR Assignment 3/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  7. 7. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Why CR Assignment Matters? Select developers considering the low fixing time yet keeping satisfactory quality Needs good knowledge on the project 4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  8. 8. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Why CR Assignment Matters? Select developers considering the low fixing time yet keeping satisfactory quality Needs good knowledge on the project However, dozens to hundreds CRs daily 4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  9. 9. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Why CR Assignment Matters? Select developers considering the low fixing time yet keeping satisfactory quality Needs good knowledge on the project However, dozens to hundreds CRs daily Labor-intensive and time consuming Susceptible to mistakes 4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  10. 10. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Why CR Assignment Matters? Select developers considering the low fixing time yet keeping satisfactory quality Needs good knowledge on the project However, dozens to hundreds CRs daily Labor-intensive and time consuming Susceptible to mistakes 37%-44% of CRs did not reach the right developer Reassignments (rework!) 4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  11. 11. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Research Objective To propose an automated approach for CR assignment Information Retrieval (IR) models Rule-based expert systems Context-aware information 5/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  12. 12. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Research Methodology 6/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  13. 13. 1 Introduction 2 Literature Review 3 Survey on CR Assignment 4 Proposal 5 Experiment 6 Conclusions
  14. 14. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Systematic Mapping Study The process 1 Research questions 2 Searches in the literature (protocol) 3 Selection of papers, tools, and services 4 Classification (two schemes) 5 Analysis and synthesis of the results 7/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  15. 15. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Research Questions Defined two questions for the mapping study 8/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  16. 16. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Research Questions Defined two questions for the mapping study Question 1 – What are the current challenges and opportunities regarding CR repositories and how do they impact software development? 8/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  17. 17. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Research Questions Defined two questions for the mapping study Question 1 – What are the current challenges and opportunities regarding CR repositories and how do they impact software development? Question 02 – Do the tools and online services for CR management address any of the challenges pointed out as a result of the answers to Question 01? If so, how do they address such challenges? 8/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  18. 18. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Criteria Inclusion: Theory, practice, and approaches CR artifacts written in natural language Unique studies Exclusion: summaries of tutorial or workshop posters keynotes studies with no scientific analysis studies published in unknown sources 9/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  19. 19. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Searches SEARCH ENGINES Automated: Google, ACM, IEEE, Citeseer, Elsevier, Scirus, ScienceDirect, Scopus, ISI, SpringerLink, and Wiley Manual: DBLP KEYWORDS Bug report, change request, modification request, defect track, software issue, bug tracking STUDIES SELECTION 1150 → superficial reading → 321 → full reading → 142 10/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  20. 20. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Tools Selection and Analysis Tools Online Services Bugzilla http://www.bugzilla.org SourceForge http://www.sourceforge.net MantisBT http://www.mantisbt.org Launchpad http://www.launchpad.net Trac http://trac.edgewall.org Code Plex http://www.codeplex.com Redmine http://www.redmine.org Google Code http://code.google.com Jira http://www.atlassian.com GitHub http://www.github.com Do they address any of the challenges? How? 11/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  21. 21. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Classification Schemes Classification Scheme 1: created a taxonomy for Research areas and topics Classification Scheme 2: used a taxonomy for Information Retrieval models 12/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  22. 22. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Classification Scheme 1 Taxonomy for Challenges and Opportunities 13/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  23. 23. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Classification Scheme 2 Information Retrieval (IR) Taxonomy Representation Reasoning Repository Query Document With logic With uncer- tainty With learning CRs(e.g.Bugzilla) CommitLog(e.g.CVS,SVN) SourceCode Keyword-based Pattern-based Structural StreamofCharacters VectorSpace Structural Logic Algebra GraphTheories ProbabilityTheories FuzzySetTheories NeuralNetwork SymbolicLearning SupportVectorMachines DecisionTrees/Table LazyLearning BayesianStatistics GeneticAlgorithms RegressionAnalysis LearntoRank Table: Taxonomy for the classification of the IR models and techniques used in each approach. This is an extension of the taxonomy created by Canfora and Cerulo [1]. 14/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  24. 24. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Concluding Remarks from the Review Automated and Semi-Automated approaches for CR challenges Combinations of software repositories Possibility of mixing up the approaches Lack of contextual information in the approaches I.e.: CR assignment needs workload, developer knowledge, priority, and politics issues Difficulty in assessing the approaches State-of-the-art still far from the state-of-the-practice 15/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  25. 25. 1 Introduction 2 Literature Review 3 Survey on CR Assignment 4 Proposal 5 Experiment 6 Conclusions
  26. 26. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Survey’s Research Questions RQ1. How much time does the CR Assignment activities take? (amount of CRs, individual time, and reassignments) RQ2. What are the strategies used to assign CRs to the appropriate developers? RQ3. What is the complexity involved in assigning CRs to developers? 16/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  27. 27. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Questionnaire 38 questions 8 open-ended 30 closed-ended (most Likert-scaled) 17/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  28. 28. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Questionnaire 38 questions 8 open-ended 30 closed-ended (most Likert-scaled) Three steps validation 17/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  29. 29. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Population Sample Around 400 software developers from Brazilian Federal Organization for Data Processing (SERPRO) From three main sites in the south of Brazil Porto Alegre, Florian´opolis, and Curitiba 18/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  30. 30. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Responses Periodically remainder emails 38 responses out of 400 (9%) Is it enough? Yes! In SERPRO, project leaders and managers are likely to have the desired profile 19/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  31. 31. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Data Analysis I RQ1. How much time does a CR assignment take? It is common to assign almost 20 CRs per day Each CR takes around 5 to 10 minutes to be assigned Reassigning CRs is not so frequent in the SERPRO organization 20 CRs ∗ 10 min = 3.3 hours (per developer/day) Plus reassignments (±10 minutes) For bigger projects and open source it gets worse 20/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  32. 32. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Data Analysis II RQ2. What are the strategies used to assign CRs? 1 Consider workload 2 Severity and criticality 3 Talk to developers before assignment 4 Select developers with more familiarity on the problem 5 Select developers who have solved similar CRs 6 Developers with better knowledge on the project 7 Developers who master the tools 8 Affinity 21/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  33. 33. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Data Analysis III RQ3. What is the complexity involved in assigning CRs? According to the strategies, CR assignments require: Good knowledge on the project(s) The ability of communicating to other people The ability of information seeking in different repositories The capability to retain the knowledge that is acquired during this cognitive process Assign CRs to different teams Assign CRs to different projects 22/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  34. 34. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Survey Replication Application of the same survey design Dataprev Instituto Recˆoncavo de Tecnologia (IRT) Confirmation of initial results 23/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  35. 35. 1 Introduction 2 Literature Review 3 Survey on CR Assignment 4 Proposal 5 Experiment 6 Conclusions
  36. 36. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions The Solution An Automated Approach to Assign Software Change Requests 24/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  37. 37. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Requirements CRs must be assigned according to their severity and criticality workload of developers developers experience interpersonal relationships rely on contextual information (software repositories) 25/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  38. 38. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Strategy to Automated CR Assignment 26/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  39. 39. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Rule-Based Expert System (RBES) rule "Critical CRs, or CRs for module C" when $cr: ChangeRequest (severity == CRITICAL || module =="C") then $cr.assignTo(developer(" johndoe@fakedev .com")) end rule "Change Requests for modules A and B" when $cr: ChangeRequest (module =="A" || module =="B") then $cr.assignTo( availableDeveloper (Workload.WEIGHTED )) end 27/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  40. 40. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Information Retrieval Model With Learning Support Vector Machine (SVM) Training (Black arrows) Recommendation (Gray arrows) 10-fold cross-validation 28/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  41. 41. 1 Introduction 2 Literature Review 3 Survey on CR Assignment 4 Proposal 5 Experiment 6 Conclusions
  42. 42. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Questions Q1: What is the accuracy of the proposed approach for automated CR assignment? Q2: What is the necessary effort to setup the approach in a software development project? Q3: Does the achieved accuracy pay the necessary effort needed in the setup? 29/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  43. 43. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Experiment Design Proposed approach versus pure SVM Proposed approach: SVM, expert system and context 30/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  44. 44. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Experiment Design Proposed approach versus pure SVM Proposed approach: SVM, expert system and context 30/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  45. 45. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Hypotheses Null Hypothesis H0: µ(accuracy with our approach) <= µ(accuracy with SVM) µ(payoff with our approach) <= µ(payoff with SVM) 31/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  46. 46. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Hypotheses Null Hypothesis H0: µ(accuracy with our approach) <= µ(accuracy with SVM) µ(payoff with our approach) <= µ(payoff with SVM) Alternative Hypothesis H1: µ(accuracy with our approach) > µ(accuracy with SVM) µ(payoff with our approach) > µ(payoff with SVM) 31/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  47. 47. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Testing dataset CRs from two modules of Novo SIAFI project (SERPRO) Module A = 781 CRs Module B = 1031 CRs 32/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  48. 48. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Configuration of the Proposed Approach Rules extraction Context information Assignment strategy configuration 33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  49. 49. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Configuration of the Proposed Approach Rules extraction Interviews with 4 workers and analysis of CR samples Total of 14 rules Context information Assignment strategy configuration 33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  50. 50. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Configuration of the Proposed Approach Rules extraction Interviews with 4 workers and analysis of CR samples Total of 14 rules Context information developers vacation developers project allocation developers experience Assignment strategy configuration 33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  51. 51. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Configuration of the Proposed Approach Rules extraction Interviews with 4 workers and analysis of CR samples Total of 14 rules Context information developers vacation developers project allocation developers experience Assignment strategy configuration 1 execute simple rules 2 execute complex rules 3 SVM (instead of manual assignment) 33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  52. 52. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Results I Q1. What is the accuracy of the proposed approach for automated CR assignment? New approach: Module A = 45% and Module B = 34% SVM: Module A = 38% and Module B = 23% An improvement of 18% on Module A and 48% on B Null hypothesis refuted 34/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  53. 53. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Results II Q2. What is the necessary effort to setup the approach in a software development project? 38 hours (rule extraction, context information, strategy) Q3. Does the achieved accuracy pay the necessary effort needed in the setup? 10 minutes for each CR assigned SVM saved 89 hours New approach saved 117 hours Economy of 28 hours vs. 38 hours for setup Null hypothesis not refuted (for this context!) 35/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  54. 54. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Threats to the Validity Generalization of the results (only CRs from one project) Variety of metrics (Precision, Recall, and F-measure) SVM learning process (quality of text data) Difficult to assess the configuration time (trial and error for rules extraction) Implementation of the approach (bug-free?) 36/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  55. 55. 1 Introduction 2 Literature Review 3 Survey on CR Assignment 4 Proposal 5 Experiment 6 Conclusions
  56. 56. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Conclusions I Research Contribution Mapping study on CR repositories investigation Questionnaire-based survey with practitioners An approach for automated CR assignment Validation of the approach Tools Prototype and plugins Test bed for new research 37/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  57. 57. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Conclusions II Academic Contributions Cavalcanti, Y. C., Neto, P. A. D. M. S., Machado, I. D. C., de Almeida, E. S., and de Lemos Meira, S. R. (2013b). Towards Understanding Software Change Request Assignment: A survey with practitioners. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE’2013), pages 195–206 Cavalcanti, Y. C., da Mota Silveira Neto, P. A., do Carmo Machado, I., Vale, T. F., de Almeida, E. S., and de Lemos Meira, S. R. (2013a). Challenges and Opportunities for Software Change Request Repositories: a systematic mapping study. Journal of Software: Evolution and Process. Online first More publications are under work: CBSoft’2014 tool session, ICSME’2014, JSEP journal 38/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  58. 58. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions Conclusions III Future work Investigate new algorithms for workload balancing Investigate methods and techniques for automatic extraction of assignment rules Perform new experimental studies Address other issues of CR management 39/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  59. 59. An Automated Approach to Assign Software Change Requests Ph.D. Thesis Yguarat˜a Cerqueira Cavalcanti Centro de Inform´atica – UFPE March 20, 2014
  60. 60. References Lotka’s Law Few developers fix the most of CRs 1/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  61. 61. References The Problem’s Characteristics There are complex factors which influence CR assignment Factors vary from one organization to another Such as developers’ workload, CRs attributes, interpersonal relationships, and developers know-how Consider different rules for the assignments Thus, automated approaches should be context-aware 2/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  62. 62. References Component Diagram of the Solution 3/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  63. 63. References Prototype Tool Architecture 4/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests
  64. 64. References References I [1] Canfora, G. and Cerulo, L. (2004). A taxonomy of information retrieval models and tools. Computing and Information Technology, 12(3), 175–194. [2] Cavalcanti, Y. C., da Mota Silveira Neto, P. A., do Carmo Machado, I., Vale, T. F., de Almeida, E. S., and de Lemos Meira, S. R. (2013a). Challenges and Opportunities for Software Change Request Repositories: a systematic mapping study. Journal of Software: Evolution and Process. Online first. [3] Cavalcanti, Y. C., Neto, P. A. D. M. S., Machado, I. D. C., de Almeida, E. S., and de Lemos Meira, S. R. (2013b). Towards Understanding Software Change Request Assignment: A survey with practitioners. In Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE’2013), pages 195–206. 5/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

×