Slides of my Ph.D. presentation on March 2014.

1. An Automated Approach to Assign Software
Change Requests
Ph.D. Thesis
Yguarat˜a Cerqueira Cavalcanti
Centro de Inform´atica – UFPE
March 20, 2014

2. Agenda
1 Introduction
2 Literature Review
3 Survey on CR Assignment
4 Proposal
5 Experiment
6 Conclusions

3. 1 Introduction
2 Literature Review
3 Survey on CR Assignment
4 Proposal
5 Experiment
6 Conclusions

4. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Change Management
Every software project changes (1st Lehman’s law)
user needs
defects
new functionalities
Changes are made during software development or after release
(software maintenance and evolution)
Changes need to be managed, instead you lose control
component versions
software versions (different clients)
1/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

5. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Change Requests (CRs)
CR describes a defect to be fixed, an adaptive or
perfective change, or a new functionality.
CRs are stored and managed through CR Repositories.
2/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

6. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
CR Assignment
3/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

7. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Why CR Assignment Matters?
Select developers considering the low fixing time yet keeping
satisfactory quality
Needs good knowledge on the project
4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

8. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Why CR Assignment Matters?
Select developers considering the low fixing time yet keeping
satisfactory quality
Needs good knowledge on the project
However, dozens to hundreds CRs daily
4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

9. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Why CR Assignment Matters?
Select developers considering the low fixing time yet keeping
satisfactory quality
Needs good knowledge on the project
However, dozens to hundreds CRs daily
Labor-intensive and time consuming
Susceptible to mistakes
4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

10. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Why CR Assignment Matters?
Select developers considering the low fixing time yet keeping
satisfactory quality
Needs good knowledge on the project
However, dozens to hundreds CRs daily
Labor-intensive and time consuming
Susceptible to mistakes
37%-44% of CRs did not reach the right developer
Reassignments (rework!)
4/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

11. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Research Objective
To propose an automated approach for CR assignment
Information Retrieval (IR) models
Rule-based expert systems
Context-aware information
5/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

12. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Research Methodology
6/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

13. 1 Introduction
2 Literature Review
3 Survey on CR Assignment
4 Proposal
5 Experiment
6 Conclusions

14. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Systematic Mapping Study
The process
1 Research questions
2 Searches in the literature (protocol)
3 Selection of papers, tools, and services
4 Classification (two schemes)
5 Analysis and synthesis of the results
7/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

15. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Research Questions
Defined two questions for the mapping study
8/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

16. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Research Questions
Defined two questions for the mapping study
Question 1 – What are the current challenges and
opportunities regarding CR repositories and how do they
impact software development?
8/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

17. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Research Questions
Defined two questions for the mapping study
Question 1 – What are the current challenges and
opportunities regarding CR repositories and how do they
impact software development?
Question 02 – Do the tools and online services for CR
management address any of the challenges pointed out as
a result of the answers to Question 01? If so, how do they
address such challenges?
8/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

18. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Criteria
Inclusion:
Theory, practice, and approaches
CR artifacts written in natural language
Unique studies
Exclusion:
summaries of tutorial or workshop
posters
keynotes
studies with no scientific analysis
studies published in unknown sources
9/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

19. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Searches
SEARCH ENGINES
Automated: Google, ACM, IEEE, Citeseer, Elsevier, Scirus,
ScienceDirect, Scopus, ISI, SpringerLink, and Wiley
Manual: DBLP
KEYWORDS
Bug report, change request, modification request, defect track,
software issue, bug tracking
STUDIES SELECTION
1150 → superficial reading → 321 → full reading → 142
10/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

20. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Tools Selection and Analysis
Tools Online Services
Bugzilla http://www.bugzilla.org SourceForge http://www.sourceforge.net
MantisBT http://www.mantisbt.org Launchpad http://www.launchpad.net
Trac http://trac.edgewall.org Code Plex http://www.codeplex.com
Redmine http://www.redmine.org Google Code http://code.google.com
Jira http://www.atlassian.com GitHub http://www.github.com
Do they address any of the challenges?
How?
11/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

21. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Classification Schemes
Classification Scheme 1: created a taxonomy for Research
areas and topics
Classification Scheme 2: used a taxonomy for Information
Retrieval models
12/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

22. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Classification Scheme 1
Taxonomy for Challenges and Opportunities
13/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

23. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Classification Scheme 2
Information Retrieval (IR) Taxonomy
Representation Reasoning Repository
Query Document
With logic
With
uncer-
tainty
With learning
CRs(e.g.Bugzilla)
CommitLog(e.g.CVS,SVN)
SourceCode
Keyword-based
Pattern-based
Structural
StreamofCharacters
VectorSpace
Structural
Logic
Algebra
GraphTheories
ProbabilityTheories
FuzzySetTheories
NeuralNetwork
SymbolicLearning
SupportVectorMachines
DecisionTrees/Table
LazyLearning
BayesianStatistics
GeneticAlgorithms
RegressionAnalysis
LearntoRank
Table: Taxonomy for the classification of the IR models and techniques
used in each approach. This is an extension of the taxonomy created by
Canfora and Cerulo [1].
14/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

24. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Concluding Remarks from the Review
Automated and Semi-Automated approaches for CR
challenges
Combinations of software repositories
Possibility of mixing up the approaches
Lack of contextual information in the approaches
I.e.: CR assignment needs workload, developer knowledge,
priority, and politics issues
Difficulty in assessing the approaches
State-of-the-art still far from the state-of-the-practice
15/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

25. 1 Introduction
2 Literature Review
3 Survey on CR Assignment
4 Proposal
5 Experiment
6 Conclusions

26. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Survey’s Research Questions
RQ1. How much time does the CR Assignment activities
take? (amount of CRs, individual time, and reassignments)
RQ2. What are the strategies used to assign CRs to the
appropriate developers?
RQ3. What is the complexity involved in assigning CRs to
developers?
16/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

27. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Questionnaire
38 questions
8 open-ended
30 closed-ended (most
Likert-scaled)
17/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

28. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Questionnaire
38 questions
8 open-ended
30 closed-ended (most
Likert-scaled)
Three steps validation
17/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

29. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Population Sample
Around 400 software developers from Brazilian Federal
Organization for Data Processing (SERPRO)
From three main sites in the south of Brazil
Porto Alegre, Florian´opolis, and Curitiba
18/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

30. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Responses
Periodically remainder emails
38 responses out of 400 (9%)
Is it enough? Yes!
In SERPRO, project leaders and managers are likely to have
the desired profile
19/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

31. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Data Analysis I
RQ1. How much time does a CR assignment take?
It is common to assign almost 20 CRs per day
Each CR takes around 5 to 10 minutes to be assigned
Reassigning CRs is not so frequent in the SERPRO
organization
20 CRs ∗ 10 min = 3.3 hours (per developer/day)
Plus reassignments (±10 minutes)
For bigger projects and open source it gets worse
20/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

32. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Data Analysis II
RQ2. What are the strategies used to assign CRs?
1 Consider workload
2 Severity and criticality
3 Talk to developers before assignment
4 Select developers with more familiarity on the problem
5 Select developers who have solved similar CRs
6 Developers with better knowledge on the project
7 Developers who master the tools
8 Affinity
21/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

33. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Data Analysis III
RQ3. What is the complexity involved in assigning CRs?
According to the strategies, CR assignments require:
Good knowledge on the project(s)
The ability of communicating to other people
The ability of information seeking in different repositories
The capability to retain the knowledge that is acquired during
this cognitive process
Assign CRs to different teams
Assign CRs to different projects
22/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

34. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Survey Replication
Application of the same survey design
Dataprev
Instituto Recˆoncavo de Tecnologia (IRT)
Confirmation of initial results
23/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

35. 1 Introduction
2 Literature Review
3 Survey on CR Assignment
4 Proposal
5 Experiment
6 Conclusions

36. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
The Solution
An Automated Approach to Assign
Software Change Requests
24/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

37. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Requirements
CRs must be assigned according to their
severity and criticality
workload of developers
developers experience
interpersonal relationships
rely on contextual information (software repositories)
25/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

38. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Strategy to Automated CR Assignment
26/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

39. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Rule-Based Expert System (RBES)
rule "Critical CRs, or CRs for module C"
when
$cr: ChangeRequest (severity == CRITICAL || module =="C")
then
$cr.assignTo(developer(" johndoe@fakedev .com"))
end
rule "Change Requests for modules A and B"
when
$cr: ChangeRequest (module =="A" || module =="B")
then
$cr.assignTo( availableDeveloper (Workload.WEIGHTED ))
end
27/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

40. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Information Retrieval Model With Learning
Support Vector Machine (SVM)
Training (Black arrows)
Recommendation (Gray arrows)
10-fold cross-validation
28/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

41. 1 Introduction
2 Literature Review
3 Survey on CR Assignment
4 Proposal
5 Experiment
6 Conclusions

42. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Questions
Q1: What is the accuracy of the proposed approach for
automated CR assignment?
Q2: What is the necessary effort to setup the approach in a
software development project?
Q3: Does the achieved accuracy pay the necessary effort
needed in the setup?
29/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

43. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Experiment Design
Proposed approach versus pure SVM
Proposed approach: SVM, expert system and context
30/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

44. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Experiment Design
Proposed approach versus pure SVM
Proposed approach: SVM, expert system and context
30/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

45. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Hypotheses
Null Hypothesis
H0: µ(accuracy with our approach) <= µ(accuracy with SVM)
µ(payoff with our approach) <= µ(payoff with SVM)
31/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

46. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Hypotheses
Null Hypothesis
H0: µ(accuracy with our approach) <= µ(accuracy with SVM)
µ(payoff with our approach) <= µ(payoff with SVM)
Alternative Hypothesis
H1: µ(accuracy with our approach) > µ(accuracy with SVM)
µ(payoff with our approach) > µ(payoff with SVM)
31/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

47. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Testing dataset
CRs from two modules of Novo SIAFI project (SERPRO)
Module A = 781 CRs
Module B = 1031 CRs
32/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

48. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Configuration of the Proposed Approach
Rules extraction
Context information
Assignment strategy configuration
33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

49. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Configuration of the Proposed Approach
Rules extraction
Interviews with 4 workers and analysis of CR samples
Total of 14 rules
Context information
Assignment strategy configuration
33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

50. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Configuration of the Proposed Approach
Rules extraction
Interviews with 4 workers and analysis of CR samples
Total of 14 rules
Context information
developers vacation
developers project allocation
developers experience
Assignment strategy configuration
33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

51. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Configuration of the Proposed Approach
Rules extraction
Interviews with 4 workers and analysis of CR samples
Total of 14 rules
Context information
developers vacation
developers project allocation
developers experience
Assignment strategy configuration
1 execute simple rules
2 execute complex rules
3 SVM (instead of manual assignment)
33/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

52. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Results I
Q1. What is the accuracy of the proposed approach for
automated CR assignment?
New approach: Module A = 45% and Module B = 34%
SVM: Module A = 38% and Module B = 23%
An improvement of 18% on Module A and 48% on B
Null hypothesis refuted
34/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

53. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Results II
Q2. What is the necessary effort to setup the approach in a
software development project?
38 hours (rule extraction, context information, strategy)
Q3. Does the achieved accuracy pay the necessary effort
needed in the setup?
10 minutes for each CR assigned
SVM saved 89 hours
New approach saved 117 hours
Economy of 28 hours vs. 38 hours for setup
Null hypothesis not refuted (for this context!)
35/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

54. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Threats to the Validity
Generalization of the results (only CRs from one project)
Variety of metrics (Precision, Recall, and F-measure)
SVM learning process (quality of text data)
Difficult to assess the configuration time (trial and error for
rules extraction)
Implementation of the approach (bug-free?)
36/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

55. 1 Introduction
2 Literature Review
3 Survey on CR Assignment
4 Proposal
5 Experiment
6 Conclusions

56. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Conclusions I
Research Contribution
Mapping study on CR repositories investigation
Questionnaire-based survey with practitioners
An approach for automated CR assignment
Validation of the approach
Tools
Prototype and plugins
Test bed for new research
37/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

57. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Conclusions II
Academic Contributions
Cavalcanti, Y. C., Neto, P. A. D. M. S., Machado, I. D. C., de Almeida, E. S.,
and de Lemos Meira, S. R. (2013b). Towards Understanding Software Change
Request Assignment: A survey with practitioners.
In Proceedings of the 17th International Conference on Evaluation and
Assessment in Software Engineering (EASE’2013), pages 195–206
Cavalcanti, Y. C., da Mota Silveira Neto, P. A., do Carmo Machado, I., Vale,
T. F., de Almeida, E. S., and de Lemos Meira, S. R. (2013a). Challenges and
Opportunities for Software Change Request Repositories: a systematic mapping
study.
Journal of Software: Evolution and Process.
Online first
More publications are under work: CBSoft’2014 tool session,
ICSME’2014, JSEP journal
38/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

58. Outline Introduction Literature Review Survey on CR Assignment Proposal Experiment Conclusions
Conclusions III
Future work
Investigate new algorithms for workload balancing
Investigate methods and techniques for automatic extraction of
assignment rules
Perform new experimental studies
Address other issues of CR management
39/39 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

59. An Automated Approach to Assign Software
Change Requests
Ph.D. Thesis
Yguarat˜a Cerqueira Cavalcanti
Centro de Inform´atica – UFPE
March 20, 2014

60. References
Lotka’s Law
Few developers fix the most of CRs
1/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

61. References
The Problem’s Characteristics
There are complex factors which influence CR assignment
Factors vary from one organization to another
Such as developers’ workload, CRs attributes, interpersonal
relationships, and developers know-how
Consider different rules for the assignments
Thus, automated approaches should be context-aware
2/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

62. References
Component Diagram of the Solution
3/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

63. References
Prototype Tool Architecture
4/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests

64. References
References I
[1] Canfora, G. and Cerulo, L. (2004). A taxonomy of information
retrieval models and tools. Computing and Information
Technology, 12(3), 175–194.
[2] Cavalcanti, Y. C., da Mota Silveira Neto, P. A.,
do Carmo Machado, I., Vale, T. F., de Almeida, E. S., and
de Lemos Meira, S. R. (2013a). Challenges and Opportunities for
Software Change Request Repositories: a systematic mapping
study. Journal of Software: Evolution and Process. Online first.
[3] Cavalcanti, Y. C., Neto, P. A. D. M. S., Machado, I. D. C.,
de Almeida, E. S., and de Lemos Meira, S. R. (2013b). Towards
Understanding Software Change Request Assignment: A survey
with practitioners. In Proceedings of the 17th International
Conference on Evaluation and Assessment in Software
Engineering (EASE’2013), pages 195–206.
5/5 Yguarat˜a Cerqueira Cavalcanti An Automated Approach to Assign Software Change Requests