Researchers have proposed several approaches to extract information from user reviews useful for maintaining and evolving mobile apps. However, most of them just perform automatic classification of user reviews according to specific keywords (e.g., bugs, features). Moreover, they do not provide any support for linking user feedback to the source code components to be changed, thus requiring a manual, time-consuming, and error-prone task. In this paper, we introduce ChangeAdvisor, a novel approach that analyzes the structure, semantics, and sentiments of sentences contained in user reviews to extract useful (user) feedback from maintenance perspectives and recommend to developers changes to software artifacts. It relies on natural language processing and clustering algorithms to group user reviews around similar user needs and suggestions for change. Then, it involves textual based heuristics to determine the code artifacts that need to be maintained according to the recommended software changes. The quantitative and qualitative studies carried out on 44 683 user reviews of 10 open source mobile apps and their original developers showed a high accuracy of ChangeAdvisor in (i) clustering similar user change requests and (ii) identifying the code components impacted by the suggested changes. Moreover, the obtained results show that ChangeAdvisor is more accurate than a baseline approach for linking user feedback clusters to the source code in terms of both precision (+47%) and recall (+38%).

1. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
Delft University of Technology
ICSE 2017
International Conference on
Software Engineering UNIVERSITÀ DEGLI STUDI
DI SALERNO
Recommending and Localizing
Change Requests
for Mobile Apps based on User Reviews
Fabio Palomba1, Pasquale Salza2, Adelina Ciurumelea3, Sebastiano Panichella3
Harald Gall3, Filomena Ferrucci2, Andrea De Lucia2
1Delft University of Technology, 2University of Salerno, 3University of Zurich

2. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Apps everywhere
Over 2 billions people rely on 5
millions of mobile apps for
social and emergency
connectivity

3. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Apps are software too
Short period releases
User reviews on the stores

4. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
User reviews are useful…
Other than non informative content, they contain
important pieces of information

5. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Reading all of them is an effort-prone task
…but they are too many!

6. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
The State-of-the-Art
But, it is not possible to:
1. Extract only the useful information hidden behind
different user reviews
2. Group together fine-grained information
3. Understand the actual impact of each change
request
SURF CLAP
Summarizes user reviews Prioritizes user reviews

7. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
ChangeAdvisor
Transforms user reviews in change requests
and localises them within the source code

8. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
ChangeAdvisor
JAVA
JAVA
JAVA
JAVA
Parser
ARdoc
HDP-LDADice Indexer
Source code
Feedback clusters
Code components
Filter
Feedback
preprocessing
Source code
preprocessing
Problem discovery and
Feature requests
Classified feedback
Ranked list
(cluster/component)
Reviews
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017

9. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
JAVA
JAVA
JAVA
JAVA
Parser
ARdoc
Source code Code components
Filter
Source code
preprocessing
Classified feedbackReviews
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Extraction of a bag of words for each class

10. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
JAVA
JAVA
JAVA
JAVA
Parser
ARdoc
Dice Indexer
Source code Code components
S
pr
Classified feedbackReviews
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Extraction and classification of user feedback

11. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
HDice Indexer
Code components
Filter
F
pre
Source code
preprocessing
Problem discovery and
Feature requests
Classified feedback
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Filtering of change requests

12. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
HDP-LDAexer
Feedback clusters
Filter
Feedback
preprocessing
Source code
preprocessing
Problem discovery and
Feature requests
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Grouping similar
user needs

13. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
ARdoc
Dice Indexer
Feedback clusters
Filter
Problem discovery and
Feature requests
Classified feedback
Ranked list
(cluster/component)
iews
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Code components
Linking feedback to components

14. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Empirical Study
10 open source apps
13,510 feedback
4,138 classes

15. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
RQ1
Does ChangeAdvisor identify cohesive
user feedback clusters representing
related change requests?

16. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Clusters evaluation
We asked the experts to rate the
cohesiveness of clusters using a
Likert scale
5+ years experience
4
3

17. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Results
Overall median = 4
Overall max = 5
8.2 clusters/app

18. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
RQ2
Does ChangeAdvisor correctly
link user feedback clusters to
source code components and
how does it compare with the
state-of-the-art?

19. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Oracle definition
We asked the experts to define the
link between clusters and classes

20. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Links evaluation
We measured the
accuracy of
ChangeAdvisor using
precision and recall

21. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Comparison
We exploited BLUiR
that links bug reports
to classes using the
Vector Space Model
BLUiR

22. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Results
ChangeAdvisor
Precision = 81 %
Recall = 70 %
BLUiR
Precision = 34 %
Recall = 32 %
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017

23. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Overlap analysis
ChangeAdvisor BLUiR
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Number of links correctly identified by one approach and
missed by the other

24. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Overlap analysis
ChangeAdvisor
BLUiR
72 % 5 %23 %
May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017

25. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Are the suggestions
provided by ChangeAdvisor
actually useful for
developers?
RQ3

26. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Surveying developers
Cohesiveness. How well are the user
reviews grouped according to the
number of source code components
that need to be modified?
Precision. How well do the proposed
classes match the actual set of those
needed to be changed in order to satisfy
the user requests?
Completeness. Evaluate the
completeness of the set of classes
suggested compared to the actual set
of those needed to be modified

27. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Results
Cohesiveness = high Precision = very high Completeness = very high

28. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Results
I know what changes I
have to make in my app
when implementing a
change. However, a tool
like this may help in
quantifying the number of
classes to be modified.

29. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Future directions
• Extend the study involving
more apps and developers
• Improve our approach to
prioritize user feedback
based on the number of
classes to be modified

30. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
One more thing…
http://bit.ly/2r5GzMq
ChangeAdvisor

31. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
NLP steps
1. Spelling correction
2. Contractions expansion
3. Nouns and verbs filtering
4. Tokenization
5. Singularization
6. Stopword removal
7. Stemming
8. Repetitions removal
9. Short tokens removal
10.Short tokens removal
11.Short documents
removal

32. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Dice similarity

33. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Characteristics of the apps in the dataset

34. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Evaluation of the clusters cohesiveness

35. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
ChangeAdvisor vs BLUiR

36. May 25th, 2017Recommending and Localizing Change Requests for Mobile Apps based on User Reviews
ICSE 2017
Surveys results