Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SURF: Summarizer of User Reviews Feedback

322 views

Published on

Andrea Di Sorbo, Sebastiano Panichella, Carol Alexandru, Corrado A. Visaggio, Gerardo Canfora, Harald Gall: SURF: Summarizer of User Reviews Feedback. Proceedings of the 39th IEEE International Conference on Software Engineering (ICSE 2017). Buenos Aires, Argentina. RANK: A*

  • Be the first to comment

  • Be the first to like this

SURF: Summarizer of User Reviews Feedback

  1. 1. SURF: Summarizer of 
 User Reviews Feedback! Andrea Sebastiano Carol V. Corrado A. Gerardo ! Di Sorbo Panichella Alexandru Visaggio Canfora ! UNIVERSITÀDEGLI STUDIDELSANNIO !
  2. 2. SURF: Summarizer of 
 User Reviews Feedback! Andrea Sebastiano Carol V. Corrado A. Gerardo ! Di Sorbo Panichella Alexandru Visaggio Canfora ! UNIVERSITÀDEGLI STUDIDELSANNIO !
  3. 3. OUTLINE Context: Manual v.s. Automated Analysis of User Reviews Proposed Solution: SURF (Summarizer of User Reviews Feedback) Case Study: Assessment of the Summaries Involving 12 developers Conclusion
  4. 4. Manual v.s. Automated Analysis of User Reviews V.S.!
  5. 5. Maintenance of Mobile Applications “About one third of app reviews! contain useful information for developers”! Pagano et. al. RE2013!
  6. 6. Manual Analysis of Reviews
  7. 7. PAST WORK Chen et al – ICSE 2014! Panichella et al – ICSME 2015!
  8. 8. The Problem Feature Requests Bug Reports
  9. 9. Summaries of User Reviews SURF (Summarizer of User Reviews Feedback)
  10. 10. USER REVIEWS MODEL Di Sorbo et. al. – FSE 2016!
  11. 11. USER REVIEWS MODEL I love this app but it crashes my whole iPad and it has to restart itself • User intention: Problem Discovery! ! • Review topics: App, Model! “…The User Reviews Model proposed by the authors is impressive in how it analyzes a review sentence by sentence and is able to characterize a sentence with multiple labels…” – one of FSE reviewers!
  12. 12. USER REVIEWS MODEL I love this app but it crashes my whole iPad and it has to restart itself • User intention: Problem Discovery! ! • Review topics: App, Model! “…The User Reviews Model proposed by the authors is impressive in how it analyzes a review sentence by sentence and is able to characterize a sentence with multiple labels…” – one of FSE reviewers!
  13. 13. SUMMARIZER OF USER REVIEW FEEDBACK
  14. 14. 1. Data Collection1!
  15. 15. 2. Intention Classification2! machine! learning!
  16. 16. 3. Topics Classification3! Can't change position of icons on main screen and can't close bookmarks icon too.! screen, trajectory, button, white, background, interface, usability, tap, switch, icon, orientation, position, picture, show, list, category, cover, scroll, touch, website, swipe, sensitive, view, roll, side, sort, click, small, colorful, glitch, page, corner, bookmark…! GUI-related dictionary! P (SENTENCE, GUI) = 5/14 = 0.357 !
  17. 17. 3. Topics Classification3! Can't change position of icons on main screen and can't close bookmarks icon too.! screen, trajectory, button, white, background, interface, usability, tap, switch, icon, orientation, position, picture, show, list, category, cover, scroll, touch, website, swipe, sensitive, view, roll, side, sort, click, small, colorful, glitch, page, corner, bookmark…! GUI-related dictionary! P (SENTENCE, GUI) = 5/14 = 0.357 !
  18. 18. 4. Sentence Scoring4! The scoring function is able to simultaneously reward: ! ! (i)  sentences containing feature requests or bug reports with respect to other kinds of feedback; ! (ii)  sentences that are likely to relate specific topics;! (iii)  longer sentences;! (iv)  sentences concerning frequently discussed features.! ! Only sentences in the top positions of the ranked list are selected.!
  19. 19. 4. Sentence Scoring4! The scoring function is able to simultaneously reward: ! ! (i)  sentences containing feature requests or bug reports with respect to other kinds of feedback; ! (ii)  sentences that are likely to relate specific topics;! (iii)  longer sentences;! (iv)  sentences concerning frequently discussed features.! ! Only sentences in the top positions of the ranked list are selected.!
  20. 20. 5. Summary Generation5! DEMO
  21. 21. Case Study 21 Involving 12 Developers
  22. 22. Case Study 22 2622 Reviews Involving 12 Developers
  23. 23. Case Study Involving 12 Developers 2622 Reviews Of 12 Apps 23
  24. 24. Study Procedure 24
  25. 25. Experiment Involving 3 Countries ITALY SWITZERLAND NETHERLAND
  26. 26. 26 ITALY SWITZERLAND NETHERLAND Experiment Involving 3 Countries
  27. 27. ITALY SWITZERLAND NETHERLAND 27 Setting of the Experiment
  28. 28. Setting of the Experiment ITALY SWITZERLAND NETHERLAND 1)  Summaries for 12 Apps 28
  29. 29. ITALY SWITZERLAND NETHERLAND 1)  Summaries for 12 Apps 2)  Involving 12 Developers 29 Setting of the Experiment
  30. 30. ITALY SWITZERLAND NETHERLAND 1)  Summaries for 12 Apps 3) We assigned to each participant an app. 30 Setting of the Experiment 2)  Involving 12 Developers
  31. 31. To what extent SURF help mobile developers better understand the users' needs? RQ31
  32. 32. 32 To what extent SURF help mobile developers better understand the users' needs?
  33. 33. 75% of participants considered the summaries HIGH USEFUL and COMPREHENSIBLE 33 To what extent SURF help mobile developers better understand the users' needs?
  34. 34. 75% of participants considered the summaries HIGH USEFUL and COMPREHENSIBLE 34 To what extent SURF help mobile developers better understand the users' needs? 92% of participants declared that without summaries, evaluating user feedback is tedious and difficult.
  35. 35. How do app review summaries generated by SURF impact the time required by developers to analyze user reviews? 35
  36. 36. The time saving capability of SURF perceived by all developers Is of at least 35%. 66% of participants believe that the time saving capability of SURF is of 75%. 36 How do app review summaries generated by SURF impact the time required by developers to analyze user reviews?
  37. 37. The time saving capability of SURF perceived by all developers Is of at least 50%. 94% of participants believe that the time saving capability of SURF is of 75%. 37 92% of manually extracted feedback appears also in the automatic generated summaries. How do app review summaries generated by SURF impact the time required by developers to analyze user reviews?
  38. 38. How do app review summaries generated! by SURF impact the time required by developers to! analyze user reviews?! The time saving capability of SURF perceived by all developers Is of at least 50%. 94% of participants believe that the time saving capability of SURF is of 75%. SURF helps to prevent more than 50% of the time required by developers for analyzing users feedback and planning software changes. 66% of feedback manually extracted by the participants also appear in the summaries automatically generated by SURF. 38
  39. 39. How do app review summaries generated! by SURF impact the time required by developers to! analyze user reviews?! The time saving capability of SURF perceived by all developers Is of at least 50%. 94% of participants believe that the time saving capability of SURF is of 75%. SURF helps to prevent more than 50% of the time required by developers for analyzing users feedback and planning software changes. 66% of feedback manually extracted by the participants also appear in the summaries automatically generated by SURF. 39
  40. 40. Quality of SURF’ Summaries 40
  41. 41. Quality of SURF’ Summaries 41
  42. 42. Quality of SURF’ Summaries 42
  43. 43. Conclusion 2) SURF helps to prevent more than half of the time required for analyzing users feedback and planning software changes. 3) 92% of manually extracted feedback appears also in the automatic generated summaries. V.S.! 4) Summaries generated by SURF are reasonably correct, adequate, concise, and expressive. 43
  44. 44. Thanks for the Attention!! Questions?! SURF (Summarizer of User Review Feedback)

×