Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]

370 views

Published on

Slides of the talk at the 6th International Workshop on Advances in Semantic Information Retrieval, ASIR 2016, Gdansk, Poland, 11-14 Sept. 2016

Published in: Data & Analytics
  • Be the first to comment

Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]

  1. 1. Predicting Star Ratings based on Annotated Reviews of Mobile Apps Talk at the 6th International Workshop on Advances in Semantic Information Retrieval ASIR 2016 Prof. Dr. Dagmar Monett, Hermann Stolte
  2. 2. D. Monett Reviews and star ratings 2Gdańsk, Poland, September 11 – 14, 2016 Example of reviews and star ratings of the Evernote App, Google Play Store (07/2016)
  3. 3. D. Monett Star ratings matter 3Gdańsk, Poland, September 11 – 14, 2016 15% would consider downloading an app with a 2-star rating 50% would consider downloading an app with a 3-star rating 96% would consider downloading an app with a 4-star rating Source: Aptentive 2015 Consumer Study The Mobile Marketer‘s Guide to App Store Ratings & Reviews
  4. 4. D. Monett Star ratings matter 4Gdańsk, Poland, September 11 – 14, 2016 © and source: Aptentive 2015 Consumer Study The Mobile Marketer‘s Guide to App Store Ratings & Reviews
  5. 5. D. Monett 5Gdańsk, Poland, September 11 – 14, 2016 Our motivation
  6. 6. D. Monett Some questions… 6Gdańsk, Poland, September 11 – 14, 2016 ■ Could we (a program) teach users how to rate apps consistently with the review they are writing for a mobile app? ■ I.e., could we (a program) suggest to users the most adequate star rating they should give to a product depending on the semantic orientation of what they have already written in the review? ■ Would it mean an improvement of users' engagement and satisfaction with the app?
  7. 7. D. Monett 7Gdańsk, Poland, September 11 – 14, 2016 Background
  8. 8. D. Monett 8Gdańsk, Poland, September 11 – 14, 2016 Review rating prediction ■ Also sentiment rating prediction: ■ …a task that deals with the inference of an author's implied numerical rating, i.e. on the prediction of a rating score, from a given written review ■ E.g., recommendation systems often suggest products based on star ratings of similar products previously rated by other users
  9. 9. D. Monett 9Gdańsk, Poland, September 11 – 14, 2016 Suggested readings
  10. 10. D. Monett 10Gdańsk, Poland, September 11 – 14, 2016 Other related work ■ Analysing textual reviews and inferring sentiment polarity –positive/negative/neutral– (Pang et al. 2002; Liu, 2010) ■ Using not only textual semantics but also other information, e.g., about the author and/or the product (Tang et al., 2015; Li et al. 2011) ■ Considering phrase-level sentiment polarity (Qu et al., 2010) ■ Considering aspect-based opinion mining (Zhang et al., 2006; Ganu et al., 2013; Klinger & Cimiano, 2013; Sänger, 2015)
  11. 11. D. Monett 11Gdańsk, Poland, September 11 – 14, 2016 Our approach
  12. 12. D. Monett 12Gdańsk, Poland, September 11 – 14, 2016 Our approach ■ We do not deal with aspect identification nor with sentiment classification ■ We are assuming that these tasks are already performed before the star ratings are predicted ■ We focus on predicting star ratings based solely on available annotated, fine-granular opinions ■ I.e., a complement to works like (Sänger, 2015) which extends (Klinger & Cimiano, 2013) and use a German annotated corpus of mobile apps
  13. 13. D. Monett 13Gdańsk, Poland, September 11 – 14, 2016 The Data
  14. 14. D. Monett 14Gdańsk, Poland, September 11 – 14, 2016 SCARE Corpus Mario Sänger, Ulf Leser, Steffen Kemmerer, Peter Adolphs, and Roman Klinger. SCARE - The Sentiment Corpus of App Reviews with Fine-grained Annotations in German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). ■ Fine-grained annotations for mobile application reviews from the Google Play Store ■ 1,760 German application reviews with 2,487 aspects and 3,959 subjective phrases ■ SCARE corpus v.1.0.0 (annotations only) ■ Available at http://www.romanklinger.de/scare/
  15. 15. D. Monett 15Gdańsk, Poland, September 11 – 14, 2016 Analysing the Data
  16. 16. D. Monett 16Gdańsk, Poland, September 11 – 14, 2016 Polarity and star ratings 69.1% 23.1% Thumbs-up-thumbs-down (Liu, 2012)
  17. 17. D. Monett Avg. of labelled star ratings vs. avg. of subjective phrases polarity 17Gdańsk, Poland, September 11 – 14, 2016
  18. 18. D. Monett Number of star ratings vs. number of subjective phrases 18Gdańsk, Poland, September 11 – 14, 2016
  19. 19. D. Monett 19Gdańsk, Poland, September 11 – 14, 2016 Predicting Star Ratings
  20. 20. D. Monett Prediction process 20Gdańsk, Poland, September 11 – 14, 2016
  21. 21. D. Monett 21Gdańsk, Poland, September 11 – 14, 2016 We “played” with different models
  22. 22. D. Monett Computational models 22Gdańsk, Poland, September 11 – 14, 2016 For example, x0=1 x1 : no. of subjective phrases with positive polarity x2 : no. of subjective phrases with negative polarity x3 : no. of subjective phrases with neutral polarity
  23. 23. D. Monett Computational models 23Gdańsk, Poland, September 11 – 14, 2016 RSS: review rating score (Ganu et al., 2009, 2013)
  24. 24. D. Monett Experiments 24Gdańsk, Poland, September 11 – 14, 2016 (1) Assessing the importance of sentiment in the reviews: ■ Neutral phrases (yes/no)? ■ Reviews with no sentiment (yes/no)? (2) Using other predictors ■ Each individual experiment is run 10,000 times ■ A Monte Carlo cross-validation: 70% training dataset and 30% testing dataset, randomly on each iteration.
  25. 25. D. Monett 25Gdańsk, Poland, September 11 – 14, 2016 Some results
  26. 26. D. Monett “Best” model, exp. (1) 26Gdańsk, Poland, September 11 – 14, 2016 ■ It considers only the average value of the polarities of a review in one feature: ■ Plus: ■ filtering both subjective phrases with neutral polarity and reviews with no sentiment orientation at all ■ No normalisation
  27. 27. D. Monett Results 27Gdańsk, Poland, September 11 – 14, 2016
  28. 28. D. Monett 28Gdańsk, Poland, September 11 – 14, 2016 Conclusion
  29. 29. D. Monett Conclusion 29Gdańsk, Poland, September 11 – 14, 2016 ■ Textually-derived rating prediction can be performed well even when only phrase-level sentiment polarity is available ■ Phrases with neutral sentiment could be filtered out of the corpus ■ Computing the overall sentiment of a review using the review rating score (Ganu et al., 2009, 2013) provides the best star rating predictions
  30. 30. D. Monett Further work 30Gdańsk, Poland, September 11 – 14, 2016 ■ To consider the aspects’ relevance ■ aspect-oriented subjective phrases ■ To analyse the strengths of the opinions (Wilson et al., 2004) ■ not only positive/negative/neutral sentiment ■ To deal with other types of models different than linear, multivariate regression ones
  31. 31. D. Monett Sources 31Gdańsk, Poland, September 11 – 14, 2016 Related work: - See references list on our paper! ■ https://www.researchgate.net/publication/304244445_Predi cting_Star_Ratings_based_on_Annotated_Reviews_of_Mo bile_Apps
  32. 32. dagmar@monettdiaz.com monettdiaz Contact:

×