Oğuzhan Çalıkkasap Natural Language Processing Course Presentation

1. Bug Report, Feature Request, or Simply
Praise? On Automatically Classifying App
Reviews
Authors: Walid Maalej and Hader Nabil
IEEE RE - 2015
Presented by:
Oğuzhan Çalıkkasap

2. App Market
• Apple App Store
• Google Play Store
https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/
2

3. Motivation
• Better reviews, higher sales
• Rich source
Information about bugs
Ideas for new features
• Benefitted by
App vendors
Developers
3

4. Purpose
• Classify reviews into four categories
Bug reports
Feature requests
User experiences
Ratings
4

5. Paper Outline
• Introduce probabilistic techniques for review classification
• Compare accuracy of the review classification techniques
• Derive insights into review analytics tool design
5

6. Review Classification Techniques
1. String matching
2. Bag of Words
3. Text Preprocessing
4. Review Metadata
5. Sentiment Analysis
6. Binary and Multiclass Classifiers
6

7. Research Method and Data
• Collect reviews from app store, extract their metadata
• Manually label reviews and create a truth set
• Implement each classifier at different parts of truth set
• Evaluate classifiers’ accuracies
• Data: Review text, title, app name, category, store, submission
date, star rating
7

8. Evaluation Data 8

9. Evaluation Metrics
• Precision: Fraction of reviews that are classified correctly for type i
• Recall: Fraction of reviews of type i which are classified correctly
• F1: Harmonic mean
9
𝑇𝑃𝑖
𝑇𝑃𝑖 + 𝐹𝑃𝑖
𝑇𝑃𝑖
𝑇𝑃𝑖 + 𝐹𝑁𝑖

10. Research Results - Features 10

11. Research Results - Classifiers 11

12. Study Outcomes
• Probabilistic approaches always outperformed the basic classifiers
• Combination of text classifiers, metadata, NLP and sentiments usually
resulted in high precision – above about 70%
• Does not mean combination of techniques always rank best
• No clear trend like ‘more NLP leads to a better result’
• Binary classifiers more accurate than multiclass classifiers
• No single classifier works best for all review types and data sources
• Naive Bayes seems to be a more appropriate classifier for review
classification
12

13. Thank you for listening 13
https://mast.informatik.uni-hamburg.de/wp-content/uploads/2015/06/review_classification_preprint.pdf
Original paper: