SlideShare a Scribd company logo
Predicting Star Ratings
based on Annotated Reviews
of Mobile Apps
Talk at the 6th International Workshop on Advances in Semantic Information Retrieval
ASIR 2016
Prof. Dr. Dagmar Monett, Hermann Stolte
D. Monett
Reviews and star ratings
2Gdańsk, Poland, September 11 – 14, 2016
Example of reviews and star ratings of the
Evernote App, Google Play Store (07/2016)
D. Monett
Star ratings matter
3Gdańsk, Poland, September 11 – 14, 2016
15% would consider downloading an app with a 2-star rating
50% would consider downloading an app with a 3-star rating
96% would consider downloading an app with a 4-star rating
Source: Aptentive 2015 Consumer Study
The Mobile Marketer‘s Guide to App Store Ratings & Reviews
D. Monett
Star ratings matter
4Gdańsk, Poland, September 11 – 14, 2016
© and source: Aptentive 2015 Consumer Study
The Mobile Marketer‘s Guide to App Store Ratings & Reviews
D. Monett 5Gdańsk, Poland, September 11 – 14, 2016
Our motivation
D. Monett
Some questions…
6Gdańsk, Poland, September 11 – 14, 2016
■ Could we (a program) teach users how to rate
apps consistently with the review they are writing
for a mobile app?
■ I.e., could we (a program) suggest to users the
most adequate star rating they should give to a
product depending on the semantic orientation of
what they have already written in the review?
■ Would it mean an improvement of users'
engagement and satisfaction with the app?
D. Monett 7Gdańsk, Poland, September 11 – 14, 2016
Background
D. Monett 8Gdańsk, Poland, September 11 – 14, 2016
Review rating prediction
■ Also sentiment rating prediction:
■ …a task that deals with the inference of an
author's implied numerical rating, i.e. on the
prediction of a rating score, from a given written
review
■ E.g., recommendation systems often suggest
products based on star ratings of similar
products previously rated by other users
D. Monett 9Gdańsk, Poland, September 11 – 14, 2016
Suggested readings
D. Monett 10Gdańsk, Poland, September 11 – 14, 2016
Other related work
■ Analysing textual reviews and inferring sentiment
polarity –positive/negative/neutral– (Pang et al. 2002;
Liu, 2010)
■ Using not only textual semantics but also other
information, e.g., about the author and/or the
product (Tang et al., 2015; Li et al. 2011)
■ Considering phrase-level sentiment polarity (Qu et
al., 2010)
■ Considering aspect-based opinion mining (Zhang et
al., 2006; Ganu et al., 2013; Klinger & Cimiano, 2013; Sänger, 2015)
D. Monett 11Gdańsk, Poland, September 11 – 14, 2016
Our approach
D. Monett 12Gdańsk, Poland, September 11 – 14, 2016
Our approach
■ We do not deal with aspect identification nor with
sentiment classification
■ We are assuming that these tasks are already
performed before the star ratings are predicted
■ We focus on predicting star ratings based solely
on available annotated, fine-granular opinions
■ I.e., a complement to works like (Sänger, 2015) which
extends (Klinger & Cimiano, 2013) and use a German
annotated corpus of mobile apps
D. Monett 13Gdańsk, Poland, September 11 – 14, 2016
The Data
D. Monett 14Gdańsk, Poland, September 11 – 14, 2016
SCARE Corpus
Mario Sänger, Ulf Leser, Steffen Kemmerer, Peter Adolphs, and Roman Klinger.
SCARE - The Sentiment Corpus of App Reviews with Fine-grained Annotations in
German. In Proceedings of the Tenth International Conference on Language
Resources and Evaluation (LREC'16), Portorož, Slovenia, May 2016. European
Language Resources Association (ELRA).
■ Fine-grained annotations for mobile application
reviews from the Google Play Store
■ 1,760 German application reviews with 2,487
aspects and 3,959 subjective phrases
■ SCARE corpus v.1.0.0 (annotations only)
■ Available at http://www.romanklinger.de/scare/
D. Monett 15Gdańsk, Poland, September 11 – 14, 2016
Analysing the Data
D. Monett 16Gdańsk, Poland, September 11 – 14, 2016
Polarity and star ratings
69.1%
23.1%
Thumbs-up-thumbs-down
(Liu, 2012)
D. Monett
Avg. of labelled star ratings vs.
avg. of subjective phrases polarity
17Gdańsk, Poland, September 11 – 14, 2016
D. Monett
Number of star ratings vs.
number of subjective phrases
18Gdańsk, Poland, September 11 – 14, 2016
D. Monett 19Gdańsk, Poland, September 11 – 14, 2016
Predicting
Star Ratings
D. Monett
Prediction process
20Gdańsk, Poland, September 11 – 14, 2016
D. Monett 21Gdańsk, Poland, September 11 – 14, 2016
We “played” with
different models
D. Monett
Computational models
22Gdańsk, Poland, September 11 – 14, 2016
For example,
x0=1
x1 : no. of subjective phrases with positive polarity
x2 : no. of subjective phrases with negative polarity
x3 : no. of subjective phrases with neutral polarity
D. Monett
Computational models
23Gdańsk, Poland, September 11 – 14, 2016
RSS: review rating score (Ganu et al., 2009, 2013)
D. Monett
Experiments
24Gdańsk, Poland, September 11 – 14, 2016
(1) Assessing the importance of sentiment in the
reviews:
■ Neutral phrases (yes/no)?
■ Reviews with no sentiment (yes/no)?
(2) Using other predictors
■ Each individual experiment is run 10,000 times
■ A Monte Carlo cross-validation: 70% training
dataset and 30% testing dataset, randomly on each
iteration.
D. Monett 25Gdańsk, Poland, September 11 – 14, 2016
Some results
D. Monett
“Best” model, exp. (1)
26Gdańsk, Poland, September 11 – 14, 2016
■ It considers only the average value of the
polarities of a review in one feature:
■ Plus:
■ filtering both subjective phrases with neutral
polarity and reviews with no sentiment
orientation at all
■ No normalisation
D. Monett
Results
27Gdańsk, Poland, September 11 – 14, 2016
D. Monett 28Gdańsk, Poland, September 11 – 14, 2016
Conclusion
D. Monett
Conclusion
29Gdańsk, Poland, September 11 – 14, 2016
■ Textually-derived rating prediction can be
performed well even when only phrase-level
sentiment polarity is available
■ Phrases with neutral sentiment could be filtered
out of the corpus
■ Computing the overall sentiment of a review using
the review rating score (Ganu et al., 2009, 2013) provides
the best star rating predictions
D. Monett
Further work
30Gdańsk, Poland, September 11 – 14, 2016
■ To consider the aspects’ relevance
■ aspect-oriented subjective phrases
■ To analyse the strengths of the opinions (Wilson et al.,
2004)
■ not only positive/negative/neutral sentiment
■ To deal with other types of models different than
linear, multivariate regression ones
D. Monett
Sources
31Gdańsk, Poland, September 11 – 14, 2016
Related work:
- See references list on our paper!
■ https://www.researchgate.net/publication/304244445_Predi
cting_Star_Ratings_based_on_Annotated_Reviews_of_Mo
bile_Apps
dagmar@monettdiaz.com
monettdiaz
Contact:

More Related Content

Viewers also liked

Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:WireTeaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Dagmar Monett
 
Index Determination in DAEs using the Library indexdet and the ADOL-C Package...
Index Determination in DAEs using the Library indexdet and the ADOL-C Package...Index Determination in DAEs using the Library indexdet and the ADOL-C Package...
Index Determination in DAEs using the Library indexdet and the ADOL-C Package...
Dagmar Monett
 
Software Requirements Elicitation Methods
Software Requirements Elicitation MethodsSoftware Requirements Elicitation Methods
Software Requirements Elicitation Methods
mnaeem22
 
Agile Project-Based Teaching and Learning
Agile Project-Based Teaching and LearningAgile Project-Based Teaching and Learning
Agile Project-Based Teaching and Learning
Dagmar Monett
 
Agent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertation
Agent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertationAgent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertation
Agent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertation
Dagmar Monett
 
Experiences in Software Testing (lecture slides)
Experiences in Software Testing (lecture slides)Experiences in Software Testing (lecture slides)
Experiences in Software Testing (lecture slides)
Dagmar Monett
 
Key Issues for Requirements Engineering (lecture slides)
Key Issues for Requirements Engineering (lecture slides)Key Issues for Requirements Engineering (lecture slides)
Key Issues for Requirements Engineering (lecture slides)
Dagmar Monett
 
E-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical StudyE-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical Study
Dagmar Monett
 
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Dagmar Monett
 
Software requirement elicitation
Software requirement elicitationSoftware requirement elicitation
Software requirement elicitation
Pankamol Srikaew
 
Introduction to Agents and Multi-agent Systems (lecture slides)
Introduction to Agents and Multi-agent Systems (lecture slides)Introduction to Agents and Multi-agent Systems (lecture slides)
Introduction to Agents and Multi-agent Systems (lecture slides)
Dagmar Monett
 
A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)
Dagmar Monett
 
Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)
Dagmar Monett
 
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Dagmar Monett
 
Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)
Dagmar Monett
 
Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...
Dagmar Monett
 
How to Gather Requirements
How to Gather RequirementsHow to Gather Requirements
How to Gather Requirements
Clearworks
 
Elicitation techniques
Elicitation techniquesElicitation techniques
Elicitation techniques
daisy92081
 
Software Requirement Elicitation Techniques http://www.imran.xyz
Software Requirement Elicitation Techniques http://www.imran.xyzSoftware Requirement Elicitation Techniques http://www.imran.xyz
Software Requirement Elicitation Techniques http://www.imran.xyz
Imran Hussain Khan
 
Requirements Gathering Best Practice Pack
Requirements Gathering Best Practice PackRequirements Gathering Best Practice Pack
Requirements Gathering Best Practice Pack
Amy Slater
 

Viewers also liked (20)

Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:WireTeaching Students Collaborative Requirements Engineering. Case Study Red:Wire
Teaching Students Collaborative Requirements Engineering. Case Study Red:Wire
 
Index Determination in DAEs using the Library indexdet and the ADOL-C Package...
Index Determination in DAEs using the Library indexdet and the ADOL-C Package...Index Determination in DAEs using the Library indexdet and the ADOL-C Package...
Index Determination in DAEs using the Library indexdet and the ADOL-C Package...
 
Software Requirements Elicitation Methods
Software Requirements Elicitation MethodsSoftware Requirements Elicitation Methods
Software Requirements Elicitation Methods
 
Agile Project-Based Teaching and Learning
Agile Project-Based Teaching and LearningAgile Project-Based Teaching and Learning
Agile Project-Based Teaching and Learning
 
Agent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertation
Agent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertationAgent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertation
Agent-Based Configuration of (Metaheuristic) Algorithms - Doctoral dissertation
 
Experiences in Software Testing (lecture slides)
Experiences in Software Testing (lecture slides)Experiences in Software Testing (lecture slides)
Experiences in Software Testing (lecture slides)
 
Key Issues for Requirements Engineering (lecture slides)
Key Issues for Requirements Engineering (lecture slides)Key Issues for Requirements Engineering (lecture slides)
Key Issues for Requirements Engineering (lecture slides)
 
E-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical StudyE-Learning Adoption in a Higher Education Setting: An Empirical Study
E-Learning Adoption in a Higher Education Setting: An Empirical Study
 
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
Requirements Engineering Techniques for Eliciting Requirements (lecture slides)
 
Software requirement elicitation
Software requirement elicitationSoftware requirement elicitation
Software requirement elicitation
 
Introduction to Agents and Multi-agent Systems (lecture slides)
Introduction to Agents and Multi-agent Systems (lecture slides)Introduction to Agents and Multi-agent Systems (lecture slides)
Introduction to Agents and Multi-agent Systems (lecture slides)
 
A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)A Structured Approach to Requirements Analysis (lecture slides)
A Structured Approach to Requirements Analysis (lecture slides)
 
Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)Methods for Validating and Testing Software Requirements (lecture slides)
Methods for Validating and Testing Software Requirements (lecture slides)
 
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)Genetic Algorithms and Ant Colony Optimisation (lecture slides)
Genetic Algorithms and Ant Colony Optimisation (lecture slides)
 
Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)Requirements Engineering Methods for Documenting Requirements (lecture slides)
Requirements Engineering Methods for Documenting Requirements (lecture slides)
 
Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...Modelling Software Requirements: Important diagrams and templates (lecture sl...
Modelling Software Requirements: Important diagrams and templates (lecture sl...
 
How to Gather Requirements
How to Gather RequirementsHow to Gather Requirements
How to Gather Requirements
 
Elicitation techniques
Elicitation techniquesElicitation techniques
Elicitation techniques
 
Software Requirement Elicitation Techniques http://www.imran.xyz
Software Requirement Elicitation Techniques http://www.imran.xyzSoftware Requirement Elicitation Techniques http://www.imran.xyz
Software Requirement Elicitation Techniques http://www.imran.xyz
 
Requirements Gathering Best Practice Pack
Requirements Gathering Best Practice PackRequirements Gathering Best Practice Pack
Requirements Gathering Best Practice Pack
 

More from Dagmar Monett

Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Dagmar Monett
 
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Dagmar Monett
 
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
Dagmar Monett
 
The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning
Dagmar Monett
 
Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.
Dagmar Monett
 
Coming to terms with intelligence in machines
Coming to terms with intelligence in machinesComing to terms with intelligence in machines
Coming to terms with intelligence in machines
Dagmar Monett
 
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...
Dagmar Monett
 
Artificial Intelligence: The Promise, the Myth, and a Dose of Reality
Artificial Intelligence: The Promise, the Myth, and a Dose of RealityArtificial Intelligence: The Promise, the Myth, and a Dose of Reality
Artificial Intelligence: The Promise, the Myth, and a Dose of Reality
Dagmar Monett
 
Intelligence, the elusive concept and general capability still not found in m...
Intelligence, the elusive concept and general capability still not found in m...Intelligence, the elusive concept and general capability still not found in m...
Intelligence, the elusive concept and general capability still not found in m...
Dagmar Monett
 
The I in AI (or why there is still none)
The I in AI (or why there is still none)The I in AI (or why there is still none)
The I in AI (or why there is still none)
Dagmar Monett
 
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Dagmar Monett
 

More from Dagmar Monett (11)

Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
Narratives that speak AI lingua? AI vocabulary in listed companies' annual re...
 
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
Game-based Learning as a Suitable Approach for Teaching Digital Ethical Think...
 
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
University-Industry Collaboration's Next Level: A Comparative Study as Basis ...
 
The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning The Changing Landscape of Digital Technologies for Learning
The Changing Landscape of Digital Technologies for Learning
 
Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.Will Robots Take all the Jobs? Not yet.
Will Robots Take all the Jobs? Not yet.
 
Coming to terms with intelligence in machines
Coming to terms with intelligence in machinesComing to terms with intelligence in machines
Coming to terms with intelligence in machines
 
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence: ...
 
Artificial Intelligence: The Promise, the Myth, and a Dose of Reality
Artificial Intelligence: The Promise, the Myth, and a Dose of RealityArtificial Intelligence: The Promise, the Myth, and a Dose of Reality
Artificial Intelligence: The Promise, the Myth, and a Dose of Reality
 
Intelligence, the elusive concept and general capability still not found in m...
Intelligence, the elusive concept and general capability still not found in m...Intelligence, the elusive concept and general capability still not found in m...
Intelligence, the elusive concept and general capability still not found in m...
 
The I in AI (or why there is still none)
The I in AI (or why there is still none)The I in AI (or why there is still none)
The I in AI (or why there is still none)
 
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
Erfahrungen aus Projektbasiertes Lernen im Informatik Studium - The Missing p...
 

Recently uploaded

potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
huseindihon
 
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
sharonblush
 
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
kuldeepsharmaks8120
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
uapta
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
girewiy968
 
DataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptxDataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptx
Kanchana Weerasinghe
 
potential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in generalpotential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in general
huseindihon
 
Data Preprocessing Cheatsheet for learners
Data Preprocessing Cheatsheet for learnersData Preprocessing Cheatsheet for learners
Data Preprocessing Cheatsheet for learners
mohamed Ibrahim
 
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
dizzycaye
 
ch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ssch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ss
MinThetLwin1
 
Celonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptxCelonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptx
AnujaGaikwad28
 
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdfWhy_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Alexander Teggin
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
44annissa
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
LINAT
 
Biometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdfBiometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdf
Joel Ngushwai
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata
 
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
satpalsheravatmumbai
 
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
fatima shekh$A17
 
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
45unexpected
 
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
kinni singh$A17
 

Recently uploaded (20)

potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
 
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
 
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
 
DataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptxDataScienceConcept_Kanchana_Weerasinghe.pptx
DataScienceConcept_Kanchana_Weerasinghe.pptx
 
potential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in generalpotential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in general
 
Data Preprocessing Cheatsheet for learners
Data Preprocessing Cheatsheet for learnersData Preprocessing Cheatsheet for learners
Data Preprocessing Cheatsheet for learners
 
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
Female Service Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Se...
 
ch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ssch8_multiplexing cs553 st07 slide share ss
ch8_multiplexing cs553 st07 slide share ss
 
Celonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptxCelonis Busniess Analyst Virtual Internship.pptx
Celonis Busniess Analyst Virtual Internship.pptx
 
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdfWhy_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
Why_are_we_hypnotizing_ourselves-_ATeggin-1.pdf
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
 
Biometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdfBiometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdf
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
 
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
 
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
BDSM Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And ...
 
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
 
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
 

Predicting Star Ratings based on Annotated Reviewss of Mobile Apps [Slides]

  • 1. Predicting Star Ratings based on Annotated Reviews of Mobile Apps Talk at the 6th International Workshop on Advances in Semantic Information Retrieval ASIR 2016 Prof. Dr. Dagmar Monett, Hermann Stolte
  • 2. D. Monett Reviews and star ratings 2Gdańsk, Poland, September 11 – 14, 2016 Example of reviews and star ratings of the Evernote App, Google Play Store (07/2016)
  • 3. D. Monett Star ratings matter 3Gdańsk, Poland, September 11 – 14, 2016 15% would consider downloading an app with a 2-star rating 50% would consider downloading an app with a 3-star rating 96% would consider downloading an app with a 4-star rating Source: Aptentive 2015 Consumer Study The Mobile Marketer‘s Guide to App Store Ratings & Reviews
  • 4. D. Monett Star ratings matter 4Gdańsk, Poland, September 11 – 14, 2016 © and source: Aptentive 2015 Consumer Study The Mobile Marketer‘s Guide to App Store Ratings & Reviews
  • 5. D. Monett 5Gdańsk, Poland, September 11 – 14, 2016 Our motivation
  • 6. D. Monett Some questions… 6Gdańsk, Poland, September 11 – 14, 2016 ■ Could we (a program) teach users how to rate apps consistently with the review they are writing for a mobile app? ■ I.e., could we (a program) suggest to users the most adequate star rating they should give to a product depending on the semantic orientation of what they have already written in the review? ■ Would it mean an improvement of users' engagement and satisfaction with the app?
  • 7. D. Monett 7Gdańsk, Poland, September 11 – 14, 2016 Background
  • 8. D. Monett 8Gdańsk, Poland, September 11 – 14, 2016 Review rating prediction ■ Also sentiment rating prediction: ■ …a task that deals with the inference of an author's implied numerical rating, i.e. on the prediction of a rating score, from a given written review ■ E.g., recommendation systems often suggest products based on star ratings of similar products previously rated by other users
  • 9. D. Monett 9Gdańsk, Poland, September 11 – 14, 2016 Suggested readings
  • 10. D. Monett 10Gdańsk, Poland, September 11 – 14, 2016 Other related work ■ Analysing textual reviews and inferring sentiment polarity –positive/negative/neutral– (Pang et al. 2002; Liu, 2010) ■ Using not only textual semantics but also other information, e.g., about the author and/or the product (Tang et al., 2015; Li et al. 2011) ■ Considering phrase-level sentiment polarity (Qu et al., 2010) ■ Considering aspect-based opinion mining (Zhang et al., 2006; Ganu et al., 2013; Klinger & Cimiano, 2013; Sänger, 2015)
  • 11. D. Monett 11Gdańsk, Poland, September 11 – 14, 2016 Our approach
  • 12. D. Monett 12Gdańsk, Poland, September 11 – 14, 2016 Our approach ■ We do not deal with aspect identification nor with sentiment classification ■ We are assuming that these tasks are already performed before the star ratings are predicted ■ We focus on predicting star ratings based solely on available annotated, fine-granular opinions ■ I.e., a complement to works like (Sänger, 2015) which extends (Klinger & Cimiano, 2013) and use a German annotated corpus of mobile apps
  • 13. D. Monett 13Gdańsk, Poland, September 11 – 14, 2016 The Data
  • 14. D. Monett 14Gdańsk, Poland, September 11 – 14, 2016 SCARE Corpus Mario Sänger, Ulf Leser, Steffen Kemmerer, Peter Adolphs, and Roman Klinger. SCARE - The Sentiment Corpus of App Reviews with Fine-grained Annotations in German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). ■ Fine-grained annotations for mobile application reviews from the Google Play Store ■ 1,760 German application reviews with 2,487 aspects and 3,959 subjective phrases ■ SCARE corpus v.1.0.0 (annotations only) ■ Available at http://www.romanklinger.de/scare/
  • 15. D. Monett 15Gdańsk, Poland, September 11 – 14, 2016 Analysing the Data
  • 16. D. Monett 16Gdańsk, Poland, September 11 – 14, 2016 Polarity and star ratings 69.1% 23.1% Thumbs-up-thumbs-down (Liu, 2012)
  • 17. D. Monett Avg. of labelled star ratings vs. avg. of subjective phrases polarity 17Gdańsk, Poland, September 11 – 14, 2016
  • 18. D. Monett Number of star ratings vs. number of subjective phrases 18Gdańsk, Poland, September 11 – 14, 2016
  • 19. D. Monett 19Gdańsk, Poland, September 11 – 14, 2016 Predicting Star Ratings
  • 20. D. Monett Prediction process 20Gdańsk, Poland, September 11 – 14, 2016
  • 21. D. Monett 21Gdańsk, Poland, September 11 – 14, 2016 We “played” with different models
  • 22. D. Monett Computational models 22Gdańsk, Poland, September 11 – 14, 2016 For example, x0=1 x1 : no. of subjective phrases with positive polarity x2 : no. of subjective phrases with negative polarity x3 : no. of subjective phrases with neutral polarity
  • 23. D. Monett Computational models 23Gdańsk, Poland, September 11 – 14, 2016 RSS: review rating score (Ganu et al., 2009, 2013)
  • 24. D. Monett Experiments 24Gdańsk, Poland, September 11 – 14, 2016 (1) Assessing the importance of sentiment in the reviews: ■ Neutral phrases (yes/no)? ■ Reviews with no sentiment (yes/no)? (2) Using other predictors ■ Each individual experiment is run 10,000 times ■ A Monte Carlo cross-validation: 70% training dataset and 30% testing dataset, randomly on each iteration.
  • 25. D. Monett 25Gdańsk, Poland, September 11 – 14, 2016 Some results
  • 26. D. Monett “Best” model, exp. (1) 26Gdańsk, Poland, September 11 – 14, 2016 ■ It considers only the average value of the polarities of a review in one feature: ■ Plus: ■ filtering both subjective phrases with neutral polarity and reviews with no sentiment orientation at all ■ No normalisation
  • 27. D. Monett Results 27Gdańsk, Poland, September 11 – 14, 2016
  • 28. D. Monett 28Gdańsk, Poland, September 11 – 14, 2016 Conclusion
  • 29. D. Monett Conclusion 29Gdańsk, Poland, September 11 – 14, 2016 ■ Textually-derived rating prediction can be performed well even when only phrase-level sentiment polarity is available ■ Phrases with neutral sentiment could be filtered out of the corpus ■ Computing the overall sentiment of a review using the review rating score (Ganu et al., 2009, 2013) provides the best star rating predictions
  • 30. D. Monett Further work 30Gdańsk, Poland, September 11 – 14, 2016 ■ To consider the aspects’ relevance ■ aspect-oriented subjective phrases ■ To analyse the strengths of the opinions (Wilson et al., 2004) ■ not only positive/negative/neutral sentiment ■ To deal with other types of models different than linear, multivariate regression ones
  • 31. D. Monett Sources 31Gdańsk, Poland, September 11 – 14, 2016 Related work: - See references list on our paper! ■ https://www.researchgate.net/publication/304244445_Predi cting_Star_Ratings_based_on_Annotated_Reviews_of_Mo bile_Apps