Justifying Recommendations through Aspect-based Sentiment Analysis of Users Reviews

@cataldomusto cataldo.musto@uniba.it
Justifying Recommendations through
Aspect-based Sentiment Analysis
of Users’ Reviews
CATALDO MUSTO, PASQUALE LOPS, MARCO DE GEMMIS, GIOVANNI SEMERARO
UNIVERSITÀ DEGLI STUDI DI BARI ‘ALDO MORO’ - ITALY
UMAP 2019 – 27th International
Conference on User Modeling,
Adaptation and Personalization
Larnaca (Cyprus) – June 10, 2019

Recommender
Systems
Cataldo Musto, Pasquale Lops, Marco de Gemmis, Giovanni Semeraro.
Justifying Recommendations through Aspect-based Sentiment Analysis of Users’ Reviews. UMAP 2019 – Larnaca (Cyprus) – June 10, 2019

Motivations
Recommendation
I suggest you

Motivations
?
Recommendation

Intuition
To identify relevant and distinguishing
characteristics of the recommended
item by mining users’ reviews, and to use
these characteristics to build a natural
language justification

Intuition
Breath-taking images
Excellent actors
Excellent sound design
Wonderful storytelling
To identify relevant and distinguishing
characteristics of the recommended
item by mining users’ reviews, and to use
these characteristics to build a natural
language justification

Intuition
"I recommend you 300 because people
who liked the movie think that the war
scenes are really well done. Moreover,
people liked 300 since it has a wonderful
storytelling and the actors are excellent"
A potential review-based justification

Solution

Aspect Extraction

Aspect Extraction
Goal: to identify the aspects that are discussed when
people talk about the item
Strategy: to use natural language processing techniques
(specifically, part-of-speech tagging) to extract the names
mentioned in users’ reviews

Aspect Extraction
reviews aspects
Input: reviews of the item i R = {ri1, ri2 … rin}
Output: aspects A = {ai1, ai2 … aik}

Aspect Extraction
reviews aspects
Input: reviews of the item i R = {ri1, ri2 … rin}
Output: aspects A = {ai1, ai2 … aik}
Why only names?
Findings from previous
work in the area
Why no bigrams?
No significant
improvement emerged

Aspect Ranking

Aspect Ranking
Aspect Extraction Goal: to identify the aspects that are
discussed when people talk about the item
Aspect Ranking Goal: to identify distinguishing aspects
that are discussed with a positive sentiment when people
talk about the item

Aspect Ranking
Aspect Extraction Goal: to identify the aspects that are
discussed when people talk about the item
Aspect Ranking Goal: to identify distinguishing aspects
that are discussed with a positive sentiment when people
talk about the item Strategy: to calculate a relevance score
for each aspect previously detected

Aspect Ranking
Goal: to identify distinguishing aspects that are discussed
with a positive sentiment when people talk about the item

Aspect Ranking
How many times aspect ‘a’ appears in the
reviews of item ‘i’ (frequency of the aspect)

Aspect Ranking
How positive is the opinion of the users
when they talk about aspect ‘a’ (opinion
towards the aspect)

Aspect Ranking
How positive is the opinion of the users
when they talk about aspect ‘a’ (opinion
towards the aspect)
How distinguishing is
the aspect ‘a’ (inverse
popularity)

Aspect Ranking
Intuition: our formula gives an higher score to the aspects that are
frequently mentioned in the reviews with a positive sentiment.
Moreover, it also rewards less popular aspects (higher IAF).

Aspect Ranking
aspects top-k aspects
Input: aspects A = {ai1, ai2 … aim}
Output: top-k aspects A = {ai1, ai2 … aik}

Generation

Generation
Goal: to generate a natural language justification that
relies on the most relevant aspects identified by the ASPECT
RANKING module.
For each aspect ’a’ returned by the ASPECT RANKING module
Browse the available reviews
Look for a compliant excerpt containing ‘a’
If the sentence has a positive sentiment
Add the sentence to the justification
Strategy

Generation
Question: when does an excerpt is a compliant sentence?
Answer: an excerpt is compliant if it follows one of the 18
justification patterns we defined
Example: the excerpt must have a third personal
singular verb

Generation
Question: when does an excerpt is a compliant sentence?
Answer: an excerpt is compliant if it follows one of the 18
justification patterns we defined
“I really liked the cast” Not compliant
“The cast was great” Compliant
Example: the excerpt must have a third personal
singular verb

Generation – Final Output
"I recommend you 300 because people who liked the
movie think that the war scenes are really well done.
Moreover, people liked 300 since it has a wonderful
storytelling and the actors are excellent"
Legenda
Blue: randomized sentences
Green: recommendation
Red: aspects (k=3)
Black: compliant excerpts

Experimental Evaluation
Research Question 1
How effective are the justifications generated through the pipeline, on varying of different
combinations of the parameters?
Research Question 2
How does our justifications perform with respect to a classic feature-based explanation [*] ?
(e.g.) “I suggest you 300 since you like other movies by Zack Snyder, as Watchmen”
Experimental Design
User Study with a Web Application
286 subjects
Movie and Books Domain
Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^]
Between-subjects for Research Question 1, Within-subjects for Research Question 2
[*] Musto, C., Narducci, F., Lops, P., De Gemmis, M., Semeraro G.
ExpLOD: A Framework for Explaining Recommendations based on
the Linked Open Data Cloud. In Proceedings of the 10th ACM
Conference on Recommender Systems. pp. 151-154. 2016
[^] Tintarev, N., & Masthoff, J. Designing and evaluating
explanations for recommender systems. In Recommender
systems handbook. pp. 479-510. Springer, Boston, MA. 2011

Parameters of the system
- Length of the justifications (short vs. long justifications)
short → top-2 aspects long -> top-4 aspects
- Vocabulary of aspects (statics vs. complete)
static → bounded to a fixed and pre-defined set of relevant aspects. No aspect
extraction, just aspect ranking
complete → not bounded. All the aspects are discovered by the aspect extractor
- Four different configurations
Implementation Details
Recommendations generated through Personalized PageRank, aspect extraction through
CoreNLP POS-tagger and sentiment analysis through Stanford algorithm

Statistics of the datasets
Movies Books
#items 307 333
#reviews 153,566 52,560
avg. reviews/item 500.21 157.80
avg. words/reviews 138.38 126.65
Dataset are obtained by mapping items from MovieLens and BookCrossing
datasets with reviews obtained from Amazon data (dataset available)

Web Application
Recommendation

Web Application (Research Question 1)
Recommendation
Justification
Questionnaire

Web Application (Research Question 2)
I propose you “Aliens”
because you sometimes like
movies edited by Canadian
film editors, American fiction
films and 1980s films, as The
Terminator.
I recommend you “Aliens”
because people who liked this
movie think that the Alien
series is one of the best sci-fi
movies and that the ending is
awesome with some fantastic
action scenes.
Review-based
Explanation
Feature-based
Explanation

Results (Research Question 1)
MOVIES Transparency Persuasion Engagement Trust Effectiveness
Static Short 3.40 3.13 3.09 3.23 0.64
Static Long 3.77 3.68 3.55 3.73 0.55
Complete Short 3.91 3.60 3.25 3.70 0.53
Complete Long 3.74 3.48 3.35 3.46 0.59

Finding 1
With the ‘complete’ set of
aspects, shorter justifications
have the best results
Static Short 3.40 3.13 3.09 3.23 0.64
Static Long 3.77 3.68 3.55 3.73 0.55
Complete Short 3.91 3.60 3.25 3.70 0.53
Complete Long 3.74 3.48 3.35 3.46 0.59

Finding 1
Finding 2
With the ‘static’ set of
aspects, longer justifications
Static Short 3.40 3.13 3.09 3.23 0.64
Static Long 3.77 3.68 3.55 3.73 0.55
Complete Short 3.91 3.60 3.25 3.70 0.53
Complete Long 3.74 3.48 3.35 3.46 0.59

Finding 1
Finding 2
With the ‘static’ set of
aspects, longer justifications
Overall
Long justifications based on
static aspects have the best
results in the Movie Domain
Static Short 3.40 3.13 3.09 3.23 0.64
Static Long 3.77 3.68 3.55 3.73 0.55
Complete Short 3.91 3.60 3.25 3.70 0.53
Complete Long 3.74 3.48 3.35 3.46 0.59

BOOKS Transparency Persuasion Engagement Trust Effectiveness
Static Short 3.57 3.40 3.07 3.31 0.74
Static Long 3.64 3.82 3.45 3.71 0.45
Complete Short 3.82 3.59 3.37 3.51 0.47
Complete Long 3.61 3.41 3.31 3.30 0.57
Findings confirmed for the Books domain.
Static list of aspects + Long justifications = Best Performance

MOVIES
Review
-based
Feature-
based
Indifferent
Transparency 47.4% 38.6% 14.0%
Persuasion 51.7% 43.3% 5.0%
Engagement 66.7% 25.0% 8.3%
Trust 53.3% 35.5% 11.7%
Effectiveness 57.9% 35.0% 7.1%
Outcome: Users preferred Review-based Justifications
Confirmed for all the metrics and both the domains
BOOKS
Review-
based
Feature-
based
Indifferent
Transparency 58.1% 36.0% 5.9%
Persuasion 61.8% 29.0% 9.2%
Engagement 54.6% 27.3% 18.1%
Trust 58.2% 27.2% 14.6%
Effectiveness 59.9% 31.1% 10.0%

Recap
Exploitation of Users’ Review to Generate Natural Language Justification
Natural Language Processing and Sentiment Analysis pipeline
◦ Aspect Extraction
◦ Aspect Ranking
◦ Generation of the Justification
Outcomes of the Experiments
◦ If we bound the generation to a static list of aspects, we can generate longer justifications
◦ Otherwise shorter justifications are preferred (due to some noise?)
◦ In general, review-based justifications are preferred to classic feature-based explanations

Future Work
To evaluate new and different domains
To improve the implementation of the single modules
◦ Aspect Extraction (try to extract bigrams or entities)
◦ Aspect Ranking (more sophisticated techniques can be exploited)
◦ Generation of the Justification (make the natural language generation more dynamic)
To introduce methods to generate hybrid explanations
◦ Combinations of review data with descriptive features

Thank you!
cataldo.musto@uniba.it
@cataldomusto
Contacts

Justifying Recommendations through Aspect-based Sentiment Analysis of Users Reviews

More Related Content

More from Cataldo Musto

Recently uploaded

Justifying Recommendations through Aspect-based Sentiment Analysis of Users Reviews