Exploiting Distributional Semantics Models for Natural Language Context-aware Justifications for Recommender Systems

@cataldomusto
cataldo.musto@uniba.it
Exploiting Distributional Semantics Models
for Natural Language Context-aware
Justifications for Recommender Systems
CATALDO MUSTO, GIUSEPPE SPILLO, PASQUALE LOPS, MARCO DE GEMMIS, GIOVANNI SEMERARO
UNIVERSITÀ DEGLI STUDI DI BARI ‘ALDO MORO’ – ITALY
SWAP RESEARCH GROUP – HTTP://WWW.DI.UNIBA.IT/~SWAP
IntRS 2020 – Joint Workshop on
Interfaces and Human Decision-Making
for Recommender Systems
jointly held with ACM RecSys 2020
Online - Worldwide– September 26, 2020

The Explanation Problem
Recommendation
2Cataldo Musto, Giuseppe Spillo, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Exploiting Distributional Semantics Models for Natural
Language Context-aware Justifications for Recommender Systems. IntRS Workshop@ACM RecSys 2020 – Online – September 26, 2020
Profile

A solution: review-based features
To identify relevant and distinguishing
characteristics of the recommended
item by mining users’ reviews
Cataldo Musto, Pasquale Lops, Marco de Gemmis, Giovanni
Semeraro. Justifying Recommendations through Aspect-based
Sentiment Analysis of Users Reviews. UMAP 2019: 4-12

To identify relevant and distinguishing
characteristics of the recommended
item by mining users’ reviews
funny yarn
memorable writing
interesting concept
romantic end….

I recommend you Stranger Than
Fiction because people who liked the
movie think that it has a memorable
writing. Moreover, people liked
Stranger Than Fiction since it has a
romantic end.

Context plays a key role for
decision-making tasks
• Contextual factors (mood, company) do
influence the selection of the most
suitable item to be recommended;
6
…What about the context?
Cataldo Musto, Giuseppe Spillo, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro. Exploiting Distributional Semantics Models for Natural
SHALL AN EXPLANATION BE
INFLUENCED BY THE CONTEXT OF
CONSUMPTION?

A Methodology to
Generate Context-aware
Post-Hoc Natural Language
Justifications Exploiting
Distributional Semantics
Models
7
Contribution

A Methodology to
Generate Context-aware
Post-Hoc Natural Language
Justifications Exploiting
Distributional Semantics
Models
8
Contribution - Hallmarks
Justifications vary
depending on the different
contexts of consumption
Justifications are
independent of the
underlying recommendation
model
Justifications are generated
by exploiting a geometrical
representation of items,
contexts and sentences

9
Workflow

10
Workflow
Step 1
We learn a vector-space representation of ‘contexts’

11
Workflow
Step 2
We identify the most suitable review excerpts,
given an item and a vector-space representation
of ‘contexts’

12
Workflow
Step 3
We put together the review excerpts, to
generate the final context-aware justification

13
Context Learner
Step 1

• Goal: to build a ‘representation’ of the contexts
• Intuition: to exploit Distributional Semantics Models (DSMs) to
obtain a vector space representation of each context
14
Context Learner
content representation
company= friends
company= colleagues
company= family

Language Context-aware Justifications for Recommender Systems. IntRS Workshop@ACM RecSys 2020 – Online – September 26, 2020 15
Distributional Semantics Models
Ludwig Wittgenstein
(Austrian philosopher)
Meaning of a word is
determined by its usage.
«Words that share a similar context
have a similar meaning»

Ludwig Wittgenstein
(Austrian philosopher)
Recent techniques to represent
textual content (Word2Vec,
BERT, etc) are all inspired by
distributional hypothesis.
«Words that share a similar context
have a similar meaning»

A vector space representation of each word based
on word usage can be obtained
17
beer
wine
glass
spoon
This is called
WordSpace

c1 c2 c3 c4 c5 c6 c7 c8 c9
beer ✔ ✔ ✔ ✔
wine ✔ ✔ ✔ ✔ ✔
spoon ✔ ✔ ✔ ✔
glass ✔ ✔ ✔ ✔ ✔
Representation based on a term-context
matrix encoding term usage.
18

Good overlap = similar meaning
Each row of the matrix is a vector
19
c1 c2 c3 c4 c5 c6 c7 c8 c9
beer ✔ ✔ ✔ ✔
wine ✔ ✔ ✔ ✔ ✔
spoon ✔ ✔ ✔ ✔
glass ✔ ✔ ✔ ✔ ✔

• Question: how can we exploit DSMs for our goals?
We designed the following pipeline
1. Contexts Definition
2. Sentence Annotation
3. Vector Space Construction
4. Output Generation
20
Context Learner

1. Contexts Definition
◦ We manually define contextual factors and contextual dimensions for
a specific domain (e.g., movie recommendation)
21
Context Learner
Attention Company Mood
High AttentionLow Attention Family PartnerFriends Bad Mood Good Mood

2. Sentence Annotation
◦ To build a representation of each context, we need to manually
annotate sentences (e.g., reviews excerpts) with the set of contexts in
which they are suitable as context-aware justifications.
22
Context Learner
Not easy to understand, requires a very careful vision’
A fairy tale, pleasant and enchanting
A very romantic movie
(…repeat over many sentences)

◦ Once the annotation step is completed, we tokenize sentences
◦ We build a term-context matrix encoding term usage (as in DSMs)
23
Context Learner
careful ✔✔ ✔
fairy ✔✔ ✔
romantic ✔✔
intense ✔
easy ✔ ✔

◦ It is important to emphasize that we are not limited to single word.
Rows of the matrix can be also bigrams, as well.
24
Context Learner
careful vision ✔ ✔
fairy tale ✔ ✔
romantic
movie
✔
intense plot ✔
easy vision ✔ ✔

◦ Column Vectors = Vector Space Representation of Each Context
◦ Lexicons = top-k lemmas with the highest score in a column
25
Context Learner
careful ✔✔ ✔
fairy ✔✔ ✔
romantic ✔✔
intense ✔
easy ✔ ✔

◦ Column Vectors = Vector Space Representation of Each Context
◦ Lexicons = top-k lemmas with the highest score in a column
26
Context Learner
= { fairy, calm, story, kids … }
= { atmoshpere, romantic, … }
= { funny, simple, smooth … }

27
Ranker
Step 2

• Goal: given an item and context of consumption, to identify the
most suitable review excerpts
• Intuition: to adopt similarity measures in geometrical spaces
28
Ranker
representation
company= friends
company= colleagues
company= family

Ranker
friends
family
partner
We start from
the output
returned by the
Context Learner

Ranker
Given a recommended
item, we encode in the
vector space the
available review
excerpts
We limit to sentences
expressing a positive
sentiment
friends
family
partner
‘it is a classy, sweet and funny movie’
‘it has a memorable writing’
‘the movie has a very romantic end’

Ranker
Next, given a context of
consumption, we
identify the top-K review
excerpts by exploiting
similarity measures in
geometrical spaces
(e.g., cosine similarity)friends
family
partner
‘it has a memorable writing’

Ranker
consumption, we
geometrical spaces
family
partner

34
Generator
Step 3

• Goal: to combine the top-k review excerpts in a natural language
justification adapted to the context of consumption
• Intuition: to exploit natural language generation techniques
• Each justification has a fixed part, which is common to all the justifications, and a
dynamic part, which is filled in based on previously identified excerpts.
35
Generator
You should watch ’Stranger than
Fiction’. It is a good movie to
watch with your partner because
it has a very romantic end.
Moreover, plot is very intense.
You should watch ’Stranger than Fiction’.
It is a good movie to watch with your
friends because it crackles with
laughther and pathos and it is a classy
sweet and funny movie.

Final Output
You should watch
’Stranger than
Fiction’. It is a good
movie to watch with
your partner
because it has a
very romantic end.
Moreover, plot is
very intense.
You should watch
’Stranger than
Fiction’. It is a good
movie to watch with
your friends because
it crackles with
laughther and pathos
and it is a classy
sweet and funny
movie.
Context-aware Natural Language
Justification based on DSMs

Experimental Evaluation
Research Question 1 (RQ1)
How effective are the justifications generated through the pipeline, on varying of different
combinations of the parameters?
Research Question 2 (RQ2)
How does our justifications perform with respect to non-contextual justifications and contextual
justifications based on a fixed lexicon?
Experimental Design
User Study with a Web Application
273 subjects - Movie Domain. 300 movies. ~150k reviews.
Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^]
Parameters: Lexicon (Unigram, Bigrams and Unigram+Bigrams)
Between-subjects for Research Question 1, Within-subjects for Research Question 2
[^] Tintarev, N., & Masthoff, J. Designing and
evaluating explanations for recommender
systems. In Recommender systems
handbook. pp. 479-510. Springer, Boston,
MA. 2011

Experimental Evaluation – WebApp (RQ1)
Welcome
Screen
Context
Selection

Generation of
the Justification
Questionnaire
Transparency
Persuasion
Engagement
Trust

Comparison
to Baselines
Questionnaire
Transparency
Persuasion
Engagement
Trust

Results (Research Question 1)
41
Question Unigrams (Uni) Bigrams (Bi) Uni+Bi
Transparency «I understood why the movie was
suggested to me»
3.38 3.81 3.64
Persuasion «The justification made the
recommendation more convincing»
3.56 3.62 3.54
Engagement «The justification allowed me to discover
more information about the movie»
3.54 3.72 3.70
Trust «The justification increased my trust in
recommender systems»
3.44 3.66 3.61

42
Question Unigrams (Uni) Bigrams (Bi) Uni+Bi
Transparency «I understood why the movie was
suggested to me»
3.38 3.81 3.64
Persuasion «The justification made the
recommendation more convincing»
3.56 3.62 3.54
Engagement «The justification allowed me to discover
more information about the movie»
3.54 3.72 3.70
Trust «The justification increased my trust in
recommender systems»
3.44 3.66 3.61
Intuition: bigrams (e.g., romantic
soundtrack) better catch semantics
of reviews excerpts

MOVIES CA+DSMs Baseline Indifferent
Transparency 52.38% 38.10% 19.32%
Persuasion 54.10% 36.33% 19.57%
Engagement 49.31% 39.23% 11.56%
Trust 42.86% 39.31% 17.83%
Improvement over a non-contextual
baseline based on DSMs

MOVIES CA+DSMs Baseline Indifferent
Transparency 53.21% 34.47% 12.32%
Persuasion 55.17% 32.33% 12.50%
Engagement 44.51% 32.75% 22.74%
Trust 42.90% 42.11% 14.99%
Improvement over a contextual
baseline based on a static lexicon

Recap
Hallmarks
◦ Diversification of the justification based on the context of consumption
◦ Adoption of DSMs to (unsupervisedly) learn a vector-space representation of context
Contribution
◦ A domain-independent framework to generate post-hoc context-aware review-based
natural language justifications
Findings
◦ A representation based on bigrams better catches the semantics of the different context
of consumptions
◦ Users tend to prefer context-aware justifications, and DSMs allow to build a more
effective representation
45

Future Work
Generation of personalized
justifications
◦ We aim to encode user preferences into
the generation process
Evaluation of the post-hoc nature
◦ To assess whether the model is solid
enough to ‘explain’ also more complex
and opaque deep learning models
Generation of hybrid justifications
◦ Combining structured features and
review-based features
46
RecSys
2021

Thank you! cataldo.musto@uniba.it
@cataldomusto
Contacts
47

Exploiting Distributional Semantics Models for Natural Language Context-aware Justifications for Recommender Systems

More Related Content

Similar to Exploiting Distributional Semantics Models for Natural Language Context-aware Justifications for Recommender Systems

More from Cataldo Musto

Recently uploaded

Exploiting Distributional Semantics Models for Natural Language Context-aware Justifications for Recommender Systems