Our lecture for the RecSys Summer School 2019 in Goteborg (Sweden).

1. @cataldomusto cataldo.musto@uniba.it
Advances in Content-based Recommender Systems
Explanation Strategies
CATALDO MUSTO
UNIVERSITÀ DEGLI STUDI DI BARI ‘ALDO MORO’ - ITALY

2. Recommender
Systems
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 2

3. The Explanation Problem
Recommendation
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
I suggest you…
3

4. The Explanation Problem
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Recommendation
4

5. A possible solution: descriptive properties
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Recommendation
I suggest you The Ring because you
often like movies with Naomi Watts
as 21 grams and Mulholland Drive.
Furthermore, you like films about
ghosts such as The Sixth Sense.
5

6. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Another solution: review-based features
I recommend you The Ring because
people who liked the movie think that
it delivers some bone-chilling terror.
Moreover, people liked The Ring
since the casting is pretty good.
6

7. An overview of content-based strategies
to build a domain-agnostic and
algorithm-agnostic explanation
supporting the recommendation.
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
In this talk
7

8. An overview of content-based strategies
to build a domain-agnostic and
algorithm-agnostic explanation
supporting the recommendation.
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
In this talk
8

9. 1. Content-based Explanations exploiting
the Linked Open Data cloud
2. Review-based Explanation exploiting
Sentiment Analysis techniques
3. Review-based Explanations exploiting
Automatic Text Summarization
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Agenda
9

10. 1. Content-based Explanations exploiting
the Linked Open Data cloud
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Agenda
Cataldo Musto, Fedelucio Narducci, Pasquale Lops, Marco de Gemmis,
Giovanni Semeraro: ExpLOD: A Framework for Explaining Recommendations
based on the Linked Open Data Cloud.
Proceedings of RecSys 2016: pp. 151-154 (Best Paper Nominee)
10

11. 2. Review-based Explanation exploiting
Sentiment Analysis techniques
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Agenda
Cataldo Musto, Pasquale Lops, Marco de Gemmis, Giovanni Semeraro:
Justifying Recommendations through Aspect-based Sentiment Analysis of
Users Reviews.
Proceedings of ACM UMAP 2019: pp. 4-12
11

12. 3. Review-based Explanations exploiting
Automatic Text Summarization
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Agenda
Cataldo Musto, Gaetano Rossiello, Marco de Gemmis, Pasquale Lops, Giovanni
Semeraro: Combining Text Summarization and Aspect-based Sentiment
Analysis of Users’ Reviews to Justify Recommendations.
To be presented at ACM RecSys 2019☺
12

13. Content-based Explanations exploiting the
Linked Open Data cloud
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
1.
13

14. Intuition
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Descriptive features of
the items can be freely
gathered from
knowledge graphs as
DBpedia
(http://dbpedia.org)
14

15. Properties from DBpedia
The Ring
Ghost Films
Hans
Zimmer
Naomi
Watts
Psychological
Horror Films
Films shot
in California
Horror
Movies
Japanese
Movies
Gore
Verbinski
dcterms:subject dbo:starring
dcterms:subjectdcterms:subject
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 15

16. Methodology
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 16

17. ExpLOD Framework
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 17

18. ExpLOD: Mapper
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 18

19. ExpLOD: Mapper
Mapper
Profile Recommendations
dbp:The_Ring_(2002_film)dbp:21_grams
Profile Recommendation
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 19

20. ExpLOD: Builder
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 20

21. ExpLOD: Builder
Recommendation
American
Films
Psychological
Movies
Films about
Ghosts
Naomi
Watts
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 21

22. ExpLOD: Ranker
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 22

23. ExpLOD: Ranker
items in the
user profile and in
the recommendation list
property
number of edges
connecting the property c
with the items in
the user profile
number of edges
connecting the property c
with the items in
the recommendation set
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 23

24. ExpLOD: Ranker
Recommendation
American
Films
Psychological
Movies
Films about
Ghosts
Naomi
Watts
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 24

25. ExpLOD: Generator
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 25

26. ExpLOD: Generator
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 26

27. ExpLOD: Generator
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Naomi
Watts
I suggest you The Ring because you
often like movies with Naomi Watts
as 21 grams and Mulholland Drive.
27

28. ExpLOD: Generator
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Naomi
Watts
Films about
Ghosts
Furthermore, you like films about
ghosts such as The Sixth Sense.
28

29. ExpLOD: final output
Recommendation
I suggest you The Ring because
you often like movies with
Naomi Watts as 21 grams and
Mulholland Drive. Furthermore,
you like films about ghosts such
as The Sixth Sense.
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 29

30. Experimental Evaluation
Research Question
How does our explanations perform with respect to other explanation strategies?
Experimental Design
User Study with a Web Application
308 subjects, Movie Domain
Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^]
Between-subjects experiment
Configurations: ExpLOD, popularity-based baseline, non-personalized baseline
[^] Tintarev, N., & Masthoff, J. Designing and evaluating
explanations for recommender systems. In Recommender
systems handbook. pp. 479-510. Springer, Boston, MA. 2011
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 30

31. Experimental Protocol
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
1. Gathering movie preferences
Users rated their favourite movies
31

32. Experimental Protocol
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
2. Recommendation is obtained
Personalized PageRank as algorithm
1. Gathering movie preferences
Users rated their favourite movies
32

33. Experimental Protocol
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
2. Recommendation is obtained
Personalized PageRank as algorithm
3. Explanation is generated
Random Configuration (users not aware)
1. Gathering movie preferences
Users rated their favourite movies
33

34. Experimental Protocol
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
1. Gathering movie preferences
Users rated their favourite movies
2. Recommendation is obtained
Personalized PageRank as algorithm
3. Explanation is generated
Random Configuration (users not aware)
4. Metrics are calculated
Through Questionnaires
34

35. Explanations - Results
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
MOVIES ExpLOD Non-personalized Popularity
transparency 4.18 3.04 3.01
persuasion 3.41 2.84 2.59
engagement 3.48 3.28 2.31
trust 3.39 2.81 2.67
effectiveness 0.72 0.66 0.93
35

36. Explanations - Results
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
MOVIES ExpLOD Non-personalized Popularity
transparency 4.18 3.04 3.01
persuasion 3.41 2.84 2.59
engagement 3.48 3.28 2.31
trust 3.39 2.81 2.67
effectiveness 0.72 0.66 0.93
«I recommend you The Ring since you should like movies by
Gore Verbinski whose music composer is Hans Zimmer»
36

37. Explanations - Results
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
MOVIES ExpLOD Non-personalized Popularity
transparency 4.18 3.04 3.01
persuasion 3.41 2.84 2.59
engagement 3.48 3.28 2.31
trust 3.39 2.81 2.67
effectiveness 0.72 0.66 0.93
«I recommend you The Ring since it is very
popular in the community»
37

38. Explanations - Results
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
MOVIES ExpLOD Non-personalized Popularity
transparency 4.18 3.04 3.01
persuasion 3.41 2.84 2.59
engagement 3.48 3.28 2.31
trust 3.39 2.81 2.67
effectiveness 0.72 0.66 0.93
Significant improvement for 4 out of 5 metrics
38

39. Explanations - Results
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Aim Question
transparency
I understood why this movie was
recommended to me
topic
director
distributor
composer
persuasion
The explanation made the
recommendation more convincing
awards
director
location
producer
engagement
The explanation helped me discover new
information
writer
director
producer
distributor
trust
The explanation increased my trust in the
recommender system
awards
composer
producer
topic
effectiveness I like this recommendation
director
writer
location
composer
39

40. Review-based Explanation exploiting
Sentiment Analysis techniques
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
2.
40

41. Intuition
To identify relevant and distinguishing
characteristics of the recommended
item by mining users’ reviews
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 41

42. Intuition
Intense thriller
Pretty good casting
Well-plotted investigation
Impressive horror
......
To identify relevant and distinguishing
characteristics of the recommended
item by mining users’ reviews
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 42

43. Workflow
43Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019

44. Aspect Extraction
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 44

45. Aspect Extraction
Goal: to identify the aspects that are
discussed when people talk about the item
Strategy: to use natural language
processing techniques (specifically, part-
of-speech tagging) to extract the names
mentioned in users’ reviews
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 45

46. Aspect Extraction
Goal: to identify the aspects that are
discussed when people talk about the item
Strategy: to use natural language
processing techniques (specifically, part-
of-speech tagging) to extract the names
mentioned in users’ reviews
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 46

47. Aspect Extraction
reviews aspects
Input: reviews of the item i R = {ri1, ri2 … rin}
Output: aspects A = {ai1, ai2 … aik}
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 47

48. Aspect Extraction
reviews aspects
Input: reviews of the item i R = {ri1, ri2 … rin}
Output: aspects A = {ai1, ai2 … aik}
Why only names?
Findings from previous
work in the area
Why no bigrams?
No significant
improvement emerged
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 48

49. Aspect Ranking
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 49

50. Aspect Ranking
Goal: to identify distinguishing aspects that are discussed
with a positive sentiment when people talk about the item
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 50

51. Aspect Ranking
Goal: to identify distinguishing aspects that are discussed
with a positive sentiment when people talk about the item
How many times aspect ‘a’ appears in the
reviews of item ‘i’ (frequency of the aspect)
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 51

52. Aspect Ranking
Goal: to identify distinguishing aspects that are discussed
with a positive sentiment when people talk about the item
How many times aspect ‘a’ appears in the
reviews of item ‘i’ (frequency of the aspect)
How positive is the opinion of the users
when they talk about aspect ‘a’ (opinion
towards the aspect)
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 52

53. Aspect Ranking
Goal: to identify distinguishing aspects that are discussed
with a positive sentiment when people talk about the item
How many times aspect ‘a’ appears in the
reviews of item ‘i’ (frequency of the aspect)
How positive is the opinion of the users
when they talk about aspect ‘a’ (opinion
towards the aspect)
How distinguishing is
the aspect ‘a’ (inverse
popularity)
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 53

54. Aspect Ranking
Goal: to identify distinguishing aspects that are discussed
with a positive sentiment when people talk about the item
Intuition: our formula gives an higher score to the aspects that are
frequently mentioned in the reviews with a positive sentiment.
Moreover, it also rewards less popular aspects (higher IAF).
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 54

55. Aspect Ranking
aspects top-k aspects
Input: aspects A = {ai1, ai2 … aim}
Output: top-k aspects A = {ai1, ai2 … aik}
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 55

56. Generation
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 56

57. Generation
Goal: to generate a template-based natural language
justification that relies on the most relevant aspects
identified by the ASPECT RANKING module.
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 57

58. Generation
Goal: to generate a template-based natural language
justification that relies on the most relevant aspects
identified by the ASPECT RANKING module.
For each aspect ’a’ returned by the ASPECT RANKING module
Browse the available reviews
Look for a compliant excerpt containing ‘a’
If the sentence has a positive sentiment
Add the sentence to the justification
Strategy
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 58

59. Generation
Question: when does an excerpt is a compliant sentence?
Answer: an excerpt is compliant if it follows one of the 18
justification patterns we defined
Example: the excerpt must have a third personal
singular verb
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 59

60. Generation
Question: when does an excerpt is a compliant sentence?
“I really liked the cast” Not compliant
“The cast was great” Compliant
Example: the excerpt must have a third personal
singular verb
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 60
Answer: an excerpt is compliant if it follows one of the 18
justification patterns we defined

61. Generation – Final Output
I recommend you The Ring because people who
liked the movie think that it delivers some bone-
chilling terror. Moreover, people liked The Ring
since the casting is pretty good.
Legenda
Red: randomized template sentences
Green: recommendation
Blue: aspects (k=2)
Black: compliant excerpts
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 61

62. Experimental Evaluation
Research Question 1
How effective are the justifications generated through the pipeline, on varying of different
combinations of the parameters?
Research Question 2
How does our justifications perform with respect to a classic feature-based explanation?
Experimental Design
User Study with a Web Application
286 subjects
Movie and Books Domain
Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness
Between-subjects for Research Question 1, Within-subjects for Research Question 2
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 62

63. Experimental Evaluation
Parameters of the system
- Length of the justifications (short vs. long justifications)
short → top-2 aspects long -> top-4 aspects
- Vocabulary of aspects (statics vs. complete)
static → bounded to a fixed and pre-defined set of relevant aspects. No aspect
extraction, just aspect ranking
complete → not bounded. All the aspects are discovered by the Aspect Extractor
- Four different configurations
Implementation Details
Recommendations generated through Personalized PageRank, aspect extraction through
CoreNLP POS-tagger and sentiment analysis through Stanford algorithm
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 63

64. Experimental Protocol
Recommendation
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 64

65. Experimental Protocol (Research Question 1)
Recommendation
Review-based
Explanation
Questionnaire
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 65

66. Experimental Protocol (Research Question 2)
I propose you “Aliens”
because you sometimes like
movies edited by Canadian
film editors, American fiction
films and 1980s films, as The
Terminator.
I recommend you “Aliens”
because people who liked this
movie think that the Alien
series is one of the best sci-fi
movies and that the ending is
awesome with some fantastic
action scenes.
Review-based
Explanation
Feature-based
Explanation
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 66

67. Results (Research Question 1)
MOVIES Transparency Persuasion Engagement Trust Effectiveness
Static Short 3.40 3.13 3.09 3.23 0.64
Static Long 3.77 3.68 3.55 3.73 0.55
Complete Short 3.91 3.60 3.25 3.70 0.53
Complete Long 3.74 3.48 3.35 3.46 0.59
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 67

68. Results (Research Question 1)
Finding 1
With the ‘complete’ set of
aspects, shorter justifications
have the best results
Finding 2
With the ‘static’ set of
aspects, longer justifications
have the best results
Overall
Long justifications based on
static aspects have the best
results in the Movie Domain
MOVIES Transparency Persuasion Engagement Trust Effectiveness
Static Short 3.40 3.13 3.09 3.23 0.64
Static Long 3.77 3.68 3.55 3.73 0.55
Complete Short 3.91 3.60 3.25 3.70 0.53
Complete Long 3.74 3.48 3.35 3.46 0.59
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 68

69. Results (Research Question 2)
MOVIES
Review-
based
Feature-
based
Indiffer.
Transparency 47.4% 38.6% 14.0%
Persuasion 51.7% 43.3% 5.0%
Engagement 66.7% 25.0% 8.3%
Trust 53.3% 35.5% 11.7%
Effectiveness 57.9% 35.0% 7.1%
Outcome: Users preferred Review-based Justifications
Confirmed for all the metrics and both the domains
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 69

70. Review-based Explanations exploiting
Automatic Text Summarization
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
3.
70

71. Why do we need another
approach that exploits
users’ reviews?
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Motivations
71

72. Why do we need another
approach that exploits
users’ reviews?
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Motivations
Our first methodology has
two main weaknesses
• Very naïve strategy for ASPECT EXTRACTION
• Very static template-based GENERATION
72

73. To exploit automatic text summarization
techniques to build an higher-quality justifications.
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Intuition
73

74. To exploit automatic text summarization
techniques to build an higher-quality justifications.
We conceive our justification as a summary of the
information conveyed by all the available reviews.
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Intuition
74

75. Workflow
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 75

76. Workflow
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Same conceptual
workflow, different
implementations of
the modules!
76

77. Aspect Extraction
Statistical approach based on the Kullback-Leibler
(KL) Divergence
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Measures the difference between the distribution of a term
in a generic corpus (e.g. BNC) and its distribution in a domain corpus
(e.g. movie reviews)
77

78. Aspect Extraction
Measures the difference between the distribution of a term
in a generic corpus (e.g. BNC) and its distribution in a domain corpus
(e.g. movie reviews)
Insight: the higher the divergence, the higher the
importance of the term in the domain
t = term
ca = corpus A
cb = corpus B
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Statistical approach based on the Kullback-Leibler
(KL) Divergence
78

79. Aspect Extraction
Measures the difference between the distribution of a term
in a generic corpus (e.g. BNC) and its distribution in a domain corpus
(e.g. movie reviews)
Insight: the higher the divergence, the higher the
importance of the term in the domain
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
KL(cast, BNC, movie-reviews) >> 0
KL(actor, BNC, movie-reviews) > 0
KL(city, BNC, movie-reviews) ~ 0
KL(woman, BNC, movie-reviews) ~ 0
We label as ‘aspects’ the
nouns whose
KL-divergence is higher
than zero
Statistical approach based on the Kullback-Leibler
(KL) Divergence
79

80. Aspect Extraction
Measures the difference between the distribution of a term
in a generic corpus (e.g. BNC) and its distribution in a domain corpus
(e.g. movie reviews)
Insight: the higher the divergence, the higher the
importance of the term in the domain
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
KL(cast, BNC, movie-reviews) >> 0 YES
KL(actor, BNC, movie-reviews) > 0 YES
KL(city, BNC, movie-reviews) ~ 0 NO
KL(woman, BNC, movie-reviews) ~ 0 NO
Statistical approach based on the Kullback-Leibler
(KL) Divergence
80

81. Aspect Ranking
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 81

82. Aspect Ranking
Goal: to identify distinguishing aspects that are discussed
with a positive sentiment when people talk about the item
Novelty: KL-divergence is used as relevance score rela,Ri
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 82

83. Generation
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Same conceptual
workflow, different
implementations of
the modules!
83

84. Generation
Intuition: we conceive our justification as a summary of the
information conveyed by all the available reviews
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 84

85. Generation
Intuition: we conceive our justification as a summary of the
information conveyed by all the available reviews
Approach: we exploited a centroid-based method for automatic text
summarization. Very good performance in multi-document
summarization scenarios.
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Assumption: each review can be considered as ‘document’ thus the
corpus of the reviews can be used to feed the algorithm
85

86. Generation
Generation process is in turn split into two steps
• Sentence Filtering
• Text Summarization
Sentence Filtering is used to feed the summarization algorithm
with compliant sentences. We selected sentences that matched
the following criterions:
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 86

87. Generation
Generation process is in turn split into two steps
• Sentence Filtering
• Text Summarization
Sentence Filtering is used to feed the summarization algorithm
with compliant sentences. We selected sentences that matched
the following criterions:
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
• The sentence contains a main aspect
• The sentence is longer than 5 tokens
• The sentence expresses a positive sentiment
• The sentence does not contain first-person personal or possessive pronouns
87

88. Generation
Text Summarization Algorithm
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Input: item i, sentences s1…sn, word limit k
Output: summary for item i consisting of k words
1. Build a vector space representation for each sentence
2. Merge all the sentences in a pseudo-document that represents the
item (centroid vector)
3. Until the word limit is not reached
3.1 Go through the sentences and calculate cosine similarity
3.2 Pick the one with the highest cosine similarity to the
centroid (and not that similar to the sentences previously
picked)
3.3 Add it to the summary
88

89. Generation
Text Summarization Algorithm
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Input: item i, sentences s1…sn, word limit k
Output: summary for item i consisting of k words
1. Build a vector space representation for each sentence
2. Merge all the sentences in a pseudo-document that represents the
item (centroid vector)
3. Until the word limit is not reached
3.1 Go through the sentences and calculate cosine similarity
3.2 Pick the one with the highest cosine similarity to the
centroid (and not that similar to the sentences previously
picked)
3.3 Add it to the summary
89

90. Generation
Text Summarization Algorithm
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Input: item i, sentences s1…sn, word limit k
Output: summary for item i consisting of k words
1. Build a vector space representation for each sentence
2. Merge all the sentences in a pseudo-document that represents the
item (centroid vector)
3. Until the word limit is not reached
3.1 Go through the sentences and calculate cosine similarity
3.2 Pick the one with the highest cosine similarity to the
centroid (and not that similar to the sentences previously
picked)
3.3 Add it to the summary
90

91. Generation
Text Summarization Algorithm
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Input: item i, sentences s1…sn, word limit k
Output: summary for item i consisting of k words
1. Build a vector space representation for each sentence
2. Merge all the sentences in a pseudo-document that represents the
item (centroid vector)
3. Until the word limit is not reached
3.1 Go through the sentences and calculate cosine similarity
3.2 Pick the one with the highest cosine similarity to the
centroid (and not that similar to the sentences previously
picked)
3.3 Add it to the summary
91

92. Generation
Text Summarization Algorithm
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Input: item i, sentences s1…sn, word limit k
Output: summary for item i consisting of k words
1. Build a vector space representation for each sentence
2. Merge all the sentences in a pseudo-document that represents the
item (centroid vector)
3. Until the word limit is not reached
3.1 Go through the sentences and calculate cosine similarity
3.2 Pick the one with the highest cosine similarity to the
centroid (and not that similar to the sentences previously
picked)
3.3 Add it to the summary
92

93. Generation – Final Output
“If you like or love the blood and gore kinds of films,
this movie will certainly disappoint you as the focus is
on character, story, mood and unique special effects.
The Ring is a story about supernatural evil therefore,
it is a horror film, done very much in the style of the
psychological thriller.”
Legenda
Red: aspects (k=4)
Black: compliant excerpts
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 93

94. Experimental Evaluation
Research Question 1
How effective are the justifications generated through the pipeline, on varying of different
combinations of the parameters?
Research Question 2
How does our justifications perform with respect to a simple review-based explanation?
Experimental Design
User Study with a Web Application
141 subjects
Movie Domain. 300 movies. ~150k reviews.
Metrics: Transparency, Engagement, Persuasion, Trust, Effectiveness [^]
Parameters: Justification Length (Short=50 words, Long=100) and #Aspects (10 and 30).
Between-subjects for Research Question 1, Within-subjects for Research Question 2
[^] Tintarev, N., & Masthoff, J. Designing and evaluating
explanations for recommender systems. In Recommender
systems handbook. pp. 479-510. Springer, Boston, MA. 2011
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 94

95. Results (Research Question 1)
MOVIES Transparency Persuasion Engagement Trust Effectiveness
Top-10 Short 2.83 3.06 3.06 2.83 0.89
Top-30 Long 3.16 3.06 2.69 3.19 0.94
Top-10 Short 3.95 3.64 3.37 3.55 0.55
Top-30 Long 3.24 3.18 3.12 3.22 0.38
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
Finding 1
Long justifications better
than short justifications,
on average
Finding 2
Top-10 aspect provide
better explanations than
Top-30 aspects
Finding 3
Long explanations based
on Top-10 aspects lead to
the best results
95

96. Results (Research Question 2)
MOVIES
Review+
Summary
Review-
based
Indiffer.
Transparency 54.5% 40.9% 4.6%
Persuasion 77.3% 13.6% 9.1%
Engagement 63.6% 27.3% 9.1%
Trust 68.2% 4.5% 27.3%
Outcome: automatic Text Summarization provides users with the best explanation
Confirmed for all the metrics and both the domains
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 96

97. Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 97
Recap and Take Home Messages

98. Recap
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
“If you like or love the blood and gore kinds of films,
this movie will certainly disappoint you as the focus
is on character, story, mood and unique special
effects. The Ring is a story about supernatural evil
therefore, it is a horror film, done very much in the
style of the psychological thriller.”
I recommend you The Ring because people who
liked the movie think that it delivers some bone-
chilling terror. Moreover, people liked The Ring since
the casting is pretty good.
I suggest The Ring because you
often like movies with Naomi Watts as 21 grams
and Mulholland Drive. Furthermore, you like films
about ghosts such as The Sixth Sense.
Feature-based
explanation
exploiting DBpedia
Review-based
explanation using
sentiment analysis
Review-based
explanation using
automatic text
summarization
98

99. Take-home Messages
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019
99
1.
2.
All the methodologies can provide the users with satisfying
explanations, that can support the suggestions returned by a
generic recommendation algorithm
How to choose the most suitable one?
Available data and explanation aims have to drive the choice!
Feature-based: easier approach, good transparency;
Review-based: improves the persuasion and the engagement;
Summarization-based: more sophisticated generation, good for
long-term usage of the explanation facilities.

100. Thank you!
cataldo.musto@uniba.it
@cataldomusto
Contacts
Cataldo Musto. Advances in Content-based Recommender Systems – Explanation Strategies.
ACM Summer School on Recommender Systems 2019, Gothenburg (Sweden), September 13 2019 100