Presentation at UMAP 2017. 25th International Conference on User Modeling Adaptation and Personalization. Bratislava, Slovakia, July 2017

1. @cataldomusto @ale_suglia
@cld_greco @SWAP_research
A Deep Architecture for
Content-based Recommendations
Exploiting Recurrent Neural Networks
ALESSANDRO SUGLIA, CLAUDIO GRECO, CATALDO MUSTO, MARCO DE GEMMIS, PASQUALE
LOPS, GIOVANNI SEMERARO
UNIVERSITÀ DEGLI STUDI DI BARI ‘ALDO MORO’ - ITALY
25th International Conference on User
Modeling, Adaptation and Personalization
Bratislava, Slovakia
July 12, 2017
cataldo.musto@uniba.it

2. Recurrent Neural Networks (RNNs)
Widespread Deep Learning Architecture
◦ Based on Neural Networks
◦ The connections between the units may contain loops which let consider past states in the
learning process
◦ Very suitable to model variable-length sequential data
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

3. Recurrent Neural Networks (RNNs)
Widespread Deep Learning Architecture
◦ Based on Neural Networks
◦ The connections between the units may contain loops which let consider past states in the
learning process
◦ Very suitable to model variable-length sequential data
PROS CONS
◦ Very good performance in different tasks
◦ Can learn short-term and long-term (temporal) dependencies
◦ Vanishing/exploding gradient problem
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

4. Recurrent Neural Networks (RNNs)
Widespread Deep Learning Architecture
◦ Based on Neural Networks
◦ The connections between the units may contain loops which let consider past states in the
learning process
◦ Very suitable to model variable-length sequential data
PROS CONS
◦ Very good performance in different tasks
◦ Can learn short-term and long-term (temporal) dependencies
◦ Vanishing/exploding gradient problem
LONG-SHORT TERM MEMORY NETWORKS (LSTMS)
◦ Introduced to solve the vanishing/exploding gradient problem
Each cell presents a complex structure which is more powerful than simple RNN cells.
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

5. Motivations
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
?
In content-based recommender systems
suggestions are generated by matching
the features stored in the user profile
with those describing the items to be
recommended

6. Motivations
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
user profile
?
items
In content-based recommender systems
suggestions are generated by matching
the features stored in the user profile
with those describing the items to be
recommended

7. Motivations
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
user profile
?
items
In content-based recommender systems
suggestions are generated by matching
the features stored in the user profile
with those describing the items to be
recommended
Content Representation
plays a key role!

8. Motivations
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
user profile
?
items
In content-based recommender systems
suggestions are generated by matching
the features stored in the user profile
with those describing the items to be
recommended
RNNs are very suitable!
Content can be considered as a
sequence of terms

9. Research Question
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

10. Research Question
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
Our contribution
AMAR (Ask Me Any Rating)
Deep Architecture inspired by a neural
network model used to solve Question
Answering toy tasks [*]
[*] J. Weston et al. “Towards AI-Complete Question
Answering: A Set of Prerequisite Toy Tasks”.
In: CoRR abs/1502.05698 (2015)

11. Research Question
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
Our contribution
AMAR (Ask Me Any Rating)
Deep Architecture inspired by a neural
network model used to solve Question
Answering toy tasks [*]
[*] J. Weston et al. “Towards AI-Complete Question
Answering: A Set of Prerequisite Toy Tasks”.
In: CoRR abs/1502.05698 (2015)
Analogy
Question:Answers = User Profile:Items

12. AMAR: Ask Me Any Rating
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

13. AMAR: Ask Me Any Rating
User and Item are modeled through two embeddings
EMBEDDINGS ARE JOINTLY LEARNED
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

14. AMAR: Ask Me Any Rating
User and Item are modeled through two embeddings
EMBEDDINGS ARE JOINTLY LEARNED
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
Given an item, its textual description w1 , ... ,wn is
represented through a RNN with LSTM cells
Each LSTM generates a latent representation h(wi)
for each word wi
The final representation of the item is obtained
through a MEAN POOLING LAYER

15. AMAR: Ask Me Any Rating
User and Item are modeled through two embeddings
EMBEDDINGS ARE JOINTLY LEARNED
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
The resulting embeddings are merged through a
CONCATENATION LAYER
Given an item, its textual description w1 , ... ,wn is
represented through a RNN with LSTM cells
Each LSTM generates a latent representation h(wi)
for each word wi
The final representation of the item is obtained
through a MEAN POOLING LAYER

16. AMAR: Ask Me Any Rating
User and Item are modeled through two embeddings
EMBEDDINGS ARE JOINTLY LEARNED
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
A LOGISTIC REGRESSION LAYER estimates user interest in
the item and builds the recommendation list.
Given an item, its textual description w1 , ... ,wn is
represented through a RNN with LSTM cells
Each LSTM generates a latent representation h(wi)
for each word wi
The final representation of the item is obtained
through a MEAN POOLING LAYER
The resulting embeddings are merged through a
CONCATENATION LAYER

17. AMAR+
AMAR has a very modular and extensible
architecture
It is possible to add extra modules to encode
more information beyond the simple description
of the item
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

18. AMAR+
AMAR has a very modular and extensible
architecture
It is possible to add extra modules to encode
more information beyond the simple description
of the item
AMAR+ introduces A GENRE EMBEDDING,which
represents the genre associated to the item to
be recommended
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

19. AMAR+
AMAR has a very modular and extensible
architecture
It is possible to add extra modules to encode
more information beyond the simple description
of the item
AMAR+ introduces A GENRE EMBEDDING,which
represents the genre associated to the item to
be recommended
For each genre g1, … , gm associated to an item
a genre embedding is learnt. All the embeddings
are averaged through a MEAN POOLING LAYER.
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

20. AMAR+
AMAR has a very modular and extensible
architecture
It is possible to add extra modules to encode
more information beyond the simple description
of the item
AMAR+ introduces A GENRE EMBEDDING,which
represents the genre associated to the item to
be recommended
For each genre g1, … , gm associated to an item
a genre embedding is learnt. All the embeddings
are averaged through a MEAN POOLING LAYER.
The new information is merged and the pipeline
estimates again the user preference in the item
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

21. Experiments
How does our deep architecture
perform when compared to other
content-based recommender
systems or state-of-the-art
baselines?
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

22. Datasets
MovieLens 1M (ML1M)
6,040 users
3,883 movies
1,000,209 ratings
57.51% positive ratings
165.59 ratings/user (avg.)
269.88 ratings/item (avg.)
99.4% sparsity
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

23. Datasets
DBbook
6,181 users
6,733 movies
72,732 ratings
45.86% positive ratings
11.71 ratings/user (avg.)
10.74 ratings/item (avg.)
99.8% sparsity
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

24. Experimental Settings
Top-N recommendation task
Metric
◦ F1@5
AMAR parameters
◦ RMSprop optimizer, 25 epochs
◦ a=0.9, learning rate 0.001
◦ Batch size 1536 (ML1M) and 512 (DBbook)
◦ Binary cross entropy as cost function
◦ User, Item and Genre embedding size = 10
Item Processing
◦ Mapping item names with Wikipedia pages
◦ Extraction of textual content from plots
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

25. Baselines
Word Embedding techniques
◦ Word2Vec
◦ Glove
◦ Doc2Vec
◦ In Word2Vec and Glove, items/profile are represented
as the centroid vector of the representation of the word
occurring in the textual descriptions
Collaborative Filtering and Matrix Factorization
techniques
U2U-CF, I2I-CF
BPRMF, BPRSlim, WRMF
Optimal parameters. All available in MyMediaLite toolkit
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

26. Baselines
Word Embedding techniques
◦ Word2Vec
◦ Glove
◦ Doc2Vec
◦ In Word2Vec and Glove, items/profile are represented
as the centroid vector of the representation of the word
occurring in the textual descriptions
Collaborative Filtering and Matrix Factorization
techniques
◦ U2U-CF, I2I-CF
◦ BPRMF[*], BPRSlim[+], WRMF
◦ Optimal parameters.
◦ All available in MyMediaLite toolkit
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
[*] S. Rendle, C.Freudenthaler, Z. Gantner, L. Schmidt-Thieme:
BPR: Bayesian Personalized Ranking from Implicit Feedback. UAI 2009.
[+] X. Ning, G. Karypis: Slim: Sparse linear methods for top-n recommender systems. ICDM 2011.

27. Results – MovieLens data
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
0.5550.558
0.49 0.482 0.485
0.427 0.431 0.425 0.423
0.446
MovieLens
AMAR AMAR+ Word2Vec Doc2Vec Glove U2U I2I BPRMF WRMF BPRSlim

28. 0.5550.558
0.49 0.482 0.485
0.427 0.431 0.425 0.423
0.446
MovieLens
AMAR AMAR+ Word2Vec Doc2Vec Glove U2U I2I BPRMF WRMF BPRSlim
Results – MovieLens data
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
Word
Embedding
techniques

29. 0.5550.558
0.49 0.482 0.485
0.427 0.431 0.425 0.423
0.446
MovieLens
AMAR AMAR+ Word2Vec Doc2Vec Glove U2U I2I BPRMF WRMF BPRSlim
Results – MovieLens data
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
Word
Embedding
techniques
Collaborative Filtering and
Matrix Factorization
techniques

30. Results – DBbook data
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
0.5640.565
0.542 0.54
0.552
0.536 0.536
0.508
0.519 0.511
MovieLens
AMAR AMAR+ Word2Vec Doc2Vec Glove U2U I2I BPRMF WRMF BPRSlim

31. 0.5640.565
0.542 0.54
0.552
0.536 0.536
0.508
0.519 0.511
MovieLens
AMAR AMAR+ Word2Vec Doc2Vec Glove U2U I2I BPRMF WRMF BPRSlim
Results – DBbook data
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
AMAR and AMAR+
overcome all the
baselines

32. Recap
AMAR: a deep architecture for content-based recommendation exploiting RNNs
◦ Neural Network predicts the likelihood that a user would like a certain item
◦ User and Item embeddings are jointly learned.
◦ LSTMs to model textual description of the items.
Results
 AMAR and AMAR+ significantly improve all the baselines
 Modular and Extensible Architecture: AMAR+ introduces a genre embedding
 High training time (ML1M=90’ per epoch , DBbook=50’ per epoch)
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017

33. Thanks!
Alessandro Suglia, Claudio Greco, Cataldo Musto, Marco de Gemmis, Pasquale Lops, Giovanni Semeraro.
A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks. UMAP 2017. Bratislava, Slovakia. July 12, 2017
cataldo.musto@uniba.it
@cataldomusto, @ale_suglia
@cld_greco, @SWAP_research