This document reviews probabilistic matrix factorization, topic modeling, collaborative topic modeling, and collaborative deep learning approaches. It describes the generative processes and graphical models of probabilistic matrix factorization, latent Dirichlet allocation, collaborative topic modeling, and collaborative deep learning. It also compares collaborative topic modeling and collaborative deep learning methods and discusses potential extensions, including using deep learning models instead of LDA to more deeply extract features from documents.

1. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Review: Collaborative Deep Learning
Hai D. Nguyen
May 5, 2016
Hai D. Nguyen Review: Collaborative Deep Learning

2. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Table of contents
1 Review of Probabilistic Matrix Factorization
2 Topic Modeling and Collaborative Topic Modeling
3 Collaborative Deep Learning
Hai D. Nguyen Review: Collaborative Deep Learning

3. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Probabilistic Matrix Factorization (PMF)
T
A GENERATIVE process:
Pick User Factor Ui
Ui ∼ N(0, σ2
u)
Pick Item Factor Vj
Vj ∼ N(0, σ2
v )
For each (User, Item) pair, pick Rij
Rij ∼ N(UT
i Vj , σ2
r )
Joint Probability:
P(U, V , R) = P(U)P(V )P(R|U, V )
Graphical Model
Hai D. Nguyen Review: Collaborative Deep Learning

4. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
MAP is equivalent to Regularized Least Square
MAP under gaussian distribution
U∗
, V ∗
= argmax
U,V
P(U, V |R)
= argmax
U,V
P(U)P(V )P(R|U, V )
= argmax
U,V
logP(U) + logP(V ) + logP(R|U, V )
= argmin
U,V
1
2 ruv
(UT
V − Ruv )2
+
λu
2
||U||2
2 +
λv
2
||V ||2
2
Hai D. Nguyen Review: Collaborative Deep Learning

5. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Latent Dirichlet Allocation
A GENERATIVE process
LDA assumes the following generative process for each
document w in a corpus D
1 For k = 1...K:
(a) φ(k)
∼ Dirichlet(β)
2 For each document d ∈ D :
(a) θd ∼ Dirichlet(α)
(b) For each word wi ∈ d :
i. zi ∼ Mult(θd )
ii. wi ∼ Mult(φ(zi )
)
Make use of Gibb Sampling or Variational Inference to train
the model
Hai D. Nguyen Review: Collaborative Deep Learning

6. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Collaborative Topic Models
Combination of LDA and PMF
In PMF, each item has a latent representation V in some
unknown latent space
In Topic modelling (LDA), each item (article) has topic
proportions θ in the learned topic space
Basic idea: Fix V = θ
Hai D. Nguyen Review: Collaborative Deep Learning

7. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Collaborative Topic Modeling: Graphical Model and
Generative Process
The generative process for CTP is defined as follow:
Hai D. Nguyen Review: Collaborative Deep Learning

8. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Collaborative Deep Learning
Same idea as CTP model previously mentioned but different
methods (Deep Learning)
Replace Topic Model (LDA here) with a certain DEEP
LEARNING model which can be DBF, SDAE or CNN.
Hai D. Nguyen Review: Collaborative Deep Learning

9. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Collaborative Deep Learning: Graphical Model and
Generative Process
The generative process for CDL is defined as follow:
Hai D. Nguyen Review: Collaborative Deep Learning

10. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Collaborative Deep Learning: Learning
Hai D. Nguyen Review: Collaborative Deep Learning

11. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Comparison between two methods: CTM vs CDL
CTM method
CDL method
Hai D. Nguyen Review: Collaborative Deep Learning

12. Review of Probabilistic Matrix Factorization
Topic Modeling and Collaborative Topic Modeling
Collaborative Deep Learning
Some thoughs and extensions
it canbe extended to multi-dimensional recommendation
system (e.g., context-based RS) with Tensor approaches
In my opinion, LDA can be viewed as a shalow model. In this
paper, authors proposed using DL which can automatically
and deeply extract features for document (features maybe
more effective than that of LDA)
How can we extend LDA to deeper model?
Hai D. Nguyen Review: Collaborative Deep Learning