The document discusses collaborative filtering and neural network approaches for recommender systems. It introduces collaborative filtering techniques like matrix factorization that aim to predict missing ratings. It then describes applying neural networks to learn nonlinear embeddings of users and items to perform collaborative filtering. The neural network model takes sparse ratings as input, encodes them into a dense representation, then reconstructs the input. It can be trained with both prediction and reconstruction objectives. The model achieves state-of-the-art performance on movie recommendation datasets. Extensions discussed include adding external metadata and using the model for other domains like images.

1. Inria, SequeL
1
Inria, Sequel – Meetup Juin 2016
Deep Learning
Collaborative
Filtering
and

2. Inria, SequeL
2
Collaborative filtering
Bob

3. Inria, SequeL
3
Bob
Collaborative filtering
Tim

4. Inria, SequeL
4
Bob
Tim
Collaborative filtering

5. Inria, SequeL
5
Collaborative filtering
Bob
Tim

6. Inria, SequeL
6
Collaborative filtering
Bob
Tim

7. Inria, SequeL
7
Collaborative filtering
Bob
Tim

8. Inria, SequeL
8
Goal is to predict the rating Tim
would have given to Jurassic Park
Collaborative filtering
Tim

9. Inria, SequeL
9
Collaborative filtering

10. Inria, SequeL
10
Goal : Predict the missing rating
Collaborative filtering
Bob ? 3 5 ?
Ana ? 4 ? 5
Elain 5 ? 4 ?
Sulman 2 ? 1 4

11. Inria, SequeL
11
Goal : Predict the missing rating
Bob 4 3 5 1
Ana 2 4 2 5
Elain 5 4 4 3
Sulman 2 3 1 4
Collaborative filtering

12. Inria, SequeL
12
Goal : Predict the missing rating
Bob 4 3 5 1
Ana 2 4 2 5
Elain 5 4 4 3
Sulman 2 3 1 4
To be recommended
Collaborative filtering

13. Inria, SequeL
13
Bob ? 3 5 ?
Ana ? 4 ? 5
Elain 5 ? 4 ?
Sulman 2 ? 1 4
Score: Average|Predicted rating – real rating|
Collaborative filtering

14. Inria, SequeL
14
Bob ? 3 5 ?
Ana ? 4 ? 5
Elain 5 ? 4 ?
Sulman 2 ? 1 4
Score = Average |Predicted rating – real rating|2
Collaborative filtering

15. Inria, SequeL
15
Neural Networks

16. Inria, SequeL
16
Input
Neural Networks

17. Inria, SequeL
17
Forward
Backward
The model is updated to fit the
target
The error is backpropagated into
the model
Neural Networks

18. Inria, SequeL
18
Neural Networks

19. Inria, SequeL
19
RUsers
Items v
u
Neural Networks

20. Inria, SequeL
20
RUsers
Items v
u
Neural Networks

21. Inria, SequeL
21
RUsers
Items v
u
Neural Networks

22. Inria, SequeL
22
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
𝑅
user
𝑅
i
t
e
m
U-CFN V-CFN
Neural Networks

23. Inria, SequeL
23
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
inputs
Dimension
reduction
Input reconstruction
5 3 4 2 1
4.9 2.8 4 .1 2.0 1.1
The model

24. Inria, SequeL
24
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
3 4 1
2.8 4 .1 1.1
Training stepTesting step
The model

25. Inria, SequeL
25
1 ? 2 ? 1
1.2 4.9 1.1
1 0 2 0 1
1.2 2.1 4.9 2.0 1.1
Forward
The model

26. Inria, SequeL
26
1 ? 2 ? 1
1.2 4.9 1.1
1 0 2 0 1
0.2 0 -0.1 0 0.1
Backward
The model

27. Inria, SequeL
27
β*0.2 0 α*1.1 0 β*1.1
1 0 2 0 1
1 0 0 0 1
Forward
1.2 2.3 3.1 4.2 1.1
Backward
1 ? 2 ? 1
True input
0.2 0 1.1 0 0.1
error 0.2 ? 1.1 ? 0.1
α = Prediction = supervized
β = Reconstruction = unsupervized
The model
1 ? 2 ? 1

28. Inria, SequeL
28
β*0.2 0 α*1.1 0 β*1.1
1 0 2 0 1
1 0 0 0 1
Forward
1.2 2.3 3.1 4.2 1.1
Backward
1 ? 2 ? 1
True input
0.2 0 1.1 0 0.1
error 0.2 ? 1.1 ? 0.1
α = Prediction = supervized
β = Reconstruction = unsupervized
The model
1 ? 2 ? 1

29. Inria, SequeL
29
The model
300.000 to 50.000 entries
Real
life

30. Inria, SequeL
30
MovieLens :
- 10 million ratings : 72.000 users / 10.000 movies
- 20 million ratings : 138.000 users / 28.000 movies
Results

31. Inria, SequeL
31
Quadratic error
MovieLens-10M MovieLens-20M
BPMF 0.8213 0.8123
ALS-WR 0.7830 0.7746
LLORMA 0.7949 0.7843
U-CFN 0.7767 0.7663
V-CFN 0.7767 0.7663
RMSE for train/test 90/10
BPMF : rank=10
ALS-WE : rank = 200
LLORMA : rank = 20, anchor point = 30
A. Mnih and R. Salakhutdinov, “Probabilistic matrix factorization,” in Advances in neural information processing systems, 2007, pp. 1257– 1264.
B. J. Lee, S. Kim, G. Lebanon, and Y. Singerm, “Local low-rank matrix approximation,” in Proc. of ICML’13, 2013, pp. 82–90.
C. Y.Zhou,D.Wilkinson,R.Schreiber,andR.Pan,“Large-scaleparallel collaborative filtering for the netflix prize,” in Algorithmic Aspects in Information and Management. Springer, 2008, pp. 337–348.
Results

32. Inria, SequeL
32
Singular value decomposition (SVD)
Other algorithms:
●Alternating Least Square Weighted Lambda-Regularization (ALS-WR)
●Probabilistic Matrix Factorization (PMF, BPMF, NLPMF)
●Local Low Rank Matrix Approximation (LLORMA)
RUsers
Items v
u
Link with matrix factorization (optional)

33. Inria, SequeL
33
RUsers
Items v
u
z = W3 𝐚𝐜𝐭𝐢𝐯𝐚𝐭𝐢𝐨𝐧2
ri = Vui
activation
Link with matrix factorization (optional)

34. Inria, SequeL
34
ui
V
Link with matrix factorization (optional)
activation
ri = Vui
RUsers
Items v
u
CFN computes a Non-Linear
Matrix Factorization

35. Inria, SequeL
35
Discussion
Why nobody tried it? Image/sounds
are dense
Few works
about sparsity
No tools!

36. Inria, SequeL
36
Discussion
Torch framework
Reproduce results

37. Inria, SequeL
37
? 3 4 ? 1
4.9 2.8 4 .1 2.0 1.1
Actor :
Dicaprio
Genre:
action
Year:
2010
Adding external information

38. Inria, SequeL
38
4.9 2.8 4 .1 2.0 1.1
Content-based Filtering
Extensions

39. Inria, SequeL
39
Thank you for listening!

40. Inria, SequeL
40
Input reconstruction
5 ? 4 ? 1
4.9 2.8 4 .1 2.0 1.1
Encoder
Decoder
Autoencoders:
Matrix factorization
z = ℎ Wui,𝐝𝐞𝐧𝐬𝐞
ui,𝐬𝐩𝐚𝐫𝐬𝐞
ui,𝐝𝐞𝐧𝐬𝐞
Link with Matrix Factorization