This document outlines an item-based collaborative filtering recommendation algorithm that has been scaled up to run on Hadoop. It first discusses collaborative filtering techniques and how they work. It then describes scaling up the item-based collaborative filtering approach by dividing it into two steps: similarity computation and prediction/recommendation. The key computations involve calculating average item ratings, similarity between item pairs, and predicted ratings for target users. An experiment tested the scaled approach on a Hadoop cluster with 3 nodes.

1. 2015/2/27
Scaling-up Item-based Collaborative Filtering
Recommendation Algorithm based on Hadoop
Jing Jiang, Jie Lu, Guangquan Zhang, Guodong Long 2011 IEEE World Congress Services

2. outline
✤ Collaborative Filtering
✤ scaling-up item-based CF
✤ experimentation and evaluation

3. Collaborative Filtering
✤ Collaborative filtering (CF) techniques have achieved
widespread success in E-commerce nowadays.

4. Collaborative Filtering
✤ Collaborative filtering is a method of making
automatic predictions (filtering) about the interests of
a user by collecting preferences or taste information
from many users (collaborating). from wiki

5. Collaborative Filtering
1. Weight all users with respect to similarity with active user
2. Select a subset of users to use as a set of predictors
3. Compute a prediction from a weighted combination of selected
neighbors’ ratings

6. 1. Weight all users with respect to similarity with active user
2. Select a subset of users to use as a set of predictors
3. Compute a prediction from a weighted combination of selected
neighbors’ ratings
simple
compute
Nathan [5,1,5]
Joe [5,2,5]
John [2,5,2.5]
Al [2,2,4]
use cosine compute similarity
cos (Nathan,Joe) 0.99
cos (Nathan,John) 0.64
cos (Nathan,Al) 0.91

7. 1. Weight all users with respect to similarity with active user
2. Select a subset of users to use as a set of predictors
3. Compute a prediction from a weighted combination of selected
neighbors’ ratings
simple
compute
cos (Nathan,Joe) 0.99
cos (Nathan,John) 0.64
cos (Nathan,Al) 0.91
(0.99*4+0.64*3+0.91*2)/(0.99+0.64+0.91) = 3.03
0.99
0.91
0.64
? = 3.03

8. Collaborative Filtering
✤ User-Based CF
✤ Item-Based CF
compute similarity base on user
compute similarity base on item

9. Collaborative Filtering
✤ User-Based CF
compute similarity base on user
if predict user A to item4 rating
user B to item4 rating is 5
user F to item4 rating is 1
user A to item4 =
5 * similarities (user A, user B) + 1 * similarities (user A, user F)
similarities (user A, user B) + similarities (user A, user F)

10. Collaborative Filtering
✤ Item-Based CF
compute similarity base on item
if predict user A to item4 rating
user A to item2 rating is 1
user A to item3 rating is 1
user A to item4 =
1 * similarities (item2, item4) + 1 * similarities (item3, item4)
similarities (item2, item4) + similarities (item3, item4)

11. scaling-up item-based CF
divide CF algorithm into two steps as follows:
Similarity computation
Prediction and Recommendation
pearson correlation(1,-1)
j

12. scaling-up item-based CF
pearson correlation(1,-1)
j
Covariance

13. scaling-up item-based CF
Similarity computation
apple milk toast
sam 2 0 4
john 5 5 3
tim 2 4 ?
u
i
j
j
Ri = (2+5+2)/3 Rj = (4+3)/2

14. scaling-up item-based CF
Similarity computation
apple milk toast
sam 2 0 4
john 5 5 3
tim 2 4 ?
u
j
i
Ru(sam) = (2+0+4)/3
Rj = (2+5+2)/3 Ri = (4+3)/2

15. scaling-up item-based CF
The three parts of intensive computation are:
(1)computing the average rating for each item
(2)computing the similarity between item pairs
(3)computing predicted items for the target user

16. item iby user j
map item i
1 2 3

17. 1
wheremeans the
set of users who rated the item kand item l

18. 2
similarity
3
map user j
map user j

19. experimentation and evaluation
3 nodes
nodes with Intel P4 CPU,
1G RAM, 80G disk
All the machines were connected
with one 100Mbps switch.

20. experimentation and evaluation
13
20