This document discusses personalization using big data and collaborative filtering techniques. It describes how collaborative filtering works by analyzing user preferences for items to predict other items a user may like. Several collaborative filtering algorithms are presented, including item-based algorithms that recommend similar items to those a user has rated highly. Matrix math and Apache Mahout are introduced as ways to implement collaborative filtering at scale using Hadoop. Real-world examples of using these techniques for recommendations are also briefly mentioned.

1. Personalization
with Big Data
21/5/13.
2013
Yossi Cohen
CTO 3Base
(Taldor Group)

2. Who we are & what we do
 Projects in cloud environment
 SAAS, Multi tenant, lots of users
 >20 Big Data projects in various technologies
(NoSQL, Hadoop etc.)
 End to end solution
 Architecture & design
 Implementation
 Maintenance
2

3. You must have seen this…
3

4. 4

5. 5

6. 6

7. 7

8. Some examples
8

9. The biggest one
9

10. Another one
10

11. Lessons
 Users have different tastes
 Users don’t have patience
 We need to show them things they like…
 Approaches
 Modeling
 Content Based
 Collaborative Filtering
11

12. Collaborative Filtering is …
 Given a user’s
preferences for items,
guess which other
items would be highly
preferred
 Only needs
preferences;
users and items
opaque
 Many algorithms!
12

13. Collaborative Filtering is …
Sean likes “Scarface” a lot
Robin likes “Scarface” somewhat
Grant likes “The Notebook” not at all
…
(123,654,5.0)
(789,654,3.0)
(345,876,1.0)
…
(345,654,4.5)
…
(Magic)
Grant may like “Scarface” quite a bit
…
13

14. Recommending people food
14

15. Item-Based Algorithm
 Recommend items similar to a user’s
highly-preferred items
15

16. Item-Based Algorithm
 Have user’s preference for items
 Know all items and can compute weighted
average to estimate user’s preference
 What is the item – item similarity notion?
for every item i that u has no preference for yet
for every item j that u has a preference for
compute a similarity s between i and j
add u's preference for j, weighted by s,
to a running average
return top items, ranked by weighted average
16

17. Item-Item Similarity
 Could be based on content…
 Two foods similar if both sweet, both cold
 BUT: In collaborative filtering, based only on
preferences (numbers)
 Pearson correlation between ratings ?
 Log-likelihood ratio ?
 Simple co-occurrence:
Items similar when appearing often in the same user’s set
of preferences
17

18. As matrix math
 User’s preferences are a vector
 Each dimension corresponds to one item
 Dimension value is the preference value
 Item-item co-occurrences are a matrix
 Row i / column j is count of item i / j co-
occurrence
 Estimating preferences:
co-occurrence matrix × preference (column) vector
18

19. As matrix math
16 9 16 5 6
9 30 19 3 2
16 19 23 5 4
5 3 5 10 20
6 2 4 20 9
16 animals ate both
hot dogs and ice
cream
10 animals ate
blueberries
0
5
5
2
0
135
251
220
60
70
19

20. Hadoop way
20

21. Apache Mahout is …
 Machine learning …
 Collaborative filtering
(recommenders)
 Clustering
 Classification
 Frequent item set mining
 and more
 … at scale
 Much implemented on Hadoop
 Efficient data structures
21

22. Thank you For more information:
Yossi@3base.co.il
www.taldor.co.il