Spark Summit East Talk

1. BUILDING A
RECOMMENDATION ENGINE
USING DIVERSE FEATURES
Divyanshu Vats
Sailthru

2. Joint work with
Max Sperlich
(Sailthru)
David Glueck
(Sailthru)
Alex Gaudio
(Alluvium)
Jeremy Stanley
(Instacart)

3. Customer Retention Cloud

5. users
items

6. users
items
Buy?

7. users
items
Buy?
Matrix Factorization
model = ALS.trainImplicit(data)
model.predict(user_id, item_id)

8. users
items
Buy?
Matrix Factorization
model = ALS.trainImplicit(data)
model.predict(user_id, item_id)
Other Methods
- Item-Item Similarity
- Content Based
- Nearest neighbor based methods

9. users
items
Buy?
Matrix Factorization
model = ALS.trainImplicit(data)
model.predict(user_id, item_id)
Other Methods
- Item-Item Similarity
- Content Based
- Nearest neighbor based methods
Open source scalable
implementations
Easily Incorporate
additional features?

10. Additional Features?

11. Additional Features?
title
image
description
item
location
browser
# clicks
user

12. Additional Features?
title
image
description
item
location
browser
# clicks
user
user-item
- purchases
- item views

13. Goal: Framework to easily integrate user,
item, and user-item interactions for
meaningful recommendations

14. prediction interval response

15. prediction interval response
1features
0

16. prediction interval response
user item user-item interactions user-item features
1
0

17. prediction interval response
Can use a combination of
collaborative filtering algorithms!
user item
1
0
user-item interactions user-item features

18. So far….
users
items
Buy?
- Easily integrate user, item, and user-item features
- Recommendation → Binary classification
- Next: Implementation and use of Spark!

19. Structured Tables:
e.g. url views,
purchases

20. Structured Tables:
e.g. url views,
purchases
item-item similarity
filtering table
filtering table
filtering table
filtering table

21. Structured Tables:
e.g. url views,
purchases
filtering table
filtering table
filtering table
filtering table
Extract features:
user, item,
user-item features
Model Building,
Validation, Reporting
item-item similarity
Recommendations

22. Know Thy Partitions!
data.select(‘user_id’, ‘url’)
data.groupBy(‘user_id’)
data.count()
data.filter(count > 10)

23. Know Thy Partitions!
data.select(‘user_id’, ‘url’)
data.groupBy(‘user_id’)
data.count()
data.filter(count > 10)
Inefficient!

24. Know Thy Partitions!
partition by user_id
data.select(‘user_id’, ‘url’)
data.groupBy(‘user_id’)
data.count()
data.filter(count > 10)

25. Know Thy Partitions!
data.select(‘user_id’, ‘url’)
data.mapPartitions(filter_url)
da
partition by user_id

26. Break up application into small chunks!
users = data.select(‘user_id’, ‘url’)
.mapPartitions(filter_url)
.collect()
BigData.filter(lambda x: x.user_id in users)

27. Break up application into small chunks!
users = data.select(‘user_id’, ‘url’)
.mapPartitions(filter_url)
.collect()
BigData.filter(lambda x: x.user_id in users)

28. Break up application into small chunks!
users = data.select(‘user_id’, ‘url’)
.mapPartitions(filter_url)
.collect()
BigData.filter(lambda x: x.user_id in users)

29. Mesos + Scheduler + Docker + Spark
A
B
C D E F
- Carefully define applications and
state a dependency graph
- Manage graph using:
github.com/sailthru/stolos

30. Mesos + Scheduler + Docker + Spark
A
B
C D E F
- Carefully define applications and
state a dependency graph
- Manage graph using:
github.com/sailthru/stolos

31. Item-Item Similarities
users
items
P(Buy) = 0.002

32. users
items
P(Buy) = 0.002
500 items
1000 items
> 5000 items
Item-Item Similarities

33. users
items
P(Buy) = 0.002
RowMatrix.computeSimilarites
item-item similarities
Item-Item Similarities

34. users
items
P(Buy) = 0.002
RowMatrix.computeSimilarites
item-item similarities
filter and score items for
each user
Item-Item Similarities

35. cluster of
users and
items

36. To Summarize..
users
items
Buy?
- Recommendations using diverse features
- Carefully designed spark apps connected using a graph
- Future work:
- More spark apps + spark-mesos cluster manager

37. THANK YOU.
email: dvats@sailthru.com