3rd Meetup- Recommendation, Containers and Resource Management at Scale

1. “Streaming Simplified”
Recommendation Engine
using Spark's MLlib
Manish Kaushal
Software Engineer

2. What is Recommendation Engine?

3. Content Based Recommendation System
Focus on properties of items.
Similarity of items is determined by measuring the similarity
in their properties.
For example in case of movies properties are genre, year of
release etc.

4. Collaborative Filtering
Focus on the relationship between users and items.
Similarity is determined by the similarity of the ratings of those
items by the users who have rated both items.

5. Recommendations

6. Recommendations

7. Recommendations

8. Alternating Least Square

9. Alternating Least Square
Ratings Matrix is factorized into UserFactors and ItemFactors for
given rank.
Starts with random UserFactor and is optimized keeping
ItemFactor constant.
In next step ItemFactor is optimized keeping UserFactor constant.
Iterate above two steps.

10. Spark's Implementation
Block wise parallel :Users/products are partitioned into blocks
and join is based on blocks instead of individual user/product.
In-line Blocks
Out-line Blocks
In memory computation.

11. Out-Links
Out-links of each item is which blocks of user it will contribute to.

12. Out-Links

13. In-Links
In-link for each user is which of the feature vectors it receives
from each product block it will depend on.

14. In-Links

15. Benchmarks
The benchmark was
conducted on EC2 using
m3.2xlarge instances.
Spark 1.0.1
https://github.com/databrick
s/als-benchmark-scripts

16. Thank You