Sparking Science up with Research Recommendations
•
8 likes•2,707 views
Mendeley builds tools to help researchers, including a personalized article recommender called Mendeley Suggest. Mendeley Suggest uses various recommender system algorithms like collaborative filtering and matrix factorization to provide article recommendations. It has implemented these algorithms on different platforms like Apache Mahout, Spark, and MLlib to evaluate performance. Testing on their dataset of 15 million documents and 1 million users showed that a tuned user-based collaborative filtering approach on Spark performed best in terms of quality and cost compared to other algorithms and platforms. Mendeley aims to continue improving Mendeley Suggest by exploring new algorithms and platforms to provide high quality recommendations efficiently.
Recommended
More Related Content
Viewers also liked
Viewers also liked (17)
Similar to Sparking Science up with Research Recommendations
Similar to Sparking Science up with Research Recommendations (20)
Recently uploaded
Recently uploaded (20)
Sparking Science up with Research Recommendations
- 1. Sparking Science up with Research Recommendations Maya Hristakeva @mayahhf
- 2. Overview • What is Mendeley Suggest? • Computation Layer • Conclusions
- 3. Read & Organize Search & Discover Collaborate & Network Experiment & Synthesize Mendeley builds tools to help researchers …
- 4. Being the best researcher you can be! • Good researchers are on top of their game • Large amount of research produced • Takes time to get what you need • Help researchers by recommending relevant research
- 6. Recommender System Components information flow (components often built in parallel) Data (Feature Engineering) Algorithms Business Logic and Analytics User Experience
- 7. Mendeley Suggest Components (Past) information flow (components often built in parallel) Data (Feature Engineering) Algorithms Business Logic and Analytics User Experience
- 8. Mendeley Suggest Components (Present) information flow (components often built in parallel) Data (Feature Engineering) Algorithms Business Logic and Analytics User Experience
- 9. Mendeley Suggest Components (Goal) information flow (components often built in parallel) Data (Feature Engineering) Algorithms Business Logic and Analytics User Experience
- 10. Overview • What is Mendeley Suggest? • Computation Layer – Algorithms – Evaluation – Implementations & Performance • Conclusions
- 11. Personalized Article Recommendations Input: User libraries Output: Suggested articles to read Algorithms: • Collaborative Filtering – Item-based – User-Based – Matrix Factorization • Content-based
- 12. Item-based Collaborative Filtering Recommend articles that are similar to the ones you read – Similarity is based on article co-occurrences in users’ libraries – “Users who read x also read y”
- 13. User-based Collaborative Filtering Find users who have similar appreciation for articles as you – Similarity is based on users’ libraries overlap Recommend new articles based on what the users similar to you read – “Users similar to you (based on a, b, c) also read x”
- 14. Matrix Factorization CF 2 4 5 5 4 1 5 ? 2 1 5 4 4 2 4 5 1 U n x k V k x m fij= <Ui*,V*j> E(U,V) = L(Xij, fij) + R(U,V) X n x m
- 15. Overview • What is Mendeley Suggest? • Computation Layer – Algorithms – Evaluation – Implementations • Conclusions
- 16. Performance Costly & GoodCostly & Bad Cheap & GoodCheap & Bad
- 17. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Performance
- 18. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Performance
- 19. How to measure quality? • Offline Evaluation – Parameter sweep is quick – Don’t offend real users • Methodology – n-fold cross-validation – time-based validation • Metrics – precision, recall and f-measure – AUC (area under roc curve), NDCG (normalized discounted cumulative gain)
- 20. Overview • What is Mendeley Suggest? • Computation Layer – Algorithms – Evaluation – Implementations • Conclusions
- 22. Setup • EMR Cluster – Master: 1 x r3.xlarge instance (4 core, 32GB) – Core: 10 x r3.2xlarge instances (8 core, 64GB) • Data: user libraries – 15mil documents >>> 1mil users – 150mil interactions • Offline Evaluation – Methodology: time-based evaluation – Metric: precision@10
- 24. Apache Mahout • Mahout (out-of-the-box) – Item-based CF • org.apache.mahout.cf.taste.hadoop.item.RecommenderJob – ALS Matrix Factorization • org.apache.mahout.cf.taste.hadoop.als.ParallelALSFactorizationJob • org.apache.mahout.cf.taste.hadoop.als.RecommenderJob • Implemented User-based CF on top of Mahout at Mendeley
- 25. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Orig. item-based mahout Tuned item-based mahout -0.5K (-60%) Performance ~$125
- 26. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Orig. item-based mahout Tuned item-based mahout -0.5K (-60%) Orig. user-based mahout Tuned user-based mahout -0.1K (-40%) Performance ~$125
- 27. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Orig. item-based mahout Tuned item-based mahout Orig. user-based mahout Tuned user-based mahout +150% -0.2K (-55%) -0.7K (-82%) Performance ~$125
- 28. Mahout Performance • Mahout’s recommender is already efficient – But your data may have unusual properties • We’ve got improvements by – Tuning Hadoop’s mapper and reducer allocation over the Recommender Job steps – Using an appropriate partitioner • Improve quality – Mahout provides Item-based CF – We have many more items than users – Typically, user-based is more appropriate
- 30. Mahout Spark • Co-occurrence Recommenders with Spark – Item-Item similarity • mahout spark-itemsimilarity SimilarityAnalysis.cooccurrencesIDSs(ratings, …) – User-User similarity • mahout spark-rowsimilarity SimilarityAnalysis.rowSimilarityIDSs(ratings, …) • Only supports Boolean data and log-likelihood similarity • Does not generate actual recommendations
- 31. Mahout Spark • Could not get to run successfully on our data • Got further by tuning parameters but still failed with OOM – spark.driver.maxResultSize – spark.kryoserializer.buffer.max – spark.default.parallelism – spark.storage.memoryFraction • Gave best runtime performance on MovieLens datasets
- 33. Mendeley Spark • Started as hack-day project – Implement Item-based and User-based CF in Spark • Can be implemented in two steps 1. Compute Item-Item or User-User Similarities • given user preferences 2. Compute Recommendations • given similarities and user preferences
- 40. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Tuned IB Mahout Tuned UB Mahout Orig. UB Spark Performance ~$50
- 41. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Tuned IB Mahout Tuned UB Mahout Orig. UB Spark Tuned UB Spark Tuned IB Spark -0.1K (-40%) Performance ~$50
- 42. Mendeley Spark Performance • Spark implementation of User-based CF performs well • Managed to run variation of Item-based CF – Uses fewer items per user to recommend similar items to – Quality not impacted much • We’ve got improvements by tuning – Resource allocation – Parallelism – http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs- part-2/
- 44. Spark MLlib DimSum • DimSum: efficient algorithm for computing all-pairs similarity – “Dimension Independent Matrix Square using MapReduce” – Contributed by Twitter • Replace similarity computation with DimSum – Only supports cosine similarity • Does not generate actual recommendations – Compute recommendations as before
- 45. MLlib DimSum Item-Item Similarity
- 46. MLlib DimSum User-User Similarity
- 47. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Tuned IB Mahout Tuned UB Mahout Tuned UB Spark Tuned IB Spark UB DimSum Spark MLlib Performance ~$50
- 48. Spark MLlib Matrix Factorization Implements alternating least squares (ALS) 1. Compute Model 2. Compute Recommendations
- 49. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Tuned IB Mahout Tuned UB Mahout Tuned UB Spark Tuned IB Spark UB DimSum Spark MLlib ALS Matrix Fact. Spark MLlib -50% Performance ~$50
- 50. MLlib Performance • Provides good alternative for computing user-user similarities – Due to data sparsity, not getting big gains in runtime – Only supports cosine similarity • Failed to compute item-item similarities – Exceeds maximum allowed value of 2G for spark.kryoserializer.buffer.max • User-based CF outperforms ALS CF • Need scalable solution for generating recommendations based on ALS CF model
- 52. Overview • What is Mendeley Suggest? • Computation Layer • Conclusions
- 53. Costly & GoodCostly & Bad Cheap & GoodCheap & Bad Tuned IB Mahout Tuned UB Mahout Tuned UB Spark Tuned IB Spark UB DimSum Spark MLlib ALS Matrix Fact. Spark MLlib Performance +100% +150% ~$50
- 54. Mendeley Suggest Components (Future) information flow (components often built in parallel) Data (Feature Engineering) Algorithms Business Logic and Analytics User Experience
- 55. Conclusions • Mendeley Suggest is a personalized article recommender • Spark is good alternative to Mahout as computation layer – Needs some love and tuning – Much fewer lines of code – easier to maintain and extend • User-based can outperform item-based and matrix factorization • Save resources and money by understanding your data • Test offline before deploying – but also need online tests to get real performance