Large-scale Social Recommendation Systems: Challenges and Opportunity
Report
Share
•8 likes•1,883 views
![Job Seeking: Socially Contagious
25
[Zhang, 2012]
• Prob. of quitting increases as the #of recently left connected colleague](https://image.slidesharecdn.com/srskeynote-140817220631-phpapp01/75/largescale-social-recommendation-systems-challenges-and-opportunity-25-2048.jpg?cb=1668329661)
![Acknowledgement
• Thanks to Data Team at LinkedIn: http://data.linkedin.com!
• We are hiring!!
• Contact: mtiwari[at]linkedin.com!
• Follow: @mitultiwari on Twitter
52
You!
Applied Reseacher/
Research Engineer](https://image.slidesharecdn.com/srskeynote-140817220631-phpapp01/75/largescale-social-recommendation-systems-challenges-and-opportunity-52-2048.jpg?cb=1668329661)
Large-scale Social Recommendation Systems: Challenges and Opportunity
•8 likes•1,883 views
Report
Share
Keynote talk at 4th International Workshop on Social Recommender Systems (SRS 2013) In conjunction with 22nd International World Wide Web Conference (WWW 2013). More details: http://cslinux0.comp.hkbu.edu.hk/~fwang/srs2013/
Recommended
More Related Content
What's hot
What's hot(7)
Similar to Large-scale Social Recommendation Systems: Challenges and Opportunity
Similar to Large-scale Social Recommendation Systems: Challenges and Opportunity(20)
More from Mitul Tiwari
More from Mitul Tiwari(6)
Recently uploaded
Recently uploaded(9)
Large-scale Social Recommendation Systems: Challenges and Opportunity
- 1. Large-Scale Social Recommendation Systems: Challenges And Opportunities Mitul Tiwari! ! Search, Network, and Analytics (SNA)! LinkedIn
- 2. Who Am I 2
- 3. Outline • About LinkedIn! • Social Recommender Systems at LinkedIn! • Social Graph Analysis! • Virality in Social Recommender Systems! • Scaling Challenges 3
- 4. Linkedin By The Numbers 4 225M members 2 new members/sec
- 5. Broad Range Of Products 5
- 7. Contacts 7
- 9. Job Search 9
- 10. Company Pages 10
- 11. Outline • About LinkedIn! • Social Recommender Systems at LinkedIn! • Social Graph Analysis! • Virality in Social Recommender Systems! • Scaling Challenges 11
- 12. Linkedin Homepage • Powered by recommendations 12
- 13. Recommender Ecosystem 13 ! Similar Profiles Connections News Skill Endorsements
- 14. Outline • Social Recommender Systems at LinkedIn! • LinkedIn Today: Recommend News! • Jobs Recommendation! • Related Searches Recommendation! • Social Graph Analysis! • Virality in Social Recommender Systems! • Scaling Challenges 14
- 15. Linkedin Today: News Recommendation • Objective: serve valuable professional news, leading to higher engagement as measured by metrics such as CTR 15
- 16. News Recommendation: Explore/Exploit 16 item j from a set of candidates User i with user features (e.g., industry, behavioral features, Demographic features,……) (i, j) : response yijvisits Algorithm selects (click or not) Which item should we select? ! The item with highest predicted CTR ! An item for which we need data to predict its CTR Exploit Explore Agarwal et. al 2012
- 17. News Recommendations: Challenges • Drop in CTR wrt Time 17
- 18. News Recommendation: Challenges • Same item shown to the same users: drop in CTR 18
- 19. News Recommendations: Revised Algorithm • Explore/Exploit scheme! • Explore: choose an item at random with a small probability (e.g., 5%)! • Exploit: choose highest scoring CTR item (e.g., 95%)! • Temporal smoothing: more weight to recent data! • Impression discounting: discount items with repeat views! • Segmented model: segment users in CTR estimation! • Opportunity: Multi-arm bandit problem 19
- 20. Outline 20 • Social Recommender Systems at LinkedIn! • LinkedIn Today: Recommend News! • Jobs Recommendations! • Related Searches Recommendation! • Social Graph Analysis! • Social Update Stream and Virality! • Scaling Challenges
- 21. Jobs Recommendation • Goal: recommend dream jobs to job seekers! • Challenges! • Lag between view and application, offer, acceptance! • High level of expectations 21
- 22. Jobs Recommendation 22 17 Corpus StatsJob User Base Filtered title geo company industry description functional area … Candidate General expertise specialties education headline geo experience Current Position title summary tenure length industry functional area … Similarity (candidate expertise, job description) 0.56 Similarity (candidate specialties, job description) 0.2 Transition probability (candidate industry, job industry) 0.43 Title Similarity 0.8 Similarity (headline, title) 0.7 . . . derived Matching Binary Exact matches: geo, industry, … Soft transition probabilities, similarity, … Text Transition probabilities Connectivity yrs of experience to reach title education needed for this title … Ensemble Scorings Bhasin et. al 2012
- 23. Magic Is In Feature Engineering • Open to relocation?! • Region similarity based on profile or network! • Region transition probability! • Predict members’ propensity to migrate and potential regions 23
- 24. What Should You Transition To And When 24 • Probability of holding a title wrt time: spikes 12 months aligned Wang et. al, WWW’13
- 25. Job Seeking: Socially Contagious 25 [Zhang, 2012] • Prob. of quitting increases as the #of recently left connected colleague
- 26. Outline 26 • Social Recommender Systems at LinkedIn! • LinkedIn Today: Recommend News! • Jobs Recommendation! • Related Searches Recommendation! • Social Graph Analysis! • Social Update Stream and Virality! • Scaling Challenges
- 27. Related Searches Recommendation • Millions of Searches everyday! • Help users to explore and refine their queries 27 Reda et. al, CIKM’12
- 29. Related Searches Recommendations • Signals! • Collaborative Filtering! • Query-Result Click graph! • Overlapping terms! • Length-bias! • Ensemble approach for unified recommendation! • Practical considerations 29
- 30. Related Searches Recommendations • Signals! • Collaborative Filtering! • Query-Result Click graph! • Overlapping terms! • Length-bias! • Ensemble approach for unified recommendation! • Practical considerations! • Opportunity: Build Personalized Search Recommendation 30
- 31. Outline • About LinkedIn! • Social Recommender Systems at LinkedIn! • Social Graph Analysis! • Virality in Social Recommender Systems! • Scaling Challenges 31
- 32. Link Prediction Over Social Graph 32
- 34. Connection Strength • Measure strength of each connection! • Applications! • Introductions! • Update stream relevance 34
- 35. Query-Result Clicks Graph • Application: Related Searches correlated by result clicks for related searches recommendation Q1 Qn R1 Rm 35
- 36. Skills Similarity Graph • Graph of all co-occurrences between LinkedIn Skills 36
- 37. Skills 37
- 38. Find Influencers In Venture Capital? 38
- 39. Outline • About LinkedIn! • Social Recommender Systems at LinkedIn! • Social Graph Analysis! • Virality in Social Recommender Systems! • Scaling Challenges 39
- 42. Viral Growth: 1B Skills Endorsements • One of the fastest growing product in LinkedIn’s history 42
- 43. Skill Tagging • Tagging: extract potential skills from profile using skills taxonomy! ! ! • Standardize skill phrase variants Profile Tokenize SkillsTagger Phrases Skills 43
- 44. Skill Recommendation • Predict a skill even if not present in the profile! • Based on likelihood of member having a skill! • Features: company, industry, skills, ... 44 Profile Tokenize SkillsTagger Phrases Skills Skills Classifier Profile features Recommended Skills
- 45. Suggested Skill Endorsements • Binary Classification! • Features! • Company overlap, School overlap, Industrial and functional area similarity, Title similarity, Site interactions, Co- interactions, ... Candidate generation Classifier Features - Company - Title - Industry ... Suggested Endorsements 45
- 46. Social Recommendation And Tagging SkillTagging Skill Recommendation Suggested Skill Endorsements 46
- 47. Skills Important For Data Scientists? 47
- 48. Outline • About LinkedIn! • Social Recommender Systems at LinkedIn! • Social Graph Analysis! • Virality in Social Recommender Systems! • Scaling Challenges 48
- 49. Scaling Challenges: Related Searches Example • Kafka: publish-subscribe messaging system! • Hadoop: MapReduce data processing system ! • Azkaban: Hadoop workflow management tool! • Voldemort: Key-value store Metaphor Hadoop Search Backend Kafka Voldemort Related Searches Backend Front End HDFS 49
- 50. Outline • About LinkedIn! • Social Recommender Systems at LinkedIn! • Social Graph Analysis! • Virality in Social Recommender Systems! • Scaling Challenges 50
- 51. References 51
- 52. Acknowledgement • Thanks to Data Team at LinkedIn: http://data.linkedin.com! • We are hiring!! • Contact: mtiwari[at]linkedin.com! • Follow: @mitultiwari on Twitter 52 You! Applied Reseacher/ Research Engineer
- 53. Questions? 53