Beyond ratings and followers (RecSys 2012)


Published on

Industry session talk on LinkedIn Recommender System @ RecSys 2012

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • 175+ M members2 members per second
  • Taking a leaf out of PaoloCremonosi’s talk.. The answer is 50%.. There I gave it away.. Its time for coffee 50% of connections are from recommendations (PYMK50% of job applications are from recommendations (JYMBII)50% of group joins are from recommendations (GYML)
  • As a colleague of mine puts it.. We are the tour de force for Recommendations..From traditional recommender problems, i.e. recommending p
  • I am spoilt for choice here.. There is so much interesting work I can talk about .. But today I picked a few interesting areas not classically considered to be mainline recommender products.but in keeping with the Ecosystem theme, this application fits right in..Let’s talk about People Recommendations.. BUT not in the context of connecting or knowing or following or rating or dating .. This is about cloning..Recruiters and Head HuntersInterview multiple people for filling one role.Hiring Managers“Hire more like the superstars on my team..”LinkedInRecommend Jobs/News/Groups that “people like you” act on..More conceivable applications : Find similar leads for making a sales pitch, or let me give you a sample of people I want to show this Ad to.. Create me a segment .. or
  • Extensive set of tooling to target the population.. Yes we sorta shoot ourselves in the foot sometimes.. But then member’s come first. Example audience, in real time.. Let’s advertise tailor their campaigns. Also give a real-time reach estimate.
  • Solve the impedance mismatch by creating the Ad representation in the user space. This concept is used extensively at LinkedIn for all kinds of user recommendations, not just ads.
  • 8000 name variants of IBMWe use the definition of entity resolution terminology k−ambiguous and k−variant from [10]. Same company name can denote multiple company entities but each occurrence of a company name references a single entity only. A name referring to k different entities is called k − ambigous. Additionally, An entity which can be referred to by k different names is called k − variant.Ranker approach does not work. A given name may not be resolvable in the sense that the company entity has not being created yet…Classification problemGiven a pair of (member position, company entity), a binary classifier would determine whether there is enough evidence to resolve the member position to the company entity. This would address the problem of the ranking approach in that an unresolvable member position would most likely remain unresolved because the classifier has insufficient evidence for any company entity. It is certainly possible that there could be multiple company entities with sufficient evidence for a member position.
  • Unreasonable effectiveness of Big Data.. This chart shows the probability of holding a title across all titles, plotted vs number of months after graduation. Notice the spikes.. They are ~12 month almost perfectly aligned.. Remember the itch that you had when you finished 2 years at your company 
  • A brand new Recommendation Delivery paradigm – Tested on LinkedIn Groups to generate 2X Group Join rate. Applicable to advertising as well..The idea is simple - Reverse the Social Proof idea . Ask the actor to recommend their connections to interact with this item. - The message comes from the individual not LinkedInInherently socially endorsedTimely and contextualCan be applied to Ads delivery which we will be testing in the next few months
  • Incredibly powerful whetted paradigm that we are excited to try to rope into our Ads offerings
  • And now the technologies that drives it all. The core our matching algorithm uses Lucene with our custom query implementation. We use Hadoop to scale our platform. It serves a variety of needs from computing Collaborative filtering features, building Lucene indices offline, doing quality analysis of recommendation and host of other exciting thingsLucene does not provide fast real-time indexing. To keep our indices up-to date, we use a real-time indexing library on top of Lucene called Zoie. We provide facets to our members for drilling down and exploring recommendation results. This is made possible by a Faceting Search library called Bobo. For storing features and for caching recommendation results, we use a key-value store Voldemort. For analyzing tracking and reporting data, we use a distributed messaging system called Kafka.Out of these Bobo, Zoie, Voldemort and Kafka are developed at LinkedIn and are open sourced. In fact, Kafka is an apache incubator project.Historically, we have used R for model training. We have recently started experimenting with Mahout for model training.
  • Beyond ratings and followers (RecSys 2012)

    1. 1. Beyond Ratings & Followers Anmol Bhasin Sr. ManagerAnalytics Engineering
    2. 2. The answer is 4
    3. 3. The Recommender Ecosystem Similar Profiles Connections Network updates Events You May Be Interested In News 11
    4. 4. LinkedIn Recommendation Engine Jobs Groups PeopleRecommendation … Ads Entities Companies Searches be interested in Similar Groups Jobs You May Jobs Browse Browse Map Similar Jobs News Browse Map TalentMatch Groups Events GYML Referral Profiles People Similar Center Map … and more Products A/B APIRecommen-dation Types Behavior Collaborative Popularity User Feedback Analysis Filtering Shared, Dynamic, Unified (R-T) Feature Extraction, Entity (R-T) matching computations Core Resolution & Enrichment Offline data munging (hadoop) Service
    5. 5. Cloning
    6. 6. Possible Approaches Naïve K Nearest Neighbor solution  Complexity is O(n 2 ) Clustering  Latent Factor Models like PLSI or LDA  Hierarchical Agglomerative clustering Self Organizing Maps Item based Collaborative Filtering  Find pairs of Users viewed in the same session
    7. 7. Challenges Scale  175+ M profiles Dimensionality  ~2M companies  ~200K schools  ~147 industries  ~200 countries  ~25K titles  ~40K Skills  ~200 Job Functions Similar means different things to different people  Similar Behavior doesn’t mean you can replace me at my job  Accuracy vs Relevance (me & my boss.. ) Realtime.. It’s a problem of accuracy.. Not recall..
    8. 8. Approach  Focus attention only on pairs likely to be similar  Filter out the possibly dis-similar pairs  Run Similarity Functions on filtered in pairs FILTERCluster Rank
    9. 9. Locality Sensitive Hashing LSH function family for Cosine Distance
    10. 10. Approach  Focus attention only on pairs likely to be similar  Filter out the possibly dis-similar pairs  Run Similarity Functions on filtered in pairs FILTERCluster Rank
    11. 11. Similarity Functions Different bands of attributes  Boolean, Jaccard or Cosine Similarities across attribute pairs.• Logisitic Regression with Elastic Penalty  Learn model params on a set of hand labeled data points  Predicted value interpreted as score
    12. 12. Ad Ranking Given U j ,{(c0, b0 ), (c1, b1 ), (c2, b2 ), (c3, b3 )..(cn, bn )}, H Objective argmax(pCTR i *bidi ) iÎC Goal:  Increase revenue  Respect daily budgets of Advertisers  Good user experience
    13. 13. Campaign creation
    14. 14. Virtual Profiling Title : Eng Mgr Company : LinkedIn Location : CA,USA Skills : ML, RecSys Title : Vice President Company : Twitter Location : CA,USA Skills : DM, ML, RecSys ……………….
    15. 15. Virtual Profiling Title :Title : Eng Mgr Sr. SE<1>, Eng Mgr<1>,Company : LinkedIn Eng Dir<1>Location : CA,USASkills : ML, RecSys Company :Title : Sr. SE LinkedIn<2>, Google<1>,Company : GoogleLocation : PA, USA Location :Skills : ML, DM CA,USA <2>, PA, USA<1> Skills :Title : Eng Dir ML<2>, RecSys<1>,Company : Linkedin Stats<1>, DM<1>Location : PA, USASkills : ML, Stats, DM
    16. 16. Virtual ProfilingInformation Gain Pick Top K overrepresented features from the clicker distribution vs the target segment A representative projection of the item in the member feature space
    17. 17. CTR Prediction – CF Similarity Ranker MEMBER FEATURESAD CREATIVE VIRTUAL PROFILE Creative Score to features pCTR pCTRi correction L2 regularized Logistic Regression (Liblinear, VW, Mahout, ADMM) For new ad creatives back-off to the advertiser / ad category nodes till they reach critical impression/click volume (explore/exploit)
    18. 18. Feature Engineering – Entity Resolution Companies ‘IBM’ has 8000+ variations - ibm – ireland - ibm research - T J Watson Labs - International Bus. Machines K-Ambiguous - Deep Blue Huge impact on the business and UE  Ad targeting  TalentMatch  Referrals Asonam’11, KDD’11 30
    19. 19. Feature Engineering – Sticky Locations Open to relocation ?  Region similarity based on profiles or network  Region transition probability  predict individuals propensity to migrate and most likely migration target Impact on job recommendations  20% lift in views/viewers/applications/applicants
    20. 20. What should you transition to .. and when ? Probability of switch Months since graduation 32
    21. 21. Social Referral
    22. 22. Social ReferralLinkedin Group: Text Analytics From: Deepak Agarwal – Engineering Director, LinkedInI found this group interesting, and I think you will tooDeepak 2X conversionLinkedin Group: Text Analytics > 2X Conversion Mohammad Amin, Baoshi Yan, Sripad Sriram, Anmol Bhasin, Christian Posse. Social Referral : Using network connections to deliver recommendations. To appear in Proceedings of the Sixth ACM conference on Recommender systems (RecSys 12)
    23. 23. Orthogonality in A/B
    24. 24. Beware of some A/B testing pitfalls1. Novelty effect  E.g., new job recommendation algorithms have week-long novelty effect that shows lifts twice the stationary (real) one job views per 5% bucket range - 6/5/11 job views 6/19/11 9,000 7,000 8,000 7,000 6,000 6,000 5,000 5,000 4,000 job views per 5% bucket range - 4,000 6/5/11 3,000 job views 6/19/11 3,000 2,000 2,000 1,000 1 week lifts 1,000 2weeks lifts 0 0 0 5 10 15 20 25 0 5 10 15 20 251. Cannibalization  Zero-sum game or real lift?2. Random sampling destroys network effect 38 38
    25. 25. Open Source Technologies BoboZoie Voldemort Kafka 40
    26. 26. CreditsEngineering : Abhishek Gupta, Adam Smyczek, Adil Aijaz,Alan Li, Baoshi Yan, Bee-Chung Chen, Deepak Agarwal,Ethan Zhang, Haishan Liu, Igor Perisic, JonathanTraupman, Liang Zhang, Lokesh Bajaj, Mario Rodriguez,Mitul Tiwari, Mohammad Amin, Monica Rogati, ParulJain, Paul Ogilvie, Sam Shah, Sanjay Dubey, Tarun Kumar,Trevor Walker, Utku IrmakProduct : Andrew Hill, Christian posse, Gyanda Sachdeva,Mike Grishaver, Parker Barrile, Sachit Kamat Alphabetically sorted 
    27. 27. A Recommendation for you.. Picture yourself with this New Job: You Applied Researcher / Research Engineer
    28. 28. Contact: abhasin@linkedin.com