Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Modeling Impression Discounting In
Large-scale Recommender Systems
KDD 2014, presented by Mitul Tiwari
1
Pei Lee, Laks V.S...
2
LinkedIn By The Numbers
300M members 2 new members/sec
Large-scale Recommender Systems
at LinkedIn3
 People You May Know  Skills Endorsements
Recommender Systems at LinkedIn
4
What is Impression Discounting?
5
 Examples
 People You May Know
 Skills Endorsement
 Problem Definition
5 days ago:
Recommended, but
not invited
Today:
If we recommend
again, would you
invite or not?
If you are not likely to i...
Example: Skills Endorsement
 If you are likely to endorse, we will show a skills
suggestion again;
 Otherwise, we should...
Problem Definition
 Impressions: recommendations shown to user
 Conversion: positive action - invite, endorse, etc.
 In...
Impression Discounting Framework
9
Outline
10
 Define an impression
 User, Item, Conversion Rate
 Features
 How many times the user saw this item?
 When...
Define an Impression
11
Features on Impressed Items
12
 ImpCount (frequency):
 the number of historical impressions before the current
impressio...
Data Analysis: PYMK Dataset
 Data: 1.09 B impression tuples
 Training dataset: 80% of data
 0.55 billion unique impress...
Skills Endorsement Dataset
14
 Total dataset size: 190 million impression tuples
 Training dataset: 80% of data
 Testin...
Tencent SearchAds Dataset
15
 Publicly available for KDD Cup 2012 by the
Tencent search engine
 Total Size: 150 million ...
Data Analysis: Conversion Rate
Changes with Impression Count
16
If we show an item to a
user repeatedly, the
conversion ra...
17
 The conversion rate
decreases with both
ImpCount and
LastSeen
PYMK Invitation Rate Changes with
Impression Count and ...
18
 Similar observation:
The conversion rate
also decreases with
both ImpCount and
LastSeen
Endorsement Rate Changes with...
Impression Discounting Model Fitting
20
 Workflow:
 Model fitting process:
 Fundamental discounting functions
 Aggrega...
Conversion Rate with
ImpCount21
Discounting Functions
22
Conversion Rate with LastSeen
23
Aggregation Models
24
 Linear Aggregation
 Multiplicative Aggregation
f(Xi) is one of
discounting function
given earlier
Anti-Noise Regression Model
26
 The sparsity of observations increases variance in the
conversion rate.
 Typically happe...
Density Weighted Regression
27
 Add a weight with square error
 Original RMSE:
 Density weighted RMSE:
vi is assessed b...
Impression Discounting Framework
28
Offline Evaluation: Precision @k
29
 Precision at top k
 Precision improvement for different behavior sets:
More feature...
Online Evaluation: Bucket Test on PYMK
30
 Use behavior set (LastSeen, ImpCount)
 On LinkedIn PYMK online system
Conclusion
31
 Impression Discounting
Model
 Learnt a discounting function
to capture ignored
impressions
 Linear or mu...
Questions?
32
 Related work in the paper
 Want to know more contact:
mtiwari@linkedin.com
Upcoming SlideShare
Loading in …5
×

Modeling Impression discounting in large-scale recommender systems

1,250 views

Published on

Recommender systems have become very important for many online activities, such as watching movies, shopping for products, and connecting with friends on social networks. User behavioral analysis and user feedback (both explicit and implicit) modeling are crucial for the improvement of any online recommender system. Widely adopted recommender systems at LinkedIn such as “People You May Know” and “Suggested Skills Endorsement” are evolving by analyzing user behaviors on impressed recommendation items.
In this paper, we address modeling impression discounting of
recommended items, that is, how to model users’ no-action feedback on impressed recommended items. The main contributions of this paper include (1) large-scale analysis of impression data from LinkedIn and KDD Cup; (2) novel anti-noise regression techniques, and its application to learn four different impression discounting functions including linear decay, inverse decay, exponential decay, and quadratic decay; (3) applying these impression discounting functions to LinkedIn’s “People You May Know” and “Suggested Skills Endorsements” recommender systems.

Published in: Data & Analytics
  • Be the first to comment

Modeling Impression discounting in large-scale recommender systems

  1. 1. Modeling Impression Discounting In Large-scale Recommender Systems KDD 2014, presented by Mitul Tiwari 1 Pei Lee, Laks V.S. Lakshmanan University of British Columbia Vancouver, BC, Canada Mitul Tiwari, Sam Shah LinkedIn Mountain View, CA, USA 2015/3/12
  2. 2. 2 LinkedIn By The Numbers 300M members 2 new members/sec
  3. 3. Large-scale Recommender Systems at LinkedIn3  People You May Know  Skills Endorsements
  4. 4. Recommender Systems at LinkedIn 4
  5. 5. What is Impression Discounting? 5  Examples  People You May Know  Skills Endorsement  Problem Definition
  6. 6. 5 days ago: Recommended, but not invited Today: If we recommend again, would you invite or not? If you are not likely to invite, we better discount this recommendation. 1 day ago: Recommended again, but not invited Example: People You May Know
  7. 7. Example: Skills Endorsement  If you are likely to endorse, we will show a skills suggestion again;  Otherwise, we should leave this space for other skills suggestions Recommended Recommended Again
  8. 8. Problem Definition  Impressions: recommendations shown to user  Conversion: positive action - invite, endorse, etc.  In natural language:  We already impressed an item several times with no conversion. How much should we discount that item?  More formally:  For an user u and item i, given an impression history (T1, T2 , …, Tn) between u and i, can we predict the conversion rate for u on i?
  9. 9. Impression Discounting Framework 9
  10. 10. Outline 10  Define an impression  User, Item, Conversion Rate  Features  How many times the user saw this item?  When is the last time the user saw this item? …  Data analysis: impression features and conversion relationship  Impression discounting model fitting  Experimental evaluation
  11. 11. Define an Impression 11
  12. 12. Features on Impressed Items 12  ImpCount (frequency):  the number of historical impressions before the current impression, associated with the same (user, item)  LastSeen (recency):  the day difference between the last impression and the current impression, associated with the same (user, item)  Position:  the offset of item in the recommendation list of user  UserFreq:  the interaction frequency of user in a recommender system
  13. 13. Data Analysis: PYMK Dataset  Data: 1.09 B impression tuples  Training dataset: 80% of data  0.55 billion unique impressions  0.87 billion total impressions  20 millions invitations  Testing dataset: 20% of data  0.14 billion unique impressions (3.7%)  0.22 billion total impressions (2.4%)  5.2 millions invitations
  14. 14. Skills Endorsement Dataset 14  Total dataset size: 190 million impression tuples  Training dataset: 80% of data  Testing dataset: 20% of data
  15. 15. Tencent SearchAds Dataset 15  Publicly available for KDD Cup 2012 by the Tencent search engine  Total Size: 150 million impression sequences  CTR of the Ad at the 1st, 2nd and 3rd position: 4.8%, 2.7%, and 1.4%
  16. 16. Data Analysis: Conversion Rate Changes with Impression Count 16 If we show an item to a user repeatedly, the conversion rate decreases
  17. 17. 17  The conversion rate decreases with both ImpCount and LastSeen PYMK Invitation Rate Changes with Impression Count and Last Seen
  18. 18. 18  Similar observation: The conversion rate also decreases with both ImpCount and LastSeen Endorsement Rate Changes with Impression Count and Last Seen
  19. 19. Impression Discounting Model Fitting 20  Workflow:  Model fitting process:  Fundamental discounting functions  Aggregation Model  Offline and online
  20. 20. Conversion Rate with ImpCount21
  21. 21. Discounting Functions 22
  22. 22. Conversion Rate with LastSeen 23
  23. 23. Aggregation Models 24  Linear Aggregation  Multiplicative Aggregation f(Xi) is one of discounting function given earlier
  24. 24. Anti-Noise Regression Model 26  The sparsity of observations increases variance in the conversion rate.  Typically happens when the number of observations are relatively low.
  25. 25. Density Weighted Regression 27  Add a weight with square error  Original RMSE:  Density weighted RMSE: vi is assessed by the number of observations in a small neighborhood of (Xi, yi)
  26. 26. Impression Discounting Framework 28
  27. 27. Offline Evaluation: Precision @k 29  Precision at top k  Precision improvement for different behavior sets: More features yield higher precision
  28. 28. Online Evaluation: Bucket Test on PYMK 30  Use behavior set (LastSeen, ImpCount)  On LinkedIn PYMK online system
  29. 29. Conclusion 31  Impression Discounting Model  Learnt a discounting function to capture ignored impressions  Linear or multiplicative aggregation model  Anti-noise regression model  Offline and Online evaluation (Bucket testing)
  30. 30. Questions? 32  Related work in the paper  Want to know more contact: mtiwari@linkedin.com

×