Recommender systems have become very important for many online activities, such as watching movies, shopping for products, and connecting with friends on social networks. User behavioral analysis and user feedback (both explicit and implicit) modeling are crucial for the improvement of any online recommender system. Widely adopted recommender systems at LinkedIn such as “People You May Know” and “Suggested Skills Endorsement” are evolving by analyzing user behaviors on impressed recommendation items. In this paper, we address modeling impression discounting of recommended items, that is, how to model users’ no-action feedback on impressed recommended items. The main contributions of this paper include (1) large-scale analysis of impression data from LinkedIn and KDD Cup; (2) novel anti-noise regression techniques, and its application to learn four different impression discounting functions including linear decay, inverse decay, exponential decay, and quadratic decay; (3) applying these impression discounting functions to LinkedIn’s “People You May Know” and “Suggested Skills Endorsements” recommender systems.

1. Modeling Impression Discounting In
Large-scale Recommender Systems
KDD 2014, presented by Mitul Tiwari
1
Pei Lee, Laks V.S. Lakshmanan
University of British Columbia
Vancouver, BC, Canada
Mitul Tiwari, Sam Shah
LinkedIn
Mountain View, CA, USA
2015/3/12

2. 2
LinkedIn By The Numbers
300M members 2 new members/sec

3. Large-scale Recommender Systems
at LinkedIn3
 People You May Know  Skills Endorsements

4. Recommender Systems at LinkedIn
4

5. What is Impression Discounting?
5
 Examples
 People You May Know
 Skills Endorsement
 Problem Definition

6. 5 days ago:
Recommended, but
not invited
Today:
If we recommend
again, would you
invite or not?
If you are not likely to invite, we better discount this recommendation.
1 day ago:
Recommended
again, but not invited
Example: People You May Know

7. Example: Skills Endorsement
 If you are likely to endorse, we will show a skills
suggestion again;
 Otherwise, we should leave this space for other skills
suggestions
Recommended Recommended Again

8. Problem Definition
 Impressions: recommendations shown to user
 Conversion: positive action - invite, endorse, etc.
 In natural language:
 We already impressed an item several times with no
conversion. How much should we discount that
item?
 More formally:
 For an user u and item i, given an impression history
(T1, T2 , …, Tn) between u and i, can we predict the
conversion rate for u on i?

9. Impression Discounting Framework
9

10. Outline
10
 Define an impression
 User, Item, Conversion Rate
 Features
 How many times the user saw this item?
 When is the last time the user saw this item? …
 Data analysis: impression features and conversion
relationship
 Impression discounting model fitting
 Experimental evaluation

11. Define an Impression
11

12. Features on Impressed Items
12
 ImpCount (frequency):
 the number of historical impressions before the current
impression, associated with the same (user, item)
 LastSeen (recency):
 the day difference between the last impression and the
current impression, associated with the same (user, item)
 Position:
 the offset of item in the recommendation list of user
 UserFreq:
 the interaction frequency of user in a recommender system

13. Data Analysis: PYMK Dataset
 Data: 1.09 B impression tuples
 Training dataset: 80% of data
 0.55 billion unique impressions
 0.87 billion total impressions
 20 millions invitations
 Testing dataset: 20% of data
 0.14 billion unique impressions (3.7%)
 0.22 billion total impressions (2.4%)
 5.2 millions invitations

14. Skills Endorsement Dataset
14
 Total dataset size: 190 million impression tuples
 Training dataset: 80% of data
 Testing dataset: 20% of data

15. Tencent SearchAds Dataset
15
 Publicly available for KDD Cup 2012 by the
Tencent search engine
 Total Size: 150 million impression sequences
 CTR of the Ad at the 1st, 2nd and 3rd
position: 4.8%, 2.7%, and 1.4%

16. Data Analysis: Conversion Rate
Changes with Impression Count
16
If we show an item to a
user repeatedly, the
conversion rate
decreases

17. 17
 The conversion rate
decreases with both
ImpCount and
LastSeen
PYMK Invitation Rate Changes with
Impression Count and Last Seen

18. 18
 Similar observation:
The conversion rate
also decreases with
both ImpCount and
LastSeen
Endorsement Rate Changes with
Impression Count and Last Seen

19. Impression Discounting Model Fitting
20
 Workflow:
 Model fitting process:
 Fundamental discounting functions
 Aggregation Model
 Offline and online

20. Conversion Rate with
ImpCount21

21. Discounting Functions
22

22. Conversion Rate with LastSeen
23

23. Aggregation Models
24
 Linear Aggregation
 Multiplicative Aggregation
f(Xi) is one of
discounting function
given earlier

24. Anti-Noise Regression Model
26
 The sparsity of observations increases variance in the
conversion rate.
 Typically happens when the number of observations are
relatively low.

25. Density Weighted Regression
27
 Add a weight with square error
 Original RMSE:
 Density weighted RMSE:
vi is assessed by the number of observations in a small
neighborhood of (Xi, yi)

26. Impression Discounting Framework
28

27. Offline Evaluation: Precision @k
29
 Precision at top k
 Precision improvement for different behavior sets:
More features yield
higher precision

28. Online Evaluation: Bucket Test on PYMK
30
 Use behavior set (LastSeen, ImpCount)
 On LinkedIn PYMK online system

29. Conclusion
31
 Impression Discounting
Model
 Learnt a discounting function
to capture ignored
impressions
 Linear or multiplicative
aggregation model
 Anti-noise regression model
 Offline and Online
evaluation (Bucket testing)

30. Questions?
32
 Related work in the paper
 Want to know more contact:
mtiwari@linkedin.com