Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
A Two Step Ranking Solution 
for Twitter User Engagement 
Behnoush Abdollahi, Mahsa Badami, Gopi Chand 
Nutakki, Wenlong S...
Outline 
• Introduction 
• Challenges 
• Summary of our approach 
• Preprocessing 
• Two step ranking model 
• Neighborhoo...
Introduction 
• Data: extended version of the MovieTweetings 
dataset 
o collected from the users of the IMDb iOS app that...
Challenges 
• High dimensionality 
• Power law distribution of the user engagement 
• Missing values 
• Outliers 
• Imbala...
Summary of our approach 
Challenge@Recsys 2014 5
Preprocessing 
• Data cleaning and completion 
o Removing 
• special characters from text-based features, 
• empty, 
• red...
Preprocessing 
• Feature Extraction + Engineering 
from Twitter + IMDb 
Features 
Context-based 
Text-based 
Tweet-Movie 
...
Context-based features 
o User Profile(Twitter) 
• # of user’s followers/friends 
• # of users that are mentioned/ replied...
Text-based features 
description 
hashtag 
movie plot 
movie 
genre 
user 
extracted bag of words, then select the 
most r...
Tweet-Movie Similarity 
• Using common lower dimensional latent space 
• Joint latent space is learned 
o using NMF 
o to ...
Tweet-Movie Similarity 
1. Building a semantic tweet representation by factoring the tweet matrix X1: 
X1(n1 ´m1) = A1(n1 ...
Two step ranking model 
1. Cost sensitive classifier 
o Classify tweets into zero engagement and multiple non-zero 
classe...
Cost sensitive classifier 
• More than 95% of the data have zero engagement value  
o classical classification methods te...
Ranker 
• Let y the grade/ label set be: 
• Let twitter_user_ids: and 
y =1,2,..., l 
u1,u2,...,um 
a set of tweets associ...
Neighborhood-based Repairing for 
Tweets with Predicted Zero 
Engagement 
non-zero 
engagment 
non-zero 
engagment 
zero 
...
Neighborhood-based 
Repairing for Tweets with 
Predicted Zero-Engagement 
• Goal: to correct predictions of non-zero tweet...
Neighborhood-based 
Repairing for Tweets with 
Predicted Zero Engagement 
• Varied neighborhood size: NT= 5,10 or 20 
• Us...
Steps in Process 
classification 
refine 
classification 
ranking the 
non-zero 
tweets 
merge the 
zeros with 
the ranked...
Results 
• Step 1- Classifier 
o Used Weka with a cost sensitive framework 
o The engagement was discretized into 6 classe...
Results 
• nDCG@10 for the test data. 
• Ranker applied indiscriminately on both zero and 
non-zero class tweets. 
• This ...
Results 
• Merging the zero and non-zero predicted 
engagement to calculate nDCG@10 
– by appending the zero-class tweets ...
Results 
• Effect of repairing the predicted zero engagement tweets on 
nDCG@10 values for varying margin sizes defined by...
Lessons Learned 
• What helped 
o Adding additional sources of data (IMDb). 
o Dedicated janitorial work on the data 
o Fe...
Lessons Learned 
• What hurt 
o Over or under-sampling to handle class 
imbalance 
o Spending too much time on filling mis...
Conclusion 
• If more time or computational power were within our 
reach, we would further explore several directions: 
o ...
Upcoming SlideShare
Loading in …5
×

A Two Step Ranking Solution for Twitter User Engagement

608 views

Published on

A two step ranking solution to the RecSys 2014 Challenge presented at the RecSys 2014 conference on Oct 10, in Foster City (CA, USA) by Behnoush Abdollahi

Published in: Science
  • Be the first to comment

  • Be the first to like this

A Two Step Ranking Solution for Twitter User Engagement

  1. 1. A Two Step Ranking Solution for Twitter User Engagement Behnoush Abdollahi, Mahsa Badami, Gopi Chand Nutakki, Wenlong Sun, Olfa Nasraoui Knowledge Discovery and Web Mining Lab University of Louisville http://webmining.spd.louisville.edu Challenge@Recsys 2014 1
  2. 2. Outline • Introduction • Challenges • Summary of our approach • Preprocessing • Two step ranking model • Neighborhood-based Repairing for Tweets with Predicted Zero Engagement • Results • Lessons learned • Conclusion Challenge@Recsys 2014 2
  3. 3. Introduction • Data: extended version of the MovieTweetings dataset o collected from the users of the IMDb iOS app that rate movies and share the rating on Twitter • Predicting user engagement for: o #favorites o #retweets • Evaluation: nDCG@10 metric learning to rank approach data statistics: Challenge@Recsys 2014 3
  4. 4. Challenges • High dimensionality • Power law distribution of the user engagement • Missing values • Outliers • Imbalanced engagement distribution o more than 95% of engagement = zero • Different from many standard prediction and recommendation problems: o user egagement may be affected by • implicit profile information of the user on Twitter • movie preference data by other users • movie content data Challenge@Recsys 2014 4
  5. 5. Summary of our approach Challenge@Recsys 2014 5
  6. 6. Preprocessing • Data cleaning and completion o Removing • special characters from text-based features, • empty, • redundant, • invalid values such as movie ratings exceeding 10 o Filling missing data such as user id by looking at the nearest tweets o Using country code as a location feature • if missing  converted other similar geographical features to the country code Challenge@Recsys 2014 6
  7. 7. Preprocessing • Feature Extraction + Engineering from Twitter + IMDb Features Context-based Text-based Tweet-Movie similarity Challenge@Recsys 2014 7
  8. 8. Context-based features o User Profile(Twitter) • # of user’s followers/friends • # of users that are mentioned/ replied to by the current user • same features for each re-tweeted tweet o Movie Profile (IMDb) • imdb movie plot, director, actors, genre, languages, countries • # of times in which a movie has been tagged in a tweet • average rating for a movie • Movie total re-tweet/favorite count number of users who have rated a particular movie o Twitter Profile • tweet/retweet flag • time delay between the tweet and movie release date • seasonality (Christmas, Halloween time,…) • time features: certain times of the day (day or night), certain days of the week (week days or weekends) Challenge@Recsys 2014 8
  9. 9. Text-based features description hashtag movie plot movie genre user extracted bag of words, then select the most relevant based on Mutual Information Gain Ratio with target Challenge@Recsys 2014 9
  10. 10. Tweet-Movie Similarity • Using common lower dimensional latent space • Joint latent space is learned o using NMF o to capture the similarity between a tweet and the movie that it mentions o to handle the problems of: • sparsity, • high dimensionality in bag of word features, • poor semantics o based on tweet & movie features, such as: • hashtags, • user description, • movie genres, • movie plot, ... Challenge@Recsys 2014 10
  11. 11. Tweet-Movie Similarity 1. Building a semantic tweet representation by factoring the tweet matrix X1: X1(n1 ´m1) = A1(n1 ´ f1)´B1( f1 ´m1) where: n1 = #tweets, m1 = #features and f1 = #factors Tweet s Words Tweet s Latent Factors Latent Factors Words x 2. Mapping the movie data X2 into the latent space defined by B1 to compute the movie coefficient matrix A2 X2(n2 ´m1) = A2 (n2 ´ f1)´B1( f1 ´m1) where: n2 = #movies 3. Computing the similarity using the dot product of the corresponding rows of A1 and A2 sim(X,X ) = A´ AT 11 2 2 Challenge@Recsys 2014 11
  12. 12. Two step ranking model 1. Cost sensitive classifier o Classify tweets into zero engagement and multiple non-zero classes o Imbalanced data  used cost sensitive classification 2. Ranker o List-wise, point-wise, pair-wise approaches o Predict the relevance values in Information Retrieval tasks o Engagement values considered as grades/labels • y =1, 2,…, l Challenge@Recsys 2014 12
  13. 13. Cost sensitive classifier • More than 95% of the data have zero engagement value  o classical classification methods tend to misclassify the minority class (non-zeros engagements ) as the majority class (zero engagements) • Cost sensitive framework o assign different weights to errors of type I and type II o assign higher weights to tweets classified in Zero engagement class o use cost matrix C, for a sample x in class i, the following Loss function is minimized to find p: j å C(i,j) = the cost of predicting class i when the true class is j • Tweets classified as non-zero engagement are then passed to the ranker actual negative actual positive predicted negative C(0,0) C(0,1) predicted positive C(1,0) C(1,1) L(x, i) = P( j | x)C(i, j) Challenge@Recsys 2014 13
  14. 14. Ranker • Let y the grade/ label set be: • Let twitter_user_ids: and y =1,2,..., l u1,u2,...,um a set of tweets associated with user ui Ti = ti,1, ti,2,..., ti,ni then y= y, y,..., yis the set of grades associated with user u(nis i i,1i,2i,ni i i the size of Tand y) i i• The classified training set is denoted as: i, j x =F( i u, S = {(ui,Ti ), m yi } i=1 • A feature vector is generated from each group-tweet pair i u, • Our goal: to train ranking models which can assign a score to a given pair • Used Random Forrest (RF) o uses independent subsets o parallelizable, robust to noisy data, capable of learning disjunctive expressions i, j t ) ( i, j t ) Challenge@Recsys 2014 14
  15. 15. Neighborhood-based Repairing for Tweets with Predicted Zero Engagement non-zero engagment non-zero engagment zero engagment zero engagment Challenge@Recsys 2014 15
  16. 16. Neighborhood-based Repairing for Tweets with Predicted Zero-Engagement • Goal: to correct predictions of non-zero tweets misclassified as the zero engagement tweets • Added neighborhood based approach after step 1: 1. Compute similarity between training and test tweets in common Latent space (computed using NMF) 2. Find the NT nearest tweets in the training for each test tweet classified as zero-engagement 3. Reassign the predicted engagement Challenge@Recsys 2014 16
  17. 17. Neighborhood-based Repairing for Tweets with Predicted Zero Engagement • Varied neighborhood size: NT= 5,10 or 20 • Used two options to reassign predicted engagements: 1. Predict non-zero engagement If either of the neighbors’ engagements is non-zero 2. Predict engagement = the most frequent engagement value among neighbors • Select a margin based on cosine similarity to consider a set of zero-predicted-engagement test tweets to become candidates to be repaired o  move to non-zero predicted engagement Challenge@Recsys 2014 17
  18. 18. Steps in Process classification refine classification ranking the non-zero tweets merge the zeros with the ranked non-zeros tweets Challenge@Recsys 2014 18
  19. 19. Results • Step 1- Classifier o Used Weka with a cost sensitive framework o The engagement was discretized into 6 classes class 1 • only 0s class 2 • only 1s class 3 • 2-10 class 4 • 11-20 class 5 • 21-50 class 6 • values > 50 o Adaboost gave the best result • 99% classification accuracy • true positive rate = 0.74 in the minority class (non-zero engagements) • false positive rate = 0.01 • Step 2- Ranker o Built global ranking model o Used RankLib implemented in Java o Engagement is considered the target value to be ranked Challenge@Recsys 2014 19
  20. 20. Results • nDCG@10 for the test data. • Ranker applied indiscriminately on both zero and non-zero class tweets. • This result is only shown to appreciate the impact of the classier in Step 1. Ranking Algorithm All features Excluding IMDb Features Excluding graph-propagated features Random Forest 0.553 0.503 0.485 LambdaMAR T 0.466 0.422 0.411 RankBoost 0.432 0.417 0.406 Challenge@Recsys 2014 20
  21. 21. Results • Merging the zero and non-zero predicted engagement to calculate nDCG@10 – by appending the zero-class tweets at the end of the non-zero tweets for each user and then sorting the tweets again based on the user id Ranking Algorithm All features Excluding IMDb Features Excluding graph-propagated features Random Forest 0.805 0.503 0.485 LambdaMART 0.466 0.422 0.411 RankBoost 0.432 0.417 0.406 Challenge@Recsys 2014 21
  22. 22. Results • Effect of repairing the predicted zero engagement tweets on nDCG@10 values for varying margin sizes defined by different similarity thresholds and neighborhood sizes • varied the neighborhood size, NT= 5,10,15 • for NT = 10 nearest neighbors and a similarity threshold of 0.9 to confine the margin where tweets are repaired  increased the nDCG@10 value to 0.817 Challenge@Recsys 2014 22
  23. 23. Lessons Learned • What helped o Adding additional sources of data (IMDb). o Dedicated janitorial work on the data o Feature Engineering: selection, extraction, and construction of relevant features. o Mapping data (tweets and movies) to a Latent Factor Space using NMF o Binary classification of zero and non-zero engagement tweets prior to ranking. o Cost-sensitive classification in Step 1 o Using Learning to Rank (LTR) methods in Step 2. o Repairing misclassified non-zero engagement tweets within a limited margin, and then re-ranking them. Challenge@Recsys 2014 23
  24. 24. Lessons Learned • What hurt o Over or under-sampling to handle class imbalance o Spending too much time on filling missing values for many features did not have any return on investment o Not having the luxury of better computational power o Building separate models based on whether or not the users and movies were common between the training and test sets Challenge@Recsys 2014 24
  25. 25. Conclusion • If more time or computational power were within our reach, we would further explore several directions: o Exploring other LTR options in addition to the pointwise approach, including pairwise and listwise LTR. o Domain-informed Transductive learning o Exploring additional extracted or constructed features that may affect engagement. o Exploring high performance computing to compute many embarrassably parallellizable tasks. Challenge@Recsys 2014 25

×