SlideShare a Scribd company logo
1 of 28
Download to read offline
Temporal Learning and Sequence Modeling
for a Job Recommender System
Kuan Liu, Xing Shi
Anoop Kumar, Linhong Zhu, and Prem Natarajan
Information Sciences Institute, University of Southern California
Team: Milk Tea
Personalized Job Post Recommendation
Task: Recommend job posts to users on Xing
Given: 1) interaction history and 2) user/item features
Goal: Obtain highest recommentation score fo active users
Challenges and Observations:
Large volume
1.5M users, 1.3M items, 8.8M interactions, 200M impression
Rich/Noisy user/item features available
eps. categorical features. e.g. >100K text tokens
Temporal dynamics/sequence form in interaction history
Observervations
Temporal Dynamics: Time as a factor to influence a user's future behavior.
Observation: users tend to re-interact with items that they did in the past.
e.g. on average 2 of 7 items in a user’s Week 45 appeared in his past interaction list.
Observation: users are more influenced by what they interacted recently than long time ago.
How to explicitly model temporal influence?
Sequence Property
User-Item interactions are NOT i.i.d. Instead, a user interacts with a sequence of items.
Conjecture: Item sequence may contain additional useful information that helps improvement recommender systems. (e.g. temporal
relation, item-item similarities.)
Does sequence really help? If so, how to model?
A. Temporal Ranking on Historical Items
Motivation:
Users have a strong tendency to re-interact with items that they already did in
the past.
More recent interactions influence a user’s future behavior more.
Historical items are important! Recency of interaction matters!
Approach:
A (time reweighted) linear ranking model.
Minimize a loss incurred on carefully constructed triplet constraints.
A. Temporal Ranking on Historical Items Model
Linear Ranking Model
Model solving based on triplet constraints
relative contribution of types of
interaction (click, save, respond,
delete) at t
• Construct constraints when u interacted with i1, i2 before t, but only interacted
with i1 at t
• Minimize number of constrait voilations
Historical interactions
beween user u and item i
at time tLikelihood of user
interacting with
item
Experimental Setup
Settings:
Training Data: 26 to 44 week
Validation: Week 45
Validation scores are reported
Submitting quota limit
Consistent validation/test scores
Evaluation metric:
Score (all): The challenge score
Score(new): The score after removing all user-item pair in the history.
Recommend from History
Scores (in thousands) only based on historical items.
(The higher, the better.)
Distribution of Weights
B. Temporal Matrix Factorization
Matrix Factorization
To recommend new items
Hybrid Matrix Factorization (HMF)
Learn categorical features
Temporal HMF (THMF)
Re-weight loss of HMF by time
Hybrid Matrix Factorization
Users/Items are represented as sums of feature embedding. (b: bias.)
User-item score is given by inner product
Model is trained by minimizing the loss
Temporal Hybrid Matrix Factorization
A non-negative weight associated with time is placed in the loss
captures contribution of interactions over time. Zero weights in
reduce training set size as well.
● Computeing .
○ can be learned jointly with other embedding parameters.
○ in our experiment are fixed as learned weights in Model A
Temporal HMF Improves HMF
C. Sequence Modeling
USER 1: ITEM 93, ITEM 5, …, ITEM 27 (-> ??, ??, ??)
Tools:
• Encoder(users)-Decoder(items) framework: next item recommendation is based on both user
and previous items
• LSTM to model ‘user encoding’ and ‘item transition’
• Embedding layer to incorporate feature learning
USER 8: ITEM 55, ITEM 24, …, ITEM 5 (-> ??, ??, ??)
...
USER 65: ITEM 47, ITEM 7, …, ITEM 62 (-> ??, ??, ??)
Sequence of items ordered by time:
LSTM Implementation
Novel Extension to Model
• Features
• Continuous embedding is used to learn categorical features
• New layer (look-up table and concatenation) is used connect input and RNN cells
• Anonymous users
• Item IDs are treated as categorical features
• User IDs are removed to prevent overfitting
• Sampling and data augmentation
• No sampling
• Original sequence gives better empirical results
Recommend via LSTMs
Does Sequence Help?
Implicit assumption: sequence or order provides additional information beyond
that provided by item frequency alone
Experiment:
Original sequence
Sub-sequence sampling
Does Sequence Help?
Implicit assumption: sequence or order provides additional information beyond
that provided by item frequency alone
Experiment:
Original sequence
Sub-sequence sampling
Approach Summary
Temporal Learning
A. Temporal based Ranking
B. Temporal MF
Wins over non-temporal counterpart significantly.
Sequence Modeling
C. LSTM based Encoder-Decoder model.
Beats the best MF model.
Ensemble Methods
• Linear Regression
• Random Forests
Conclusion
• History based model attains the best individual score
• Proposed RNN-based model outperforms the commonly used matrix
factorization
• Ensemble based model obtained 5th place in the competition
Component History HMF THMF LSTM Ensemble
Score (thousands) 524 312 366 391 613
Q & A
Thanks you!
Other slides
Outline
RecSys Challenge 2016
Approach Overview
Temporal Learning
Sequence modeling
Experiments
Conclude
Recommend via MF
Recommend via LSTMs
Final score before/after ensemble
Temporal Learning and Sequence Modeling for a Job Recommender System

More Related Content

What's hot

Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at NetflixJustin Basilico
 
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender SystemsHybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender SystemsMatthias Braunhofer
 
GTC 2021: Counterfactual Learning to Rank in E-commerce
GTC 2021: Counterfactual Learning to Rank in E-commerceGTC 2021: Counterfactual Learning to Rank in E-commerce
GTC 2021: Counterfactual Learning to Rank in E-commerceGrubhubTech
 
Handling Missing Attributes using Matrix Factorization 
Handling Missing Attributes using Matrix Factorization Handling Missing Attributes using Matrix Factorization 
Handling Missing Attributes using Matrix Factorization CS, NcState
 
Download
DownloadDownload
Downloadbutest
 
Collaborative Metric Learning (WWW'17)
Collaborative Metric Learning (WWW'17)Collaborative Metric Learning (WWW'17)
Collaborative Metric Learning (WWW'17)承剛 謝
 
Sentiment analysis of tweets using Neural Networks
Sentiment analysis of tweets using Neural NetworksSentiment analysis of tweets using Neural Networks
Sentiment analysis of tweets using Neural NetworksAdrián Palacios Corella
 
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...Alejandro Bellogin
 
Collaborative Filtering with Spark
Collaborative Filtering with SparkCollaborative Filtering with Spark
Collaborative Filtering with SparkChris Johnson
 
Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...
Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...
Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...Alan Said
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender SystemsLior Rokach
 
Collaborative Filtering 1: User-based CF
Collaborative Filtering 1: User-based CFCollaborative Filtering 1: User-based CF
Collaborative Filtering 1: User-based CFYusuke Yamamoto
 
Slides ppt
Slides pptSlides ppt
Slides pptbutest
 
Replicable Evaluation of Recommender Systems
Replicable Evaluation of Recommender SystemsReplicable Evaluation of Recommender Systems
Replicable Evaluation of Recommender SystemsAlejandro Bellogin
 
Alleviating cold-user start problem with users' social network data in recomm...
Alleviating cold-user start problem with users' social network data in recomm...Alleviating cold-user start problem with users' social network data in recomm...
Alleviating cold-user start problem with users' social network data in recomm...Eduardo Castillejo Gil
 
Active Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a SurveyActive Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a SurveyUniversity of Bergen
 
Collaborative Filtering using KNN
Collaborative Filtering using KNNCollaborative Filtering using KNN
Collaborative Filtering using KNNŞeyda Hatipoğlu
 

What's hot (20)

Artwork Personalization at Netflix
Artwork Personalization at NetflixArtwork Personalization at Netflix
Artwork Personalization at Netflix
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender SystemsHybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
Hybrid Solution of the Cold-Start Problem in Context-Aware Recommender Systems
 
GTC 2021: Counterfactual Learning to Rank in E-commerce
GTC 2021: Counterfactual Learning to Rank in E-commerceGTC 2021: Counterfactual Learning to Rank in E-commerce
GTC 2021: Counterfactual Learning to Rank in E-commerce
 
Active learning
Active learningActive learning
Active learning
 
Handling Missing Attributes using Matrix Factorization 
Handling Missing Attributes using Matrix Factorization Handling Missing Attributes using Matrix Factorization 
Handling Missing Attributes using Matrix Factorization 
 
Download
DownloadDownload
Download
 
Collaborative Metric Learning (WWW'17)
Collaborative Metric Learning (WWW'17)Collaborative Metric Learning (WWW'17)
Collaborative Metric Learning (WWW'17)
 
Sentiment analysis of tweets using Neural Networks
Sentiment analysis of tweets using Neural NetworksSentiment analysis of tweets using Neural Networks
Sentiment analysis of tweets using Neural Networks
 
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
HT2014 Tutorial: Evaluating Recommender Systems - Ensuring Replicability of E...
 
Collaborative Filtering with Spark
Collaborative Filtering with SparkCollaborative Filtering with Spark
Collaborative Filtering with Spark
 
Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...
Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...
Comparative Recommender System Evaluation: Benchmarking Recommendation Frame...
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Collaborative Filtering 1: User-based CF
Collaborative Filtering 1: User-based CFCollaborative Filtering 1: User-based CF
Collaborative Filtering 1: User-based CF
 
Slides ppt
Slides pptSlides ppt
Slides ppt
 
Temporal based Recommendation System
Temporal based Recommendation SystemTemporal based Recommendation System
Temporal based Recommendation System
 
Replicable Evaluation of Recommender Systems
Replicable Evaluation of Recommender SystemsReplicable Evaluation of Recommender Systems
Replicable Evaluation of Recommender Systems
 
Alleviating cold-user start problem with users' social network data in recomm...
Alleviating cold-user start problem with users' social network data in recomm...Alleviating cold-user start problem with users' social network data in recomm...
Alleviating cold-user start problem with users' social network data in recomm...
 
Active Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a SurveyActive Learning in Collaborative Filtering Recommender Systems : a Survey
Active Learning in Collaborative Filtering Recommender Systems : a Survey
 
Collaborative Filtering using KNN
Collaborative Filtering using KNNCollaborative Filtering using KNN
Collaborative Filtering using KNN
 

Viewers also liked

RecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-Pie
RecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-PieRecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-Pie
RecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-PieTommaso Carpi
 
Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...
Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...
Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...Vasily Leksin
 
Jobandtalent at recsys challenge 2016
Jobandtalent at recsys challenge 2016Jobandtalent at recsys challenge 2016
Jobandtalent at recsys challenge 2016Oscar Huarte
 
RecSys Challenge 2016
RecSys Challenge 2016RecSys Challenge 2016
RecSys Challenge 2016Fabian Abel
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Xavier Amatriain
 
Machine learning @ Spotify - Madison Big Data Meetup
Machine learning @ Spotify - Madison Big Data MeetupMachine learning @ Spotify - Madison Big Data Meetup
Machine learning @ Spotify - Madison Big Data MeetupAndy Sloane
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Xavier Amatriain
 

Viewers also liked (9)

RecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-Pie
RecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-PieRecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-Pie
RecSys Multi-Stack Ensemble for Job Recommendation, Pumpkin-Pie
 
Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...
Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...
Avito recsys-challenge-2016RecSys Challenge 2016: Job Recommendation Based on...
 
Jobandtalent at recsys challenge 2016
Jobandtalent at recsys challenge 2016Jobandtalent at recsys challenge 2016
Jobandtalent at recsys challenge 2016
 
RecSys Challenge 2016
RecSys Challenge 2016RecSys Challenge 2016
RecSys Challenge 2016
 
Recruit recsys-review-magambo
Recruit recsys-review-magamboRecruit recsys-review-magambo
Recruit recsys-review-magambo
 
Recsys 2016
Recsys 2016Recsys 2016
Recsys 2016
 
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
 
Machine learning @ Spotify - Madison Big Data Meetup
Machine learning @ Spotify - Madison Big Data MeetupMachine learning @ Spotify - Madison Big Data Meetup
Machine learning @ Spotify - Madison Big Data Meetup
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
 

Similar to Temporal Learning and Sequence Modeling for a Job Recommender System

V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...
V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...
V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...eMadrid network
 
Collaborative Filtering Survey
Collaborative Filtering SurveyCollaborative Filtering Survey
Collaborative Filtering Surveymobilizer1000
 
CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"
CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"
CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"Byungjoo Lee
 
Filtering out improper user accounts from twitter user accounts for discoveri...
Filtering out improper user accounts from twitter user accounts for discoveri...Filtering out improper user accounts from twitter user accounts for discoveri...
Filtering out improper user accounts from twitter user accounts for discoveri...siramatu-lab
 
Measuring the New Wikipedia Community (PyData SV 2013)
Measuring the New Wikipedia Community (PyData SV 2013)Measuring the New Wikipedia Community (PyData SV 2013)
Measuring the New Wikipedia Community (PyData SV 2013)PyData
 
Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...
Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...
Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...Kishor Datta Gupta
 
How can pre-training help to solve the cold start problem?
How can pre-training help to solve the cold start problem?How can pre-training help to solve the cold start problem?
How can pre-training help to solve the cold start problem?Lokesh Vadlamudi
 
Recommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learningRecommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learningArithmer Inc.
 
Preference amplification in recommendation system
Preference amplification in recommendation system Preference amplification in recommendation system
Preference amplification in recommendation system taeseon ryu
 
Recommendation system based on adaptive ontological graphs and weighted ranking
Recommendation system based on adaptive ontological graphs and weighted rankingRecommendation system based on adaptive ontological graphs and weighted ranking
Recommendation system based on adaptive ontological graphs and weighted rankingvikramadityajakkula
 
Recommending Sequences RecTour 2017
Recommending Sequences RecTour 2017Recommending Sequences RecTour 2017
Recommending Sequences RecTour 2017Gunjan Kumar
 
case based recommendation approach for market basket data
case based recommendation approach for market basket datacase based recommendation approach for market basket data
case based recommendation approach for market basket datamniranjanmurthy
 
2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...Ed Chi
 
Recommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative FilteringRecommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative FilteringChangsung Moon
 
Recommender systems for E-commerce
Recommender systems for E-commerceRecommender systems for E-commerce
Recommender systems for E-commerceAlexander Konduforov
 
Bridging the gap between closed and open items or how to make CALL more intel...
Bridging the gap between closed and open items or how to make CALL more intel...Bridging the gap between closed and open items or how to make CALL more intel...
Bridging the gap between closed and open items or how to make CALL more intel...bwylin
 
FLEAT VI - Harvard University - Piet Desmet & Bert Wylin
FLEAT VI - Harvard University - Piet Desmet & Bert WylinFLEAT VI - Harvard University - Piet Desmet & Bert Wylin
FLEAT VI - Harvard University - Piet Desmet & Bert WylinPiet Desmet
 
Helping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCTHelping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCTcies
 
Interactive Machine Learning Appendix
Interactive  Machine Learning AppendixInteractive  Machine Learning Appendix
Interactive Machine Learning AppendixZitao Liu
 

Similar to Temporal Learning and Sequence Modeling for a Job Recommender System (20)

V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...
V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...
V Jornadas eMadrid sobre "Educación Digital". Cristina Conati, University of ...
 
Collaborative Filtering Survey
Collaborative Filtering SurveyCollaborative Filtering Survey
Collaborative Filtering Survey
 
CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"
CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"
CHI'16 Full Paper "Spotlights: Attention-Optimised Highlights for Skim Reading"
 
Filtering out improper user accounts from twitter user accounts for discoveri...
Filtering out improper user accounts from twitter user accounts for discoveri...Filtering out improper user accounts from twitter user accounts for discoveri...
Filtering out improper user accounts from twitter user accounts for discoveri...
 
Measuring the New Wikipedia Community (PyData SV 2013)
Measuring the New Wikipedia Community (PyData SV 2013)Measuring the New Wikipedia Community (PyData SV 2013)
Measuring the New Wikipedia Community (PyData SV 2013)
 
Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...
Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...
Deep Reinforcement Learning based Recommendation with Explicit User-ItemInter...
 
How can pre-training help to solve the cold start problem?
How can pre-training help to solve the cold start problem?How can pre-training help to solve the cold start problem?
How can pre-training help to solve the cold start problem?
 
Recommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learningRecommendation algorithm using reinforcement learning
Recommendation algorithm using reinforcement learning
 
Preference amplification in recommendation system
Preference amplification in recommendation system Preference amplification in recommendation system
Preference amplification in recommendation system
 
Recommendation system based on adaptive ontological graphs and weighted ranking
Recommendation system based on adaptive ontological graphs and weighted rankingRecommendation system based on adaptive ontological graphs and weighted ranking
Recommendation system based on adaptive ontological graphs and weighted ranking
 
Recommending Sequences RecTour 2017
Recommending Sequences RecTour 2017Recommending Sequences RecTour 2017
Recommending Sequences RecTour 2017
 
Chounta@paws
Chounta@pawsChounta@paws
Chounta@paws
 
case based recommendation approach for market basket data
case based recommendation approach for market basket datacase based recommendation approach for market basket data
case based recommendation approach for market basket data
 
2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
 
Recommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative FilteringRecommender Systems: Advances in Collaborative Filtering
Recommender Systems: Advances in Collaborative Filtering
 
Recommender systems for E-commerce
Recommender systems for E-commerceRecommender systems for E-commerce
Recommender systems for E-commerce
 
Bridging the gap between closed and open items or how to make CALL more intel...
Bridging the gap between closed and open items or how to make CALL more intel...Bridging the gap between closed and open items or how to make CALL more intel...
Bridging the gap between closed and open items or how to make CALL more intel...
 
FLEAT VI - Harvard University - Piet Desmet & Bert Wylin
FLEAT VI - Harvard University - Piet Desmet & Bert WylinFLEAT VI - Harvard University - Piet Desmet & Bert Wylin
FLEAT VI - Harvard University - Piet Desmet & Bert Wylin
 
Helping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCTHelping teachers understand their learners and their needs better in WebCT
Helping teachers understand their learners and their needs better in WebCT
 
Interactive Machine Learning Appendix
Interactive  Machine Learning AppendixInteractive  Machine Learning Appendix
Interactive Machine Learning Appendix
 

Recently uploaded

一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理cyebo
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Jon Hansen
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Valters Lauzums
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp onlinebalibahu1313
 
The Significance of Transliteration Enhancing
The Significance of Transliteration EnhancingThe Significance of Transliteration Enhancing
The Significance of Transliteration Enhancingmohamed Elzalabany
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictJack Cole
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonPayment Village
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理pyhepag
 
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理pyhepag
 
社内勉強会資料  Mamba - A new era or ephemeral
社内勉強会資料   Mamba - A new era or ephemeral社内勉強会資料   Mamba - A new era or ephemeral
社内勉強会資料  Mamba - A new era or ephemeralNABLAS株式会社
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyRafigAliyev2
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfEmmanuel Dauda
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...Amil baba
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理pyhepag
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfMichaelSenkow
 
Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfRobertoOcampo24
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...ssuserf63bd7
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxStephen266013
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Calllward7
 

Recently uploaded (20)

一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理一比一原版纽卡斯尔大学毕业证成绩单如何办理
一比一原版纽卡斯尔大学毕业证成绩单如何办理
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp online
 
The Significance of Transliteration Enhancing
The Significance of Transliteration EnhancingThe Significance of Transliteration Enhancing
The Significance of Transliteration Enhancing
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prison
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
 
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
 
社内勉強会資料  Mamba - A new era or ephemeral
社内勉強会資料   Mamba - A new era or ephemeral社内勉強会資料   Mamba - A new era or ephemeral
社内勉強会資料  Mamba - A new era or ephemeral
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertainty
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdf
 
Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdf
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptx
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call
 

Temporal Learning and Sequence Modeling for a Job Recommender System

  • 1. Temporal Learning and Sequence Modeling for a Job Recommender System Kuan Liu, Xing Shi Anoop Kumar, Linhong Zhu, and Prem Natarajan Information Sciences Institute, University of Southern California Team: Milk Tea
  • 2. Personalized Job Post Recommendation Task: Recommend job posts to users on Xing Given: 1) interaction history and 2) user/item features Goal: Obtain highest recommentation score fo active users Challenges and Observations: Large volume 1.5M users, 1.3M items, 8.8M interactions, 200M impression Rich/Noisy user/item features available eps. categorical features. e.g. >100K text tokens Temporal dynamics/sequence form in interaction history
  • 3. Observervations Temporal Dynamics: Time as a factor to influence a user's future behavior. Observation: users tend to re-interact with items that they did in the past. e.g. on average 2 of 7 items in a user’s Week 45 appeared in his past interaction list. Observation: users are more influenced by what they interacted recently than long time ago. How to explicitly model temporal influence? Sequence Property User-Item interactions are NOT i.i.d. Instead, a user interacts with a sequence of items. Conjecture: Item sequence may contain additional useful information that helps improvement recommender systems. (e.g. temporal relation, item-item similarities.) Does sequence really help? If so, how to model?
  • 4. A. Temporal Ranking on Historical Items Motivation: Users have a strong tendency to re-interact with items that they already did in the past. More recent interactions influence a user’s future behavior more. Historical items are important! Recency of interaction matters! Approach: A (time reweighted) linear ranking model. Minimize a loss incurred on carefully constructed triplet constraints.
  • 5. A. Temporal Ranking on Historical Items Model Linear Ranking Model Model solving based on triplet constraints relative contribution of types of interaction (click, save, respond, delete) at t • Construct constraints when u interacted with i1, i2 before t, but only interacted with i1 at t • Minimize number of constrait voilations Historical interactions beween user u and item i at time tLikelihood of user interacting with item
  • 6. Experimental Setup Settings: Training Data: 26 to 44 week Validation: Week 45 Validation scores are reported Submitting quota limit Consistent validation/test scores Evaluation metric: Score (all): The challenge score Score(new): The score after removing all user-item pair in the history.
  • 7. Recommend from History Scores (in thousands) only based on historical items. (The higher, the better.)
  • 9. B. Temporal Matrix Factorization Matrix Factorization To recommend new items Hybrid Matrix Factorization (HMF) Learn categorical features Temporal HMF (THMF) Re-weight loss of HMF by time
  • 10. Hybrid Matrix Factorization Users/Items are represented as sums of feature embedding. (b: bias.) User-item score is given by inner product Model is trained by minimizing the loss
  • 11. Temporal Hybrid Matrix Factorization A non-negative weight associated with time is placed in the loss captures contribution of interactions over time. Zero weights in reduce training set size as well. ● Computeing . ○ can be learned jointly with other embedding parameters. ○ in our experiment are fixed as learned weights in Model A
  • 13. C. Sequence Modeling USER 1: ITEM 93, ITEM 5, …, ITEM 27 (-> ??, ??, ??) Tools: • Encoder(users)-Decoder(items) framework: next item recommendation is based on both user and previous items • LSTM to model ‘user encoding’ and ‘item transition’ • Embedding layer to incorporate feature learning USER 8: ITEM 55, ITEM 24, …, ITEM 5 (-> ??, ??, ??) ... USER 65: ITEM 47, ITEM 7, …, ITEM 62 (-> ??, ??, ??) Sequence of items ordered by time:
  • 15. Novel Extension to Model • Features • Continuous embedding is used to learn categorical features • New layer (look-up table and concatenation) is used connect input and RNN cells • Anonymous users • Item IDs are treated as categorical features • User IDs are removed to prevent overfitting • Sampling and data augmentation • No sampling • Original sequence gives better empirical results
  • 17. Does Sequence Help? Implicit assumption: sequence or order provides additional information beyond that provided by item frequency alone Experiment: Original sequence Sub-sequence sampling
  • 18. Does Sequence Help? Implicit assumption: sequence or order provides additional information beyond that provided by item frequency alone Experiment: Original sequence Sub-sequence sampling
  • 19. Approach Summary Temporal Learning A. Temporal based Ranking B. Temporal MF Wins over non-temporal counterpart significantly. Sequence Modeling C. LSTM based Encoder-Decoder model. Beats the best MF model. Ensemble Methods • Linear Regression • Random Forests
  • 20. Conclusion • History based model attains the best individual score • Proposed RNN-based model outperforms the commonly used matrix factorization • Ensemble based model obtained 5th place in the competition Component History HMF THMF LSTM Ensemble Score (thousands) 524 312 366 391 613
  • 21. Q & A Thanks you!
  • 22.
  • 24. Outline RecSys Challenge 2016 Approach Overview Temporal Learning Sequence modeling Experiments Conclude