Crossing Minds, profile picture
Uploaded byCrossing Minds
PDF, PPTX701 views

Recommender Systems from A to Z – Model Evaluation

The document outlines the process of building recommender systems, covering key areas such as dataset selection, model training, evaluation metrics, and deployment. It details various loss functions and optimization strategies, alongside practical recommendations for addressing overfitting and hyperparameter optimization. Additionally, it emphasizes the importance of user feedback and comparing different model performances using multiple metrics.

Embed presentation

Download as PDF, PPTX
Recommender Systems from A to Z Part 1: The Right Dataset Part 2: Model Training Part 3: Model Evaluation Part 4: Real-Time Deployment
Recommender Systems from A to Z Part 1: The Right Dataset Part 2: Model Training Part 3: Model Evaluation Part 4: Real-Time Deployment
1. Introduction Train/Valid split, underfitting and overfitting Learning Curve in recommendation engines 2. Evaluation functions Basic metrics for recommender engines (Precision, Recall, TPR, TNR...) Regression, Classification, Ranking metrics 3. Loss functions Optimization problems and Losses functions properties Regression, Classification, Ranking losses 4. Practical recommendations Regularization, HP optimization, Embeddings evaluations.
Introduction
Previous Meetup Recap: Recommendation Engine Types Recommendation engine Content-based Collaborative-filtering Hybrid engine Memory-based Model-based Item-Item User-User User-Item Model When? Problem definition Solutions strategies Content-based Item Cold start Least Square, Deep Learning Item-Item n_users >> n_items Affinity Matrix User-User n_user << n_items KNN, Affinity Matrix User-Item Better performance Matrix Factorization, Deep Learning
Previous Meetup Recap: Recommendation Engine Models Global Avg User Avg Item-Item Linear Linear + Reg Matrix Fact Deep Learning Domains Baseline Baseline users >> items Known “I” Known “I” Unknown “I” Extra datasets Model Complexity Trivial Trivial Simple Linear Linear Linear Non-linear Time Complexity + + +++ ++++ ++++ ++++ ++ Overfit/Underfit Underfit Underfit May Underfit May Overfit May Perform Bad May Overfit Can Overfit Hyper-Params 0 0 0 1 2 2–3 many Implementation Numpy Numpy Numpy Numpy Numpy LightFM, Spark NNet libraries
Optimization Problem – Matrix Factorization Example (or R)
Optimization Problem – Matrix Factorization Example Optimization problem (definitions) Sparse matrix of ratings with m users and n items Dense matrix of users embeddings Dense matrix of items embeddings
Optimization Problem – Matrix Factorization Example Optimization problem (definitions) Ratings of User #1 Embedding of User #1 Embedding of Item #1 Sparse matrix of ratings with m users and n items Dense matrix of users embeddings Dense matrix of items embeddings Ratings of User #m To Item #n
Optimization Problem – Matrix Factorization Example Optimization problem (definitions) AVAILABLE DATASET ? ? Sparse matrix of ratings with m users and n items Dense matrix of users embeddings Dense matrix of items embeddings
Optimization Problem – Matrix Factorization Example Our goal is to find U and I, such as the difference between each datapoint in R and and the product between each user and item is minimal. (or R)
Optimization Problem – Matrix Factorization Example Our goal is to find U and I, such as the difference between each datapoint in R and and the product between each user and item is minimal. (or R) 3. How are we going to solve the problem? 2. What properties are we looking in our outputs? - Exact rating vs like/dislike vs ranking predictions 1. What type of data do we have?
Ask the Right Questions (1) What type of data do we have? (2) What properties are we looking in our outputs? (3) How are we going to solve the problem? (4) Which hyper-parameters of my model are the best? (5) Which model is the best? Business decisions Technical decisions
Ask the Right Questions (1) What type of data do we have? (2) What properties are we looking in our outputs? (3) How are we going to solve the problem? (4) Which hyper-parameters of my model are the best? (5) Which model is the best? EVALUATION FUNCTIONS LOSS FUNCTIONS RANDOM SEARCH, GP COMPARE METRICS ML FOR RECOMMENDATION Business decisions Technical decisions
Objectives Types (from data point of view) Classification ● clic/no-click ● like/dislike/missing ● estimated probability of like (e.g. watch time) Regression ● absolute rating (e.g. from 1/5 to 5/5) ● number of interactions Ranking ● estimated order of preference (e.g. watch time) ● pairwise comparisons Unsupervised ● clustering of items ● clustering of users
Choosing the Right Objective (from business point of view) Absolute Predictions vs Relative Predictions Does only the order of the predictions matter? Sensitivity vs Specificity Is false positive worst than false negative? Skewness Is misclassifying an all-star favorite worst than misclassifying a casual like?
Choosing the Right Objective (from business point of view) Absolute Predictions vs Relative Predictions Does only the order of the predictions matter? Sensitivity vs Specificity Is false positive worst than false negative? Skewness Is misclassifying an all-star favorite worst than misclassifying a casual like? LOSS FUNCTION THAT PENALIZE MORE ERRORS IN ALL-STAR RATING RANKING LOSS FUNCTION CLASSIFICATION LOSS FUNCTION
Cross Validation – In Traditional Machine Learning 1 2 3 4 4 1 2 3 3 4 1 2 2 3 4 1
Cross Validation – In Recommendation Engines Dataset
Cross Validation – In Recommendation Engines Split such as every user is present in train and valid More stronger: split as every user have 80/20 train and valid Dataset
Underfitting and Overfitting
Underfitting and Overfitting Model fails to learn relations in data Model is a good fit for the data Model fails to generalize New samples New samples New samples + Complex
Underfitting and Overfitting Validation Sample + Complex
Underfitting and Overfitting Validation Sample + Complex OverfittingUnderfitting
Underfitting and Overfitting epoch Loss Function or Metric Mini-Batch Gradient Descent for epoch in n_epochs: ● shuffle the batches ● for batch in n_batches: ○ compute the predictions for the batch ○ compute the error for the batch ○ compute the gradient for the batch ○ update the parameters of the model ● plot error vs epoch
Underfitting and Overfitting A very simple way of checking underfitting Ground truth Y Model predictions Model is predicting always the same Predicted Y Underfitting
Evaluation Functions
What do we want to evaluate? Classification ● True Positive Rate (TPR) ● True Negative Rate (TNR) ● Precision ● F-measure Regression ● Mean Square Error (MSE) Ranking ● Recall@K ● Precision@K ● CG, DCG, nDCG Ranking/Classification metrics ● AUC Some common evaluation functions
Regression Mean Square Error (MSE) ● Easy to compute ● Linear gradient ● Can also be used as loss function Mean Absolute Error (MAE) ● Easy to compute ● Easy to interpret ● Discontinuous gradient ● Can’t be used as loss function
Classification – Precision vs Recall TS = Toy Story KP = Kung Fu Panda TD = How to train your dragon A = Annabelle Model 1 Model 2 TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2 User’s likes User’s dislikes Model recommendations TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2
Classification – Precision vs Recall TS = Toy Story KP = Kung Fu Panda TD = How to train your dragon A = Annabelle User’s likes User’s dislikes Model recommendations Recall = 5/7 Precision = 5/5 = 1 Recall = 7/7 = 1 Precision = 7/9 Model 1 Model 2 TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2 TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2
Classification 1/2 True Positive Rate (a.k.a TPR, Recall, Sensitivity) ● Easy to understand ● Useful for likes/dislikes datasets ● Measure of global bias of a model ● 0 <= TPR <=1 (higher is better) True Negative Rate (a.k.a TNR, Selectivity, Specificity) ● Easy to understand ● Useful for likes/dislikes datasets ● Measure of global bias of a model ● 0 <= TNR <=1 (higher is better)
Classification 2/2 Precision ● Easy to understand ● Useful for likes/dislikes datasets ● Measure quality of recommendation ● 0 <= Precision <=1 (higher is better) F-measure ● Balance precision and recall ● Not good for recommendation, because doesn’t take into account True Negatives ● 0 <= F-measure <= 1 (higher is better)
Ranking 1/3 Recall@K ● Count the positive items of the top K items predicted for each user ● Divides that number by the number of positive items for each user ● A perfect score is 1 if the user has K or less positive items and they all appear in the predicted top K ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = ? TOP K Positive = ? Total Positive = ? Recall@K = ?
Ranking 1/3 Recall@K ● Count the positive items of the top K items predicted for each user ● Divides that number by the number of positive items for each user ● A perfect score is 1 if the user has K or less positive items and they all appear in the predicted top K ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = {TS1, TS2, A1} TOP K Positive = {TS1, TS2} = 2 Total Positive = 4 Recall@K = 2 / 4 top 1 top 2 top 3
Ranking 1/3 Recall@K ● In math terms:
Ranking 2/3 Precision@K ● Count the positive items of the top K items predicted for each user ● Divides that number by K for each user ● A perfect score is 1 if the user has K or more positive items and the top K only contains positives ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = ? TOP K Positive = ? Recall@K = ?
Ranking 2/3 Precision@K ● Count the positive items of the top K items predicted for each user ● Divides that number by K for each user ● A perfect score is 1 if the user has K or more positive items and the top K only contains positives ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = {TS1, TS2, A1} TOP K Positive = {TS1, TS2} = 2 Recall@K = 2 / 3 top 1 top 2 top 3
Ranking 2/3 Precision@K ● In math terms:
Ranking 3/3 CG, DCG, and nDCG ● CG: Sum the true ratings of the Top K items predicted for each user ● DCG: Weight by position in Top K; nDCG: Normalize in [0, 1] ● A perfect score is 1 if the ranking of the prediction is the same as the ranking of the true ratings ● The bigger the score the better Movie Toy Story 1 1 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = ? CG = ? DCG = ?
Ranking 3/3 CG, DCG, and nDCG ● CG: Sum the true ratings of the Top K items predicted for each user ● DCG: Weight by position in Top K; nDCG: Normalize in [0, 1] ● A perfect nDCG is 1 if the ranking of the prediction is the same as the ranking of the true ratings ● The bigger the score the better Movie Toy Story 1 1 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = {TS1, TS2, A1} CG = 1.0 + 0.9 - 0.2 DCG = 1/1 + 0.9/2 - 0.2/3 top 1 top 2 top 3
Ranking 3/3 CG, DCG, and nDCG ● CG: Sum the true ratings of the Top K items predicted for each user ● DCG: Weight by position in Top K; nDCG: Normalize in [0, 1] ● A perfect nDCG is 1 if the ranking of the prediction is the same as the ranking of the true ratings
Hybrid Ranking/Classification AUC ● Vary positive prediction threshold (not just 0) ● Compute TPR and FPR for all possible positive thresholds ● Build Receiver Operating Characteristic (ROC) curve ● Integrate Area Under the ROC Curve (AUC)
Loss functions
Loss Functions vs Evaluation Functions Evaluation Metrics ● Expensive to evaluate ● Often not smooth ● Often not even derivable Loss Functions ● Smooth approximations of your evaluation metric ● Well suited for SGD
Loss Functions: How we are going to solve the problem? Classification loss ● Logistic ● Cross Entropy ● Kullback-Leibler Divergence Regression loss ● Mean Square Error (MSE) Ranking loss ● WARP ● BPR Some common loss functions
Optimization Problems – Basic Formulation with RMSE Goal: find U and I s.t. the difference between each datapoint in R and and the product between each user and item is minimal (or R)
Optimization Problems – General Formulation Goal: find U and I s.t. the loss function J is minimized. (or R)
Convex vs Non-Convex Optimization Convex Non-convex
Convex Optimization
Non-Convex Optimization
Loss Functions – Regression Mean Square Error ● Typical used loss function for regression. It’s a smooth function. It’s easy to understand. Regularized Mean Square Error ● Mean square error plus regularization to avoid overfitting.
Loss Functions – Classification Logistic ● Typical used loss function for classification. Smooth gradient around zero and steep for large errors.
Loss Functions – Classification Logistic
Loss Functions – Ranking Weighted Approximate-Rank Pairwise (WARP) ● Approximates DCG-like evaluation metrics ● Smooth and tractable computation Bayesian Personalised Ranking (BPR) ● Approximates AUC ● Smooth and tractable computation ● Requires binary comparisons (good for binary comparison feedback)
Practical Recommendations
Practical Recommendations (1) Always compute baseline metrics (2) Always analyze underfitting vs overfitting (3) Always do hyperparameter optimization (4) Always compute multiple metrics for your models (5) Always analyze the clustering properties of the items/users (6) Always ask feedback from end users
Practical Recommendations (1) Always compute baseline metrics (2) Always analyze underfitting vs overfitting (3) Always do hyperparameter optimization (4) Always compute multiple metrics for your models (5) Always analyze the clustering properties of the items/users (6) Always ask feedback from end users COMPARE WITH GLOBAL MODELS IT’S EASY IF OVERFITTING, USE REGULARIZATION GRID SEARCH OR GAUSSIAN PROCESS TPR, TNR, PRECISION, ETC. ITEM/ITEM SIMILARITIES EVERYTHING IS ABOUT USER TASTE
(1) Always compute baseline metrics Global Avg User Avg Item-Item Linear Linear + Reg Matrix Fact Deep Learning Domains Baseline Baseline users >> items Known “I” Known “I” Unknown “I” Extra datasets Model Complexity Trivial Trivial Simple Linear Linear Linear Non-linear Time Complexity + + +++ ++++ ++++ ++++ ++ Overfit/Underfit Underfit Underfit May Underfit May Overfit May Perform Bad May Overfit Can Overfit Hyper-Params 0 0 0 1 2 2–3 many Implementation Numpy Numpy Numpy Numpy Numpy LightFM, Spark NNet libraries
(2) Always analyze underfitting vs overfitting Model-based ● Dropout ● Bagging Loss-based normalization ● norm: best approximation of sparsity-inducing norm ● norm: very smooth, easy to optimize Data Augmentation ● Negative Sampling
(3) Always do hyperparameter optimization Grid Search Brute force over all the combinations of the parameters Exponential cost: for 20 parameters, to get only 10 evaluations each, you need 10^20 complete runs Random Search Uniformly sample combinations of the parameters Very easy to implement, very useful in practice Gaussian Process Optimization Meta-learning of the validation error given hyper-parameters Solve exploration/exploitation tradeoff
(3) Always do hyperparameter optimization Metric to minimize Metric to maximize
(4) Always compute multiple metrics for your models
(5) Always analyze the clustering properties of the items/users Items embeddings ● In general, we combine items embeddings with: FEATURES | IMAGE EMBS | NLP EMBS ● After getting the embeddings, we always compute Top-K similarities in well known items ● We use the items embeddings to create clusters and analyze how good they are
(5) Always ask for final users feedback RECOMMENDATION IS ALL ABOUT USERS TASTE ASK THEM FOR FEEDBACK!!
Conclusions
Losses and metrics summary table Name Category loss eval batch-SGD support implicit Comments MSE Regr ✓ ✓ ✓ ✓ linear gradient MAE Regr ✓ ✓ easy to interpret Logistic / XE / KL Classif ✓ ✓ ✓ ✓ flexible truth Exponential Classif ✓ ✓ exploding gradient Recall (global) Classif ✓ ✓ ✓ requires negative Precision (global) Classif ✓ ✓ ✓ requires negative F-measure (global) Classif ✓ ✓ ✓ requires negative MRR Ranking ✓ considers only 1 item nDCG Ranking ✓ requires rank WARP Ranking ✓ for nDCG, p@k, r@k AUC Hybrid ✓ ✓ ✓ requires negative BPR Hybrid ✓ ✓ for AUC Recall@k Hybrid ✓ requires≤k positives Precision@k Hybrid ✓ requires ≥k positives
Questions
Thank YOU!
Negative Sampling Problem ● Unary feedback: the best model will always predict “1” for each user and item. ● In general: ○ your model is used in real life to predict (user, item) outside sparse dataset. ○ can’t train on the full (#users x #items) dense matrix. Negative Sampling Solution ● unary→binary (e.g. click/missing) binary→ternary (e.g. like/dislike/missing) ● sample strategy matters a lot (i.e. how to split train and valid) ● how many negative samples matters a lot
Negative Sampling
Negative Sampling Split negative feedback in the same proportion
Underfitting and Overfitting – Take Home (1) For doing cross-validation split data such as almost all users are in training and validation (2) Use negative sampling to avoid overfitting in your models (3) Always use learning curves to get more insights about underfitting vs overfitting (4) Compute mean and variance of your predictions to get insights about underfitting vs overfitting
Loss Functions – Classification ● Equivalent to cross-entropy between the truth and the predicted probability (for 2-classes model) ● Equivalent to Kullback-Leibler divergence between the truth and the predicted probability ● Often used for deep-learning based recommendation engines ● Smooth gradient around zero and steep for large errors Logistic

More Related Content

PPTX
Recommender Systems from A to Z – Model Training
byCrossing Minds
 
PDF
Time, Context and Causality in Recommender Systems
byYves Raimond
 
PDF
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix Scale
byXavier Amatriain
 
PDF
Recommender Systems from A to Z – Real-Time Deployment
byCrossing Minds
 
PDF
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
byAnoop Deoras
 
PDF
Personalizing "The Netflix Experience" with Deep Learning
byAnoop Deoras
 
PDF
Missing values in recommender models
byParmeshwar Khurd
 
PDF
Calibrated Recommendations
byHarald Steck
 
Recommender Systems from A to Z – Model Training
byCrossing Minds
 
Time, Context and Causality in Recommender Systems
byYves Raimond
 
Qcon SF 2013 - Machine Learning & Recommender Systems @ Netflix Scale
byXavier Amatriain
 
Recommender Systems from A to Z – Real-Time Deployment
byCrossing Minds
 
Tutorial on Deep Learning in Recommender System, Lars summer school 2019
byAnoop Deoras
 
Personalizing "The Netflix Experience" with Deep Learning
byAnoop Deoras
 
Missing values in recommender models
byParmeshwar Khurd
 
Calibrated Recommendations
byHarald Steck
 

What's hot

PDF
Sequential Decision Making in Recommendations
byJaya Kawale
 
PDF
Boston ML - Architecting Recommender Systems
byJames Kirk
 
PPTX
Recommender systems using collaborative filtering
byD Yogendra Rao
 
PDF
How to build a recommender system?
byblueace
 
PDF
Recent Trends in Personalization at Netflix
byJustin Basilico
 
PDF
Past, present, and future of Recommender Systems: an industry perspective
byXavier Amatriain
 
PPTX
Recommender system
byNilotpal Pramanik
 
PPTX
Recommendation system
byRishabh Mehta
 
PPTX
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...
byJustin Basilico
 
PDF
Overview of recommender system
byStanley Wang
 
PDF
Homepage Personalization at Spotify
byOguz Semerci
 
PDF
Recommender system algorithm and architecture
byLiang Xiang
 
PDF
Personalizing the listening experience
byMounia Lalmas-Roelleke
 
PPTX
Recommendation system
bySAIFUR RAHMAN
 
PDF
Past, Present & Future of Recommender Systems: An Industry Perspective
byJustin Basilico
 
PDF
RecSysOps: Best Practices for Operating a Large-Scale Recommender System
byEhsan38
 
PDF
Kdd 2014 Tutorial - the recommender problem revisited
byXavier Amatriain
 
PDF
Deep Learning for Recommender Systems
byYves Raimond
 
PDF
Engagement, metrics and "recommenders"
byMounia Lalmas-Roelleke
 
PPT
Item Based Collaborative Filtering Recommendation Algorithms
bynextlib
 
Sequential Decision Making in Recommendations
byJaya Kawale
 
Boston ML - Architecting Recommender Systems
byJames Kirk
 
Recommender systems using collaborative filtering
byD Yogendra Rao
 
How to build a recommender system?
byblueace
 
Recent Trends in Personalization at Netflix
byJustin Basilico
 
Past, present, and future of Recommender Systems: an industry perspective
byXavier Amatriain
 
Recommender system
byNilotpal Pramanik
 
Recommendation system
byRishabh Mehta
 
Is that a Time Machine? Some Design Patterns for Real World Machine Learning ...
byJustin Basilico
 
Overview of recommender system
byStanley Wang
 
Homepage Personalization at Spotify
byOguz Semerci
 
Recommender system algorithm and architecture
byLiang Xiang
 
Personalizing the listening experience
byMounia Lalmas-Roelleke
 
Recommendation system
bySAIFUR RAHMAN
 
Past, Present & Future of Recommender Systems: An Industry Perspective
byJustin Basilico
 
RecSysOps: Best Practices for Operating a Large-Scale Recommender System
byEhsan38
 
Kdd 2014 Tutorial - the recommender problem revisited
byXavier Amatriain
 
Deep Learning for Recommender Systems
byYves Raimond
 
Engagement, metrics and "recommenders"
byMounia Lalmas-Roelleke
 
Item Based Collaborative Filtering Recommendation Algorithms
bynextlib
 

Similar to Recommender Systems from A to Z – Model Evaluation

PDF
Introduction to behavior based recommendation system
byKimikazu Kato
 
PPTX
Lessons learnt at building recommendation services at industry scale
byDomonkos Tikk
 
PPTX
Recommendation system
byDing Li
 
PPTX
Collaborative Filtering Recommendation System
byMilind Gokhale
 
PDF
Tutorial: Context In Recommender Systems
byYONG ZHENG
 
PDF
Recommendation System --Theory and Practice
byKimikazu Kato
 
PDF
SIGIR 2018 - From the Probability Ranking Principle to the Low Prior Discover...
byRocío Cañamares
 
PDF
IJCAI Workshop Presentation
byNeal Lathia
 
PDF
Ensemble Methods and Recommender Systems
byrosni
 
PDF
C3_W2.pdf
byShaheenKolimi
 
PPTX
Recommender Systems from A to Z – The Right Dataset
byCrossing Minds
 
PDF
Lecture 10 - Model Testing and Evaluation, a lecture in subject module Statis...
byManinda Edirisooriya
 
PPTX
RS NAIVE BAYES ASSOCIATION RULE MINING AND BLACK BOX
bykhsbharadwaj123
 
PDF
Recommender Systems, Matrices and Graphs
byRoelof Pieters
 
PDF
Real-world News Recommender Systems
bykib_83
 
PPTX
Recommender Systems: Advances in Collaborative Filtering
byChangsung Moon
 
PPT
Recommender systems
byTamer Rezk
 
PPTX
Data Summit 2022 – How do I Know My Machine Learning Data Model is Good?
byDatavail
 
PDF
Building your first recommender - Jettro Coenradie - Codemotion Amsterdam 2018
byCodemotion
 
PDF
Netflix Recommendations - Beyond the 5 Stars
byXavier Amatriain
 
Introduction to behavior based recommendation system
byKimikazu Kato
 
Lessons learnt at building recommendation services at industry scale
byDomonkos Tikk
 
Recommendation system
byDing Li
 
Collaborative Filtering Recommendation System
byMilind Gokhale
 
Tutorial: Context In Recommender Systems
byYONG ZHENG
 
Recommendation System --Theory and Practice
byKimikazu Kato
 
SIGIR 2018 - From the Probability Ranking Principle to the Low Prior Discover...
byRocío Cañamares
 
IJCAI Workshop Presentation
byNeal Lathia
 
Ensemble Methods and Recommender Systems
byrosni
 
C3_W2.pdf
byShaheenKolimi
 
Recommender Systems from A to Z – The Right Dataset
byCrossing Minds
 
Lecture 10 - Model Testing and Evaluation, a lecture in subject module Statis...
byManinda Edirisooriya
 
RS NAIVE BAYES ASSOCIATION RULE MINING AND BLACK BOX
bykhsbharadwaj123
 
Recommender Systems, Matrices and Graphs
byRoelof Pieters
 
Real-world News Recommender Systems
bykib_83
 
Recommender Systems: Advances in Collaborative Filtering
byChangsung Moon
 
Recommender systems
byTamer Rezk
 
Data Summit 2022 – How do I Know My Machine Learning Data Model is Good?
byDatavail
 
Building your first recommender - Jettro Coenradie - Codemotion Amsterdam 2018
byCodemotion
 
Netflix Recommendations - Beyond the 5 Stars
byXavier Amatriain
 

Recently uploaded

PPTX
CLASSIFICATION OF LIVING ORGANISMS: THE THREE DOMAINS OF LIFE
byHOME
 
PDF
CosmologywithVoidsfromtheNancyGraceRomanSpaceTelescope
bySérgio Sacani
 
PPTX
Metabolism-Electron Transport Chain and Oxidative Phosphorylation.pptx
byDepartment of Biochemistry, Dr.N.G.P Arts and Science College, Coimbatore
 
PDF
Multiple outflows and delayed ejections revealed by early imaging of novae
bySérgio Sacani
 
PDF
glyoproteins and its types n o gpi linked pdf.pdf
byrutikbait012
 
PDF
Cognitive Architectures: Information Compression, Meaning-Making, and Drift i...
byCognitive Drift Institute
 
PPTX
UNIT -5 PPT PP-1.pptx buffer second year
byyashchaudhari2256
 
PDF
Ch3-phys701-Maram optics for 1234566.pdf
byssuserd90459
 
PPTX
Quality of irrigation water and brackish water management.pptx
bySaurabh Upadhyay
 
PDF
Kaplan and Sadock's Comprehensive Text of Psychiatry 11th Edition PDF
bymokotey464
 
DOCX
Siwes technical report done at hospital2025
bykm9056517
 
PDF
BP504T PHARMACOGNOSY UNIT 02 SUMMARY PART 01
byAnubhav Gupta
 
PPTX
Lecture 15 hospital acquired infection.pptx
byAmri559698
 
PPTX
FILTERATION filter aid and filter media.pptx
byyashchaudhari2256
 
PPTX
Epigenetics (Protein translation, Genetic Code, Steps involved in Translation...
byRashmiMG2
 
PDF
Amino acid cost and supply chain analysis for cultivated meat
byThe Good Food Institute
 
PPT
1-2_SterilisationValidationQualification.ppt
byUrvashiPatni1
 
PPTX
ASTHMA. CHE (1).pptx- diagnosis and management
bymbchb420221
 
PPTX
Physical Science SHS 24.1 Special and General Theory of Relativity.pptx
byJessonSuaga1
 
PDF
AThickVolatileAtmosphereontheUltrahotSuper-EarthTOI-561b
bySérgio Sacani
 
CLASSIFICATION OF LIVING ORGANISMS: THE THREE DOMAINS OF LIFE
byHOME
 
CosmologywithVoidsfromtheNancyGraceRomanSpaceTelescope
bySérgio Sacani
 
Metabolism-Electron Transport Chain and Oxidative Phosphorylation.pptx
byDepartment of Biochemistry, Dr.N.G.P Arts and Science College, Coimbatore
 
Multiple outflows and delayed ejections revealed by early imaging of novae
bySérgio Sacani
 
glyoproteins and its types n o gpi linked pdf.pdf
byrutikbait012
 
Cognitive Architectures: Information Compression, Meaning-Making, and Drift i...
byCognitive Drift Institute
 
UNIT -5 PPT PP-1.pptx buffer second year
byyashchaudhari2256
 
Ch3-phys701-Maram optics for 1234566.pdf
byssuserd90459
 
Quality of irrigation water and brackish water management.pptx
bySaurabh Upadhyay
 
Kaplan and Sadock's Comprehensive Text of Psychiatry 11th Edition PDF
bymokotey464
 
Siwes technical report done at hospital2025
bykm9056517
 
BP504T PHARMACOGNOSY UNIT 02 SUMMARY PART 01
byAnubhav Gupta
 
Lecture 15 hospital acquired infection.pptx
byAmri559698
 
FILTERATION filter aid and filter media.pptx
byyashchaudhari2256
 
Epigenetics (Protein translation, Genetic Code, Steps involved in Translation...
byRashmiMG2
 
Amino acid cost and supply chain analysis for cultivated meat
byThe Good Food Institute
 
1-2_SterilisationValidationQualification.ppt
byUrvashiPatni1
 
ASTHMA. CHE (1).pptx- diagnosis and management
bymbchb420221
 
Physical Science SHS 24.1 Special and General Theory of Relativity.pptx
byJessonSuaga1
 
AThickVolatileAtmosphereontheUltrahotSuper-EarthTOI-561b
bySérgio Sacani
 

Recommender Systems from A to Z – Model Evaluation

  • 2.
    Recommender Systems fromA to Z Part 1: The Right Dataset Part 2: Model Training Part 3: Model Evaluation Part 4: Real-Time Deployment
  • 3.
    Recommender Systems fromA to Z Part 1: The Right Dataset Part 2: Model Training Part 3: Model Evaluation Part 4: Real-Time Deployment
  • 4.
    1. Introduction Train/Valid split,underfitting and overfitting Learning Curve in recommendation engines 2. Evaluation functions Basic metrics for recommender engines (Precision, Recall, TPR, TNR...) Regression, Classification, Ranking metrics 3. Loss functions Optimization problems and Losses functions properties Regression, Classification, Ranking losses 4. Practical recommendations Regularization, HP optimization, Embeddings evaluations.
  • 5.
  • 6.
    Previous Meetup Recap:Recommendation Engine Types Recommendation engine Content-based Collaborative-filtering Hybrid engine Memory-based Model-based Item-Item User-User User-Item Model When? Problem definition Solutions strategies Content-based Item Cold start Least Square, Deep Learning Item-Item n_users >> n_items Affinity Matrix User-User n_user << n_items KNN, Affinity Matrix User-Item Better performance Matrix Factorization, Deep Learning
  • 7.
    Previous Meetup Recap:Recommendation Engine Models Global Avg User Avg Item-Item Linear Linear + Reg Matrix Fact Deep Learning Domains Baseline Baseline users >> items Known “I” Known “I” Unknown “I” Extra datasets Model Complexity Trivial Trivial Simple Linear Linear Linear Non-linear Time Complexity + + +++ ++++ ++++ ++++ ++ Overfit/Underfit Underfit Underfit May Underfit May Overfit May Perform Bad May Overfit Can Overfit Hyper-Params 0 0 0 1 2 2–3 many Implementation Numpy Numpy Numpy Numpy Numpy LightFM, Spark NNet libraries
  • 8.
    Optimization Problem –Matrix Factorization Example (or R)
  • 9.
    Optimization Problem – MatrixFactorization Example Optimization problem (definitions) Sparse matrix of ratings with m users and n items Dense matrix of users embeddings Dense matrix of items embeddings
  • 10.
    Optimization Problem –Matrix Factorization Example Optimization problem (definitions) Ratings of User #1 Embedding of User #1 Embedding of Item #1 Sparse matrix of ratings with m users and n items Dense matrix of users embeddings Dense matrix of items embeddings Ratings of User #m To Item #n
  • 11.
    Optimization Problem –Matrix Factorization Example Optimization problem (definitions) AVAILABLE DATASET ? ? Sparse matrix of ratings with m users and n items Dense matrix of users embeddings Dense matrix of items embeddings
  • 12.
    Optimization Problem –Matrix Factorization Example Our goal is to find U and I, such as the difference between each datapoint in R and and the product between each user and item is minimal. (or R)
  • 13.
    Optimization Problem –Matrix Factorization Example Our goal is to find U and I, such as the difference between each datapoint in R and and the product between each user and item is minimal. (or R) 3. How are we going to solve the problem? 2. What properties are we looking in our outputs? - Exact rating vs like/dislike vs ranking predictions 1. What type of data do we have?
  • 14.
    Ask the RightQuestions (1) What type of data do we have? (2) What properties are we looking in our outputs? (3) How are we going to solve the problem? (4) Which hyper-parameters of my model are the best? (5) Which model is the best? Business decisions Technical decisions
  • 15.
    Ask the RightQuestions (1) What type of data do we have? (2) What properties are we looking in our outputs? (3) How are we going to solve the problem? (4) Which hyper-parameters of my model are the best? (5) Which model is the best? EVALUATION FUNCTIONS LOSS FUNCTIONS RANDOM SEARCH, GP COMPARE METRICS ML FOR RECOMMENDATION Business decisions Technical decisions
  • 16.
    Objectives Types (fromdata point of view) Classification ● clic/no-click ● like/dislike/missing ● estimated probability of like (e.g. watch time) Regression ● absolute rating (e.g. from 1/5 to 5/5) ● number of interactions Ranking ● estimated order of preference (e.g. watch time) ● pairwise comparisons Unsupervised ● clustering of items ● clustering of users
  • 17.
    Choosing the RightObjective (from business point of view) Absolute Predictions vs Relative Predictions Does only the order of the predictions matter? Sensitivity vs Specificity Is false positive worst than false negative? Skewness Is misclassifying an all-star favorite worst than misclassifying a casual like?
  • 18.
    Choosing the RightObjective (from business point of view) Absolute Predictions vs Relative Predictions Does only the order of the predictions matter? Sensitivity vs Specificity Is false positive worst than false negative? Skewness Is misclassifying an all-star favorite worst than misclassifying a casual like? LOSS FUNCTION THAT PENALIZE MORE ERRORS IN ALL-STAR RATING RANKING LOSS FUNCTION CLASSIFICATION LOSS FUNCTION
  • 19.
    Cross Validation –In Traditional Machine Learning 1 2 3 4 4 1 2 3 3 4 1 2 2 3 4 1
  • 20.
    Cross Validation –In Recommendation Engines Dataset
  • 21.
    Cross Validation –In Recommendation Engines Split such as every user is present in train and valid More stronger: split as every user have 80/20 train and valid Dataset
  • 22.
  • 23.
    Underfitting and Overfitting Modelfails to learn relations in data Model is a good fit for the data Model fails to generalize New samples New samples New samples + Complex
  • 24.
  • 25.
    Underfitting and Overfitting Validation Sample +Complex OverfittingUnderfitting
  • 26.
    Underfitting and Overfitting epoch Loss Function or Metric Mini-BatchGradient Descent for epoch in n_epochs: ● shuffle the batches ● for batch in n_batches: ○ compute the predictions for the batch ○ compute the error for the batch ○ compute the gradient for the batch ○ update the parameters of the model ● plot error vs epoch
  • 27.
    Underfitting and Overfitting Avery simple way of checking underfitting Ground truth Y Model predictions Model is predicting always the same Predicted Y Underfitting
  • 28.
  • 29.
    What do wewant to evaluate? Classification ● True Positive Rate (TPR) ● True Negative Rate (TNR) ● Precision ● F-measure Regression ● Mean Square Error (MSE) Ranking ● Recall@K ● Precision@K ● CG, DCG, nDCG Ranking/Classification metrics ● AUC Some common evaluation functions
  • 30.
    Regression Mean Square Error(MSE) ● Easy to compute ● Linear gradient ● Can also be used as loss function Mean Absolute Error (MAE) ● Easy to compute ● Easy to interpret ● Discontinuous gradient ● Can’t be used as loss function
  • 31.
    Classification – Precisionvs Recall TS = Toy Story KP = Kung Fu Panda TD = How to train your dragon A = Annabelle Model 1 Model 2 TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2 User’s likes User’s dislikes Model recommendations TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2
  • 32.
    Classification – Precisionvs Recall TS = Toy Story KP = Kung Fu Panda TD = How to train your dragon A = Annabelle User’s likes User’s dislikes Model recommendations Recall = 5/7 Precision = 5/5 = 1 Recall = 7/7 = 1 Precision = 7/9 Model 1 Model 2 TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2 TS1 TS2 TS3 KP1 KP2 TS4 KP3 A1 A2
  • 33.
    Classification 1/2 True PositiveRate (a.k.a TPR, Recall, Sensitivity) ● Easy to understand ● Useful for likes/dislikes datasets ● Measure of global bias of a model ● 0 <= TPR <=1 (higher is better) True Negative Rate (a.k.a TNR, Selectivity, Specificity) ● Easy to understand ● Useful for likes/dislikes datasets ● Measure of global bias of a model ● 0 <= TNR <=1 (higher is better)
  • 34.
    Classification 2/2 Precision ● Easyto understand ● Useful for likes/dislikes datasets ● Measure quality of recommendation ● 0 <= Precision <=1 (higher is better) F-measure ● Balance precision and recall ● Not good for recommendation, because doesn’t take into account True Negatives ● 0 <= F-measure <= 1 (higher is better)
  • 35.
    Ranking 1/3 Recall@K ● Countthe positive items of the top K items predicted for each user ● Divides that number by the number of positive items for each user ● A perfect score is 1 if the user has K or less positive items and they all appear in the predicted top K ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = ? TOP K Positive = ? Total Positive = ? Recall@K = ?
  • 36.
    Ranking 1/3 Recall@K ● Countthe positive items of the top K items predicted for each user ● Divides that number by the number of positive items for each user ● A perfect score is 1 if the user has K or less positive items and they all appear in the predicted top K ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = {TS1, TS2, A1} TOP K Positive = {TS1, TS2} = 2 Total Positive = 4 Recall@K = 2 / 4 top 1 top 2 top 3
  • 37.
  • 38.
    Ranking 2/3 Precision@K ● Countthe positive items of the top K items predicted for each user ● Divides that number by K for each user ● A perfect score is 1 if the user has K or more positive items and the top K only contains positives ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = ? TOP K Positive = ? Recall@K = ?
  • 39.
    Ranking 2/3 Precision@K ● Countthe positive items of the top K items predicted for each user ● Divides that number by K for each user ● A perfect score is 1 if the user has K or more positive items and the top K only contains positives ● Independent of the exact values of the predictions, only their relative rank matters Movie Toy Story 1 1.0 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = {TS1, TS2, A1} TOP K Positive = {TS1, TS2} = 2 Recall@K = 2 / 3 top 1 top 2 top 3
  • 40.
  • 41.
    Ranking 3/3 CG, DCG,and nDCG ● CG: Sum the true ratings of the Top K items predicted for each user ● DCG: Weight by position in Top K; nDCG: Normalize in [0, 1] ● A perfect score is 1 if the ranking of the prediction is the same as the ranking of the true ratings ● The bigger the score the better Movie Toy Story 1 1 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = ? CG = ? DCG = ?
  • 42.
    Ranking 3/3 CG, DCG,and nDCG ● CG: Sum the true ratings of the Top K items predicted for each user ● DCG: Weight by position in Top K; nDCG: Normalize in [0, 1] ● A perfect nDCG is 1 if the ranking of the prediction is the same as the ranking of the true ratings ● The bigger the score the better Movie Toy Story 1 1 0.9 Toy Story 2 0.9 0.7 Kung Fu Panda 1 0.7 0.1 Kung Fu Panda 2 0.6 -0.1 Annabelle 1 -0.2 0.4 K = 3 TOP K = {TS1, TS2, A1} CG = 1.0 + 0.9 - 0.2 DCG = 1/1 + 0.9/2 - 0.2/3 top 1 top 2 top 3
  • 43.
    Ranking 3/3 CG, DCG,and nDCG ● CG: Sum the true ratings of the Top K items predicted for each user ● DCG: Weight by position in Top K; nDCG: Normalize in [0, 1] ● A perfect nDCG is 1 if the ranking of the prediction is the same as the ranking of the true ratings
  • 44.
    Hybrid Ranking/Classification AUC ● Varypositive prediction threshold (not just 0) ● Compute TPR and FPR for all possible positive thresholds ● Build Receiver Operating Characteristic (ROC) curve ● Integrate Area Under the ROC Curve (AUC)
  • 45.
  • 46.
    Loss Functions vsEvaluation Functions Evaluation Metrics ● Expensive to evaluate ● Often not smooth ● Often not even derivable Loss Functions ● Smooth approximations of your evaluation metric ● Well suited for SGD
  • 47.
    Loss Functions: Howwe are going to solve the problem? Classification loss ● Logistic ● Cross Entropy ● Kullback-Leibler Divergence Regression loss ● Mean Square Error (MSE) Ranking loss ● WARP ● BPR Some common loss functions
  • 48.
    Optimization Problems –Basic Formulation with RMSE Goal: find U and I s.t. the difference between each datapoint in R and and the product between each user and item is minimal (or R)
  • 49.
    Optimization Problems –General Formulation Goal: find U and I s.t. the loss function J is minimized. (or R)
  • 50.
    Convex vs Non-ConvexOptimization Convex Non-convex
  • 51.
  • 52.
  • 53.
    Loss Functions –Regression Mean Square Error ● Typical used loss function for regression. It’s a smooth function. It’s easy to understand. Regularized Mean Square Error ● Mean square error plus regularization to avoid overfitting.
  • 54.
    Loss Functions –Classification Logistic ● Typical used loss function for classification. Smooth gradient around zero and steep for large errors.
  • 55.
    Loss Functions –Classification Logistic
  • 56.
    Loss Functions –Ranking Weighted Approximate-Rank Pairwise (WARP) ● Approximates DCG-like evaluation metrics ● Smooth and tractable computation Bayesian Personalised Ranking (BPR) ● Approximates AUC ● Smooth and tractable computation ● Requires binary comparisons (good for binary comparison feedback)
  • 57.
  • 58.
    Practical Recommendations (1) Alwayscompute baseline metrics (2) Always analyze underfitting vs overfitting (3) Always do hyperparameter optimization (4) Always compute multiple metrics for your models (5) Always analyze the clustering properties of the items/users (6) Always ask feedback from end users
  • 59.
    Practical Recommendations (1) Alwayscompute baseline metrics (2) Always analyze underfitting vs overfitting (3) Always do hyperparameter optimization (4) Always compute multiple metrics for your models (5) Always analyze the clustering properties of the items/users (6) Always ask feedback from end users COMPARE WITH GLOBAL MODELS IT’S EASY IF OVERFITTING, USE REGULARIZATION GRID SEARCH OR GAUSSIAN PROCESS TPR, TNR, PRECISION, ETC. ITEM/ITEM SIMILARITIES EVERYTHING IS ABOUT USER TASTE
  • 60.
    (1) Always computebaseline metrics Global Avg User Avg Item-Item Linear Linear + Reg Matrix Fact Deep Learning Domains Baseline Baseline users >> items Known “I” Known “I” Unknown “I” Extra datasets Model Complexity Trivial Trivial Simple Linear Linear Linear Non-linear Time Complexity + + +++ ++++ ++++ ++++ ++ Overfit/Underfit Underfit Underfit May Underfit May Overfit May Perform Bad May Overfit Can Overfit Hyper-Params 0 0 0 1 2 2–3 many Implementation Numpy Numpy Numpy Numpy Numpy LightFM, Spark NNet libraries
  • 61.
    (2) Always analyzeunderfitting vs overfitting Model-based ● Dropout ● Bagging Loss-based normalization ● norm: best approximation of sparsity-inducing norm ● norm: very smooth, easy to optimize Data Augmentation ● Negative Sampling
  • 62.
    (3) Always dohyperparameter optimization Grid Search Brute force over all the combinations of the parameters Exponential cost: for 20 parameters, to get only 10 evaluations each, you need 10^20 complete runs Random Search Uniformly sample combinations of the parameters Very easy to implement, very useful in practice Gaussian Process Optimization Meta-learning of the validation error given hyper-parameters Solve exploration/exploitation tradeoff
  • 63.
    (3) Always dohyperparameter optimization Metric to minimize Metric to maximize
  • 64.
    (4) Always computemultiple metrics for your models
  • 65.
    (5) Always analyzethe clustering properties of the items/users Items embeddings ● In general, we combine items embeddings with: FEATURES | IMAGE EMBS | NLP EMBS ● After getting the embeddings, we always compute Top-K similarities in well known items ● We use the items embeddings to create clusters and analyze how good they are
  • 66.
    (5) Always askfor final users feedback RECOMMENDATION IS ALL ABOUT USERS TASTE ASK THEM FOR FEEDBACK!!
  • 67.
  • 68.
    Losses and metricssummary table Name Category loss eval batch-SGD support implicit Comments MSE Regr ✓ ✓ ✓ ✓ linear gradient MAE Regr ✓ ✓ easy to interpret Logistic / XE / KL Classif ✓ ✓ ✓ ✓ flexible truth Exponential Classif ✓ ✓ exploding gradient Recall (global) Classif ✓ ✓ ✓ requires negative Precision (global) Classif ✓ ✓ ✓ requires negative F-measure (global) Classif ✓ ✓ ✓ requires negative MRR Ranking ✓ considers only 1 item nDCG Ranking ✓ requires rank WARP Ranking ✓ for nDCG, p@k, r@k AUC Hybrid ✓ ✓ ✓ requires negative BPR Hybrid ✓ ✓ for AUC Recall@k Hybrid ✓ requires≤k positives Precision@k Hybrid ✓ requires ≥k positives
  • 69.
  • 70.
  • 71.
    Negative Sampling Problem ● Unaryfeedback: the best model will always predict “1” for each user and item. ● In general: ○ your model is used in real life to predict (user, item) outside sparse dataset. ○ can’t train on the full (#users x #items) dense matrix. Negative Sampling Solution ● unary→binary (e.g. click/missing) binary→ternary (e.g. like/dislike/missing) ● sample strategy matters a lot (i.e. how to split train and valid) ● how many negative samples matters a lot
  • 72.
  • 73.
    Negative Sampling Split negativefeedback in the same proportion
  • 74.
    Underfitting and Overfitting– Take Home (1) For doing cross-validation split data such as almost all users are in training and validation (2) Use negative sampling to avoid overfitting in your models (3) Always use learning curves to get more insights about underfitting vs overfitting (4) Compute mean and variance of your predictions to get insights about underfitting vs overfitting
  • 75.
    Loss Functions –Classification ● Equivalent to cross-entropy between the truth and the predicted probability (for 2-classes model) ● Equivalent to Kullback-Leibler divergence between the truth and the predicted probability ● Often used for deep-learning based recommendation engines ● Smooth gradient around zero and steep for large errors Logistic