SlideShare a Scribd company logo
1 of 52
Download to read offline
Background Motivation Model & Metric Experimental Setup Results Summary
Incorporating Clicks, Attention and Satisfaction
into a SERP Evaluation Model
Aleksandr Chuklin¶,§ Maarten de Rijke§
chuklin@google.com derijke@uva.nl
¶Google Research Europe
§University of Amsterdam
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 1
Background
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Main problem
Combining relevance of individual SERP items (Rk) into a
whole-page metric.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 3
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
document 3
document 4
document 1
document 2
document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
Precision at N:
P@N =
1
N
N
k=1
Rk
document 3
document 4
document 1
document 2
document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
Precision at N:
P@N =
1
N
N
k=1
Rk
Discounted Cumulative Gain (DCG):
DCG@N =
N
k=1
1
log2 (1 + k)
· Rk
document 3
document 4
document 1
document 2
document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
Precision at N:
P@N =
1
N
N
k=1
Rk
Discounted Cumulative Gain (DCG):
DCG@N =
N
k=1
1
log2 (1 + k)
· Rk
Model-Based Metrics (Chuklin et al. 2013):
Utility@N =
N
k=1
P(Ck = 1) · Rk
document 3
document 4
document 1
document 2
document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Main Goal of This Paper
Better measure for SERP utility
Namely, improve this (Chuklin et al. 2013):
N
k=1
P(Ck = 1) · Rk
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 5
Motivation
Background Motivation Model & Metric Experimental Setup Results Summary
Complex Heterogeneous SERPs
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 7
Background Motivation Model & Metric Experimental Setup Results Summary
Motivation 1: Non-Trivial Attention Patterns
4
ement
9
1
3
5
6
7
8
4
2
(c) Mouse Data
data. The session sequence for this data would be
Image credits: F. Diaz, R.W. White, G. Buscher, and D. Liebling. Robust models of mouse movement on dynamic
web search results pages. In CIKM, 2013. ACM Press
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 8
Background Motivation Model & Metric Experimental Setup Results Summary
Motivation 2: Satisfaction Without Clicks
High direct page utility (measured by DCG or ERR) leads to higher
abandonment rate (SERPs with no clicks)
direct page utility
Image credits: from A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In WWW, 2012.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 9
Background Motivation Model & Metric Experimental Setup Results Summary
Problems of Existing Models and Evaluation Metrics
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Background Motivation Model & Metric Experimental Setup Results Summary
Problems of Existing Models and Evaluation Metrics
existing models mostly do not model non-trivial user
attention patterns
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Background Motivation Model & Metric Experimental Setup Results Summary
Problems of Existing Models and Evaluation Metrics
existing models mostly do not model non-trivial user
attention patterns
existing models do not use explicit user satisfaction data
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Model & Metric
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
𝜑&
𝐸&
𝐶&
𝜑)
𝐸)
𝐶)
𝜑*
𝐸*
𝐶*
𝑆
…
Utility
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 12
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
𝜑&
𝐸&
𝐶&
𝜑)
𝐸)
𝐶)
𝜑*
𝐸*
𝐶*
𝑆
…
Utility
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 13
Background Motivation Model & Metric Experimental Setup Results Summary
Click Model
Examination assumption: click happens only when an item was
examined and attractive:
P(Ck = 1) = P(Ek = 1) · P(Ck = 1 | Ek = 1)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14
Background Motivation Model & Metric Experimental Setup Results Summary
Click Model
Examination assumption: click happens only when an item was
examined and attractive:
P(Ck = 1) = P(Ek = 1) · P(Ck = 1 | Ek = 1)
N.B. Here we assume that P(Ck = 1 | Ek = 1) = α(Rk) where Rk
comes from the raters and α is a logistic function.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
𝜑&
𝐸&
𝐶&
𝜑)
𝐸)
𝐶)
𝜑*
𝐸*
𝐶*
𝑆
…
Utility
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 15
Background Motivation Model & Metric Experimental Setup Results Summary
Attention (Examination) Model
Logistic regression model:
P(Ek = 1) = ε(ϕk),
where ϕk is a vector of features for SERP item k.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Attention (Examination) Model
Logistic regression model:
P(Ek = 1) = ε(ϕk),
where ϕk is a vector of features for SERP item k.
Feature group Features # of features
rank user-perceived rank of the SERP item
(can be different from k)
1
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Attention (Examination) Model
Logistic regression model:
P(Ek = 1) = ε(ϕk),
where ϕk is a vector of features for SERP item k.
Feature group Features # of features
rank user-perceived rank of the SERP item
(can be different from k)
1
CSS classes SERP item type (Web, News,
Weather, Currency, Knowledge
Panel, etc.)
10
geometry offset from the top, first or second col-
umn (binary), width (w), height (h),
w × h
5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
𝜑&
𝐸&
𝐶&
𝜑)
𝐸)
𝐶)
𝜑*
𝐸*
𝐶*
𝑆
…
Utility
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 17
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
in our model it also comes from the SERP items that simply
attracted attention;
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
in our model it also comes from the SERP items that simply
attracted attention;
P(S = 1) = σ(τ0 + U) =
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
in our model it also comes from the SERP items that simply
attracted attention;
P(S = 1) = σ(τ0 + U) =
σ τ0 +
k
P(Ek = 1)ud (Dk) +
k
P(Ck = 1)ur (Rk)
where Dk and Rk are ratings assigned by the raters for direct
snippet relevance and result relevance respectively. ud and ur are
linear functions of rating histograms.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
The CAS Metric
Utility that determines the satisfaction probability:
U =
k
P(Ek = 1)ud (Dk) +
k
P(Ck = 1)ur (Rk)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Background Motivation Model & Metric Experimental Setup Results Summary
The CAS Metric
Utility that determines the satisfaction probability:
U =
k
P(Ek = 1)ud (Dk)
NEW
+
k
P(Ck = 1)ur (Rk)
Chuklin et al. 2013
has an additional term
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Background Motivation Model & Metric Experimental Setup Results Summary
The CAS Metric
Utility that determines the satisfaction probability:
U =
k
P(Ek = 1)ud (Dk)
NEW
+
k
P(Ck = 1)ur (Rk)
Chuklin et al. 2013
has an additional term
trained on mousing and satisfaction (in addition to clicks)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Experimental Setup
Background Motivation Model & Metric Experimental Setup Results Summary
Dataset
199 queries with explicit unambiguous
feedback (satisfied / not satisfied);
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 21
Background Motivation Model & Metric Experimental Setup Results Summary
Dataset
199 queries with explicit unambiguous
feedback (satisfied / not satisfied);
1,739 rated results
direct snippet relevance (D)
result relevance (R)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 21
Background Motivation Model & Metric Experimental Setup Results Summary
Baselines and CAS Model Variants
UBM model that agrees
well with online team-draft
experimental outcomes;
PBM position-based model,
a robust model with fewer
parameters than UBM;
random model that predicts
click and satisfaction with
fixed probabilities (learned
from the data).
uUBM from
Chuklin et al. 2013. Similar
to UBM, but parameters are
trained on a different and
much bigger dataset.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 22
Background Motivation Model & Metric Experimental Setup Results Summary
Baselines and CAS Model Variants
UBM model that agrees
well with online team-draft
experimental outcomes;
PBM position-based model,
a robust model with fewer
parameters than UBM;
random model that predicts
click and satisfaction with
fixed probabilities (learned
from the data).
uUBM from
Chuklin et al. 2013. Similar
to UBM, but parameters are
trained on a different and
much bigger dataset.
CASnod is a stripped-down
version that does not use
(D) labels;
CASnosat is a version of
the CAS model that does
not include the satisfaction
term while optimizing the
model;
CASnoreg is a version of
the CAS model that does
not use regularization while
training. All other models
were trained with
L2-regularization.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 22
Results
Background Motivation Model & Metric Experimental Setup Results Summary
Is the New Metric Really New?
Correlation Between Metrics
Table: Correlation between metrics measured by average Pearson’s
correlation coefficient.
CASnosat CASnoreg CAS UBM PBM DCG uUBM
CASnod 0.593 0.564 0.633 0.470 0.487 0.546 0.441
CASnosat 0.664 0.715 0.707 0.668 0.735 0.684
CASnoreg 0.974 0.363 0.379 0.417 0.341
CAS 0.377 0.394 0.440 0.360
UBM 0.814 0.972 0.882
PBM 0.906 0.965
DCG 0.943
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 24
Background Motivation Model & Metric Experimental Setup Results Summary
Is the New Metric Measuring the Right Thing?
Metric Correlation with True Satisfaction
CASnod
CASnosat
CASnoreg
CAS
UBM PBM
random DCG
uUBM
0.2
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Pearson correlation coefficient between different model-based
metrics and the user-reported satisfaction.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 25
Background Motivation Model & Metric Experimental Setup Results Summary
Bonus Point
Log-Likelihood of Click Prediction
CASnod
CASnosat
CASnoreg
CAS
UBM PBM
random
uUBM
4.5
4.0
3.5
3.0
2.5
2.0
1.5
Log-likelihood of the click data. Note that uUBM was trained on a
totally different dataset.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 26
Summary
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
Direct snippet relevance (D) is essential for predicting
satisfaction.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
Direct snippet relevance (D) is essential for predicting
satisfaction.
The CAS metric is quite different from the previously used
metrics, making it an interesting addition to TREC.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
Direct snippet relevance (D) is essential for predicting
satisfaction.
The CAS metric is quite different from the previously used
metrics, making it an interesting addition to TREC.
When used as a model, CAS consistently predicts user
satisfaction with a relatively small penalty in click prediction.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Acknowledgments
All content represents the opinion of the authors which is not necessarily shared or endorsed by their respective
employers and/or sponsors.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 29
Background Motivation Model & Metric Experimental Setup Results Summary
Evaluating the User Model
Log-Likelihood of Satisfaction Prediction
CASnod
CASnosat
CASnoreg
CAS
UBM PBM
random
uUBM
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Log-likelihood of the satisfaction prediction. Some models have
log-likelihood below −0.8, hence there are no boxes for them.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 31
Background Motivation Model & Metric Experimental Setup Results Summary
Analyzing the Attention Features
CASrank is the
model that only uses
the rank to predict
attention;
CASnogeom only
uses the rank and
SERP item type
information and does
not use geometry;
CASnoclass does not
use the CSS class
features (SERP item
type).
Pearson correlation with satisfaction
CASrank
CASnogeom
CASnoclass
CASnod
CAS
0.2
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Log-likelihood of clicks / satisfaction
CASrank
CASnogeom
CASnoclass
CASnod
CAS
2.5
2.4
2.3
2.2
2.1
2.0
1.9
1.8
1.7
CASrank
CASnogeom
CASnoclass
CASnod
CAS
0.65
0.60
0.55
0.50
0.45
0.40
0.35
0.30
0.25
0.20
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 32
Background Motivation Model & Metric Experimental Setup Results Summary
Heterogeneous SERPs
12% of the SERPs in our data are heterogeneous and our metric
does well for them.
Table: Pearson correlation between utility of heterogeneous SERP and
user-reported satisfaction.
CAS UBM PBM random DCG uUBM
0.60 0.38 -0.05 -0.39 0.24 -0.08
CASrank CASnogeom CASclass CASnod CASnosat CASnoreg
0.15 -0.04 0.27 -0.04 0.48 0.67
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 33
Background Motivation Model & Metric Experimental Setup Results Summary
Spammers
Some raters were filtered out as spammers, but there was still
some natural disagreement:
Table: Filtered out workers and agreement scores for remaining workers.
% of workers % of ratings Cohen’s Krippendorf’s
label removed removed kappa alpha
(D) 32% 27% 0.339 0.144
(R) 41% 29% 0.348 0.117
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 34

More Related Content

Viewers also liked

Isomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master ClassIsomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master Class
Spike Brehm
 

Viewers also liked (20)

Strategies to Drive Web Traffic in the Real Estate World
Strategies to Drive Web Traffic in the Real Estate WorldStrategies to Drive Web Traffic in the Real Estate World
Strategies to Drive Web Traffic in the Real Estate World
 
The Paradox of Great Content
The Paradox of Great ContentThe Paradox of Great Content
The Paradox of Great Content
 
The Search Landscape in 2017
The Search Landscape in 2017The Search Landscape in 2017
The Search Landscape in 2017
 
SEO & UX: So Happy Together
SEO & UX: So Happy TogetherSEO & UX: So Happy Together
SEO & UX: So Happy Together
 
Link Building's Tipping Point
Link Building's Tipping PointLink Building's Tipping Point
Link Building's Tipping Point
 
Keeping Up With SEO in 2017 & Beyond
Keeping Up With SEO in 2017 & BeyondKeeping Up With SEO in 2017 & Beyond
Keeping Up With SEO in 2017 & Beyond
 
Intro to Mozcon 2016
Intro to Mozcon 2016Intro to Mozcon 2016
Intro to Mozcon 2016
 
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
 
The Measure of a Marketer's Worth
The Measure of a Marketer's WorthThe Measure of a Marketer's Worth
The Measure of a Marketer's Worth
 
The Worst Lessons Marketing Ever Taught Content
The Worst Lessons Marketing Ever Taught ContentThe Worst Lessons Marketing Ever Taught Content
The Worst Lessons Marketing Ever Taught Content
 
The Remarkable SEO Power of Republishing
The Remarkable SEO Power of RepublishingThe Remarkable SEO Power of Republishing
The Remarkable SEO Power of Republishing
 
SEO: Crawl Budget Optimierung & Onsite SEO
SEO: Crawl Budget Optimierung & Onsite SEOSEO: Crawl Budget Optimierung & Onsite SEO
SEO: Crawl Budget Optimierung & Onsite SEO
 
SEO: SERPs im Wandel - SMX Munich 2017
SEO: SERPs im Wandel - SMX Munich 2017SEO: SERPs im Wandel - SMX Munich 2017
SEO: SERPs im Wandel - SMX Munich 2017
 
Crawl Budget Optimization - SMX München 2016
Crawl Budget Optimization - SMX München 2016Crawl Budget Optimization - SMX München 2016
Crawl Budget Optimization - SMX München 2016
 
Burman GSC Gurgaon
Burman GSC GurgaonBurman GSC Gurgaon
Burman GSC Gurgaon
 
Frontend talk for backenders
Frontend talk for backendersFrontend talk for backenders
Frontend talk for backenders
 
Alloy Cybersecurity
Alloy CybersecurityAlloy Cybersecurity
Alloy Cybersecurity
 
Isomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master ClassIsomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master Class
 
SEO in a Two Algorithm World
SEO in a Two Algorithm WorldSEO in a Two Algorithm World
SEO in a Two Algorithm World
 
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
 

Similar to Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model

Visual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningVisual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learning
Benjamin Bengfort
 
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Simplilearn
 

Similar to Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model (20)

Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
Evaluate deep q learning for sequential targeted marketing with 10-fold cross...Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
 
Workshop: Your first machine learning project
Workshop: Your first machine learning projectWorkshop: Your first machine learning project
Workshop: Your first machine learning project
 
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
 
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
 
Tech meetup Data Driven - Codemotion
Tech meetup Data Driven - Codemotion Tech meetup Data Driven - Codemotion
Tech meetup Data Driven - Codemotion
 
Predire il futuro con Machine Learning & Big Data
Predire il futuro con Machine Learning & Big DataPredire il futuro con Machine Learning & Big Data
Predire il futuro con Machine Learning & Big Data
 
Data Science Machine
Data Science Machine Data Science Machine
Data Science Machine
 
Kaggle KDD Cup Report
Kaggle KDD Cup ReportKaggle KDD Cup Report
Kaggle KDD Cup Report
 
Phase 2 of Predicting Payment default on Vehicle Loan EMI
Phase 2 of Predicting Payment default on Vehicle Loan EMIPhase 2 of Predicting Payment default on Vehicle Loan EMI
Phase 2 of Predicting Payment default on Vehicle Loan EMI
 
Recommending job ads to people
Recommending job ads to peopleRecommending job ads to people
Recommending job ads to people
 
Visual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningVisual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learning
 
AlgorithmsModelsNov13.pptx
AlgorithmsModelsNov13.pptxAlgorithmsModelsNov13.pptx
AlgorithmsModelsNov13.pptx
 
Fast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA HardwareFast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA Hardware
 
Developing Web-scale Machine Learning at LinkedIn - From Soup to Nuts
Developing Web-scale Machine Learning at LinkedIn - From Soup to NutsDeveloping Web-scale Machine Learning at LinkedIn - From Soup to Nuts
Developing Web-scale Machine Learning at LinkedIn - From Soup to Nuts
 
Empirical Model of Supervised Learning Approach for Opinion Mining
Empirical Model of Supervised Learning Approach for Opinion MiningEmpirical Model of Supervised Learning Approach for Opinion Mining
Empirical Model of Supervised Learning Approach for Opinion Mining
 
Telecom Churn Analysis
Telecom Churn AnalysisTelecom Churn Analysis
Telecom Churn Analysis
 
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
 
Keynote at IWLS 2017
Keynote at IWLS 2017Keynote at IWLS 2017
Keynote at IWLS 2017
 
Envelopment Analysis In Economics
Envelopment Analysis In EconomicsEnvelopment Analysis In Economics
Envelopment Analysis In Economics
 
Web Page Ranking using Machine Learning
Web Page Ranking using Machine LearningWeb Page Ranking using Machine Learning
Web Page Ranking using Machine Learning
 

More from Rand Fishkin

More from Rand Fishkin (20)

SparkToro Beta Sneak Peek
SparkToro Beta Sneak PeekSparkToro Beta Sneak Peek
SparkToro Beta Sneak Peek
 
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & MoreThe Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
 
Building Influence in 2019
Building Influence in 2019Building Influence in 2019
Building Influence in 2019
 
Influence Not Influencers
Influence Not InfluencersInfluence Not Influencers
Influence Not Influencers
 
The Next Era of Web Marketing: 2019 & Beyond
The Next Era of Web Marketing: 2019 & BeyondThe Next Era of Web Marketing: 2019 & Beyond
The Next Era of Web Marketing: 2019 & Beyond
 
How to Kick Butt with Your Email Outreach
How to Kick Butt with Your Email OutreachHow to Kick Butt with Your Email Outreach
How to Kick Butt with Your Email Outreach
 
The Big 7 Startup Marketing Mistakes
The Big 7 Startup Marketing MistakesThe Big 7 Startup Marketing Mistakes
The Big 7 Startup Marketing Mistakes
 
Why Your Saas Marketing Sucks (and how to fix it)
Why Your Saas Marketing Sucks (and how to fix it)Why Your Saas Marketing Sucks (and how to fix it)
Why Your Saas Marketing Sucks (and how to fix it)
 
SEO on the SERPs - Brighton SEO Closing Talk
SEO on the SERPs - Brighton SEO Closing TalkSEO on the SERPs - Brighton SEO Closing Talk
SEO on the SERPs - Brighton SEO Closing Talk
 
7 Lessons That Would Have Made Me a Better Entrepreneur
7 Lessons That Would Have Made Me a Better Entrepreneur7 Lessons That Would Have Made Me a Better Entrepreneur
7 Lessons That Would Have Made Me a Better Entrepreneur
 
The Search & SEO World in 2018
The Search & SEO World in 2018The Search & SEO World in 2018
The Search & SEO World in 2018
 
SEO in 2017/18
SEO in 2017/18SEO in 2017/18
SEO in 2017/18
 
Why Startups Suck at Marketing
Why Startups Suck at MarketingWhy Startups Suck at Marketing
Why Startups Suck at Marketing
 
The Invisible Giant that Mucks Up Our Marketing
The Invisible Giant that Mucks Up Our MarketingThe Invisible Giant that Mucks Up Our Marketing
The Invisible Giant that Mucks Up Our Marketing
 
B2B SEO in 2017
B2B SEO in 2017B2B SEO in 2017
B2B SEO in 2017
 
How to Survive Google's Trojan Horsing of the Web
How to Survive Google's Trojan Horsing of the WebHow to Survive Google's Trojan Horsing of the Web
How to Survive Google's Trojan Horsing of the Web
 
What Startup Execs Need to Know About SEO in 2017
What Startup Execs Need to Know About SEO in 2017What Startup Execs Need to Know About SEO in 2017
What Startup Execs Need to Know About SEO in 2017
 
Inside Google's Numbers in 2017
Inside Google's Numbers in 2017Inside Google's Numbers in 2017
Inside Google's Numbers in 2017
 
Why We Can't Do SEO WIthout CRO
Why We Can't Do SEO WIthout CROWhy We Can't Do SEO WIthout CRO
Why We Can't Do SEO WIthout CRO
 
The Digital Marketer's Framework
The Digital Marketer's FrameworkThe Digital Marketer's Framework
The Digital Marketer's Framework
 

Recently uploaded

一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
AS
 
一比一原版美国北卡罗莱纳大学毕业证如何办理
一比一原版美国北卡罗莱纳大学毕业证如何办理一比一原版美国北卡罗莱纳大学毕业证如何办理
一比一原版美国北卡罗莱纳大学毕业证如何办理
A
 
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
Fir
 
Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...
Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...
Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...
mikehavy0
 
原版定制美国加州大学河滨分校毕业证原件一模一样
原版定制美国加州大学河滨分校毕业证原件一模一样原版定制美国加州大学河滨分校毕业证原件一模一样
原版定制美国加州大学河滨分校毕业证原件一模一样
A
 
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
Fi
 
一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样
一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样
一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样
AS
 
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
musaddumba454
 
一比一定制美国罗格斯大学毕业证学位证书
一比一定制美国罗格斯大学毕业证学位证书一比一定制美国罗格斯大学毕业证学位证书
一比一定制美国罗格斯大学毕业证学位证书
A
 

Recently uploaded (20)

一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
一比一原版(Wintec毕业证书)新西兰怀卡托理工学院毕业证原件一模一样
 
The Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdfThe Rise of Subscription-Based Digital Services.pdf
The Rise of Subscription-Based Digital Services.pdf
 
一比一原版美国北卡罗莱纳大学毕业证如何办理
一比一原版美国北卡罗莱纳大学毕业证如何办理一比一原版美国北卡罗莱纳大学毕业证如何办理
一比一原版美国北卡罗莱纳大学毕业证如何办理
 
APNIC Updates presented by Paul Wilson at CaribNOG 27
APNIC Updates presented by Paul Wilson at  CaribNOG 27APNIC Updates presented by Paul Wilson at  CaribNOG 27
APNIC Updates presented by Paul Wilson at CaribNOG 27
 
Beyond Inbound: Unlocking the Secrets of API Egress Traffic Management
Beyond Inbound: Unlocking the Secrets of API Egress Traffic ManagementBeyond Inbound: Unlocking the Secrets of API Egress Traffic Management
Beyond Inbound: Unlocking the Secrets of API Egress Traffic Management
 
SOC Analyst Guide For Beginners SOC analysts work as members of a managed sec...
SOC Analyst Guide For Beginners SOC analysts work as members of a managed sec...SOC Analyst Guide For Beginners SOC analysts work as members of a managed sec...
SOC Analyst Guide For Beginners SOC analysts work as members of a managed sec...
 
Lowongan Kerja LC Yogyakarta Terbaru 085746015303
Lowongan Kerja LC Yogyakarta Terbaru 085746015303Lowongan Kerja LC Yogyakarta Terbaru 085746015303
Lowongan Kerja LC Yogyakarta Terbaru 085746015303
 
HUMANIZE YOUR BRAND - FREE E-WORKBOOK Download Now
HUMANIZE YOUR BRAND - FREE E-WORKBOOK Download NowHUMANIZE YOUR BRAND - FREE E-WORKBOOK Download Now
HUMANIZE YOUR BRAND - FREE E-WORKBOOK Download Now
 
GOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdfGOOGLE Io 2024 At takes center stage.pdf
GOOGLE Io 2024 At takes center stage.pdf
 
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
一比一原版(PSU毕业证书)美国宾州州立大学毕业证如何办理
 
Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...
Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...
Abortion Clinic in Kwa thema +27791653574 Kwa thema WhatsApp Abortion Clinic ...
 
原版定制美国加州大学河滨分校毕业证原件一模一样
原版定制美国加州大学河滨分校毕业证原件一模一样原版定制美国加州大学河滨分校毕业证原件一模一样
原版定制美国加州大学河滨分校毕业证原件一模一样
 
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
一比一原版(UWE毕业证书)西英格兰大学毕业证原件一模一样
 
Dan Quinn Commanders Feather Dad Hat Hoodie
Dan Quinn Commanders Feather Dad Hat HoodieDan Quinn Commanders Feather Dad Hat Hoodie
Dan Quinn Commanders Feather Dad Hat Hoodie
 
APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0
APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0
APNIC Policy Roundup presented by Sunny Chendi at TWNOG 5.0
 
Subdomain enumeration is a crucial phase in cybersecurity, particularly durin...
Subdomain enumeration is a crucial phase in cybersecurity, particularly durin...Subdomain enumeration is a crucial phase in cybersecurity, particularly durin...
Subdomain enumeration is a crucial phase in cybersecurity, particularly durin...
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
 
一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样
一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样
一比一原版(Polytechnic毕业证书)新加坡理工学院毕业证原件一模一样
 
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
100^%)( POLOKWANE))(*((+27838792658))*))௹ )Abortion Pills for Sale in Sibasa,...
 
一比一定制美国罗格斯大学毕业证学位证书
一比一定制美国罗格斯大学毕业证学位证书一比一定制美国罗格斯大学毕业证学位证书
一比一定制美国罗格斯大学毕业证学位证书
 

Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model

  • 1. Background Motivation Model & Metric Experimental Setup Results Summary Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model Aleksandr Chuklin¶,§ Maarten de Rijke§ chuklin@google.com derijke@uva.nl ¶Google Research Europe §University of Amsterdam AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 1
  • 3. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Main problem Combining relevance of individual SERP items (Rk) into a whole-page metric. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 3
  • 4. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples document 3 document 4 document 1 document 2 document 5 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 5. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: P@N = 1 N N k=1 Rk document 3 document 4 document 1 document 2 document 5 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 6. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: P@N = 1 N N k=1 Rk Discounted Cumulative Gain (DCG): DCG@N = N k=1 1 log2 (1 + k) · Rk document 3 document 4 document 1 document 2 document 5 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 7. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: P@N = 1 N N k=1 Rk Discounted Cumulative Gain (DCG): DCG@N = N k=1 1 log2 (1 + k) · Rk Model-Based Metrics (Chuklin et al. 2013): Utility@N = N k=1 P(Ck = 1) · Rk document 3 document 4 document 1 document 2 document 5 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 8. Background Motivation Model & Metric Experimental Setup Results Summary Main Goal of This Paper Better measure for SERP utility Namely, improve this (Chuklin et al. 2013): N k=1 P(Ck = 1) · Rk AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 5
  • 10. Background Motivation Model & Metric Experimental Setup Results Summary Complex Heterogeneous SERPs AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 7
  • 11. Background Motivation Model & Metric Experimental Setup Results Summary Motivation 1: Non-Trivial Attention Patterns 4 ement 9 1 3 5 6 7 8 4 2 (c) Mouse Data data. The session sequence for this data would be Image credits: F. Diaz, R.W. White, G. Buscher, and D. Liebling. Robust models of mouse movement on dynamic web search results pages. In CIKM, 2013. ACM Press AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 8
  • 12. Background Motivation Model & Metric Experimental Setup Results Summary Motivation 2: Satisfaction Without Clicks High direct page utility (measured by DCG or ERR) leads to higher abandonment rate (SERPs with no clicks) direct page utility Image credits: from A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In WWW, 2012. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 9
  • 13. Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
  • 14. Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics existing models mostly do not model non-trivial user attention patterns AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
  • 15. Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics existing models mostly do not model non-trivial user attention patterns existing models do not use explicit user satisfaction data AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
  • 17. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜑& 𝐸& 𝐶& 𝜑) 𝐸) 𝐶) 𝜑* 𝐸* 𝐶* 𝑆 … Utility AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 12
  • 18. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜑& 𝐸& 𝐶& 𝜑) 𝐸) 𝐶) 𝜑* 𝐸* 𝐶* 𝑆 … Utility AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 13
  • 19. Background Motivation Model & Metric Experimental Setup Results Summary Click Model Examination assumption: click happens only when an item was examined and attractive: P(Ck = 1) = P(Ek = 1) · P(Ck = 1 | Ek = 1) AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14
  • 20. Background Motivation Model & Metric Experimental Setup Results Summary Click Model Examination assumption: click happens only when an item was examined and attractive: P(Ck = 1) = P(Ek = 1) · P(Ck = 1 | Ek = 1) N.B. Here we assume that P(Ck = 1 | Ek = 1) = α(Rk) where Rk comes from the raters and α is a logistic function. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14
  • 21. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜑& 𝐸& 𝐶& 𝜑) 𝐸) 𝐶) 𝜑* 𝐸* 𝐶* 𝑆 … Utility AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 15
  • 22. Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P(Ek = 1) = ε(ϕk), where ϕk is a vector of features for SERP item k. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
  • 23. Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P(Ek = 1) = ε(ϕk), where ϕk is a vector of features for SERP item k. Feature group Features # of features rank user-perceived rank of the SERP item (can be different from k) 1 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
  • 24. Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P(Ek = 1) = ε(ϕk), where ϕk is a vector of features for SERP item k. Feature group Features # of features rank user-perceived rank of the SERP item (can be different from k) 1 CSS classes SERP item type (Web, News, Weather, Currency, Knowledge Panel, etc.) 10 geometry offset from the top, first or second col- umn (binary), width (w), height (h), w × h 5 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
  • 25. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP 𝜑& 𝐸& 𝐶& 𝜑) 𝐸) 𝐶) 𝜑* 𝐸* 𝐶* 𝑆 … Utility AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 17
  • 26. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 27. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 28. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; P(S = 1) = σ(τ0 + U) = AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 29. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; P(S = 1) = σ(τ0 + U) = σ τ0 + k P(Ek = 1)ud (Dk) + k P(Ck = 1)ur (Rk) where Dk and Rk are ratings assigned by the raters for direct snippet relevance and result relevance respectively. ud and ur are linear functions of rating histograms. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 30. Background Motivation Model & Metric Experimental Setup Results Summary The CAS Metric Utility that determines the satisfaction probability: U = k P(Ek = 1)ud (Dk) + k P(Ck = 1)ur (Rk) AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
  • 31. Background Motivation Model & Metric Experimental Setup Results Summary The CAS Metric Utility that determines the satisfaction probability: U = k P(Ek = 1)ud (Dk) NEW + k P(Ck = 1)ur (Rk) Chuklin et al. 2013 has an additional term AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
  • 32. Background Motivation Model & Metric Experimental Setup Results Summary The CAS Metric Utility that determines the satisfaction probability: U = k P(Ek = 1)ud (Dk) NEW + k P(Ck = 1)ur (Rk) Chuklin et al. 2013 has an additional term trained on mousing and satisfaction (in addition to clicks) AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
  • 34. Background Motivation Model & Metric Experimental Setup Results Summary Dataset 199 queries with explicit unambiguous feedback (satisfied / not satisfied); AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 21
  • 35. Background Motivation Model & Metric Experimental Setup Results Summary Dataset 199 queries with explicit unambiguous feedback (satisfied / not satisfied); 1,739 rated results direct snippet relevance (D) result relevance (R) AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 21
  • 36. Background Motivation Model & Metric Experimental Setup Results Summary Baselines and CAS Model Variants UBM model that agrees well with online team-draft experimental outcomes; PBM position-based model, a robust model with fewer parameters than UBM; random model that predicts click and satisfaction with fixed probabilities (learned from the data). uUBM from Chuklin et al. 2013. Similar to UBM, but parameters are trained on a different and much bigger dataset. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 22
  • 37. Background Motivation Model & Metric Experimental Setup Results Summary Baselines and CAS Model Variants UBM model that agrees well with online team-draft experimental outcomes; PBM position-based model, a robust model with fewer parameters than UBM; random model that predicts click and satisfaction with fixed probabilities (learned from the data). uUBM from Chuklin et al. 2013. Similar to UBM, but parameters are trained on a different and much bigger dataset. CASnod is a stripped-down version that does not use (D) labels; CASnosat is a version of the CAS model that does not include the satisfaction term while optimizing the model; CASnoreg is a version of the CAS model that does not use regularization while training. All other models were trained with L2-regularization. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 22
  • 39. Background Motivation Model & Metric Experimental Setup Results Summary Is the New Metric Really New? Correlation Between Metrics Table: Correlation between metrics measured by average Pearson’s correlation coefficient. CASnosat CASnoreg CAS UBM PBM DCG uUBM CASnod 0.593 0.564 0.633 0.470 0.487 0.546 0.441 CASnosat 0.664 0.715 0.707 0.668 0.735 0.684 CASnoreg 0.974 0.363 0.379 0.417 0.341 CAS 0.377 0.394 0.440 0.360 UBM 0.814 0.972 0.882 PBM 0.906 0.965 DCG 0.943 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 24
  • 40. Background Motivation Model & Metric Experimental Setup Results Summary Is the New Metric Measuring the Right Thing? Metric Correlation with True Satisfaction CASnod CASnosat CASnoreg CAS UBM PBM random DCG uUBM 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Pearson correlation coefficient between different model-based metrics and the user-reported satisfaction. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 25
  • 41. Background Motivation Model & Metric Experimental Setup Results Summary Bonus Point Log-Likelihood of Click Prediction CASnod CASnosat CASnoreg CAS UBM PBM random uUBM 4.5 4.0 3.5 3.0 2.5 2.0 1.5 Log-likelihood of the click data. Note that uUBM was trained on a totally different dataset. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 26
  • 43. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 44. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 45. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. The CAS metric is quite different from the previously used metrics, making it an interesting addition to TREC. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 46. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. The CAS metric is quite different from the previously used metrics, making it an interesting addition to TREC. When used as a model, CAS consistently predicts user satisfaction with a relatively small penalty in click prediction. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 47. Background Motivation Model & Metric Experimental Setup Results Summary Acknowledgments All content represents the opinion of the authors which is not necessarily shared or endorsed by their respective employers and/or sponsors. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 29
  • 48.
  • 49. Background Motivation Model & Metric Experimental Setup Results Summary Evaluating the User Model Log-Likelihood of Satisfaction Prediction CASnod CASnosat CASnoreg CAS UBM PBM random uUBM 0.8 0.7 0.6 0.5 0.4 0.3 0.2 Log-likelihood of the satisfaction prediction. Some models have log-likelihood below −0.8, hence there are no boxes for them. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 31
  • 50. Background Motivation Model & Metric Experimental Setup Results Summary Analyzing the Attention Features CASrank is the model that only uses the rank to predict attention; CASnogeom only uses the rank and SERP item type information and does not use geometry; CASnoclass does not use the CSS class features (SERP item type). Pearson correlation with satisfaction CASrank CASnogeom CASnoclass CASnod CAS 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Log-likelihood of clicks / satisfaction CASrank CASnogeom CASnoclass CASnod CAS 2.5 2.4 2.3 2.2 2.1 2.0 1.9 1.8 1.7 CASrank CASnogeom CASnoclass CASnod CAS 0.65 0.60 0.55 0.50 0.45 0.40 0.35 0.30 0.25 0.20 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 32
  • 51. Background Motivation Model & Metric Experimental Setup Results Summary Heterogeneous SERPs 12% of the SERPs in our data are heterogeneous and our metric does well for them. Table: Pearson correlation between utility of heterogeneous SERP and user-reported satisfaction. CAS UBM PBM random DCG uUBM 0.60 0.38 -0.05 -0.39 0.24 -0.08 CASrank CASnogeom CASclass CASnod CASnosat CASnoreg 0.15 -0.04 0.27 -0.04 0.48 0.67 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 33
  • 52. Background Motivation Model & Metric Experimental Setup Results Summary Spammers Some raters were filtered out as spammers, but there was still some natural disagreement: Table: Filtered out workers and agreement scores for remaining workers. % of workers % of ratings Cohen’s Krippendorf’s label removed removed kappa alpha (D) 32% 27% 0.339 0.144 (R) 41% 29% 0.348 0.117 AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 34