SlideShare a Scribd company logo
1 of 52
Download to read offline
Background Motivation Model & Metric Experimental Setup Results Summary
Incorporating Clicks, Attention and Satisfaction
into a SERP Evaluation Model
Aleksandr ChuklinĀ¶,Ā§ Maarten de RijkeĀ§
chuklin@google.com derijke@uva.nl
Ā¶Google Research Europe
Ā§University of Amsterdam
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 1
Background
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Main problem
Combining relevance of individual SERP items (Rk) into a
whole-page metric.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 3
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
document 3
document 4
document 1
document 2
document 5
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
Precision at N:
P@N =
1
N
N
k=1
Rk
document 3
document 4
document 1
document 2
document 5
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
Precision at N:
P@N =
1
N
N
k=1
Rk
Discounted Cumulative Gain (DCG):
DCG@N =
N
k=1
1
log2 (1 + k)
Ā· Rk
document 3
document 4
document 1
document 2
document 5
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Search Engine Result Page (SERP) Evaluation
Examples
Precision at N:
P@N =
1
N
N
k=1
Rk
Discounted Cumulative Gain (DCG):
DCG@N =
N
k=1
1
log2 (1 + k)
Ā· Rk
Model-Based Metrics (Chuklin et al. 2013):
Utility@N =
N
k=1
P(Ck = 1) Ā· Rk
document 3
document 4
document 1
document 2
document 5
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Main Goal of This Paper
Better measure for SERP utility
Namely, improve this (Chuklin et al. 2013):
N
k=1
P(Ck = 1) Ā· Rk
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 5
Motivation
Background Motivation Model & Metric Experimental Setup Results Summary
Complex Heterogeneous SERPs
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 7
Background Motivation Model & Metric Experimental Setup Results Summary
Motivation 1: Non-Trivial Attention Patterns
4
ement
9
1
3
5
6
7
8
4
2
(c) Mouse Data
data. The session sequence for this data would be
Image credits: F. Diaz, R.W. White, G. Buscher, and D. Liebling. Robust models of mouse movement on dynamic
web search results pages. In CIKM, 2013. ACM Press
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 8
Background Motivation Model & Metric Experimental Setup Results Summary
Motivation 2: Satisfaction Without Clicks
High direct page utility (measured by DCG or ERR) leads to higher
abandonment rate (SERPs with no clicks)
direct page utility
Image credits: from A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In WWW, 2012.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 9
Background Motivation Model & Metric Experimental Setup Results Summary
Problems of Existing Models and Evaluation Metrics
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Background Motivation Model & Metric Experimental Setup Results Summary
Problems of Existing Models and Evaluation Metrics
existing models mostly do not model non-trivial user
attention patterns
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Background Motivation Model & Metric Experimental Setup Results Summary
Problems of Existing Models and Evaluation Metrics
existing models mostly do not model non-trivial user
attention patterns
existing models do not use explicit user satisfaction data
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Model & Metric
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
šœ‘&
šø&
š¶&
šœ‘)
šø)
š¶)
šœ‘*
šø*
š¶*
š‘†
ā€¦
Utility
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 12
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
šœ‘&
šø&
š¶&
šœ‘)
šø)
š¶)
šœ‘*
šø*
š¶*
š‘†
ā€¦
Utility
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 13
Background Motivation Model & Metric Experimental Setup Results Summary
Click Model
Examination assumption: click happens only when an item was
examined and attractive:
P(Ck = 1) = P(Ek = 1) Ā· P(Ck = 1 | Ek = 1)
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 14
Background Motivation Model & Metric Experimental Setup Results Summary
Click Model
Examination assumption: click happens only when an item was
examined and attractive:
P(Ck = 1) = P(Ek = 1) Ā· P(Ck = 1 | Ek = 1)
N.B. Here we assume that P(Ck = 1 | Ek = 1) = Ī±(Rk) where Rk
comes from the raters and Ī± is a logistic function.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 14
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
šœ‘&
šø&
š¶&
šœ‘)
šø)
š¶)
šœ‘*
šø*
š¶*
š‘†
ā€¦
Utility
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 15
Background Motivation Model & Metric Experimental Setup Results Summary
Attention (Examination) Model
Logistic regression model:
P(Ek = 1) = Īµ(Ļ•k),
where Ļ•k is a vector of features for SERP item k.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Attention (Examination) Model
Logistic regression model:
P(Ek = 1) = Īµ(Ļ•k),
where Ļ•k is a vector of features for SERP item k.
Feature group Features # of features
rank user-perceived rank of the SERP item
(can be diļ¬€erent from k)
1
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Attention (Examination) Model
Logistic regression model:
P(Ek = 1) = Īµ(Ļ•k),
where Ļ•k is a vector of features for SERP item k.
Feature group Features # of features
rank user-perceived rank of the SERP item
(can be diļ¬€erent from k)
1
CSS classes SERP item type (Web, News,
Weather, Currency, Knowledge
Panel, etc.)
10
geometry oļ¬€set from the top, ļ¬rst or second col-
umn (binary), width (w), height (h),
w Ɨ h
5
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Clicks + Attention + Satisfaction (CAS) Model
SERP
šœ‘&
šø&
š¶&
šœ‘)
šø)
š¶)
šœ‘*
šø*
š¶*
š‘†
ā€¦
Utility
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 17
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
in our model it also comes from the SERP items that simply
attracted attention;
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
in our model it also comes from the SERP items that simply
attracted attention;
P(S = 1) = Ļƒ(Ļ„0 + U) =
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
Satisfaction Model
in previous models, satisfaction comes only from clicked
results;
in our model it also comes from the SERP items that simply
attracted attention;
P(S = 1) = Ļƒ(Ļ„0 + U) =
Ļƒ Ļ„0 +
k
P(Ek = 1)ud (Dk) +
k
P(Ck = 1)ur (Rk)
where Dk and Rk are ratings assigned by the raters for direct
snippet relevance and result relevance respectively. ud and ur are
linear functions of rating histograms.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
The CAS Metric
Utility that determines the satisfaction probability:
U =
k
P(Ek = 1)ud (Dk) +
k
P(Ck = 1)ur (Rk)
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Background Motivation Model & Metric Experimental Setup Results Summary
The CAS Metric
Utility that determines the satisfaction probability:
U =
k
P(Ek = 1)ud (Dk)
NEW
+
k
P(Ck = 1)ur (Rk)
Chuklin et al. 2013
has an additional term
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Background Motivation Model & Metric Experimental Setup Results Summary
The CAS Metric
Utility that determines the satisfaction probability:
U =
k
P(Ek = 1)ud (Dk)
NEW
+
k
P(Ck = 1)ur (Rk)
Chuklin et al. 2013
has an additional term
trained on mousing and satisfaction (in addition to clicks)
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Experimental Setup
Background Motivation Model & Metric Experimental Setup Results Summary
Dataset
199 queries with explicit unambiguous
feedback (satisļ¬ed / not satisļ¬ed);
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 21
Background Motivation Model & Metric Experimental Setup Results Summary
Dataset
199 queries with explicit unambiguous
feedback (satisļ¬ed / not satisļ¬ed);
1,739 rated results
direct snippet relevance (D)
result relevance (R)
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 21
Background Motivation Model & Metric Experimental Setup Results Summary
Baselines and CAS Model Variants
UBM model that agrees
well with online team-draft
experimental outcomes;
PBM position-based model,
a robust model with fewer
parameters than UBM;
random model that predicts
click and satisfaction with
ļ¬xed probabilities (learned
from the data).
uUBM from
Chuklin et al. 2013. Similar
to UBM, but parameters are
trained on a diļ¬€erent and
much bigger dataset.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 22
Background Motivation Model & Metric Experimental Setup Results Summary
Baselines and CAS Model Variants
UBM model that agrees
well with online team-draft
experimental outcomes;
PBM position-based model,
a robust model with fewer
parameters than UBM;
random model that predicts
click and satisfaction with
ļ¬xed probabilities (learned
from the data).
uUBM from
Chuklin et al. 2013. Similar
to UBM, but parameters are
trained on a diļ¬€erent and
much bigger dataset.
CASnod is a stripped-down
version that does not use
(D) labels;
CASnosat is a version of
the CAS model that does
not include the satisfaction
term while optimizing the
model;
CASnoreg is a version of
the CAS model that does
not use regularization while
training. All other models
were trained with
L2-regularization.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 22
Results
Background Motivation Model & Metric Experimental Setup Results Summary
Is the New Metric Really New?
Correlation Between Metrics
Table: Correlation between metrics measured by average Pearsonā€™s
correlation coeļ¬ƒcient.
CASnosat CASnoreg CAS UBM PBM DCG uUBM
CASnod 0.593 0.564 0.633 0.470 0.487 0.546 0.441
CASnosat 0.664 0.715 0.707 0.668 0.735 0.684
CASnoreg 0.974 0.363 0.379 0.417 0.341
CAS 0.377 0.394 0.440 0.360
UBM 0.814 0.972 0.882
PBM 0.906 0.965
DCG 0.943
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 24
Background Motivation Model & Metric Experimental Setup Results Summary
Is the New Metric Measuring the Right Thing?
Metric Correlation with True Satisfaction
CASnod
CASnosat
CASnoreg
CAS
UBM PBM
random DCG
uUBM
0.2
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Pearson correlation coeļ¬ƒcient between diļ¬€erent model-based
metrics and the user-reported satisfaction.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 25
Background Motivation Model & Metric Experimental Setup Results Summary
Bonus Point
Log-Likelihood of Click Prediction
CASnod
CASnosat
CASnoreg
CAS
UBM PBM
random
uUBM
4.5
4.0
3.5
3.0
2.5
2.0
1.5
Log-likelihood of the click data. Note that uUBM was trained on a
totally diļ¬€erent dataset.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 26
Summary
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
Direct snippet relevance (D) is essential for predicting
satisfaction.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
Direct snippet relevance (D) is essential for predicting
satisfaction.
The CAS metric is quite diļ¬€erent from the previously used
metrics, making it an interesting addition to TREC.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Summary
A model-based metric needs to model satisfaction explicitly
and use it for training.
Direct snippet relevance (D) is essential for predicting
satisfaction.
The CAS metric is quite diļ¬€erent from the previously used
metrics, making it an interesting addition to TREC.
When used as a model, CAS consistently predicts user
satisfaction with a relatively small penalty in click prediction.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
Acknowledgments
All content represents the opinion of the authors which is not necessarily shared or endorsed by their respective
employers and/or sponsors.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 29
Background Motivation Model & Metric Experimental Setup Results Summary
Evaluating the User Model
Log-Likelihood of Satisfaction Prediction
CASnod
CASnosat
CASnoreg
CAS
UBM PBM
random
uUBM
0.8
0.7
0.6
0.5
0.4
0.3
0.2
Log-likelihood of the satisfaction prediction. Some models have
log-likelihood below āˆ’0.8, hence there are no boxes for them.
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 31
Background Motivation Model & Metric Experimental Setup Results Summary
Analyzing the Attention Features
CASrank is the
model that only uses
the rank to predict
attention;
CASnogeom only
uses the rank and
SERP item type
information and does
not use geometry;
CASnoclass does not
use the CSS class
features (SERP item
type).
Pearson correlation with satisfaction
CASrank
CASnogeom
CASnoclass
CASnod
CAS
0.2
0.1
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Log-likelihood of clicks / satisfaction
CASrank
CASnogeom
CASnoclass
CASnod
CAS
2.5
2.4
2.3
2.2
2.1
2.0
1.9
1.8
1.7
CASrank
CASnogeom
CASnoclass
CASnod
CAS
0.65
0.60
0.55
0.50
0.45
0.40
0.35
0.30
0.25
0.20
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 32
Background Motivation Model & Metric Experimental Setup Results Summary
Heterogeneous SERPs
12% of the SERPs in our data are heterogeneous and our metric
does well for them.
Table: Pearson correlation between utility of heterogeneous SERP and
user-reported satisfaction.
CAS UBM PBM random DCG uUBM
0.60 0.38 -0.05 -0.39 0.24 -0.08
CASrank CASnogeom CASclass CASnod CASnosat CASnoreg
0.15 -0.04 0.27 -0.04 0.48 0.67
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 33
Background Motivation Model & Metric Experimental Setup Results Summary
Spammers
Some raters were ļ¬ltered out as spammers, but there was still
some natural disagreement:
Table: Filtered out workers and agreement scores for remaining workers.
% of workers % of ratings Cohenā€™s Krippendorfā€™s
label removed removed kappa alpha
(D) 32% 27% 0.339 0.144
(R) 41% 29% 0.348 0.117
ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 34

More Related Content

Viewers also liked

Strategies to Drive Web Traffic in the Real Estate World
Strategies to Drive Web Traffic in the Real Estate WorldStrategies to Drive Web Traffic in the Real Estate World
Strategies to Drive Web Traffic in the Real Estate WorldRand Fishkin
Ā 
The Paradox of Great Content
The Paradox of Great ContentThe Paradox of Great Content
The Paradox of Great ContentRand Fishkin
Ā 
The Search Landscape in 2017
The Search Landscape in 2017The Search Landscape in 2017
The Search Landscape in 2017Rand Fishkin
Ā 
SEO & UX: So Happy Together
SEO & UX: So Happy TogetherSEO & UX: So Happy Together
SEO & UX: So Happy TogetherRand Fishkin
Ā 
Link Building's Tipping Point
Link Building's Tipping PointLink Building's Tipping Point
Link Building's Tipping PointRand Fishkin
Ā 
Keeping Up With SEO in 2017 & Beyond
Keeping Up With SEO in 2017 & BeyondKeeping Up With SEO in 2017 & Beyond
Keeping Up With SEO in 2017 & BeyondRand Fishkin
Ā 
Intro to Mozcon 2016
Intro to Mozcon 2016Intro to Mozcon 2016
Intro to Mozcon 2016Rand Fishkin
Ā 
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...Rand Fishkin
Ā 
The Measure of a Marketer's Worth
The Measure of a Marketer's WorthThe Measure of a Marketer's Worth
The Measure of a Marketer's WorthRand Fishkin
Ā 
The Worst Lessons Marketing Ever Taught Content
The Worst Lessons Marketing Ever Taught ContentThe Worst Lessons Marketing Ever Taught Content
The Worst Lessons Marketing Ever Taught ContentRand Fishkin
Ā 
The Remarkable SEO Power of Republishing
The Remarkable SEO Power of RepublishingThe Remarkable SEO Power of Republishing
The Remarkable SEO Power of RepublishingRand Fishkin
Ā 
SEO: Crawl Budget Optimierung & Onsite SEO
SEO: Crawl Budget Optimierung & Onsite SEOSEO: Crawl Budget Optimierung & Onsite SEO
SEO: Crawl Budget Optimierung & Onsite SEOPhilipp Klƶckner
Ā 
SEO: SERPs im Wandel - SMX Munich 2017
SEO: SERPs im Wandel - SMX Munich 2017SEO: SERPs im Wandel - SMX Munich 2017
SEO: SERPs im Wandel - SMX Munich 2017Philipp Klƶckner
Ā 
Crawl Budget Optimization - SMX MĆ¼nchen 2016
Crawl Budget Optimization - SMX MĆ¼nchen 2016Crawl Budget Optimization - SMX MĆ¼nchen 2016
Crawl Budget Optimization - SMX MĆ¼nchen 2016Bastian Grimm
Ā 
Burman GSC Gurgaon
Burman GSC GurgaonBurman GSC Gurgaon
Burman GSC GurgaonManish Kumar
Ā 
Alloy Cybersecurity
Alloy CybersecurityAlloy Cybersecurity
Alloy CybersecurityMark Stockman
Ā 
Isomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master ClassIsomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master ClassSpike Brehm
Ā 
SEO in a Two Algorithm World
SEO in a Two Algorithm WorldSEO in a Two Algorithm World
SEO in a Two Algorithm WorldRand Fishkin
Ā 
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016Everett Sizemore
Ā 

Viewers also liked (20)

Strategies to Drive Web Traffic in the Real Estate World
Strategies to Drive Web Traffic in the Real Estate WorldStrategies to Drive Web Traffic in the Real Estate World
Strategies to Drive Web Traffic in the Real Estate World
Ā 
The Paradox of Great Content
The Paradox of Great ContentThe Paradox of Great Content
The Paradox of Great Content
Ā 
The Search Landscape in 2017
The Search Landscape in 2017The Search Landscape in 2017
The Search Landscape in 2017
Ā 
SEO & UX: So Happy Together
SEO & UX: So Happy TogetherSEO & UX: So Happy Together
SEO & UX: So Happy Together
Ā 
Link Building's Tipping Point
Link Building's Tipping PointLink Building's Tipping Point
Link Building's Tipping Point
Ā 
Keeping Up With SEO in 2017 & Beyond
Keeping Up With SEO in 2017 & BeyondKeeping Up With SEO in 2017 & Beyond
Keeping Up With SEO in 2017 & Beyond
Ā 
Intro to Mozcon 2016
Intro to Mozcon 2016Intro to Mozcon 2016
Intro to Mozcon 2016
Ā 
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
Fight Back Against Back: How Search Engines & Social Networks' AI Impacts Mar...
Ā 
The Measure of a Marketer's Worth
The Measure of a Marketer's WorthThe Measure of a Marketer's Worth
The Measure of a Marketer's Worth
Ā 
The Worst Lessons Marketing Ever Taught Content
The Worst Lessons Marketing Ever Taught ContentThe Worst Lessons Marketing Ever Taught Content
The Worst Lessons Marketing Ever Taught Content
Ā 
The Remarkable SEO Power of Republishing
The Remarkable SEO Power of RepublishingThe Remarkable SEO Power of Republishing
The Remarkable SEO Power of Republishing
Ā 
SEO: Crawl Budget Optimierung & Onsite SEO
SEO: Crawl Budget Optimierung & Onsite SEOSEO: Crawl Budget Optimierung & Onsite SEO
SEO: Crawl Budget Optimierung & Onsite SEO
Ā 
SEO: SERPs im Wandel - SMX Munich 2017
SEO: SERPs im Wandel - SMX Munich 2017SEO: SERPs im Wandel - SMX Munich 2017
SEO: SERPs im Wandel - SMX Munich 2017
Ā 
Crawl Budget Optimization - SMX MĆ¼nchen 2016
Crawl Budget Optimization - SMX MĆ¼nchen 2016Crawl Budget Optimization - SMX MĆ¼nchen 2016
Crawl Budget Optimization - SMX MĆ¼nchen 2016
Ā 
Burman GSC Gurgaon
Burman GSC GurgaonBurman GSC Gurgaon
Burman GSC Gurgaon
Ā 
Frontend talk for backenders
Frontend talk for backendersFrontend talk for backenders
Frontend talk for backenders
Ā 
Alloy Cybersecurity
Alloy CybersecurityAlloy Cybersecurity
Alloy Cybersecurity
Ā 
Isomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master ClassIsomorphic JavaScript: #DevBeat Master Class
Isomorphic JavaScript: #DevBeat Master Class
Ā 
SEO in a Two Algorithm World
SEO in a Two Algorithm WorldSEO in a Two Algorithm World
SEO in a Two Algorithm World
Ā 
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
Cut The Cruft - Everett Sizemore - MozTalk Denver - 2016
Ā 

Similar to Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model

Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
Evaluate deep q learning for sequential targeted marketing with 10-fold cross...Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
Evaluate deep q learning for sequential targeted marketing with 10-fold cross...Jian Wu
Ā 
Workshop: Your first machine learning project
Workshop: Your first machine learning projectWorkshop: Your first machine learning project
Workshop: Your first machine learning projectAlex Austin
Ā 
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...Ognjen Scekic
Ā 
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...Edureka!
Ā 
Tech meetup Data Driven - Codemotion
Tech meetup Data Driven - Codemotion Tech meetup Data Driven - Codemotion
Tech meetup Data Driven - Codemotion antimo musone
Ā 
Predire il futuro con Machine Learning & Big Data
Predire il futuro con Machine Learning & Big DataPredire il futuro con Machine Learning & Big Data
Predire il futuro con Machine Learning & Big DataData Driven Innovation
Ā 
Phase 2 of Predicting Payment default on Vehicle Loan EMI
Phase 2 of Predicting Payment default on Vehicle Loan EMIPhase 2 of Predicting Payment default on Vehicle Loan EMI
Phase 2 of Predicting Payment default on Vehicle Loan EMIVikas Virani
Ā 
Recommending job ads to people
Recommending job ads to peopleRecommending job ads to people
Recommending job ads to peopleFabian Abel
Ā 
Visual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningVisual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningBenjamin Bengfort
Ā 
AlgorithmsModelsNov13.pptx
AlgorithmsModelsNov13.pptxAlgorithmsModelsNov13.pptx
AlgorithmsModelsNov13.pptxPerumalPitchandi
Ā 
Fast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA HardwareFast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA HardwareTigerGraph
Ā 
Developing Web-scale Machine Learning at LinkedIn - From Soup to Nuts
Developing Web-scale Machine Learning at LinkedIn - From Soup to NutsDeveloping Web-scale Machine Learning at LinkedIn - From Soup to Nuts
Developing Web-scale Machine Learning at LinkedIn - From Soup to NutsKun Liu
Ā 
Empirical Model of Supervised Learning Approach for Opinion Mining
Empirical Model of Supervised Learning Approach for Opinion MiningEmpirical Model of Supervised Learning Approach for Opinion Mining
Empirical Model of Supervised Learning Approach for Opinion MiningIRJET Journal
Ā 
Telecom Churn Analysis
Telecom Churn AnalysisTelecom Churn Analysis
Telecom Churn AnalysisVasudev pendyala
Ā 
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...Simplilearn
Ā 
Keynote at IWLS 2017
Keynote at IWLS 2017Keynote at IWLS 2017
Keynote at IWLS 2017Manish Pandey
Ā 
Envelopment Analysis In Economics
Envelopment Analysis In EconomicsEnvelopment Analysis In Economics
Envelopment Analysis In EconomicsAmber Rodriguez
Ā 
Web Page Ranking using Machine Learning
Web Page Ranking using Machine LearningWeb Page Ranking using Machine Learning
Web Page Ranking using Machine LearningPradip Rahul
Ā 

Similar to Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model (20)

Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
Evaluate deep q learning for sequential targeted marketing with 10-fold cross...Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
Evaluate deep q learning for sequential targeted marketing with 10-fold cross...
Ā 
Workshop: Your first machine learning project
Workshop: Your first machine learning projectWorkshop: Your first machine learning project
Workshop: Your first machine learning project
Ā 
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
Simulation-Based Modeling and Evaluation of Incentive Schemes in Crowdsourcin...
Ā 
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Data Analytics For Beginners | Introduction To Data Analytics | Data Analytic...
Ā 
Tech meetup Data Driven - Codemotion
Tech meetup Data Driven - Codemotion Tech meetup Data Driven - Codemotion
Tech meetup Data Driven - Codemotion
Ā 
Predire il futuro con Machine Learning & Big Data
Predire il futuro con Machine Learning & Big DataPredire il futuro con Machine Learning & Big Data
Predire il futuro con Machine Learning & Big Data
Ā 
Data Science Machine
Data Science Machine Data Science Machine
Data Science Machine
Ā 
Kaggle KDD Cup Report
Kaggle KDD Cup ReportKaggle KDD Cup Report
Kaggle KDD Cup Report
Ā 
Phase 2 of Predicting Payment default on Vehicle Loan EMI
Phase 2 of Predicting Payment default on Vehicle Loan EMIPhase 2 of Predicting Payment default on Vehicle Loan EMI
Phase 2 of Predicting Payment default on Vehicle Loan EMI
Ā 
Recommending job ads to people
Recommending job ads to peopleRecommending job ads to people
Recommending job ads to people
Ā 
Visual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learningVisual diagnostics for more effective machine learning
Visual diagnostics for more effective machine learning
Ā 
AlgorithmsModelsNov13.pptx
AlgorithmsModelsNov13.pptxAlgorithmsModelsNov13.pptx
AlgorithmsModelsNov13.pptx
Ā 
Fast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA HardwareFast Parallel Similarity Calculations with FPGA Hardware
Fast Parallel Similarity Calculations with FPGA Hardware
Ā 
Developing Web-scale Machine Learning at LinkedIn - From Soup to Nuts
Developing Web-scale Machine Learning at LinkedIn - From Soup to NutsDeveloping Web-scale Machine Learning at LinkedIn - From Soup to Nuts
Developing Web-scale Machine Learning at LinkedIn - From Soup to Nuts
Ā 
Empirical Model of Supervised Learning Approach for Opinion Mining
Empirical Model of Supervised Learning Approach for Opinion MiningEmpirical Model of Supervised Learning Approach for Opinion Mining
Empirical Model of Supervised Learning Approach for Opinion Mining
Ā 
Telecom Churn Analysis
Telecom Churn AnalysisTelecom Churn Analysis
Telecom Churn Analysis
Ā 
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Machine Learning Algorithms | Machine Learning Tutorial | Data Science Algori...
Ā 
Keynote at IWLS 2017
Keynote at IWLS 2017Keynote at IWLS 2017
Keynote at IWLS 2017
Ā 
Envelopment Analysis In Economics
Envelopment Analysis In EconomicsEnvelopment Analysis In Economics
Envelopment Analysis In Economics
Ā 
Web Page Ranking using Machine Learning
Web Page Ranking using Machine LearningWeb Page Ranking using Machine Learning
Web Page Ranking using Machine Learning
Ā 

More from Rand Fishkin

SparkToro Beta Sneak Peek
SparkToro Beta Sneak PeekSparkToro Beta Sneak Peek
SparkToro Beta Sneak PeekRand Fishkin
Ā 
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & MoreThe Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & MoreRand Fishkin
Ā 
Building Influence in 2019
Building Influence in 2019Building Influence in 2019
Building Influence in 2019Rand Fishkin
Ā 
Influence Not Influencers
Influence Not InfluencersInfluence Not Influencers
Influence Not InfluencersRand Fishkin
Ā 
The Next Era of Web Marketing: 2019 & Beyond
The Next Era of Web Marketing: 2019 & BeyondThe Next Era of Web Marketing: 2019 & Beyond
The Next Era of Web Marketing: 2019 & BeyondRand Fishkin
Ā 
How to Kick Butt with Your Email Outreach
How to Kick Butt with Your Email OutreachHow to Kick Butt with Your Email Outreach
How to Kick Butt with Your Email OutreachRand Fishkin
Ā 
The Big 7 Startup Marketing Mistakes
The Big 7 Startup Marketing MistakesThe Big 7 Startup Marketing Mistakes
The Big 7 Startup Marketing MistakesRand Fishkin
Ā 
Why Your Saas Marketing Sucks (and how to fix it)
Why Your Saas Marketing Sucks (and how to fix it)Why Your Saas Marketing Sucks (and how to fix it)
Why Your Saas Marketing Sucks (and how to fix it)Rand Fishkin
Ā 
SEO on the SERPs - Brighton SEO Closing Talk
SEO on the SERPs - Brighton SEO Closing TalkSEO on the SERPs - Brighton SEO Closing Talk
SEO on the SERPs - Brighton SEO Closing TalkRand Fishkin
Ā 
7 Lessons That Would Have Made Me a Better Entrepreneur
7 Lessons That Would Have Made Me a Better Entrepreneur7 Lessons That Would Have Made Me a Better Entrepreneur
7 Lessons That Would Have Made Me a Better EntrepreneurRand Fishkin
Ā 
The Search & SEO World in 2018
The Search & SEO World in 2018The Search & SEO World in 2018
The Search & SEO World in 2018Rand Fishkin
Ā 
SEO in 2017/18
SEO in 2017/18SEO in 2017/18
SEO in 2017/18Rand Fishkin
Ā 
Why Startups Suck at Marketing
Why Startups Suck at MarketingWhy Startups Suck at Marketing
Why Startups Suck at MarketingRand Fishkin
Ā 
The Invisible Giant that Mucks Up Our Marketing
The Invisible Giant that Mucks Up Our MarketingThe Invisible Giant that Mucks Up Our Marketing
The Invisible Giant that Mucks Up Our MarketingRand Fishkin
Ā 
B2B SEO in 2017
B2B SEO in 2017B2B SEO in 2017
B2B SEO in 2017Rand Fishkin
Ā 
How to Survive Google's Trojan Horsing of the Web
How to Survive Google's Trojan Horsing of the WebHow to Survive Google's Trojan Horsing of the Web
How to Survive Google's Trojan Horsing of the WebRand Fishkin
Ā 
What Startup Execs Need to Know About SEO in 2017
What Startup Execs Need to Know About SEO in 2017What Startup Execs Need to Know About SEO in 2017
What Startup Execs Need to Know About SEO in 2017Rand Fishkin
Ā 
Inside Google's Numbers in 2017
Inside Google's Numbers in 2017Inside Google's Numbers in 2017
Inside Google's Numbers in 2017Rand Fishkin
Ā 
Why We Can't Do SEO WIthout CRO
Why We Can't Do SEO WIthout CROWhy We Can't Do SEO WIthout CRO
Why We Can't Do SEO WIthout CRORand Fishkin
Ā 
The Digital Marketer's Framework
The Digital Marketer's FrameworkThe Digital Marketer's Framework
The Digital Marketer's FrameworkRand Fishkin
Ā 

More from Rand Fishkin (20)

SparkToro Beta Sneak Peek
SparkToro Beta Sneak PeekSparkToro Beta Sneak Peek
SparkToro Beta Sneak Peek
Ā 
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & MoreThe Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
The Healthcare Search Landscape in 2019: SEO, Content Marketing, & More
Ā 
Building Influence in 2019
Building Influence in 2019Building Influence in 2019
Building Influence in 2019
Ā 
Influence Not Influencers
Influence Not InfluencersInfluence Not Influencers
Influence Not Influencers
Ā 
The Next Era of Web Marketing: 2019 & Beyond
The Next Era of Web Marketing: 2019 & BeyondThe Next Era of Web Marketing: 2019 & Beyond
The Next Era of Web Marketing: 2019 & Beyond
Ā 
How to Kick Butt with Your Email Outreach
How to Kick Butt with Your Email OutreachHow to Kick Butt with Your Email Outreach
How to Kick Butt with Your Email Outreach
Ā 
The Big 7 Startup Marketing Mistakes
The Big 7 Startup Marketing MistakesThe Big 7 Startup Marketing Mistakes
The Big 7 Startup Marketing Mistakes
Ā 
Why Your Saas Marketing Sucks (and how to fix it)
Why Your Saas Marketing Sucks (and how to fix it)Why Your Saas Marketing Sucks (and how to fix it)
Why Your Saas Marketing Sucks (and how to fix it)
Ā 
SEO on the SERPs - Brighton SEO Closing Talk
SEO on the SERPs - Brighton SEO Closing TalkSEO on the SERPs - Brighton SEO Closing Talk
SEO on the SERPs - Brighton SEO Closing Talk
Ā 
7 Lessons That Would Have Made Me a Better Entrepreneur
7 Lessons That Would Have Made Me a Better Entrepreneur7 Lessons That Would Have Made Me a Better Entrepreneur
7 Lessons That Would Have Made Me a Better Entrepreneur
Ā 
The Search & SEO World in 2018
The Search & SEO World in 2018The Search & SEO World in 2018
The Search & SEO World in 2018
Ā 
SEO in 2017/18
SEO in 2017/18SEO in 2017/18
SEO in 2017/18
Ā 
Why Startups Suck at Marketing
Why Startups Suck at MarketingWhy Startups Suck at Marketing
Why Startups Suck at Marketing
Ā 
The Invisible Giant that Mucks Up Our Marketing
The Invisible Giant that Mucks Up Our MarketingThe Invisible Giant that Mucks Up Our Marketing
The Invisible Giant that Mucks Up Our Marketing
Ā 
B2B SEO in 2017
B2B SEO in 2017B2B SEO in 2017
B2B SEO in 2017
Ā 
How to Survive Google's Trojan Horsing of the Web
How to Survive Google's Trojan Horsing of the WebHow to Survive Google's Trojan Horsing of the Web
How to Survive Google's Trojan Horsing of the Web
Ā 
What Startup Execs Need to Know About SEO in 2017
What Startup Execs Need to Know About SEO in 2017What Startup Execs Need to Know About SEO in 2017
What Startup Execs Need to Know About SEO in 2017
Ā 
Inside Google's Numbers in 2017
Inside Google's Numbers in 2017Inside Google's Numbers in 2017
Inside Google's Numbers in 2017
Ā 
Why We Can't Do SEO WIthout CRO
Why We Can't Do SEO WIthout CROWhy We Can't Do SEO WIthout CRO
Why We Can't Do SEO WIthout CRO
Ā 
The Digital Marketer's Framework
The Digital Marketer's FrameworkThe Digital Marketer's Framework
The Digital Marketer's Framework
Ā 

Recently uploaded

Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”soniya singh
Ā 
Pune Airport ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready...
Pune Airport ( Call Girls ) Pune  6297143586  Hot Model With Sexy Bhabi Ready...Pune Airport ( Call Girls ) Pune  6297143586  Hot Model With Sexy Bhabi Ready...
Pune Airport ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready...tanu pandey
Ā 
š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...
š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...
š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...Neha Pandey
Ā 
Moving Beyond Twitter/X and Facebook - Social Media for local news providers
Moving Beyond Twitter/X and Facebook - Social Media for local news providersMoving Beyond Twitter/X and Facebook - Social Media for local news providers
Moving Beyond Twitter/X and Facebook - Social Media for local news providersDamian Radcliffe
Ā 
VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...
VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...
VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...aditipandeya
Ā 
Networking in the Penumbra presented by Geoff Huston at NZNOG
Networking in the Penumbra presented by Geoff Huston at NZNOGNetworking in the Penumbra presented by Geoff Huston at NZNOG
Networking in the Penumbra presented by Geoff Huston at NZNOGAPNIC
Ā 
Hot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night Stand
Hot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night StandHot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night Stand
Hot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night Standkumarajju5765
Ā 
Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”soniya singh
Ā 
VIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call Girl
VIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call GirlVIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call Girl
VIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call Girladitipandeya
Ā 
SEO Growth Program-Digital optimization Specialist
SEO Growth Program-Digital optimization SpecialistSEO Growth Program-Digital optimization Specialist
SEO Growth Program-Digital optimization SpecialistKHM Anwar
Ā 
10.pdfMature Call girls in Dubai +971563133746 Dubai Call girls
10.pdfMature Call girls in Dubai +971563133746 Dubai Call girls10.pdfMature Call girls in Dubai +971563133746 Dubai Call girls
10.pdfMature Call girls in Dubai +971563133746 Dubai Call girlsstephieert
Ā 
Enjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort Service
Enjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort ServiceEnjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort Service
Enjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort ServiceDelhi Call girls
Ā 
Top Rated Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...
Top Rated  Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...Top Rated  Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...
Top Rated Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...Call Girls in Nagpur High Profile
Ā 
horny (9316020077 ) Goa Call Girls Service by VIP Call Girls in Goa
horny (9316020077 ) Goa  Call Girls Service by VIP Call Girls in Goahorny (9316020077 ) Goa  Call Girls Service by VIP Call Girls in Goa
horny (9316020077 ) Goa Call Girls Service by VIP Call Girls in Goasexy call girls service in goa
Ā 
Lucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRL
Lucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRLLucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRL
Lucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRLimonikaupta
Ā 
Radiant Call girls in Dubai O56338O268 Dubai Call girls
Radiant Call girls in Dubai O56338O268 Dubai Call girlsRadiant Call girls in Dubai O56338O268 Dubai Call girls
Radiant Call girls in Dubai O56338O268 Dubai Call girlsstephieert
Ā 

Recently uploaded (20)

Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Saket Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Ā 
Pune Airport ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready...
Pune Airport ( Call Girls ) Pune  6297143586  Hot Model With Sexy Bhabi Ready...Pune Airport ( Call Girls ) Pune  6297143586  Hot Model With Sexy Bhabi Ready...
Pune Airport ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready...
Ā 
š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...
š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...
š“€¤Call On 7877925207 š“€¤ Ahmedguda Call Girls Hot Model With Sexy Bhabi Ready Fo...
Ā 
Moving Beyond Twitter/X and Facebook - Social Media for local news providers
Moving Beyond Twitter/X and Facebook - Social Media for local news providersMoving Beyond Twitter/X and Facebook - Social Media for local news providers
Moving Beyond Twitter/X and Facebook - Social Media for local news providers
Ā 
Rohini Sector 6 Call Girls Delhi 9999965857 @Sabina Saikh No Advance
Rohini Sector 6 Call Girls Delhi 9999965857 @Sabina Saikh No AdvanceRohini Sector 6 Call Girls Delhi 9999965857 @Sabina Saikh No Advance
Rohini Sector 6 Call Girls Delhi 9999965857 @Sabina Saikh No Advance
Ā 
VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...
VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...
VIP 7001035870 Find & Meet Hyderabad Call Girls Dilsukhnagar high-profile Cal...
Ā 
Networking in the Penumbra presented by Geoff Huston at NZNOG
Networking in the Penumbra presented by Geoff Huston at NZNOGNetworking in the Penumbra presented by Geoff Huston at NZNOG
Networking in the Penumbra presented by Geoff Huston at NZNOG
Ā 
Hot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night Stand
Hot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night StandHot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night Stand
Hot Call Girls |Delhi |Hauz Khas ā˜Ž 9711199171 Book Your One night Stand
Ā 
Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Call Girls In Pratap Nagar Delhi šŸ’ÆCall Us šŸ”8264348440šŸ”
Ā 
Dwarka Sector 26 Call Girls | Delhi | 9999965857 šŸ«¦ Vanshika Verma More Our Se...
Dwarka Sector 26 Call Girls | Delhi | 9999965857 šŸ«¦ Vanshika Verma More Our Se...Dwarka Sector 26 Call Girls | Delhi | 9999965857 šŸ«¦ Vanshika Verma More Our Se...
Dwarka Sector 26 Call Girls | Delhi | 9999965857 šŸ«¦ Vanshika Verma More Our Se...
Ā 
VIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call Girl
VIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call GirlVIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call Girl
VIP 7001035870 Find & Meet Hyderabad Call Girls LB Nagar high-profile Call Girl
Ā 
SEO Growth Program-Digital optimization Specialist
SEO Growth Program-Digital optimization SpecialistSEO Growth Program-Digital optimization Specialist
SEO Growth Program-Digital optimization Specialist
Ā 
Rohini Sector 22 Call Girls Delhi 9999965857 @Sabina Saikh No Advance
Rohini Sector 22 Call Girls Delhi 9999965857 @Sabina Saikh No AdvanceRohini Sector 22 Call Girls Delhi 9999965857 @Sabina Saikh No Advance
Rohini Sector 22 Call Girls Delhi 9999965857 @Sabina Saikh No Advance
Ā 
10.pdfMature Call girls in Dubai +971563133746 Dubai Call girls
10.pdfMature Call girls in Dubai +971563133746 Dubai Call girls10.pdfMature Call girls in Dubai +971563133746 Dubai Call girls
10.pdfMature Call girls in Dubai +971563133746 Dubai Call girls
Ā 
Enjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort Service
Enjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort ServiceEnjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort Service
Enjoy Nightāš”Call Girls Dlf City Phase 3 Gurgaon >ą¼’8448380779 Escort Service
Ā 
Top Rated Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...
Top Rated  Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...Top Rated  Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...
Top Rated Pune Call Girls Daund āŸŸ 6297143586 āŸŸ Call Me For Genuine Sex Servi...
Ā 
horny (9316020077 ) Goa Call Girls Service by VIP Call Girls in Goa
horny (9316020077 ) Goa  Call Girls Service by VIP Call Girls in Goahorny (9316020077 ) Goa  Call Girls Service by VIP Call Girls in Goa
horny (9316020077 ) Goa Call Girls Service by VIP Call Girls in Goa
Ā 
Lucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRL
Lucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRLLucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRL
Lucknow ā¤CALL GIRL 88759*99948 ā¤CALL GIRLS IN Lucknow ESCORT SERVICEā¤CALL GIRL
Ā 
Call Girls In Noida šŸ“± 9999965857 šŸ¤© Delhi šŸ«¦ HOT AND SEXY VVIP šŸŽ SERVICE
Call Girls In Noida šŸ“±  9999965857  šŸ¤© Delhi šŸ«¦ HOT AND SEXY VVIP šŸŽ SERVICECall Girls In Noida šŸ“±  9999965857  šŸ¤© Delhi šŸ«¦ HOT AND SEXY VVIP šŸŽ SERVICE
Call Girls In Noida šŸ“± 9999965857 šŸ¤© Delhi šŸ«¦ HOT AND SEXY VVIP šŸŽ SERVICE
Ā 
Radiant Call girls in Dubai O56338O268 Dubai Call girls
Radiant Call girls in Dubai O56338O268 Dubai Call girlsRadiant Call girls in Dubai O56338O268 Dubai Call girls
Radiant Call girls in Dubai O56338O268 Dubai Call girls
Ā 

Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model

  • 1. Background Motivation Model & Metric Experimental Setup Results Summary Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model Aleksandr ChuklinĀ¶,Ā§ Maarten de RijkeĀ§ chuklin@google.com derijke@uva.nl Ā¶Google Research Europe Ā§University of Amsterdam ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 1
  • 3. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Main problem Combining relevance of individual SERP items (Rk) into a whole-page metric. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 3
  • 4. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples document 3 document 4 document 1 document 2 document 5 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 5. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: P@N = 1 N N k=1 Rk document 3 document 4 document 1 document 2 document 5 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 6. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: P@N = 1 N N k=1 Rk Discounted Cumulative Gain (DCG): DCG@N = N k=1 1 log2 (1 + k) Ā· Rk document 3 document 4 document 1 document 2 document 5 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 7. Background Motivation Model & Metric Experimental Setup Results Summary Search Engine Result Page (SERP) Evaluation Examples Precision at N: P@N = 1 N N k=1 Rk Discounted Cumulative Gain (DCG): DCG@N = N k=1 1 log2 (1 + k) Ā· Rk Model-Based Metrics (Chuklin et al. 2013): Utility@N = N k=1 P(Ck = 1) Ā· Rk document 3 document 4 document 1 document 2 document 5 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 4
  • 8. Background Motivation Model & Metric Experimental Setup Results Summary Main Goal of This Paper Better measure for SERP utility Namely, improve this (Chuklin et al. 2013): N k=1 P(Ck = 1) Ā· Rk ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 5
  • 10. Background Motivation Model & Metric Experimental Setup Results Summary Complex Heterogeneous SERPs ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 7
  • 11. Background Motivation Model & Metric Experimental Setup Results Summary Motivation 1: Non-Trivial Attention Patterns 4 ement 9 1 3 5 6 7 8 4 2 (c) Mouse Data data. The session sequence for this data would be Image credits: F. Diaz, R.W. White, G. Buscher, and D. Liebling. Robust models of mouse movement on dynamic web search results pages. In CIKM, 2013. ACM Press ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 8
  • 12. Background Motivation Model & Metric Experimental Setup Results Summary Motivation 2: Satisfaction Without Clicks High direct page utility (measured by DCG or ERR) leads to higher abandonment rate (SERPs with no clicks) direct page utility Image credits: from A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In WWW, 2012. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 9
  • 13. Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 10
  • 14. Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics existing models mostly do not model non-trivial user attention patterns ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 10
  • 15. Background Motivation Model & Metric Experimental Setup Results Summary Problems of Existing Models and Evaluation Metrics existing models mostly do not model non-trivial user attention patterns existing models do not use explicit user satisfaction data ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 10
  • 17. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP šœ‘& šø& š¶& šœ‘) šø) š¶) šœ‘* šø* š¶* š‘† ā€¦ Utility ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 12
  • 18. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP šœ‘& šø& š¶& šœ‘) šø) š¶) šœ‘* šø* š¶* š‘† ā€¦ Utility ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 13
  • 19. Background Motivation Model & Metric Experimental Setup Results Summary Click Model Examination assumption: click happens only when an item was examined and attractive: P(Ck = 1) = P(Ek = 1) Ā· P(Ck = 1 | Ek = 1) ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 14
  • 20. Background Motivation Model & Metric Experimental Setup Results Summary Click Model Examination assumption: click happens only when an item was examined and attractive: P(Ck = 1) = P(Ek = 1) Ā· P(Ck = 1 | Ek = 1) N.B. Here we assume that P(Ck = 1 | Ek = 1) = Ī±(Rk) where Rk comes from the raters and Ī± is a logistic function. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 14
  • 21. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP šœ‘& šø& š¶& šœ‘) šø) š¶) šœ‘* šø* š¶* š‘† ā€¦ Utility ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 15
  • 22. Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P(Ek = 1) = Īµ(Ļ•k), where Ļ•k is a vector of features for SERP item k. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 16
  • 23. Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P(Ek = 1) = Īµ(Ļ•k), where Ļ•k is a vector of features for SERP item k. Feature group Features # of features rank user-perceived rank of the SERP item (can be diļ¬€erent from k) 1 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 16
  • 24. Background Motivation Model & Metric Experimental Setup Results Summary Attention (Examination) Model Logistic regression model: P(Ek = 1) = Īµ(Ļ•k), where Ļ•k is a vector of features for SERP item k. Feature group Features # of features rank user-perceived rank of the SERP item (can be diļ¬€erent from k) 1 CSS classes SERP item type (Web, News, Weather, Currency, Knowledge Panel, etc.) 10 geometry oļ¬€set from the top, ļ¬rst or second col- umn (binary), width (w), height (h), w Ɨ h 5 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 16
  • 25. Background Motivation Model & Metric Experimental Setup Results Summary Clicks + Attention + Satisfaction (CAS) Model SERP šœ‘& šø& š¶& šœ‘) šø) š¶) šœ‘* šø* š¶* š‘† ā€¦ Utility ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 17
  • 26. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 27. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 28. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; P(S = 1) = Ļƒ(Ļ„0 + U) = ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 29. Background Motivation Model & Metric Experimental Setup Results Summary Satisfaction Model in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; P(S = 1) = Ļƒ(Ļ„0 + U) = Ļƒ Ļ„0 + k P(Ek = 1)ud (Dk) + k P(Ck = 1)ur (Rk) where Dk and Rk are ratings assigned by the raters for direct snippet relevance and result relevance respectively. ud and ur are linear functions of rating histograms. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 18
  • 30. Background Motivation Model & Metric Experimental Setup Results Summary The CAS Metric Utility that determines the satisfaction probability: U = k P(Ek = 1)ud (Dk) + k P(Ck = 1)ur (Rk) ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 19
  • 31. Background Motivation Model & Metric Experimental Setup Results Summary The CAS Metric Utility that determines the satisfaction probability: U = k P(Ek = 1)ud (Dk) NEW + k P(Ck = 1)ur (Rk) Chuklin et al. 2013 has an additional term ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 19
  • 32. Background Motivation Model & Metric Experimental Setup Results Summary The CAS Metric Utility that determines the satisfaction probability: U = k P(Ek = 1)ud (Dk) NEW + k P(Ck = 1)ur (Rk) Chuklin et al. 2013 has an additional term trained on mousing and satisfaction (in addition to clicks) ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 19
  • 34. Background Motivation Model & Metric Experimental Setup Results Summary Dataset 199 queries with explicit unambiguous feedback (satisļ¬ed / not satisļ¬ed); ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 21
  • 35. Background Motivation Model & Metric Experimental Setup Results Summary Dataset 199 queries with explicit unambiguous feedback (satisļ¬ed / not satisļ¬ed); 1,739 rated results direct snippet relevance (D) result relevance (R) ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 21
  • 36. Background Motivation Model & Metric Experimental Setup Results Summary Baselines and CAS Model Variants UBM model that agrees well with online team-draft experimental outcomes; PBM position-based model, a robust model with fewer parameters than UBM; random model that predicts click and satisfaction with ļ¬xed probabilities (learned from the data). uUBM from Chuklin et al. 2013. Similar to UBM, but parameters are trained on a diļ¬€erent and much bigger dataset. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 22
  • 37. Background Motivation Model & Metric Experimental Setup Results Summary Baselines and CAS Model Variants UBM model that agrees well with online team-draft experimental outcomes; PBM position-based model, a robust model with fewer parameters than UBM; random model that predicts click and satisfaction with ļ¬xed probabilities (learned from the data). uUBM from Chuklin et al. 2013. Similar to UBM, but parameters are trained on a diļ¬€erent and much bigger dataset. CASnod is a stripped-down version that does not use (D) labels; CASnosat is a version of the CAS model that does not include the satisfaction term while optimizing the model; CASnoreg is a version of the CAS model that does not use regularization while training. All other models were trained with L2-regularization. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 22
  • 39. Background Motivation Model & Metric Experimental Setup Results Summary Is the New Metric Really New? Correlation Between Metrics Table: Correlation between metrics measured by average Pearsonā€™s correlation coeļ¬ƒcient. CASnosat CASnoreg CAS UBM PBM DCG uUBM CASnod 0.593 0.564 0.633 0.470 0.487 0.546 0.441 CASnosat 0.664 0.715 0.707 0.668 0.735 0.684 CASnoreg 0.974 0.363 0.379 0.417 0.341 CAS 0.377 0.394 0.440 0.360 UBM 0.814 0.972 0.882 PBM 0.906 0.965 DCG 0.943 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 24
  • 40. Background Motivation Model & Metric Experimental Setup Results Summary Is the New Metric Measuring the Right Thing? Metric Correlation with True Satisfaction CASnod CASnosat CASnoreg CAS UBM PBM random DCG uUBM 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Pearson correlation coeļ¬ƒcient between diļ¬€erent model-based metrics and the user-reported satisfaction. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 25
  • 41. Background Motivation Model & Metric Experimental Setup Results Summary Bonus Point Log-Likelihood of Click Prediction CASnod CASnosat CASnoreg CAS UBM PBM random uUBM 4.5 4.0 3.5 3.0 2.5 2.0 1.5 Log-likelihood of the click data. Note that uUBM was trained on a totally diļ¬€erent dataset. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 26
  • 43. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 44. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 45. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. The CAS metric is quite diļ¬€erent from the previously used metrics, making it an interesting addition to TREC. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 46. Background Motivation Model & Metric Experimental Setup Results Summary Summary A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. The CAS metric is quite diļ¬€erent from the previously used metrics, making it an interesting addition to TREC. When used as a model, CAS consistently predicts user satisfaction with a relatively small penalty in click prediction. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 28
  • 47. Background Motivation Model & Metric Experimental Setup Results Summary Acknowledgments All content represents the opinion of the authors which is not necessarily shared or endorsed by their respective employers and/or sponsors. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 29
  • 48.
  • 49. Background Motivation Model & Metric Experimental Setup Results Summary Evaluating the User Model Log-Likelihood of Satisfaction Prediction CASnod CASnosat CASnoreg CAS UBM PBM random uUBM 0.8 0.7 0.6 0.5 0.4 0.3 0.2 Log-likelihood of the satisfaction prediction. Some models have log-likelihood below āˆ’0.8, hence there are no boxes for them. ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 31
  • 50. Background Motivation Model & Metric Experimental Setup Results Summary Analyzing the Attention Features CASrank is the model that only uses the rank to predict attention; CASnogeom only uses the rank and SERP item type information and does not use geometry; CASnoclass does not use the CSS class features (SERP item type). Pearson correlation with satisfaction CASrank CASnogeom CASnoclass CASnod CAS 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Log-likelihood of clicks / satisfaction CASrank CASnogeom CASnoclass CASnod CAS 2.5 2.4 2.3 2.2 2.1 2.0 1.9 1.8 1.7 CASrank CASnogeom CASnoclass CASnod CAS 0.65 0.60 0.55 0.50 0.45 0.40 0.35 0.30 0.25 0.20 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 32
  • 51. Background Motivation Model & Metric Experimental Setup Results Summary Heterogeneous SERPs 12% of the SERPs in our data are heterogeneous and our metric does well for them. Table: Pearson correlation between utility of heterogeneous SERP and user-reported satisfaction. CAS UBM PBM random DCG uUBM 0.60 0.38 -0.05 -0.39 0.24 -0.08 CASrank CASnogeom CASclass CASnod CASnosat CASnoreg 0.15 -0.04 0.27 -0.04 0.48 0.67 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 33
  • 52. Background Motivation Model & Metric Experimental Setup Results Summary Spammers Some raters were ļ¬ltered out as spammers, but there was still some natural disagreement: Table: Filtered out workers and agreement scores for remaining workers. % of workers % of ratings Cohenā€™s Krippendorfā€™s label removed removed kappa alpha (D) 32% 27% 0.339 0.144 (R) 41% 29% 0.348 0.117 ACā€“MdR Incorporating Clicks, Attention and Satisfaction. . . 34